Creating a custom Nginx build for Ubuntu/Debian

I’ve been using the RTCamp Ubuntu package for Nginx because it had ngx_cache_purge and ngx_pagespeed modules builtin. The problem with is that it’s still stuck on Nginx 1.8 version which doesn’t support HTTP/2 so I had to figure out how to do my own build based on the latest Nginx mainline version. Continue reading


Moving a project from one git repository to another while retaining it’s history

I recently had to move a project from one git repository to another existing repository under different source tree and I wanted to retain the history of each file. This is rather easy once you know how to do it but you can easily jack up things if you don’t know. So I wanted to write clear instructions for the next time I have to do this and hopefully it also helps someone else. Continue reading


Setting up https with Let’s Encrypt on Nginx

Let’s Encrypt is an awesome free, automated and open way of protecting your site with https. As you may have noticed this site is using Let’s Encrypt certificate and I’ve started rolling it out to all my other sites too. With free https certificate there’s really no excuse not to use https only. In fact if you want to take advantage of HTTP/2 you’ll need https since no one currently supports it unencrypted even though the spec doesn’t mandate it. Continue reading

Monitoring Apache HTTPd with New Relic

When figuring out what’s wrong with a sites performance it’s important to get facts about every aspect and component involved with that site. Apache is quite often used in front of java applications and it’s the app server for php applications. Bad apache configuration can make a site seem sluggish even when there are plenty of other resources available so it’s important to see what’s going on here. From the first screen of Apache HTTPd plugin you’ll get a nice overview of all your monitored apaches.

New Relic Plugins Apache Listing

Drilling down to a single server overview shows request velocity, cpu load, busy/idle workers and even bytes sent over time.

New Relic Plugins Apache Overview

Going to throughtput shows throughtput details over time.

New Relic Plugins Apache Throughtput

Workers section shows you what is happening with the workers. If you have a lot of busy workers you can see in what state their are and that might provide some insight to what is going on.

New Relic Plugins Apache Workers

Installing Apache HTTPd agent for New Relic

1) I’m using the MeetMe New Relic Agent to monitor Apache HTTPd. It’s written in python and we’ll need to install pip. The following is using the Ubuntu python-pip package. You can find alternative install methods from pip docs.

apt-get install python-pip libyaml-dev python-dev

2) Next use pip to install newrelic-plugin-agent. When I ran it I got some errors but it still worked.

pip install newrelic-plugin-agent

3) Next we’ll need to create the configuration file for the agent. You can start by copying /opt/newrelic-plugin-agent/newrelic-plugin-agent.cfg or just use what I have posted below. The first thing you need is to set license_key. You can find your license key from your account settings page on The second one is to add apache_httpd configuration. You can add multiple httpds to monitor.

cat - <<EOF>> /etc/newrelic/newrelic-plugin-agent.cfg
%YAML 1.2
 license_key: YOUR_LICENSE_KEY
 wake_interval: 60
 #newrelic_api_timeout: 10
 #proxy: http://localhost:8080

  - name: localhost
    scheme: http
    host: localhost
    verify_ssl_cert: true
    port: 80
    path: /server-status

 user: newrelic
 pidfile: /var/run/newrelic/

     format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -15s %(threadName)-10s %(name) -45s %(funcName) -25s L%(lineno)-6d: %(message)s'
     class : logging.handlers.RotatingFileHandler
     formatter: verbose
     filename: /var/log/newrelic/newrelic-plugin-agent.log
     maxBytes: 10485760
     backupCount: 3
     level: INFO
     propagate: True
     handlers: [console, file]
     level: ERROR
     propagate: True
     handlers: [console, file]

4) Make sure you have enabled mod_status in your apache and you’ve allowed access from the host your agent is running if it’s not running in the same host as your apache.

5) Then we need to add a init script for the newrelic-plugin-agent. There’s one under /opt/newrelic-plugin-agent but for me it was incomplete file so I just downloaded the one from github.

mv newrelic-plugin-agent.deb /etc/init.d/newrelic-plugin-agent
chmod 755 /etc/init.d/newrelic-plugin-agent
update-rc.d newrelic-plugin-agent defaults

Now you can start the newrelic-plugin-agent with

service newrelic-plugin-agent start

Now in few minutes you should see your Apache HTTPd server(s) listed under Plugins HTTPd on

Monitoring Nginx with New Relic

Apache HTTPd has always been my goto httpd, reverse proxy and load balancer but lately I’ve grown more interested in Nginx. It’s very high performance and lightweight not to mention easy to configure. Of course with my currently a single Nginx I wanted to see how could I hook it up my monitoring. Turns our there’s a New Relic agent directly from Nginx.

New Relic Plugins Nginx Listing

From the overview you can see the number of active and idle connections as well as the request rate.

New Relic Plugins Nginx Overview

From connections you’ll even more connection details. With very little connections and requests my graphs are currently slightly boring. In addition to connection details you can find more details about requests, upstreams, servers and cache.

New Relic Plugins Nginx Connections
Installing New Relic Monitoring Agent for Nginx

1) First you need to add the Ubuntu package repository for Nginx. If you’ve done this already when you installed Nginx you can skip to next step. If you are not using Ubuntu 14.04 like I am you can find the other Linux packages from Nginx website.

apt-key add nginx_signing.key

cat - <<-EOF >> /etc/apt/sources.list.d/nginx.list
deb trusty nginx
deb-src trusty nginx

apt-get update

2) Next you need to install the Nginx New Relic Agent

apt-get install nginx-nr-agent

3) Next you’ll need to edit the agent configuration file in /etc/nginx-nr-agent/nginx-nr-agent.ini. You need to add your license key which you can find from your account settings page on


Additionally you need to add a new source which points to your Nginx status url.


4) You’ll need to add a server block to Nginx for the status. Since I had very simple configuration in my Nginx I just added the following to /etc/nginx/sites-enabled/default

server {
   server_name localhost;

   location = /nginx_stub_status {
     stub_status on;
     deny all;

5) Last thing you need to do is reload Nginx and start the Nginx New Relic Agent.

service nginx reload
service nginx-nr-agent start

Now in few minutes you should start seeing your Nginx server listed under Plugins Nginx on

Monitoring MariaDB / MySQL with New Relic

In order to size your system correctly you need metrics and you need those also from your database. Not only is it important to know what’s happening in your database for capacity planning it’s also invaluable information when something goes wrong. If you have a database related performance problem in your code seeing how the fix effects the database can help you see if you actually solved the problem. Also looking at database statistics you might be able to spot a issue before it becomes a serious problem.

New Relic has a wonderful plugin framework and there’s a ton of ready made plugins and also SDK and API for things it doesn’t already support. MySQL plugin is one of those ready made plugins and it provides all the key information you’ll need. The MySQL plugin page quickly shows what’s going on all monitored databases.
New Relic Plugins MySQL

When you drill down to a individual database server the overview shows the SQL volume and how it’s split between reads and writes. More key metrics are displayed under Key Utilizations. You’ll also find database connections and network traffic on this page.

New Relic Plugins MySQL Overview

Going further down to Query analysis you’ll see in more details about the queries.

New Relic Plugins MySQL Query Analysis

If you are using InnoDB there’s a separate page to show key metrics from InnoDB.

New Relic Plugins MySQL InnoDB Metrics

Installing MySQL / MariaDB Monitoring

1) MySQL plugin can easily be installed with New Relic platform installer. So the first thing you need to do is install the platform installer. You’ll need your New Relic license key which you can find from account settings on Once you have that you can install it with following one liner which is for 64bit Debian and Ubuntu.


2) Next go to the newly created newrelic-npi directory and run install. You’ll want to answer yes to all the questions and when prompted to configure the plugin grab the configuration from the next step.

./npi install nrmysql

3) If you skipped configuration you can configure the plugin afterward too. You can find the configuration file under newrelic-npi from plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/config/plugin.json. Below is a sample configuration for MariaDB (works for MySQL) running on localhost and we’ll be creating a separate user newrelic with password somepassword which the plugin will use to gather data. You can connect to multiple databases with the same agent. I usually install this agent on the same server my nagios is running on.

 "agents": [
     "name" : "MariaDB on localhost",
     "host" : "localhost",
     "metrics" : "status,newrelic,buffer_pool_stats,innodb_status,innodb_mutex",
     "user" : "newrelic",
     "passwd" : "somepassword"

4) Now we need to create a user in the database and grant some rights to it.

cat - <<EOF | mysql -u root -p
CREATE USER newrelic@'%' IDENTIFIED BY 'somepassword';

5) Last thing is to start the service but before we do make sure you have Java installed as this agent is written in Java. If you don’t have Java installed check my unattended Java install script. Otherwise you can proceed to start the service that should have been created during npi install if you answered all the questions correctly.

service newrelic_plugin_com.newrelic.plugins.mysql.instance start

Now it may take few minutes before you see your server under Plugins MySQL in If it doesn’t check the log under plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/logs/ for hints and make sure the agent actually started.

Monitoring Ubuntu / Debian Server with New Relic

With New Relic Server Monitoring you’ll see all the important information about your system with just one glance. This is a essential tool for troubleshooting performance issues and also seeing that your system is properly sized. Sometimes poor application performance has nothing to do with the application but rather the system it’s running. If the system is not correctly sized you might be running out of memory, cpu or the bottle neck could be disk io. Without proper monitoring it is very hard to pinpoint the cause.

The servers listing gives a nice overview of all servers and you an easily see if there’s any issues.

New Relic Servers

New Relic Servers

When looking at a specific server you’ll see a history of it’s CPU and memory usage as well as load average and network I/O. If you have any APM enabled applications installed you’ll see a overview of their response times, throughput and error rate. You’ll also see some of the top processes running on the server.

New Relic Servers Overview

New Relic Server Overview

When you drill down to processes listing you’ll quickly see the top memory and cpu consumers. You can also look at the history of individual processes.

New Relic Server Processes

New Relic Server Processes

Installing New Relic Server Monitoring on a Ubuntu / Debian Server

1) Add an apt source for New Relic.

cat - <<-EOF >> /etc/apt/sources.list.d/newrelic.list 
# newrelic repository list 
deb newrelic non-free

2) You’ll need to get the key for New Relic repository and then update apt sources. After that you can install newrelic-sysmond.

apt-key adv --recv-keys --keyserver 0xB31B29E5548C16BF
apt-get update
apt-get install newrelic-sysmond

3) Next you’ll need to tell it your license key so that it reports the data to your account. You can find your license key from your account settings page on You can either edit the configuration file or you can set the license like shown below:

/usr/sbin/nrsysmond-config --set license_key=YOUR_LICENSE_KEY

4) Finally once everything is configured you can start the system monitor daemon.

service newrelic-sysmond start

Now in few minutes you should start seeing your server listed under Servers on

Monitoring with New Relic

New Relic is a wonderful software analytics suite that is 100% SaaS. I love it because it’s so easy to setup compared to Nagios, MRTG and other on premise software. Also their Lite edition is free with 24 hour data retention and for 30 days you’ll get to see the power of the Pro version. I still use nagios for my main monitoring and create some key graphs with MRTG but the data junkie in me loves all the data New Relic gathers and shows in nice graphs.

New Relic has seven parts or products as they call them. Those are APM, Insights, Mobile, Browser, Synthetics, Servers and Plugins. I have myself used only APM, Browser, Server and Plugins which are included in the free Lite edition.


APM is the application monitoring part. It focuses on providing information about the application itself. The Lite edition shows you response times, throughput, web transaction information. It’s basically a low impact profiler. With the Pro subscription you get much deeper analysis of time spent on executing SQL, JVM statistics etc.

New Relic APM Java Overview


The browser provides insights on client side performance. Even though your application might respond quickly the users perceived performance could be poor because of network performance or even how the page is rendering on the browser.


Servers as the name suggests provides performance information about the actual server your applications are running on.

New Relic Servers Overview


There’s a ton of plugins to provide monitoring capabilities to systems not otherwise supported by New Relic and with it’s SDKs and API you can build your own plugins. Some of the plugins I have used are for MySQL/MariaDB, Nginx and Apache.

New Relic Plugins MySQL Overview


Mobile is APM for mobile applications.


Synthetics allows you to test your application from around the world. It can check business critical user flows and interaction to make sure your site available and functioning from around the world.


Insights is a paid feature that combines business metrics with performance data. It can combine data from APM, Browser, Mobile and Synthetics for deeper analysis and segmentation and filtering.

Since I recently installed bunch of new servers and I had to refresh my memory on how I installed and configured each of the agents so I decided to write a series of articles on each of them. Here’s a list of topics I’m going to publish and as I publish them I’ll link the topic to the article. These topics will cover APM with Java and PHP, Servers and Plugins for MySQL, Nginx and Apache

Unattended Java install on Ubuntu 14.04

I like to to automate all the tasks I do often and of the things many of my virtual servers need is Java JDK. Unfortunately the Oracle JDK is not available as debian package but there’s a way to make it. This is where WebUpd8 Team PPA comes in as they provide installer for java6, java7 and java8.

Below is the script I use to install it unattended. You can download it also from github gist. If you want Java6 then just use oracle-java6-installer and for Java 8 oracle-java8-installer. This also works other ubuntu versions just substitute trusty with the code of your ubuntu release like precise for Ubuntu 12.04. Hope you find this useful.

cat - <<-EOF >> /etc/apt/sources.list.d/webupd8team-java.list
# webupd8team repository list 
deb trusty main
# deb-src trusty main

apt-key adv --recv-keys --keyserver 0xEEA14886

echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections

apt-get update
apt-get install oracle-java7-installer

New Site New Focus

It’s been a while since I last posted anything and the site has been quite stale and out dated. So I thought I’d update it to a more modern responsive site that looks good on both desktop and mobile. As I wanted to focus more on actual writing and less on building a site I moved back to WordPress which is still the best blogging platform. My only concern is now security as php apps are much more prone to being hacked than Java ones.

With this new focus I’m also going to write about much more broad topics than just Liferay. As Liferay is moving it’s core towards OSGi I’ve studied it a lot and grown to love it although it doesn’t come without it’s own challenges especially for some moving to it from Java EE. As some of you may know I’ve been running and administering my own Linux servers for more than 15 years so some of the new topics will be about virtualization, containers, monitoring etc. I still read a lot of books so when I read one that deserves a mention I’ll write a book review. I hope you enjoy what’s to come.

As I built this new site I only migrated posts that had been read in the past year so if you run into something that’s no longer available do email me I still have them saved. Also all the old urls should be automatically redirected to the new ones.