Monitoring Apache HTTPd with New Relic

When figuring out what’s wrong with a sites performance it’s important to get facts about every aspect and component involved with that site. Apache is quite often used in front of java applications and it’s the app server for php applications. Bad apache configuration can make a site seem sluggish even when there are plenty of other resources available so it’s important to see what’s going on here. From the first screen of Apache HTTPd plugin you’ll get a nice overview of all your monitored apaches.

New Relic Plugins Apache Listing

Drilling down to a single server overview shows request velocity, cpu load, busy/idle workers and even bytes sent over time.

New Relic Plugins Apache Overview

Going to throughtput shows throughtput details over time.

New Relic Plugins Apache Throughtput

Workers section shows you what is happening with the workers. If you have a lot of busy workers you can see in what state their are and that might provide some insight to what is going on.

New Relic Plugins Apache Workers

Installing Apache HTTPd agent for New Relic

1) I’m using the MeetMe New Relic Agent to monitor Apache HTTPd. It’s written in python and we’ll need to install pip. The following is using the Ubuntu python-pip package. You can find alternative install methods from pip docs.

apt-get install python-pip libyaml-dev python-dev

2) Next use pip to install newrelic-plugin-agent. When I ran it I got some errors but it still worked.

pip install newrelic-plugin-agent

3) Next we’ll need to create the configuration file for the agent. You can start by copying /opt/newrelic-plugin-agent/newrelic-plugin-agent.cfg or just use what I have posted below. The first thing you need is to set license_key. You can find your license key from your account settings page on The second one is to add apache_httpd configuration. You can add multiple httpds to monitor.

cat - <<EOF>> /etc/newrelic/newrelic-plugin-agent.cfg
%YAML 1.2
 license_key: YOUR_LICENSE_KEY
 wake_interval: 60
 #newrelic_api_timeout: 10
 #proxy: http://localhost:8080

  - name: localhost
    scheme: http
    host: localhost
    verify_ssl_cert: true
    port: 80
    path: /server-status

 user: newrelic
 pidfile: /var/run/newrelic/

     format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -15s %(threadName)-10s %(name) -45s %(funcName) -25s L%(lineno)-6d: %(message)s'
     class : logging.handlers.RotatingFileHandler
     formatter: verbose
     filename: /var/log/newrelic/newrelic-plugin-agent.log
     maxBytes: 10485760
     backupCount: 3
     level: INFO
     propagate: True
     handlers: [console, file]
     level: ERROR
     propagate: True
     handlers: [console, file]

4) Make sure you have enabled mod_status in your apache and you’ve allowed access from the host your agent is running if it’s not running in the same host as your apache.

5) Then we need to add a init script for the newrelic-plugin-agent. There’s one under /opt/newrelic-plugin-agent but for me it was incomplete file so I just downloaded the one from github.

mv newrelic-plugin-agent.deb /etc/init.d/newrelic-plugin-agent
chmod 755 /etc/init.d/newrelic-plugin-agent
update-rc.d newrelic-plugin-agent defaults

Now you can start the newrelic-plugin-agent with

service newrelic-plugin-agent start

Now in few minutes you should see your Apache HTTPd server(s) listed under Plugins HTTPd on

Monitoring Nginx with New Relic

Apache HTTPd has always been my goto httpd, reverse proxy and load balancer but lately I’ve grown more interested in Nginx. It’s very high performance and lightweight not to mention easy to configure. Of course with my currently a single Nginx I wanted to see how could I hook it up my monitoring. Turns our there’s a New Relic agent directly from Nginx.

New Relic Plugins Nginx Listing

From the overview you can see the number of active and idle connections as well as the request rate.

New Relic Plugins Nginx Overview

From connections you’ll even more connection details. With very little connections and requests my graphs are currently slightly boring. In addition to connection details you can find more details about requests, upstreams, servers and cache.

New Relic Plugins Nginx Connections
Installing New Relic Monitoring Agent for Nginx

1) First you need to add the Ubuntu package repository for Nginx. If you’ve done this already when you installed Nginx you can skip to next step. If you are not using Ubuntu 14.04 like I am you can find the other Linux packages from Nginx website.

apt-key add nginx_signing.key

cat - <<-EOF >> /etc/apt/sources.list.d/nginx.list
deb trusty nginx
deb-src trusty nginx

apt-get update

2) Next you need to install the Nginx New Relic Agent

apt-get install nginx-nr-agent

3) Next you’ll need to edit the agent configuration file in /etc/nginx-nr-agent/nginx-nr-agent.ini. You need to add your license key which you can find from your account settings page on


Additionally you need to add a new source which points to your Nginx status url.


4) You’ll need to add a server block to Nginx for the status. Since I had very simple configuration in my Nginx I just added the following to /etc/nginx/sites-enabled/default

server {
   server_name localhost;

   location = /nginx_stub_status {
     stub_status on;
     deny all;

5) Last thing you need to do is reload Nginx and start the Nginx New Relic Agent.

service nginx reload
service nginx-nr-agent start

Now in few minutes you should start seeing your Nginx server listed under Plugins Nginx on

Monitoring MariaDB / MySQL with New Relic

In order to size your system correctly you need metrics and you need those also from your database. Not only is it important to know what’s happening in your database for capacity planning it’s also invaluable information when something goes wrong. If you have a database related performance problem in your code seeing how the fix effects the database can help you see if you actually solved the problem. Also looking at database statistics you might be able to spot a issue before it becomes a serious problem.

New Relic has a wonderful plugin framework and there’s a ton of ready made plugins and also SDK and API for things it doesn’t already support. MySQL plugin is one of those ready made plugins and it provides all the key information you’ll need. The MySQL plugin page quickly shows what’s going on all monitored databases.
New Relic Plugins MySQL

When you drill down to a individual database server the overview shows the SQL volume and how it’s split between reads and writes. More key metrics are displayed under Key Utilizations. You’ll also find database connections and network traffic on this page.

New Relic Plugins MySQL Overview

Going further down to Query analysis you’ll see in more details about the queries.

New Relic Plugins MySQL Query Analysis

If you are using InnoDB there’s a separate page to show key metrics from InnoDB.

New Relic Plugins MySQL InnoDB Metrics

Installing MySQL / MariaDB Monitoring

1) MySQL plugin can easily be installed with New Relic platform installer. So the first thing you need to do is install the platform installer. You’ll need your New Relic license key which you can find from account settings on Once you have that you can install it with following one liner which is for 64bit Debian and Ubuntu.


2) Next go to the newly created newrelic-npi directory and run install. You’ll want to answer yes to all the questions and when prompted to configure the plugin grab the configuration from the next step.

./npi install nrmysql

3) If you skipped configuration you can configure the plugin afterward too. You can find the configuration file under newrelic-npi from plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/config/plugin.json. Below is a sample configuration for MariaDB (works for MySQL) running on localhost and we’ll be creating a separate user newrelic with password somepassword which the plugin will use to gather data. You can connect to multiple databases with the same agent. I usually install this agent on the same server my nagios is running on.

 "agents": [
     "name" : "MariaDB on localhost",
     "host" : "localhost",
     "metrics" : "status,newrelic,buffer_pool_stats,innodb_status,innodb_mutex",
     "user" : "newrelic",
     "passwd" : "somepassword"

4) Now we need to create a user in the database and grant some rights to it.

cat - <<EOF | mysql -u root -p
CREATE USER newrelic@'%' IDENTIFIED BY 'somepassword';

5) Last thing is to start the service but before we do make sure you have Java installed as this agent is written in Java. If you don’t have Java installed check my unattended Java install script. Otherwise you can proceed to start the service that should have been created during npi install if you answered all the questions correctly.

service newrelic_plugin_com.newrelic.plugins.mysql.instance start

Now it may take few minutes before you see your server under Plugins MySQL in If it doesn’t check the log under plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/logs/ for hints and make sure the agent actually started.

Monitoring Ubuntu / Debian Server with New Relic

With New Relic Server Monitoring you’ll see all the important information about your system with just one glance. This is a essential tool for troubleshooting performance issues and also seeing that your system is properly sized. Sometimes poor application performance has nothing to do with the application but rather the system it’s running. If the system is not correctly sized you might be running out of memory, cpu or the bottle neck could be disk io. Without proper monitoring it is very hard to pinpoint the cause.

The servers listing gives a nice overview of all servers and you an easily see if there’s any issues.

New Relic Servers

New Relic Servers

When looking at a specific server you’ll see a history of it’s CPU and memory usage as well as load average and network I/O. If you have any APM enabled applications installed you’ll see a overview of their response times, throughput and error rate. You’ll also see some of the top processes running on the server.

New Relic Servers Overview

New Relic Server Overview

When you drill down to processes listing you’ll quickly see the top memory and cpu consumers. You can also look at the history of individual processes.

New Relic Server Processes

New Relic Server Processes

Installing New Relic Server Monitoring on a Ubuntu / Debian Server

1) Add an apt source for New Relic.

cat - <<-EOF >> /etc/apt/sources.list.d/newrelic.list 
# newrelic repository list 
deb newrelic non-free

2) You’ll need to get the key for New Relic repository and then update apt sources. After that you can install newrelic-sysmond.

apt-key adv --recv-keys --keyserver 0xB31B29E5548C16BF
apt-get update
apt-get install newrelic-sysmond

3) Next you’ll need to tell it your license key so that it reports the data to your account. You can find your license key from your account settings page on You can either edit the configuration file or you can set the license like shown below:

/usr/sbin/nrsysmond-config --set license_key=YOUR_LICENSE_KEY

4) Finally once everything is configured you can start the system monitor daemon.

service newrelic-sysmond start

Now in few minutes you should start seeing your server listed under Servers on

Monitoring with New Relic

New Relic is a wonderful software analytics suite that is 100% SaaS. I love it because it’s so easy to setup compared to Nagios, MRTG and other on premise software. Also their Lite edition is free with 24 hour data retention and for 30 days you’ll get to see the power of the Pro version. I still use nagios for my main monitoring and create some key graphs with MRTG but the data junkie in me loves all the data New Relic gathers and shows in nice graphs.

New Relic has seven parts or products as they call them. Those are APM, Insights, Mobile, Browser, Synthetics, Servers and Plugins. I have myself used only APM, Browser, Server and Plugins which are included in the free Lite edition.


APM is the application monitoring part. It focuses on providing information about the application itself. The Lite edition shows you response times, throughput, web transaction information. It’s basically a low impact profiler. With the Pro subscription you get much deeper analysis of time spent on executing SQL, JVM statistics etc.

New Relic APM Java Overview


The browser provides insights on client side performance. Even though your application might respond quickly the users perceived performance could be poor because of network performance or even how the page is rendering on the browser.


Servers as the name suggests provides performance information about the actual server your applications are running on.

New Relic Servers Overview


There’s a ton of plugins to provide monitoring capabilities to systems not otherwise supported by New Relic and with it’s SDKs and API you can build your own plugins. Some of the plugins I have used are for MySQL/MariaDB, Nginx and Apache.

New Relic Plugins MySQL Overview


Mobile is APM for mobile applications.


Synthetics allows you to test your application from around the world. It can check business critical user flows and interaction to make sure your site available and functioning from around the world.


Insights is a paid feature that combines business metrics with performance data. It can combine data from APM, Browser, Mobile and Synthetics for deeper analysis and segmentation and filtering.

Since I recently installed bunch of new servers and I had to refresh my memory on how I installed and configured each of the agents so I decided to write a series of articles on each of them. Here’s a list of topics I’m going to publish and as I publish them I’ll link the topic to the article. These topics will cover APM with Java and PHP, Servers and Plugins for MySQL, Nginx and Apache

Unattended Java install on Ubuntu 14.04

I like to to automate all the tasks I do often and of the things many of my virtual servers need is Java JDK. Unfortunately the Oracle JDK is not available as debian package but there’s a way to make it. This is where WebUpd8 Team PPA comes in as they provide installer for java6, java7 and java8.

Below is the script I use to install it unattended. You can download it also from github gist. If you want Java6 then just use oracle-java6-installer and for Java 8 oracle-java8-installer. This also works other ubuntu versions just substitute trusty with the code of your ubuntu release like precise for Ubuntu 12.04. Hope you find this useful.

cat - <<-EOF >> /etc/apt/sources.list.d/webupd8team-java.list
# webupd8team repository list 
deb trusty main
# deb-src trusty main

apt-key adv --recv-keys --keyserver 0xEEA14886

echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections

apt-get update
apt-get install oracle-java7-installer

New Site New Focus

It’s been a while since I last posted anything and the site has been quite stale and out dated. So I thought I’d update it to a more modern responsive site that looks good on both desktop and mobile. As I wanted to focus more on actual writing and less on building a site I moved back to WordPress which is still the best blogging platform. My only concern is now security as php apps are much more prone to being hacked than Java ones.

With this new focus I’m also going to write about much more broad topics than just Liferay. As Liferay is moving it’s core towards OSGi I’ve studied it a lot and grown to love it although it doesn’t come without it’s own challenges especially for some moving to it from Java EE. As some of you may know I’ve been running and administering my own Linux servers for more than 15 years so some of the new topics will be about virtualization, containers, monitoring etc. I still read a lot of books so when I read one that deserves a mention I’ll write a book review. I hope you enjoy what’s to come.

As I built this new site I only migrated posts that had been read in the past year so if you run into something that’s no longer available do email me I still have them saved. Also all the old urls should be automatically redirected to the new ones.

Monitoring and Graphing Liferay with MRTG

MRTG (The Multi Router Traffic Grapher) is usually used to monitor SNMP enabled network devices and draw graphs of how much traffic has passed through each interface. It can also be used to graph any two values (in/out) and I use it for graphing cpu usage, loadavg, iowait, used memory, disk space and temperature sensor values that I can read through SNMP. Liferay however doesn’t support SNMP so I developed a perl script that can read JMX MBean values using JMX4Perl and Jolokia. I’m going to assume you have JMX4Perl and Jolokia setup already the way I describe it in my earlier post: Monitoring Liferay with Nagios, Jolokia and JMX4Perl. You should also note that MRTG won’t send you any alerts so it’s a good idea to setup Nagios to do just that.

Now you might wonder why would you need MRTG if you already have Nagios. Nagios operates on the present value although there’s a add-on nagios grapher that can create graphs like MRTG does but I like MRTG more because you can see all the graphs on one page. Being able to see a full overview of the system is very important when trying to identify performance problems. This is also why you want to get more information out of the application, Liferay in this case. My script will help in reading connection pool and thread pool utilization as well as heap usage. Those are essential when doing troubleshooting.

First you’ll need to install and setup MRTG. I’m not going to go into details on that because it dependes on your system and the internet is full of guides to do it. Once you have it done you’ll need to download my script which is available in my github repository. For rest of this post I’m going to assume it’s located in /usr/local/bin/ but it’s up to you where you put it. Just adjust the script path accordingly.

Monitoring c3p0 connection pool

Getting the values for c3p0 is a little bit tricky because it will generate a unique mbean name based on the identity token it generates for the connection pool every time the server is started. Because of this my script assumes you only have one c3p0 connection pool if you have multiple pool you’ll need to add additional logic in to the mrtg-jmx4perl to find the correct mbean. Notice that this is the case if you configure Liferay to use connection pool from instead of using a JNDI resource. We can read the mbean for c3p0 by using mbean name as “com.mchange.v2.c3p0:type=*,*” and the attributes we are most interested are numConnectionsAllUsers and numBusyConnectionsAllUsers. Below is a sample mrtg configuration snippet.

Target[dbpool]: `/usr/local/bin/ --server=servername --mbean="com.mchange.v2.c3p0:type=*,*" --attribute="numConnectionsAllUsers numBusyConnectionsAllUsers"`
MaxBytes[dbpool]: 20
Title[dbpool]: DB Pool
PageTop[dbpool]: <h1>DB Pool</h1>
WithPeak[dbpool]: dwmy
Unscaled[dbpool]: dwmy
Options[dbpool]: growright,unknaszero,nopercent,gauge
YLegend[dbpool]: Connections
LegendI[dbpool]: Connections
LegendO[dbpool]: Busy Connections
Legend1[dbpool]: Connections
Legend2[dbpool]: Busy Connections
Legend3[dbpool]: Peak Connections
Legend4[dbpool]: Peak Busy Connections

Here’s a daily graph from one of my Liferay portal servers.

mrtg - db pool connections

Monitoring Tomcat AJP Thread Pool

This one is pretty easy because the the mbean name is static but it does vary depending on do tomcat version and connector you are using. In Tomcat 7 with native library the name for ajp thread pool is Catalina:type=ThreadPool,name=”ajp-apr-8009″. Without native library it would be ajp-bio-8009. In tomcat 6 my ajp pool mbean name is Catalina:type=ThreadPool,name=jk-8009. Notice the lack of double quotes in the name. You can easily check the name using jconsole. So for this one the config looks like:

Target[ajp-threadpool]: `/usr/local/bin/ --server=servername --mbean="Catalina:type=ThreadPool,name=\"ajp-apr-8009\"" --attribute="currentThreadCount currentThreadsBusy"`
MaxBytes[ajp-threadpool]: 50
Title[ajp-threadpool]: AJP Thread Pool
PageTop[ajp-threadpool]: <h1>AJP Thread Pool</h1>
WithPeak[ajp-threadpool]: dwmy
#Unscaled[ajp-threadpool]: dwmy
Options[ajp-threadpool]:  growright,unknaszero,nopercent,gauge
YLegend[ajp-threadpool]: Threads
LegendI[ajp-threadpool]: Threads
LegendO[ajp-threadpool]: Busy Threads
Legend1[ajp-threadpool]: Threads
Legend2[ajp-threadpool]: Busy Threads
Legend3[ajp-threadpool]: Peak Threads
Legend4[ajp-threadpool]: Peak Busy Threads

Here’s a daily graph of a thread pool.

mrtg - thread pool

Monitoring Heap Usage

The last one we are going to monitor is Java Heap usage. It can be read from java.lang:type=Memory using attribute HeapMemoryUsage and path used. Now we are reading only one value.

Target[heap]: `/usr/local/bin/ --server=servername --mbean="java.lang:type=Memory" --attribute="HeapMemoryUsage" --path="used"`
MaxBytes[heap]: 1296302080
Title[heap]: Heap
PageTop[heap]: <h1>Heap</h1>
WithPeak[heap]: dwmy
Unscaled[heap]: dwmy
Options[heap]:  growright,unknaszero,nopercent,gauge,noo
YLegend[heap]: bytes
kilo[heap]: 1024
LegendI[heap]: Used
Legend1[heap]: Used
Legend3[heap]: Peak Used

Here’s a daily graph of heap memory usage.

mrtg - heap usage

You can download the full sample-mrtg.cfg from github.

That’s how easy it is to monitor and graph Liferay or pretty much any Java webapp using MRTG. You could easily use this to monitor ehcache utilization or anything else that’s accessible via JMX.

Installing MariaDB on Ubuntu

I’ve been using MariaDB for some time now and it’s perfect replacement for MySQL especially with the latest news onOracle’s move to hinder MySQL developer community despite it’s promise to EU. Now is a perfect time to ditch MySQL and move to something that’s backed by the original authors of MySQL and that something is MariaDB.

1. First pick your Ubuntu version repository mirror close to you from MariaDB downloads page. Once you’ve picked up your mirror then add them to /etc/apt/source.list.d/mariadb.list. I’m still running 10.04 so here’s what I put in my mariadb.list:

# MariaDB repository list - created 2012-07-04 18:04 UTC
deb lucid main
deb-src lucid main

2. Next you’ll need to import the signing key

sudo apt-key adv --recv-keys --keyserver 0xcbcb082a1bb943db

3. Update

aptitude update

4. Install

aptitude install mariadb-server-5.5

Now you have MariaDB 5.5 installed and you can configure it exactly like you would configure MySQL.

Tips for securing your Liferay installation

There’s few security related things that I see people constantly doing wrong. The very first thing is assuming Liferay bundle with it’s default settings is secure for production. It is far from secure. Don’t get me wrong this doesn’t mean that Liferay isn’t secure it just means that shouldn’t deploy Liferay with it’s default settings and assume it’s secure. So let’s go over some things you should consider.

Default admin user

Everyone knows the default admin user and some attacks have taken advantage knowing this user and even it’s userid which is predictable. What I would suggest is not only to change the email address and screenname of this user but actually create a completely new admin user and remove this user.

Portal instance web id

The default company web id is and it goes without saying you should change it unless you are actually deploying You can do this simply by setting property in your This must be done before you start your portal and let it generate the database.

Encryption algorithm

By default Liferay is configured to use 56bit DES encryption algorithm. I believe this legacy is due to US encryption export laws. The problem with 56bit DES is that it was cracked back in the 90s and is not considered secure encryption anymore. Liferay encrypts certaing things with this like your password in Remember Me cookie. If someone get’s a hold of that cookie they can crack your password. I would recommend using at least 128bit AES. To do that you’ll just need to set following properties before starting your portal against a clean database.


Password hashing

Recently there has been a lot of sites that have their passwords being compromised because they weren’t using salt with their password hash. Liferay by default uses SHA-1 to hash your password. That hash is a one way algorithm that doesn’t allow reversing the password from the hash but if someone gets a hold of your password hash it’s still possible to crack with brute force or by using rainbow tables. Rainbow tables are precalculated hashes that allow very easily and fast find unsalted passwords. Salt is something we add to the password before hashing it and it’s preferrable unique of each password so that even if two users have the same password their hash is different. Liferay comes with SSHA algorithm that salts the password before calculating the SHA-1 hash from it. You can enable it by setting following in your


Unused SSO hooks

The default Liferay bundle comes with all SSO hooks included even thought they are not all enabled it’s a good idea to remove any hooks your are not using. There’s a property called auto.login.hooks and you should remove all hooks your are not using. Also remember to disable their associated filters.

Unused Remote APIs

Liferay has several different remote APIs such as JSON, JSONWS, Web service, Atom, WebDAV, Sharepoint etc. You should go through them and disable everything your site is not using. Please note that some Liferay builtin portlets rely on some of these APIs. All the APIs are accessible under /api URL.

Mixed HTTP and HTTPS

Everyone should by now know about Firesheep a firefox extension that allows an attacker to sniff a wifi network they are connected to and hijack a users authenticated session. This attack can compromise any website that doesn’t use all authenticated traffic over https. If you use https for just part of the site and your users can access rest of the site as authenticated user over http then your are vulnerable to Firesheep attack. This is especially bad with Liferay if you are using the default encryption and you use Remember me functionality because then the attacker could even compromise your password and use it login to any system where you use the same password. I’m sad to say that even is vulnerable to this attack.

Shared Secrets

Don’t forget to change any shared secrets. The auth.token.shared.secret has a default value that you want to change so that no-one can try to exploit it. This tip came from Jelmer who has found many vulnerabilities in Liferay.  Another one you don’t want to overlook is auth.mac.shared.key which has default value of blank. That one is relevant if you auth.mac.allowset to true.

This is not an exhaustive list but this should make your Liferay installation much more secure than it is by default. For more tips on what to configure before going to production check out Liferay whitepapers. You should especially read the deployment checklist. If you can think of any other things that should be on this list comment them or tweet them to me @koivimik

Update: Added shared secret tip from Jelmer