New Site New Focus

It’s been a while since I last posted anything and the site has been quite stale and out dated. So I thought I’d update it to a more modern responsive site that looks good on both desktop and mobile. As I wanted to focus more on actual writing and less on building a site I moved back to WordPress which is still the best blogging platform. My only concern is now security as php apps are much more prone to being hacked than Java ones.

With this new focus I’m also going to write about much more broad topics than just Liferay. As Liferay is moving it’s core towards OSGi I’ve studied it a lot and grown to love it although it doesn’t come without it’s own challenges especially for some moving to it from Java EE. As some of you may know I’ve been running and administering my own Linux servers for more than 15 years so some of the new topics will be about virtualization, containers, monitoring etc. I still read a lot of books so when I read one that deserves a mention I’ll write a book review. I hope you enjoy what’s to come.

As I built this new site I only migrated posts that had been read in the past year so if you run into something that’s no longer available do email me I still have them saved. Also all the old urls should be automatically redirected to the new ones.

Monitoring and Graphing Liferay with MRTG

MRTG (The Multi Router Traffic Grapher) is usually used to monitor SNMP enabled network devices and draw graphs of how much traffic has passed through each interface. It can also be used to graph any two values (in/out) and I use it for graphing cpu usage, loadavg, iowait, used memory, disk space and temperature sensor values that I can read through SNMP. Liferay however doesn’t support SNMP so I developed a perl script that can read JMX MBean values using JMX4Perl and Jolokia. I’m going to assume you have JMX4Perl and Jolokia setup already the way I describe it in my earlier post: Monitoring Liferay with Nagios, Jolokia and JMX4Perl. You should also note that MRTG won’t send you any alerts so it’s a good idea to setup Nagios to do just that.

Now you might wonder why would you need MRTG if you already have Nagios. Nagios operates on the present value although there’s a add-on nagios grapher that can create graphs like MRTG does but I like MRTG more because you can see all the graphs on one page. Being able to see a full overview of the system is very important when trying to identify performance problems. This is also why you want to get more information out of the application, Liferay in this case. My script will help in reading connection pool and thread pool utilization as well as heap usage. Those are essential when doing troubleshooting.

First you’ll need to install and setup MRTG. I’m not going to go into details on that because it dependes on your system and the internet is full of guides to do it. Once you have it done you’ll need to download my mrtg-jmx4perl.pl script which is available in my github repository. For rest of this post I’m going to assume it’s located in /usr/local/bin/mrtg-jmx4perl.pl but it’s up to you where you put it. Just adjust the script path accordingly.

Monitoring c3p0 connection pool

Getting the values for c3p0 is a little bit tricky because it will generate a unique mbean name based on the identity token it generates for the connection pool every time the server is started. Because of this my script assumes you only have one c3p0 connection pool if you have multiple pool you’ll need to add additional logic in to the mrtg-jmx4perl to find the correct mbean. Notice that this is the case if you configure Liferay to use connection pool from portal-ext.properties instead of using a JNDI resource. We can read the mbean for c3p0 by using mbean name as “com.mchange.v2.c3p0:type=*,*” and the attributes we are most interested are numConnectionsAllUsers and numBusyConnectionsAllUsers. Below is a sample mrtg configuration snippet.

Target[dbpool]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="com.mchange.v2.c3p0:type=*,*" --attribute="numConnectionsAllUsers numBusyConnectionsAllUsers"`
MaxBytes[dbpool]: 20
Title[dbpool]: DB Pool
PageTop[dbpool]: <h1>DB Pool</h1>
WithPeak[dbpool]: dwmy
Unscaled[dbpool]: dwmy
Options[dbpool]: growright,unknaszero,nopercent,gauge
YLegend[dbpool]: Connections
ShortLegend[dbpool]: 
LegendI[dbpool]: Connections
LegendO[dbpool]: Busy Connections
Legend1[dbpool]: Connections
Legend2[dbpool]: Busy Connections
Legend3[dbpool]: Peak Connections
Legend4[dbpool]: Peak Busy Connections

Here’s a daily graph from one of my Liferay portal servers.

mrtg - db pool connections

Monitoring Tomcat AJP Thread Pool

This one is pretty easy because the the mbean name is static but it does vary depending on do tomcat version and connector you are using. In Tomcat 7 with native library the name for ajp thread pool is Catalina:type=ThreadPool,name=”ajp-apr-8009″. Without native library it would be ajp-bio-8009. In tomcat 6 my ajp pool mbean name is Catalina:type=ThreadPool,name=jk-8009. Notice the lack of double quotes in the name. You can easily check the name using jconsole. So for this one the config looks like:

Target[ajp-threadpool]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="Catalina:type=ThreadPool,name=\"ajp-apr-8009\"" --attribute="currentThreadCount currentThreadsBusy"`
MaxBytes[ajp-threadpool]: 50
Title[ajp-threadpool]: AJP Thread Pool
PageTop[ajp-threadpool]: <h1>AJP Thread Pool</h1>
WithPeak[ajp-threadpool]: dwmy
#Unscaled[ajp-threadpool]: dwmy
Options[ajp-threadpool]:  growright,unknaszero,nopercent,gauge
YLegend[ajp-threadpool]: Threads
ShortLegend[ajp-threadpool]: 
LegendI[ajp-threadpool]: Threads
LegendO[ajp-threadpool]: Busy Threads
Legend1[ajp-threadpool]: Threads
Legend2[ajp-threadpool]: Busy Threads
Legend3[ajp-threadpool]: Peak Threads
Legend4[ajp-threadpool]: Peak Busy Threads

Here’s a daily graph of a thread pool.

mrtg - thread pool

Monitoring Heap Usage

The last one we are going to monitor is Java Heap usage. It can be read from java.lang:type=Memory using attribute HeapMemoryUsage and path used. Now we are reading only one value.

Target[heap]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="java.lang:type=Memory" --attribute="HeapMemoryUsage" --path="used"`
MaxBytes[heap]: 1296302080
Title[heap]: Heap
PageTop[heap]: <h1>Heap</h1>
WithPeak[heap]: dwmy
Unscaled[heap]: dwmy
Options[heap]:  growright,unknaszero,nopercent,gauge,noo
YLegend[heap]: bytes
ShortLegend[heap]: 
kilo[heap]: 1024
LegendI[heap]: Used
Legend1[heap]: Used
Legend3[heap]: Peak Used

Here’s a daily graph of heap memory usage.

mrtg - heap usage

You can download the full sample-mrtg.cfg from github.

That’s how easy it is to monitor and graph Liferay or pretty much any Java webapp using MRTG. You could easily use this to monitor ehcache utilization or anything else that’s accessible via JMX.

Installing MariaDB on Ubuntu

I’ve been using MariaDB for some time now and it’s perfect replacement for MySQL especially with the latest news onOracle’s move to hinder MySQL developer community despite it’s promise to EU. Now is a perfect time to ditch MySQL and move to something that’s backed by the original authors of MySQL and that something is MariaDB.

1. First pick your Ubuntu version repository mirror close to you from MariaDB downloads page. Once you’ve picked up your mirror then add them to /etc/apt/source.list.d/mariadb.list. I’m still running 10.04 so here’s what I put in my mariadb.list:

# MariaDB repository list - created 2012-07-04 18:04 UTC
# http://downloads.mariadb.org/mariadb/repositories/
deb http://ftp.heanet.ie/mirrors/mariadb/repo/5.5/ubuntu lucid main
deb-src http://ftp.heanet.ie/mirrors/mariadb/repo/5.5/ubuntu lucid main

2. Next you’ll need to import the signing key

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xcbcb082a1bb943db

3. Update

aptitude update

4. Install

aptitude install mariadb-server-5.5

Now you have MariaDB 5.5 installed and you can configure it exactly like you would configure MySQL.

Tips for securing your Liferay installation

There’s few security related things that I see people constantly doing wrong. The very first thing is assuming Liferay bundle with it’s default settings is secure for production. It is far from secure. Don’t get me wrong this doesn’t mean that Liferay isn’t secure it just means that shouldn’t deploy Liferay with it’s default settings and assume it’s secure. So let’s go over some things you should consider.

Default admin user

Everyone knows the default admin user test@liferay.com and some attacks have taken advantage knowing this user and even it’s userid which is predictable. What I would suggest is not only to change the email address and screenname of this user but actually create a completely new admin user and remove this user.

Portal instance web id

The default company web id is liferay.com and it goes without saying you should change it unless you are actually deploying liferay.com. You can do this simply by setting company.default.web.id property in your portal-ext.properties. This must be done before you start your portal and let it generate the database.

Encryption algorithm

By default Liferay is configured to use 56bit DES encryption algorithm. I believe this legacy is due to US encryption export laws. The problem with 56bit DES is that it was cracked back in the 90s and is not considered secure encryption anymore. Liferay encrypts certaing things with this like your password in Remember Me cookie. If someone get’s a hold of that cookie they can crack your password. I would recommend using at least 128bit AES. To do that you’ll just need to set following properties before starting your portal against a clean database.

company.encryption.algorithm=AES
company.encryption.key.size=128

Password hashing

Recently there has been a lot of sites that have their passwords being compromised because they weren’t using salt with their password hash. Liferay by default uses SHA-1 to hash your password. That hash is a one way algorithm that doesn’t allow reversing the password from the hash but if someone gets a hold of your password hash it’s still possible to crack with brute force or by using rainbow tables. Rainbow tables are precalculated hashes that allow very easily and fast find unsalted passwords. Salt is something we add to the password before hashing it and it’s preferrable unique of each password so that even if two users have the same password their hash is different. Liferay comes with SSHA algorithm that salts the password before calculating the SHA-1 hash from it. You can enable it by setting following in your portal-ext.properties

password.encryption.algorithm=SSHA

Unused SSO hooks

The default Liferay bundle comes with all SSO hooks included even thought they are not all enabled it’s a good idea to remove any hooks your are not using. There’s a property called auto.login.hooks and you should remove all hooks your are not using. Also remember to disable their associated filters.

Unused Remote APIs

Liferay has several different remote APIs such as JSON, JSONWS, Web service, Atom, WebDAV, Sharepoint etc. You should go through them and disable everything your site is not using. Please note that some Liferay builtin portlets rely on some of these APIs. All the APIs are accessible under /api URL.

Mixed HTTP and HTTPS

Everyone should by now know about Firesheep a firefox extension that allows an attacker to sniff a wifi network they are connected to and hijack a users authenticated session. This attack can compromise any website that doesn’t use all authenticated traffic over https. If you use https for just part of the site and your users can access rest of the site as authenticated user over http then your are vulnerable to Firesheep attack. This is especially bad with Liferay if you are using the default encryption and you use Remember me functionality because then the attacker could even compromise your password and use it login to any system where you use the same password. I’m sad to say that even Liferay.com is vulnerable to this attack.

Shared Secrets

Don’t forget to change any shared secrets. The auth.token.shared.secret has a default value that you want to change so that no-one can try to exploit it. This tip came from Jelmer who has found many vulnerabilities in Liferay.  Another one you don’t want to overlook is auth.mac.shared.key which has default value of blank. That one is relevant if you auth.mac.allowset to true.

This is not an exhaustive list but this should make your Liferay installation much more secure than it is by default. For more tips on what to configure before going to production check out Liferay whitepapers. You should especially read the deployment checklist. If you can think of any other things that should be on this list comment them or tweet them to me @koivimik

Update: Added shared secret tip from Jelmer

Monitoring Liferay with Nagios, Jolokia and JMX4Perl

How do I monitor Liferay? That’s a question I’ve heard a lot lately. Well the standard way of getting some information about the application is by using JMX. The downside of JMX is that it’s a Java only standard and the only remote connection is by using RMI which doesn’t really sit well with non Java monitoring software like very popular Nagios. Another hurdle might be that your network admin might not be inclined to open up RMI access to the jvm.

There’s a nice agent called Jolokia that can provide a http bridge to JMX. You can install it as java agent in pretty much any java app or deploy it as a webapp. With Jolokia installed you can query any MBeans for their values using a simple http GET and get the data as JSON objects. JMX4Perl is a perl module and scripts that provide a easy way to run those queries through Jolokia. One of those scripts is check_jmx4perl which can be used in nagios service checks.

Okay so now we know that we are going to need Nagios, Jolokia and JMX4Perl to monitor the Liferay JVM but what should we monitor? Well that depends on what information you are interested in but at minimum I would monitor ajp or http thread usage as well as heap utilization. Just by monitoring those values you’ll know when your JVM is becomes unresponsive and can also get some early warning that there’s issues for example heap usage goes over warning threashold and never returns to normal or keeps constantly going over the threshold which could indicate they you don’t have enough heap allocated.

I’m going to assume that you have  nagios installed and configured and I will only go through how to install Jolokia and configure some checks for threads and heap. So let’s start by installing JMX4Perl.

Installing JMX4Perl is pretty simple with cpan. You just launch cpan command line client and install it like this:

cpan> install JMX::Jmx4Perl

Next you’ll need to download Jolokia and deploy the jolokia.war to your app server. For this example I’m going to assume that you are using Tomcat 7. Once you’ve deployed Jolokia it’s usually good idea to restrict who can query it. For this example we are just going to restrict it to a certain IP address (the Nagios server) and limit it to read operations only. Since I don’t like modifying the war we are going to tell Jolokia where to find the policy file through a context parameter. Create a jolokia.xml in tomcat/conf/Catalina/localhost with following content:

<Context path="/jolokia">
        <Parameter name="policyLocation" value="file:///etc/jolokia/jolokia-access.xml" />
</Context>

That tells Jolokia to look for the policy file jolokia-access.xml from /etc/jolokia/jolokia-access.xml. This is great when you are running multiple tomcats in the same server and want them to share the jolokia policy file.

Now go ahead and create the jolokia-access.xml in /etc/jolokia

<?xml version="1.0" encoding="utf-8"?>
<restrict>
        <remote>
                <host>[YOUR NAGIOS SERVER IP]</host>
        </remote>
        <http>
                <method>get</method>
                <method>post</method>
        </http>
        <commands>
                <command>read</command>
        </commands>
</restrict>

Next we need to create configuration for jmx4perl. In /etc/jmx4perl/jmx4perl.cfg we are going to include some preconfigured checks extend them. Tomcat 7 you need to add quotes around the thread pool name. We also need to set warning and critical levels for alerts. You’ll also need to add a Server for each tomcat you want to monitor.

# Default definitions
include default/memory.cfg
include default/tomcat.cfg

# ==========================
# Check definitions

<Check tc7_connector_threads>
	Use = relative_base($1,$2)
	Label = Connector $0 : $BASE
	Value = Catalina:name="$0",type=ThreadPool/currentThreadCount
	Base = Catalina:name="$0",type=ThreadPool/maxThreads
	Critical 95
	Warning 90
</Check>

<Check j4p_memory_heap>
	Use memory_heap
	Critical 95
	Warning 90
</Check>

<Server tomcat>
	Url http://MY_TOMCAT_HOSTNAME:8080/jolokia
</Server>

Then in /etc/nagios3/commands.cfg we’ll need to add a check command for jmx4perl and we’ll use the check_jmx4perl script to do that.

define command {
	command_name    check_j4p_cmd
	command_line    /usr/local/bin/check_jmx4perl --unknown-is-critical --config /etc/jmx4perl/jmx4perl.cfg --server $ARG1$ --check $ARG2$ $ARG3$
}

Then we need to define a service to monitor in /etc/nagios3/conf.d/host-MY_TOMCAT_HOSTNAME.cfg

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat Heap Memory
	check_command check_j4p_cmd!tomcat!j4p_memory_heap!x
}

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat AJP Threads
	check_command check_j4p_cmd!tomcat!tc7_connector_threads!ajp-bio-8009
}

The check above is for your tomcat heap and the other one is for Tomcat 7 AJP threads.

Now you should all the pieces to implement your own monitoring using Nagios, Jolokia and JMX4Perl. You should also remember that you can apply this to any JEE application not just Liferay.

Configuring c3p0 connection pool for Liferay on Tomcat

There’s several ways you could configure a connection pool for Liferay on Tomcat but the way I’m going to show is the JEE way and the only one I consider correct.

The first thing is to copy or move the c3p0.jar from webapps/ROOT/WEB-INF/lib/ to lib/. Also make sure you have your dabase driver there. In this example it would be mysql.jar.

Then we need to tell Liferay that you want to use a connection pool from JNDI and this you can do by adding following line to your portal-ext.properties which can be placed in Liferay Home directory (the directory above tomcat).

jdbc.default.jndi.name=jdbc/LiferayPool

Add following snippet to conf/server.xml inside GlobalNamingResources. Adjust the pool size and idle time and connection test period according to your environment. They are particularly important when you have a firewall between your Liferay and database or when the database server drops connections after certain idle period.

<Resource
    name="jdbc/LiferayPool"
    auth="Container"
    type="com.mchange.v2.c3p0.ComboPooledDataSource"
    factory="org.apache.naming.factory.BeanFactory"
    driverClass="com.mysql.jdbc.Driver"
    jdbcUrl="jdbc:mysql://localhost/lportaluseUnicode=true&amp;characterEncoding=UTF-8&amp;useFastDateParsing=false"
    user="lportal"
    password="test"
    minPoolSize="10"
    maxPoolSize="20"
    maxIdleTime="600"
    preferredTestQuery="select 1 from dual"
    idleConnectionTestPeriod="180"
    numHelperThreads="5"
    maxStatementsPerConnection="100"
/>

Now we need to link the jdbc/LiferayPool name defined in portal-ext.properties to the jdbc/LiferayPool defined in server.xml and this definition goes to conf/Catalina/localhost/ROOT.xml

<ResourceLink name="jdbc/LiferayPool" global="jdbc/LiferayPool" type="javax.sql.DataSource"/>

Now we are done and you can start your tomcat with the new connection pool. Note you can follow similar process to configure MailSession from JNDI.

Why is my java process taking more memory than I gave it?

It seems to be quite common misconseption that the memory you give to java process with -Xmx and -Xms command line arguments is the amount of memory the process will consume but in fact that is only the amount of memory your java object heap will have. The heap is just one factor in how much memory the java process will consume. To better understand how much memory your java application will consume from the system you need to understand all the factors that account for the memory usage. Those factors are:

  • Objects
  • Classes
  • Threads
  • Native data structures
  • Native code

The memory consumption associated with each item varies across applications, runtime environments and platforms.  So how do you calculate the total memory? Well, it’s not really all that easy to get accurate number because you have little control over the native part. The only parts you can really control is the amount of heap -Xmx, memory consumed by classes -XX:MaxPermSize and thread stack -Xss which controls the amount of memory each thread takes. Be careful when adjusting stack size as too low size will cause StackOverflow exceptions and your application won’t work correctly. So the formula is:

(-Xmx) + (-XX:MaxPermSize) + numberofthreads * (-Xss) + Other mem

The other mem part depends on how much native code is used like NIO, socket buffers, JNI etc. It’s anywhere from 5% of total jvm memory and up. So assume we have following JVM arguments and 100 threads

-Xmx1024m -XX:MaxPermSize=256m -Xss512k

That would mean that the jvm process would take at least: 1024m + 256m + 100*512k + (0.05 * 1330m) = 1396.5m.

I usually use a quick approximation rule of 1.5 * max heap to be the minimum amont of RAM a tomcat process will require. This can be higher if you have large application that requires you to increase MaxPermSize beyond 256m. If you use this to size how much memory your system will require remember that you need to leave memory for the OS and other applications running on the system otherwise you might end up using a lot of virtual memory that will affect your application performance negatively.

What to Do When You Get “Error listerStart” with Tomcat

I’m sure many people other than me have banged their head in the wall trying to figure out an error like this:

SEVERE: Error listenerStart 
26-May-2012 13:44:27 org.apache.catalina.core.StandardContext startInternal 
SEVERE: Context [] startup failed due to previous errors

That basically means that tomcat failed to start the webapp because there was an error with some listener, quite often Spring context listener. The really annoying part is that it doesn’t actually show you what went wrong. There’s actually pretty simple way to get tomcat to log the actual error. You just need to create a logging.properties in WEB-INF/classes of the failing webapp and add following lines to it:

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

Then just reload the webapp to see the error in tomcat console log. I hope this tip saves you a lot of hasle from trying to figure out the root cause of the problem.

How to Create a Consistent Liferay Backup

This is a question I’ve gotten asked in nearly all the Liferay System Administrator trainings I’ve given. Most people will just backup their database and Liferay data directory separately but any competent system admin will tell you that it’s not guaranteed to be consistent because someone could upload or delete files between the time you took the database dump and the time you copied the data directory. Now I’m assuming that you are storing your document library binaries to filesystem instead of database.

Now to achieve a consistent backup with minimal interruption to your portal what you need to do is get a read lock on all your Liferay tables. This will prevent writes to the database. Then you dump the database to file with a tool like mysqldump and then you take a quick snapshot of the filesystem before you unlock the tables. You need to keep the connection that locked the tables open until this whole process is done. Once you have the database dump and filesystem snapshot ready only then you can release the lock and then you can backup the data directory using what ever method you would normally use.

For the PoC I’m using MySQL and my filesystem is on Linux LVM volume which supports taking snapshots. I’ve written a Perl script to execute all the commands. I’m sharing the script under GPL and it’s available in Github. Feel free to fork it and modify it to suit your needs and if you have good ideas send me a pull request.

The way the script works is you pass in bunch of parameters like database details, lvm volume location, source and target directories. Here’s an example:

backup_liferay.pl -u dba-backup -p mypassword -d lportal -h localhost \
--lvm-volume-path /dev/vg0/opt --lvm-snapshot-volume-path /dev/vg0/opt-snapshot \
--lvm-snapshot-volume-name opt-snapshot --lvm-snapshot-volume-size 50G \
--snapshot-mount-path /backups/snapshot \
--source-path /liferay-portal-6.1.0/data/document_library \
--db-target-path /backups/mysql/lportal.sql.gz \
--data-target-path /backups/liferay --compress

Now even if that doesn’t exactly match your system I hope it gives you an idea how to roll your own Liferay backup.

Debugging Maven Plugins

When developing maven plugins things don’t always work the way you expect so you need to debug the Mojo to see what’s really going on. I had a weird case where my plugin worked when I ran it independently but when I ran it with mvn clean package it always failed. First thing you can do is run in debug mode which produces a lot more output and shows all the plugin execution configuration. You can enable it with -X argument like this:

mvn -X clean package

Now that didn’t quite help with my case so next thing I did was to run it with remote debugger. That way I could step through the code line by line and inspect all the variables. To do that you just modify the MAVEN_OPTS environment variable in the shell where you are executing you maven plugin and add java debugger agentlib config like this:

MAVEN_OPTS="-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=y"

I used suspend=y so that it would wait for my debugger to attach before continuing the execution. Then you just add some breakpoints in you IDE and remote debug it like any java application. That by the way solved my issue as I realized each of my Liferay maven plugins were initialing Liferay configuration but since they were all run after each other in the same context only the first one mattered.