When optimizing - don't forget the Java Virtual Machine (JVM) | Big Data

When optimizing - don't forget the Java Virtual Machine (JVM)

Recently, I was working on a project that was coming to a close. It was related to optimizing a database using a Java based in-memory cache to reduce the load. The application had to process up to a million objects per day and was characterized by its heavy use of memory and the high number of read, write and update operations. These operations were found to be the most costly, which meant that optimization efforts were concentrated here.

The project had already achieved impressive performance increases, but one question remained unanswered - would changing the JVM increase performance?

As the code was written in Java, the first idea which came to mind was to optimize the Java Virtual Machine (JVM). Many optimization configuration settings are available for each vendors JVM offering. e.g. Sun Microsystems. Some examples of optimization parameters include: memory allocation, garbage collector configuration, thread configuration and a lot more. Andrig T. Miller presents his experiences here and stackoverflow has useful tips.

In this post I don't want to concentrate on the individual tweaking of JVM configuration settings as I wanted to test how the application performs on other vendors JVMs, rather than the default one from Sun Microsystems. It is often better to replace the engine than to tweak an existing one (In the software world it is usually not true).

TESTED JVMs
First I had to define the list of JVMs which I wanted to test. They had to be available for free as I didn't have time to ask any company to send me their version of JVM for evaluation. I tested the following JVMs:

Sun Java 1.6 Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
IBM Java 1.6 IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Linux amd64-64)
Oracle JRockIt Java 1.6 x64 BEA JRockit(R) (build R27.6.3-40_o-112056-1.6.0_11-20090318)
OpenJDK 1.6 OpenJDK Runtime Environment (IcedTea6 1.5) (fedora-22.b16.fc11-x86_64)
Sun Java was the obvious choice, it is commonly used in industry, so there is no need for explanation. IBM's JVM was chosen as IBM has a proud heritage and has been developing its own JVMs for years. Oracle JVM is pushing hard into the market, so I had to check it. Why OpenJDK? I like open source, I'm using Fedora Linux and it was in the repository.

HOW I TESTED
Now it was time to test these toys. But how to test them? I had two choices: write my own testing suite or use existing tools which are proven to be reliable in testing of Java performance. I decided to do both.

Simple application
As for my own testing framework, I wrote a very simple Java application which was performing many add/delete/modify operations on a HashMap object. Additionally this application was executing the garbage collector every X iterations, why? To replicate the operations that were limiting performance in the in-memory cache software, especially the frequent execution of the garbage collector.

Testing tools
Regarding some existing tools, I quickly researched and found two suitable solutions. The first one was Phoronix Test Suite (PTS), which is a very nice framework for performance testing using:

Sunflow Rendering System -  tests rendering time of sample scene
Bork File Encrypter - tests file encryption performance
SciMark - Java benchmark for scientific and numerical computing.
Application to speedup
As for the final testing tool, I decided to use the in-memory cache application which required speeding up. The results from this test would determine if the application should be migrated to another JVM.

RESULTS
So what about the results?

Using the Phoronix Test Suite SunJDK6, OpenJDK were almost equally fast. Oracle RockIt was much slower than these two, but IBM JDK6 was the slowest performing JVM. So don't use these two with PTS, you will not see any speedup.

My own simple testing application surprised me in terms of performance. IBM JDK6 was the fastest JVM in this test and by fastest I mean really fast. Average time of an iteration in my app was 13 seconds (Sun JDK6, OpenJDK, JrockIt). IBM JVM averaged 4 seconds.

And now for the most important results for the in-memory cache application to be sped up. The results can be seen in the graph below:

What we can see there? A big surprise, our application was performing almost the same on each JVM. The presented results are average, I ran the application 10 times on each JVM. No real improvement, so it did not make sense to migrate to another JVM and invest time in proving its stability.

CONCLUSION
The main finding was that no speedup was observed for the in-memory cache application that I was working on. This resulted in no JVM migration and subsequent testing. The project was closed faster and there were no lingering doubts about whether a change in JVM would increase performance.

Interestingly, the simple application that I developed executed 3X faster on IBM JDK. This indicates that application characteristics can result in significant differences in their performance on JVMs.

The main point is to test the application that you want to tune on different JVMs. It is simple and quick to do and you may be surprised by the performance increases that you see.