Java Remoting: Protocol Benchmarks

I’ve been analyzing Java remoting protocols at work over the past couple of days, and thought I’d share some of the results here. Specifically, I’m going to share the results of our protocol benchmarking.

Every application will have different requirements in this area, but most criteria will include performance. At the very least, you will want to know how much (if any) performance you are sacrificing for the sake of your other requirements.

The Protocols

The protocols under consideration are Java’s RMI/JRMP, Oracle’s ORMI (with and without HTTP tunneling enabled), Spring’s HttpInvoker, Caucho’s Hessian and Burlap, and three flavors of Apache XML-RPC (Sun-based, HttpClient-based and Lite-based).

RMI/JRMP is Java’s default binary remoting protocol, and its big advantage is that it supports full object serialization of Serializable and Externalizable objects. RMI/JRMP has historically not been as easy to use as some of the other options, but if you’re using Spring (as we are), it’s as easy to use as anything else (except Apache XML-RPC; see below).

ORMI is OC4J’s EJB remoting protocol, which we’re currently using in production. This protocol was originally developed as part of the Orion application server, whose codebase formed the basis for OC4J. The only differentiator between ORMI and RMI/JRMP is likely to be performance. Of course, this protocol probably won’t be an option unless your serverside component is deployed on OC4J. ORMI supports tunneling via HTTP and HTTPS, should you run into firewall or proxy problems.

Spring’s HttpInvoker is another Java-to-Java binary remoting protocol. It’s also very easy to use and supports full object serialization of Serializable and Externalizable objects. The big difference between HttpInvoker and RMI/JRMP is that HttpInvoker transports data via HTTP, which makes it easier to use when you start running into proxy and firewall issues. Unfortunately, HttpInvoker isn’t an option if you’re not using Spring on both the server and the client.

Caucho’s Hessian protocol is a slim, binary cross-platform remoting protocol. Most cross-platform protocols are XML-based, and thus sacrifice a significant amount of performance in order to achieve interoperability. Hessian’s draw is that it achieves cross-platform interoperability with minimal performance degradation. However, Hessian uses a custom reflection-based serialization mechanism which can be troublesome in certain scenarios (think Hibernate proxies).

Currently in a draft state, Hessian 2 is the second incarnation of the Hessian protocol. You probably don’t want to use it in your production applications quite yet, but it looks very promising. Of course, it’s so easy to switch between Hessian and Hessian 2 that you might want to have a peek just for the heck of it [1].

As far as I can tell, Caucho’s Burlap is essentially Hessian-in-XML. As such, Burlap is not a binary protocol and should be expected to suffer accordingly in the performance department. This being the case, I haven’t been able to figure out exactly why anyone would choose to use Burlap instead of Hessian… aside from a requirement from the marketing department that your RPC be buzzword compliant.

Apache XML-RPC is Apache’s implementation of XML-RPC, an XML-based RPC specification which focuses on interoperability. Apache XML-RPC may be a bit more difficult to set up than the other options if you’re using Spring. The infrastructure for the other protocols is built into the Spring framework, but you’ll have to write the Apache XML-RPC / Spring integration yourself.

Apache XML-RPC supports a number of transport factories under the covers. The three transport factories tested were the default transport factory (based on Sun’s HttpURLConnection), the HttpClient transport factory (based on HttpClient) and the Lite transport factory (based on an Apache XML-RPC minimal HTTP client implementation).

The Configuration

The test consisted of remote invocations to a single method which takes an integer size parameter and returns a list of the specified size. The objects in the returned list each contained 10 instance variables (4 strings of about 40 characters each, 2 dates, 1 long, 1 integer, 1 double and 1 float, none of which were null). The lists returned were static, so there was no list construction overhead.

The results of the first 10 invocations were thrown away, in order to allow the VMs to “warm up.” The next 100 invocations were tallied and averaged.

The tests were performed on a Windows XP SP2 machine with 2GB of RAM and a 2Ghz Core 2 Duo CPU, using Sun’s JVM 1.5.0_09. The client and server were run in two separate VMs on the same host, in order to avoid spurious networking artifacts [2].

The following library versions were used: Spring 2.0.6, Jetty 6.1.5, OC4J 10.1.3.2.0, slf4j 1.4.3, log4j 1.2.14, Apache XML-RPC 3.1, Commons HttpClient 3.1, and Caucho Hessian 3.1.3.

Spring was used on both the clientside and on the serverside in order to hide the remoting details from the interface consumer and implementor. All serverside components were run inside a single Jetty servlet container, except for the ORMI serverside testing components which were run in OC4J.

Vendor extensions and GZIP requesting were enabled for all Apache XML-RPC tests. Streaming and GZIP compressing were enabled for all Apache XML-RPC tests except the Lite tests (the Lite transport factory does not support these options). These configurations represent the best performance optimizations available to the various Apache XML-RPC flavors.

The Results

The following graph illustrates the response times of the various protocols for invocations returning smaller lists:

protocol-benchmark-smaller-lists-3.png

The following graph illustrates the response times of the various protocols for invocations returning larger lists:

protocol-benchmark-larger-lists-3.png

Conclusion

It’s important to note that no general conclusions can be derived from the absolute numbers represented above. Rather, the numbers must be examined relative to each other.

That said, the following observations may be made:

  1. The binary protocols (RMI, ORMI, HttpInvoker and Hessian) are always faster than the XML-based protocols (Burlap and the Apache XML-RPC variants) — except for ORMI with HTTP tunneling enabled.
  2. Performance is pretty even amongst the binary protocols — except for Hessian, which performs well only when compared to the XML-based protocols, and ORMI with HTTP tunneling enabled, which performs on a par with the XML-RPC variants.
  3. Burlap has much better performance than the XML-RPC variants.
  4. Native RMI and Hessian 2 have the best performance until the remote method invocations start returning larger lists, at which point vanilla ORMI takes a slight lead.
  5. Changing Apache XML-RPC’s transport factory does not seem to have a very large effect on performance.
  6. It’s amazing how fast standard ORMI is, compared to ORMI with HTTP tunneling enabled.
  7. Hessian 2 bears watching!

Other Interesting Links

JBoss Remoting Framework Benchmarks: Tom Elrod’s benchmark of the JBoss Remoting framework. Includes part of the Spring Remoting framework.

Nominet’s Protocol Benchmarks: Interesting benchmark of many of the protocols here considered. Intriguingly, HttpInvoker is found to have consistently better performance than RMI/JRMP, a finding which contradicts our results.

Footnotes

[1] If you’re using Spring’s Hessian integration and the latest Hessian JARs, moving to Hessian 2 is as easy as adding the following property to your HessianProxyFactoryBean:

    <property name="proxyFactory">
        <bean class="com.caucho.hessian.client.HessianProxyFactory">
            <property name="hessian2Request" value="true" />
            <property name="hessian2Reply" value="true" />
        </bean>
    </property>

[2] The tests were later run on two separate machines over a controlled network, in order to verify that the local tests are representative; they are.

About these ads

54 Comments

  1. Emil Ong said,

    January 8, 2008 at 1:44 am

    Great article! It might be nice to do a comparison of bandwidth used as well. I think Hessian 2 should beat RMI in a lot of the tests in that case.

    Emil

  2. Calum Shaw-Mackay said,

    January 8, 2008 at 6:26 am

    It would be interesting to see how JERI (Jini Extensible Remote Invocation) compares to Standard JRMP, especialliy as you can more or less configure the entire Invocation Layer stack for things like security etc, and there are differing endpoints (HTTP, TCP and Memory) etc.

    Calum

  3. Shoaib Akhtar said,

    January 8, 2008 at 10:01 am

    Can you please provide the link for Hessian 2?

    Thanks,
    Shoaib

  4. anonymous said,

    January 8, 2008 at 10:22 am

    http://hessian.caucho.com/

  5. Anonymous said,

    January 8, 2008 at 11:24 am

    Corba?

  6. Daniel Gredler said,

    January 8, 2008 at 12:37 pm

    Emil: You’re right. As Christian noted in the mailing list discussion, these numbers assume optimal network conditions. It would be interesting to have a test which weighs bandwidth use appropriately. Maybe another time :-)

    Calum and Anonymous: Yep, both JERI and CORBA would be interesting additions. But I had to draw the line somewhere.

    Shoaib: Just follow the Hessian/Burlap link at the top of the article; the link is the same for Hessian and Hessian 2. The latest Hessian JAR includes support for both protocols.

  7. Paul Keeble said,

    January 8, 2008 at 5:08 pm

    Would it be possible to get hold of the source code? I’d like to add a few other tests within the same framework (such as Axis and other webservices implementations) to see just how much those web service calls are really costing comparatively to the tests you’ve already done.

  8. January 9, 2008 at 3:52 am

    Daniel —

    Thanks for the interesting article.

    You may also want to consider JSON-RPC (http://json-rpc.org) which has several open source Java implementations, including http://jabsorb.org. The protocol is simple and has good chances to be fast. There are implementations in many languages, most notably in JavaScript which makes it a good choice for Web and hybrid thin/fat-client applications.

    — Sasha

  9. Richard Henderson said,

    January 9, 2008 at 4:10 am

    httpinvoker! Very quick. So it isn’t about the base64 or http protocol overhead. Just the XML tax?

  10. January 9, 2008 at 4:26 am

    CORBA would be interesting for companies sill running CORBA legacy apps.

    Peace
    -stephan

  11. Shiv Prakash Ojha said,

    January 9, 2008 at 5:34 am

    Quite inlined with what we did some 12 months back.

    I wanted to share my experience regarding Hessian & Burlap. They have issues in serialization when

    – Object Graph is complex and contains Inner/annonymous classes
    – They use reflection to create object instance – so if you have validations in constructors & don’t have support for default constructors – they fail to de-serialize.

    I hope this would be useful info for others as well.

    Thanks,
    Shiv

  12. Giampaolo Tranchida said,

    January 9, 2008 at 5:55 am

    Good job.

    it will be interesting to compare also with the last JAX-WS Web Service implementation (CXF or Axis2)

    Thanks

  13. athanazio said,

    January 9, 2008 at 8:04 am

    would be great to have the code of the benchmark so we an add other protocols to the test, and have a richer result data… what you think ?
    I would suggest to test HTTP request with JSON result data.

  14. January 9, 2008 at 8:08 am

    [...]  http://daniel.gredler.net/2008/01/07/java-remoting-protocol-benchmarks/ [...]

  15. Daniel Gredler said,

    January 9, 2008 at 12:14 pm

    Paul and Athanazio:

    Sure thing! I’ll get you the code within the next couple of days.

    Two caveats:

    1. Please share the modified code and results, so we all benefit;

    2. I can’t provide the OC4J serverside component, because it’s grafted onto a proprietary EAR which I had laying around. You’ll be able to test all of the other protocols, though.

    Suggestions for enhancements have included: Jini’s JERI, CORBA, JSON-RPC, CXF, Axis, Axis2 and probably some others I’ve forgotten :-)

    Also, people have suggested testing different sizes of object graphs, in addition to the existing list tests.

  16. Daniel Gredler said,

    January 9, 2008 at 1:05 pm

    Richard: It looks that way. Keep in mind that the binary protocols are free to add all kinds of optimizations, too. For example, I think Hessian 2 has special short encodings for common integers.

  17. Mileta Cekovic said,

    January 10, 2008 at 8:28 am

    Including ICE would be nice also, as ICE was event faster then the RMI in my latests tests.

    Also do not expect wonders from CORBA, it is 10-20% slower then RMI as it has to use platform interoperable protocol IIOP, and because of that it has to be slower then Java optimized RMI.

    At the end, if your environment needs best performance and lowest bandwidth ‘custom socketing’ with custom protocol and custom binary serialization is the way to go. Even Java binary serialization has a lot of overhead compared to custom serialization as Java serialization has to be aware of JVM and class versions and thus stores full class and property names with the serialized objects which can be such an overhead.

  18. codist said,

    January 10, 2008 at 11:38 am

    Be interesting (but difficult) to compare these with TerraCotta’s approach.

  19. January 11, 2008 at 2:54 pm

    Good work! I would like to add a few notes;
    1) Why did you forget about BEA’s t3 RMI if you are looking for performance? You’d better give it a try and I’m sure you will be fascinated by the results. The same about JBoss JNP and (dead!) JBoss serialization! But still t3 rules ;)
    2) I guess you should add another chart showing JVM’s thread number for each remoting scenario. I did this once and results were very interesting in fact. To have the best performance it great, but to reach it with minimal thread/resources is greater. You will see that some protocols do not fully utilize thread pooling/multiplexing while some others do.
    3) Again if you really care about performance (and can ignore standards a bit) have a look at MINA in combination with its object serialization codec.
    4) Space based remoting (gigaspaces for example) is also a great absent of the show :)
    Regards

  20. January 14, 2008 at 5:21 pm

    [...] Java Remoting: Protocol Benchmarks « A Public Scratchpad (tags: remoting java performance webservices) [...]

  21. sanjay Radia said,

    January 25, 2008 at 12:36 pm

    Good benchmark.
    2 points:
    1. If you could publish the sources to your benchmark I would like to add RMI/JERI and ICE.
    I will post back the additional code.

    2. Failure semantics.
    It would be useful to list the failure semantics of each of the RPC systems.
    For example, RMI is at-most-once — very useful for building diverse distributed applications.
    For example Hessian does not state their semantics on their specs page (or at least I could not
    find it).

    thanks

    sanjay

  22. Daniel Gredler said,

    January 25, 2008 at 12:44 pm

    Sanjay: I just sent you the code via email (unfortunately I’m not allowed to post ZIP files here). Looking forward to seeing your results!

  23. February 7, 2008 at 4:27 pm

    [...] Gredler wrote a very good article about the performance of Java remoting protocols. Hessian is fast as we can see but what about a [...]

  24. February 24, 2008 at 4:05 am

    [...] remoting spectrum in order to ship a large number of objects across the net. Here I found a great article that experimented several popular remoting protocols and did a comparison. The protocols covered [...]

  25. afuentes said,

    March 16, 2008 at 10:51 am

    Very interesting.
    Questions ???
    Do you know performance issues with client c++ to have http keep connection alive (5) ???
    This client make many request per second on same connection.
    For me hessian is good options for that.
    Waht do you think ??

  26. surender said,

    June 4, 2008 at 9:47 am

    Wonderful comparison.And as someone mentioned above, for really loaded distributed systems, threads/resource consumption should also be considered. For example, Sun’s RMI creates thread on each connection and doesn’t use thread pool where hessian can use containers thread pool. I wonder though when it can become bottleneck if production grade servers run on 4GB ram on clustered environment.

  27. SaaSMan said,

    June 8, 2008 at 9:53 am

    I found this blog entry while researching Java serialization performance. My objective is to efficiently serialize 5,000 to 10,000 pojo objects and store these as a blob on Amazon S3 storage service. Does the wide range of results here effectively highlight the performance of the underlying serialization implementations?

  28. Jimmy Zhang said,

    August 2, 2008 at 5:38 pm

    I think you guys should investigate latest XML parsing technology such as VTD-XML
    http//vtd-xml.sf.net

  29. Maneesh Chauhan said,

    August 14, 2008 at 1:15 pm

    Hi
    We are using the Hessian for the data transfer from server to client side. I have few question how the Hessian works in case of the Bulk Data

    Is there any constraint of bulk data ?

    Can we use the Hessian to send the 200 , 000 objects and each object contains 100 bytes

    Thanks in advance for your prompt answer

  30. Abhishek Kumar said,

    September 11, 2008 at 6:25 am

    Hey guys i rebuilt the test, but for ArrayList with 5000 elements i am getting for Hessian about 40 ms and for HTTP Invoker 45 ms avg time ..its way better than in graph .. i am wondering why!!!!

  31. October 3, 2008 at 6:29 am

    [...] is a binary web service protocol. It’s quite efficient (see Daniel Gredler’s and Miquel’s benchmarks) and it’s dynamically [...]

  32. October 3, 2008 at 5:07 pm

    [...] is a binary web service protocol. It’s quite efficient (see Daniel Gredler’s and Miquel’s benchmarks) and it’s dynamically [...]

  33. October 23, 2008 at 6:48 am

    Interesting comparison. Thanks for sharing it.

  34. October 31, 2008 at 8:50 am

    Daniel – I’d be interested in receiving the test code package. Would love to update it with a CORBA test just for comparison, and possibly also CXF and Axis. Anything I did I’d of course ship back/make available. Thanks.

  35. November 30, 2008 at 11:56 pm

    [...] is an interesting benchmark, demonistrating how hessian perform against classical binary [...]

  36. Nilesh Wani said,

    December 15, 2008 at 6:33 am

    Hi Daniel,

    Is it possible for you to please share the source code for this comparison? That would be very helpful for many developers. We have to decide on a remoting protocol and the code can be very help us in taking the right decision.

    Thanks
    Nilesh

  37. Nullio said,

    January 13, 2009 at 3:51 pm

    Danil,

    I would really appreciate if you could send me a copy of the source code used in this test.
    It will be very helpfull for my current assignments.

    Thanks in Advance

    Nullio.

  38. Abhishek Kumar said,

    February 2, 2009 at 6:37 am

    hey nullio send me ur mail id at abhishek9852@gmail.com
    ill mail u the source code , i also did the same thing for my project !

  39. Rich said,

    April 7, 2009 at 3:17 pm

    I think it’s important to contract remoting framework overhead (RMI, Spring, Servlet, HttpInvoker) and serialization (Hessian1+2, java Object*Stream, JBoss serialization).

    Im my testing of serialization libraries over plain TCP and UDP sockets, Hessian2 was significantly compressed in relation to its competitors and performed much better.

    The latencies resulting from the remoting frameworks themselves are a separate issue.

  40. Rich said,

    April 7, 2009 at 3:18 pm

    contract = constrast

  41. Stefan Scheidt said,

    April 9, 2009 at 2:53 pm

    Wouldn’t it be nice to make your test suite open source, e.g. on googlecode?

    Stefan

  42. April 10, 2009 at 5:05 am

    [...] from Caucho is released under the Apache License 2.0. Hessian is well integrated into Spring and seems to perform well for version 3.2 (one year old!). An ORM tool which supports Hessian out of the box is [...]

  43. Juan said,

    April 21, 2009 at 9:32 am

    For updated bechmarks:

    http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking

  44. Mojahedul Hoque Abul Hasanat said,

    May 12, 2009 at 12:23 am

    A very useful article. I was just looking for this information. Thanks for sharing.

  45. Chris said,

    May 28, 2009 at 7:10 am

    Nice piece of work! Again, thanks for sharing this information.

    In case speed an excellent scaling behaviour is an issue, it would be indespensable to obey main semantic rules of the web – i.e. using GET where clients need a view to a centralized model and POST where clients change the model. GET requests meight be cached by clients and proxies, POST requests cannot.

    As far as I can see, all the lightweight candidates (Hessian, Burlap, HttpInvoker) allways use POST, which means the whole HTTP Caching infrastructure is turned off.

    Does anybody know a framework which uses HTTP, is leightweight AND offers to switch to GET where it is appropriate? (And, yes, I know it is a challenge to map method parameters to URL parameters.)

  46. Brian Bonner said,

    June 18, 2009 at 3:36 pm

    I’m curious have you been able to stream complex objects across the wire with burlap, hessian or hessian2, Strings seem to work fine, but any complex user defined objects seem to fail.

  47. September 24, 2009 at 4:42 pm

    [...] Verwendung2 Siehe auch zu “>Weblinks zu zu Burlap Design NotesJava Remoting: Protocol Benchmarks, Performancevergleich von Daniel Gredler zwischen Hessian, Burlap, Oracles ORMI, RMI, XML-RPC [...]

  48. September 29, 2009 at 6:26 pm

    [...] are using SOAP as an RPC protocol although there are no advantages in following this approach. Measurementshave shown that the overhead of SOAP compared to RMI-JRMP is significant. Performance degradation by [...]

  49. November 22, 2009 at 1:16 pm

    [...] Update 22.11.2009 Wireless Java RMI Efficient Support of Java RMI over Heterogeneous Wireless Networks Wireless Optimization for Java RMI Java: Expected overhead of the rmi protocol Java Remoting: Protocol Benchmarks [...]

  50. April 27, 2010 at 3:28 pm

    I was recently required to start writing services using RMI instead of RESTful JSON transfer and wrote a little tutorial on how to support both automatically in Grails, http://techscursions.blogspot.com/2010/04/easy-rmi-services-in-grails-with-json.html . Enjoy!

  51. Kam said,

    June 23, 2010 at 1:25 pm

    Hello,

    Does anyone knows how does Hessian manage the HTTP connections? Is it using a pool of connections or it creates a new one for each new request?

    I have not luck with google so far :(

    Thanks!
    Kam

  52. April 11, 2011 at 5:11 am

    [...] ya tiene algunos años y las librerías y las JVMs han ido mejorando, sobre todo en rendimiento, este estudio es ilustrativo en términos comparativos sobre los tiempos de respuestas de diferentes tecnologías [...]

  53. August 20, 2011 at 6:20 pm

    [...] instances is much faster too. These are two charts depicting the speeds of the various protocols (source). SOAP can be compared to XML-RPC. Clearly, RMI is superior (click to enlarge): Also [...]


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: