SOAP Benchmark Results for Linux Dual Xenon over Gigabit Ethernet

Last updated: $Date: 2004/06/22 23:08:29 $

Purpose

Make sure to read benchmark description.

This test was performed to gather data on performance of long running SOAP services on client/server situations when network is very fast (for example Grids).

Ultimate goal of this benchmark is to provide a tool that helps to evaluate and in process of using it to improve current SOAP toolkits and their suitability for Scientific Computing that requires ability to transfer large arrays and process quickly lot of small message. This latter requirement is common with the business world and even in business services there is need to transfer large BASE64 encoded chunk of data so even former goal can be useful.

The same service was implemented using different toolkits and services were deployed and then accessed using the same benchmark test driver so this test only measures server side behavior.

Also as it is testing long running services we are very concerned about average response time over longer time periods and not interested in quick results for few iterations ...

Setup

Hardware: both client and server are machines with two 2GHz Pentium 4 Xeons processors, 1 GB DDR Ram, and a 15K RPM 18GB Ultra-160 SCSI drive. The nodes are by Gigabit Ethernet.

Software: JDK 1.4.2 (build 1.4.2_04-b05, mixed mode) using Java HotSpot (TM) Server VM

Tests SOAP toolkits:

Tests were run for long time (minimum few minutes) and averages were computed. Tests were executed for arrays of size 10, 100, 1000, 5000, 10000, 25000, 50000, 75000, 100000 (including BASE64 encoded array of bytes of such sizes) using following script (executed multiple times).

gSOAP and XSOAP4 completed all tests.

Axis C++ could not get past size 5000: we did not have enough time to debug why it was failing or why sendInts etc. methods did not work (see test log).

Axis Java was able to complete all tests but the memory overhead was approaching 1GB (AXIS/tomcat) - good that machine had 1GB of memory and pent of swap ....

Ping test

This test just measures bottom line speed of sending and receiving (echo) of empty (void) SOAP message.

It seems AXIS-Java has problems mainly because machine starts swapping and even though we kept changing order fo running streaming and non-streaming versions it seems that AXIS-Java non-streaming is better when processing short messages.

AXIS-C++ is surprisingly slower compared to fast gSOAP and XSOAP4 (Java!) so it suggests there is room for improvement in HTTP transport in AXIS-C++.

 

Arrays Test

We send arrays and receive (echo) arrays of doubles, ints, and strings.

Small arrays: all toolkits finished test.  gSOAP and XSOAP4 has similar speeds as the only limitation is HTTP 1.0/TCP connection. Even though AXIS-C++ should be on par with gSOAP (C++) it is more like XSOAP4 (Java).

AXIS-Java is much slower and even for such small size - there is a difference between streaming and non streaming version but it is only about 10-20%.

 

Medium size arrays: AXIS-C++ was not able to finish this test and is not shown.

gSOAP is clearly the fastest, XSOAP4 close second (about 2x slower) and AXIS-Java is very very slow.

 

Large arrays: AXIS-Java was again very slow.

 

So we have chosen to zoom in more details of gSOAP and XSOAP4 performance:

gSOAP is very well optimized and clear winner when sending numerical arrays.

 

 

Binary (Base64) Test

We send and received (echo) bytes encoded as BASE64.

AXIS-C++ did not get test past 5000 so we do not show it later.

 

First results for small payloads.

Zooming into the fast toolkits:

And then processing large data

Zoomed to show the fastest gSOAP and XSOAP4 performance:

AXIS-Java is surprisingly slow for small byte arrays but reasonably efficient for large arrays which suggest that there is big constant overhead in BASE64 handling in AXIS-Java engine.

gSOAP is again clear leader in performance.

 

Observations

gSOAP is the fastest toolkit available and it especially shines when transferring large amount of data. XSOAP4 even though relatively new (in alpha stage) and not yet optimized for performance turned out to be surprisingly stable and well performing (as for Java).

AXIS-C++ was not able to finish all tests and we expect that this will improve over time. However it seems that AXIS-C++ HTTP transport is very inefficient as even for ping (echoVoid) method that has empty body it was 4x slower than gSOAP or XSOAP4 (and CPU usage is 1/3 indicating that there is lot of IO waits or some other blocking ...)

It seems that AXIS-Java has huge memory leak - test was not completed as JVM ran out of memory even though it was started with -Xmx1024m (1GB!) and it actually managed not only to take all memory but also all swap space leading to machine freezing which is very bad sign if you plans to run AXIS-Java based services for this kind of payloads ...

 

[BACK]


For more info contact Aleksander Slominski.

Valid XHTML 1.0!