SOAP Benchmark Suite for Scientific Computing

Introduction
Authors
Why do we need benchmarks for SOAP?
Who will benefit?
Benchmark Suite
Download Test Driver and Run Benchmark
Performance Results
Results for new SOAP Toolkits
Future Work
Contact

NOTE: another version of this page is at http://grid.cs.binghamton.edu/projects/soap_bench/

Introduction

The SOAP protocol has emerged as the standard mechanism for exchanging information between various Web services. Web services have been adopted as the underlying architecture for Grid computing. Consequently, many applications will transport data via SOAP. The advantages of SOAP include extensibility, robustness and interoperability, but not performance. SOAP implementations have been developed in many languages including C, C++, Java, Python and C#. These toolkits differ in their design and supported performance optimizations.

We have designed a benchmark suite to compare and quantify the performance of various toolkits. Based on our experience of developing three different, independent toolkits (gSOAP, XSOAP/XSUL and bSOAP), we propose a benchmark suite of SOAP benchmarks that have been designed to exercise the performance of each toolkit for a wide range of representative use patterns.

Authors

The design and development of these benchmarks was done by a team of faculty and students from three different universities.

• Indiana University
  • Aleksander Slominski
  • Kenneth Chiu
• Florida State University
  • Robert van Engelen
• State University of New York (SUNY) at Binghamton
  • Madhusudhan Govindaraju
  • Michael J. Lewis
  • Pu Liu

Why do we need SOAP benchmarks?

1. The requirements of Web services based applications is diverse. As a result, toolkits are designed to incorporate various features including end-to-end performance, serialization and deserialization efficiency, small memory footprint, specific security requirements and scalability. The benchmark suite will help decide which toolkit is best suited for a particular application.
2. SOAP presents some unique design and implementation challenges which require smart techniques. The naive approach can lead to unacceptable performance penalties. The benchmark is designed to highlight clever techniques.
3. Many SOAP toolkits are currently being developed, in addition to the large number of existing ones.

Who Will Benefit?

Our work is designed to benefit both SOAP library developers and application programmers who use SOAP based Web services. We provide many insights into the design and development of SOAP toolkits thereby guiding SOAP developers in building toolkits with enhanced performance. Web service users can compare and contrast the performance and features of various toolkits, based on our benchmark suite, and select the one that best suites their needs.

Benchmark Suite

We propose a benchmark suite consisting of WSDL files with operations to test the performance for various features, including:

• Latency of a toolkit: the overhead imposed by a toolkit to send and receive a SOAP call when no parameters have to be serialized or deserialized.
• Base64: performance for various array sizes of base64 encoded data.
• Doubles: performance for various array sizes of floating point data.
• Strings: performance for various sizes of String data.
• Integers: performance for various sizes of integer data.
• Structs: performance for various sizes of structs.
• Linked-Lists: performance for various sizes of linked-lists.
• Multi-ref: performance in the presence of a large number of co-referenced objects.

Currently we have only arrays of bytes (BASE64 encoded), integers, doubles, and strings as representatives.

The following WSDL file contains operations for most of the above mentioned features.

Download Tests Driver And Run Tests

Test Driver

Use the driver to test a SOAP toolkit - it has simple to use command line client.

AXIS Java

AXIS Java tests source code is in axis_java

To run tests using AXIS, please make sure to do ant clean, and ant deploy in axis_java.

It is also necessary to modify WEB-INF\server-config.wsdd:

<parameter name="sendXsiTypes" value="false"/>
<parameter name="sendMultiRefs" value="false"/>

AXIS C++

AXIS C++ tests source code is in axis_cpp

At time of running tests (June 2004) AXIS C++ had very long way to go as far as usability is considered. Getting it to run required expert knowledge and access to libtool/autoconf/automake gurus and lot of effort to figure out all 20+ steps. Getting it to compile and run sample felt like accomplishing Mission Impossible - hopefully now it is easier :)

XSOAP/XSUL

Test driver contains already XSUL test service code.

Performance Results

This test was performed to determine suitability and limiting factors for using long running Web Services for Scientific Computing purposes. We conducted experiments to study the performance of the following toolkits: gSOAP 2.4, XSOAP4/XSUL 1.1, AXIS Java 1.2 (CVS 2004/06/11) with and without streaming enabled), Axis C++ (CVS 2004/05/28), and .NET 1.1.4322. The .NET framework was run on a Dell Dimension 4500 with Intel Pentium 4 2.26GHz processor, 1GB of DDR and 80GB Ultra ATA/100 hard drive. The other toolkits were run on dual processor 2.0 GHz Pentium 4 Xeon machines with 1 GB DDR SDRAM and a 15K RPM 18GB Ultra-160 SCSI drive running Debian Linux version 2.4.24. The machines were connected by a gigabit ethernet switch. gSOAP and AXIS C++ were compiled with gcc 2.95.4 with optimization flag ``-O2." AXIS Java and XSOAP4 were compiled with Java 1.4.2_04.

Discussion of results

See detailed report and paper Toward Characterizing the Performance of SOAP Toolkits (PDF 190KB) accepted for Publication in Procceedings of 5th IEEE/ACM International Workshop on Grid Computing, Pittsburgh, November 2004.

Graphs from paper:

• End-to-End Performance of Base64 data
• End-to-End Performance of Doubles
• End-to-End Performance of Ints
• End-to-End Performance of Strings
• Deserialization Performance of Base64 data
• Deserialization Performance of Doubles
• Deserialization of Performance of Doubles (focussed)
• Deserialization Performance of Ints
• Deserialization Performance of Strings
• Serialization Performance of Base64 data
• Serialization Performance of Base64 data (focused)
• Serialization Performance of Doubles
• Serialization Performance of Ints
• Serialization Performance of Strings

Loopback performance

This test was executed on a single machine. So, it shows performance results without network vagaries. Results can be viewed here

.NET and XSOAP4/XSUL

Want to send in your results?

If you would like to inform us of the results of your SOAP toolkit, and include the results in this space, please contact us.

Future Work

We plan to add to the benchmark suite to test the following features:

• Memory footprint of the toolkit and suitability for hand-held devices
• Compliance and performance for emerging Web service security features

Contact

For more information, please contact Aleksander Slominski and Madhusudhan Govindaraju