Software Tree, Inc.

STORM™ Benchmark for Object-Relational Mapping Engines by

INTRODUCTION
Upon the request of some potential and existing customers, Software Tree has developed a benchmark framework that tests the performance of object-relational mapping (OR-Mapping) engines. The benchmark is called STORM™, which stands for Software Tree’s Object Relational Mapping benchmark. Although there are many different performance aspects of an OR-Mapping engine that can be measured in different ways, we set the following major goals for the STORM benchmark:

  • Easy to understand, configure, and use
  • Measure the OR-Mapping engine performance for the two most common use cases – inserts and queries
  • Produce meaningful results matrix
  • Easily portable for different OR-Mapping products
  • Don't require a web server or an application server to run the benchmark

This benchmark does not prescribe any particular hardware or software setup. You can use any combination of hardware, JVM, RDBMS and JDBC driver. The idea is to easily and quickly run the benchmark programs and get meaningful results. We believe that by running the benchmark programs in your particular environment and settings, you can get both absolute and relative performance numbers, which will make sense for your particular situation.

We hope that our efforts on STORM benchmark will help the community in evaluating the two most basic and important performance characteristics (inserts and queries) of different OR-Mapping technologies and help us improve our offerings.

OVERVIEW
The STORM benchmark uses the instances of an “Account” class, which has ten attributes of different types and maps into a database table with a record size of 100 bytes. The benchmark program does the operations in the following order:

  • Initialize the OR-Mapping subsystem
  • Delete all the old database records for the Account class
  • Create and insert configured number of new Account objects in each transaction for configured number of transactions (iterations)
  • Print the statistics for insert operations
  • Query and print the total number of Account objects in the database
  • Query configured number of Account objects in each transaction for configured number of transactions (iterations)
  • Print the statistics of the query operations

This high-level overview and the actual reference implementation of the benchmark for JDX constitute the current specification of the STORM benchmark.

CURRENT IMPLEMENTATIONS
Software Tree has developed the STORM benchmark program for its JDX OR-Mapping engine. Software Tree has also ported the benchmark for Hibernate. These implementations are available with the JDX evaluation kit in their own subdirectories under the examples\STORMBenchmark directory. Each subdirectory is self-contained and comes with extensive documentation including ANT scripts to configure and run the benchmark tests. Implementations for other OR-Mapping engines may be added later on.

Your performance numbers would depend upon your setup (including CPU, Memory, JDK version, RDBMS, JDBC driver, etc.) but the results from running the STORM benchmark programs using different OR-Mapping engines under similar setup should give a reasonable indication of the relative efficiency and scalability of those engines.

AVAILABILITY
STORM benchmark package is shipped with the JDX evaluation kit, which can be downloaded from Software Tree’s web site. The package is available under the examples\STORMBenchmark directory,

FEEDBACK
Your suggestions and contributions for improvements to STORM are welcome. Please send them to storm@softwaretree.com. We will try to incorporate them in STORM as and when possible.

Software Tree, LLC
2953 Bunker Hill Lane, Suite 400
Santa Clara, CA 95054 (USA)