STORM™ Benchmark for
Object-Relational Mapping Engines
by
Software Tree, LLC
INTRODUCTION
Upon the request of
some potential and existing customers, Software Tree has developed a
benchmark framework that tests the performance of object-relational
mapping (OR-Mapping) engines. The benchmark is called STORM™, which stands for Software Tree’s Object Relational Mapping
benchmark. Although there
are many different performance aspects of an OR-Mapping engine that can be
measured in different ways, we set the following major goals for the
STORM benchmark:
- Easy to understand, configure, and use
- Measure the OR-Mapping engine performance
for the two most common use cases – inserts and queries
- Produce meaningful results matrix
- Easily portable for different OR-Mapping
products
- Don't require a
web server or an application server to run the benchmark
This benchmark does
not prescribe any particular hardware or software setup. You can use any combination of
hardware, JVM, RDBMS and JDBC driver. The idea is to easily and quickly run the benchmark
programs and get meaningful results. We believe that by running the benchmark programs in
your particular environment and settings, you can get both absolute and
relative performance numbers, which will make sense for your particular
situation.
We hope that our
efforts on STORM benchmark will help the community in evaluating the two
most basic and important performance characteristics (inserts and
queries) of different OR-Mapping technologies and help us improve our
offerings.
OVERVIEW
The STORM benchmark
uses the instances of an “Account” class, which has ten attributes of different types and
maps into a database table with a record size of 100 bytes. The benchmark program does the
operations in the following order:
·
Initialize the OR-Mapping subsystem
·
Delete all the old database records
for the Account class
·
Create and insert configured number
of new Account objects in each transaction for configured number of
transactions (iterations)
·
Print the statistics for insert
operations
·
Query and print the total number of
Account objects in the database
·
Query configured number of Account
objects in each transaction for configured number of transactions
(iterations)
·
Print the statistics of the query
operations
This
high-level overview and the actual reference implementation of the
benchmark for JDX constitute the current specification of the STORM
benchmark.
CURRENT IMPLEMENTATIONS
Software Tree has
developed the STORM benchmark program for its JDX OR-Mapping engine. Software Tree has also ported the
benchmark for Hibernate. These implementations are available with the JDX evaluation kit in
their own subdirectories under the examples\STORMBenchmark
directory. Each subdirectory
is self-contained and comes with extensive documentation including ANT
scripts to configure and run the benchmark tests. Implementations for other
OR-Mapping engines may be added later on.
Your performance
numbers would depend upon your setup (including CPU, Memory, JDK version,
RDBMS, JDBC driver, etc.) but the results from running the STORM
benchmark programs using different OR-Mapping engines under similar setup
should give a reasonable indication of the relative efficiency and
scalability of those engines.
AVAILABILITY
STORM benchmark
package is shipped with the JDX evaluation kit, which can be downloaded
from Software Tree’s web site. The package is available under
the examples\STORMBenchmark directory,
FEEDBACK
Your suggestions and
contributions for improvements to STORM are welcome. Please send them to storm@softwaretree.com. We will try to incorporate them
in STORM as and when possible.
Software Tree, LLC
2953 Bunker Hill Lane, Suite 400
Santa Clara, CA 95054
(USA) |
