Navigating the Database Performance Landscape: A Comprehensive Guide to Database Benchmarking Suites (2024)

OLTP Database Benchmarking Suites

At the core of many database systems, OLTP (Online Transaction Processing) is the beating heart that powers the transactional interactions vital for daily business operations.

OLTP refers to a class of database workloads characterized by their emphasis on quick, low-latency transactions. These transactions typically involve frequent inserts, updates, and deletes of relatively small amounts of data. In essence, OLTP systems cater to the swift and concurrent processing of numerous transactions, mirroring the dynamic nature of applications like e-commerce platforms, banking systems, and airline reservation systems.

Key Metrics in OLTP Benchmarking:

• Transaction Throughput: Measures the number of transactions processed per unit of time, indicating the system's ability to handle a high transaction load.
• Response Time/Latency: Evaluate the time taken to complete a transaction, emphasizing the system's responsiveness to user requests.
• Concurrency Scalability: Assesses how well the database system handles an increasing number of concurrent transactions without compromising performance or data integrity.

Sysbench

• Origin: Open-source, see Gitlab
• Purpose: Sysbench serves as a versatile benchmarking tool for assessing system performance under various workloads. Originally developed for MySQL, it has evolved to support multiple database engines.
• Real-world Use-Cases: Sysbench can be understood as a microbenchmark, which allows the generation of several different queries for benchmarking and stress-testing databases, making it a go-to tool for database administrators and developers. It's ideal for assessing general system performance and database scalability.
• Key Features: Flexible and extensible, Sysbench allows users to create custom test scenarios. It includes also CPU, memory, file I/O, and database benchmarks, making it a comprehensive solution for system-level testing.
• benchANT Integration: Yes

TPC-C (BenchBase)

• Origin: Transaction Processing Performance Council (TPC), now integrated into the BenchBase framework
• Purpose: TPC-C evaluates the performance of OLTP systems by simulating the workload of an order-entry environment with multiple users. It is one of the first and maybe the most famous TPC benchmark. It was integrated into the BenchBase framework by the Carnegie Mellon University. They also added several other database integrations. We recommend using this branch instead of the original TPC version.
• Real-world Use-Cases: Ideal in scenarios mirroring order processing systems, such as those found in retail and manufacturing industries.
• Key Features: TPC-C generates a mix of transactions, including new order creation, order status checks, and delivery updates. It stresses the system by simulating a busy, multi-user environment with varying transaction types.
• benchANT Integration: Yes

HammerDB - TPROC-C

• Origin: Transaction Processing Performance Council (TPC), GitHub HammerDB Project
• Purpose: HammerDB, with its TPROC-C benchmark, assesses the performance of database systems using the TPC-C standard. It simulates an order-entry environment with multiple users and complex transactions.
• Real-world Use-Cases: Valuable for organizations evaluating in scenarios resembling order processing and transaction-intensive environments.
• Key Features: The TPROC-C benchmark within HammerDB stresses databases with a mix of transactions, including order creation, order status checks, and delivery updates.
• benchANT Integration: No

TPC-E

• Origin: Transaction Processing Performance Council (TPC), see TPC.org
• Purpose: TPC-E benchmarks measure the performance of online transaction processing (OLTP) systems by simulating the activities of a brokerage firm. It focuses on complex, realistic transactions.
• Real-world Use-Cases: TPC-E is relevant for handling financial transactions, providing insights into how well a system can handle demanding OLTP workloads.
• Key Features: TPC-E emphasizes transactional complexity, including intricate order processing and financial calculations. This benchmark provides a more modern and challenging alternative to its predecessor, TPC-C.
• benchANT Integration: No

Twitter

• Origin: BenchBase
• Purpose: Part of the BenchBase suite, the Twitter benchmark simulates the workload of a social media platform, focusing on tweet creation, retrieval, and user interactions.
• Real-world Use-Cases: Relevant for assessing the performance of databases powering social media platforms, providing insights into how well a system handles high-volume, short-duration transactions.
• Key Features: The Twitter benchmark emulates user activities such as tweeting, retweeting, and following, simulating the dynamic and real-time nature of social media interactions.
• benchANT Integration: Yes

TATP (BenchBase)

• Origin: BenchBase, originally dveloped at Nokia, called TM1
• Purpose: TATP assesses the performance of databases handling telecommunications transaction processing.
• Real-world Use-Cases: Relevant for telecommunications scenarios, where high-throughput, low-latency transaction processing is crucial.
• Key Features: TATP benchmarks focus on complex transactional operations typical in telecommunications, including subscriber data updates and call detail record generation.
• benchANT Integration: Yes

Wikipedia (BenchBase)

• Origin: BenchBase, originally from OLTPbench
• Purpose: The Wikipedia benchmark, part of the BenchBase suite, simulates a scenario based on a collaborative content creation platform, stressing the database with various read and write operations.
• Real-world Use-Cases: Suitable for content management systems and collaborative platforms, such as wikis.
• Key Features: The Wikipedia benchmark simulates user interactions like page edits, searches, and revisions, providing insights into the efficiency of databases handling content-centric workloads.
• benchANT Integration: No

Epinions.com (BenchBase)

• Origin: BenchBase
• Purpose: AuctionMark, within the BenchBase suite, emulates an online auction platform's workload, stressing the database with bid submissions, auction status checks, and user interactions.
• Real-world Use-Cases: Relevant for evaluating the performance of databases supporting online auction platforms and e-commerce systems with dynamic bidding processes.
• Key Features: AuctionMark generates transactional scenarios including bidding, auction creation, and result retrieval, reflecting the complex interactions of online auction environments.
• benchANT Integration: No

Resource Stresser (BenchBase)

• Origin: BenchBase
• Purpose: The Resource Stresser benchmark is a microbenchmark for extreme resource contention scenarios, such as high CPU or memory loads.
• Real-world Use-Cases: Valuable for understanding how a database responds and maintains stability under resource-intensive conditions, aiding in capacity planning and resource optimization.
• Key Features: Resource Stresser applies extreme stress to a database system, evaluating its resilience and resource management capabilities under challenging conditions.
• benchANT Integration: No

SEATS (BenchBase)

• Origin: BenchBase
• Purpose: SEATS, a part of the BenchBase suite, simulates a scenario based on an airline reservation system.
• Real-world Use-Cases: Relevant for (airline) reservation and booking systems.
• Key Features: SEATS generates transactional scenarios, including seat reservations, ticket purchases, and flight status checks, offering insights into database efficiency in the context of airline operations.
• benchANT Integration: No

SIBench (BenchBase)

• Origin: BenchBase
• Purpose: SIBench, within the BenchBase suite, simulates a workload based on a stock trading platform.
• Real-world Use-Cases: Applicable for evaluating the performance of databases supporting stock trading and financial platforms, simulating high-throughput, transactional scenarios.
• Key Features: SIBench generates transactions related to stock trading, account management, and financial operations, providing insights into database efficiency in the context of financial markets.
• benchANT Integration: No

SmallBank (BenchBase)

• Origin: BenchBase, originally developed by Michael J. Cahill at Oracle
• Purpose: SmallBank, part of the BenchBase suite, emulates the workload of a small banking application, stressing the database with various financial transactions, account operations, and user interactions.
• Real-world Use-Cases: Suitable for assessing the performance of databases supporting small-scale banking applications, offering insights into transactional efficiency and data integrity.
• Key Features: SmallBank generates transactions like deposits, withdrawals, and fund transfers, simulating the transactional nature of small-scale banking operations.
• benchANT Integration: No

Voter (BenchBase)

• Origin: BenchBase, originally created from a VoltDB app
• Purpose: The Voter benchmark, within the BenchBase suite, simulates a workload based on an election voting system.
• Real-world Use-Cases: Relevant for assessing the performance of databases supporting election systems, providing insights into transactional efficiency during peak voting periods.
• Key Features: The Voter benchmark generates scenarios including voter registrations, ballot submissions, and result queries, simulating the transactional nature of election systems.
• benchANT Integration: No

YCSB (BenchBase)

• Origin: original by Yahoo! Research, integrated and adjusted in BenchBase
• Purpose: YCSB (Yahoo! Cloud Serving Benchmark), part of the BenchBase suite, is designed for benchmarking and comparing the performance of cloud-based and distributed databases. In comparison to the original YCSB, which we will explain in the NoSQL section, the operations of this implementation are packed into transactions.
• Real-world Use-Cases: Widely used for evaluating the efficiency of databases in cloud environments, helping organizations choose suitable solutions for distributed and scalable applications.
• Key Features: YCSB supports a variety of workloads, including read and write-intensive operations, providing a standardized approach for assessing the performance of cloud databases under different scenarios.
• benchANT Integration: No, benchANT uses its own YCSB fork with updated and a lot more database integrations, which originate from the original non-transactional YCSB.

PgBench

• Origin: PostgreSQL Community
• Purpose: PgBench is a benchmarking tool that comes bundled with the PostgreSQL database. It is designed to simulate various transactional workloads to assess the performance of PostgreSQL.
• Real-world Use-Cases: Widely used for benchmarking PostgreSQL installations, helping database administrators and developers understand the system's capabilities and identify areas for optimization.
• Key Features: PgBench supports multiple built-in scenarios, including simple read and write operations, making it a valuable tool for evaluating PostgreSQL performance in diverse usage scenarios.
• benchANT Integration: No

CockroachPKG

• Origin: Cockroach Labs
• Purpose: Cockroach-PKG, developed by Cockroach Labs, is a benchmarking suite specifically tailored for the CockroachDB distributed SQL database, evaluating its performance under various workloads from existing benchmarks like YCSB, TPC-C and so on. The benchmarking suite is integrated into the source code of Cockroach.
• Real-world Use-Cases: Applicable for organizations considering or utilizing CockroachDB as their distributed database solution, providing insights into its performance characteristics for several use cases, based on the selected benchmarking suite.
• Key Features: Cockroach-PKG includes tests for transactional workloads, schema changes, and distributed consistency, allowing users to assess CockroachDB's capabilities in a variety of scenarios.
• benchANT Integration: No

Deprecated OLTP Database Benchmarking Suites

In recent years, numerous benchmarking suites have been created, particularly from university research, but were discontinued after a short time. The following benchmarking suites are no longer current or actively maintained. Running these benchmarks should be done with caution and is only possible for older database versions and drivers.

• FinTime: Benchmarking suite for OLTP finance applications for Oracle and DB2 developed by NYU around the year 2000.
• BenchFoundry: A suite developed by 2017 for consistency benchmarks and TPC-C-like benchmarks with workload traces.

HTAP Database Benchmarking Suites

Hybrid Transactional/Analytical Processing (HTAP) workloads emerge as a convergence of OLTP (Online Transactional Processing) and OLAP (Online Analytical Processing).

HTAP benchmarks execute simultaneous transactional processing and analytical queries. This stands in contrast to OLAP, which traditionally involves periodic batch processing for analytical tasks.

In consequence, HTAP workloads are mixed workloads with high levels of transactional activity and, at the same time, executing business analytics queries directly over production data.

CH-benCHmark (BenchBase)

• Origin: BenchBase framework, but developed by the Technical University of Munich
• Purpose: CH-benCHmark, within the BenchBase suite, is a benchmark specifically designed to create an HTAP workload based on the BenchBase TPC-C implementation with the additional 22 analytical queries of the TPC-H.
• Real-world Use-Cases: Relevant for organizations which have simultaneous OLTP and OLAP workloads on the same database systems in eCommerce-similar scenarios, selling and analyzing the orders.
• Key Features: The CH-benCHmark allows the parallel execution of transactional and analytical queries to the same database and database tables, simulating a hybrid workload which can often be seen in monolithic applications.
• benchANT Integration: No

HATtrick

• Origin: University of Wisoncsin, based on TPC-C and the SSB benchmark
• Purpose: HATtrick was created by researchers at the University of Wisconsin to define a new HTAP benchmark which covers the gap between pure OLTP and OLAP workloads.
• Real-world Use-Cases: The HTAP workload that is created by the HATtrick benchmark is similar to eCommerce applications with integrated analytical queries to generate a mixture of OLTP and OLAP.
• Key Features: A unique features of this benchmark is the “freshness” KPI for the analytical queries, which describes how new the results of the analytical queries in reference to the transactional write queries.
• benchANT Integration: No

Deprecated HTAP Database Benchmarking Suites

• HTAPBench: A parallelized workload from TPC-C and TPC-H to create an HTAP workload, which was developed in 2019 in a university environment, but has not been updated recently. It follows a similar approach as the CH-benCHmark.