Navigating the Database Performance Landscape: A Comprehensive Guide to Database Benchmarking Suites (2026)

OLTP Database Benchmarking Suites

At the core of many database systems, OLTP (Online Transaction Processing) is the beating heart that powers the transactional interactions vital for daily business operations.

OLTP refers to a class of database workloads characterized by their emphasis on quick, low-latency transactions. These transactions typically involve frequent inserts, updates, and deletes of relatively small amounts of data. In essence, OLTP systems cater to the swift and concurrent processing of numerous transactions, mirroring the dynamic nature of applications like e-commerce platforms, banking systems, and airline reservation systems.

Typical characteristics of OLTP workloads include:

• High transaction rates with short execution paths
• Low-latency response requirements
• Frequent inserts, updates, and deletes
• Small, index-driven data access patterns
• Strong consistency and isolation guarantees
• High levels of concurrent client activity

In distributed or cloud-native architectures, additional factors such as replication topology, coordination overhead, and cross-node latency significantly influence observed performance.

Key Metrics in OLTP Benchmarking:

• Transaction Throughput (TPS): Measures the number of successfully committed transactions per unit of time, indicating the system's ability to handle a high transaction load.
• Response Time / Latency Distribution: Evaluates average latency as well as percentile metrics (e.g., p95, p99, p99.9) to capture tail behavior.
• Concurrency Scalability: Assesses how performance evolves as the number of concurrent clients increases, without compromising performance or data integrity.
• Consistency & Replication Impact: Measures the cost of synchronous replication, quorum writes/reads, and cross-node commit protocols.
• Resource Efficiency: Evaluates CPU, memory, and I/O utilization under sustained transactional load.

Sysbench

• Origin: Open-source, see Gitlab
• Purpose: Sysbench serves as a versatile benchmarking tool for assessing system performance under various workloads. Originally developed for MySQL, it has evolved to support multiple database engines.
• Real-world Use-Cases: Sysbench can be understood as a microbenchmark, which allows the generation of several different queries for benchmarking and stress-testing databases, making it a go-to tool for database administrators and developers. It's ideal for assessing general system performance and database scalability.
• Key Features: Flexible and extensible, Sysbench allows users to create custom test scenarios. It includes also CPU, memory, file I/O, and database benchmarks, making it a comprehensive solution for system-level testing.
• benchANT Integration: Yes

TPC-C (BenchBase)

• Origin: Transaction Processing Performance Council (TPC), now integrated into the BenchBase framework
• Purpose: TPC-C evaluates the performance of OLTP systems by simulating the workload of an order-entry environment with multiple users. It is one of the first and maybe the most famous TPC benchmark. It was integrated into the BenchBase framework by the Carnegie Mellon University. They also added several other database integrations. We recommend using this branch instead of the original TPC version.
• Real-world Use-Cases: Ideal in scenarios mirroring order processing systems, such as those found in retail and manufacturing industries.
• Key Features: TPC-C generates a mix of transactions, including new order creation, order status checks, and delivery updates. It stresses the system by simulating a busy, multi-user environment with varying transaction types.
• benchANT Integration: Yes

HammerDB - TPROC-C

• Origin: Transaction Processing Performance Council (TPC), GitHub HammerDB Project
• Purpose: HammerDB, with its TPROC-C benchmark, assesses the performance of database systems using the TPC-C standard. It simulates an order-entry environment with multiple users and complex transactions.
• Real-world Use-Cases: Valuable for organizations evaluating in scenarios resembling order processing and transaction-intensive environments.
• Key Features: The TPROC-C benchmark within HammerDB stresses databases with a mix of transactions, including order creation, order status checks, and delivery updates.
• benchANT Integration: No

TPC-E

• Origin: Transaction Processing Performance Council (TPC), see TPC.org
• Purpose: TPC-E benchmarks measure the performance of online transaction processing (OLTP) systems by simulating the activities of a brokerage firm. It focuses on complex, realistic transactions.
• Real-world Use-Cases: TPC-E is relevant for handling financial transactions, providing insights into how well a system can handle demanding OLTP workloads.
• Key Features: TPC-E emphasizes transactional complexity, including intricate order processing and financial calculations. This benchmark provides a more modern and challenging alternative to its predecessor, TPC-C.
• benchANT Integration: No

Twitter

• Origin: BenchBase
• Purpose: Part of the BenchBase suite, the Twitter benchmark simulates the workload of a social media platform, focusing on tweet creation, retrieval, and user interactions.
• Real-world Use-Cases: Relevant for assessing the performance of databases powering social media platforms, providing insights into how well a system handles high-volume, short-duration transactions.
• Key Features: The Twitter benchmark emulates user activities such as tweeting, retweeting, and following, simulating the dynamic and real-time nature of social media interactions.
• benchANT Integration: Yes

TATP (BenchBase)

• Origin: BenchBase, originally dveloped at Nokia, called TM1
• Purpose: TATP assesses the performance of databases handling telecommunications transaction processing.
• Real-world Use-Cases: Relevant for telecommunications scenarios, where high-throughput, low-latency transaction processing is crucial.
• Key Features: TATP benchmarks focus on complex transactional operations typical in telecommunications, including subscriber data updates and call detail record generation.
• benchANT Integration: Yes

Wikipedia (BenchBase)

• Origin: BenchBase, originally from OLTPbench
• Purpose: The Wikipedia benchmark, part of the BenchBase suite, simulates a scenario based on a collaborative content creation platform, stressing the database with various read and write operations.
• Real-world Use-Cases: Suitable for content management systems and collaborative platforms, such as wikis.
• Key Features: The Wikipedia benchmark simulates user interactions like page edits, searches, and revisions, providing insights into the efficiency of databases handling content-centric workloads.
• benchANT Integration: No

AuctionMark (BenchBase)

• Origin: BenchBase
• Purpose: AuctionMark, within the BenchBase suite, emulates an online auction platform's workload, stressing the database with bid submissions, auction status checks, and user interactions.
• Real-world Use-Cases: Relevant for evaluating the performance of databases supporting online auction platforms and e-commerce systems with dynamic bidding processes.
• Key Features: AuctionMark generates transactional scenarios including bidding, auction creation, and result retrieval, reflecting the complex interactions of online auction environments.
• benchANT Integration: No

Epinions.com (BenchBase)

• Origin: BenchBase, originally from OLTPbench
• Purpose: The Epinions.com benchmark, part of the BenchBase suite, is derived from the Epinions.com consumer review website. It is built on data from a previous study combined with additional statistics extracted from the website. The workload centers on users interacting with other users and writing reviews for items such as products.
• Real-world Use-Cases: Relevant for review platforms, social networks and other community-driven websites, where users rate items and build trust relationships with other users.
• Key Features: The workload consists of nine transactions: four interact with user records only, four with item records only, and one with all tables of the database. Users have an n-to-m relationship with items (reviews and ratings) and an n-to-m relationship with other users (trust). It is similar to the Twitter benchmark, but its many-to-many relationships are traversed using SQL joins instead of application-side joins, and its tables interact in more complex ways.
• benchANT Integration: No

Resource Stresser (BenchBase)

• Origin: BenchBase
• Purpose: The Resource Stresser benchmark is a microbenchmark for extreme resource contention scenarios, such as high CPU or memory loads.
• Real-world Use-Cases: Valuable for understanding how a database responds and maintains stability under resource-intensive conditions, aiding in capacity planning and resource optimization.
• Key Features: Resource Stresser applies extreme stress to a database system, evaluating its resilience and resource management capabilities under challenging conditions.
• benchANT Integration: No

SEATS (BenchBase)

• Origin: BenchBase
• Purpose: SEATS, a part of the BenchBase suite, simulates a scenario based on an airline reservation system.
• Real-world Use-Cases: Relevant for (airline) reservation and booking systems.
• Key Features: SEATS generates transactional scenarios, including seat reservations, ticket purchases, and flight status checks, offering insights into database efficiency in the context of airline operations.
• benchANT Integration: No

SIBench (BenchBase)

• Origin: BenchBase
• Purpose: SIBench, within the BenchBase suite, simulates a workload based on a stock trading platform.
• Real-world Use-Cases: Applicable for evaluating the performance of databases supporting stock trading and financial platforms, simulating high-throughput, transactional scenarios.
• Key Features: SIBench generates transactions related to stock trading, account management, and financial operations, providing insights into database efficiency in the context of financial markets.
• benchANT Integration: No

SmallBank (BenchBase)

• Origin: BenchBase, originally developed by Michael J. Cahill at Oracle
• Purpose: SmallBank, part of the BenchBase suite, emulates the workload of a small banking application, stressing the database with various financial transactions, account operations, and user interactions.
• Real-world Use-Cases: Suitable for assessing the performance of databases supporting small-scale banking applications, offering insights into transactional efficiency and data integrity.
• Key Features: SmallBank generates transactions like deposits, withdrawals, and fund transfers, simulating the transactional nature of small-scale banking operations.
• benchANT Integration: No

Voter (BenchBase)

• Origin: BenchBase, originally created from a VoltDB app
• Purpose: The Voter benchmark, within the BenchBase suite, simulates a workload based on an election voting system.
• Real-world Use-Cases: Relevant for assessing the performance of databases supporting election systems, providing insights into transactional efficiency during peak voting periods.
• Key Features: The Voter benchmark generates scenarios including voter registrations, ballot submissions, and result queries, simulating the transactional nature of election systems.
• benchANT Integration: No

YCSB (BenchBase)

• Origin: original by Yahoo! Research, integrated and adjusted in BenchBase
• Purpose: YCSB (Yahoo! Cloud Serving Benchmark), part of the BenchBase suite, is designed for benchmarking and comparing the performance of cloud-based and distributed databases. In comparison to the original YCSB, which we will explain in the NoSQL section, the operations of this implementation are packed into transactions.
• Real-world Use-Cases: Widely used for evaluating the efficiency of databases in cloud environments, helping organizations choose suitable solutions for distributed and scalable applications.
• Key Features: YCSB supports a variety of workloads, including read and write-intensive operations, providing a standardized approach for assessing the performance of cloud databases under different scenarios.
• benchANT Integration: No, benchANT uses its own YCSB fork with updated and a lot more database integrations, which originate from the original non-transactional YCSB.

PgBench

• Origin: PostgreSQL Community
• Purpose: PgBench is a benchmarking tool that comes bundled with the PostgreSQL database. It is designed to simulate various transactional workloads to assess the performance of PostgreSQL.
• Real-world Use-Cases: Widely used for benchmarking PostgreSQL installations, helping database administrators and developers understand the system's capabilities and identify areas for optimization.
• Key Features: PgBench supports multiple built-in scenarios, including simple read and write operations, making it a valuable tool for evaluating PostgreSQL performance in diverse usage scenarios.
• benchANT Integration: No

CockroachPKG

• Origin: Cockroach Labs
• Purpose: Cockroach-PKG, developed by Cockroach Labs, is a benchmarking suite specifically tailored for the CockroachDB distributed SQL database, evaluating its performance under various workloads from existing benchmarks like YCSB, TPC-C and so on. The benchmarking suite is integrated into the source code of Cockroach.
• Real-world Use-Cases: Applicable for organizations considering or utilizing CockroachDB as their distributed database solution, providing insights into its performance characteristics for several use cases, based on the selected benchmarking suite.
• Key Features: Cockroach-PKG includes tests for transactional workloads, schema changes, and distributed consistency, allowing users to assess CockroachDB's capabilities in a variety of scenarios.
• benchANT Integration: No

pgdistbench

• Origin: Xata.io – GitHub Repository
• Purpose: pgdistbench is a distributed PostgreSQL benchmarking tool designed to run standardized benchmarks at scale in Kubernetes environments. It provides orchestration and repeatability for benchmark execution across one or more PostgreSQL instances, including distributed load generation via multiple benchmark driver replicas.
• Real-world Use-Cases: Suitable for benchmarking Kubernetes-native PostgreSQL deployments (e.g., CloudNativePG), pre-existing PostgreSQL clusters, and Postgres-based platforms where the benchmark setup itself (deployment orchestration, runner scaling, and cluster-level stress behavior) is part of the evaluation.
• Key Features: Supports multiple benchmark workloads including TPC-C (OLTP), TPC-H (OLAP), and CH-BenCHmark (mixed OLTP/OLAP). Uses a two-component architecture: a bench driver (HTTP server that executes the benchmark against the system under test) and a k8 runner (CLI that deploys and configures systems under test, coordinates multiple bench driver replicas, and manages benchmark lifecycle phases such as prepare/run/cleanup). Supports scenario configuration via YAML or KCL (KusionStack Configuration Language) to enable validated and reusable benchmark definitions. Includes Kubernetes-focused stress modes that can exercise cluster behavior with dynamic PostgreSQL instance lifecycle management.
• benchANT Integration: No

Swingbench

• Origin: Developed and maintained by Dominic Giles – Project Page
• Purpose: Swingbench is a load generator and set of utilities for benchmarking database systems by generating transactional workloads and capturing response-time and throughput behavior. It supports both GUI and command-line execution and includes utilities to chart transaction rates and response-time distributions.
• Real-world Use-Cases: Commonly used to validate and stress-test database features and operational scenarios such as Oracle Real Application Clusters (RAC), standby databases, online maintenance operations (e.g., table rebuilds), and backup/recovery workflows. It is also used for migration validation, performance regression testing, and hardware sizing under controlled load.
• Key Features: Ships with multiple built-in benchmarks including OrderEntry, SalesHistory, StressTest, JSON, MovieStream, and “TPC-like” workloads (TPC-DS Like, TPC-H Like). OrderEntry (based on Oracle’s OE schema) is designed for continuous execution and introduces heavy contention on a small set of tables to stress interconnect and memory; it includes both JDBC and PL/SQL variants. SalesHistory (based on Oracle’s SH schema) targets complex read-only queries on large tables and supports scaling (e.g., 1GB to 1TB). The framework is Java-based (cross-platform) and provides an API for implementing custom benchmark workloads.
• benchANT Integration: No

Deprecated OLTP Database Benchmarking Suites

In recent years, numerous benchmarking suites have been created, particularly from university research, but were discontinued after a short time. The following benchmarking suites are no longer current or actively maintained. Running these benchmarks should be done with caution and is only possible for older database versions and drivers.

• FinTime: Benchmarking suite for OLTP finance applications for Oracle and DB2 developed by NYU around the year 2000.
• BenchFoundry: A suite developed by 2017 for consistency benchmarks and TPC-C-like benchmarks with workload traces.

HTAP Database Benchmarking Suites

Hybrid Transactional/Analytical Processing (HTAP) workloads emerge as a convergence of OLTP (Online Transactional Processing) and OLAP (Online Analytical Processing).

HTAP benchmarks execute simultaneous transactional processing and analytical queries. This stands in contrast to OLAP, which traditionally involves periodic batch processing for analytical tasks.

In consequence, HTAP workloads are mixed workloads with high levels of transactional activity and, at the same time, executing business analytics queries directly over production data.

Typical characteristics of HTAP workloads include:

• Concurrent execution of transactional and analytical queries
• Shared storage or execution layers across workload types
• Continuous data ingestion with near real-time analytical visibility
• Mixed read/write patterns with varying query complexity
• Resource contention across CPU, memory, network, and I/O
• Freshness requirements for analytical queries over recently updated data

Because HTAP systems remove traditional ETL boundaries, benchmarking must explicitly measure trade-offs between transactional throughput and analytical performance.

Key Metrics in HTAP Benchmarking:

• Transactional Throughput (TP): Measures committed transactions per unit of time while analytical queries are running.
• Analytical Query Latency (AP): Evaluates execution time of analytical queries under concurrent transactional load.
• Workload Interference Impact: Quantifies performance degradation caused by concurrent workload classes.
• Data Freshness / Visibility Delay: Measures how quickly analytical queries reflect newly committed transactional updates.
• Resource Isolation Efficiency: Assesses how effectively CPU, memory, and I/O resources are balanced across workload types.
• Scalability Under Mixed Load: Examines performance behavior as data volume, concurrency, and workload intensity increase simultaneously.

CH-benCHmark (BenchBase)

• Origin: BenchBase framework, but developed by the Technical University of Munich
• Purpose: CH-benCHmark, within the BenchBase suite, is a benchmark specifically designed to create an HTAP workload based on the BenchBase TPC-C implementation with the additional 22 analytical queries of the TPC-H.
• Real-world Use-Cases: Relevant for organizations which have simultaneous OLTP and OLAP workloads on the same database systems in eCommerce-similar scenarios, selling and analyzing the orders.
• Key Features: The CH-benCHmark allows the parallel execution of transactional and analytical queries to the same database and database tables, simulating a hybrid workload which can often be seen in monolithic applications.
• benchANT Integration: No

HATtrick

• Origin: University of Wisoncsin, based on TPC-C and the SSB benchmark
• Purpose: HATtrick was created by researchers at the University of Wisconsin to define a new HTAP benchmark which covers the gap between pure OLTP and OLAP workloads.
• Real-world Use-Cases: The HTAP workload that is created by the HATtrick benchmark is similar to eCommerce applications with integrated analytical queries to generate a mixture of OLTP and OLAP.
• Key Features: A unique features of this benchmark is the “freshness” KPI for the analytical queries, which describes how new the results of the analytical queries in reference to the transactional write queries.
• benchANT Integration: No

Web3Bench

• Origin: Community project – GitHub Repository
• Purpose: Web3Bench is a benchmark suite designed for evaluating database systems in Web3 and decentralized application scenarios. It provides a unified workload model that combines transactional and analytical tasks, reflecting hybrid access patterns commonly observed in blockchain-based systems.
• Real-world Use-Cases: Suitable for benchmarking databases used in decentralized applications, blockchain analytics platforms, and systems that must handle both high-frequency transactional updates (e.g., token transfers) and analytical queries over blockchain state.
• Key Features: Based on a simplified Ethereum-inspired data model including blocks, transactions, smart contracts, and token transfers. Derived from a 20GB real Ethereum dataset and extended through a proprietary data generator to support scalable dataset sizes. Workloads are categorized by latency requirements (real-time, online serving, and batch processing) and include explicit latency limits for real-time and online queries, as well as throughput measurement for batch tasks. A specialized benchmark driver measures both latency and throughput across mixed workload scenarios.
• benchANT Integration: No

HyBench-2024

• Origin: Academic project (Tsinghua University) – GitHub Repository | Paper (VLDB 2024)
• Purpose: HyBench is a benchmark suite designed to evaluate HTAP (Hybrid Transactional and Analytical Processing) database systems under tightly coupled mixed workloads. It models concurrent transactional (TP) and analytical (AP) execution and explicitly measures workload interference, analytical freshness, and hybrid performance trade-offs.
• Real-world Use-Cases: Suitable for evaluating operational analytics platforms where transactional updates and analytical queries must operate on the same dataset with minimal delay. Representative scenarios include financial systems, e-commerce platforms with live dashboards, and data platforms requiring continuous ingestion with near real-time analytical visibility.
• Key Features: HyBench defines distinct workload components for TP, AP, and XP (cross-phase evaluation). The benchmark measures transactional throughput (TP), analytical performance (AP), mixed-workload behavior under concurrent execution, and freshness metrics, quantifying how quickly analytical queries reflect recent transactional updates. It supports configurable scale factors (e.g., 1x, 10x, 100x), adjustable TP/AP concurrency levels, workload composition ratios, and independent JDBC configurations for transactional and analytical clients. The execution workflow includes data generation, loading, index building, and coordinated mixed-workload runs. Implemented in Java (Java 17+, Maven-based), HyBench emphasizes reproducible configuration through structured parameter files and clearly documented execution procedures to ensure methodological consistency across systems.
• benchANT Integration: No

CostBench

• Origin: ClickHouse – GitHub Repository | Blog: Introducing CostBench
• Purpose: CostBench is an open benchmark for the cost-performance of real-time analytics along the complete analytics path. Instead of starting after the data has been bulk-loaded, as static query benchmarks do, it measures the work and cost required to make continuously arriving data query-ready and to serve analytical queries over it while ingestion and maintenance are still running. We list CostBench in the HTAP category as it combines a continuous write workload with concurrent analytical queries over the same, constantly changing data and explicitly measures data freshness. However, it is not a transactional workload: the write side consists of high-volume, append-only ingestion instead of OLTP-style transactions with updates, deletes and transactional guarantees.
• Real-world Use-Cases: Relevant for selecting a cloud data warehouse or real-time analytics platform for live dashboards and drill-down analysis over streaming data, e.g. market data, observability or product analytics, where both the query latency and the total cost of keeping the data fresh matter.
• Key Features: The current full-path workload replays NBBO-style stock-market quotes (about 113 billion rows, 651 GB of compressed Parquet files) at a fixed target rate of 1 million events per second. The dataset is not open data: it was captured from the commercial market-data provider Massive and may not be redistributed, so reproducing the benchmark requires a licensed quotes dataset with the same schema. While ingest continues, the providers maintain a query-ready raw layout for two drill-down queries and a derived aggregate for four dashboard queries, both executed on a fixed schedule. Besides ingest progress and query latency under active ingestion, CostBench captures the freshness of the pre-aggregated data and the complete cost of the fresh data path, including provider-specific components such as background refresh compute, serverless ingestion services or a broker layer. Results are combined into a single score, (fresh-data-path cost + matched query cost) × accumulated query runtime (lower is better), with ClickHouse Cloud as the reference system for pairwise comparisons with Snowflake, Google BigQuery, Amazon Redshift Serverless and Databricks Serverless SQL. Query observations are aligned by ingested row progress, and workloads, pricing assumptions, raw results and cost calculations are published for review. In addition, a legacy read-side benchmark compares the query cost-performance of the 43 ClickBench queries over pre-loaded data at 1B, 10B and 100B rows. The repository is published under the CC BY-NC-SA 4.0 license, which excludes commercial use.
• benchANT Integration: No

Deprecated HTAP Database Benchmarking Suites

• HTAPBench: A parallelized workload from TPC-C and TPC-H to create an HTAP workload, which was developed in 2019 in a university environment, but has not been updated recently. It follows a similar approach as the CH-benCHmark.

Full-Text Search Database Benchmarking Suites

Full-text search workloads retrieve documents based on their textual content instead of exact key or attribute matches. They power site and product search, document management systems, log analytics and observability platforms, and are served by dedicated search engines such as Elasticsearch or OpenSearch, by search libraries such as Apache Lucene or Tantivy, and increasingly by search extensions of general-purpose database management systems.

Under the hood, these systems rely on inverted indexes, text analysis (tokenization, stemming, stop words) and relevance scoring functions such as BM25. Their workloads therefore differ significantly from the CRUD-style workloads of the NoSQL category as well as from the scan- and aggregation-heavy workloads of the OLAP category. Semantic search over embeddings is covered in the Vector Benchmarking section, while hybrid search combines both worlds.

Typical characteristics of full-text search workloads include:

• Indexing of large document collections with text analysis
• Term, boolean (intersection, union, negation) and phrase queries
• Ranking of results by relevance (top-k retrieval) in addition to counting matches
• Continuous indexing and segment merging while queries are served
• Combinations of text search with filters, aggregations and facets

Key Metrics in Full-Text Search Benchmarking:

• Query Latency per Query Type: Measures response times for term, boolean and phrase queries, including percentile distributions.
• Query Throughput: Evaluates the number of search requests per second under increasing client concurrency.
• Indexing Throughput: Measures how many documents per second can be ingested and made searchable.
• Index Size & Resource Usage: Quantifies storage footprint, memory and CPU consumption of the index.
• Relevance: Evaluates the quality of the returned results. As with recall in vector benchmarking, performance results should only be compared for equivalent result quality.

Rally

• Origin: Elastic, see GitHub and the workload definitions (tracks)
• Purpose: Rally is the official macrobenchmarking framework for Elasticsearch, developed by Elastic. It covers the complete benchmarking lifecycle from cluster setup to result comparison.
• Real-world Use Cases: Valuable for evaluating Elasticsearch deployments for full-text search, log analytics, observability and security analytics, e.g. for capacity planning, version upgrades and configuration tuning.
• Key Features: Rally can set up and tear down Elasticsearch clusters or run against existing clusters. Workloads are defined as so-called tracks, and a large set of ready-to-use tracks with real-world datasets (e.g. geonames, HTTP logs, NYC taxis, StackOverflow posts) is maintained separately. Rally records indexing and query metrics such as throughput, service time and latency percentiles, supports telemetry devices to capture low-level system metrics, and can compare the results of different benchmark runs. It puts a strong emphasis on reproducible benchmark results.
• benchANT Integration: No

Elastic-benchmark

• Origin: Elasticsearch, see GitHub
• Purpose: Elastic-benchmark is a benchmarking tool designed for Elasticsearch and OpenSearch, focusing on evaluating cluster performance and scalability under a search-related workload.
• Real-world Use Cases: Valuable for assessing the performance of Elastic and OpenSearch clusters in scenarios involving full-text search, log analytics, and real-time data processing.
• Key Features: Elastic-benchmark is a Rally-based setup for a side-by-side comparison of Elasticsearch and OpenSearch clusters on Kubernetes. It supports customizable workload configurations, including indexing, querying, and data retrieval operations.
• benchANT Integration: No

OpenSearch Benchmark

• Origin: OpenSearch Project – GitHub Repository
• Purpose: OpenSearch Benchmark is a performance evaluation framework designed specifically for OpenSearch clusters. It enables structured and reproducible benchmarking of indexing and search workloads, supporting both comparative performance analysis and regression testing across OpenSearch versions. Originally forked from Rally, it is the official benchmarking tool of the OpenSearch project.
• Real-world Use Cases: Suitable for evaluating search and log analytics deployments, observability platforms, security analytics systems, and document search engines built on OpenSearch. It is commonly used for performance regression testing, capacity planning, infrastructure validation, and tuning cluster configurations under realistic query and indexing workloads.
• Key Features: OpenSearch Benchmark supports:
  • Automated provisioning and teardown of OpenSearch clusters
  • Execution of predefined and custom benchmark workloads
  • Management of datasets and benchmark specifications across OpenSearch versions
  • Performance result recording and comparison
  • Integration of telemetry devices to capture low-level system metrics
  • Reproducible benchmarking workflows

The framework allows users to define custom workloads tailored to specific indexing patterns, query mixes, and concurrency levels. Telemetry components can collect system-level metrics to help identify bottlenecks and performance regressions. Emphasis is placed on methodological reproducibility to ensure consistent and comparable results across benchmark runs.

• benchANT Integration: No

FTSB

• Origin: Redis, see GitHub
• Purpose: ftsb (Full Text Search Benchmark) is a benchmarking tool designed to evaluate the performance of full-text search engines, focusing on indexing and query processing efficiency.
• Real-world Use Cases: Relevant for assessing the performance of full-text search engines in applications such as document management systems, content indexing, and search-based applications.
• Key Features: ftsb supports customizable workload profiles for indexing and querying large volumes of textual data. It provides metrics for evaluating indexing speed, query response time, and resource utilization of full-text search engines. Currently, only RediSearch is supported.
• benchANT Integration: No

Search Benchmark, the Game

• Origin: Originally created by Jason Wolfe, continued by the Tantivy/Quickwit team, see GitHub and the published results. SereneDB maintains a fork that adds its IResearch search library.
• Purpose: Search Benchmark, the Game is a standardized benchmark for comparing the query performance of search engine libraries. It addresses both users comparing different libraries and library developers looking for optimization opportunities.
• Real-world Use Cases: Relevant for selecting or optimizing the search library underneath a search engine or a database with full-text search capabilities. As it benchmarks embedded libraries such as Apache Lucene, Tantivy, PISA, Bleve or IResearch rather than complete search engines or database systems, aspects such as networking, distribution and concurrency are not covered.
• Key Features: The corpus is the English Wikipedia, and the queries are derived from the AOL query dataset. Each query is executed as intersection, union and phrase query, complemented by negated and term queries with different term frequencies. Each query type is measured with three collection modes: counting all matches, retrieving the top 10 documents by BM25 score, and both combined. The benchmark runs single-threaded on warmed-up engines and keeps the best of 10 runs, and the results can be explored in an interactive web view. New engines can be added by implementing a small Makefile-based interface for building the index and serving queries via stdin/stdout.
• benchANT Integration: No

TextBench

• Origin: ClickHouse, see GitHub
• Purpose: TextBench is a reproducible benchmark that compares ClickHouse and Elasticsearch for full-text search workloads over log data at scale. It aims for a fair comparison by focusing exclusively on capabilities available in both systems, measuring query latency and storage efficiency.
• Real-world Use Cases: Relevant for log analytics and observability platforms, where log messages are searched for tokens and combined with structured filters and aggregations, e.g. for troubleshooting or dashboards.
• Key Features: The dataset consists of synthetic OpenTelemetry logs generated by the OpenTelemetry demo application, with the log message as full-text indexed field. It is publicly available as Parquet files in an S3 bucket and is benchmarked at scales of 1, 10 and 50 billion rows. The nine queries cover single-term and boolean (AND/OR) token searches, full-text search combined with structured predicates (service, severity, time range), and aggregations driven by full-text filters such as faceting, GROUP BY and time histograms. Features that are only available in Elasticsearch, such as phrase matching and wildcards, are excluded. After ingestion, the service is restarted and each query is executed three times to capture cold and hot runtimes; in addition, the on-disk sizes of the data and the text index are reported. The published results were measured on single-node AWS m6i.8xlarge instances.
• benchANT Integration: No

SearchBench

• Origin: SereneDB, see GitHub and the published results. The queries are partly derived from TextBench, and the runner design follows ClickBench.
• Purpose: SearchBench is a benchmark for full-text search engines on log-shaped data. In contrast to library-level benchmarks, it compares complete systems end to end, from data loading and index building to query execution.
• Real-world Use Cases: Relevant for log analytics, observability and security analytics platforms, where large volumes of machine-generated log data need to be searched and aggregated. It is particularly useful for comparing dedicated search engines with databases that offer full-text search capabilities.
• Key Features: The workload consists of 92 queries: Q01–Q83 cover full-text search and aggregations, and Q84–Q92 cover joins. In addition to the TextBench-derived queries, which allow comparisons with the TextBench leaderboard, SearchBench adds BM25-scored top-k queries. The dataset is the public, synthetic OpenTelemetry logs corpus of TextBench in Parquet format, available in scales from 1 to 50 billion rows plus small smoke-test slices. Each query is executed three times (one cold and two hot runs), with an engine restart and cleared OS caches between queries; in addition, load time and on-disk data size are reported. Engines are integrated via a small set of scripts (install, start, stop, check, load, query, data-size), and adapters exist for ArangoDB, ClickHouse, CrateDB, Elasticsearch, Manticore Search, OpenSearch, ParadeDB, PostgreSQL, Quickwit, SereneDB, TimescaleDB and VictoriaLogs. All published results were measured on the same GCP instance type (n2-standard-32). The results of the different engines are not validated against each other for correctness.
• benchANT Integration: No