117 Independent Database and Cloud Performance Benchmarks

Why Performance Matters!

Welcome to a groundbreaking whitepaper that unveils the untold truths behind database and cloud performance benchmarks. In an era dominated by data-driven decision making and cloud adoption, understanding the intricate details of database and infrastructure performance is crucial for organizations seeking to gain a competitive edge. This whitepaper, based on real-world project experience, will empower you with valuable, untold insights.

Enter performance benchmarks, a scientific approach to measuring, comparing, and evaluating the performance of different databases and cloud platforms. These benchmarks act as a compass, guiding you towards informed decision-making and ensuring optimal utilization of your resources.

While database vendors and cloud providers entice you with claims of unparalleled performance and linear scalability, they conveniently omit critical information that could impact your business's efficiency and cost-effectiveness. This whitepaper is designed to expose these hidden truths, empowering you with knowledge that will level the playing field.

117 Comprehensive Benchmarks:
Empowering You with Real-World Data

Drawing from a vast array of real-world projects, this whitepaper features an astounding 117 independent performance benchmarks. Each benchmark scrutinizes a specific aspect of database and cloud performance, providing you with practical insights and real-world examples. These benchmarks cover a wide range of crucial parameters, including latency, throughput, scalability, costs, and performance/cost ratios.

By leveraging the power of these benchmarks, you can make data-driven decisions that align with your business goals. Rather than relying on marketing hype or biased vendor claims, you can understand the inhomogeneity of database and cloud infrastructure products and learn how to measure what is important.

Knowledge is power, and the insights contained within these benchmarks will equip you with the information necessary to make intelligent, evidence-based decisions. Embrace transparency, demand performance, and position your tech stack for the growing amount of data in the front position.

First steps towards Database Performance

Database performance is a selling point since the rise of database management systems in 1990. There is nearly no database producer who is not doing marketing with words like “fastest”, “scaling” or “real-time”. Also, nearly every vendor is publishing performance benchmark reports, done by their own engineering team or third parties. At benchANT, our mission is to bring independent and reliable performance data for databases, Database-as-a-Service products, and cloud resources to everyone. Our approach is based on a scientifically approved benchmarking methodology and toolset and enables reliable and efficient measurement automation. More information about benchANT and our benchmarking process is available on our website.

The most important things when doing database benchmarks are:

• Relevance
• Reproducibility
• Fairness
• Verifiability
• Usability

In the following chapters we show you a large amount of highly interesting information for the most important KPIs for database performance measurement. Based on that data, we explain why and what is important. And sometimes, you will see surprising results.

Throughput for Popular NoSQL Databases

Data-intense application need to handle not only a huge amount of stored data, but also many database operations per second to store and read data. This KPI, the so-called Throughput, is measured in database operations (or transactions) per second.

Let’s compare the throughput of some modern popular NoSQL databases with a simple CRUD workload when hosted on AWS EC2.

Transactional Throughput for Relational Databases

For relational databases the throughput is often measured in transactions per second/per hour. Hereby, multiple database operations are bundled into database transactions.

Let’s compare the throughput of some open-source relational databases for a more complex and transactional workload.

Latency: The Real-Time Experience

While throughput is very important to handle all operations, many applications also need a real-time experience with low latency. As applications often consist of several layers, the database latency is only one part of the total latency. Database technologies usually show different latencies for specific database operations, like writing, reading, updating....

Let’s have a look at the NoSQL scenario from before, but focus on read latency.

Database Strengths: MongoDB vs Cassandra

Already in the previous chapters one can see that databases have specific strengths and that the workload has an impact on the performance. Let’s visualize these strengths with three different workload types for read latency.

When scaling matters: MongoDB vs Cassandra

The performance of present workload is one thing, but what will it be in the future? The ability to scale, as linear as possible, is important for growing applications. A simplified scalability model based on compute power assumes that the scalability factor is reflected by the increased compute capacity from the small to large cluster size, i.e. the theoretical throughout scaling factor is 400% from small to large.

Looking Beyond the Mainstream: A Positive Example

While many architects only know the most popular databases, far over 300 database technologies exist. One of them is ScyllaDB, a relatively new NoSQL database that adopts many concepts of Apache Cassandra and enhances them with its close-to-the metal design. It is built specifically for applications that require high throughput and predictable low latency.

Looking Beyond the Mainstream: A Less Positive Example

Among the more than 300 databases, there are of course numerous products that are not yet 100% technically mature and whose performance leaves much to be desired in comparison to established database solutions.

Costs? Who is interested in Performance per Dollar?

While techies are looking for performance and scalability, you should never forget the cost perspective, especially if you want to impress your manager. Costs per transaction is one of the most important key metrics for businesses.

The Purpose of Databases: Handling Your Workload

While the first chapter gave us some first insights about some databases and shows the enormous potential of such performance measurements, it also gave us a hint about one further benchmarking fact: workload matters.

Every application is unique and has an individual workload. Of course, you can categorize most applications like ERP, eCommerce shop, IoT application or AI algorithm, but still it is unique regarding

• the amount of data,
• the data set size,
• the number of (parallel) users,
• the requests,
• the distribution of the requests regarding database operations
• the distribution of the requests regarding the specific data sets
• … and many more

And yes, each of these workload specifics has an impact on the performance, the best database solution for the application and the best database tuning. In Chapter 2, we provide you with performance data regarding workload variations – form traditional relational databases to modern database technologies.

Handling Different Workload Types

MySQL and PostgreSQL are among the most popular databases, used for nearly every workload, from eCommerce to analytics. The following diagram shows why performance depends so much on the workload.

Workload Variation: Relational Databases

Not always are the workloads that different as in the last example, but also simple CRUD workloads with a shift in the distribution of database operations can lead to different performance results and showing the strengths and weaknesses of databases.

Workload Variation: NoSQL Databases

Not always are the workloads that different than in the last example, but also simple CRUD workloads can lead to different performance results and showing the strengths of databases.

A Shift in the Read-Write Ratio for MongoDB

In the last example, the performance sensitivity of MongoDB due to variation in the workload regarding the read-write-ratio was already visible, here comes an in-depth analysis of this phenomenon.

It is important to understand, that shifts of the workload due to new features can have an overall impact on the database performance.

A Shift in the Read-Write Ratio for Cassandra

The same scenarios as above, we also did for Apache Cassandra to find out if more write operations always deliver lower throughput, due to more cost intense internal operation, or if this is also database specific. The results were paritally surprising.

Impact of the Infrastructure

In the last two chapters, we digged into database performance and the impact on the workload on performance, but we never questioned the underlying resources. Which impact do the following variables have?

• cloud provider
• VM types
• storage types
• number of IOPS
• cloud vs on prem infrastructure

If you are assuming that Cloud resources from AWS or Azure, or European cloud providers like IONOS cloud and Open Telekom Cloud are delivering identical performance for comparable virtual machines, we will prove you wrong. While pricing is nearly equal for many cloud providers for comparable resources, the performance of the technical cloud solution, regarding hardware, software virtualization or provisioning rules, is extremely divers.

Cloud does not equal Cloud

Finding the right resources for your application can increase the performance and especially the performance per costs significantly.

Cloud Providers Impacting PostgreSQL Performance

A first good example, how the underlying cloud infrastructure has an impact on the outcoming performance of the database running on the infrastructure.

Measuring this performance differences is not only important for performance, but even more for performance per cost comparisons before choosing cloud resources.

Cloud Providers Not Impacting Cassandra

The above example shows an extreme performance impact on PostgreSQL, but this is not necessarily the same for other databases. This example shows the performance impact for the same workload and infrastructure resources for Apache Cassandra.

Performance Differences by VM Types

Beside the differences of Cloud providers, their hardware and virtualization, there are also big differences in the available Virtual Machine types at one provider.

Price/Performance of ARM Resources for Databases

The number of VM types at public cloud providers are complemented with ARM Graviton processors since 2021. ARM VMs are usually lower priced due to lower hardware costs. Yet, do these resources work properly with databases? What’s their price/performance?

Storage Types Performance Impact

Besides VM types, it is also important to have a suitable storage for the database technology and workload requirements. The performance increase of better, but more expensive storage solutions, can’t be calculated but easily measured.

Sometimes the results are very surprising due to internal hardware limitations, like in this example of IONOS cloud, a smaller European Cloud provider.

Benchmarking Database-as-a-Service

Over the last years Database-as-a-Service (DBaaS) have become the de-facto standard of modern database management. DBaaS provide a fully managed solution for deployment, operations and support of various database technologies. DBaaS are offered by database providers, cloud providers and specialized DBaaS companies.

In one of our market studies we identified 25 DBaaS solutions for MySQL only and over 30 DBaaS solutions for PostgreSQL. In total, we found more than 175 commercially available DBaaS products, numbers still growing.

In the last few chapters, we saw that the performance of a database installation depends on the dedicated DBMS technology, the application workload, and the underlying infrastructure resources. Could this mean that Database-as-a-Service products with identical DBMs technology, the same workload hosted on the same resources show identical performance behavior? No, not really. The way a DBaaS technology is implemented and orchestrated does further influence the DBaaS performance.

DBaaS Performance Does Not Only Depend on the DBMS Technology

MySQL and MariaDB DBaaS: Market Overview

MariaDB is a DBMs technology, based on an early MySQL fork. Both are open-source databases and many DBaaS products are implemented for these two technologies. Yet, the performance and of course the performance per costs vary immensely.

MySQL and MariaDB: DBaaS Performance/Cost Comparison

For the example above, a price/performance comparison is more than interesting. The different cost structures of DBaaS solutions are more than complex and divers. Besides compute costs, storage, network, backup, and support costs need to be considered for a fair comparison.

A Document DBaaS Market Comparison

For document-oriented DBaaS products many DBaaS products similar to MongoDB Atlas, the official DBaaS service of MongoDB, can be found on the market.

Document DBaaS Latencies

Not only Throughput and throughput per costs are important KPIs for DBaaS, also the latencies can vary a lot, depending on the software and hardware implementation.

DBaaS vs self-managed DBMS

DBaaS products are a high-tech implementation for databases, but their performance compared to self-managed databases on similar resources is not identical. When migrating to a DBaaS product often the resource sizing needs to be adapted.

Conclusion

Wow, you really made it to the very end. That's a lot of data points you fought your way through. We hope you found some interesting and relevant metrics for your daily work.

At least, you saw a lot of differences and surprising results. In many use cases performance measurements can help you making good data-driven decisions. If you are interested in doing measurements on your own, or are struggling with some questions, or have concerns regarding the results, please reach out.

Disclaimers

This document is based on the work of benchANT. All insights come from the daily work in performance testing for clients and research. benchANT is specialized on database and infrastructure benchmarking to deliver data-driven technology recommendations and optimization suggestions for existing IT applications. This article was first released in August 2023 as a pdf document behind a registraton wall. It has been migrated to the blog section in 2026 without any changes.

All performance measurements shown here were done with benchANT's benchmarking framework in an automated, reliable, and re-producible way in 2022 and 2023. Accordingly, all performance numbers shown represent the state of the play at that point in time.

The primary goal of this article is to demonstrate the insights that can be gained by systematic benchmarking; it is not to depreciate or rant at cloud providers or database vendors because they show worse results than others. As the results presented throughout the document show, systems performing worse in one scenario could be superior in others. It is absolutely necessary to evaluate and benchmark on a per use case level.

The raw data backing the results presented in this article are not available online. We will make them accessible upon request.