The Cloud Database Benchmarking Blog

Following a lot of hype surrounding standalone vector databases, many in the industry have been realizing that such specialized, dedicated engines do not always fit the bill. Instead, keeping your vectors next to your relational data is oftentimes more manageable and straightforward.

In a series of blog posts, we will explore how appropriate extensions allow you to turn PostgreSQL into a hybrid database that supports both relational data and vectors. In this first part, we take a look at the essential concepts of vector databases and what the PostgreSQL ecosystem provides for this purpose. Later in a second part, we will dive into the performance characteristics of different vector extensions and how they compare against other offerings. We discussed Benchmarking Vector Databases in an earlier blog post.

As organizations increasingly rely on managed database services, selecting the right DBaaS offering requires balancing performance, cost, and operational trade-offs. Structured benchmarking frameworks help bring clarity to this complex decision space.

The article introduces benchANT’s framework for evaluating PostgreSQL DBaaS offerings, focusing on how standardized benchmarking can support objective comparisons across cloud providers. It outlines a methodology that combines workload modelling, automated benchmark execution, and consistent configuration selection to ensure reproducible results. Using representative analytical and latency-sensitive workloads, the framework enables assessment of throughput, latency, and price–performance characteristics. The article also highlights variability across providers despite similar base technologies, reinforcing the need for systematic, data-driven evaluation when selecting managed PostgreSQL solutions.

As database selection becomes increasingly complex in modern data-driven architectures, reliable performance evaluation is more critical than ever. Benchmarking frameworks help bring structure and comparability to this process.

The article presents the latest updates to benchANT’s database benchmarking compendium, highlighting their growing role in delivering reliable and comparable performance insights. It describes how standardized workloads—ranging from OLTP over HTAP to OLAP—enable consistent evaluation across databases and cloud platforms. The piece also reinforces the importance of transparent, independent benchmarking for sound technical decision-making, while hinting at ongoing database workload trends.

Choosing the right database for your application is a critical decision that can significantly impact performance, scalability, and cost-efficiency. With the rapid evolution of database technologies and the proliferation of cloud services, this choice has become increasingly complex. With more than 1,000 products to choose from, the landscape is vast and varied.

The benchANT database ranking is a public source of information that aims to simplify this decision-making process by providing clear, objective, and comparable performance metrics across a wide range of databases and cloud environments.

TimechoDB, an enterprise-grade time-series database system based on Apache IoTDB, has taken the lead in the benchANT Time-Series Database Performance Ranking. Evaluated using the Time-Series Benchmark Suite (TSBS) with the DevOps workload, TimechoDB (built on Apache IoTDB v1.3.4) delivered standout results in the small-scale benchmark category running on a single AWS EC2 node.

All benchmarks followed benchANT’s reproducible TSBS methodology, ensuring transparency and comparability across vendors. These findings demonstrate TimechoDB’s ability to sustain high-ingestion, low-latency workloads.

Traditional database benchmarks often stop at measuring throughput and latency. While those metrics quantify raw speed, they fail to capture how systems behave when the environment changes, e.g., when nodes fail, workloads spike, or capacity must grow. In production, these behaviour in these moments is just as important as steady-state performance.

This article discusses how the benchANT benchmarking framework handles such cases, and presents results of a case study which evaluate Aerospike and Cassandra under controlled scale-out and node-failure scenarios. By examining recovery times, redistribution phases, and post-incident performance, the study highlights architectural trade-offs between speed, stability, and resilience.

PostgreSQL is available as a managed service from nearly every cloud provider, but performance and cost can differ significantly under the hood. In this independent study, we benchmark OVHcloud’s PostgreSQL DBaaS against AWS RDS PostgreSQL, Azure Flexible PostgreSQL, and DigitalOcean PostgreSQL using OLTP workloads and a strict price-performance methodology. We compared throughput, latency, and efficiency on equivalent instance types to provide an objective view of how these services perform.

For engineering and product teams, our results highlight the real-world impact of choosing one DBaaS provider over another—not just on performance but also on budget efficiency. Explore the full benchmark to see how OVHcloud positions itself in the 2025 PostgreSQL DBaaS market.

Curious about how general-purpose databases stack up against native vector databases in search performance? Our latest benchmark study pits SingleStore Helios against Pinecone and Zilliz using the established VectorDBBench suite. With 10 million 768-dimensional vectors as the battleground, we provide technical insights into how a general-purpose database like SingleStore performans against this specialized database domainn.

The capabilities to store and search for vectors, also called embeddings, is a crucial capability in many fields of artificial intelligence (AI).

When selecting a database technology for vector workloads, many selection criteria are similar to other application domains for databases. Yet, some new ones show up as well. This article revisits what vector databases are and what makes them different to other use cases.

The popular database ranking consists of performance and cost metrics for the most important databases and selected DBaaS products.

We have compared the current cloud and DBaaS prices with the prices for 2022 and are quite surprised at the different trends.

Many benchmarking suites make use of synthetic data what makes them very flexible. Yet, it introduces some fuzziness when evaluating very specific use cases such as the storage, retrieval, and analytics of IT monitoring data.

Telegraf is a popular tool for collecting that data and storing it into a data backend. For choosing the right data backend for your analytics queries, Telegraf-TS has been introduced. It is an extension to YCSB, specifically tailored to evaluate the performance of different types of analytical queries over data stored by Telegraf.

In the dynamic database management systems market, performance is a key differentiator.

Database benchmarking suites provide the tool to assess performance, guide us through the vast landscape of options, and help developers, engineers, and decision-makers make informed choices.

This article dives into the various benchmarking suites for different workload domains and provides an overview of their origin, implementations, and unique features.

Join us on a journey through YCSB, TSBS, TPC-C, and others.

Vector databases are the latest trend in the database market, addressing the needs of AI applications through dedicated vector search features.

And with every new type of database there always comes the challenge for the fastest database. But what about performance benchmarking of vector databases under the shadow of the DeWitt clause?

We have analyzed the current vector database market with respect to their DeWitt clause and which provider might sue you for running and publishing vector database benchmark results.

DBaaS Navigator - holistic comparison of the most popular Database-as-a-Service products. More than 60 attributes on technical features, management tools, support contracts, performance measurements and price-performance assessment.

• Step-by-Step Guide
• Roadmap
• Disclaimer

The Yahoo Cloud Serving Benchmark (YCSB) is a well-known tool to measure and evaluate the performance of database management systems (DBMS). One of the major advantages of YCSB is the very simple workload it uses as this allows support for many different DBMS. On the down-side, YCSB standard workloads do not make use of any even marginally sophisticated DBMS features.

In this article, we sketch how to exploit YCSB in order to build different workloads and also make use of other types of queries and updates to the dataset. While such an approach is still work, it relieves the programmer from building any feature beyond workload handling that are still important for benchmarking: metric collection, thread handling, and start-up and tear-down handling.

Learn the basics of cloud database benchmarking from scratch.

Learn everything worth knowing from classic database benchmarking to cloud database benchmarking.

Start here!

Database-as-a-Service (DBaaS) has the potential to reduce required manpower, lower time-to-delivery, and has the potential to reduce the total cost of ownership (TCO) of database services. As with most other cloud services, the DBaaS market worldwide is dominated by the American companies, in particular the hyperscalers AWS, Microsoft Azure, and Google Cloud Platform (GCP). Unfortunately, relying solely on U.S. companies is not without risk for European customers, for instance because of the interplay of Cloud Act and GDPR.

Therefore, it is beneficial to keep an eye on the European market and to evaluate European alternatives such as OVHcloud's recently released MongoDB DBaaS. In this article, we evaluate this new European service against AWS DocumentDB and Azure CosmosDB MongoDB vCore.

2024 is here and it's time to look back at the past year 2023.

We naturally want to focus on databases, Database-as-a-service, (cloud) infrastructures and their performance.

Let's review the year 2023 together.

In the fall of 2023, Microsoft Azure launched a new generation of virtual machines with the latest hardware.

Our colleagues at Cloud-Mercato have already carried out CPU performance measurements and identified significant performance benefits. Do databases also benefit from these new VMs? And is it worth switching?

We have measured it!

Database benchmarking is a proven and clearly defined method for analysing and comparing the performance characteristics of databases or database management systems (DBMS).

Performance and scalability is a top current issue in IT. Data-driven applications and cloud infrastructures are current challenges for databases. Database benchmarking can provide reliable, quantitative answers to many questions and enable the selection and optimisation of performance, but also e.g. cloud costs.

Learn everything about (cloud) database benchmarking from classic SQL databases to NoSQL and NewSQL databases in distributed cloud systems.

There is always a lack of information and expertise regarding database configuration parameters. Tuning databases for specific use cases and workloads usually requires experience.

In this article, we show you how to start with database tuning and some alternative methods instead of hiring high-paid database consultants but still get better performance from your data infrastructure.

End of November, Daniel, CTO & Founder of benchANT) had the chance to attend the AWS re--Invent in Las Vegas and it was (again) a blast.

Apart from getting first-hand insights into AWS's brand new cloud features, meeting with old acquaintances and making new ones, and enjoying the crazy vibe of Las Vegas, of course, there are also many database vendors presenting their latest features in the exhibition.

AlloyDB from Google Cloud is the new modern DBaaS solution with a 100% PostgreSQL-API and competing with AWS Aurora and CosmosDB for PostgreSQL from MS Azure.

It claims to be more than 2x faster than standard PostgreSQL for transactional workloads, and a 100 times faster for analytical queries.

Since a few week AlloyDB Omni, a downloadable, self-managed database version of AlloyDB is available.

Let's get some independent numbers!

In the last weeks, we identified huge performance differences for the PostgreSQL DBaaS solutions from AWS, GCP and MS Azure.

We already did an in-depth query performance analysis for different queries and are now going to take a deeper look into performance-critical settings of PostgreSQL for these 3 DBaaS products.

We are getting closer!

Im November 2023 veröffentlichten die Gründer von benchANT einen Artikel in iX, einer renommierten deutschen IT-Zeitschrift. Hier fassen wir den ursprünglich deutschen Artikel in einer englischen Version zusammen. Der Artikel enthält Auszüge aus der DBaaS-Studie von benchANT und anderen Artikeln, die wir kürzlich über DBaaS veröffentlicht haben.

Der Artikel geht noch einmal auf den DBaaS-Markt und die Arten von Anbietern ein, erörtert, wann sich der Einsatz von DBaaS im Vergleich zum Betrieb eines selbstverwalteten DBMS lohnen kann, und zeigt schließlich anhand eines Beispiels, wie Benchmarking bei der Auswahl der richtigen DBaaS-Lösung helfen kann.

Ideally, an IT application grows its user base over time and in consequence the underlying data infrastructure needs to scale to handle the increasing workload demands.

We show a data-driven approach to efficient capacity planning for a self-hosted MongoDB-based IT application that takes both costs and performance into account.

This approach leads to an efficient data infrastructure - both from a technical and a business perspective.

A few weeks ago we published an article about cost/performance results for PostgreSQL DBaaS products of the three hyperscalers AWS, MS Azure and Google Cloud Platform.

We were surprised by the massive performance differences and decided to take a closer look at the query performance to identify possible reasons.

This is going to be an in-depth dive into query performance.

Database Management Systems (DBMS) are a central part of every modern application, and their proper configuration is crucial to guarantee quality of service on application level, but also cost-efficient operation of IT services.

Yet, finding a configuration of a DBMS that suits the workload and specific use case is difficult, time-consuming, and error-prone when done manually. The rise of AI-based tool support offers an opportunity to let algorithms do the heavy lifting.

In recent years some companies have been founded with the goal to help their users with database performance tweaking. In this article, we introduce three of them who have focused on traditional open source, relational DBMS such as PostgreSQL and MySQL. We glance over their history, solution, and marketing claims.

"One accurate measurement is worth a thousand expert opinions." - Grace Hopper

This statement from one of the IT pioneers is still relevant today with the vastly growing amount of technologies to build data intensive and performance critical applications.

This article will explain what we learned and show you a how-to about performance benchmarks of one of the most popular NoSQL databases - MongoDB.

Let's dive in!

Managed cloud databases, also called Database-as-a-Service (DBaaS), have experienced enormous growth in recent years.

We have analyzed more than 208 offers from more than 50 different providers and compiled them in a structured list with an exemplary cost calculation.

Last week we analyzed the PostgreSQL DBaaS products of the three hyperscalers AWS, MS Azure and Google Cloud Platform to spot deviations and understand their differences.

As the insights are more than interesting and useful, we proceed with this analysis for the MySQL DBaaS products of the same providers.

Check out the results of this week's DBaaS performance/cost analysis of the MySQL DBaaS products.

MongoDB and Apache Cassandra are two of the most important NoSQL database management systems. For which use case is which DB suitable?

If you search for the differences, you will find many articles with theoretical statements about performance and scalability.

For us, this information was not enough.

We have measured!

Discover a variety of informative links to internal and external guides and lists.

These can help you choose the right database technologies and make the best decisions.

The cheat sheets for professionals.

Database-as-a-Service (DBaaS) products are considered the future of database management, known to accelerate IT development, to compensate for skill shortages and to save costs.

However, our research shows that there are significant differences in terms of performance and cost structures across the market.

Recently, we analyzed the PostgreSQL DBaaS offering of the three hyperscalers AWS, MS Azure and Google Cloud Platform to spot deviations and understand their differences

Last week, the newest major version of one of the most popular database management systems got released: PostgreSQL v16.

The new release comes with a lot of new features and improvements that also include performance optimizations for specific query types. But which performance improvements can you expect for standard workloads, or are there even performance regressions?

We ran some measurements. And here are the results!

We compare the performance and scalability capabilities of MongoDB Atlas and ScyllaDB Cloud, two important NoSQL Database-as-a-Service solutions.

In more than 132 measurement results, we show the strengths of these databases for 5 important workloads - from caching to IoT sensors.

See the results of this extensive in-depth benchmarking study!

In the course of our DBaaS benchmark measurements, another interesting metric drops - deployment time.

Here you can find the measurement results and some background on deployment time.

Apache IoTDB is an emerging open source time series database. It is an Apache top-level project and from now on also represented in our database ranking.

And not just somehow!

Apache IoTDB convinces with outstanding performance results right from the start.

See for yourself.

Learn step-by-step how to efficiently measure cloud and database performance with the innovative benchmarking platform.

Learn the tips & tricks for the correct operation of benchANT.

Become a benchANT expert!

Over the last few years, we have conducted many performance measurements in the area of cloud & database. We have bundled the most exciting results of these projects here in a whitepaper and also arranged them in an educational way. Discover exciting measurement results for the most popular databases and cloud technologies such as MongoDB, PostgreSQL, MySQL or AWS.

Everything database vendors and cloud providers won't tell you. This comprehensive article reveals the results of 117 independent performance benchmarks across popular databases (MySQL, PostgreSQL, MongoDB, Cassandra, ScyllaDB), cloud infrastructure (AWS, Azure, Alibaba Cloud), and DBaaS solutions. Discover why workload characteristics and infrastructure choices matter more than marketing claims when it comes to throughput, latency, scalability, and cost optimization.

MongoDB Atlas and CosmosDB are two of the most popular Database-as-a-Service (DBaaS) products on the market. But which is the better one for your requirements?

In this article, we will perform a deep and comprehensive qualitative and quantitative benchmark of these two DBaaS products. We will discuss the differences, benefits and technical aspects to give you the information you need to make an informed decision.

See what our holistic benchmarking analysis revealed.

Learn the basics of cloud database benchmarking from scratch.

Learn everything worth knowing from classic database benchmarking to cloud database benchmarking.

Start here!

Data centers and public cloud providers advertise low-cost ARM virtual machines.

But are they suitable for running database management systems?

We looked at quite small ARM instances for small-scale applications and benchmarked them against comparably large Intel-based x86 VMs.

The test results are available here!

Data centers and public cloud providers advertise low-cost ARM virtual machines.

But are they suitable for running database management systems?

We looked at quite small ARM instances for small-scale applications and benchmarked them against comparably large Intel-based x86 VMs.

The test results are available here!

Which impact does the distribution of read and write operations have on a database management system?

Does such an influence exist? Is it always similar? How does a specific database behave when the workload changes?

We did what we do best (and fastest), we benchmarked.

9 workload variations - 2 distributed cloud databases - 3 key insights.

The Yahoo! Cloud Serving Benchmark (YCSB) is the most popular NoSQL benchmark suite.

Learn all about the YCSB in this guide, including 4 exclusive articles from YCSB developers.

We show you what it can do and how you can customize it for your situation.