The Cloud Database Benchmarking Blog

The capabilities to store and search for vectors, also called embeddings, is a crucial capability in many fields of artificial intelligence (AI).

When selecting a database technology for vector workloads, many selection criteria are similar to other application domains for databases. Yet, some new ones show up as well. This article revisits what vector databases are and what makes them different to other use cases.

The popular database ranking consists of performance and cost metrics for the most important databases and selected DBaaS products.

We have compared the current cloud and DBaaS prices with the prices for 2022 and are quite surprised at the different trends.

Many benchmarking suites make use of synthetic data what makes them very flexible. Yet, it introduces some fuzziness when evaluating very specific use cases such as the storage, retrieval, and analytics of IT monitoring data.

Telegraf is a popular tool for collecting that data and storing it into a data backend. For choosing the right data backend for your analytics queries, Telegraf-TS has been introduced. It is an extension to YCSB, specifically tailored to evaluate the performance of different types of analytical queries over data stored by Telegraf.

In the dynamic database management systems market, performance is a key differentiator.

Database benchmarking suites provide the tool to assess performance, guide us through the vast landscape of options, and help developers, engineers, and decision-makers make informed choices.

This article dives into the various benchmarking suites for different workload domains and provides an overview of their origin, implementations, and unique features.

Join us on a journey through YCSB, TSBS, TPC-C, and others.

Vector databases are the latest trend in the database market, addressing the needs of AI applications through dedicated vector search features.

And with every new type of database there always comes the challenge for the fastest database. But what about performance benchmarking of vector databases under the shadow of the DeWitt clause?

We have analyzed the current vector database market with respect to their DeWitt clause and which provider might sue you for running and publishing vector database benchmark results.

The Yahoo Cloud Serving Benchmark (YCSB) is a well-known tool to measure and evaluate the performance of database management systems (DBMS). One of the major advantages of YCSB is the very simple workload it uses as this allows support for many different DBMS. On the down-side, YCSB standard workloads do not make use of any even marginally sophisticated DBMS features.

In this article, we sketch how to exploit YCSB in order to build different workloads and also make use of other types of queries and updates to the dataset. While such an approach is still work, it relieves the programmer from building any feature beyond workload handling that are still important for benchmarking: metric collection, thread handling, and start-up and tear-down handling.

Database-as-a-Service (DBaaS) has the potential to reduce required manpower, lower time-to-delivery, and has the potential to reduce the total cost of ownership (TCO) of database services. As with most other cloud services, the DBaaS market worldwide is dominated by the American companies, in particular the hyperscalers AWS, Microsoft Azure, and Google Cloud Platform (GCP). Unfortunately, relying solely on U.S. companies is not without risk for European customers, for instance because of the interplay of Cloud Act and GDPR.

Therefore, it is beneficial to keep an eye on the European market and to evaluate European alternatives such as OVHcloud's recently released MongoDB DBaaS. In this article, we evaluate this new European service against AWS DocumentDB and Azure CosmosDB MongoDB vCore.

2024 is here and it's time to look back at the past year 2023.

We naturally want to focus on databases, Database-as-a-service, (cloud) infrastructures and their performance.

Let's review the year 2023 together.

In the fall of 2023, Microsoft Azure launched a new generation of virtual machines with the latest hardware.

Our colleagues at Cloud-Mercato have already carried out CPU performance measurements and identified significant performance benefits. Do databases also benefit from these new VMs? And is it worth switching?

We have measured it!

Database benchmarking is a proven and clearly defined method for analysing and comparing the performance characteristics of databases or database management systems (DBMS).

Performance and scalability is a top current issue in IT. Data-driven applications and cloud infrastructures are current challenges for databases. Database benchmarking can provide reliable, quantitative answers to many questions and enable the selection and optimisation of performance, but also e.g. cloud costs.

Learn everything about (cloud) database benchmarking from classic SQL databases to NoSQL and NewSQL databases in distributed cloud systems.

There is always a lack of information and expertise regarding database configuration parameters. Tuning databases for specific use cases and workloads usually requires experience.

In this article, we show you how to start with database tuning and some alternative methods instead of hiring high-paid database consultants but still get better performance from your data infrastructure.

End of November, Daniel, CTO & Founder of benchANT) had the chance to attend the AWS re--Invent in Las Vegas and it was (again) a blast.

Apart from getting first-hand insights into AWS's brand new cloud features, meeting with old acquaintances and making new ones, and enjoying the crazy vibe of Las Vegas, of course, there are also many database vendors presenting their latest features in the exhibition.

AlloyDB from Google Cloud is the new modern DBaaS solution with a 100% PostgreSQL-API and competing with AWS Aurora and CosmosDB for PostgreSQL from MS Azure.

It claims to be more than 2x faster than standard PostgreSQL for transactional workloads, and a 100 times faster for analytical queries.

Since a few week AlloyDB Omni, a downloadable, self-managed database version of AlloyDB is available.

Let's get some independent numbers!

In the last weeks, we identified huge performance differences for the PostgreSQL DBaaS solutions from AWS, GCP and MS Azure.

We already did an in-depth query performance analysis for different queries and are now going to take a deeper look into performance-critical settings of PostgreSQL for these 3 DBaaS products.

We are getting closer!

Im November 2023 veröffentlichten die Gründer von benchANT einen Artikel in iX, einer renommierten deutschen IT-Zeitschrift. Hier fassen wir den ursprünglich deutschen Artikel in einer englischen Version zusammen. Der Artikel enthält Auszüge aus der DBaaS-Studie von benchANT und anderen Artikeln, die wir kürzlich über DBaaS veröffentlicht haben.

Der Artikel geht noch einmal auf den DBaaS-Markt und die Arten von Anbietern ein, erörtert, wann sich der Einsatz von DBaaS im Vergleich zum Betrieb eines selbstverwalteten DBMS lohnen kann, und zeigt schließlich anhand eines Beispiels, wie Benchmarking bei der Auswahl der richtigen DBaaS-Lösung helfen kann.

Ideally, an IT application grows its user base over time and in consequence the underlying data infrastructure needs to scale to handle the increasing workload demands.

We show a data-driven approach to efficient capacity planning for a self-hosted MongoDB-based IT application that takes both costs and performance into account.

This approach leads to an efficient data infrastructure - both from a technical and a business perspective.

A few weeks ago we published an article about cost/performance results for PostgreSQL DBaaS products of the three hyperscalers AWS, MS Azure and Google Cloud Platform.

We were surprised by the massive performance differences and decided to take a closer look at the query performance to identify possible reasons.

This is going to be an in-depth dive into query performance.

Database Management Systems (DBMS) are a central part of every modern application, and their proper configuration is crucial to guarantee quality of service on application level, but also cost-efficient operation of IT services.

Yet, finding a configuration of a DBMS that suits the workload and specific use case is difficult, time-consuming, and error-prone when done manually. The rise of AI-based tool support offers an opportunity to let algorithms do the heavy lifting.

In recent years some companies have been founded with the goal to help their users with database performance tweaking. In this article, we introduce three of them who have focused on traditional open source, relational DBMS such as PostgreSQL and MySQL. We glance over their history, solution, and marketing claims.

"One accurate measurement is worth a thousand expert opinions." - Grace Hopper

This statement from one of the IT pioneers is still relevant today with the vastly growing amount of technologies to build data intensive and performance critical applications.

This article will explain what we learned and show you a how-to about performance benchmarks of one of the most popular NoSQL databases - MongoDB.

Let's dive in!

Last week we analyzed the PostgreSQL DBaaS products of the three hyperscalers AWS, MS Azure and Google Cloud Platform to spot deviations and understand their differences.

As the insights are more than interesting and useful, we proceed with this analysis for the MySQL DBaaS products of the same providers.

Check out the results of this week's DBaaS performance/cost analysis of the MySQL DBaaS products.

Database-as-a-Service (DBaaS) products are considered the future of database management, known to accelerate IT development, to compensate for skill shortages and to save costs.

However, our research shows that there are significant differences in terms of performance and cost structures across the market.

Recently, we analyzed the PostgreSQL DBaaS offering of the three hyperscalers AWS, MS Azure and Google Cloud Platform to spot deviations and understand their differences

Last week, the newest major version of one of the most popular database management systems got released: PostgreSQL v16.

The new release comes with a lot of new features and improvements that also include performance optimizations for specific query types. But which performance improvements can you expect for standard workloads, or are there even performance regressions?

We ran some measurements. And here are the results!

We compare the performance and scalability capabilities of MongoDB Atlas and ScyllaDB Cloud, two important NoSQL Database-as-a-Service solutions.

In more than 132 measurement results, we show the strengths of these databases for 5 important workloads - from caching to IoT sensors.

See the results of this extensive in-depth benchmarking study!

In the course of our DBaaS benchmark measurements, another interesting metric drops - deployment time.

Here you can find the measurement results and some background on deployment time.

Apache IoTDB is an emerging open source time series database. It is an Apache top-level project and from now on also represented in our database ranking.

And not just somehow!

Apache IoTDB convinces with outstanding performance results right from the start.

See for yourself.

MongoDB Atlas and CosmosDB are two of the most popular Database-as-a-Service (DBaaS) products on the market. But which is the better one for your requirements?

In this article, we will perform a deep and comprehensive qualitative and quantitative benchmark of these two DBaaS products. We will discuss the differences, benefits and technical aspects to give you the information you need to make an informed decision.

See what our holistic benchmarking analysis revealed.

DBaaS Navigator - holistic comparison of the most popular Database-as-a-Service products. More than 60 attributes on technical features, management tools, support contracts, performance measurements and price-performance assessment.

• Step-by-Step Guide
• Roadmap
• Disclaimer

Managed cloud databases, also called Database-as-a-Service (DBaaS), have experienced enormous growth in recent years.

We have analyzed more than 208 offers from more than 50 different providers and compiled them in a structured list with an exemplary cost calculation.

MongoDB and Apache Cassandra are two of the most important NoSQL database management systems. For which use case is which DB suitable?

If you search for the differences, you will find many articles with theoretical statements about performance and scalability.

For us, this information was not enough.

We have measured!

Discover a variety of informative links to internal and external guides and lists.

These can help you choose the right database technologies and make the best decisions.

The cheat sheets for professionals.

Data centers and public cloud providers advertise low-cost ARM virtual machines.

But are they suitable for running database management systems?

We looked at quite small ARM instances for small-scale applications and benchmarked them against comparably large Intel-based x86 VMs.

The test results are available here!

Data centers and public cloud providers advertise low-cost ARM virtual machines.

But are they suitable for running database management systems?

We looked at quite small ARM instances for small-scale applications and benchmarked them against comparably large Intel-based x86 VMs.

The test results are available here!

Which impact does the distribution of read and write operations have on a database management system?

Does such an influence exist? Is it always similar? How does a specific database behave when the workload changes?

We did what we do best (and fastest), we benchmarked.

9 workload variations - 2 distributed cloud databases - 3 key insights.

Learn step-by-step how to efficiently measure cloud and database performance with the innovative benchmarking platform.

Learn the tips & tricks for the correct operation of benchANT.

Become a benchANT expert!

The Yahoo! Cloud Serving Benchmark (YCSB) is the most popular NoSQL benchmark suite.

Learn all about the YCSB in this guide, including 4 exclusive articles from YCSB developers.

We show you what it can do and how you can customize it for your situation.

Apache Cassandra has released a new major release of their popular open-source, column-oriented No-SQL database management system (DBMS) after more than 6 years.

In doing so, they announce major improvements in terms of quality and performance!

How fast is the new Cassandra 4.0.0? And how does it perform against the previous versions?

We have made benchmark measurements on throughput and latency.

Learn the basics of cloud database benchmarking from scratch.

Learn everything worth knowing from classic database benchmarking to cloud database benchmarking.

Start here!