A Database Benchmarking Recap on 2023

2024 is here and it's time to look back at the past year 2023.

We naturally want to focus on databases, Database-as-a-service, (cloud) infrastructures and their performance.

Let's review the year 2023 together.

What Happened in the Database World?

This chapter alone would be worth a post of its own. Fortunately, our esteemed database researcher and entrepreneur Andy Pavlo from Ottertune has already described this post in detail in his Recap 2023 article. You can find the full article here. We think it's more than worth reading - Kudos to Andy Pavlo!

We will therefore only briefly summarise the most important points:

Numerous new Vector databases have been founded and funded: Marqo, Qdrant, Chroma, Weaviate and Pinecone
Many established DBMS vendors have also developed Vector extensions for their solutions: SingleStore, Oracle, Rockset, Clickhouse, PGVector, MongoDB and Cassandra
SQL evolved immensely in 2023 with read-only property graph queries and multi-dimensional arrays.
MariaDB is struggling with some structural and financial problems, including its DBaaS solution SkySQL, which has now been launched as an independent company. In this context, the DBaaS offering of Microsoft Azure Database for MariaDB will also be discontinued at the end of 2025.

Database Performance in 2023

In addition to the market entry of numerous database auto-tuning solutions, we presented in this article. Several companies also published performance benchmarking studies last year.

TDEngine, a new time-series database has created its own benchmark report about her own DBMS vs TimescaleDB and InfluxDB.
Couchbase has commissioned Altoros, an IT consulting firm, for their annual benchmarking report against AWS DynamoDB, MongoDB and Redis for various YCSB workloads. The results can be found here
ScyllaDB conducted an in-depth benchmarking study with benchANT with over 115 scalability and cost measurements. You can find our measurement results of ScyllaDB and MongoDB in our article: NoSQL Benchmark: MongoDB vs ScyllaDB

But that was not all that happened in 2023. benchANT alone has carried out numerous database, DBaaS and cloud performance measurements. The portion of Database-as-a-Service benchmarks compared to self-managed DBMS is certainly interesting here.

DBMS DBaaS Performance Measurement Distribution 2023

We have analyzed our internal database and compiled a few statistics for you on the technologies measured in 2023. In the end, we ran measurements on

54 different DBMS & DBaaS technologies
10 different cloud providers
using 8 different workload benchmarking suites.

Here are some stats!

Benchmarking Measurement Distribution - 2023 - Database

Benchmarking Measurement Distribution - 2023 - Cloud

Benchmarking Measurement Distribution - 2023 - Workload

<tl;dr>

While DBaaS now accounts for a large proportion of database performance measurements in general, self-managed PostgreSQL dominates when it comes to individual data storage technologies. This is partly due to customer projects, but also to internal research projects.
benchANT has integrated numerous DBaaS solutions into the benchmarking framework over the past year, and this variance can now also be seen in the measurements used for various market comparisons and technology analyses.
In terms of infrastructure, the cloud market leader AWS dominates in our measurements; this is partly due to the customers, but also to the DBaaS solutions, which usually run on AWS first.
The almost equally frequently used measurements on on-premise infrastructures or private cloud solutions stem from customer projects and the internal research projects already mentioned above.
With OVH, OTC, IONOS and STACKIT, there are also 4 European cloud infrastructures in our most frequent cloud measurements, far ahead of GCP.
The well-known YCSB dominates the benchmarking suites, partly because there exist many database integrations, making it easy to compare technologies. But also the simple rudimentary queries in the CRUD workload enable comparisons between very different database technologies easily.
benchANT is also in the process of developing its own time series benchmarking suite based on the YCSB, and numerous test measurements have already been carried out. But more on this in a separate article when the time comes.

In August 2023, we started the DataScaleFail newsletter. Since then, we have prepared independent measurements and information about databases every 2 weeks. Our readership has grown considerably in the last 5 months. Needless to say, we look forward to passing on a wealth of information from our daily business to you also in 2024.

The most popular newsletter articles in 2023 were in descending order:

Yet, the information benchANT provides is not limited to the newsletter. First and foremost, the benchANT website contains the Database Ranking and the DBaaS Navigator in addition to other content such as:

Outlook 2024

We will continue our work to provide independent database and infrastructure information with our newsletter, our blog and of course our daily work as database experts and performance engineers.

We wish you a successful year 2024 and may the performance be with you.

A Database Benchmarking Recap on 2023

What Happened in the Database World?

Database Performance in 2023

DataScaleFail – The Beginning of our Newsletter in 2023

Outlook 2024