Open Telekom Cloud PostgreSQL DBaaS: A Shining Star

Key Findings

Features: The features of DBaaS range from automatic deployment, backup and alerting to additional support. A high-level comparison of the three DBaaS services shows:

• Features: The DBaaS products compared here are native MySQL DBaaS solutions with a similar technical structure and comparable basic features in deployment, backup, alerting and management.
• Pricing: Identical DBaaS pricing structure for all offers, and estimated monthly costs at nearly identical levels.
• Support: Only OTC offers free email and telephone support at the business level, instead of up to 10% additional DBaaS costs charged by the US hyperscalers AWS and MS Azure.
• GDPR compliance: Only OTC's DBaaS solution is not subject to the US Cloud Act.

Throughput and Latency: The results of the technical performance measurements for the two IT application workloads - Telecommunication & Analytics - show performant results for the OTC DBaaS solution.

• Workload "Telecommunication": OTC shows a strong performance here - both in terms of throughput and latency - and is only surpassed by AWS but relegates MS Azure to the bottom. With the OTC setting "Private IP", even the performance of AWS is surpassed.
• Workload "Analytics": Azure Database for PostgreSQL shows above-average performance for the analytical workload based on TPC-H. Here too, OTC is in second place, followed by AWS.

Price/Performance: When the DBaaS instance and storage costs are taken into account, the PostgreSQL DBaaS solution from the Open Telekom Cloud is more than competitive overall.

• In both scenarios, OTC PostgreSQL is on the same price-performance level as AWS.
• In the telecommunication scenario, it also shows an 86% better price-performance ratio than MS Azure.
• OTC is thus establishing itself as a strong European alternative in the PostgreSQL DBaaS sector.

1. Price/Performance
The Decisive Indicator for DBaaS

Database-as-a-Service (DBaaS) solutions offer numerous advantages over self-managed database management systems. These include time savings during deployment and application development thanks to simple APIs, high guaranteed availability (SLAs) and an overall lower Total Cost of Ownership (TCO).

For these reasons, many companies rely on DBaaS solutions from Amazon Web Service (AWS), Microsoft Azure or the European solution from Open Telekom Cloud (OTC).

A comparison of these DBaaS solutions must be made holistically on various levels:

• Features: What technical and organizational features for deployment, management, support and compliance does the respective DBaaS product offer?
• Price: What are the monthly costs for resources, storage, backup, network and support for the respective DBaaS?
• Performance: What technical performance do the respective DBaaS flavors offer?

Currently, over 40 PostgreSQL DBaaS products exist in the market. A lot of them offer a native PostgreSQL managed database, but there are also advanced PostgreSQL-compatible DBaaS products like MS Azure CosmosDB for PostgreSQL out there.

In this article, we only compare the following native PostgreSQL DBaaS products with a focus on price and performance:

• AWS RDS for PostgreSQL
• MS Azure Database for PostgreSQL - Flexible Server
• OTC RDS PostgreSQL

While AWS and MS Azure offer their product in multiple regions, OTC is providing a region for Germany, the Netherlands and Switzerland. Apart from the fact that AWS and MS Azure offer a larger number of instances, there is no significant technical difference in the basic functions for deployment, backup, alerting and management of the three DBaaS products.

However, one key point of difference is data protection for European companies. While Open Telekom Cloud, as a German company, is 100% EU GDPR compliant, its US competitors are subject to the US Cloud Act, which undermines GDPR compliance.

The price model of the four DBaaS solutions hardly differs and enables a fair price comparison. There are costs for:

• The DBaaS instance with the number of nodes per number of vCPU and RAM
• Storage costs for data storage per GB
• Network costs for data ingress and egress
• Costs for backup storage per GB
• Support costs

While all providers offer a "free support tier", which is access to the documentation, only Open Telekom Cloud offers free business support by email and telephone.

A cost overview of the 3 DBaaS flavors we used for the performance measurements in this technical analysis can be found in the appendix.

2. Performance
Same Sizing, Same Database, Same Performance?

The performance of the DBaaS products is often underestimated and classified as almost identical for the same database technology i.e. PostgreSQL. However, analyses show that the database configuration, the underlying hardware used internally and the implementation of the DBaaS layer have a major influence on the performance of the DBaaS and can result in significant differences.

For the evaluation, we consider two different types of use cases.

• Analytics: A read-only workload, based on the TPC-H, with complex analytical queries.
• Telecommunication: A read-write workload, based on the TATP benchmarking suite, with simple, but high-frequent, database operations.

For the performance measurements, we use comparable DBaaS resources.

For all DBaaS providers, the public endpoints of the DBaaS instances were used to run the benchmarks. In addition, for OTC we also measured the DBaaS performance by using the private endpoint which is equivalent to the virtual private network options of the US hyperscalers. It is important to emphasize that by using the virtual private network endpoints of the hyperscaler DBaaS, a performance improvement might be achieved.

The default PostgreSQL configuration of each DBaaS offer was used and not adjusted.

The full workload and benchmarking details can be found in the appendix.

2.1 Throughput Results

The throughput is one of the most important database performance indicators. It shows how many operations can be handled by the database infrastructure. The higher the throughput the more parallel operations by users, applications or sensors can be written, read, updated or deleted.

• For the telecommunication workload, we see a strong performance of the AWS RDS instance. With over 56,000 operations per second, it outperforms Azure and the default OTC option.
• The results for OTC RDS are also outperforming the throughput results for MS Azure Database for PostgreSQL.
• Especially the OTC private IP setting is on the same top level as AWS with over 57,000 operations per second.

• Note: The analytical workload contains more complex, transactional operations, which is why the results are significantly lower than for the telecommunication workload.
• For the analytical workload, we see a different leader with Azure Database for PostgreSQL with a strong result with 1.6 transactions per second.
• OTC RDS is in second rank, providing a stable throughput with around 1.3 transactions per second for both configurations.

2.2 Price-Performance Ratio

Based on the throughput results and the transparent online available pricing details, we can calculate an even more important KPI. The price-performance-ratio.

The KPI is calculated by dividing the monthly costs by the operations per second to demonstrate how costly one operation per second is. A detailed cost calculation can be found in the appendix.

• AWS RDS and OTC RDS (with private IP) are leading the price-performance-ratio ranking with 0.022€ (0.023€) per operation per second for the telecommunication use case.
• Azure Database for PostgreSQL has significantly higher costs per operations due to its weaker throughput performance.

• For the analytical workload we see a completely different picture. The throughput performance of Azure Database for PostgreSQL leads also to a price-performance-ratio top rank.
• On the second rank we can find OTC RDS PostgreSQL, again. It has slightly lower costs per operation than AWS RDS.

2.3 Latency

To complete the technical performance analysis, we also look at the database latencies of the relational database services.

The database latency indicates the delay of the operation. Database latency is an important parameter for many applications with users, as it influences the quality of interaction with the application from the user's perspective.

The specified latency is the 95th percentile of all database operations, which means that 95% of all operations were faster or equal to the specified values. For the telecommunication use case, this is the combined latency of read and write operations. The analytical workload is purely the read latency, as no write operations are performed here.

• For the telecommunication use case, the measurements show top-level results for OTC RDS and AWS RDS around 6 milliseconds.
• Azure Database for PostgreSQL has a latency of over 21 milliseconds, which is nearly 4x slower than OTC and AWS.

• The analytical use case has significantly higher latencies, involving complex and long-running queries.
• Azure Database for PostgreSQL has the lowest latency for this use case at just under 20,000 milliseconds. Only OTC with the private IP setting comes close to this low latency.
• AWS RDS and OTC RDS with the public IP setting are in the range of 30,000 milliseconds.

3. Conclusion

Database management systems are the backbone of every modern application and Database-as-a-Service (DBaaS) may reduce the total cost of ownership of database operations.

The technical evaluation that we have undertaken in this white paper shows a holistic comparison of the most important PostgreSQL DBaaS offers of AWS, MS Azure and the Open Telekom Cloud.

Overall, the results show that the Open Telekom Cloud with its PostgreSQL RDS service is at par in performance and price-performance with the DBaaS solutions of the US hyperscalers in the selected benchmarking scenarios.

Open Telekom Cloud is the only provider to offer inclusive e-mail and call support in the pricing. Such typical service is chargeable on other providers. In each scenario, it outperforms one of the competitors for the KPIs throughput, cost per operation and latency.

The performance gain using the private IP is certainly also interesting. These settings show a significant performance advantage, at least in the telecommunication use case.

Together with the GDPR advantage, OTC offers a well-rounded and high-performance MySQL DBaaS product for European businesses.

Appendix

About the Authors

Jan Ocker is one of the founders and CGO of benchANT, a company that specializes in benchmarking and performance testing of databases. Jan started his career as an IT project manager in eCommerce. Additionally, he focused on data analytics and the optimization of SQL queries, before he founded benchANT.

Daniel Seybold is one of the founders and CTO of benchANT. Daniel started his career as a researcher focusing on distributed systems and databases. During his academic career he published over 20 papers on cloud and database performance-related topics at renowned scientific conferences and completed his PhD with the thesis An automation-based approach for reproducible evaluations of distributed DBMS on elastic infrastructures.

These research results are the technical foundation of benchANT which pursues the goal of supporting organizations in selecting the right database for their use case.

From his point of view, there is no "best" database, but only a better-suited and more efficient database solution for each use case.

Disclaimer

Open Telekom Cloud commissioned the work presented in this document. Open Telekom Cloud chose the competitors, the test, and the database sizes. benchANT chose the most compatible configurations for the other tested DBaaS, ran the benchmarks, evaluated the results, and wrote the text.

Cost Calculation

As already mentioned in the introduction, the price models of the 4 DBaaS offerings are comparable. The following price structure for DBaaS and 500GB storage resulted in the tested DBaaS flavors as of 01.02.2024.

The monthly costs were calculated for 730 hours and with an exchange rate of 1.08 USD/EUR in the region Frankfurt or comparable.

We have not included the costs for network, backup and support in the cost calculation, as these are very application-specific. Moreover, in our experience, these are in the region of 10% and are also almost identical, except for free support from OTC. Open Telekom Cloud is the only provider offering inclusive call and e-mail support.

Since all performance measurements were carried out on on-demand resources, we also used the on-demand prices for the cost calculation and price/performance calculation.

For completeness, we mention that all DBaaS providers also offer discount scales for 1- and multi-year reserved DBaaS instances. With OTC, cost savings of over 45% are possible. More detailed information on the pricing of reserved instances can be found in the price calculators of the respective providers.

Benchmarking Methodology

All benchmarks are executed with benchANT’s benchmarking platform that fully automates the entire benchmarking process to ensure deterministic and reproducible benchmark results (for more details, see the associated publications on Mowgli and benchANT). A single benchmark execution comprises the following steps carried out by the benchANT platform:

• deploy a new (PostgreSQL) DBaaS instance
• deploy a benchmark VM (16 vCores / 64 GB RAM) to run the YCSB on the same cloud provider in the same region
• trigger the LOAD phase and fill the database with the initial data set
• wait for 5 minutes as a stabilization period before triggering the RUN phase
• trigger the RUN phase of 30 minutes for the actual workload
• collect and process the results and metadata
• tear down the DBaaS instance and the benchmark VM

Each benchmark scenario has been measured three times, i.e. the described process has been carried out three times per DBaaS provider and workload, resulting in 24 data sets. For each scenario, the three results sets are analyzed towards outliers, but no significant outliers were detected. The presented results are averages over the three runs.

The applied analytical and telecommunication workloads are part of the open-source benchmark suite BenchBase which is maintained by Carnegie Mellon University.

The following table describes the most relevant benchmark settings per workload and the full benchmarking details and results can be found on GitHub.