PostgreSQL DBaaS – A Performance/Cost Evaluation

Table of Contents

• Chapter 1: Key Insights
• Chapter 2: PostgreSQL 16 PostgreSQL DBaaS Products Overview
  • AWS RDS for PostgreSQL
  • MS Azure Database PostgreSQL
  • GCP CloudSQL Postgresql
• Chapter 3: Performance Results
  • Throughput
  • Latency
• Chapter 4: Performance/Cost Analysis
  • Costs
  • Performance / Cost Results
• Chapter 5: Conclusion
• Appendix
  • Benchmarking Specifications
  • Workload Specifications

Key Insights – Performance/Cost Results PostgreSQL DBaaS

• AWS RDS for PostgreSQL outperforms Azure PostgreSQL and GCP Cloud SQL for PostgreSQL for the OLTP workload.
• Azure Database for PostgreSQL outperforms AWS RDS and GCP Cloud SQL for the OLAP workload.
• GCP CloudSQL has the most attractive pricing, but not the best price-performance.
• Price/performance is a useful indicator for pricy products such as Database-as-a-Service products
• Looking at performance/costs will help you to save money or to get more performance for your money.

PostgreSQL DBaaS Products Overview

There are over 40 commercial Database-as-a-Service products for PostgreSQL, one of the most popular open-source SQL databases on the market, right now. At first glance, the products appear similar – technically and usually also in terms of cost structure and pricing.

Yet, performance measurements show greater differences than one would expect. This means that a pure cost analysis for decision-making lacks significance, and instead a price-performance evaluation is required.

Before we go into the results, let us take a quick look at the DBaaS PostgreSQL products from AWS, Azure and GCP. An in-detail comparison can be found in our DBaaS Navigator.

AWS RDS for PostgreSQL

AWS Relational Database System (RDS) was one of the first Database-as-a-Service products released for PostgreSQL v9 (2015). AWS RDS for PostgreSQL offers a fully managed databases service with all necessary deployment and management features, such as backup, alerting etc. AWS RDS for PostgreSQL can be deployed on selected virtual machines from AWS as single-node, read replica or multi-availability-zone setup with one or two standby replica and load balancing.

The pricing model is similar to the regular virtual machine pricing. Users select dedicated resources with respect to machine size, storage type and backup sizing. They can either go with the “OnDemand” plan, or “Reserved Instances” with a 1- or 3-years contract period and respective reduction in price. Additionally, network costs and snapshot costs can occur.

An alternative DBaaS option for PostgreSQL on AWS is the distributed PostgreSQL solution from AWS Aurora. AWS Aurora is a cloud-native, automatic-scaling database, which is 100% compatible with MySQL and PostgreSQL.

Azure Database for PostgreSQL

Azure Database for PostgreSQL is the Microsoft Azure counterpart for AWS RDS for PostgreSQL. The single server edition was released in 2017, but will be retired in 2025. The successor is called “Flexible Server” and allows more flexible server configurations. Features and pricing model are similar to the AWS product.

A major difference is that Azure Database for PostgreSQL only offers a high-availability setup with 2 nodes and no read load balancing. Further, it is not possible to choose a storage type.

An alternative distributed DBaaS option for PostgreSQL on Azure is the CosmosDB API for PostgreSQL. CosmosDB for PostgreSQL is based on Citus, an open-source PostgreSQL extension transforming PostgreSQL into a distributed database with horizontal scalability.

GCP CloudSQL for PostgreSQL

Google Cloud’s DBaaS for PostgreSQL is called CloudSQL for PostgreSQL that was released in 2017. Compared to AWS RDS for PostgreSQL and Azure Database for PostgreSQL, one main difference is the flexible resource selection of the DBaaS flavor that allows the fine-grained selection of CPU and RAM per instance. CloudSQL also offers single-node and HA setups with one replica, like Azure.

The AlloyDB for PostgreSQL is the cloud-native, distributed PostgreSQL DBaaS alternative from GCP, and the equivalent product to AWS Aurora and Azure CosmosDB.

Note: In this evaluation, we only compare the standard PostgreSQL DBaaS products. Comparing the cloud-native solutions would increase the complexity enormously due to different pricing concepts and technical features. Such a comparison is planned for a separate study.

Performance Results – PostgreSQL DBaaS

The first step of comparing Database-as-a-Service options for the Price/Performance KPI, are performance measurements.

We run the measurements with our approved benchANT framework for the following workloads:

• BenchBase TATP: read-heavy, high-throughput application workload (OLTP)
• BenchBase TPC-H: read-heavy, analytical workload (OLAP)

The full workload specifications can be found in the corresponding appendix section.

The performance measurements were executed on comparable resources on the respective DBaaS infrastructure, namely

• AWS RDS for PostgreSQL: db.m5.4xlarge (16 vCPU/ 6 4 GB RAM), 500 GB GP3 storage
• Azure Database for PostgreSQL: Gen5 General Purpose 16vCore (16 vCPU/ 64 GB RAM), 500 GB SSD storage
• CloudSQL for PostgreSQL: flex 16 vCPU/ 64 GB RAM, 500 GB SSD storage

We used the default configuration of PostgreSQL of each provider.

All measurement scenarios were executed 3 times to capture variances coming from cloud resources. The full benchmarking specifications can be found in the corresponding appendix section.

Throughput Results – PostgreSQL DBaaS

• For the TATP workload, AWS RDS for PostgreSQL outperforms both rivals by nearly 100 %! Azure Database for PostgreSQL and CloudSQL are nearly on the same level, around 28,000 transactions per second.
• For the analytical workload, TPC-H, we see a completely different pattern. Azure Database for PostgreSQL outperforms AWS RDS for PostgreSQL by 42% and CloudSQL by 110%.

Note: Please acknowledge the different result metrics for the different workload types: TPC-C: transactions per second, TPC-H: transactions per hour. The results of different benchmarks are not comparable.

Latency 95th Results – PostgreSQL DBaaS

• For the TATP workload, we also find the best results at AWS RDS for PostgreSQL again. AWS is 3.5 times faster than Azure Database for PostgreSQL and 2.3 times faster than GCP Cloud SQL.
• For the analytical TPC-H workload, Azure Database shows significantly lower latencies than AWS RDS (~33%) and GCP CloudSQL (~53%)

Note: Please acknowledge the different result metric dimensions (transactions per second vs operations per second). Also, TPC-C and TPC-H latency results are not solely for read operations, but for whole transactions. Neither benchmark separates read and update latency due to transactional bundling of the single operations.

Performance / Cost Analysis

Cost Comparison

Estimating the monthly costs of a DBaaS product is more complex than for simple infrastructure resources. In the DBaaS Navigator we calculate the monthly costs adding up multiple components for a fair calculation due to different pricing models

• DBaaS flavor resource costs
• Storage costs
• Backup costs
• Network costs
• Support costs

Our research shows that the pricing models of the 3 hyperscalers, AWS, MS Azure and GCP are nearly identical. In consequence, we can simplify the calculation and just look at the two main cost positions – DBaaS flavor resource costs and storage costs.

Note: Please compare the detailed resource specs.

• The DBaaS flavor resource costs for AWS and MS Azure are nearly identical. The same goes for the storage pricing.
• GCP offers the lowest costs for the DBMS flavor, but has the most pricy storage. For our resource selection GCP CloudSQL has the lowest monthly costs, which is 15% lower than MS Azure.
• Note: Prices for cloud resources on hyperscalers are very dynamic. The costs were calculated in July 2023 and can be different now.

Performance / Costs

Considering only the costs, GCP CloudSQL would be the DBaaS product to select. But if we are looking of the performance per costs, which is a more reasonable KPI for decision-making, the picture changes.

Calculating price -performance ratios can be done in different ways (see Database Ranking). In this article, we decided to calculate the costs per transaction. The mathematically unit would be ‘costs per month / transaction per second’.

• For the TATP workload, AWS RDS for PostgreSQL shows the best price-performance due to its outstanding performance result in this benchmarking scenario. GCP CloudSQL and MS Azure Database had similar performance results, but GCP lower monthly costs.
• For the OLAP workload, Azure Database for PostgreSQL offers 33% better price-performance than AWS and 45% better than CloudSQL

Conclusion

The benchmarking results provide interesting insights for directly comparable DBaaS solutions. On the one hand, Google CloudSQL offers the most attractive pricing, but AWS RDS and MS Azure Database show the better performance and the better price/performance results for different workload types.

The reasons for the huge differences can be on several layers, resources, storage, software or DBaaS configuration. To identify these reasons in particular, much more measurement runs and in-depth analysis are required that are planned for the next months.

In conclusion, database benchmarking is very helpful in the DBaaS market to identify the best technical fit and the best economical solution. For a given workload, such a benchmarking study can help to identify the best solution and save a lot of money.

Appendix – Benchmarking Specification

Appendix – Workload & Database Specification