ARM Instances for DBMS: Part 2 – Are Larger ARM Instances Better for Databases?

The BenchmarkSetting

The AWS ARM chips are the talk of the town: more economical consumption, more efficient performance, and cheaper cloud prices! The German-language trade magazine iX has also published a large article with performance analyses.

In the first part of the series we gave small ARM instances (2 cores, 4 GB RAM) and asked ourselves:

Are ARM VMs also suitable for efficiently running a DBMS instance for typical web-oriented workloads?

The small ARM VMs could not really convince us. They were even inferior to the x86 instances in their advertised core competence (price/performance).

However, we didn't leave it alone with these results and now also measured larger ARM VMs (8 cores x 16 GiB RAM).

Cloud

For a fair comparison, we chose the following three instance types, each with 8 cores and 16 GiB:

1. a1.2xlarge: AWS Graviton processors with 64-bit ARM Neoverse cores.
2. c5.2xlare: AWS Intel Xeon Scalable processor (2nd Gen) for data processing and compute-intensive workloads.
3. c6i.2xlarge: AWS Intel Xeon Scalable processor (3rd Gen) for data processing and compute-intensive workloads with increased bandwidth.

Virtual machines of this size as a single-node instance are suitable for running a database for a small web application, according to AWS.

Database

As a database, we use one of the most popular and widespread NoSQL databases.

• MongoDB (version 4.4.6) in the vanilla default configuration.
• as a single-node instance without replication.

This database is installed identically and fully automated by our platform on all VMs and corresponds to the identical database setting of the first part of the series.

Benchmark

The workload was slightly changed compared to the first part of the series and scaled along to match the larger instances.

As a benchmark, we again use the Yahoo Cloud Serving Benchmark in version 0.17.0 with the same parameters.

• Data set size: 5 KB
• Initial data size: 10 GB
• Read-Write distribution:
  • A) Social-Media: 50% Read / 50% Write
  • B) eCommerce: 90% Read / 10% Read
• Read access pattern:
  • A) Social-Media: ZIPFIAN
  • B) eCommerce: LATEST

The full parameters can be found in the YCSB command:

The two defined workloads A) social media and B) eCommerce reflect typical web-based workloads. With the help of these ideal-typical workloads, the performance of the different setups is evaluated and compared.

The workload is sized to be suitable for the selected instances while not representing a pure in-memory workload. The entire data volume cannot be kept completely in memory due to the size.

Benchmark execution

The benchmarks were measured in a fully automated manner using the benchANT platform.

benchANT's benchmarking platform is based on over 7 years of research & development in a university context and is an integral part of the completed PhD of my colleague Dr. Daniel Seybold.

Each configuration setup was measured three times and the data aggregated at the end. The performance measurement under load ran for 30 minutes per benchmark run.

As performance KPIs we consider

1. the Throughput,
2. the Costs (cloud compute costs + storage)
3. the Throughput/Cost Ratio
4. the Read Latency (95% quantile) and
5. the Write Latency (95% quantile).

Throughput, Costs and Throughput/Cost-Ratio

Which VM type is best suited for running a cloud database for web-based workloads.

We look at the throughput (operations / second) and relate this value to the costs to make a statement about the economic efficiency of the instances.

Cost

AWS promotes its ARM instances as cost-saving alternatives. In this comparison, the ARM instance is 40% cheaper than the identically priced c5 & c6 VMs.

The on-demand prices (compute + storage) of the different instances for one month are:

• a1.2xlarge: $174 / 153 €
• c5.2xlarge: $273 / 241 €
• c6i.2xlarge: $273 / 241 €

Note: Prices differ depending on the AWS cloud region. In addition, significantly lower monthly prices can be achieved through reserved instances.

Throughput

The small ARM instances in the first benchmarking part fell off significantly against their x86 alternatives. This is not as extreme the case with the larger a1.2xlarge instance.

eCommerce

1. c6i.2xlarge: 22,900 ops/s
2. c5.2xlarge: 21,900 Ops/s
3. a1.2xlarge: 18,200 Ops/s

Social Media

1. c5.2xlarge: 19,800 ops/s
2. c6i.2xlarge: 19,000 ops/s
3. a1.2xlarge: 15,200 Ops/s

The ARM instances only show a 20-24% lower throughput than the expensive x86 VMs here. For comparison: The small ARM instances had over 50% less throughput in some cases.

Throughput/cost ratio

From the results so far, it quickly becomes clear that the a1.2xlarge have a much more attractive price/performance ratio.

eCommerce

1. a1.2xlarge: 118 ops/€
2. c6i.2xlarge: 95 Ops/€
3. c5.2xlarge: 91 Ops/€

Social Media

1. a1.2xlarge: 99 Ops/€
2. c5.2xlarge: 82 Ops/€
3. c6i.2xlarge: 79 Ops/€

For both workloads, the ARM instance shows a better price/performance ratio. This was not the case for the smaller a1.large, which ended up in last place in each case.

Read-Latency & Write-Latency

In addition to looking at throughput and cost, for some web applications it is also useful to look at the read & write latency of the cloud database setup to provide a fast response to the end user.

Lower latency is always better than high latency in this regard.

In each case, we look at the 95% percentile, meaning that 95% of all operations were faster than the specified values.

Write latency

As with the small ARM instance, the larger a1.2xlarge also shows significantly worse write latency. This applies to both workloads, but is once again significantly worse for the more write-intensive social media workload.

1. c6i.2xlarge: 1.0 ms (social media) - 2.0 ms (eCommerce)
2. c5.2xlarge: 1.5 ms (social media) - 2.4 ms (eCommerce)
3. a1.2xlarge: 6.8 ms (eCommerce) - 9.8 ms (social media)

However, write latency is more often much less important than read latency for web applications.

Read latency

In contrast, the ARM instances are again significantly faster in terms of read latency. This was already the case with the small a1.large instance.

1. a1.2xlarge: 7.1 ms (eCommerce) - 10.5 ms (social media)
2. c5.2xlarge: 15.0 ms (eCommerce) - 17.6 ms (social media)
3. c6i.2xlarge: 14.6 ms (eCommerce) - 20.4 ms (Social-Media)

Just as with the smaller instances, the a1 VM is faster and the c6i VM has significantly higher latencies.

Conclusion

This second ARM instance evaluation provides a different assessment than in part 1. The a1.2xlarge VM convinces with a super price-performance ratio as well as low read latencies and thus offers itself as a low-cost database instance.

In this case, the ARM instances fulfill the promises and the users' expectations.

Once again it shows: “Measure everything, assume nothing!”.

While the small ARM instances delivered rather sobering performance in Part 1, the larger ARM instances can deliver value to customers from a price-performance perspective.

This results in some new VM options, especially since other cloud providers have already announced that they will also offer ARM instances, or have already done so, such as Oracle. Also, many database vendors are currently porting their software to the ARM architecture, so the necessary bindings and drivers will exist in the future.

What about other ARM instances on AWS, or other cloud providers? Measure for yourself!

About benchANT

benchANT is a performance engineering company from Germany. The two technical co-founders have more than 20 years of combined research experience with distributed systems and performance engineering, specifically in the area of database management systems and cloud computing.

With the benchANT platform, they have developed an automated benchmarking tool that enables any IT architect and system/database administrator to quickly and efficiently, run cloud database benchmarks and make decisions based on performance measurements.

In addition, benchANT also advises on on-prem vs. cloud decisions and helps with resource selection and performance optimization - always based on reliable performance measurements.