New Azure VMs - Database Benchmark Performance Analysis (2024)

Table of Contents

• Chapter 1: Key Insights
• Chapter 2: About the Azure VM Update 2023
  • Excursion: Azure VM Types
    • Cloud Mercato vCPU Results
• Chapter 3: MongoDB Performance Benchmarks On New Azure VMs
  • Throughput
  • Read Latency
  • Write Latency
  • Price/Performance
• Chapter 4: Conclusion
• Appendix
  • Benchmarking Specifications
  • Workload Specifications

Key Insights - Azure VMs

• Microsoft Azure has released a new VM generation (v5) of the latest CPUs in fall 2023.
• The new VMs are priced (almost everywhere) identically to their v4 generation counterparts.
• Azure advertises a significantly better price/performance ratio of the v5 generation compared to the v4 generation
• Cloud Mercato has published a hardware benchmarking study on this, in which the new generation performs significantly better in CPU benchmarks.
• We benchmarked the identical VMs for MongoDB for two different VM types against a read-only and rea-update workload and measured significant performance gains.
• MongoDB throughput has improved by 50%, and 38% respectively.
• The read latency has decreased by 12%, respectively 70%.
• Write latency has benefited slightly (8%) from the new v5 VM generation.
• The identical pricing of the v4 and v5 VM generation results in clear price/performance advantages for the V5 generation
• The 2 different workloads deliver extremely different performance figures, but there is a clear tendency to recognize that the new v5 VM machines represent a technical upgrade.

About the Azure VM Update 2023

In the fall of 2023, Microsoft Azure released a new VM generation on newer processors.

According to Documentation, these include the following processors:

• Dv5/Dsv5: Intel Xeon Platinum 8473C or Intel Xeon Platinum (Ice Lake) in a hyperthreaded configuration
• Ddv5/Ddsv5: Intel® Xeon® Platinum 8370C (Ice Lake)
• Dasv5/ Dadsv5: AMD EPYCTM 7763v of the 3rd generation

Microsoft Azure advertises a significantly better price-performance ratio. Interestingly, the VMs of the new generation v5 are priced identically to comparable VM types of the v4 generation for almost all VM types and in all regions.

Is it worth switching to the new VMs?

Excursion: Azure VM Types

Excursion: Azure VM Types Before we look at the measurement results, let's make a small digression to understand the cryptic VM designation of Azure. Capital letters denote the VM series and the intended purpose or suitability of the VM:

• D: General purpose VM instances with a balanced ratio of vCPU to RAM
• F: Compute-optimized - high ratio of vCPU to RAM
• E: Memory-optimized - high ratio of RAM to vCPU
• M: Memory-optimized with very high vCPU
• L: Data memory-optimized
• N: GPU-optimized

Lowercase letters indicate additional features of the VM instances:

• a: AMD-based processor
• p: ARM CPU
• d: local disk available
• i: isolated VM
• l: low RAM to vCPU ratio
• m: higher RAM to vCPU ratio
• t: very low RAM
• s: Premium storage available, incl. SSD Ultra

Cloud Mercato vCPU Benchmarking Results

Shortly after the release, our esteemed colleagues from Cloud Mercato published a first performance measurement series of the new v5 VMs.

Cloud Mercator measures the performance and price/performance for various hardware benchmarks such as Geekbench and Sysbench .

• In the CPU performance measurements, all new v5 VMs impress with a performance increase of over 100% in some cases.
• The identical pricing of the new v5 to the older v4 machines also results in a significantly better price/performance ratio.
• Only the RAM bandwidth measurements with Sysbench give a mixed picture. Here, some older versions deliver significantly better performance (see link above).

How does this apply to databases?

Do database management systems also benefit from the improved hardware?

MongoDB Performance Benchmarks on New Azure VMs

To better understand the impact of the hardware update on databases, we have designed the following benchmarking scenario.

• D8s_v4 vs D8s_v5
• Disk: standard_LRS
• region: westeurope (identical to Cloud Mercatos choice)
• DBMS: MongoDB v6
• 2 workloads
  • A: read-only, memory-bound (YCSB)
  • B: read-update, disk/memory-bound (YCSB)

The workload A is so designed, that nearly the whole dataset is cached and the database requests are handled by the cache. With workload B, on the other hand, only part of the data is in the cache, the remaining data must be read from the disk. Of course, all write operations go to the disk.

This workload design allows us to measure the pure differences between the CPU and the two VM generations in a realistic scenario.

The full details of the benchmarking configuration can be found in Appendix

Throughput Results - MongoDB on Azure VMs

• MongoDB benefits significantly from the new hardware in the read-only workload. A performance increase of over 50% is possible with the D8s_v5 VM.
• MongoDB also shows an increase of almost 38% for the read-update workload with the new hardware.
• The throughput values for the read-update workload are significantly lower, as this workload is also disk-bound and not completely handled by the cache as the read-only workload. In this scenario, a lower throughput can certainly be expected for the read-update, however not to this extent which will entail a more detailed analysis by also considering different storage types besides the standard_LRS.

Read Latency Results - MongoDB on Azure VMs

• The read latency of MongoDB decreases by almost 12% in our measurements for the read-only workload.
• With the read-update workload, on the other hand, the read latency is reduced by almost 70% from 42 ms with the D8s_v4 to 14.5 ms with the D8s_v5.
• In general, the read latency is significantly higher with the read-update workload, as here the data is also read from the disk and not just from the cache.

Write Latency Results - MongoDB on Azure VMs

• The write latency improves marginally by 8% for the read-update workload.
• However, it must also be mentioned here that the write latency is very high at over 500ms. Together with the low throughput figures, this is a point that should be analyzed in greater depth.

Note: For the read-only workload the write latency does not exists due to no write operations.

Price/Performance Results - MongoDB on Azure VMs

• The selected VMs are both priced identically at $0.46/h (or $335.80/month) in the Western Europe region.
• Based on the throughput/figures above, this also results in significant price/performance advantages - so you get more performance for the same money!

Conclusion

The new VMs introduced by Microsoft Azure deliver a significant increase in performance. This is not only measurable at the hardware level, but also for databases. We did not measure a 100% improvement in performance like Cloud Mercato on the hardware level, but also significant improvements for database applications.

For the use case shown above, it is worth switching to the new VM generation, as not only the throughput but also the latencies improve significantly - and at the same price.

Of course, this result is a snapshot, as always in the cloud, and is only meaningful for specific use cases. However, a certain trend is recognisable overall. We are going to extend these measurements to other hardware configurations and publish them if anything interesting shows up.

Please contact us if you are interested in performance figures for other VM sizes, databases, and setups. We will be happy to help you!

Appendix - Benchmark Specification

All measurement results can be found in our GitHub repository, including all applied configs and a full audit of the benchmarking runs.

Appendix – Workload Specification