FAQ
You have questions? We may already have the answer to your questions.
If not, you are welcome to contact us directly.
We regularly maintain our FAQs, expand them and keep them up to date.
benchANT is company striving to enable data-driven decision making by proving decision makers, product owners, and marketing departments an easy-to-use solution for measuring the capabilities of complex, cloud-based systems.
Herewith, benchANT's main focus lies on all types of Cloud-hosted Database Management Systems (DBMS) be they run on Infrastructure-as-a-Service, as a managed service, or as a Database-as-a-Service. Yet, with a sophisticated technology stack in-place we are also able to address servers, storage systems, and message queues.
benchANT's approach is to perform batches of benchmarks to evaluate the performance of different technologies, different cloud providers, and different hardware/software configurations. Performance here not only includes classical performance metrics such as throughput, but can also include further non-functional properties such as availability. The data gained through benchmarking can be used to address many problems including capacity planning, optimization, cost reduction, and forecasting.
Load testing aims evaluating the system behaviour under an anticipated load and compare the results with the expected (or required) outcome. Based on the comparison between should and is, it can be decided whether or not to keep a software build / software configuration or not.
The goal of software stress testing, however, is to identify the saturation point and the first bottleneck of the application under test.
In that respect benchmarking can be used as both stress and load test. Yet, usually, benchmarking is also used to produce statistically significant outcomes when comparing the performance of two different systems under test. Furthermore, benchmarks should come with a clear methodology of how to measure, what to measure, and how to evaluate the data.
For the majority of users, we recommend the use of our Professional Service at least for the first few projects. Doing so, you benefit from our years of experience when it comes to planning, configuring, and setting up benchmarks leading to a way higher benchmarking quality. Due to our high level of automation when it comes to benchmarking, only a few consulting hours are required for the the services.
For well-trained users, users that have run multiple projects with us, and users that have experience in database benchmarking, we can also open up the self-service API of our platform. The good thing is, the first consulting hour are free of charge so that you can find out if your expertise suffices or not without any risk.
- Assessment meeting: Discussion of objectives, technologies, workload, and budget. Usually for free.
- Tailored, individual offer.
- Execution of measurements (often in iterative steps).
- Discussion of results.
- Project closing or planning of next project.
Besides their benchmarking capabilities, benchANT offer several further services. These include
- Write-up service: We write a whitepaper for you based on our results
- Our data infrastructure consulting services cover all aspects of data storage and data access. We emphasize data-driven decision-making and performance/cost considerations.
- Cloud migration: Should you plan to migrate your data(base) to a cloud service we are here to support you. Our services include the choice of database technology, cloud provider, service model, and cloud resource.
- Data centre migration: You moved to the cloud, but had to find out that it does not suit you? We support you when migrating back to on-prem or third party data centres. We even cover the case of server and storage selection.
- Database migration: in case you want to get rid of legacy systems including Oracle we help you by finding out which other technology is required at what configuration. We also extend your migration workforce.
In IT, a benchmark has the goal to reliably determine the capabilities of a certain system either for the purpose to understand its maximum performance (with respect to one or multiple KPIs) or to compare it to another system that provides the same services in the same domain or compare it to a different configuration of the same system.
In any of these cases, the system to be benchmarked (system under test) will need to execute one or multiple tasks. The composition of tasks and the sequence they are executed in is called the benchmark's workload. In consequence, the choice of the workload defines the conditions under which the system is tested.
For benchmarks run to optimize configurations for production workloads, matching the workload as close as possible to the production workload is a key challenge for the benchmark design that requires a lot of experience."
To ensure comparability and fairness between different DBMS providers, benchANT usually relies on established, open source benchmark suites. For the domain of Database Management Systems these include TPC-C, TSBS, YCSB, and many others. Most benchmarks suites provide a rich set of knobs in order to fine-tune and parametrize the benchmark to run.
Which benchmark suite is used in which scenario, and how its parameters are set, depend on the goals of the evaluation. If required, benchANT will extend the benchmark suites to match the use case demands. If desired by the customer or deemed necessary for the project, benchANT will also define a custom benchmark.
For scenarios that require the optimization of production workloads, benchANT provides tools to enable a replay of production workloads.
During the benchmarking process, the workload is issued from one or multiple workload driver instances. Usually, these driver instances are being deployed automatically by the benchANT framework and reside as close as possible to the DBMS instance. Usually, this means in the same data centre or same cloud region.
For specific scenarios workloads can also be placed in different data centres and even be distributed all over the world.
Yes. This is the longest available benchmarking mode we support.
For each benchmark run, the benchANT platform acquires a new virtual machine on the IaaS and installs the DBMS there. Afterwards, it installs a workload driver instance on the same cloud provider and the same region.
The costs for the IaaS services are paid by our customers.
Yes.
For each benchmark run, the benchANT platform acquires a new DBMS instance through the provider's DBaaS API, installs a workload driver instance on a nearby location (e.g. if DBaaS is backed at AWS, the workload driver will be run at AWS, in the very same region).
The costs for the IaaS services are paid by our customers.
Yes.
If the API of the private IaaS is publicly available, things are not different from a public cloud. Otherwise, a benchANT proxy will need to be run on premises. Metric gathering may be limited on private clouds.",
Yes, this is totally possible. In this case, the benchANT platform will skip the DBMS installation step and bind to a pre-existing DBMS. Similarly, the workload driver can be pre-installed so that no SSH access to a server will be needed.
Note that using pre-installed servers may harm reproducibility of benchmarks.
Based on our years of experience in performance engineering and database benchmarking, an efficient process has been established to perform our benchmarking processes:
- Orientation: Building understanding and setting goals (this step is free and is not charged, get in contact)
- Ideation: Problem definition and option space evaluation (2-8 hours)
- Measurements: Performance measurements (2-3 days)
- Decisions: Analysis of measurements (2-4 hours)
- Implementation: Deployment, configuration, coding (by effort, if wished)
Steps 3 and 4 are usually repeated (several times) in order to be able to optimize the measurement scenarios on the basis of the initial results and to get closer to the problem or ideal solution.
You will get your data through a results dashboard provided to you in the member area of the benchANT platform. Further, you will get access to the raw data of the benchmarks that are available for download also in the member area.
Management summary and technical write-ups are also possible.
Please note that publication of the results is only permitted under the conditions regulated in the Terms & Conditions.