Was my load test successful?

May 25, 2017

Testable is now the first load testing tool to let you specify success criteria that determine if a test succeeded or failed. Read on for more details.

Outcomes and Success Criteria

With functional and unit tests the notion of success is simple. A piece of functionality or code either works or it doesn’t. Automated tests typically use assertions, which are expected to all pass when run successfully.

With load testing the situation is fuzzier. If the median response time of a service goes down by 10% is that a failure? Maybe yes, maybe no. There can be acceptable and expected variance between test results that makes defining success criteria difficult.

Load testing tools generally just give you the results and let a person manually analyze and decide whether those results are acceptable or not. Our goal with this feature is to automate that human analysis process.

Testable now supports the notion of success criteria. One or more conditions will be evaluated when the test finishes to determine whether it passed or failed. Visual indicators of success can be found across the tool to quickly identify issues. Example success criteria can include:

  • Median response time must not degrade by more than 25%

  • % of requests that return a 200 status equals 100%

  • Average throughput is greater than 1MB/sec

  • 95th percentile page load time is less than 1 second

These criteria can operate on any metric, including both Testable standard and user defined ones. Criteria can operate on the value of the metric during the test or the change in value as compared to recent test runs.

The success status is displayed across the tool in many places.

Default Success Criteria

Default success criteria are defined for every Testable account. These can be found under Account => Settings and customized according to your team’s needs.

Success criteria can also be customized per test configuration, overriding the default ones.

How is this different from breaking points?

Breaking points are evaluated continuously while your test runs and serve two purposes:

  1. Find how many concurrent users your service or website can handle before performance degrades below your team’s acceptable standards.

  2. Protect your infrastructure from the load test, stopping it if it fails to meet your performance standards. This is important especially when testing against production infrastructure.

Success criteria are different in that they only evaluate once when the test finishes. The outcome is then visible in many places across the tool to make it clear to everyone on the team.

How do I get started?

The default success criteria will be applied from today going forward. Feel free to customize these under Account => Settings to more appropriate settings for your team.

Node.js fs module

The read functions of the Node.js fs module are now available for use in your Node.js scripts. This allows you to read files that were uploaded into your test scenario as part of your test.

Try out the new features and let us know if you have any feedback or concerns. Happy testing!