Concepts

Testable is focused on simplifying load testing without sacrificing flexibility and power. The key concepts are described here.

Scenario

A scenario is the set of steps that define a use case we want to execute at scale.

For example a scenario for an HTTP based REST API might involve:

  1. Send GET / to http://mywebsite.com/api
  2. Wait 100ms for a reply, check the status code to ensure a 200 OK.

There are a few ways to define a scenario:

  • Create a Recording: Testable will dynamically generate a URL that you can use to record all traffic to and from your service as you use the gateway URL (i.e. a recording man in the middle proxy).
  • Write a Node.js script: Write a script using the Javascript language running within a sandboxed Node.js environment to execute more complex scenarios. You can start from scratch or convert a Recording into a script as a starting point.
  • Run a JMeter Test Plan: JMeter, a popular source tool, can be used to define the scenario. Then simply use Testable to execute that test plan at scale.
  • Load Url: The simplest option. Ask Testable to either HTTP GET your URL or simulate loading your URL in a Chrome browser.
  • Write a PhantomJS or SlimerJS script: Write a script for PhantomJS, a headless WebKit browser implementation or for SlimerJS a headless Gecko (Firefox) browser implementation.
  • Write a Webdriver.io Selenium script: Write a Webdriver.io Selenium script for Nodejs.
  • Run a Gatling Simulation: Gatling, another popular source tool, can be used to define the scenario. Then simply use Testable to execute that simulation at scale.
Configuration

A test configuration define the load parameters that specify which scenario to run and how much load to generate. This includes the load profile, number of concurrent clients, iterations or duration, and the regions to execute in. The work is then split up and distributed out to eligible Testable agents.

Once the configuration is defined it can be executed many times and results of each execution can be analyzed individually or compared.

Test Runners

Test Runners do exactly that: actually run your test. We currently provide three types of test runners:

  1. A public shared grid distributed across the globe on the infrustructure of various cloud providers.
  2. A private shared grid distributed across your own infrastructure. This private grid can span both your own data centers and any cloud provider. This grid is then shared across tests run in your account only.
  3. On demand grid for a single test execution ensures full isolation. Currently this supports AWS EC2 instances only, either using Testable's account or your own. Other cloud platforms will be added in the future.

See the test runners guide for more details.

Triggers

Tests can be triggered in a variety of ways:

  1. Manual: Manually start the test execution from the Testable website.
  2. Trigger: A URL that accepts an HTTP POST request which kicks off the test. Find the trigger URL on the test configuration page -> Triggers tab. This provides a simple way to trigger test execution that can be used in Continuous Integration and other dev ops tools.
  3. Continuous Integration : Execute a test case when code is checked in by using a Trigger URL as part of your continuous integration process.
  4. Scheduled: Most CI tools support scheduled execution. Use the Trigger URL for your test configuration to launch your test via your CI tool on a regular schedule.
Analysis

The results of a test case execution are only useful if you can analyze and report on them. Testable will offer a few different ways to do this:

  1. Testable Website: Analysis, reporting, and sharing functionality on test results can be found via our web tool.
  2. API: Access results of test cases via our API and analyze them however you want. Summary stats (i.e. mean, 95% latency, etc) are also calculated and provided for you via this route.
  3. Third Party Integration: Push the results to a third party product for analysis and reporting. Currently only New Relic is supported. Stay tuned for more details on other integration partners.
Alerts

Test results are only really useful if you can define the conditions under which you want to be alerted of potential issues. Set success criteria and enable notifications on test failure. Use our platform to detect scale issues but also to monitor your infrastructure more generally.

API

Testable was designed from the ground up using the same API that is provided to customers. This means every feature you find on the website is available via the API. We hope this allows customers to build and integrate the solutions they need around the Testable core product.

Enterprise/Portability

Testable was built with the intention of being easily installable within a corporate data center or on corporate run cloud infrastructure. We will always offer our own cloud hosted infrastructure, but will also offer up an appliance that can execute completely on your infrastructure in the future. The Testable infrastructure is basically a set of Docker containers, so it is very easy to run on any hardware.

We already offer as well as a hybrid solution where the test runners are on premises and Testable hosts everything else.