Route Description
GET /test-cases/:testCaseId/confs Get list of test configurations
GET /test-confs/:confId Get test configuration details
POST /test-cases/:testCaseId/confs Create test configuration
HEAD /test-confs/:confId Check if test configuration exists
PATCH /test-confs/:confId Update test configuration
DELETE /test-confs/:confId Delete test configuration
GET /test-confs/:confId/summary-history Get summary history
PATCH /test-schedules/:scheduleId Pause/resume schedule

Overview

API routes related to test configurations. A test configuration is a collection of parameters that define how to execute a load test including the scenario, concurrency, duration, regions, and breaking points.

Get list of test configurations

Retrieve a list of all test configurations within a given test case.

GET /test-cases/:testCaseId/confs

Request Parameters

None

Sample Response

[
   {
      "id":183,
      "createdAt":"2016-03-10T00:53:12.278Z",
      "updatedAt":"2016-03-11T06:58:35.080Z",
      "name":"Demo",
      "providers":[  
         {  
            "id":33,
            "createdAt":"2016-05-23T22:14:23.088Z",
            "updatedAt":"2016-05-23T22:14:23.088Z",
            "numRegions":1,
            "params":{  
               "regions":"1"
            },
            "provider":{  
               "id":0,
               "createdAt":"2016-05-06T13:56:53.328Z",
               "updatedAt":"2016-05-06T13:56:53.328Z",
               "name":"Public Shared Grid",
               "public":true,
               "providerType":"Shared",
               "active":true
            }
         }
      ],
      "durationSecs":60,
      "iterationSleepSecs":10,
      "rampupSecs":60,
      "concurrentClients":5,
      "scriptParams":"{}",
      "isTryItOut":false,
      "active":true,
      "script":{
         "id":295,
         "createdAt":"2016-03-10T00:52:50.567Z",
         "updatedAt":"2016-03-10T00:52:59.779Z",
         "name":"Load In Chrome",
         "sourceType":"Recording",
         "sourceId":187,
         "length":3702,
         "hasInit":false,
         "hasTeardown":false,
         "scriptType":"Code"
      }
   }
]

Get test configuration details

Retrieve details about a specific test configuration.

GET /test-confs/:confId

Request Parameters

  • down: Whether to include all child objects including executions and tokens. Defaults to false.
  • up: Whether or not to include the parent, the organization details. Defaults to true.
  • offset: Offset within the executions to start the response. Defaults to 0. When combined with limit it is useful for paging through the results.
  • limit: Number of executions to return. Returns all by default or if set to -1.

Sample Response

{
   "id":183,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"Demo 12",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":60,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":5,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Create test configuration

Creates a new test configuration.

POST /test-cases/:testCaseId/confs

Request Body

  • name Required: Name of the configuration.
  • scriptId Required: Scenario to execute.
  • providers Required: An array of provider configurations (i.e. test runners). Use the /providers route to find all available providers (aka test runners). For more details see the test runners guide.
  • profileType: The load profile type for this test. Flat is the default if not specified.
    • Flat: Rampup (rampupSecs) to a constant number of concurrent clients per region (concurrentClients) for the duration of the test. concurrentClients parameter is required. Both duration and iteration based tests are supported for this load profile type.
    • Step: Gradually increase the number of concurrent users in a step function. Specify the number of concurrent clients to start (startConcurrentClients), finish (concurrentClients), and the step size (step). The steps will be evenly distributed across the duration of your test. 50 steps is the most allowed per region. Only duration based tests are supported for this load profile type.
    • BreakingPoint: Let Testable find the maximum number of users that can concurrently execute your scenario without performance degrading below your acceptable performance standards. This means you must specify one or more breaking points to express what you consider unacceptable performance (triggers). Testable will gradually increase the number of concurrent clients until either one or more breaking points hit or your infrastructure successfully handles the maximum concurrent clients per test allowed under your account. Only duration based tests are supported for this load profile type.
  • durationSecs: Duration of the test in seconds. Either this field or iterations is required.
  • iterations: Number of times to execute the scenario on each concurrent client. Either this field or iterations is required. Only supported if the profileType is Flat.
  • iterationSleepSecs: Sleep time in seconds between iterations. Defaults to 10 seconds, minimum is 1 second.
  • rampupSecs: Time to ramp up to the desired number of concurrent clients. Defaults to 60 seconds, minimum is 0 seconds. Only supported if the profileType is Flat.
  • concurrentClients: Number of threads that will execute the scenario concurrently during the test per region. For Step load profiles this is the number of concurrent clients to finish with. Not required if the profileType is BreakingPoint.
  • startConcurrentClients: Number of concurrent clients to start the test with. Only required if the profileType is Step.
  • step: How many concurrent clients to add each time the test steps up during execution. Only required if the profileType is Step.
  • scriptParams: If any script parameters were configured, a string containing a JSON object of key/value pairs should be provided here.
  • triggers: Also referred to as breaking points in the UI. Each trigger has a set of conditions that cause it to fire and a set of actions that should be applied in case the trigger fires. Currently any one condition being met will cause the trigger to fire and the only supported action is to stop the test execution. Required if the profileType is BreakingPoint.
  • schedule: See below for details on how to schedule your test to run later or on a recurring schedule.

Request Example:

{
   "name": "New Test Case",
   "scriptId": 295,
   "providers":[  
      {  
         "providerId":0,
         "params":{  
            "regions":"1"
         },
         "numRegions":1
      }
   ],
   "durationSecs": 120,
   "iterationSleepSecs": 10,
   "rampupSecs": 60,
   "concurrentClients": 50,
   "triggers": [
      {
         "conditions": [ {
            "metric": "firstReceivedMs",
            "metricKey": "p50",
            "isPercentage": false,
            "operator": "GTE", 
            "threshold": 1000,
            "timePeriod": "LastMinute",
            "valueType": "Absolute"
         } ],
         "actions": [ { "actionType": "StopExecution" } ]
      }
   ]
}

Response

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Test Case",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   },
   "triggers":[  
      {  
         "id":23,
         "createdAt":"2016-09-07T05:10:14.011Z",
         "updatedAt":"2016-09-07T05:10:14.011Z",
         "conditions":[  
            {  
               "id":24,
               "createdAt":"2016-09-07T05:10:14.013Z",
               "updatedAt":"2016-09-07T05:10:14.013Z",
               "metric":"firstReceivedMs",
               "metricKey":"p50",
               "isPercentage":false,
               "timePeriod":"LastMinute",
               "operator":"GTE",
               "valueType":"Absolute",
               "threshold":1000.00
            }
         ],
         "actions":[  
            {  
               "id":22,
               "createdAt":"2016-09-07T05:10:14.017Z",
               "updatedAt":"2016-09-07T05:10:14.017Z",
               "actionType":"StopExecution"
            }
         ]
      }
   ]
}

Scheduling

Tests can be scheduled to run later or on a recurring schedule using the schedule property of the request body.

Run Once Later

{ 
   "scheduleType": "Single", 
   "nextScheduled": 1492796529000 
}

Recurring Schedule

Minute example (Every 90 minutes):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Minute", 
   "minute": 90 
}

Hourly example (Every 2 hours at 30 minutes passed the hour):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Hour", 
   "hour": 2, 
   "minute": 30 
}

Week day example (Every week day at 13:30 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "WeekDay", 
   "hour": 13, 
   "minute": 30 
}

Daily example (Every day at 13:30 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Day", 
   "hour": 13, 
   "minute": 30 
}

Weekly example #1 (Bi-weekly on Thursday at 01:00 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Week", 
   "week": 2, 
   "dayOfWeek": 4, 
   "hour": 1, 
   "minute": 0 
}

Weekly example #2 (Every Monday at 01:00 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Week", 
   "dayOfWeek": 1, 
   "hour": 1, 
   "minute": 0 
}

Monthly example (Every 2 months on the 1st at 13:00 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Month", 
   "month": 2, 
   "hour": 13, 
   "minute": 0 
}

Schedule Exclusions

If your test runs on a recurring schedule you can define one or more exclusion windows in which the test should not run. If the next run falls inside any exclusion window that test run will be skipped.

Exclusions can be on the following recurring intervals: Month, Week, WeekDay, Day, Hour. All examples below assume the following recurring schedule (every 10 minutes):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Minute", 
   "minute": 90,
   "exclusions": [ ... ]
}

Monthly exclusion (from the 1st @ 22:00 UTC until the 3rd @ 02:00 UTC):

{
   "recurringInterval": "Month",
   "startDay": 1,
   "endDay": 3,
   "startHour": 22,
   "endHour": 2,
   "startMinute": 0,
   "endMinute": 0
}

Weekly exclusion (from Friday @ 17:00 UTC until Monday @ 05:00 UTC):

{
   "recurringInterval": "Week",
   "startDayOfWeek": 5,
   "endDayOfWeek": 1,
   "startHour": 17,
   "endHour": 5,
   "startMinute": 0,
   "endMinute": 0
}

Daily exclusion (from 22:00 UTC until 02:00 UTC):

{
   "recurringInterval": "Day",
   "startHour": 22,
   "endHour": 2,
   "startMinute": 0,
   "endMinute": 0
}

Week day exclusion (from 22:00 UTC until 02:00 UTC):

{
   "recurringInterval": "WeekDay",
   "startHour": 22,
   "endHour": 2,
   "startMinute": 0,
   "endMinute": 0
}

Hourly exclusion (from 0-15 minutes passed the hour):

{
   "recurringInterval": "Hour",
   "startMinute": 0,
   "endMinute": 15
}

Check if test configuration exists

HEAD /test-confs/:confId

Response

HTTP Status: 200 OK or 404 NOT FOUND

Update test configuration

Update the test configuration parameters. All parameters available when creating can be specified during update. To delete the associated schedule use "deleteSchedule": true in your request.

PATCH /test-confs/:confId

Request Body

{
    "name": "New Name"
}

Response

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Name",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Delete test configuration

DELETE /test-confs/:confId

Response

The deleted test configuration

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Test Case",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Get test configuration details

Retrieves the metric summary for the most recent executions of this test configuration.

GET /test-confs/:confId/summary-history

Request Parameters

  • down: Whether to include metrics as part of each summary. Defaults to false.
  • region: Retrieve the summaries for one specific region if specified. This is the region id found in the GET /region API. Defaults to empty (i.e. returns summary aggregated across all test regions).
  • maxExecutionId: The last test execution to include in the results. No executions after that point will be returned. Defaults to empty which means the most recent executions are returned.
  • limit: Number of recent executions to return in the results. Defaults to 10.

Sample Response

[
   {
      "id":383367,
      "createdAt":"2016-04-21T03:32:08.840Z",
      "updatedAt":"2016-04-21T03:32:08.840Z",
      "executionId":1135,
      "intervalStart":0,
      "iterationsExecuted":42,
      "count":294,
      "concurrentClients":6,
      "actualConcurrentClients":0,
      "maxConcurrentClients":6,
      "memory":100118528,
      "cpu":12.0,
      "timespan":80000,
      "metrics":[
         {
            "metricDef":"httpResponseCode",
            "metricType":"Histogram",
            "metricValueMap":{
               "200":210,
               "204":42,
               "301":42
            }
         },
         {
            "metricDef":"dnsLookupMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":2,
               "min":1,
               "p95":5,
               "max":21,
               "p99":9,
               "sd":1,
               "p50":2,
               "sum":699,
               "var":3
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"success",
            "metricType":"Counter",
            "metricValue":294,
            "metricUnits":"requests"
         },
         {
            "metricDef":"connectionOpenMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":3,
               "min":2,
               "p95":8,
               "max":23,
               "p99":11,
               "sd":2,
               "p50":3,
               "sum":1153,
               "var":4
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"bandwidth",
            "metricType":"Counter",
            "metricValue":10242207,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"httpMethod",
            "metricType":"Histogram",
            "metricValueMap":{
               "GET":294
            }
         },
         {
            "metricDef":"bytesReceived",
            "metricType":"Counter",
            "metricValue":10146405,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"bytesSent",
            "metricType":"Counter",
            "metricValue":95802,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"firstReceivedMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":20,
               "min":3,
               "p95":61,
               "max":170,
               "p99":73,
               "sd":21,
               "p50":15,
               "sum":6146,
               "var":453
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"dataPacketsReceived",
            "metricType":"Counter",
            "metricValue":1645,
            "metricUnits":"packets"
         },
         {
            "metricDef":"connectionCloseMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":22,
               "min":4,
               "p95":63,
               "max":173,
               "p99":76,
               "sd":21,
               "p50":17,
               "sum":6717,
               "var":463
            },
            "metricUnits":"ms"
         }
      ]
   }
]

Pause/resume Schedule

If your test configuration has a recurring schedule, it can be paused or resumed using this endpoint. To find the scheduleId load the test configuration JSON and it can be found at schedules[0].id.

PATCH /test-schedules/:scheduleId

Request Body

{
   "active": true
}

Pass active value of true to resume the schedule or false to pause it.

Result

Returns 200 status code if successful with no body.