Route Description
GET /test-cases/:testCaseId/confs Get list of test configurations
GET /test-confs/:confId Get test configuration details
POST /test-cases/:testCaseId/confs Create test configuration
HEAD /test-confs/:confId Check if test configuration exists
PATCH /test-confs/:confId Update test configuration
DELETE /test-confs/:confId Delete test configuration
GET /test-confs/:confId/summary-history Get summary history

Overview

API routes related to test configurations. A test configuration is a collection of parameters that define how to execute a load test including the scenario, concurrency, duration, regions, and breaking points.

Get list of test configurations

Retrieve a list of all test configurations within a given test case.

GET /test-cases/:testCaseId/confs

Request Parameters

None

Sample Response

[
   {
      "id":183,
      "createdAt":"2016-03-10T00:53:12.278Z",
      "updatedAt":"2016-03-11T06:58:35.080Z",
      "name":"Demo",
      "providers":[  
         {  
            "id":33,
            "createdAt":"2016-05-23T22:14:23.088Z",
            "updatedAt":"2016-05-23T22:14:23.088Z",
            "numRegions":1,
            "params":{  
               "regions":"1"
            },
            "provider":{  
               "id":0,
               "createdAt":"2016-05-06T13:56:53.328Z",
               "updatedAt":"2016-05-06T13:56:53.328Z",
               "name":"Public Shared Grid",
               "public":true,
               "providerType":"Shared",
               "active":true
            }
         }
      ],
      "durationSecs":60,
      "iterationSleepSecs":10,
      "rampupSecs":60,
      "concurrentClients":5,
      "scriptParams":"{}",
      "isTryItOut":false,
      "active":true,
      "script":{
         "id":295,
         "createdAt":"2016-03-10T00:52:50.567Z",
         "updatedAt":"2016-03-10T00:52:59.779Z",
         "name":"Load In Chrome",
         "sourceType":"Recording",
         "sourceId":187,
         "length":3702,
         "hasInit":false,
         "hasTeardown":false,
         "scriptType":"Code"
      }
   }
]

Get test configuration details

Retrieve details about a specific test configuration.

GET /test-confs/:confId

Request Parameters

  • down: Whether to include all child objects including executions and tokens. Defaults to false.
  • up: Whether or not to include the parent, the organization details. Defaults to true.
  • offset: Offset within the executions to start the response. Defaults to 0. When combined with limit it is useful for paging through the results.
  • limit: Number of executions to return. Returns all by default or if set to -1.

Sample Response

{
   "id":183,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"Demo 12",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":60,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":5,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Create test configuration

Creates a new test configuration.

POST /test-cases/:testCaseId/confs

Request Body

  • name Required: Name of the configuration.
  • scriptId Required: Scenario to execute.
  • providers Required: An array of provider configurations (i.e. test runners). Use the /providers route to find all available providers (aka test runners). For more details see the test runners guide.
  • profileType: The load profile type for this test. Flat is the default if not specified.
    • Flat: Rampup (rampupSecs) to a constant number of concurrent clients per region (concurrentClients) for the duration of the test. concurrentClients parameter is required. Both duration and iteration based tests are supported for this load profile type.
    • Step: Gradually increase the number of concurrent users in a step function. Specify the number of concurrent clients to start (startConcurrentClients), finish (concurrentClients), and the step size (step). The steps will be evenly distributed across the duration of your test. 50 steps is the most allowed per region. Only duration based tests are supported for this load profile type.
    • BreakingPoint: Let Testable find the maximum number of users that can concurrently execute your scenario without performance degrading below your acceptable performance standards. This means you must specify one or more breaking points to express what you consider unacceptable performance (triggers). Testable will gradually increase the number of concurrent clients until either one or more breaking points hit or your infrastructure successfully handles the maximum concurrent clients per test allowed under your account. Only duration based tests are supported for this load profile type.
  • durationSecs: Duration of the test in seconds. Either this field or iterations is required.
  • iterations: Number of times to execute the scenario on each concurrent client. Either this field or iterations is required. Only supported if the profileType is Flat.
  • iterationSleepSecs: Sleep time in seconds between iterations. Defaults to 10 seconds, minimum is 1 second.
  • rampupSecs: Time to ramp up to the desired number of concurrent clients. Defaults to 60 seconds, minimum is 0 seconds. Only supported if the profileType is Flat.
  • concurrentClients: Number of threads that will execute the scenario concurrently during the test per region. For Step load profiles this is the number of concurrent clients to finish with. Not required if the profileType is BreakingPoint.
  • startConcurrentClients: Number of concurrent clients to start the test with. Only required if the profileType is Step.
  • step: How many concurrent clients to add each time the test steps up during execution. Only required if the profileType is Step.
  • scriptParams: If any script parameters were configured, a string containing a JSON object of key/value pairs should be provided here.
  • triggers: Also referred to as breaking points in the UI. Each trigger has a set of conditions that cause it to fire and a set of actions that should be applied in case the trigger fires. Currently any one condition being met will cause the trigger to fire and the only supported action is to stop the test execution. Required if the profileType is BreakingPoint.
  • schedule: To schedule this test to run later or on a recurring schedule use this parameter. To run once later: { "scheduleType": "Single", "nextScheduled": 1492796529000 }. To run on a recurring schedule: { "scheduleType": "Recurring", "recurringInterval": "Day", "hour": 13, "minute": 30 } (i.e. run every day at 13:30 UTC). Available recurring interval types: Month, Week, Day, WeekDay, Hour, Minute. Use corresponding fields month, week, day (1-31), dayOfWeek (1-7 = Monday-Sunday), hour, minute. Some examples:
    • Every 90 minutes: { "scheduleType": "Recurring", "recurringInterval": "Minute", "minute": 90 }
    • Every 2 hours at 30 minutes past the hour: `{ "scheduleType": "Recurring", "recurringInterval": "Hour", "hour": 2, "minute": 30 }
    • Every Monday at 01:00 UTC: { "scheduleType": "Recurring", "recurringInterval": "Week", "dayOfWeek": 1, "hour": 1, "minute": 0 }
    • Every 2 months on the 1st at 13:00 UTC: { "scheduleType": "Recurring", "recurringInterval": "Month", "month": 2, "hour": 13, "minute": 0 }
    • Bi-weekly on Thursday at 01:00 UTC: { "scheduleType": "Recurring", "recurringInterval": "Week", "week": 2, "dayOfWeek": 4, "hour": 1, "minute": 0 }

Example:

{
    "name": "New Test Case",
    "scriptId": 295,
   "providers":[  
      {  
         "providerId":0,
         "params":{  
            "regions":"1"
         },
         "numRegions":1
      }
   ],
   "durationSecs": 120,
   "iterationSleepSecs": 10,
   "rampupSecs": 60,
   "concurrentClients": 50,
   "triggers": [
      {
         "conditions": [ {
            "metric": "firstReceivedMs",
            "metricKey": "p50",
            "isPercentage": false,
            "operator": "GTE", 
            "threshold": 1000,
            "timePeriod": "LastMinute",
            "valueType": "Absolute"
         } ],
         "actions": [ { "actionType": "StopExecution" } ]
      }
   ]
}

Response

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Test Case",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   },
   "triggers":[  
      {  
         "id":23,
         "createdAt":"2016-09-07T05:10:14.011Z",
         "updatedAt":"2016-09-07T05:10:14.011Z",
         "conditions":[  
            {  
               "id":24,
               "createdAt":"2016-09-07T05:10:14.013Z",
               "updatedAt":"2016-09-07T05:10:14.013Z",
               "metric":"firstReceivedMs",
               "metricKey":"p50",
               "isPercentage":false,
               "timePeriod":"LastMinute",
               "operator":"GTE",
               "valueType":"Absolute",
               "threshold":1000.00
            }
         ],
         "actions":[  
            {  
               "id":22,
               "createdAt":"2016-09-07T05:10:14.017Z",
               "updatedAt":"2016-09-07T05:10:14.017Z",
               "actionType":"StopExecution"
            }
         ]
      }
   ]
}

Check if test configuration exists

HEAD /test-confs/:confId

Response

HTTP Status: 200 OK or 404 NOT FOUND

Update test configuration

Update the test configuration parameters. All parameters available when creating can be specified during update. To delete the associated schedule use "deleteSchedule": true in your request.

PATCH /test-confs/:confId

Request Body

{
    "name": "New Name"
}

Response

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Name",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Delete test configuration

DELETE /test-confs/:confId

Response

The deleted test configuration

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Test Case",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Get test configuration details

Retrieves the metric summary for the most recent executions of this test configuration.

GET /test-confs/:confId/summary-history

Request Parameters

  • down: Whether to include metrics as part of each summary. Defaults to false.
  • region: Retrieve the summaries for one specific region if specified. This is the region id found in the GET /region API. Defaults to empty (i.e. returns summary aggregated across all test regions).
  • maxExecutionId: The last test execution to include in the results. No executions after that point will be returned. Defaults to empty which means the most recent executions are returned.
  • limit: Number of recent executions to return in the results. Defaults to 10.

Sample Response

[
   {
      "id":383367,
      "createdAt":"2016-04-21T03:32:08.840Z",
      "updatedAt":"2016-04-21T03:32:08.840Z",
      "executionId":1135,
      "intervalStart":0,
      "iterationsExecuted":42,
      "count":294,
      "concurrentClients":6,
      "actualConcurrentClients":0,
      "maxConcurrentClients":6,
      "memory":100118528,
      "cpu":12.0,
      "timespan":80000,
      "metrics":[
         {
            "metricDef":"httpResponseCode",
            "metricType":"Histogram",
            "metricValueMap":{
               "200":210,
               "204":42,
               "301":42
            }
         },
         {
            "metricDef":"dnsLookupMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":2,
               "min":1,
               "p95":5,
               "max":21,
               "p99":9,
               "sd":1,
               "p50":2,
               "sum":699,
               "var":3
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"success",
            "metricType":"Counter",
            "metricValue":294,
            "metricUnits":"requests"
         },
         {
            "metricDef":"connectionOpenMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":3,
               "min":2,
               "p95":8,
               "max":23,
               "p99":11,
               "sd":2,
               "p50":3,
               "sum":1153,
               "var":4
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"bandwidth",
            "metricType":"Counter",
            "metricValue":10242207,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"httpMethod",
            "metricType":"Histogram",
            "metricValueMap":{
               "GET":294
            }
         },
         {
            "metricDef":"bytesReceived",
            "metricType":"Counter",
            "metricValue":10146405,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"bytesSent",
            "metricType":"Counter",
            "metricValue":95802,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"firstReceivedMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":20,
               "min":3,
               "p95":61,
               "max":170,
               "p99":73,
               "sd":21,
               "p50":15,
               "sum":6146,
               "var":453
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"dataPacketsReceived",
            "metricType":"Counter",
            "metricValue":1645,
            "metricUnits":"packets"
         },
         {
            "metricDef":"connectionCloseMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":22,
               "min":4,
               "p95":63,
               "max":173,
               "p99":76,
               "sd":21,
               "p50":17,
               "sum":6717,
               "var":463
            },
            "metricUnits":"ms"
         }
      ]
   }
]