Monitor Success Rate Using Our API

In this example we do the following:

  1. Start a test execution
  2. Wait for it to finish. If the success rate while running dips below 50% after we have at least 5 results, stop the test immediately.
  3. Save all results to a CSV.

It assumes you have already created a test configuration and scenario via the API or website first and have the trigger URL for the configuration available, new configuration guide, new scenario guide, configuration API, and scenario API for more details.

A trigger URL is required in the first API call below.

#!/bin/bash

echo "[$(date)] Start a new execution for existing trigger"
execution_id=$(curl -X POST --silent https://api.testable.io/public/lcjsdlijvx | jq -r ".id")

# This next part keeps checking the success rate once we have 5 results until the test is done.
# If it dips below 50% it stops execution.

echo "[$(date)] Waiting for execution to complete (view online at https://a.testable.io/results/$execution_id)"
echo "[$(date)] Will stop execution if the success rate dips below 50%"
while : ; do
  echo -n "."
  sleep 5
  details=$(curl -H "X-Testable-Key:$API_KEY" --silent https://api.testable.io/executions/$execution_id)
  running=$(echo "$details" | jq -r ".running")
  if [[ $running = "true" ]]; then
    count=$(echo "$details" | jq -r ".summary.count")
    success=$(echo "$details" | jq -r '.summary.metrics | .[] | select(.metricDef=="success") | .metricValue')
    if [[ $count > 5 ]]; then
      pct=$(( $success / $count * 100 ))
      if [[ $pct < 50 ]]; then
        echo "[$(date)] Success rate down to $pct %, stopping execution"
        curl -H "X-Testable-Key:$API_KEY" -X PATCH --silent https://api.testable.io/executions/$execution_id/stop &>/dev/null
        echo "[$(date)] Stopped execution"
      fi
    fi
  fi
  [[ $(echo "$details" | jq -r ".completed") = "false" ]] || break
done

epoch=$(date +"%s")
echo "[$(date)] Storing CSV results at results-$epoch.csv"
curl -H "X-Testable-Key:$API_KEY" --silent https://api.testable.io/executions/$execution_id/results.csv > results-$epoch.csv