On Premises Test Execution

Introduction

Testable test runners are responsible for executing test cases and reporting back the results. Shared runners are provided in regions across the globe for your use. In certain cases it makes sense for users to have their own private runners. These situations include:

  • Your product sits behind a firewall and is not reachable from outside that firewall.
  • Your test requires IP address whitelisting. The global shared agents are dynamically allocated and removed so a fixed range of IPs cannot be guaranteed.
  • Ensure complete isolation from other user's test cases.
  • Leverage spare capacity on your infrastructure.

Prerequisites

Install Docker

Docker must be installed before you can start the agent. Instructions on how to install docker can be found at https://docs.docker.com/engine/installation/.

Create an API Key

Login to Testable and go to Settings -> Api Key. Create a new API key. You will need this to run the agent.

Outbound Network Access

The test runner will attempt to connect to https://agents.testable.io (port 443) to receive work and report on results. Make sure outbound network access is available from the host.

A quick way to check is to run: telnet agents.testable.io 443 and ensure it connects successfully.

Running

docker run -e AGENT_REGION_NAME=[region-name] -e AGENT_KEY=[api-key] -v /logs --privileged --security-opt seccomp:unconfined -d testable/agent:alpha

That's it. The region name can be anything you want. This name will appear in the Regions multi-select when you create or update your test configuration and select the My Shared Grid test runner. The AGENT_KEY is the API key you created earlier (Settings -> Api Keys).

The equivalent docker-compose.yml if using Docker Compose:

testable:
  image: testable/agent:alpha
  network: bridge
  environment:
    - AGENT_REGION_NAME=[region-name]
    - AGENT_KEY=[api-key]
  volumes:
    - /logs
  security_opts:
    - seccomp:unconfined
  privileged: true

Full set of accepted environment variables:

  • AGENT_REGION_NAME (required): The name of the region.
  • AGENT_KEY (required): Api key to authenticate your agent with (create api keys at Settings -> Api Keys)
  • AGENT_LOG_LEVEL: The logger level. Choices are (error, warn, info, verbose, debug). Defaults to info.
  • AGENT_LOG_CONSOLE: Whether or not to log to the console. Defaults to false.
  • AGENT_CONCURRENT_CLIENTS: Number of concurrent clients the agent can execute at any one time. Defaults to 500.
  • AGENT_REGION_DESCRIPTION: The description that appears with the region name when configuring test cases. Defaults to the region name if not specified.
  • AGENT_UPDATE_LAT_LONG: A boolean that indicates whether or not to update the location of your agent at runtime. Defaults to false. If true, Testable will geo locate your IP address. On the test results, your agent will appear at this location on the map.
  • AGENT_LATITUDE: The latitude at which your agents are located. If set along with AGENT_LONGITUDE, your region's location will be updated to these values and the geo location of your IP address ignored.
  • AGENT_LONGITUDE: The longitude at which your agents are located. If set along with AGENT_LATITUDE, your region's location will be updated to these values and the geo location of your IP address ignored.
  • PUBLIC_HOSTNAME: Optionally specify the hostname of the server running the docker container. This is important if you run multiple test runners on one server so that Testable does not count available memory/cpu multiple times.

Logging

Logging is written to /logs/agent.log within the container. Mount the /logs volume if you would like to access the log from outside of Docker. 10 days worth of logs are kept in the container using a daily rolling strategy.

Execute a Test Case In Your Region

Create a new test case, and during the Configuration step select My Shared Grid as a Test Runner. You should see your region in the list. Select it and when your test starts it will execute on one of your test runners.

Note that you can have multiple regions of your own each with multiple agents.

Upgrading

To upgrade to the latest version of the test runner software:

docker pull testable/agent:alpha

Flash enabled Docker image

The default Testable Docker image, testable/agent:alpha, does not have the Flash plugin installed.

To test a Flash enabled website, use the testable/agent-flash:alpha image instead.

Troubleshooting

I don't see my region in the list?

Try reloading the Testable page and it should appear as a region to select for your test configuration. If it still doesn't appear, check the outbound network access to agents.testable.io.

My tests keep crashing when run on my test runner.

If you set a -m or --memory option with docker run it is possible you have assigned the container a limited amount of memory that is insufficient to run your test. Use docker inspect [container-id] to check the memory settings of your container.

I use Selenium Webdriver.io + Chrome and the browser is crashing.

There are two potential issues here:

1. Shared Memory

Chrome relies heavily on shared memory and Docker by default does not always provide much shared memory to the container.

To check this issue first run ipcs -lm on the instance. Check the output for max total shared memory (kbytes).

Next run the same command on your Docker container: docker exec [container-id] ipcs -lm and observe the output for max total shared memory (kbytes). If the shared memory in your container is lower this is likely an issue.

Recent version of docker support a setting for shared memory: --shm-size 4G. If that does not work you can try mapping shared memory as a volume: -v /dev/shm:/dev/shm or -v /run/shm:/dev/shm depending on the location of shared memory on the host. Run mount -t tmpfs to find out the location of shared memory on your host looking for tmpfs on /XXX/shm type tmpfs (rw,nosuid,nodev) in the output. /XXX/shm would be the shared memory location on your host.

2. Docker security settings

On some Linux installations (e.g. Ubuntu 16.04) the default security settings for your docker container are not sufficient for Chrome to launch.

We can fix this by ensuring our docker run command includes --security-opt seccomp:unconfined. Check out this for a more restricted security profile that should be sufficient for Chrome.