OpenFin application load testing

Typically if you want to load test an OpenFin application you would need reproduce the user journeys through your application using a tool like JMeter, Gatling, or Locust. This can require significant test engineering time and maintenance as your application evolves.

Testable now allows you to take your Selenium driven OpenFin functional tests and run them with multiple concurrent virtual users across many instances and locations as a load test. An example of such a functional-test-turned-load-test can be found at openfin-wdio-testable-example.

In this post we will walk you through step by step how this test works and how to get it running on Testable.

The basic approach

The basic idea here relies on the fact that an OpenFin application is simply a wrapper around a Chromium application. This means we can use Selenium + chromedriver to launch our OpenFin application and drive it’s behavior using any Selenium bindings. For our example we choose Webdriver.io as our bindings of choice.

The Webdriver.io project

Our Webdriver.io project follows a familiar pattern with a couple of small adjustments to get it to launch our OpenFin application. Let’s review each file in the project, it’s purpose, and any special settings related to OpenFin testing.

package.json

Webdriver.io runs on the Node.js runtime so we need to provide a package.json file with all dependencies required to run our test. No special dependencies are required to run an OpenFin application.

{
  "name": "openfin-wdio-testable-example",
  "version": "1.0.0",
  "private": true,
  "dependencies": {
    "chai": "4.2.0",
    "lodash": "4.17.14",
    "webdriverio": "5.10.0",
    "@wdio/cli": "5.10.0",
    "@wdio/dot-reporter": "5.9.3",
    "@wdio/concise-reporter": "^5.9.3",
    "@wdio/local-runner": "5.10.0",
    "@wdio/sync": "5.10.0",
    "@wdio/mocha-framework": "5.9.4"
  }
}
wdio.conf.js

In our Webdriver.io test runner configuration we need to set it up so that Chromedriver runs OpenFin instead of the Chrome browser. Luckily chromeOptions -> binary serves this exact purpose. On Testable, the config file will be found at process.env.CONFIG_URL. When running locally we need to make sure to point it at the right config URL depending on operating system:

const isWinOS = process.platform === 'win32';
const launchTarget = isWinOS ? 'RunOpenFin.bat' : `${process.cwd()}/RunOpenFin.sh`;
const CONFIG_URL = process.env.CONFIG_URL || (isWinOS ? `${process.cwd()}\\app_sample.json` : `${process.cwd()}/app_sample.json`);

exports.config = {
  specs : [
    'test.js'
  ],
  capabilities : [
    {
      browserName   : 'chrome',
      chromeOptions : {
        extensions : [],
        binary     : launchTarget,
        args       : [
          `--config=${CONFIG_URL}`
        ]
      }
    }
  ],
  host           : 'localhost',
  port           : 9515,
  reporters      : ['dot', 'concise'],
  path           : '/',
  logLevel       : 'error',
  coloredLogs    : true,
  framework      : 'mocha',
  waitforTimeout : 20000,
  mochaOpts      : {
    ui        : 'bdd',
    timeout   : 500000
  }
};

Notice that when Chromedriver launches the “browser” it will also include the URL to the OpenFin application configuration.

RunOpenFin.bat

The Chrome launch file for Windows is in charge of launching our OpenFin runtime. Extracts the config URL and remote debug port from the command line arguments passed from Chromedriver:

:loop  
 IF "%1"=="" GOTO cont  
 SET opt=%1
 IF "%opt%" == "--remote-debugging-port" (
    SET debuggingPort=%2
 )
 IF "%opt%" == "--config" (
    SET startupURL=%2
 )
 SHIFT & GOTO loop

It also sets the path to the OpenFin runtime binary on the Testable test runner:

SET openfinLocation=C:\Users\Administrator\AppData\Local\OpenFin

Finally it launches the OpenFin runtime:

%openfinLocation%\OpenFinRVM.exe --config=%startupURL% --runtime-arguments="--remote-debugging-port=%debuggingPort%"
app.json

A standard OpenFin application config file is required.

test.js

The test specification that uses Webdriver.io commands to drive the OpenFin application. See the test.js code for more details. There are a couple of special steps to switch into an OpenFin application window and to wait for the OpenFin runtime to initialize.

Turning it into a Testable load test

Start by signing up and creating a new test case using the New Test Case button on the dashboard.

Enter the test case name (e.g. OpenFin Demo) and press Next.

Scenario

Select Selenium as the scenario type.

Selenium Scenario

Let’s use the following settings:

1. Bindings: Webdriver.io.
2. Source: Version Control.
3. Repository: Add New… use Name = openfin-wdio-testable-example, Url = https://github.com/testable/openfin-wdio-testable-example.git.
4. Branch: master for Webdriver.io v5, wdio-v4 for Webdriver.io v4.
5. Webdriver.io Conf File: wdio.conf.js
6. Runtime Requirements: Windows. OpenFin is built to run on Windows. Testable’s test runner is available for Windows. You can also run OpenFin on Linux and our demo repository supports that as well.
7. Browser Binary: OpenFin.
8. OpenFin Config URL: app_sample.json.

General Settings

Next, click on the Configuration tab or press the Next button at the bottom to move to the next step of configuring your test.

Configuration

Now that we have the scenario for our test, we need to configure our load parameters including:

1. Concurrent Users Per Location: Number of users that will execute in parallel in each region selected. Each user will execute the scenario (i.e. launch OpenFin and perform the test steps).
2. Test Length: Select Iterations to have each client execute the scenario a set number of times regardless of how long it takes. Choose Duration if you want each client to continue executing the scenario for a set amount of time (in minutes).
3. Location(s): Choose the location in which to run your test and the [test runner source](/guides/test-runners.html) that indicates which test runners to use in that location to run the load test. For OpenFin tests, you will need to select either AWS On Demand – Testable Account or your own AWS account.
4. Test Runners: We can use the default test runner recommendation of a single c5.xlarge instance.

And that’s it! Press Start Test and watch the results start to flow in.

For the sake of this example, let’s use the following parameters:

Test Configuration

View Results

Once the test starts executing, Testable will distribute the work out to the EC2 instances that Testable spins up.

Test Results

In each region, the Testable runner runs the OpenFin test with 5 virtual users on 1 EC2 instance, each running the test 3 times with a 10 second pause between test iterations.

The results will include screenshots, assertions, traces, performance metrics, logging, breakdown by URL, analysis, comparison against previous test runs, and more.

That’s it! Go ahead and try these same steps with your own scripts and feel free to
contact us with any questions.