WebSocket Performance Testing in 4 Simple Steps

September 10, 2018

This post is focused on how we can use Testable to performance test a WebSocket server, capture some useful custom metrics, and analyze the results.

We will use an example websocket server that supports subscribing to a stock symbol and receiving streaming summy price updates (dummy) once a second. More details on this service can be found here.

Step 1: Create a Test Case

Make sure you sign up for a Testable account first. After logging in, click the New Test Case button and give it a name (Websocket Demo).

Step 2: Write Test Script

Choose Node.js Script as the scenario type. Use the template dropdown and select Custom Metrics -> Measure subscription to first tick latency on a websocket.

You should see the following Javascript code inserted into the Code tab:

var subscribeSentAt = 0;
const ws = new WebSocket("ws://sample.testable.io/streaming/websocket");

ws.on('open', function open() {
subscribeSentAt = moment().valueOf();
ws.send('{ "subscribe": "IBM" }');

ws.on('message', function(data, flags) {
results('IBM').timing({ namespace: 'User', name: 'sub2tick', val: moment().valueOf() - subscribeSentAt, units: 'ms' });

If you are familiar with Node.js and the ws module then this code should look pretty clear already. If not let’s go through each block.

var ws = new WebSocket("ws://sample.testable.io/streaming/websocket");

This line opens a websocket connection to the sample service.

ws.on('open', function open() {
    subscribeSentAt = moment().valueOf();
    ws.send('{ "subscribe": "IBM" }');

Listen for the websocket open event. Once received, capture the current timestamp (ms) and send a subscription request for IBM.

ws.on('message', function(data, flags) {
    results('IBM').timing({ namespace: 'User', name: 'sub2tick', val: moment().valueOf() - subscribeSentAt, units: 'ms' });

There are a few things happening here:

  1. Subscribe to the message event on the websocket which gets fired every time the server sends a message.
  2. On receiving a message (the price quote), measure the latency since opening the websocket. Capture this latency as a custom metric called sub2tick, grouped by symbol (IBM), which can be aggregated and analyzed when our test runs. If grouping by symbol was not useful, simply replace results('IBM') with results().
  3. Close the websocket. Otherwise price ticks will continue indefinitely until the test times out.

To see what the price update message looks like, add a log.trace(data) line to the above code for testing. The TRACE logging level will only be available when you smoke test your script and not when running it as a load test.

This code now defines the Scenario to execute at scale.

Test out your script by pressing the Smoke Test button in the upper right. This executes it one time on a shared Testable test runner. Any captured metrics and logging will appear in the Smoke Test Output tab.

Example Smoke Test Output

Notice that Testable captures a bunch of metrics automatically in addition to the custom metric we added in our script.

Step 3: Configure a Load Test

Click Next to move onto the Configuration step. We now define exactly how to execute the scenario we defined in Step 2.

  • 10 virtual users in each region.
  • 1 minute to ramp up the virtual users.
  • 1 minute duration.
  • Two regions (AWS N Virginia and AWS Oregon).

Click the Start Test button and your test is off and running! Congratulations you have officially created and run a load test. Now let’s look at analyzing the results.

Step 4: View the Results

By now you should see results flowing in as the test executes. The default dashboard will show logging, summary, results grid, and graphs of the system captured metrics.

After the test finishes you get a nice summary of the results across the top as well.

The “sub2tick” metric can also be added to the overview, results grid, or to a new chart. Read this for more information on customizing the results view.

And that’s it! We’ve setup a test case, captured custom metrics, run it at scale, and analyzed the results.

If you don’t want to write the script yourself you can also record a websocket scenario as well.