Load Testing a TCP Based Streaming Quote Service

February 3, 2016

In the financial world, there are plenty of custom protocols and performance sensitive APIs that work over raw TCP and UDP sockets. Most of the load testing services out there do not offer any solutions in this space and at Testable we believe this is one area that sets us apart.

In this post we will show you how to use Testable to load test a streaming stock quote service that uses a simple custom protocol over TCP. We will utilize the S&P 500 symbols as part of the test, divided equally amongst the iterations of the test.

Streaming Quote Service API

The Quote Service API is available at sample.testable.io:8091.

As you can see from above the basic flow is as follows:

  1. Open TCP socket connection to the quote service
  2. Send a subscribe packet in the format “subscribe|[symbol]n”
  3. Receive quote packets once a second per subscribed symbol in the following JSON format: { "symbol": "[symbol]", "price":87.85, "exchange": "NYSE" }n.
  4. To unsubscribe, send “unsubscribe|[symbol]n”
  5. Close the TCP socket

Test Overview

Our test scenario will be as follows:

  1. Connect to the Quote Service
  2. Subscribe to 5 symbols evenly distributed from the pool of 500 potential symbols
  3. Receive updates for 30 seconds
  4. Disconnect from the Quote Service

We will then execute this scenario at scale under the following configuration:

  1. 50 total concurrent clients spread across 2 global regions (N Virginia, Oregon). Number chosen so it can be run on our free acount!
  2. Each client will execute the above scenario for 5 minutes.

Step 1: Create a Test Case

Make sure you sign up for a Testable account first. After logging in click the New Test Case button, give it a name, and specify the URL (tcp://sample.testable.io:8091 in our example).

Step 2: Define Test Scenario

The goal here is to convert the test scenario described above into executable code. This can be done by creating a recording or by writing a script. In our case let’s choose the script option.

Testable scripts are simply Javascript that executes in a sandboxed Node.JS environment.

Click Next and select Script as the scenario type. Before we write our script though we need to upload the S&P 500 symbols that we will use for testing.

Upload Symbols CSV

The S&P500 symbols can be found here. Go ahead and download it.

Click on the Data tab and then click Add Files. Select the symbols.csv file. It’ll give you a preview of the first 10 rows. Feel free to ignore the parse error since our CSV only has one column. Click Looks Good! and the CSV will be uploaded.

Write Script

Now that we have our symbols it is time to write the script! You can get started by using the template dropdown and selecting TCP -> TCP connection as a starting point, but in this case let’s use the following code instead:

var client = net.connect({ host: 'sample.testable.io', port: 8091 }, function() {
    dataTable.open('symbols.csv').next({ rows: 5 }, function(row) {
        client.write('subscribe|' + row.data['Symbols'] + 'n');
    });
});

client.on('data', function(data) {
    _.forEach(data.toString().split('n'), function(packet) {
        if (packet.length > 0) {
            var quote = JSON.parse(packet);
            results(quote.symbol).counter('quotesReceived', 1);
        } 
    });
});

setTimeout(function() {
    client.end();
}, 30000);

This code uses the Node.js net module to communicate with our quote service. Let’s go through the code block by block.

var client = net.connect({ host: 'sample.testable.io', port: 8091 }, function() {

This line opens a TCP socket connection to the sample service.

dataTable.open('symbols.csv').next({ rows: 5 }, function(row) {

Get the next 5 rows from the CSV file we uploaded earlier with the S&P 500 symbols. Don’t worry Testable ensures that each concurrent client gets different rows using a global test wide iterator. The callback function is called once per row received.

client.write('subscribe|' + row.data['Symbols'] + 'n');

Subscribe to the symbol found in the Symbols column of the CSV (i.e. the only column!). Once the server processes this request we expect to start receiving quotes for that symbol in the format:

{ 
    "symbol": "[symbol]", 
    "price":87.85, 
    "exchange": "NYSE" 
}

Let’s keep moving and review the quote processing code.

client.on('data', function(data) {
    _.forEach(data.toString().split('n'), function(packet) {
        if (packet.length > 0) {
            var quote = JSON.parse(packet);
            results(quote.symbol).counter('quotesReceived', 1);
            results('All Symbols').counter('quotesReceived', 1);
        } 
    });
});

There are a few things happening here:

  1. Subscribe to the data event on the TCP socket.
  2. On receiving a message (the price quotes), split it using the new line character in case multiple updates came together.
  3. For each quote, parse it into a JSON object.
  4. Increment a per symbol and an overall “quotes received” counter by 1. We will use these metrics later on when analyzing the test results. Note that you can capture custom timings and histograms as well.
setTimeout(function() {
    client.end();
}, 30000);

The final part simply waits 30 seconds and then closes the socket.

This code now defines the Scenario to execute at scale!

Test out your script by pressing the Run button in the upper right. This executes it one time on a shared Testable agent. Any captured metrics and logging will appear in the Run Console tab.

Example Run Console Output

02/02/2016 03:54:25.395 PM Result Received for tcp://sample.testable.io:8091
    dnsLookupMs [Timing]: 32
    connectionOpenMs [Timing]: 35
    success [Counter]: +1
    dataPacketsSent [Counter]: +5
    dataPacketsReceived [Counter]: +10
    firstReceivedMs [Timing]: 836
    connectionCloseMs [Timing]: 5007
    bytesSent [Counter]: +71
    bytesReceived [Counter]: +1546
    bandwidth [Counter]: +1617
02/02/2016 03:54:26.231 PM Result Received for ACE
    quotesReceived [Counter]: +5
02/02/2016 03:54:26.231 PM Result Received for All Symbols
    quotesReceived [Counter]: +25

Notice that Testable captures a bunch of metrics automatically in addition to the custom metric we added in our script.

Step 3: Configure a Load Test

Click Next to move onto the Configuration step. We now define exactly how to execute the scenario we defined in Step 2.

  • 25 concurrent clients in each region. Each concurrent client represents a user connecting to the quote service in parallel.
  • 1 second between iterations.
  • 5 minute duration.
  • Two regions (AWS N Virginia and AWS Oregon).

Click the Start Test button and your Quote Service load test is off and running!

Step 4: View the Results

By now you should see results flowing in as the test executes. The default dashboard will show a summary, results grid, and graphs of the system captured metrics.

Let’s chart our quotesReceived metric and add it the to the Summary and Results Grid.

Add a quotesReceived chart

In the upper right click the Dashboard.. button and select Add Chart.

Give it a name like Quotes Received. Select a Line chart and let’s chart quotesReceived -> count, and quotesReceived -> total.

Add the chart and it will appear on your dashboard.

Add quotesReceived to summary

Scroll down to the summary and press the configure icon. Press Add Column and select metric quotesReceived and press Update Summary. You can reorder the columns by dragging the new one up or down in the configuration window.

After updating the configuration the summary will look as follows:

Not only does it show in the summary but the next time you run the test it will show you the percentage change in this metric from the last test run.

Add quotesReceived to results grid

Same deal as adding it to the summary except now we can see the number of quotes received by symbol.

Now save this dashboard as a new one using Dashboard.. -> Save As. Read this for more information on customizing the dashboard.

And that’s it! We’ve load tested our quote service while capturing some custom metrics, run it at scale, and analyzed the results.

SHARE: