Load testing a Cassandra DB

This post continues our series on how to test various types of technology with Testable. Today we will be load testing a Cassandra DB, capturing some useful custom metrics, and analyzing the results.

The test will include creating and dropping a DB schema and then timing insert latency under load.

Step 1: Create a Test Case

Make sure you sign up for a Testable account first. After logging in click the New Test Case button, give it a name, and specify the URL to your DB (i.e. db.mydomain.com:3306).

Step 2: Write Test Script

Testable scripts are simply Javascript that executes in a sandboxed Node.JS environment. Once you finish Step 1, click Next and select Write Script as the scenario type. We will use the cassandra-driver NPM module for communicating with our DB.

Init Code

The Init code will only execute once globally at the start of our test. Let’s use the following code for Init:

var cassandra = require('cassandra-driver');
var async = require('async');
var client = new cassandra.Client( { contactPoints : [ 'myserver.com' ] } );
async.series([
  function connect(next) {
    client.connect(next);
  },
  function createKeyspace(next) {
    var query = "CREATE KEYSPACE IF NOT EXISTS loadtest WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3' }";
    client.execute(query, next);
  },
  function createTable(next) {
    var query = "CREATE TABLE IF NOT EXISTS loadtest.person (id uuid, name text, address text, PRIMARY KEY(id))";
    client.execute(query, next);
  }
], function (err) {
  if (err) {
    console.error('There was an error', err.message, err.stack);
  }
  console.log('Init completed');
  client.shutdown();
});

This code connects to the DB (update the host, user, password for your use case), creates the loadtest keyspace if it doesn’t exist, and within that keyspace the person table if it doesn’t exist.

Teardown Code

At the end of the test we want to drop the database we just tested. Use the following code for Teardown:

var cassandra = require('cassandra-driver');
var async = require('async');

var client = new cassandra.Client( { contactPoints : [ 'dev.testable.io' ] } );
async.series([
  function connect(next) {
    client.connect(next);
  },
  function dropKeyspace(next) {
    var query = "DROP KEYSPACE loadtest";
    client.execute(query, next);
  }
], function (err) {
  if (err) {
    console.error('There was an error', err.message, err.stack);
  }
  console.log('Teardown completed');
  client.shutdown();
});

Similar to the init code, except this time we drop the keyspace to cleanup our test.

Main Code

Now that we have the init/teardown setup, let’s define the actual test scenario:

var cassandra = require('cassandra-driver');
var async = require('async');

var client = new cassandra.Client( { contactPoints : [ 'dev.testable.io' ] } );
async.series([
  function connect(next) {
    client.connect(next);
  },
  function insertRow(next) {
    var query = "INSERT INTO loadtest.person (id, name, address) VALUES (?, ?, ?)";
    timing(function(done) {
      client.execute(query, [cassandra.types.Uuid.random(), info.currentId(), 'Dummy address'], { prepare: true }, function() {
        done();
        results().counter('inserts', 1);
        next.apply(this, arguments);
      });
    }, 'insertLatency');
  }
], function (err) {
  if (err) {
    console.error('There was an error', err.message, err.stack);
  }
  client.shutdown();
});

This code does the following:

  1. Connect to our DB
  2. Time our INSERT query. The INSERT uses info.currentId() (i.e. a unique identifier for the test iteration) as the name and a dummy value for the address.
  3. Capture the timer as the insertLatency custom metric.
  4. Count the number of inserts executed as the inserts counter.

This code now defines the Scenario to execute at scale.

Test out your script by pressing the Run button in the upper right. This executes it one time on a shared Testable agent, including init/teardown. Any captured metrics and logging will appear in the Run Console tab.

Notice that Testable captures a bunch of low level metrics automatically in addition to the custom metric we added in our script.

Step 3: Configure a Load Test

Click Next to move onto the Configuration step. We now define exactly how to execute the scenario we defined in Step 2.

  • 10 concurrent clients in each region. Each concurrent client is like a user connecting to your DB and runs on a Testable agent in parallel.
  • 1 minute duration.
  • Three regions (AWS N Virginia, AWS Oregon, AWS Singapore). Note that if your DB is behind a firewall you can also run the test on your own hardware.

Click the Start Test button and your test is off and running! Congratulations you have officially created and run a load test. Now let’s look at analyzing the results.

Step 4: View the Results

By now you should see results flowing in as the test executes. The default dashboard will show a summary, results grid, and graphs of the system captured metrics.

Let’s chart our insertLatency metric and add it the to the Summary and Results Grid.

Add a insertLatency chart

In the upper right click the Dashboard.. button and select Add Chart.

Give it a name like Insert Latency. Select a Line chart and let’s chart insertLatency -> mean, insertLatency -> p95 (95th percentile), and insertLatency -> p99 (99th percentile).

Add the chart and it will appear on your dashboard.

Add insertLatency to summary

Scroll down to the summary and press the configure icon. Press Add Column and select metric insertLatency-mean and press Update Summary. You can reorder the columns by dragging the new one up or down in the configuration window.

After updating the configuration the summary will look as follows:

You can save this new dashboard for future use with the Dashboard menu in the upper right. And that’s it! We’ve setup a Cassandra test scenario, captured custom metrics, run it at scale, and analyzed the results.

Load Testing a TCP Based Streaming Quote Service

In the financial world, there are plenty of custom protocols and performance sensitive APIs that work over raw TCP and UDP sockets. Most of the load testing services out there do not offer any solutions in this space and at Testable we believe this is one area that sets us apart.

In this post we will show you how to use Testable to load test a streaming stock quote service that uses a simple custom protocol over TCP. We will utilize the S&P 500 symbols as part of the test, divided equally amongst the iterations of the test.

Streaming Quote Service API

The Quote Service API is available at sample.testable.io:8091.

As you can see from above the basic flow is as follows:

  1. Open TCP socket connection to the quote service
  2. Send a subscribe packet in the format “subscribe|[symbol]n”
  3. Receive quote packets once a second per subscribed symbol in the following JSON format: { "symbol": "[symbol]", "price":87.85, "exchange": "NYSE" }n.
  4. To unsubscribe, send “unsubscribe|[symbol]n”
  5. Close the TCP socket

Test Overview

Our test scenario will be as follows:

  1. Connect to the Quote Service
  2. Subscribe to 5 symbols evenly distributed from the pool of 500 potential symbols
  3. Receive updates for 30 seconds
  4. Disconnect from the Quote Service

We will then execute this scenario at scale under the following configuration:

  1. 50 total concurrent clients spread across 2 global regions (N Virginia, Oregon). Number chosen so it can be run on our free acount!
  2. Each client will execute the above scenario for 5 minutes.

Step 1: Create a Test Case

Make sure you sign up for a Testable account first. After logging in click the New Test Case button, give it a name, and specify the URL (tcp://sample.testable.io:8091 in our example).

Step 2: Define Test Scenario

The goal here is to convert the test scenario described above into executable code. This can be done by creating a recording or by writing a script. In our case let’s choose the script option.

Testable scripts are simply Javascript that executes in a sandboxed Node.JS environment.

Click Next and select Script as the scenario type. Before we write our script though we need to upload the S&P 500 symbols that we will use for testing.

Upload Symbols CSV

The S&P500 symbols can be found here. Go ahead and download it.

Click on the Data tab and then click Add Files. Select the symbols.csv file. It’ll give you a preview of the first 10 rows. Feel free to ignore the parse error since our CSV only has one column. Click Looks Good! and the CSV will be uploaded.

Write Script

Now that we have our symbols it is time to write the script! You can get started by using the template dropdown and selecting TCP -> TCP connection as a starting point, but in this case let’s use the following code instead:

var client = net.connect({ host: 'sample.testable.io', port: 8091 }, function() {
    dataTable.open('symbols.csv').next({ rows: 5 }, function(row) {
        client.write('subscribe|' + row.data['Symbols'] + 'n');
    });
});

client.on('data', function(data) {
    _.forEach(data.toString().split('n'), function(packet) {
        if (packet.length > 0) {
            var quote = JSON.parse(packet);
            results(quote.symbol).counter('quotesReceived', 1);
        } 
    });
});

setTimeout(function() {
    client.end();
}, 30000);

This code uses the Node.js net module to communicate with our quote service. Let’s go through the code block by block.

var client = net.connect({ host: 'sample.testable.io', port: 8091 }, function() {

This line opens a TCP socket connection to the sample service.

dataTable.open('symbols.csv').next({ rows: 5 }, function(row) {

Get the next 5 rows from the CSV file we uploaded earlier with the S&P 500 symbols. Don’t worry Testable ensures that each concurrent client gets different rows using a global test wide iterator. The callback function is called once per row received.

client.write('subscribe|' + row.data['Symbols'] + 'n');

Subscribe to the symbol found in the Symbols column of the CSV (i.e. the only column!). Once the server processes this request we expect to start receiving quotes for that symbol in the format:

{ 
    "symbol": "[symbol]", 
    "price":87.85, 
    "exchange": "NYSE" 
}

Let’s keep moving and review the quote processing code.

client.on('data', function(data) {
    _.forEach(data.toString().split('n'), function(packet) {
        if (packet.length > 0) {
            var quote = JSON.parse(packet);
            results(quote.symbol).counter('quotesReceived', 1);
            results('All Symbols').counter('quotesReceived', 1);
        } 
    });
});

There are a few things happening here:

  1. Subscribe to the data event on the TCP socket.
  2. On receiving a message (the price quotes), split it using the new line character in case multiple updates came together.
  3. For each quote, parse it into a JSON object.
  4. Increment a per symbol and an overall “quotes received” counter by 1. We will use these metrics later on when analyzing the test results. Note that you can capture custom timings and histograms as well.
setTimeout(function() {
    client.end();
}, 30000);

The final part simply waits 30 seconds and then closes the socket.

This code now defines the Scenario to execute at scale!

Test out your script by pressing the Run button in the upper right. This executes it one time on a shared Testable agent. Any captured metrics and logging will appear in the Run Console tab.

Example Run Console Output

02/02/2016 03:54:25.395 PM Result Received for tcp://sample.testable.io:8091
    dnsLookupMs [Timing]: 32
    connectionOpenMs [Timing]: 35
    success [Counter]: +1
    dataPacketsSent [Counter]: +5
    dataPacketsReceived [Counter]: +10
    firstReceivedMs [Timing]: 836
    connectionCloseMs [Timing]: 5007
    bytesSent [Counter]: +71
    bytesReceived [Counter]: +1546
    bandwidth [Counter]: +1617
02/02/2016 03:54:26.231 PM Result Received for ACE
    quotesReceived [Counter]: +5
02/02/2016 03:54:26.231 PM Result Received for All Symbols
    quotesReceived [Counter]: +25

Notice that Testable captures a bunch of metrics automatically in addition to the custom metric we added in our script.

Step 3: Configure a Load Test

Click Next to move onto the Configuration step. We now define exactly how to execute the scenario we defined in Step 2.

  • 25 concurrent clients in each region. Each concurrent client represents a user connecting to the quote service in parallel.
  • 1 second between iterations.
  • 5 minute duration.
  • Two regions (AWS N Virginia and AWS Oregon).

Click the Start Test button and your Quote Service load test is off and running!

Step 4: View the Results

By now you should see results flowing in as the test executes. The default dashboard will show a summary, results grid, and graphs of the system captured metrics.

Let’s chart our quotesReceived metric and add it the to the Summary and Results Grid.

Add a quotesReceived chart

In the upper right click the Dashboard.. button and select Add Chart.

Give it a name like Quotes Received. Select a Line chart and let’s chart quotesReceived -> count, and quotesReceived -> total.

Add the chart and it will appear on your dashboard.

Add quotesReceived to summary

Scroll down to the summary and press the configure icon. Press Add Column and select metric quotesReceived and press Update Summary. You can reorder the columns by dragging the new one up or down in the configuration window.

After updating the configuration the summary will look as follows:

Not only does it show in the summary but the next time you run the test it will show you the percentage change in this metric from the last test run.

Add quotesReceived to results grid

Same deal as adding it to the summary except now we can see the number of quotes received by symbol.

Now save this dashboard as a new one using Dashboard.. -> Save As. Read this for more information on customizing the dashboard.

And that’s it! We’ve load tested our quote service while capturing some custom metrics, run it at scale, and analyzed the results.

Using Unique Credentials For Each Test Iteration

Often times when running a load test we need to authenticate as a user before performing actions. This could anything from a website that uses OAuth to a custom streaming service with its own protocol for authentication. Testable scripts support CSV data to solve this problem.

For this blog post we use the sample service /login endpoint to simulate testing with different users. The only valid credentials that this endpoint accepts is admin/admin.

Step 1: Create a Test Case

Make sure you sign up for a Testable account first. After logging in click the New Test Case button, give it a name, and specify the URL (http://sample.testable.io/login in our example).

Step 2: Upload CSV Data

Click Next and select Script as the scenario type. On the Data tab upload a CSV file with the following 3 rows of data:

USER,PASSWORD
admin,admin
invalid1,admin123
invalid2,invalid123

In our script we will use one row from this file each time it executes. Testable gives us an iterator which cycles through the rows in the file across all global test iterations. The global iterator can even fail once all rows have been iterated over if you do not want to repeat credentials, but in our case we are happy to loop through the rows multiple times.

The code for our script will look as follows:

var dt = dataTable.open('credentials.csv');
dt.next(function(row) {
    var credentials = { 
        user: row.data['USER'], 
        password: row.data['PASSWORD'] 
    };
    http.request({ 
        hostname: 'sample.testable.io', 
        path: '/login', 
        method: 'POST'
    }).end(JSON.stringify(credentials));
    results(credentials.user).counter('count', 1, 'requests');
});

This code does the following:

  1. Gets the next row in the CSV file. Remember that the iterator is global across all concurrent clients.
  2. Build a credentials object using the row assigned to the scenario.
  3. Call POST http://sample.testable.io/login and pass the credentials in the body.
  4. Increment a custom counter that keeps track of the number of requests per username.

You can try out your script by pressing the Run button in the upper right.

Step 3: Configure and Execute Test

After pressing Next we configure the load test using the following parameters that are well within the free account limit:

  • Concurrent Clients Per Region: 10
  • Iterations: 20
  • Regions: AWS N. Virginia, AWS Oregon, AWS Singapore
  • Ramp up: 60 seconds
  • Iteration Sleep: 1 second

Press Start Test and watch the results start to flow in.

That’s it! Read the upload data documentation for the full set of options.