Execute a JMeter test plan with additional custom metrics

We are excited to announce support for executing JMeter test plans the Testable way! Our approach provides the most flexibility in the industry while still harnessing the full power of JMeter.

Background

At Testable we strongly believe the easiest way to define a performance test scenario is to either:

  1. Write a Javascript script that will execute in a sandboxed Node.js environment. Testable provides templates to help get you started.
  2. Record a set of interactions with your service and either use it directly or convert that recording into a script that can be tweaked.

If you are starting from scratch we would recommend one of the above approaches depending on the use case. However we also realize that there are technology teams that have invested heavily in, or would prefer to define their tests, using JMeter, a popular open source load testing framework. We would love to have these users leverage the Testable distribution, scale, metric gathering, reporting, analytics, and sharing capabilities to execute their JMeter test plans.

Getting Started

In order to support JMeter testing we added a new scenario type.

For this example we will use a simple JMeter test plan that calls http://sample.testable.io/stocks/IBM. The test plan uses a thread group of 5 threads that calls the sample service URL 10 times each. On the Testable side, let’s also add a couple of custom counters that keep track of “slow” (> 500ms) and “fast” (< 200ms) requests that we can graph later.

Step 1: Create a Test Case

Make sure you sign up for a Testable account first. After logging in click the New Test Case button, give it a name, and specify the URL (http://sample.testable.io/stocks/IBM in our example).

Click Next and select Run JMeter Test Plan as the Scenario Type.

Step 2: Upload Test Plan

Click on the Add Files button and upload the test plan to your test case.


You can try out your test plan by pressing the Run button in the upper right corner. This runs your JMeter test plan on a Testable test runner with 1 thread for 1 iteration.

Step 4: Configure the Load Test

Now that we have defined our test scenario, let’s configure how we want to execute it at scale on the Testable platform.

Each Iteration of a Concurrent User of the test will execute your JMeter test plan, using whatever Thread Group Parameters are defined in the plan, in parallel. Note that this behavior can be overridden but is beyond the scope of this post (see our documentation for more details).

In our example test plan the Thread Group Parameters specify 5 threads, using a loop controller, with 10 loop iterations. So if had 10 Testable Concurrent Users then we would have 50 total threads (10 test plan executions * 5 threads each). If we wanted 100 total iterations we would specify 10 Testable Iterations since the test plan has 10 iterations built in. It’s up to the user to decide whether you want Testable or JMeter handling the concurrency and iterations. Since having many JMeter instances running carries overhead, it is usually preferable to let JMeter handle the concurrency and run fewer Testable Concurrent Clients.

In our case let’s target 10 threads and 50 iterations in 3 regions overall. This means the following Testable parameters:

  • Concurrent Clients Per Region = 2
  • Type = Iteration
  • Iterations = 5
  • Regions = AWS N Virginia, AWS Oregon, and AWS Singapore

Click the Start Test button and your test is off and running! Congratulations you have officially created and run your first JMeter test. Now let’s look at analyzing the results.

Step 5: View the Results

By now you should see results flowing in as the test executes. The default dashboard will show a summary, results grid, and graphs of the system captured metrics.

And that’s it! We’ve setup a JMeter test case, run it at scale, and analyzed the results.

Check out our JMeter documentation for the full set of options. And as always feel free to contact us with any thoughts or suggestions.