Advanced Metrics

Any load testing tool will report the page load time. Load Tester™ goes far beyond this simple view of web performance, recording 15 metrics for every page and 20 for every transaction in the test.

• Duration: min, max, average and standard deviation
• Time to first byte: min, max, average and standard deviation
• Completions, attempts, failures, failure rate and others
• Total and average size, average speed
• Authentication attempts and duration

Waiting Users

One of two metrics available exclusively in Load Tester™, our test engineers developed the Waiting Users metric to get a better view of what is happening to users during a load test. This metric, which is available at each level in the test results (entire test, individual testcase, web page and transaction), reports how many users are waiting for the server to respond to one or more requests. It was originally developed to help determine which pages or transactions were holding up the most users.

In real-world usage, we have found that this metric is also an excellent leading indicator of performance problems - it tends rise earlier than other performance indicators and point to an impending bottleneck. The chart above is from an actual test run for one of our services clients. The Waiting Users metric increases quickly and is soon followed by the Average Page Duration (beige). The Average Page Duration increases much later -- because the slower pages are not reported until they finally complete. The waiting users metric makes it obvious that the rush of new users during the ramp-up is not handled gracefully by the server. This is not obvious when looking only at the Average Page Duration.

For another example, read this post about solving a tough performance puzzle using these metrics.

Average Wait Time

Closely related to Waiting Users, the Average Wait Time metric captures how long, on average, each user (who is currently waiting on a response) has been waiting. This metric has also shown to be a leading indicator of capacity limits. This is illustrated on the above chart, where the Average Wait Time (green) increase before the overall average.

Because Average wait time does not include completed requests, it is excellent for identifying cases where most requests are being serviced quickly but some are much slower. When the number of waiting users remains constant but average wait time is increasing, then the server is falling behind and leaving some users waiting - possibly indefinitely. When the waiting users is increasing but the average wait time hold steady, then everything may be ok - as long as the waiting users is increasing out of proportion with increases in the total number of users.

Note that both of these metrics, at the level of individual transactions, would be impossible to capture using real browsers, because the browser does not expose this level of detail to testing tools. Load Tester can drill down into a page to find out which transaction is causing the delay. This feature is increasing in importance as modern applications employ AJAX technologies to make multiple requests to the server to populate a single page with data.