Performance Testing Basics

Squeeze the app before release. If the app withstands that, it is fit for release. But how to squeeze? How will we determine the number of users, data volume etc.? Let us take this step by step and learn. Performance tests are usually postponed until customers feel the pinch. The primary reason is the cost of the tools and the capability to use the tools. If one wants to earn millions of dollars thru a hosted app, a good, proven and simple way is to increase users and reduce price. If one does this, the business volume will grow - but it brings the performance issues as well along with that.

Most of the tools use the same concept of emulating the requests from the client side of the application. This has to be done programmatically. When one is able to generate requests, processing response is a relatively easier task. When you choose the tools, it is better you look for the must-be-in features and then for nice-if features.

The must-be-in features are listed below.

- Select protocols (HTTP, FTP, SMTP etc.)
- Record the user sequence and generate script
- Parameterize the script to supply a variety of data
- Process dynamic data sent by server side (correlation)
- Configure user count, iterations and pacing between iterations
- Configure user ramp-up
- Process secondary requests
- Configure network speed, browser types
- Check for specific patterns in the response
- Execute multiple scripts in parallel
- Measure hits, throughput, response time for every page
- Log important details and server response data for troubleshooting
- Provide custom coding facility to add additional logic

The nice-if features are listed below.

- Configure performance counters for OS, webserver, app server, database server. This way, you can get all results under one single tool
- Automatically correlate standard dynamic texts based on java or .net framework. This will reduce scripting time
- Provide a visual UI to script and build logic
- Generate data as needed - sequential, random and unique data
- Provide a flexible licensing model - permanent as well as pay-per-use will be great
- Integrate withe profiling tools to pinpoint issues at code level

When one evaluates a performance testing tool, one must do a simple proof of concept on the above features, to see how effectively the tool handles these features. No need to say, that the tool must be simple to use.

Here are a few simple terms you need to be clear - at least academically. There are so many different definitions for the phrases given below, but we try to take the mostly accepted definitions from various project groups.

Load Testing - Test the app for an expected number of users. Usually customers know their current user base (example - total number of account holders in a bank). The number of online users will be usually between 5% and 10% of customer base. But an online user may be just a logged in user, doing no transaction with server. Our interest is always on the concurrent users. Concurrent users are usually between 5% and 10% of online users. So if 100,000 is the total customer base, then 10% of it, 10,000 will be online users and 10% of that, 1000 will be concurrent users.

Stress Testing - Overload the system by x%. That x% may be 10% more than normal load or even 300% more than the normal load. But usually load tests happen for a longer duration and stress tests happen for a shorter duration as spikes, with abnormally more users. Stress is like a flash flood.

Scalability/Capacity Testing - See the level at which the system crashes. Keep increasing users and you will see a lot of failures and eventually crash. Some companies use the term stress testing itself to include the capacity testing as well.

Volume Testing - keep increasing the data size for requests as well as process requests when the application database has 100s of millions of records. This usually checks the robustness and speed of the data retrieval and processing.

Endurance/Availability Tests - test the system for a very long period of time. Let the users keep sending requests 24 by 7 may be even for a week or month. See if system consistently behaves over a period of time.