Survey of Performance Testing Tools


Performance testing is done to find out one or more risks related to cost, expense and opportunity affecting business reputation and continuity. Main goal of performance testing is to test for the availability, scalability, speed and performance of the software as well as hardware

Shweta Tyagi

Fig. 1: Load testing process
Fig. 1: Load testing process

Performance testing is a non-functional type of testing implemented and executed to evaluate and characterise the performance-related aspects of an application such as reliability, speed and scalability. Various types of performance tests, each focussing on different objectives, are implemented throughout the software development life cycle comprising different iterations.

In the architecture iteration, performance testing focusses on identifying and eliminating architectural-related issues and bottlenecks. In the construction phase, performance tests are performed to verify that the application can handle peak loads and stress conditions such as a large number of users, transactions or volumes.

Building a successful application requires two main elements—functionality and performance.

Explore Circuits and Projects Explore Videos and Tutorials

Functionality indicates the ability of software products such as web application and computer software to satisfy the need of an end user.

Performance is the ability of a system to accomplish useful work measured against some known standards of cost, completeness, speed and accuracy.

The main goal of performance testing is to find and remove all bottlenecks before the product is launched into the market. Poor or inadequate performance testing may lead to applications that are prone to breakdowns.

Fig. 2: Soak testing

Performance testing is necessary for critical uses such as life-saving applications in healthcare, medical, drug and biotech discovery systems and mission-critical applications such as missile programs, aerospace, flight, automotive and defence to ensure that they work for longer periods of time without performance glitches. In this paper, performance testing methods, techniques and tools are summarised and analysed.

Performance testing concept
Resources that are monitored and reported as a part of performance testing include CPU usage, cache coherence, data consistency, memory usage, power consumption and network bandwidth. Response time and throughput related to router, application servers and web servers are also considered in performance testing. Put together, an application’s reliability, availability, response time, throughput, security, extensibility and scalability is observed and analysed during performance testing.

Fig. 3: Working of LoadRunner
Fig. 3: Working of LoadRunner

Why performance testing
In today’s world of e-business, almost all businesses are focussing on performance testing as it directly influences business revenue, brand name, organisation’s reputation and, most importantly, customer satisfaction. Ignoring performance testing can cause severe losses.

E64_Table_1In 1999, Hershey’s Food Company launched a new supply chain management system (SCMS). Warehouse implementation and other issues caused Hershey to miss important Halloween shipments. This caused the company to lose $150 million in revenue, a drop of 19 per cent in annual profit and 34 per cent depreciation in the value of their stock.

Businesses that fully depend on web application for their sales can be greatly affected if application response time is poor. A minor mechanical problem in a switch at one of AT&T’s 114 switching centres resulted in 75 million phone calls across the USA going unanswered.

3ZC_Table_2Studies show that user-visiting rate of an application starts to suffer when its response time takes over four seconds. Unsatisfied customer may never want to come back and influence others not to visit the application. This can make huge impact on business.

With these examples, one can see the importance of performance testing. Ignoring performance testing can lead to some major problems such as physical costs, as explained above, and some invisible and non-physical costs such as embarrassment, damage to reputation, tension, lower morale of workers and frustration in the organisation. Poor performance may cause loss in customer’s faith in the company.

Automated Test Equipment

Types of performance testing
Performance testing can be categorised into following types:

Load testing. Load testing is the simplest form of performance testing. It is used to define system’s behaviour under specific load conditions. The load can be multiple users accessing application simultaneously within a certain time frame. Load testing helps find the maximum load capacity of an application and also identifies the bottlenecks in the application that are the root causes of degradation.

Load testing is most suited for multiuser systems built using client/server architecture such as application and web server.

Load testing of an application uses the following approach: When a user visits a site, a script recorder records the conversation and then creates interaction scripts. A load generator replays the recorded scripts with modified parameters. During replay, both hardware and software aspects such as CPU, memory, disk I/O of physical server, response time and throughput of server will be monitored and analysed. Then, a load testing report will be generated.

Stress testing. Stress testing is a form of performance testing that is used to determine system’s availability, robustness and reliability under peak load conditions. The goal of stress testing is to test the application under extreme load conditions to find any loopholes. These conditions can include high concurrency, limited computer resources and heavy loads.

Fig. 4: Working of NeoLoad
Fig. 4: Working of NeoLoad

The idea behind stress testing is to test the system beyond operational capacity in order to find defects which can lead to system breakdown and possibly end in a disastrous failure.

Proper stress testing is fruitful in detecting interlock problems, synchronisation and timing bugs, priority problems and resource bugs.

There are two types of stress testing: Hardware stress testing and software stress testing.

Hardware stress testing is used to ensure the stability of computer hardware in a normal environment under excessive levels of stress. Stress levels can include types of task, extremes of workload, memory uses, thermal load voltage or clock speed.

In software testing, emphasis is on availability, robustness and stability of the system under heavy load conditions. The goal of stress testing is to ensure that the software does not crash in conditions that lack computational resources, usually denial of service attacks or high concurrency.

Soak testing. Soak testing is a process of running an application at an extreme load condition for a longer period of time. A soak test would normally be executed over a weekend or an entire day to identify any performance problems that occur due to a large number of transactions.

In some cases, system may stop working due to certain defects such as memory leaks and CPU failures. Load test and stress test cannot find such defects because they are limited to short durations. In such cases, soak tests are useful to find such defects.

A soak test can run as long as possible with some limitation on testing. Weekends are a good time for such tests.

Fig. 2 shows the login activity of some sites. The soak test shown here is at a level of 540 logins per hour, with some typical usage for each login session. In this example, the average number of logins per day is 5308 and it can take eight hours at 540 logins per hour to complete an entire day’s activity through the application.

Soak tests are useful in finding any degradation in performance under extreme situations. Soak tests help in finding major problems such as memory leaks and degradation in response time.

A soak test is suitable for applications which are run for longer periods of time, for example, ERP and CRM systems, where user executes a number of business transactions and is logged in for many hours. Soak testing is useful in systems such as air-traffic control system. Soak tests for such systems may have multi-day or even multi-month durations.

Big Data in Test and Measurement

Spike testing. Spike test is a non-functional test in which a web application is tested against a varying load over a longer period of time. In this, a load is applied to an application continuously varying between 50 and 100 per cent over load adjusters. Performance and response of an application is constantly monitored and analysed during this process. There are possibilities that a system might break under these varying loads.

Performance of a web application can be improved by using bandwidth throttling. Bandwidth throttling is intentionally slowing the Internet server in order to regulate Internet traffic and minimise bandwidth congestion.

Spike testing is a type of performance testing in which we evaluate the performance of a web application in unexpected traffic. Performance of a web page degrades when there is a sudden increase in number of users accessing it. The requirement of a web application varies for a number of reasons, some are event-driven like launch of a product, and some are time-driven like time of day. Spike testing tests all these issues and guarantees that all the performance-related issues are detected and fixed before the fulfilment of requirements.

The goal of a spike test is to check that when the performance degrades, whether the system will fail or it will handle the load.

Performance testing tools
Performance testing tools include automatic as well as semi-automatic ways of measuring web application performance and load stress capacity. It can monitor the performance of a web application in peak traffic and extreme load conditions. A good performance testing tool must satisfy some criteria such as capability for testing using case-based load scenario testing. It should be easily configurable, capable for stress testing and provide monitoring of parameters.

There are many open source as well as licenced performance testing tools available in the market. A few examples are.

1. HP LoadRunner. HP LoadRunner is an automated performance testing tool and a product from HP for testing the behaviour and performance of systems under actual loads. This tool improves the performance of an application by detecting bottlenecks before a product is launched or deployed. It enables testing of rich Internet applications, Web 2.0 technologies, ERP and CRM applications and legacy applications.

How it works. The HP LoadRunner is a combination of different tools; namely virtual user, generator, controller, load generator and analyser.

The virtual user generator allows, creating scripts that generate a series of actions, such as navigating through the software, amongst others.

The controller takes the script and runs it through a schedule that is set up for testing.

Analyser tells all the results of the load test in various forms. It provides summaries of data, as well as the details of the load test for bottlenecks.

LoadRunner can simulate hundreds or thousands of concurrent users. LoadRunner supports lots of application protocols: Remote Desktop protocol (RDP), Citrix ICA, Flex AF, ERP/CRM, Mail Clients, Database and AJAX Truclient.

Some of the advantages of using LoadRunner are that it is easy to use with nice GUI, does not need installation on servers under testing, uses native monitors, has excellent monitoring and analysis interface and excellent and active support from HP. The only disadvantage associated with this tool is its cost.

2. IBM rational performance tester. Rational performance tester is an automated performance testing tool for web- and server-based applications developed by the Rational Software division of IBM. It is a creation, execution and analysis tool that helps developers to validate the reliability and scalability of a web application before its deployment.

Archimedes: A Free Monte Carlo Simulator for Semiconductor Device Simulations

How it works. Creation of performance test in IBM Rational performance tester is a three-step process: build scripts, schedule workload, execute and analyse.

Build scripts: Script represents user’s activities in the application, scripts are real-world usage scenarios built by the tester based on a design plan—without any programming, and it is built by pointing and clicking through the target application’s user interface (UI).

Schedule workload: Schedule defines the sequence and ordering of script execution. It dynamically adds load during the run to find breakpoints.

Execute and analyse: Results after load testing give a clear picture of whether the system can or cannot handle the load. The data provided helps to locate the root cause of a problem.

Protocols supported by Rational Performance tester are Citrix, Socket Recording, SOA, SAP, and Web HTTP.

(i) Supports XML and is significantly cheaper than LoadRunner
(ii) No programming knowledge is required
(iii) Diagnoses Weblogic and Websphere application servers
(iv) Gives you the option of using a persistent cursor for data tables

(i) Does not support Java applet-based application
(ii) Scheduling is not possible
(iii) Connection handling is not visible to the user

DA1_Table33. NeoLoad. NeoLoad is a load and performance testing tool used to measure and analyse the performance of web and mobile applications.

NeoLoad helps in analysing the performance of a web application by increasing traffic to the application and monitoring performance under heavy load conditions. It is designed and developed by a French company named Neotys. It is written in Java and is available in English and French languages.

1FE_Table4How it works. NeoLoad has two main components: the controller and the load generator

The controller. Provides a graphical interface from which the user can create and record scenarios, run tests and analyse the results. It monitors the web infrastructure’s various components without installing any agents on the server

ZD3_Table5The load generator. The load generator simulates users accessing the application being tested. It also collects information about the application’s performance

NeoLoad is compatible with all web servers and web application servers like J2EE, .NET, PHP, ASP, CGI, AJAX and SOAP. The main features of this tool are that it supports Digest, Basic, NTLM and form-based authentication. It also supports client certificate authentication. It supports hybrid and native mobile application, and also JSON and SPDY protocol. NeoLoad does not cover all bases on protocols such as Citrix ICA or RDP.

4. SOASTA’s CloudTest. Cloud testing is a type of software testing in which web applications use cloud infrastructures to simulate real-world traffic. SOASTA’s CloudTest is one of the industry’s best selling software to perform cloud performance testing. It has the capacity to simulate thousands of users to use the application at the same time. It increases the traffic on the website to monitor the actual performance under peak load condition.

CloudTest is designed and developed by the American company SOASTA, Inc. This is not s free software. The trial version is available.

Some of the benefits of using this tool are that it lives in the cloud, so there is no need for the associated headaches of installation, managing injectors, controllers and analysis tools. Software also has advanced deploy and diagnostic capabilities—if a service fails on a cloud host, it detects and redeploys automatically. SOASTA provides run-time load analysis. This allows detection and investigation of issues as they happen.

Some other tools are listed in Table V.

The author is currently working as a project engineer at CDAC, Mumbai. Her research interests are software testing and reliability, and software metrics