Performance Test Automation

This chapter delves on the automation aspects connected with performance issues. Adopting a manual method of testing the performance is neither desirable nor feasible. Automating the testing mode is an inevitable necessity. The strategy lies in choosing not only the best of the available tools but also to blend their productivity with human analytical skills. In this regard, it is important to bear in mind that use of tools as such does not amount to automating the entire testing process. This chapter addresses the task of setting up a process for automating the tests and highlights various issues involved in it. It also discusses some of the strategies for success in the automation of performance testing and concludes with a survey on tools currently available for testing in automation mode.

Test Preparation Phase I

The core theme of the previous chapters is that a methodical and structured approach to PT is rather necessary right from the early phases of the development cycle. This will ensure predictability and controllability of application performance in practice. The preparatory activities associated with PT (Figure 4.1) are of great importance and are distributed over the life cycle phases of requirement elicitation and analysis, High Level Design (HLD), Detail Level Design (DLD), and several sets of system builds. These activities help define and provide continuity between the high level requirements for application performance, strategies for testing, a framework for designing the tests (see Barber, 2004), and artifacts used to plan and carry out tests. This chapter contains a detailed consideration of the definition phase while Chapters 5 and 6 highlight issues related to the design and build phases associated with the preparatory activities as shown in Figure 4.1.

Performance Testing

It is an accepted fact that no system is perfect from the viewpoint of performance. Problems pertaining to performance affect all types of systems, regardless as to whether they are client/server or Web application systems. It is imperative to understand the factors affecting performance of the system before embarking on the task of tackling them. The constraints affecting the performance may be many but can be broadly classified into (i) technical and (ii) project related factors. The former is very much pronounced in the case of Web-based systems, as the relevant technology has not attained the stage of maturity to address the issues on its own. On the other hand, the project managers struggle to develop an optimal system within the existing technology framework. This in turn, generates many project related factors sub optimizing the performance. Likewise, many factors affecting the performance of the system are discussed in this chapter. The subsequent chapters deal at length with the methods and strategies to tackle these factors.

Web-Based Systems and Performance Testing

For many years, the World Wide Web (Web) functioned quite well without any concern about the quality of performance. The designers of the Web page, as well as the users were not much worried about the performance attributes. The Web, in the initial stages of development, was primarily meant to be an information provider rather than a medium to transact business, into which it has grown. The expectations from the users were also limited only to seek the information available on the Web. Thanks to the ever growing population of Web surfers (now in the millions), information found on the Web underwent a dimensional change in terms of nature, content, and depth.

Performance Test Execution Phase

The focus of this chapter is toward the factors needing attention while conducting the tests of performance. Chapters 5 and 6 discuss in detail the test plan as well as the test scripts required for conducting the performance tests. The successful execution of PT calls for coordination of a complex set of activities such as management of personnel, scheduling the tests at an appropriate time, configuring tools for various performance parameters, verification of application readiness for the test, and, above all, management of multiple test runs versus cost of resources. Conducting the tests may spread over a number of days, and it may also warrant customization of the operating system. The team responsible for carrying out the tests must be well aware of these factors. Typical test execution (see Test execution, 2004) phases are shown in Figure 7.1. Each phase must be planned carefully. If these phases are followed in sequence, the probability of test execution errors could be reduced. The main reason behind introducing these phases is to reduce the cost. To illustrate, if we directly run tests without passing through initial phases like elaboration test (say) and an error creeps in later, it will result in unnecessary investment of effort on test runs, which is very expensive. Specialized skills are required to run tests. Sometimes, the test environment (including tools) has to be taken on hire during the test runs. All these involve effort and time, which will go to waste if the tests are not executed properly. Each phase is explained in Figure 7.1.

Test Preparation Phase II

The focus of Chapter 4 was on the planning and definition phase of PT while this chapter provides an in-depth view of the test design phase. All the necessary inputs for PT are captured under the test definition phase. These inputs are the building blocks for the design phase of PT.

Introduction to Performance Monitoring and Tuning

For any applications to be performance conscious, its performance must be monitored continuously. Monitoring performance is a necessary part of the preventive maintenance of the application. By monitoring, we obtain performance data which are useful in diagnosing performance problems under operational conditions. Based on data collected through monitoring, one can define a baseline — a range of measurements that represent acceptable performance under typical operating conditions. This baseline provides a reference point that makes it easier to spot problems when they occur. In addition, during troubleshooting system problems, performance data give information about the behavior of system resources at the time the problem occurs, which is useful in pinpointing the cause. In order to monitor the system, the operational environment provides various parameters implemented through counters for collection of performance data. Applications developed must ultimately be installed and run on a specific operating system. Hence, applications performance also depends on factors that govern the operating system. Each operating system has its own set of performance parameters to monitor and tune for better performance. Performance of applications also depends on the architectural level monitoring and tuning. However, architectural design depends on specific technology. Hence, technology level monitoring and tuning must be addressed for better results. To achieve all these, proper guidelines must be enforced at various stages for monitoring and tuning. All the previous chapters, together, described the performance testing from concept to reality whereas this chapter highlights aspects of monitoring and tuning to specific technologies. This chapter provides an overview of monitoring and tuning applications with frameworks in Java and Microsoft .NET technologies. Before addressing the technology specific performance issues, we need to know the overall bottlenecks that arise in Web applications.

Performance Testing

Performance testing (PT) is not to be construed as features testing even though it has a definite link with the latter. In fact, PT begins from where the feature testing ends. It presupposes successful completion of features testing, meaning that all the desired functional requirements expected from the system are fully met. However, it is worth noting that conducting PT without features testing is often fraught with great difficulties. This is because the residual defects in the system, which otherwise would have been known in features testing, would give misleading results about the performance. Previous chapters emphasized the need for PT and various factors that have impact the performance of Web-based applications. This chapter provides an insight about the technology aspects, including the software languages, necessary for Web development. Needless to add, that technology as well as software languages have a critical impact on the performance of the system.

Post Test Execution Phase

The phase pertaining to post test execution comprises not only multifaceted activities but is also a tedious task. It is not uncommon to find many testers who normally underestimate the complexity involved in this phase and face an uphill task later, while fine tuning the system for optimum performance. This chapter discusses points arising in the post execution phase by considering: • Specific test execution scenarios through logs; • Method/strategy for analysis; • Results with standard benchmarks; • Areas for improvement.

Test Preparation Phase III

The next phase, that is, the build phase, commences after the completion of the design phase. Design provides the guidelines, but the build phase really implements the design so that execution of the test can be carried out later. The first activity in the build phase is to plan the various activities for testing. However, building a comprehensive test plan is as important as executing the test itself. Apart from the test plan, testing the build phase also focuses on setting up a proper test environment, test schedule, and testing processes.