Leveraging the Cloud for Large-Scale Software Testing – A Case Study

Testing is a critical phase in the software life-cycle. While small-scale component-wise testing is done routinely as part of development and maintenance of large-scale software, the system level testing of the whole software is much more problematic due to low level of coverage of potential usage scenarios by test cases and high costs associated with wide-scale testing of large software. Here, the authors investigate the use of cloud computing to facilitate the testing of large-scale software. They discuss the aspects of cloud-based testing and provide an example application of this. They describe the testing of the functional importance of methods of classes in the Google Chrome software. The methods that we test are predicted to be functionally important with respect to a functionality of the software. The authors use network analysis applied to dynamic analysis data generated by the software to make these predictions. They check the validity of these predictions by mutation testing of a large number of mutated variants of the Google Chrome. The chapter provides details of how to set up the testing process on the cloud and discusses relevant technical issues.

Cloud-Enabled Software Testing Based on Program Understanding

Cloud computing provides an innovative technology that enables Software as a Service (SaaS) to its customers. With cloud computing technologies, a suite of program understanding tools is suggested to be deployed in a cloud to aid the generation of test cases for software testing. This cloud-enabled service allows customers to use these tools through an on-demand, flexible, and pay-per-use model. Lastly, the issues and challenges of cloud computing are presented.

Cloud Environment Controls Assessment Framework

Recent years have seen the rapid growth of on-demand, flexible, low-cost cloud-based information technology services. Government and business organizations around the world have started transforming their traditional in-house data center environments to cloud-based outsourced data centers. This transformation is opening doors to new risks given that the cloud computing delivery models, related services, and technologies are still maturing and evolving. Before deployment, organizations must implement cloud environment assessment methodologies to comply with the applicable standards and regulations. They must evaluate the environment’s quality attributes of Internet connectivity, user access control, privacy and confidentiality, asset protection, multiple platforms locality, availability, reliability, performance, and scalability. The purpose of this chapter is to assist organizations that are considering providing and consuming cloud-based services in developing an assessment plan specific to organizational policies, strategies and their business and applicable legal and regulatory requirements; and assessing the cloud environment controls for infrastructure, platform, and software services.

D-Cloud

Various information systems are widely used in the information society era, and the demand for highly dependable system is increasing year after year. However, software testing for such a system becomes more difficult due to the enlargement and the complexity of the system. In particular, it is often difficult to test parallel and distributed systems in the real world after deployment, although reliable systems, such as high-availability servers, are parallel and distributed systems. To solve these problems, the authors propose a software testing environment for dependable parallel and distributed system using the cloud computing technology, named D-Cloud. D-Cloud consists of the cloud management software as the role of the resource management, and a lot of virtual machine monitors with fault injection facility in order to simulate hardware faults. In addition, D-Cloud introduces the scenario manager, and it makes a number of different tests perform automatically. Currently, D-Cloud is realized by the use of Eucalyptus as the cloud management software. Furthermore, the authors introduce FaultVM based on QEMU as the virtualization software, and D-Cloud frontend that interprets test scenario, constructs test environment, and dispatches commands. D-Cloud enables automating the system configuration and the test procedure as well as performing a number of test cases simultaneously and emulating hardware faults flexibly. This chapter presents the concept and design of D-Cloud, and describes how to specify the system configuration and the test scenario. Furthermore, the preliminary test example as the software testing using D-Cloud is presented. As the result, the authors show that D-Cloud allows easy setup of the environment, and to test the software testing for the distributed system.

Software Testing as a Service

This chapter describes a qualitative study whose aim was to explore and understand the conditions that influence software testing as a service. Interviews were conducted with software professionals from 16 organizations. The study used qualitative grounded theory as its research method. The level of domain knowledge required by testers was an initial indication of whether testing could be delivered as a service. The benefits of software testing as a service included flexibility and cost effectiveness. Among top requirements were security and pricing. Cloud computing was envisaged as the delivery model for software testing as a service. Some potential research areas suggested were pricing models and handling of test data. There was an indication that the demand for software testing as a service was on the rise, albeit with mixed feelings. Organizations would have to make careful considerations before embarking on testing their systems and applications over the internet.

Towards Migrating Genetic Algorithms for Test Data Generation to the Cloud

Search-Based Software Testing is a well-established research area, whose goal is to apply meta-heuristic approaches, like Genetic Algorithms, to address optimization problems in the testing domain. Even if many interesting results have been achieved in this field, the heavy computational resources required by these approaches are limiting their practical application in the industrial domain. In this chapter, the authors propose the migration of Search-Based Software Testing techniques to the Cloud aiming to improve their performance and scalability. Moreover, they show how the use of the MapReduce paradigm can support the parallelization of Genetic Algorithms for test data generation and their migration in the Cloud, thus relieving software company from the management and maintenance of the overall IT infrastructure and developers from handling the communication and synchronization of parallel tasks. Some preliminary results are reported, gathered by a proof-of-concept developed on the Google’s Cloud Infrastructure.

Testing in the Cloud

Cloud-based applications offer great value and benefits to businesses and other application consumers. However, unlike traditional in-house developed systems or commercial-off-the-shelf (COTS) applications, the customer has little or no control over when and how functionality may change. The cloud consumer also has little or no control over how the data controlled by the application is processed, stored, and secured. This chapter explores how the testing of cloud applications is fundamentally different from other contexts where the customer has a greater degree of control. The limitations of risk mitigation are discussed as well as cloud computing models that may also reduce the cloud consumer’s risk.

Cloud Scalability Measurement and Testing

Cloud computing introduces a new paradigm for software deployment, hosting, and service renting. Based on the XaaS architecture, a large number of users may share computing resources, platform services, and application software in a multi-tenancy approach. To ensure service availability, the system needs to support an advanced level of massive scalability so that it can provide necessary resources on demand following the pay-per-use pricing model. This chapter analyzes the unique requirements of cloud performance and scalability, compared with traditional distributed systems. Measurements are proposed with performance indicators, meters, and metrics identified from different perspectives. To support scalability testing in a dynamic environment, an agent-based testing framework is proposed to facilitate adaptive load generation and simulation using a two-layer control architecture.

Concolic Test Generation and the Cloud

Software testing in the cloud can reduce the need for hardware and software resources and offer a flexible and efficient alternative to the traditional software testing process. A major obstacle to the wider use of testing in the cloud is related to security issues. This chapter focuses on test generation techniques that combine concrete and symbolic execution of the program under test. Their deployment in the cloud leads to complex technical and security issues that do not occur for other testing methods. This chapter describes recent online deployment of such a technique implemented by the PathCrawler test generation tool for C programs, where the author faced, studied, and solved many of these issues. Mixed concrete/symbolic testing techniques not only constitute a challenging target for deployment in the cloud, but they also provide a promising way to improve the reliability of cloud environments. The author argues that these techniques can be efficiently used to help to create trustworthy cloud environments.

Building a Cloud-Based Mobile Application Testbed

A proliferation of mobile smartphone platforms, including Android devices, has triggered a rise in mobile application development for a diverse set of situations. Testing of these smartphone applications can be exceptionally difficult, due to the challenges of orchestrating production-scale quantities of smartphones such as difficulty in managing thousands of sensory inputs to each individual smartphone device. This work presents the Android Tactical Application Assessment and Knowledge (ATAACK) Cloud, which utilizes a cloud computing environment to allow smartphone-based security, sensing, and social networking researchers to rapidly use model-based tools to provision experiments with a combination of 1,000+ emulated smartphone instances and tens of actual devices. The ATAACK Cloud provides a large-scale smartphone application research testbed.