Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster

REST web services are widely popular in industry, and search techniques have been successfully used to automatically generate system-level test cases for those systems. In this article, we propose a novel mutation operator which is designed specifically for test generation at system-level, with a particular focus on REST APIs. In REST API testing, and often in system testing in general, an individual can have a long and complex chromosome. Furthermore, there are two specific issues: (1) fitness evaluation in system testing is highly costly compared with the number of objectives (e.g., testing targets) to optimize for; and (2) a large part of the genotype might have no impact on the phenotype of the individuals (e.g., input data that has no impact on the execution flow in the tested program). Due to these issues, it might be not suitable to apply a typical low mutation rate like 1/ n (where n is the number of genes in an individual), which would lead to mutating only one gene on average. Therefore, in this article, we propose an adaptive weight-based hypermutation, which is aware of the different characteristics of the mutated genes. We developed adaptive strategies that enable the selection and mutation of genes adaptively based on their fitness impact and mutation history throughout the search. To assess our novel proposed mutation operator, we implemented it in the EvoMaster tool, integrated in the MIO algorithm, and further conducted an empirical study with three artificial REST APIs and four real-world REST APIs. Results show that our novel mutation operator demonstrates noticeable improvements over the default MIO. It provides a significant improvement in performance for six out of the seven case studies, where the relative improvement is up to +12.09% for target coverage, +12.69% for line coverage, and +32.51% for branch coverage.

Evolutionary Approaches to Test Data Generation for Object-Oriented Software

In object-oriented evolutionary testing, metaheuristics are employed to select or generate test data for object-oriented software. Techniques that analyse program structures are predominant among the panoply of studies available in current literature. For object-oriented evolutionary testing, the common objective is to reach some coverage criteria, usually in the form of statement or branch coverage. This chapter explores, reviews, and contextualizes relevant literature, tools, and techniques in this area, while identifying open problems and setting ground for future work.

Rtkaller: State-aware Task Generation for RTOS Fuzzing

A real-time operating system (RTOS) is an operating system designed to meet certain real-time requirements. It is widely used in embedded applications, and its correctness is safety-critical. However, the validation of RTOS is challenging due to its complex real-time features and large code base. In this paper, we propose Rtkaller , a state-aware kernel fuzzer for the vulnerability detection in RTOS. First, Rtkaller implements an automatic task initialization to transform the syscall sequences into initial tasks with more real-time information. Then, a coverage-guided task mutation is designed to generate those tasks that explore more in-depth real-time related code for parallel execution. Moreover, Rtkaller realizes a task modification to correct those tasks that may hang during fuzzing. We evaluated it on recent versions of rt-Linux, which is one of the most widely used RTOS. Compared to the state-of-the-art kernel fuzzers Syzkaller and Moonshine, Rtkaller achieves the same code coverage at the speed of 1.7X and 1.6X, gains an increase of 26.1% and 22.0% branch coverage within 24 hours respectively. More importantly, Rtkaller has confirmed 28 previously unknown vulnerabilities that are missed by other fuzzers.

An Adaptive Search Budget Allocation Approach for Search-Based Test Case Generation

Search-based techniques have been successfully used to automate test case generation. Such approaches allocate a fixed search budget to generate test cases aiming at maximizing code coverage. The search budget plays a crucial role; due to the hugeness of the search space, the higher the assigned budget, the higher the expected coverage. Code components have different structural properties that may affect the ability of search-based techniques to achieve a high coverage level. Thus, allocating a fixed search budget for all the components is not recommended and a component-specific search budget should be preferred. However, deciding the budget to assign to a given component is not a trivial task. In this article, we introduce Budget Optimization for Testing (BOT), an approach to adaptively allocate the search budget to the classes under test. BOT requires information about the branch coverage that will be achieved on each class with a given search budget. Therefore, we also introduce BRANCHOS, an approach that predicts coverage in a budget-aware way. The results of our experiments show that (i) BRANCHOS can approximate the branch coverage in time with a low error, and (ii) BOT can significantly increase the coverage achieved by a test generation tool and the effectiveness of generated tests.

Test Suite Optimization Using Chaotic Firefly Algorithm in Software Testing

Software testing is time consuming and a costly activity. Effective generation of test cases is necessary in order to perform rigorous testing. There exist various techniques for effective test case generation. These techniques are based on various test adequacy criteria such as statement coverage, branch coverage etc. Automatic generation of test data has been the primary focus of software testing research in recent past. In this paper a novel approach based on chaotic behavior of firefly algorithm is proposed for test suite optimization. Test suite optimization problem is modeled in the framework of firefly algorithm. An Algorithm for test optimization based on firefly algorithm is also proposed. Experiments are performed on some benchmark Program and simulation results are compared for ABC algorithm, ACO algorithm, GA with Chaotic firefly algorithm. Major research findings are that chaotic firefly algorithm outperforms other bio inspired algorithm such as artificial bee colony, Ant colony optimization and Genetic Algorithm in terms of Branch coverage in software testing.

Status Report on Software Testing: Test-Comp 2021

This report describes Test-Comp 2021, the 3rd edition of the Competition on Software Testing. The competition is a series of annual comparative evaluations of fully automatic software test generators for C programs. The competition has a strong focus on reproducibility of its results and its main goal is to provide an overview of the current state of the art in the area of automatic test-generation. The competition was based on 3 173 test-generation tasks for C programs. Each test-generation task consisted of a program and a test specification (error coverage, branch coverage). Test-Comp 2021 had 11 participating test generators from 6 countries.