Parallel Test Prioritization

Although regression testing is important to guarantee the software quality in software evolution, it suffers from the widely known cost problem. To address this problem, existing researchers made dedicated efforts on test prioritization, which optimizes the execution order of tests to detect faults earlier; while practitioners in industry leveraged more computing resources to save the time cost of regression testing. By combining these two orthogonal solutions, in this article, we define the problem of parallel test prioritization, which is to conduct test prioritization in the scenario of parallel test execution to reduce the cost of regression testing. Different from traditional sequential test prioritization, parallel test prioritization aims at generating a set of test sequences, each of which is allocated in an individual computing resource and executed in parallel. In particular, we propose eight parallel test prioritization techniques by adapting the existing four sequential test prioritization techniques, by including and excluding testing time in prioritization. To investigate the performance of the eight parallel test prioritization techniques, we conducted an extensive study on 54 open-source projects and a case study on 16 commercial projects from Baidu , a famous search service provider with 600M monthly active users. According to the two studies, parallel test prioritization does improve the efficiency of regression testing, and cost-aware additional parallel test prioritization technique significantly outperforms the other techniques, indicating that this technique is a good choice for practical parallel testing. Besides, we also investigated the influence of two external factors, the number of computing resources and time allowed for parallel testing, and find that more computing resources indeed improve the performance of parallel test prioritization. In addition, we investigated the influence of two more factors, test granularity and coverage criterion, and find that parallel test prioritization can still accelerate regression testing in parallel scenario. Moreover, we investigated the benefit of parallel test prioritization on the regression testing process of continuous integration, considering both the cumulative acceleration performance and the overhead of prioritization techniques, and the results demonstrate the superiority of parallel test prioritization.

Modification Point Aware Test Prioritization and Sampling to Improve Patch Validation in Automatic Program Repair

Recently, Automatic Program Repair (APR) has shown a high capability of repairing software bugs automatically. In general, most of the APR techniques require test suites to validate automatically generated patches. However, the test suites used for patch validation might contain thousands of test cases. Running these whole test suites to validate every program variant makes the validation process not only time-consuming but also expensive. To mitigate this issue and to enhance the patch validation in APR, we introduce (1) MPTPS (Modification Point-aware Test Prioritization and Sampling), which iteratively records test execution. Based on the failed test information, it performs test prioritization, then sampling to reduce the test execution time by moving forward the test cases that are most likely to fail in the test suite; and (2) a new fitness function that refines the existing one to improve repair efficiency. We implemented our MPPEngine approach in the Astor workspace by extending jGenProg. And the experiments on the Defects4j benchmark against jGenProg show that, on average, jGenProg consumes 79.27 s to validate one program variant, where MPPEngine takes only 33.70 s for results in 57.50% of validation time reduction. Also, MPPEngine outperforms jGenProg by finding patches for six more bugs than jGenProg.