Rule-based systems are typically tested using a set of inputs which will produce known outputs. However, one does not know how thoroughly the software has been exercised. Traditional test-coverage metrics do not account for the dynamic data-driven flow of control in rule-based systems. Our literature review found that there has been little prior work on coverage metrics for rule-based systems. This paper proposes test-coverage metrics for rule-based systems derived from metrics defined by prior work, and presents an industrial scale case study. We conducted a case study to evaluate the practicality and usefulness of the proposed metrics. The case study applied the metrics to a system for computational fluid-dynamics models based on a rule-based application framework. These models were tested using a regression-test suite. The data-flow structure built by the application framework, along with the regression-test suite, provided case-study data. The test suite was evaluated against three kinds of coverage. The measurements indicated that complete coverage was not achieved, even at the lowest level definition. Lists of rules not covered provided insight into how to improve the test suite. The case study illustrated that structural coverage measures can be utilized to measure the completeness of rule-based system testing.
Tightening Test Coverage Metrics: A Case Study in Equivalence Checking Using k-Induction
