Calibrating a Discrete-Event Simulation for Quantification of Sex-Specific Colorectal Neoplasia Development

Background: Medical evidence collected from new observational studies can sometimes significantly alter our understanding of disease incidence and progression. This requires efficient and accurate calibration of disease models to help quantify the differences between observed cohorts. However, in model calibration, it is common to encounter overfitting with many model parameters but few observational outcomes. Additionally, the difficulty in evaluating fitting performance is significant due to a large degree of outcome variation and expensive computations for even a single simulation run. Methods: We developed a two-phase calibration procedure to address the above challenges. As a proof-of-the-concept study, we verified the procedure with a discrete-event-simulation-based study on sex-specific colorectal neoplasia development. For the study, we estimated eight disease model parameters that govern colorectal adenoma incidence risk and growth rates at three distinct states: non-advanced, advanced adenoma, and adenoma becoming cancerous. For the calibration, we defined the likelihood measure by a relative weighted sum-of-squares difference between the three actual prevalence values reported in a recent publication and those predicted by a discrete-event colorectal cancer simulation. In phase I of the calibration procedure, we performed a series of low-dimensional sampling-based grid searches to identify reasonably good candidate parameter designs. In phase II, we performed a local search-based approach to further improve the model fit.Results: Overall, our two-phase procedure showed better goodness of fit than a straightforward implementation of the Nelder-Mead algorithm, yielding a 10-fold reduction in calibration error (0.0025 vs. 0.0251 for an all-white mixed-family-history male cohort on the likelihood measure defined above). Further, the two-phase procedure was more effective in calibrating a validated simulation model for a female cohort than a male cohort. Finally, in phase II, performing local search on each of the parameters sequentially is more effective than searching the entire parameter space simultaneously. Conclusions: The proposed two-phase calibration procedure is effective for estimating computationally expensive stochastic dynamic disease models. In addition, initial parameter search range truncation and sensitivity analysis on various parameters can be computationally cost-effective.