144 For Audit's Sake: An Audit of Audits

Aim Audit is a mandatory requirement for completion of Dental Core Training. Consequently, many audits undertaken by trainees are to “tick a box”, leading to incomplete or poor-quality audits, with change either not implemented or never measured. This audit assesses the standard of local audits. Method All audits registered with the local audit department from 2017-2020 were assessed using a standard that “100% of audits should be full cycle” (BMJ), and “100% audits must be measured against explicit standards" (NICE). The outcome forms and submitted audit presentations were used to evaluate if the completed audits needed re-auditing, and whether they had been re-audited. Whether they were actually audits was also assessed. Results 38 audit titles were registered. 7 had not been completed. Of 31 remaining audits, 24 needed re-auditing and 7 did not. Of 24 audits needing re-audit, 4 audits had not reached the proposed re-audit deadline, 7 re-audits were completed and 13 were not, leading to 35% (7/20) re-audit compliance. 5 did not have pre-determined standards, leading to 84% standards compliance. Conclusions As shown by the results, many audits are either not audits, not completed or never re-audited, leading to no change or no measurement of change. Re-evaluation of mandatory trainee involvement in audit and an emphasis on a more department lead approach to clinical governance could enable improved continuity of audits after rotation of trainees. This may lead to higher quality work and improvement of service provision.

Robustness of Adaptive Measurement of Change to Item Parameter Estimation Error

Adaptive measurement of change (AMC) is a psychometric method for measuring intra-individual change on one or more latent traits across testing occasions. Three hypothesis tests—a Z test, likelihood ratio test, and score ratio index—have demonstrated desirable statistical properties in this context, including low false positive rates and high true positive rates. However, the extant AMC research has assumed that the item parameter values in the simulated item banks were devoid of estimation error. This assumption is unrealistic for applied testing settings, where item parameters are estimated from a calibration sample before test administration. Using Monte Carlo simulation, this study evaluated the robustness of the common AMC hypothesis tests to the presence of item parameter estimation error when measuring omnibus change across four testing occasions. Results indicated that item parameter estimation error had at most a small effect on false positive rates and latent trait change recovery, and these effects were largely explained by the computerized adaptive testing item bank information functions. Differences in AMC performance as a function of item parameter estimation error and choice of hypothesis test were generally limited to simulees with particularly low or high latent trait values, where the item bank provided relatively lower information. These simulations highlight how AMC can accurately measure intra-individual change in the presence of item parameter estimation error when paired with an informative item bank. Limitations and future directions for AMC research are discussed.