Multi-Method Vs Single Method Appraisal of Clinical Quality Indicators for the Emergency Medical Services
ABSTRACT Background Quality Indicator (QI) appraisal protocols are a novel methodology that combines multiple appraisal methods to comprehensively assess the "appropriateness" of QIs for a particular healthcare setting. However, they remain inadequately explored compared to the single appraisal method approach. This paper aimed to describe and test a QI appraisal protocol versus the single method approach, against a series of QIs potentially relevant to the South African Prehospital Emergency Care setting. Methods An appraisal protocol was developed consisting of two categorical-based appraisal methods, combined with the qualitative analysis of the discussion generated during the consensus application of each method. The output of the protocol was assessed and compared with the application and output of each method. Inter-rater reliability of each particular method was evaluated prior to group consensus rating. Variation in the number of non-valid QIs and the proportion of non-valid QIs identified between each method and the protocol were compared and assessed. Results There was mixed IRR of the individual methods. There was similarly low to moderate correlation of the results obtained between the particular methods (Spearman’s rank correlation=0.42,p<0.001). From a series of 104 QIs, 11 non-valid QIs were identified that were shared between the individual methods. A further 19 non-valid QIs were identified and not shared by each method, highlighting the benefits of a multi-method approach. The outcomes were additionally evident in the group discussion analysis, which in and of itself added further input that would not have otherwise been captured by the individual methods alone. Conclusion The utilization of a multi-method appraisal protocol offers multiple benefits, when compared to the single appraisal approach, and can provide the confidence that the outcomes of the appraisal will ensure a strong foundation on which the QI framework can be successfully implemented.