Abstract
Background and Aims
Funnel plots are used to evaluate healthcare quality by comparing hospital performances on certain outcomes. So far, in nephrology, only clinical outcomes like mortality and complications are used for this purpose. However, with the increasing importance of patient-reported outcomes (PROs; e.g. health related quality of life [HRQOL]), PROs should also be taken into account in this quality assessment. Understanding the underlying methods and challenges is needed for optimal use of PROs in this context. Therefore, we aim to provide insight into the use and interpretation of funnel plots by presenting an overview of the basic principles and considerations when applied to PROs, using examples from Dutch routine dialysis care.
Method
Data on PROs (HRQOL and symptom burden), sociodemographic and clinical characteristics of patients receiving dialysis were obtained from the Dutch renal registry and were used to illustrate and explain the different components of a funnel plot, the underlying concepts (e.g. case mix and indirect standardization) and the interpretation of funnel plots. Additionally, some methodological issues are highlighted that should be considered when these methods are applied to PROs.
Results
A funnel plot is a graphical aid that consists of four components: an indicator, reference standard, measure of precision (usually the sample size) and control limits (see Figure). Funnel plots provide insight into hospitals’ performances by comparing the observed outcome to the expected outcome. A hospital’s expected outcome is calculated using the scores in the reference population (given the hospital’s patient population, i.e. case mix) and represents the outcome that would have been observed if the hospital had performed equal to the reference standard. Hospitals may be considered as under- or overperforming when exceeding the upper or lower 95% control limit. Advantages of funnel plots include: clearly visualized precision, detection of volume-effects, discouragement of ranking hospitals and easy interpretation of results. However, without basic knowledge of underlying methods, it is easy to fall into pitfalls, including: overinterpretation of standardized scores, incorrect direct comparisons of hospitals and to assume a hospital to be in-control based on under-powered comparisons. Furthermore, application to PROs is accompanied with additional challenges such as the multidimensional nature of PROs (e.g. HRQOL) which makes adequate case mix correction more difficult, and difficulties with measuring PROs, for instance to achieve high response rates or the dependence on good psychometric properties of the PRO-measures.
Conclusion
PROs partly determine the value of nephrological care and should therefore be considered in healthcare quality assessment. Understanding of the underlying methods using funnel plots is necessary for optimal use and correct interpretation of hospital comparisons on PROs. Some challenges need to be addressed before these methods can be applied to PROs, namely: high and consistent response rates, adequate case mix correction and high-quality PRO measures.