Engaging citizens in the development of a health system performance assessment framework: a case study in Ireland

Background The launch in 2017 of the Irish 10-year reform programme Sláintecare represents a key commitment in the future of the health system. An important component of the programme was the development of a health system performance assessment (HSPA) framework. In 2019, the Department of Health of Ireland (DoH) and Health Service Executive (HSE) commissioned the technical support of researchers to develop an outcome-oriented HSPA framework which should reflect the shared priorities of multiple stakeholders, including citizens. This study describes the method applied in the Irish context and reflects on the added value of using a citizen panel in the development of an HSPA framework. Methods A panel of 15 citizens was convened, recruited by a third-party company using a sampling strategy to achieve a balanced mix representing the Irish society. Panellists received lay-language preparatory materials before the meeting. Panellists used a three-colour scheme to signal the importance of performance measures. An exit questionnaire was administered to understand how participants experienced being part of the panel. The citizen panel was the first in a series of three panels towards the development of the HSPA framework, followed by panels including representatives of the DoH and HSE, and representatives from professional associations and special interest groups. Results The citizen panel generated 249 health performance measures ranging across 13 domains. Top-ranking domains to the citizen panel (people-centredness, coordination of care, and coverage) were less prioritized by the other panels; domains less prioritized by the citizen panel, such as accessibility, responsiveness, efficiency, and effectiveness, were of higher priority in the other panels. Citizen panellists shared a similar understanding of what a citizen panel involves and described their experience at the panel as enjoyable, interesting, and informative. Conclusions The priorities of the citizen panel were accounted for during all phases of developing the HSPA framework. This was possible by adopting an inclusive development process and by engaging citizens early on. Citizen engagement in HSPA development is essential for realizing citizen-driven healthcare system performance and generating trust and ownership in performance intelligence. Future research could expand the use of citizen panels to assess, monitor, and report on the performance of healthcare systems.

Framework for assessing the performance of the Ghanaian health system: study protocol

Background Assessing the performance of health systems through quantitative and qualitative methods is recognized as an effective approach to strengthening national health systems. However, while many high-income countries have institutionalized health system performance assessment (HSPA) as an integral component of their respective health systems, few studies on HSPA have been documented in low- and middle-income countries, including Ghana. This study aims at providing a comprehensive framework for periodic assessment of the performance of the entire health system in Ghana. Methods The study will have four work packages. First, a structured review will be conducted to identify both international and national HSPA frameworks that could be applied to the Ghanaian context. Second, based on the structured review, an assessment framework tailored to the Ghanaian health system context will be developed. Third, the draft framework will be presented and discussed with experts and stakeholders for its appropriateness, feasibility and acceptability. Finally, the framework will be piloted to assess its effectiveness and suitability for full-scale implementation. Discussion Currently, Ghana does not have a full-fledged HSPA tool that provides a holistic health sector-wide approach to assessing health system performance. Thus, developing this HSPA framework for the country will provide a tool for periodic and comprehensive assessment of the performance of the health system, which can be compared with that of other countries. Such a comparison will offer the opportunity for mutual learning and for exploring new options for formulating more effective national health policies. As this is expected to be the first attempt to develop a comprehensive HSPA framework in Ghana, this study will provide a basis for future discussions on how to further develop and implement HSPA programmes in the country.

9.I. Workshop: Health systems performance – making the links between what we do and what we want to achieve

In this session, WHO together with the UHC2030 Technical Working Group on Health System Assessment and the European Observatory on Health Systems and Policies discuss the main elements of their joint approach to HSPA. The workshop intends to propose a common and practical framework that links descriptive health system analysis with evaluation, focussing on health system outcomes. The framework's approach to uncovering health system challenges and bottlenecks and the sources of variation will be discussed - showcasing how Health System Performance Assessment can be used as a means to uncover sources of health system inefficiencies and inequities. The workshop intends to identify the links between intermediate and ultimate goals and relate these to specific actions and actors within a health system, thus providing policy makers with an actionable tool to assess performance. The session is structured into two parts. In the first part the process of developing the HSPA framework and the framework itself will be introduced. The presentation will provide insights into the joint framework based on health system functions and emphasise the links to the intermediate and final goals of the HSPA framework. This will be followed by four short case studies pertaining to the four functions - governance, financing, resource generation and service delivery - and linking specific assessment areas to intermediate and final goals. The second part is an interactive discussion, to be kicked off by the input of Kanitsorn Sumriddetchkajorn and moderated by Josep Figueras, to jointly identify the strengths and weaknesses in performance of a given health system. Key messages Health Systems Performance Assessments should focus on what health systems do and how well, rather than on what they are. Health System Performance Assessment should focus on performance and use their evaluative angle to get to the root causes of not achieving final health system goals.

Engaging citizens in the co-production of a health system performance assessment framework: a case study in Ireland

Background The launch in 2017 of the Irish 10-year reform programme Sláintecare represents a key commitment in the future of the health system. An important component of the programme was the development of a health system performance assessment (HSPA) framework. In 2019, the Department of Health of Ireland (DoH) and Health Service Executive (HSE) commissioned the technical support of researchers to develop an outcome-oriented HSPA framework, which should reflect the shared priorities of multiple stakeholders, including citizens. This study describes the method applied in the Irish context and reflects on the added value of using a citizen panel in the co-production of an HSPA framework. Methods A panel of 15 citizens was convened, recruited by a third-party company using a sampling strategy to achieve a balanced mix representing the Irish society. Panelists received lay-language preparatory materials prior to the meeting. Panelists used a three-color scheme to signal the inclusion and importance of performance measures. An exit questionnaire was administered to understand how participants experienced being part of the panel. The citizen panel was the first in a series of three panels towards the development of the HSPA framework, followed by panels including representatives of the DoH and HSE, and representatives from professional associations and special interest groups. Results The citizen panel generated 249 health performance measures ranging across 13 domains. Domains assessed as the most important included people-centeredness, coordination of care, and coverage. Prioritization of domains differed between panels. Citizen panelists shared a similar understanding of what a citizen panel involves and described their experience at the panel as enjoyable, interesting, and informative. Conclusions The engagement of citizens early on in the co-production process of the HSPA framework shaped the processes that followed, with the restating of priorities of the citizen panel informing decision-making throughout. Citizen engagement in HSPA development is essential for realizing value-based people-centered health systems and assuring an inclusive process that helps to generate trust and ownership of performance intelligence. Future research could expand on how citizen panels could be further engaged in co-creating mechanisms to assess, monitor, and report on the performance of health care systems.

Framework for Assessing the Performance of the Ghanaian Health System: Study Protocol

Background: Assessing the performance of health systems through quantitative and qualitative methods is recognized as an effective approach to strengthening national health systems. However, while many high-income countries have institutionalized health system performance assessment (HSPA) as an integral component of their respective health systems, few studies on HSPA have been documented in low and middle-income countries, including Ghana. This study aims at providing a comprehensive framework for periodic assessment of the performance of the entire health system in Ghana.Methods: The study will have four work packages. First, a structured review will be conducted to identify both international and national HSPA frameworks that could be applied to the Ghanaian context. Second, based on the structured review, an assessment framework tailored to the Ghanaian health system context will be developed. Third, the draft framework will be presented and discussed with experts and stakeholders for its appropriateness, feasibility and acceptability. Finally, the framework will be piloted to assess its effectiveness and suitability for full-scale implementation.Discussion: Currently, Ghana does not have a full-fledged HSPA tool that provides a holistic health sector-wide approach to assessing health system performance. Thus, developing this HSPA framework for the country will provide a tool for periodic and comprehensive assessment of the performance of the health system, which can be compared with that of other countries. Such a comparison will offer the opportunity for mutual learning and for exploring new options for formulating more effective national health policies. As this is expected to be the first attempt to develop a comprehensive HSPA framework in Ghana, this study will provide a basis for future discussions on how to further develop and implement HSPA programs in the country.

31.F. Workshop: Health system (performance) assessment: working towards a common approach

Since the publication of the World Health Report 2000, which aimed at assessing health systems performance globally, a number of health systems performance assessment (HSPA) frameworks and tools, designed for various purposes, have emerged. In 2017, the joint Universal Health Coverage 2030 Technical Working Group on Health Systems Assessment (TWG), consisting of health policy makers and global actors, such as the WHO, World Bank, USAID, Unicef, European Commission, OECD, etc., was formed with the aim to align the existing tools and develop a common approach to understanding and measuring health systems performance globally. As part of this work, the European Observatory on Health Systems and Policies (OBS) has developed a Health Systems Performance Assessment framework for Universal Health Coverage (HSPA Framework for UHC), that allows to evaluate health system performance, largely by drawing on the information available in the existing tools or through global data collection initiatives. Suggested approach is based on the detailed assessment of the four health system functions (governance, financing, resource generation and service delivery) and their sub-functions, the interaction of which jointly leads to the successes and failures in the achievement of health system goals. We propose distinct assessment areas for specific functions and sub-functions as a way to identify potential strengths and weaknesses, which can then be related to actions or responsibilities of specific actors within a health system, or linkages to other sectors. The objective of the workshop is to introduce the background, rationale, methods and process of developing of the HSPA framework for UHC; present a systematic approach to assessing various functions of health system and their interactions; relate the assessment areas of specific functions and sub-functions to the final health system goals; and, finally, to test the use of the framework in practice. The workshop is structured as follows: the workshop will start from the four 10-minute presentations (outlined below), followed by an interactive session with the audience to discuss the content of the framework and elaborate on its potential use for HSPA and applications globally. Key messages The HSPA framework for UHC is an approach that allows to bring together existing HSPA efforts and moves towards a more systematic and universally applicable health system performance assessment. The approach assesses performance through attribution of various health system outcomes to specific functions and sub-functions, allowing to identify strengths and weaknesses that affect performance.

Health system performance assessment from the private health insured persons’ perspective in Germany

Background About 11% of the German population has full private health insurance (PHI) and mainly consists of self-employed persons, civil servants or persons with an income above a certain threshold (in 2020: 62,550 €/year) who choose to opt out from statutory health insurance. It can be assumed that these persons represent a distinct population group in Germany. Therefore, the assessment of the German health system performance from the perspective of persons with PHI was subject to this research (project IPHA “Integrating the Population Perspective in Health System Performance Assessment”). Methods A paper/online survey was conducted in 2018 among 20,000 persons with PHI in Germany. The items for this survey were based on the intermediate (access, coverage, quality and safety) and final goals (improved health, responsiveness, improved efficiency, and social and financial risk protection) of the WHO Health Systems Framework. Results The survey was completed by 3,601 participants (18.0%). Participants (age 58.6 years ± 14.6; 64.6% male) assessed the German health system very differently, exemplarily shown for the intermediate goal “access” and final goal “responsiveness”: Whereas access to off-hour care was perceived as difficult by 54.1% of the respondents, 6.7% of the respondents reported unmet needs within the last 12 months due to waiting times. 51,5% of these persons with unmet needs due to waiting times were still (very) satisfied with the overall waiting time for physicians' appointments. 73,6% of persons with unmet needs who perceived discrimination in their care reported waiting times as the area of discrimination (vs. 53,4% of all participants who perceived discrimination). Conclusions Privately insured persons in Germany perceived the performance of the health system very differently and also partially inconsistent. Further analyses will complete the picture of the persons with PHI's perspective in health system performance assessment. Key messages Persons with private health insurance perceive the performance of the German health care system very differently and also partially inconsistent. Despite unmet needs due to waiting time in 6.7% of the respondents, 51,5% of these persons were still (very) satisfied with waiting times.

Testing the framework in real life – application for country HSPA process

The ability to use the existing health system assessment tools, HSPA processes and widely available indicators is an important, although not exhaustive, part of the practical application of the framework. With this in mind, we created an annotated template that can be used along with HSPA framework for UHC to facilitate information collection and harmonisation. This annotated template was tested for Estonia and Azerbaijan with the aim to help with evaluating the transferability of HSPA information between existing approaches and in different settings. More specifically, we tested the extent to which information on assessment areas of the framework can be extracted from the existing sources; the ease of use and clarity of the annotated template; and how the information collected through the annotated template can be combined into a meaningful health system performance assessment, in line with health system goals. The test showed that the information for Estonia and Azerbaijan could be obtained to complete much of the template. This can be done through a country-led performance assessment process that ensures that the indicative measures in the annotated template are relevant to the country context. The information collected also needs to be assessed in the context of policy priorities of the national health systems. Thus, more detailed guidance on the use of the framework can help to streamline such national processes and clarify the linkages between performance of specific functions and overall health system performance.

Integrating the Population Perspective into Health System Performance Assessment (IPHA): Study Protocol for a Cross-Sectional Study in Germany Linking Survey and Claims Data of Statutorily and Privately Insured

Background: Health system performance assessment (HSPA) is a major tool for evidence-based governance in health systems and patient/population-orientation is increasingly considered as an important aspect. The IPHA study aims (1) to undertake a comprehensive performance assessment of the German health system from a population perspective based on the intermediate and final dimensions defined by the World Health Organization (WHO) and (2) to identify differences in HSPA between (a) common user characteristics and (b) user types, which differ in their interactions and patterns of action within the health system. Methods and Analysis: A cross-sectional survey was conducted between October and December 2018 with statutorily and privately health insured to assess the German health system from a population perspective related to the past 12 months. The random sample consists of 32 000 persons insured by AOK Nordost and 20 000 persons insured by Debeka. Data from the survey will subsequently be linked with health insurance claims data at the individual level for each respondent who has given consent for data linkage. Claims data covers the time period January 1, 2017 to June 30, 2018. The combination of the 2 data sources allows to identify associations between insured patient characteristics and differences in the assessment of health system performance. The survey consists of 71 items measuring all final and intermediate health system goals defined by the WHO and user characteristics like health literacy, self-efficacy, the attention an individual pays to his or her health or disease, the personal network, autonomy, compliance and sociodemographics. The claims data contains information on morbidity, care delivery, service utilization, (co)payments and sociodemography. Discussion: The study represents a promising attempt to perform a holistic HSPA using a population perspective. For this purpose, a questionnaire was designed that contains both validated and new items in order to collect data on all relevant health system dimensions. In particular, linking survey data on HSPA with claims data is of high potential for assessing and analysing determinants of health system performance from the population perspective.