Wisdom of the Experts versus Wisdom of the Crowd in Hospital Quality Ratings: Cross-Sectional Analyses of Google Ratings (Preprint)

BACKGROUND Popular online portals provide free and convenient access to user-generated quality reviews. Centers for Medicare and Medicaid Services (CMS) also provide patients with Hospital Compare Star Ratings (HCSR), a single public measure of hospital quality aggregating multiple quality dimensions. Consumers often use crowdsourced hospital ratings on platforms such as Google to select hospitals, but it is unknown if these ratings reflect a comprehensive measure of clinical quality. OBJECTIVE We analyze if Google online quality ratings, which reflect the wisdom of the crowd, are associated with HCSR, which reflect the wisdom of the experts. CMS revised the methodology of assigning star ratings to hospitals. Therefore, we analyze these associations before and after the 2021 revisions of the CMS rating system. METHODS We extracted Google ratings using Application Programming Interface (API) in June 2020. The HCSR data of April 2020 (before the revision of HCSR methodology) and April 2021 (after the revision of HCSR methodology) were obtained from CMS’ Hospital Compare (HC) website. We also extracted scores for the individual components of hospital quality for each of the hospitals in our sample using the code provided by HC. Fractional Response Model (FRM) was used to estimate the association between Google Ratings and HCSR and individual components of quality. RESULTS Results indicate that Google ratings are statistically associated with HCSR (P<.001) after controlling for hospital level effects. A one star improvement in CMS ratings before the change in methodology (after the change in methodology) is expected to increase the Google ratings by 0.145 (0.135) on average (95% CI 0.127- 0.163; P<.001, 95% CI 0.116-0.153; P<.001). The analyses with individual components of hospital quality reveal that Google ratings are not associated with components of HCSR that require medical expertise such as ‘Safety of care’ or ‘Readmissions’. The revised CMS rating system ameliorates previous partial inconsistencies in association between Google ratings and component scores of HCSR. CONCLUSIONS Overall, crowd sourced Google hospital ratings are informative about expert CMS hospital quality ratings and several individual quality components that are easier for patients to evaluate. Therefore, hospitals should not expect improvements in quality metrics that require expertise to assess such as safety of care and readmission to result in improved Google star ratings. Hospitals can benefit from using crowd-sourced ratings as timely, easily available, and dynamic indicators of their quality performance.

Toward a More Complete Picture of Readmission-Decreasing Initiatives

Readmissions are common and costly. This study examines the effectiveness of two initiatives known to help reduce readmissions. Using data from the American Hospital Association, the Census Bureau, and the Center for Medicare and Medicaid Services’ Hospital Compare database, we found that a higher quality of hospital care does not reduce, but in fact increases readmission rates. Although health information sharing decreases readmission rates, the effect is statistically significant only among the lowest-quality hospitals, not among mid- and high-quality hospitals. The results of our study have important policy implications for providers and hospital administrators with respect to efforts to reduce readmission rates.

The Impact of Public Reporting Schemes and Market Competition on Hospital Efficiency

In the wake of growing attempts to assess the validity of public reporting, much research has examined the effectiveness of public reporting regarding cost or quality of care. However, relatively little is known about whether transparency through public reporting significantly influences hospital efficiency despite its emerging expectations for providing value-based care. This study aims to identify the dynamics that transparency brought to the healthcare market regarding hospital technical efficiency, taking the role of competition into account. We compare the two public reporting schemes, All-Payer Claims Database (APCD) and Hospital Compare. Employing Data Envelopment Analysis (DEA) and a cross-sectional time-series Tobit regression analysis, we found that APCD is negatively associated with hospital technical efficiency, while hospitals facing less competition responded significantly to increasingly transparent information by enhancing their efficiency relative to hospitals in more competitive markets. We recommend that policymakers take market mechanisms into consideration jointly with the introduction of public reporting schemes in order to produce the best outcomes in healthcare.

Demographics matter: the potentially disproportionate effect of COVID-19 on hospital ratings

Objective To identify how features of the community in which a hospital serves differentially relate to its patients' experiences based on the quality of that hospital. Design A Finite Mixture Model (FMM) is used to uncover a mix of two latent groups of hospitals that differ in quality. In the FMM, a multinomial logistic equation relates hospital-level factors to the odds of being in either group. A multiple linear regression relates the characteristics of communities served by hospitals to the patients' expected ratings of their experiences at hospitals in each group. Thus, this association potentially varies with hospital quality. The analysis was conducted via Stata. Setting Hospital ratings are measured by Hospital Compare using the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, a patient satisfaction survey required by the Centers for Medicare and Medicaid Services for hospitals in the USA. Participants 2,816 Medicare-certified acute care hospitals across all US states. Intervention None. Main Outcome Measure Differences in the marginal impacts of key community demographics on patient experiences between the two groups of hospitals. Results We provide evidence that low-rated hospitals have much more variability in patient experience ratings than high-rated ones. Moreover, the experiences at low-rated hospitals are more sensitive to county demographic factors, which means exogenous shocks, like coronavirus disease-2019 (COVID-19), will likely affect these hospitals differently, as such shocks are known to disproportionately affect their communities. Conclusions Our results imply that low-rated hospitals with more variability in their HCAHPS responses are more likely to face adverse patient experiences due to COVID-19 than high-rated hospitals. Pandemics like COVID-19 create conditions that intensify the already high demands placed on hospitals and care providers and make it even more challenging to deliver quality care.

An approach to exploring associations between hospital structural measures and patient satisfaction by distance‐based analysis

Background Patient satisfaction studies have explored domains of patient satisfaction, the determinants of domains, and score differences of domains by patient/hospital structural measures but reports on the structure of patient satisfaction with respect to similarities among domains are scarce. This study is to explore by distance-based analysis whether similarities among patient-satisfaction domains are influenced by hospital structural measures, and to design a model evaluating relationships between the structure of patient satisfaction and hospital structural measures. Methods The Hospital Consumer Assessment of Healthcare Providers and Systems 2012 survey scores and their structural measures from the Hospital Compare website reported adjusted percentages of scale for each hospital. Contingency tables of nine measures and their ratings were designed based on hospital structural measures, followed by three different distance-based analyses - clustering, correspondence analysis, and ordinal multidimensional scaling – for robustness to identify homogenous groups with respect to similarities. Results Of 4,677 hospitals, 3,711 (79.3%) met the inclusion criteria and were analyzed. The measures were divided into three groups plus cleanliness. Certain combinations of these groups were shown to be dependent on hospital structural measures. High value ratings for communication and low value ratings for medication explanation, quietness and staff responsiveness were not influenced by hospital structural measures, but the varied-ratings domain group similarities, including items such as global evaluation and pain management, were affected by hospital structural measures. Conclusions Distance-based analysis can reveal the hidden structure of patient satisfaction. This study suggests that hospital structural measures including hospital size, the ability to provide acute surgical treatment, and hospital interest in improving medical care quality are factors which may influence the structure of patient satisfaction.

An Approach to Exploring Associations between Hospital Structural Measures and Patient Satisfaction by Distance-based Analysis

Background: Patient satisfaction studies have explored domains of patient satisfaction, the determinants of domains, and score differences of domains by patient/hospital structural measures but reports on the structure of patient satisfaction with respect to similarities among domains are scarce. This study is to explore by distance-based analysis whether similarities among patient-satisfaction domains are influenced by hospital structural measures, and to design a model evaluating relationships between the structure of patient satisfaction and hospital structural measures.Methods: The Hospital Consumer Assessment of Healthcare Providers and Systems 2012 survey scores and their structural measures from the Hospital Compare website reported adjusted percentages of scale for each hospital. Contingency tables of nine measures and their ratings were designed based on hospital structural measures, followed by three different distance-based analyses - clustering, correspondence analysis, and ordinal multidimensional scaling – for robustness to identify homogenous groups with respect to similarities.Results: Of 4,677 hospitals, 3,711 (79.3%) met the inclusion criteria and were analyzed. The measures were divided into three groups plus cleanliness. Certain combinations of these groups were shown to be dependent on hospital structural measures. High value ratings for communication and low value ratings for medication explanation, quietness and staff responsiveness were not influenced by hospital structural measures, but the varied-ratings domain group similarities, including items such as global evaluation and pain management, were affected by hospital structural measures.Conclusions: Distance-based analysis can reveal the hidden structure of patient satisfaction. This study suggests that hospital structural measures including hospital size, the ability to provide acute surgical treatment, and hospital interest in improving medical care quality are factors which may influence the structure of patient satisfaction.

An Approach to Assessing Patient Satisfaction in HCAHPS Databases by Analyses Using Distances.

Background: Patient satisfaction studies have explored domains of patient satisfaction, the determinants of domains, and score differences of domains by patient/hospital characteristics but reports on the structure of patient satisfaction with respect to similarities among domains are scarce.Objective: To explore whether similarities among patient-satisfaction domains are influenced by hospital characteristics by analyses using distances, and to design a model evaluating relationships between the structure of patient satisfaction and hospital characteristics.Methods: Hospital Consumer Assessment of Healthcare Providers and Systems 2012 survey scores and their structural measures from the Hospital Compare website reported the adjusted percentages of scale for each hospital. Contingency tables of nine measures and their ratings were designed based on hospital characteristics, followed by three different analyses using distances - clustering, correspondence analysis and ordinal multidimensional scaling – for robustness to identify homogenous groups with respect to similarities.Results: Of 4,677 hospitals, 3,711 (79.3%) met the inclusion criteria and were analyzed. The measures were divided into three groups plus cleanliness. Certain combinations of these groups were shown to be dependent on hospital characteristics. High value ratings for communication and low value ratings for medication explanation, quietness and staff responsiveness were not influenced by hospital characteristics, but the varied-ratings domain group similarities, including items such as global evaluation and pain management, were affected by hospital characteristics.Conclusions: Analyses using distances can reveal the hidden structure of patient satisfaction. This study suggests that hospital characteristics including hospital size, the ability to provide acute surgical treatment, and hospital interest in improving medical care quality are factors which may influence the structure of patient satisfaction.