Methodology in core outcome set (COS) development: the impact of patient interviews and using a 5-point versus a 9-point Delphi rating scale on core outcome selection in a COS development study

Abstract Background As the development of core outcome sets (COS) increases, guidance for developing and reporting high-quality COS continues to evolve; however, a number of methodological uncertainties still remain. The objectives of this study were: (1) to explore the impact of including patient interviews in developing a COS, (2) to examine the impact of using a 5-point versus a 9-point rating scale during Delphi consensus methods on outcome selection and (3) to inform and contribute to COS development methodology by advancing the evidence base on COS development techniques. Methods Semi-structured patient interviews and a nested randomised controlled parallel group trial as part of the Pelvic Girdle Pain Core Outcome Set project (PGP-COS). Patient interviews, as an adjunct to a systematic review of outcomes reported in previous studies, were undertaken to identify preliminary outcomes for including in a Delphi consensus survey. In the Delphi survey, participants were randomised (1:1) to a 5-point or 9-point rating scale for rating the importance of the list of preliminary outcomes. Results Four of the eight patient interview derived outcomes were included in the preliminary COS, however, none of these outcomes were included in the final PGP-COS. The 5-point rating scale resulted in twice as many outcomes reaching consensus after the 3-round Delphi survey compared to the 9-point scale. Consensus on all five outcomes included in the final PGP-COS was achieved by participants allocated the 5-point rating scale, whereas consensus on four of these was achieved by those using the 9-point scale. Conclusions Using patient interviews to identify preliminary outcomes as an adjunct to conducting a systematic review of outcomes measured in the literature did not appear to influence outcome selection in developing the COS in this study. The use of different rating scales in a Delphi survey, however, did appear to impact on outcome selection. The 5-point scale demonstrated greater congruency than the 9-point scale with the outcomes included in the final PGP-COS. Future research to substantiate our findings and to explore the impact of other rating scales on outcome selection during COS development, however, is warranted.

Download Full-text

Methodology in core outcome set (COS) development: the impact of patient interviews and using a 5-point versus a 9-point Delphi rating scale on core outcome selection in a COS development study.

10.21203/rs.3.rs-25991/v3 ◽

2020 ◽

Author(s):

Alexandria Remus ◽

Valerie Smith ◽

Francesca Wuytack

Keyword(s):

Rating Scales ◽

Rating Scale ◽

Core Outcome Set ◽

Delphi Survey ◽

Core Outcome ◽

Point Scale ◽

Patient Interviews ◽

Point Rating Scale ◽

Outcome Selection ◽

The Impact

Abstract Background: As the development of core outcome sets (COS) increases, guidance for developing and reporting high-quality COS continues to evolve; however, a number of methodological uncertainties still remain. The objectives of this study were: (1) to explore the impact of including patient interviews in developing a COS, (2) to examine the impact of using a 5-point versus a 9-point rating scale during Delphi consensus methods on outcome selection and (3) to inform and contribute to COS development methodology by advancing the evidence base on COS development techniques. Methods: Semi-structured patient interviews and a nested randomised controlled parallel group trial as part of the Pelvic Girdle Pain Core Outcome Set project (PGP-COS). Patient interviews, as an adjunct to a systematic review of outcomes reported in previous studies, were undertaken to identify preliminary outcomes for including in a Delphi consensus survey. In the Delphi survey, participants were randomised (1:1) to a 5-point or 9-point rating scale for rating the importance of the list of preliminary outcomes. Results: Four of the eight patient interview derived outcomes were included in the preliminary COS, however, none of these outcomes were included in the final PGP-COS. The 5-point rating scale resulted in twice as many outcomes reaching consensus after the 3-round Delphi survey compared to the 9-point scale. Consensus on all five outcomes included in the final PGP-COS was achieved by participants allocated the 5-point rating scale, whereas consensus on four of these was achieved by those using the 9-point scale. Conclusions: Using patient interviews to identify preliminary outcomes as an adjunct to conducting a systematic review of outcomes measured in the literature did not appear to influence outcome selection in developing the COS in this study. The use of different rating scales in a Delphi survey, however, did appear to impact on outcome selection. The 5-point scale demonstrated greater congruency than the 9-point scale with the outcomes included in the final PGP-COS. Future research to substantiate our findings and to explore the impact of other rating scales on outcome selection during COS development, however, is warranted.

Download Full-text

Methodology in core outcome set (COS) development: the impact of patient interviews and using a 5-point versus a 9-point Delphi rating scale on core outcome selection in a COS development study.

10.21203/rs.3.rs-25991/v2 ◽

2020 ◽

Author(s):

Alexandria Remus ◽

Valerie Smith ◽

Francesca Wuytack

Keyword(s):

Rating Scales ◽

Rating Scale ◽

Core Outcome Set ◽

Delphi Survey ◽

Core Outcome ◽

Point Scale ◽

Patient Interviews ◽

Point Rating Scale ◽

Outcome Selection ◽

The Impact

Abstract Background: As the development of core outcome sets (COS) increases, guidance for developing and reporting high-quality COS continues to evolve; however, a number of methodological uncertainties still remain. The objectives of this study were: (1) to explore the impact of including patient interviews in developing a COS, (2) to examine the impact of using a 5-point versus a 9-point rating scale during Delphi consensus methods on outcome selection and the COS and (3) to inform and contribute to COS development methodology by advancing the evidence base on COS development techniques.Methods: Semi-structured patient interviews and a nested randomised controlled parallel group trial as part of the Pelvic Girdle Pain Core Outcome Set project (PGP-COS). Patient interviews, as an adjunct to a systematic review of outcomes reported in previous studies, were undertaken to identify preliminary outcomes for including in a Delphi consensus survey. In the Delphi survey, participants were randomised (1:1) to a 5-point or 9-point scale for rating the importance of the list of preliminary outcomes.Results: None of the outcomes identified from the patient interviews were included in the final PGP-COS. The 5-point rating scale resulted in twice as many outcomes reaching consensus after the 3-round Delphi survey compared to the 9-point scale. Consensus on all five outcomes included in the final PGP-COS was achieved by participants allocated the 5-point rating scale, whereas consensus on four of these was achieved by those using the 9-point scale.Conclusions: Using patient interviews to identify preliminary outcomes as an adjunct to conducting a systematic review of outcomes measured in the literature does not appear to influence outcome selection in developing a COS. The use of different rating scales in a Delphi survey, however, does appear to impact on outcome selection. The 5-point scale demonstrated greater congruency than the 9-point scale with the outcomes included in the final PGP-COS. Use of a 5-point scale might be considered for Delphi surveys in COS development studies. Future research to substantiate our findings and to explore the impact of other rating scales on outcome selection during COS development, however, is still warranted.

Download Full-text

Methodology in core outcome set (COS) development: the impact of patient interviews and using a 5-point versus a 9-point Delphi rating scale on core outcome selection in a COS development study.

10.21203/rs.3.rs-25991/v1 ◽

2020 ◽

Author(s):

Alexandria Remus ◽

Valerie Smith ◽

Francesca Wuytack

Keyword(s):

Rating Scales ◽

Rating Scale ◽

Core Outcome Set ◽

Delphi Survey ◽

Core Outcome ◽

Point Scale ◽

Patient Interviews ◽

Point Rating Scale ◽

Outcome Selection ◽

The Impact

Abstract Background As the development of core outcome sets (COS) increases, guidance for developing and reporting high-quality COS continues to evolve; however, a number of methodological uncertainties still remain. The objectives of this study were: (1) to explore the impact of including patient interviews in developing a COS (2) to examine the impact of using a 5-point versus a 9-point rating scale during Delphi consensus methods on outcome selection and the COS and (3) to inform and contribute to COS development methodology by advancing the evidence base on COS development techniques. Methods Semi-structured patient interviews (i) and a nested randomised controlled parallel group trial (ii) as part of the Pelvic Girdle Pain Core Outcome Set project (PGP-COS). Patient interviews, as an adjunct to a systematic review of outcomes reported in previous studies, were undertaken to identify preliminary outcomes for including in a Delphi consensus survey. In the Delphi survey, participants were randomised (1:1) to a 5-point or 9-point Likert scale for rating the importance of the list of preliminary outcomes. Results None of the outcomes identified from the patient interviews only were included in the final PGP-COS. The 5-point rating scale resulted in twice as many outcomes reaching consensus after the 3-round Delphi survey compared to the 9-point scale. Consensus on all five outcomes included in the final PGP-COS was achieved by participants allocated the 5-point rating scale, whereas consensus on four of these was achieved by those using the 9-point scale. Conclusions Using patient interviews to identify preliminary outcomes as an adjunct to conducting a systematic review of outcomes measured in the literature, does not appear to influence outcome selection in developing a COS. The use of different rating scales in a Delphi survey, however, does appear to impact on outcome selection. The 5-point scale demonstrated greater congruency than the 9-point scale with the outcomes included in the final PGP-COS. Use of a 5-point scale can thus be considered for Delphi surveys in COS development studies. Future research to explore the impact of other rating scales on outcome selection during COS development is still warranted.

Download Full-text

Delphi procedure in core outcome set development: rating scale and consensus criteria determined outcome selection

Journal of Clinical Epidemiology ◽

10.1016/j.jclinepi.2019.03.011 ◽

2019 ◽

Vol 111 ◽

pp. 23-31 ◽

Cited By ~ 10

Author(s):

Dorien De Meyer ◽

Jan Kottner ◽

Hilde Beele ◽

Jochen Schmitt ◽

Toni Lange ◽

...

Keyword(s):

Rating Scale ◽

Core Outcome Set ◽

Core Outcome ◽

Outcome Set ◽

Delphi Procedure ◽

Outcome Selection

Download Full-text

Important research outcomes for treatment studies of perinatal depression: systematic overview and development of a core outcome set

10.22541/au.160801147.70898783/v1 ◽

2020 ◽

Author(s):

Christel Hellberg ◽

Marie Österberg ◽

AnnKristine Jonsson ◽

Sara Fundell ◽

Frida Trönnberg ◽

...

Keyword(s):

Perinatal Depression ◽

Core Outcome Set ◽

Nominal Group Technique ◽

Antenatal Depression ◽

Delphi Survey ◽

Nominal Group ◽

Core Outcome ◽

Systematic Overview ◽

Outcome Set ◽

Outcome Selection

Objective To develop a Core Outcome Set (COS) for treatment of perinatal depression Design Systematic overview of outcomes reported in the literature and consensus development study using a Delphi survey and modified nominal group technique. Setting International. Population Two hundred and twenty-two participants, representing thirteen countries. Methods A systematic overview of outcomes reported in recently published research, a two-round Delphi survey, a consensus meeting at which the final COS was decided. Main results In the literature search, 1772 abstracts were identified and evaluated, 284 papers/protocols were assessed in full and 165 studies were finally included in the review. In all, 106 outcomes were identified and thus included in the Delphi survey. 222 participants registered for the first round of the Delphi survey and 151 (68%) responded. In the second round, 123 (55%) participants responded. The following 9 outcomes were agreed upon for inclusion in the final COS: self-assessed symptoms of depression, diagnosis of depression by a clinician, parent to infant bonding, self-assessed symptoms of anxiety, quality of life, satisfaction with intervention, suicidal thoughts, attempted or committed suicide, thoughts of harming the baby, and adverse events. Conclusions The relevant stakeholders prioritised outcomes and reached consensus on a COS comprising nine outcomes. We hope that this COS will contribute to consistency and uniformity of outcome selection and reporting in future clinical trials involving treatment of perinatal depression Funding This article is adapted from a report by SBU, which provided funding for the study. Keywords: perinatal depression, postpartum depression, antenatal depression, COS

Download Full-text

Childbirth and Posttraumatic Stress Responses

European Journal of Psychological Assessment ◽

10.1027/1015-5759.22.4.259 ◽

2006 ◽

Vol 22 (4) ◽

pp. 259-267 ◽

Cited By ~ 40

Author(s):

Eelco Olde ◽

Rolf J. Kleber ◽

Onno van der Hart ◽

Victor J.M. Pop

Keyword(s):

Posttraumatic Stress Disorder ◽

Posttraumatic Stress ◽

Stress Responses ◽

Rating Scales ◽

Rating Scale ◽

Stress Disorder ◽

Traumatic Experience ◽

Event Scale ◽

Self Rating ◽

The Impact

Childbirth has been identified as a possible traumatic experience, leading to traumatic stress responses and even to the development of posttraumatic stress disorder (PTSD). The current study investigated the psychometric properties of the Dutch version of the Impact of Event Scale-Revised (IES-R) in a group of women who recently gave birth (N = 435). In addition, a comparison was made between the original IES and the IES-R. The scale showed high internal consistency (α = 0.88). Using confirmatory factor analysis no support was found for a three-factor structure of an intrusion, an avoidance, and a hyperarousal factor. Goodness of fit was only reasonable, even after fitting one intrusion item on the hyperarousal scale. The IES-R correlated significantly with scores on depression and anxiety self-rating scales, as well as with scores on a self-rating scale of posttraumatic stress disorder. Although the IES-R can be used for studying posttraumatic stress reactions in women who recently gave birth, the original IES proved to be a better instrument compared to the IES-R. It is concluded that adding the hyperarousal scale to the IES-R did not make the scale stronger.

Download Full-text

Application of Psychometric Theory to the Measurement of Voice Quality Using Rating Scales

Journal of Speech Language and Hearing Research ◽

10.1044/1092-4388(2005/022) ◽

2005 ◽

Vol 48 (2) ◽

pp. 323-335 ◽

Cited By ~ 89

Author(s):

Rahul Shrivastav ◽

Christine M. Sapienza ◽

Vuday Nandur

Keyword(s):

Measurement Errors ◽

Rating Scales ◽

Rating Scale ◽

Voice Quality ◽

Random Order ◽

Past Research ◽

Poor Agreement ◽

Psychometric Theory ◽

Point Rating Scale ◽

Exact Agreement

Rating scales are commonly used to study voice quality. However, recent research has demonstrated that perceptual measures of voice quality obtained using rating scales suffer from poor interjudge agreement and reliability, especially in the midrange of the scale. These findings, along with those obtained using multidimensional scaling (MDS), have been interpreted to show that listeners perceive voice quality in an idiosyncratic manner. Based on psychometric theory, the present research explored an alternative explanation for the poor interlistener agreement observed in previous research. This approach suggests that poor agreement between listeners may result, in part, from measurement errors related to a variety of factors rather than true differences in the perception of voice quality. In this study, 10 listeners rated breathiness for 27 vowel stimuli using a 5-point rating scale. Each stimulus was presented to the listeners 10 times in random order. Interlistener agreement and reliability were calculated from these ratings. Agreement and reliability were observed to improve when multiple ratings of each stimulus from each listener were averaged and when standardized scores were used instead of absolute ratings. The probability of exact agreement was found to be approximately .9 when using averaged ratings and standardized scores. In contrast, the probability of exact agreement was only .4 when a single rating from each listener was used to measure agreement. These findings support the hypothesis that poor agreement reported in past research partly arises from errors in measurement rather than individual differences in the perception of voice quality.

Download Full-text

Speech-Language Pathologists' Ratings of Speech Accuracy in Children With Speech Sound Disorders

American Journal of Speech-Language Pathology ◽

10.1044/2021_ajslp-20-00381 ◽

2021 ◽

pp. 1-12

Author(s):

Linye Jing ◽

Maria I. Grigos

Keyword(s):

Interrater Reliability ◽

Rating Scales ◽

Rating Scale ◽

Speech Sound ◽

Future Research ◽

Training Procedure ◽

Speech Sound Disorders ◽

Speech Language Pathologists ◽

Point Rating Scale ◽

Whole Word

Purpose: Forming accurate and consistent speech judgments can be challenging when working with children with speech sound disorders who produce a large number and varied types of error patterns. Rating scales offer a systematic approach to assessing the whole word rather than individual sounds. Thus, these scales can be an efficient way for speech-language pathologists (SLPs) to monitor treatment progress. This study evaluated the interrater reliability of an existing 3-point rating scale using a large group of SLPs as raters. Method: Utilizing an online platform, 30 SLPs completed a brief training and then rated single words produced by children with typical speech patterns and children with speech sound disorders. Words were closely balanced across the three rating categories of the scale. The interrater reliability of the SLPs ratings to a consensus judgment was examined. Results: The majority of SLPs (87%) reached substantial interrater reliability to a consensus judgment using the 3-point rating scale. Correct productions had the highest interrater reliability. Productions with extensive errors had higher agreement than those with minor errors. Certain error types, such as vowel distortions, were especially challenging for SLPs to judge. Conclusions: This study demonstrated substantial interrater reliability to a consensus judgment among a large majority of 30 SLPs using a 3-point rating. The clinical implications of the findings are discussed along with proposed modifications to the training procedure to guide future research.

Download Full-text