Validation of rating processes within an argument-based framework

2017 ◽  
Vol 35 (4) ◽  
pp. 477-499 ◽  
Author(s):  
Ute Knoch ◽  
Carol A. Chapelle
Keyword(s):  
Validation Studies ◽  
Test Score ◽  
Rating Scale ◽  
Language Testing ◽  
Test Interpretation ◽  
Validity Argument ◽  
Linguistic Performance ◽  
Manual Search ◽  
Rating Process

Argument-based validation requires test developers and researchers to specify what is entailed in test interpretation and use. Doing so has been shown to yield advantages (Chapelle, Enright, & Jamieson, 2010), but it also requires an analysis of how the concerns of language testers can be conceptualized in the terms used to construct a validity argument. This article presents one such analysis by examining how issues associated with the rating of test takers’ linguistic performance can be included in a validity argument. Through a manual search of published language testing research, we gathered examples of research studies investigating the quality of rating processes and products. We then analyzed them in terms of how the research could be framed within a validity argument. Drawing on Kane’s (2001, 2006, 2013) conceptualization of inferences, warrants, and assumptions, we show that the relevance of research about the rating of test performances extends beyond one or two inferences about rater reliability. Such research results, for example, provide backing for assumptions about the correspondence of the rating scale to the test construct (explanation inference) and the context of extrapolation as well as the decisions made based on the ratings and their consequences. Our analysis reveals a picture of the extensive reach of the rating process into many aspects of test score meaning as well as concrete suggestions for integrating rating issues into future argument-based validation studies.

scholarly journals A systematic review of the validity and reliability of apathy scales in neurodegenerative conditions

2014 ◽  
Vol 27 (6) ◽  
pp. 903-923 ◽  
Author(s):  
Ratko Radakovic ◽  
Catherine Harley ◽  
Sharon Abrahams ◽  
John M. Starr
Keyword(s):  
Systematic Review ◽  
Neurodegenerative Diseases ◽  
Neurodegenerative Disease ◽  
Psychometric Properties ◽  
Validation Studies ◽  
Methodological Quality ◽  
Rating Scale ◽  
Validity And Reliability ◽  
Prevalent Symptom

ABSTRACTBackground:There are several scales used to detect apathy in disease populations. Since apathy is a prevalent symptom in many neurodegenerative diseases, this is an especially important context in which to identify and compare scales.Aims:To provide an overview of apathy scales validated in generic and specific neurodegenerative disease populations, compare validation studies’ methodological quality and the psychometric properties of the validated apathy scales.Methods:A systematic review of literature was conducted of articles published between 1980 and 2013. The final articles selected for review were rated on methodological quality and the psychometric properties of the scales used were interpreted.Results:Sixteen articles validating apathy scales were included in the review, five in a generic neurodegenerative sample and eleven in specific neurodegenerative samples. The methodological quality of specific studies varied from poor to excellent. The highest quality, which had psychometrically favorable scales, were the dementia apathy interview and rating (DAIR) and the apathy evaluation scale-clinical version (AES-C) in Alzheimer's disease and the Lille apathy rating scale (LARS) in Parkinson's disease. Generic neurodegenerative disease validation studies were of average methodological quality and yielded inconsistent psychometric properties.Conclusions:Several instruments can be recommended for use in some specific neurodegenerative diseases. Other instruments should either be validated or developed to assess apathy in more generic populations.

The Assessment of Disability in Patients With Psychotic Disorders: An Application of the ICIDH-2

2002 ◽  
Vol 36 (1) ◽  
pp. 127-132 ◽  
Author(s):  
Prem Chopra ◽  
Jeremy Couper ◽  
Helen Herrman
Keyword(s):  
Subjective Experience ◽  
Psychotic Disorders ◽  
Rating Scale ◽  
International Classification Of Functioning ◽  
Perfect Agreement ◽  
Rating Process ◽  
Participation Restrictions ◽  
Comprehensive Framework

Objectives: To describe the application of the International Classification of Functioning and Disability (ICIDH-2) in the assessment of disability in patients with psychotic disorders. Method: The study was conducted at the inpatient unit of St Vincent's Mental Health Service, Melbourne. Twenty inpatients with psychotic disorders were concurrently assessed by two raters. An evaluation of the feasibility of the instrument was made during this process. Inter-rater agreement was determined using raw agreement percentages and weighted kappas. Results: The ICIDH-2 provides a comprehensive framework for the description and measurement of disability, including the dimensions of impairments in body structures and functions, activity limitations, participation restrictions and environmental factors. Certain codes are difficult for clinicians to rate because of the subjective manner in which they are defined. Our inter-rater reliability assessment varied across the dimensions from poor to almost perfect agreement. Conclusions: The ICIDH-2 is potentially useful in the clinical assessment of disability in patients with psychotic disorders although modifications are needed. These could include the use of anchor points defined for each dimension or code to assist the rating process and an alternative rating scale in which categories for rating are more broadly defined. Formal training may also be necessary to enable standardization of the rating process. There is also a need to augment measurements by clinicians with the assessment of the subjective experience of patients using self-rated disability and assessment of quality of life.

The functionality and quality of nutrition-related mobile applications in China (Preprint)

2018 ◽  
Author(s):  
Yuan Li ◽  
Jingmin Ding ◽  
Yishan Wang ◽  
Chengyao Tang ◽  
Puhong Zhang
Keyword(s):  
Disease Management ◽  
Nutrition Education ◽  
Mobile Applications ◽  
Rating Scale ◽  
Dietary Guidelines ◽  
Diet Analysis ◽  
Innovative Technology ◽  
Nutritional Knowledge ◽  
Food Record

BACKGROUND There are an increasing number of mobile applications that provide dietary guidance in supporting healthy lifestyle and disease management. However, the characteristics of these nutrition-related apps are not well analyzed. OBJECTIVE This study aimed to evaluate the functionality and quality of nutrition-related apps in China. METHODS Mobile apps providing dietary guidance were screened in Chinese iOS and android app stores using stepwise searching criteria in November 2017. Primary review was conducted by extracting information from the description of apps. Free apps that contain all the information of diet and nutrition, with the last update after 1st Jan 2016 were downloaded for further analysis on the nutritional functionality features based on the framework of Chinese Dietary Guidelines and on the market related features as well. The user version of the Mobile Application Rating Scale (uMARS) was used to assess the quality of apps. RESULTS Among the 44 downloaded nutrition-related apps screened from 628 apps with dietary guidance content, only 11(25%) were aimed exclusively for dietary guidance, the others were aimed for fitness guidance (17, 39%), disease management (11, 25%) and maternal health (5, 11%) respectively. The nutritional functionalities of the 44 apps included nutritional information enquiry (40, 91%), nutrition education (35, 80%), food record (34, 77%), diet analysis (34, 77%), and personalized recipes (21, 48%). Twelve out of 44 apps contained all of the above five nutrition related functionalities. The diet analysis and suggestions were mainly focused on energy (33/44, 75%), less on other factors like dietary structure (10/44, 23%). About 96% (42/44) of the apps provided social communication functionality and 59% (26/44) of the apps supported user incentives. Eight out of 44 apps (18%) also applied intelligent recognition technology. Using 5-point uMARS as scales of quality, the median scores of the 44 apps was 3.6 (IQR = 0.7). CONCLUSIONS Most nutrition-related apps are developed in the service of health management other than for dietary guidance exclusively. Although basic energy balance theory was generally applied, the nutritional functionality of the nutrition-related apps is relatively limited and not individualized. More efforts should be taken to integrate with the scientific nutritional knowledge and innovative technology in developing an app followed with complete and personalized dietary guidance.

Psychometric evaluation of the Mobile Application Rating Scale (MARS) (Preprint)

2020 ◽  
Author(s):  
Yannik Terhorst ◽  
Paula Philippi ◽  
Lasse Sander ◽  
Dana Schultchen ◽  
Sarah Paganini ◽  
...  
Keyword(s):  
Health Care ◽  
Construct Validity ◽  
Mobile Application ◽  
Concurrent Validity ◽  
Rating Scale ◽  
Psychometric Evaluation ◽  
Confirmatory Factor ◽  
Health Domains ◽  
Psychometric Quality

BACKGROUND Mobile health apps (MHA) have the potential to improve health care. The commercial MHA market is rapidly growing, but the content and quality of available MHA are unknown. Consequently, instruments of high psychometric quality for the assessment of the quality and content of MHA are highly needed. The Mobile Application Rating Scale (MARS) is one of the most widely used tools to evaluate the quality of MHA in various health domains. Only few validation studies investigating its psychometric quality exist with selected samples of MHAs. No study has evaluated the construct validity of the MARS and concurrent validity to other instruments. OBJECTIVE This study evaluates the construct validity, concurrent validity, reliability, and objectivity, of the MARS. METHODS MARS scoring data was pooled from 15 international app quality reviews to evaluate the psychometric properties of the MARS. The MARS measures app quality across four dimensions: engagement, functionality, aesthetics and information quality. App quality is determined for each dimension and overall. Construct validity was evaluated by assessing related competing confirmatory models that were explored by confirmatory factor analysis (CFA). A combination of non-centrality (RMSEA), incremental (CFI, TLI) and residual (SRMR) fit indices was used to evaluate the goodness of fit. As a measure of concurrent validity, the correlations between the MARS and 1) another quality assessment tool called ENLIGHT, and 2) user star-rating extracted from app stores were investigated. Reliability was determined using Omega. Objectivity was assessed in terms of intra-class correlation. RESULTS In total, MARS ratings from 1,299 MHA covering 15 different health domains were pooled for the analysis. Confirmatory factor analysis confirmed a bifactor model with a general quality factor and an additional factor for each subdimension (RMSEA=0.074, TLI=0.922, CFI=0.940, SRMR=0.059). Reliability was good to excellent (Omega 0.79 to 0.93). Objectivity was high (ICC=0.82). The overall MARS rating was positively associated with ENLIGHT (r=0.91, P<0.01) and user-ratings (r=0.14, P<0.01). CONCLUSIONS he psychometric evaluation of the MARS demonstrated its suitability for the quality assessment of MHAs. As such, the MARS could be used to make the quality of MHA transparent to health care stakeholders and patients. Future studies could extend the present findings by investigating the re-test reliability and predictive validity of the MARS.

scholarly journals A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code

SLEEP ◽  
2020 ◽  
Author(s):  
Luca Menghini ◽  
Nicola Cellini ◽  
Aimee Goldstone ◽  
Fiona C Baker ◽  
Massimiliano de Zambotti
Keyword(s):  
Open Source ◽  
Validation Studies ◽  
Large Scale ◽  
Analytical Framework ◽  
Clinical Settings ◽  
Large Scale Data ◽  
Fast Pace ◽  
Epoch Analysis ◽  
Tracking Devices

Abstract Sleep-tracking devices, particularly within the consumer sleep technology (CST) space, are increasingly used in both research and clinical settings, providing new opportunities for large-scale data collection in highly ecological conditions. Due to the fast pace of the CST industry combined with the lack of a standardized framework to evaluate the performance of sleep trackers, their accuracy and reliability in measuring sleep remains largely unknown. Here, we provide a step-by-step analytical framework for evaluating the performance of sleep trackers (including standard actigraphy), as compared with gold-standard polysomnography (PSG) or other reference methods. The analytical guidelines are based on recent recommendations for evaluating and using CST from our group and others (de Zambotti and colleagues; Depner and colleagues), and include raw data organization as well as critical analytical procedures, including discrepancy analysis, Bland–Altman plots, and epoch-by-epoch analysis. Analytical steps are accompanied by open-source R functions (depicted at https://sri-human-sleep.github.io/sleep-trackers-performance/AnalyticalPipeline_v1.0.0.html). In addition, an empirical sample dataset is used to describe and discuss the main outcomes of the proposed pipeline. The guidelines and the accompanying functions are aimed at standardizing the testing of CSTs performance, to not only increase the replicability of validation studies, but also to provide ready-to-use tools to researchers and clinicians. All in all, this work can help to increase the efficiency, interpretation, and quality of validation studies, and to improve the informed adoption of CST in research and clinical settings.

scholarly journals The effects of intensive home treatment on self-efficacy in patients recovering from a psychiatric crisis

2021 ◽  
Vol 15 (1) ◽  
Author(s):  
Ansam Barakat ◽  
Matthijs Blankers ◽  
Jurgen E Cornelis ◽  
Nick M Lommerse ◽  
Aartjan T F Beekman ◽  
...  
Keyword(s):  
Quality Of Life ◽  
Mental Disorders ◽  
Self Efficacy ◽  
Psychotic Disorders ◽  
Rating Scale ◽  
Small Sample ◽  
Home Treatment ◽  
Differential Effects ◽  
Psychiatric Crisis

Abstract Background This study evaluated whether providing intensive home treatment (IHT) to patients experiencing a psychiatric crisis has more effect on self-efficacy when compared to care as usual (CAU). Self-efficacy is a psychological concept closely related to one of the aims of IHT. Additionally, differential effects on self-efficacy among patients with different mental disorders and associations between self-efficacy and symptomatic recovery or quality of life were examined. Methods Data stem from a Zelen double consent randomised controlled trial (RCT), which assesses the effects of IHT compared to CAU on patients who experienced a psychiatric crisis. Data were collected at baseline, 6 and 26 weeks follow-up. Self-efficacy was measured using the Mental Health Confidence Scale. The 5-dimensional EuroQol instrument and the Brief Psychiatric Rating Scale (BPRS) were used to measure quality of life and symptomatic recovery, respectively. We used linear mixed modelling to estimate the associations with self-efficacy. Results Data of 142 participants were used. Overall, no difference between IHT and CAU was found with respect to self-efficacy (B = − 0.08, SE = 0.15, p = 0.57), and self-efficacy did not change over the period of 26 weeks (B = − 0.01, SE = 0.12, t (103.95) = − 0.06, p = 0.95). However, differential effects on self-efficacy over time were found for patients with different mental disorders (F(8, 219.33) = 3.75, p < 0.001). Additionally, self-efficacy was strongly associated with symptomatic recovery (total BPRS B = − 0.10, SE = 0.02, p < 0.00) and quality of life (B = 0.14, SE = 0.01, p < 0.001). Conclusions Although self-efficacy was associated with symptomatic recovery and quality of life, IHT does not have a supplementary effect on self-efficacy when compared to CAU. This result raises the question whether, and how, crisis care could be adapted to enhance self-efficacy, keeping in mind the development of self-efficacy in depressive, bipolar, personality, and schizophrenia spectrum and other psychotic disorders. The findings should be considered with some caution. This study lacked sufficient power to test small changes in self-efficacy and some mental disorders had a small sample size. Trial registration This trial is registered at Trialregister.nl, number NL6020.

scholarly journals P121 The efficacy of treatment interventions on fatigue in adolescents with juvenile idiopathic arthritis: a literature review

Rheumatology ◽  
2021 ◽  
Vol 60 (Supplement_1) ◽  
Author(s):  
Sebastian Moshtael ◽  
Sonia Khanom ◽  
Janet E McDonagh
Keyword(s):  
Quality Of Life ◽  
Juvenile Idiopathic Arthritis ◽  
Young People ◽  
Rating Scale ◽  
Outcome Measure ◽  
Therapeutic Interventions ◽  
Current Evidence ◽  
Original Research ◽  
Qualitative Synthesis

Abstract Background/Aims  Fatigue has been reported by young people and professionals alike as a major challenge for people living with juvenile idiopathic arthritis (JIA) . It remains unclear as to how therapeutic interventions impacts on this major symptom. The aim of this review was to determine the current evidence for the effectiveness of therapeutic interventions, non- pharmacological and pharmacological, on improving fatigue in adolescents with JIA. Methods  Three electronic databases (MEDLINE, EMBASE, PsycINFO) were searched from 2000 to Feb 2020; in addition to manual searches. Articles were eligible for inclusion if they (i) were original research papers, (ii) had fatigue as a primary outcome measure (iii) included adolescents (10-25 years) and (iv) were available in the English language. Results  Of the 3,142 records identified, 31 underwent full text assessment and 4 studies were included in the qualitative synthesis including 2 from the Netherlands, 1 Germany, and 1 from Canada. The total number of participants across the 4 studies was 824. Three were randomised controlled trials of nonpharmacological interventions and one was a cohort study from a national drug registry including older adolescents JIA (mean: 19-years). Measurement tools included the Paediatric Quality of life multidimensional fatigue scale, (2 studies), the Checklist for Individual Strength CIS-20 (1 study), a numeric rating scale (1 study). In one study a visual analogue scale for energy level was also utilised. In 2 studies baseline prevalence of fatigue was reported as 60% and 76% respectively. Significant reduction in fatigue was observed in all three nonpharmacological studies. In the registry study, fatigue was noted to be prevalent in spite of the disease being in inactive or minimally active on biologic therapy. In 3 studies the relationship between health-related quality of life and fatigue was highlighted and showed a decreased quality of life in fatigued young people. Conclusion  Fatigue is a significant problem in JIA during adolescence and influences quality of life. Improvements in fatigue with non-pharmacological interventions have been reported. However, at present, data is insufficient to conclusively decide which treatment intervention is most efficacious in treating fatigue in young people with JIA. Fatigue should be considered as an important outcome measure for the management of JIA in future evaluations of interventions. Disclosure  S. Moshtael: None. S. Khanom: None. J.E. McDonagh: None.

Association of Progressive Supranuclear Palsy Rating Scale with Progressive Supranuclear Palsy Quality of Life Scale

2021 ◽  
pp. 1-8
Author(s):  
Alexander Pantelyat ◽  
Lenora Higginbotham ◽  
Liana Rosenthal ◽  
Diane Lanham ◽  
Vanessa Nesspor ◽  
...  
Keyword(s):  
Quality Of Life ◽  
Clinical Trials ◽  
Progressive Supranuclear Palsy ◽  
Rating Scale ◽  
Linear Regression Analysis ◽  
Disease Evolution ◽  
Quality Of Life Scale ◽  
Patient Reported ◽  
Scale Scores

<b><i>Introduction:</i></b> There is growing interest in using patient-reported outcomes as end points in clinical trials, such as the progressive supranuclear palsy quality of life (PSP-QoL) scale. However, this tool has not been widely validated and its correlation with validated motor scales has not been explored. To evaluate the potential utility of using PSP-QoL as an outcome, it is important to examine its relationship with a standard scale used to evaluate neurologic parameters, such as the PSP Rating Scale. <b><i>Methods:</i></b> PSP-QoL and PSP Rating Scale scores were gathered from 60 clinically diagnosed PSP patients, including patients with Richardson syndrome PSP (PSP-RS, <i>n</i> = 43) and those with non-RS PSP variants (<i>n</i> = 17). Linear regression analysis adjusted for age, sex, and disease duration was used to evaluate the cross-sectional relationship between the total and subscale scores of the 2 instruments. <b><i>Results:</i></b> Among 60 PSP patients, there was a significant correlation between total PSP-QoL and PSP Rating Scale scores. The physical and mentation subscales of each instrument also demonstrated significant correlations. Comparisons among PSP subtypes indicated that worsening PSP-QoL Total and Physical subscale scores correlated with worsening PSP Rating Scale gait subscale scores more strongly for the non-RS PSP variants than for PSP-RS. <b><i>Discussion:</i></b> There is a significant association between the total scores and many of the subscale scores of the PSP-QoL and the PSP Rating Scale. Additionally, the relationship between these measures may differ for PSP-RS and non-RS variants. These findings suggest that the PSP-QoL may be useful in clinical trials as a patient-reported outcome measure. Large prospective multicenter studies utilizing the PSP-QoL are necessary to examine its relationship to disease evolution and changes in the PSP Rating Scale.

The Relationship between Restless Legs Syndrome and Quality of Life in Patients with Myasthenia Gravis

2019 ◽  
Vol 81 (3-4) ◽  
pp. 205-208
Author(s):  
Monica F. Ataide ◽  
Carolina da Cunha-Correia ◽  
Katia C.L. Petribú
Keyword(s):  
Quality Of Life ◽  
Myasthenia Gravis ◽  
Restless Legs Syndrome ◽  
Rating Scale ◽  
Cross Sectional Study ◽  
Cross Sectional ◽  
Restless Legs ◽  
Study Population ◽  
Clinical Conditions

Background: Restless legs syndrome (RLS) is characterized for an uncomfortable sensation in legs and an irresistible desire to move them. This disorder has been more recently recognized in patients with myasthenia gravis (MG) and can interfere with the quality of life (QOL). Objectives: The aims of this study are to describe the prevalence of RLS and its severity and influence on the QOL in patients with MG. Method: This was a cross-sectional study conducted from May to June 2016 in Recife, Brazil. A sample of 42 patients was interviewed using a sociodemographic questionnaire, MG QOL questionnaire-15 and The RLS Rating Scale. Results: RLS was present in 47.6% of patients and of these 40.5% met moderate to severe RLS criteria. Patients were 45 years on average (SD ± 14.4) and women represented 57.1% of the study population. Among patients with RSL, the quality-of-life scores were worse (p = 0.010) on average. There was no association of RLS with the duration of MG, use of immunosuppressant or clinical conditions that could mimic the occurrence of RLS. Conclusion: RLS is a prevalent condition in patients with MG, and may be severe enough to negatively impact QOL.

scholarly journals Influence of catastrophizing, anxiety, and depression on in-hospital opioid consumption, pain, and quality of recovery after adult spine surgery

2018 ◽  
Vol 28 (1) ◽  
pp. 119-126 ◽  
Author(s):  
Lauren K. Dunn ◽  
Marcel E. Durieux ◽  
Lucas G. Fernández ◽  
Siny Tsang ◽  
Emily E. Smith-Straesser ◽  
...  
Keyword(s):  
Spine Surgery ◽  
Rating Scale ◽  
Psychological Intervention ◽  
Opioid Consumption ◽  
Anxiety And Depression ◽  
Opioid Use ◽  
Quality Of Recovery ◽  
Pain Scores ◽  
The Impact

OBJECTIVEPerception of perioperative pain is influenced by various psychological factors. The aim of this study was to determine the impact of catastrophizing, anxiety, and depression on in-hospital opioid consumption, pain scores, and quality of recovery in adults who underwent spine surgery.METHODSPatients undergoing spine surgery were enrolled in this study, and the preoperatively completed questionnaires included the verbal rating scale (VRS), Pain Catastrophizing Scale (PCS), Hospital Anxiety and Depression Scale (HADS), and Oswestry Disability Index (ODI). Quality of recovery was assessed using the 40-item Quality of Recovery questionnaire (QoR40). Opioid consumption and pain scores according to the VRS were recorded daily until discharge.RESULTSOne hundred thirty-nine patients were recruited for the study, and 101 completed the QoR40 assessment postoperatively. Patients with higher catastrophizing scores were more likely to have higher maximum pain scores postoperatively (estimate: 0.03, SE: 0.01, p = 0.02), without increased opioid use (estimate: 0.44, SE: 0.27, p = 0.11). Preoperative anxiety (estimate: 1.18, SE: 0.65, p = 0.07) and depression scores (estimate: 1.06, SE: 0.71, p = 0.14) did not correlate with increased postoperative opioid use; however, patients with higher preoperative depression scores had lower quality of recovery after surgery (estimate: −1.9, SE: 0.56, p < 0.001).CONCLUSIONSCatastrophizing, anxiety, and depression play important roles in modulating postoperative pain. Preoperative evaluation of these factors, utilizing a validated tool, helps to identify patients at risk. This might allow for earlier psychological intervention that could reduce pain severity and improve the quality of recovery.

Sign in / Sign up

Export Citation Format

Share Document