A Second Step in Development of a Checklist for Screening Risk for Violence in Acute Psychiatric Patients: Evaluation of Interrater Reliability of the Preliminary Scheme 33

2007 ◽  
Vol 101 (3_suppl) ◽  
pp. 1145-1161
Author(s):  
Stål Bjørkly ◽  
Tron A. Moger
Keyword(s):  
Interrater Reliability ◽  
Psychiatric Patients ◽  
Intraclass Correlation ◽  
Second Step ◽  
Psychiatric Admission ◽  
Clinical Vignettes ◽  
Management Scheme ◽  
Content Domain ◽  
The Individual ◽  
Further Development

The Acute Project is a research project conducted on acute psychiatric admission wards in Norway. The objective is to develop and validate a structured, easy-to-use screening checklist for assessment of risk for violence in patients both during their stay in the ward and after discharge. The Preliminary Scheme 33 is a 33-item screening checklist with content domain inspired by the Historical-Clinical-Risk Management Scheme (HCR–20), the Brøset Violence Checklist, and eight risk factors extracted from the literature on risk assessment. The Preliminary Scheme 33 was designed and tested in two steps by a research group which includes the authors. The common aim of both steps was to develop this into a time economical, reliable, and valid checklist. In the first step in 2006 the predictive validity of the individual items was tested. The present work presents results from the second step, a study conducted to assess the interrater reliability of the 33 items. Eight clinicians working in an acute psychiatric unit volunteered to be raters and were trained to score the 33 items on a three-point scale in relation to 15 clinical vignettes, which contained information from 15 acute psychiatric patients' files. Analysis showed high interrater reliability for the total score with an intraclass correlation coefficient (ICC) of .86 (95% CI: 0.74-0.94). However, a substantial proportion of the items had medium to low ICCs. Consequences of this finding for further development of these items into a brief screen are discussed.

Interrater Reliability of the New Sport-Specific Evidence-Based Classification System for Para Va'a

2020 ◽  
Vol 37 (3) ◽  
pp. 241-252
Author(s):  
Johanna S. Rosén ◽  
Victoria L. Goosey-Tolfrey ◽  
Keith Tolfrey ◽  
Anton Arndt ◽  
Anna Bjerkefors
Keyword(s):  
Classification System ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Test Battery ◽  
Evidence Based ◽  
Water Test ◽  
The Individual ◽  
New Evidence ◽  
Test Level ◽  
Total Agreement

The purpose of this study was to examine the interrater reliability of a new evidence-based classification system for Para Va'a. Twelve Para Va'a athletes were classified by three classifier teams each consisting of a medical and a technical classifier. Interrater reliability was assessed by calculating intraclass correlation for the overall class allocation and total scores of trunk, leg, and on-water test batteries and by calculating Fleiss’s kappa and percentage of total agreement in the individual tests of each test battery. All classifier teams agreed with the overall class allocation of all athletes, and all three test batteries exhibited excellent interrater reliability. At a test level, agreement between classifiers was almost perfect in 14 tests, substantial in four tests, moderate in four tests, and fair in one test. The results suggest that a Para Va'a athlete can expect to be allocated to the same class regardless of which classifier team conducts the classification.

Interobserver Reliability Using the Phonetic Level Evaluation With Severely and Profoundly Hearing-Impaired Children

1991 ◽  
Vol 34 (5) ◽  
pp. 989-999 ◽  
Author(s):  
Stephanie Shaw ◽  
Truman E. Coggins
Keyword(s):  
Interrater Reliability ◽  
Interobserver Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Hearing Impaired ◽  
Intraclass Correlation Coefficients ◽  
Assessment Measure ◽  
Impaired Children ◽  
Speech Assessment ◽  
Hearing Impaired Children

This study examines whether observers reliably categorize selected speech production behaviors in hearing-impaired children. A group of experienced speech-language pathologists was trained to score the elicited imitations of 5 profoundly and 5 severely hearing-impaired subjects using the Phonetic Level Evaluation (Ling, 1976). Interrater reliability was calculated using intraclass correlation coefficients. Overall, the magnitude of the coefficients was found to be considerably below what would be accepted in published behavioral research. Failure to obtain acceptably high levels of reliability suggests that the Phonetic Level Evaluation may not yet be an accurate and objective speech assessment measure for hearing-impaired children.

CODEM Instrument

GeroPsych ◽  
2014 ◽  
Vol 27 (1) ◽  
pp. 23-31 ◽  
Author(s):  
Anne Kuemmel (This author contributed eq ◽  
Julia Haberstroh (This author contributed ◽  
Johannes Pantel
Keyword(s):  
Convergent Validity ◽  
Interrater Reliability ◽  
Discriminant Validity ◽  
Assessment Tool ◽  
Intraclass Correlation ◽  
Well Being ◽  
Communication Behavior ◽  
People With Dementia ◽  
Pearson's R ◽  
Pearson’S R

Communication and communication behaviors in situational contexts are essential conditions for well-being and quality of life in people with dementia. Measuring methods, however, are limited. The CODEM instrument, a standardized observational communication behavior assessment tool, was developed and evaluated on the basis of the current state of research in dementia care and social-communicative behavior. Initially, interrater reliability was examined by means of videoratings (N = 10 people with dementia). Thereupon, six caregivers in six German nursing homes observed 69 residents suffering from dementia and used CODEM to rate their communication behavior. The interrater reliability of CODEM was excellent (mean κ = .79; intraclass correlation = .91). Statistical analysis indicated that CODEM had excellent internal consistency (Cronbach’s α = .95). CODEM also showed excellent convergent validity (Pearson’s R = .88) as well as discriminant validity (Pearson’s R = .63). Confirmatory factor analysis verified the two-factor solution of verbal/content aspects and nonverbal/relationship aspects. With regard to the severity of the disease, the content and relational aspects of communication exhibited different trends. CODEM proved to be a reliable, valid, and sensitive assessment tool for examining communication behavior in the field of dementia. CODEM also provides researchers a feasible examination tool for measuring effects of psychosocial intervention studies that strive to improve communication behavior and well-being in dementia.

Reliability of a Modified Medication Appropriateness Index in Community Pharmacies

2003 ◽  
Vol 37 (1) ◽  
pp. 40-46
Author(s):  
Rosemin Kassam ◽  
Linda G Martin ◽  
Karen B Farris ◽  
Homero A Monsanto ◽  
Jean-Marie Kaiser
Keyword(s):  
Community Pharmacy ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Community Setting ◽  
Medication Appropriateness Index ◽  
Paired Samples ◽  
Psychometric Data ◽  
Ambulatory Patients ◽  
Pharmacy Setting ◽  
Medication Appropriateness

Background The medication appropriateness index (MAI) has demonstrated reliability in selected outpatient clinics where medical data were easily accessible from medical charts. However, its use in the community setting where patient data may be limited has not been examined. Objective To evaluate the usefulness of a modified MAI for use in the community pharmacy setting by testing interrater reliability using 3 different rating schemes. Methods Two raters evaluated 160 medications for 32 elderly ambulatory patients. Patient information was acquired using community pharmacist-collected medication histories. A summated MAI score, percent agreement, κ, positive agreement, negative agreement, and intraclass correlation coefficient were calculated for each criterion using 3 scoring schemes. A paired samples t-test (95% CI) was used to test interrater reliability. Results The κ statistics were >0.75 for indication and effectiveness, but good (0.41–0.66) for the remaining criteria using the Hanlon scoring scheme. The intraclass coefficients (0.82, 0.86, 0.87) and overall κ (0.65, 0.66, 0.61) were similar for the 3 schemes. Conclusions This study suggests that the modified MAI has the potential to detect medication appropriateness and inappropriateness in the community pharmacy setting; however, it is not without limitations. Because the MAI has the most clinimetric and psychometric data available, the instrument should be studied further to increase its reliability and generalizability.

A Taxonomy for Psycho-Oncological Intervention Techniques in an Acute Care Hospital in Germany

2021 ◽  
pp. 1-7
Author(s):  
Martina Madl ◽  
Marietta Lieb ◽  
Katharina Schieber ◽  
Tobias Hepp ◽  
Yesim Erim
Keyword(s):  
Acute Care ◽  
Interrater Reliability ◽  
Acute Care Hospital ◽  
Future Research ◽  
Systematic Research ◽  
Care Hospital ◽  
Certification System ◽  
Stepwise Procedure ◽  
Further Development

<b><i>Background:</i></b> Due to the establishment of a nationwide certification system for cancer centers in Germany, the availability of psycho-oncological services for cancer patients has increased substantially. However, little is known about the specific intervention techniques that are applied during sessions in an acute care hospital, since a standardized taxonomy is lacking. With this study, we aimed at the investigation of psycho-oncological intervention techniques and the development of a comprehensive and structured taxonomy thereof. <b><i>Methods:</i></b> In a stepwise procedure, a team of psycho-oncologists generated a data pool of interventions and definitions that were tested in clinical practice during a pilot phase. After an adaptation of intervention techniques, interrater reliability (IRR) was attained by rating 10 previously recorded psycho-oncological sessions. A classification of interventions into superordinate categories was performed, supported by cluster analysis. <b><i>Results:</i></b> Between April and June 2017, 980 psycho-oncological sessions took place. The experts agreed on a total number of 22 intervention techniques. An IRR of 89% for 2 independent psycho-oncological raters was reached. The 22 techniques were classified into 5 superordinate categories. <b><i>Discussion/Conclusion:</i></b> We developed a comprehensive and structured taxonomy of psycho-oncological intervention techniques in an acute care hospital that provides a standardized basis for systematic research and applied care. We expect our work to be continuously subjected to further development: future research should evaluate and expand our taxonomy to other contexts and care settings.

scholarly journals Understanding decisions about antibiotic prescribing in ICU: an application of the Necessity Concerns Framework

2021 ◽  
pp. bmjqs-2020-012479
Author(s):  
Alyssa M Pandolfo ◽  
Robert Horne ◽  
Yogini Jani ◽  
Tom W Reader ◽  
Natalie Bidad ◽  
...  
Keyword(s):  
Focus Groups ◽  
Antibiotic Prescribing ◽  
Negative Consequences ◽  
Out Of Hours ◽  
Team Members ◽  
Clinical Vignettes ◽  
Molecular Microbiology ◽  
Semistructured Interviews ◽  
Decision Making Processes ◽  
The Individual

BackgroundAntibiotics are extensively prescribed in intensive care units (ICUs), yet little is known about how antibiotic-related decisions are made in this setting. We explored how beliefs, perceptions and contextual factors influenced ICU clinicians’ antibiotic prescribing.MethodsWe conducted 4 focus groups and 34 semistructured interviews with clinicians involved in antibiotic prescribing in four English ICUs. Focus groups explored factors influencing prescribing, whereas interviews examined decision-making processes using two clinical vignettes. Data were analysed using thematic analysis, applying the Necessity Concerns Framework.ResultsClinicians’ antibiotic decisions were influenced by their judgement of the necessity for prescribing/not prescribing, relative to their concerns about potential adverse consequences. Antibiotic necessity perceptions were strongly influenced by beliefs that antibiotics would protect patients from deterioration and themselves from the ethical and legal consequences of undertreatment. Clinicians also reported concerns about prescribing antibiotics. These generally centred on antimicrobial resistance; however, protecting the individual patient was prioritised over these societal concerns. Few participants identified antibiotic toxicity concerns as a key influencer. Clinical uncertainty often complicated balancing antibiotic necessity against concerns. Decisions to start or continue antibiotics often represented ‘erring on the side of caution’ as a protective response in uncertainty. This approach was reinforced by previous experiences of negative consequences (‘being burnt’) which motivated prescribing ‘just in case’ of an infection. Prescribing decisions were also context-dependent, exemplified by a lower perceived threshold to prescribe antibiotics out-of-hours, input from external team members and local prescribing norms.ConclusionEfforts to improve antibiotic stewardship should consider clinicians’ desire to protect with a prescription. Rapid molecular microbiology, with appropriate communication, may diminish clinicians’ fears of not prescribing or of using narrower-spectrum antibiotics.

scholarly journals Using Differential Item Functioning to Test for Interrater Reliability in Constructed Response Items

2020 ◽  
Vol 80 (4) ◽  
pp. 808-820
Author(s):  
Cindy M. Walker ◽  
Sakine Göçer Şahin
Keyword(s):  
Differential Item Functioning ◽  
Interrater Reliability ◽  
Rating Scales ◽  
Rating Scale ◽  
Intraclass Correlation ◽  
Kappa Statistic ◽  
Promising Alternative ◽  
Constructed Response ◽  
Polytomous Item ◽  
Item Functioning

The purpose of this study was to investigate a new way of evaluating interrater reliability that can allow one to determine if two raters differ with respect to their rating on a polytomous rating scale or constructed response item. Specifically, differential item functioning (DIF) analyses were used to assess interrater reliability and compared with traditional interrater reliability measures. Three different procedures that can be used as measures of interrater reliability were compared: (1) intraclass correlation coefficient (ICC), (2) Cohen’s kappa statistic, and (3) DIF statistic obtained from Poly-SIBTEST. The results of this investigation indicated that DIF procedures appear to be a promising alternative to assess the interrater reliability of constructed response items, or other polytomous types of items, such as rating scales. Furthermore, using DIF to assess interrater reliability does not require a fully crossed design and allows one to determine if a rater is either more severe, or more lenient, in their scoring of each individual polytomous item on a test or rating scale.

Development and Initial Validation of a Project-Based Rubric to Assess the Systems-Based Practice Competency of Residents in the Clinical Chemistry Rotation of a Pathology Residency

2014 ◽  
Vol 138 (6) ◽  
pp. 809-813
Author(s):  
Carolyn R. Vitek ◽  
Jane C. Dale ◽  
Henry A. Homburger ◽  
Sandra C. Bryant ◽  
Amy K. Saenger ◽  
...  
Keyword(s):  
Critical Thinking ◽  
Interrater Reliability ◽  
Clinical Chemistry ◽  
Core Competencies ◽  
Intraclass Correlation ◽  
Reliability And Validity ◽  
Correlation Coefficients ◽  
Thinking Skills ◽  
Project Evaluation ◽  
Critical Thinking Skills

Context.— Systems-based practice (SBP) is 1 of 6 core competencies required in all resident training programs accredited by the Accreditation Council for Graduate Medical Education. Reliable methods of assessing resident competency in SBP have not been described in the medical literature. Objective.— To develop and validate an analytic grading rubric to assess pathology residents' analyses of SBP problems in clinical chemistry. Design.— Residents were assigned an SBP project based upon unmet clinical needs in the clinical chemistry laboratories. Using an iterative method, we created an analytic grading rubric based on critical thinking principles. Four faculty raters used the SBP project evaluation rubric to independently grade 11 residents' projects during their clinical chemistry rotations. Interrater reliability and Cronbach α were calculated to determine the reliability and validity of the rubric. Project mean scores and range were also assessed to determine whether the rubric differentiated resident critical thinking skills related to the SBP projects. Results.— Overall project scores ranged from 6.56 to 16.50 out of a possible 20 points. Cronbach α ranged from 0.91 to 0.96, indicating that the 4 rubric categories were internally consistent without significant overlap. Intraclass correlation coefficients ranged from 0.63 to 0.81, indicating moderate to strong interrater reliability. Conclusions.— We report development and statistical analysis of a novel SBP project evaluation rubric. The results indicate the rubric can be used to reliably assess pathology residents' critical thinking skills in SBP.

Evaluation of Functionality in Families with Depressed Patient by Using the FFS - Poster Prezentation

2009 ◽  
Vol 24 (S1) ◽  
pp. 1-1
Author(s):  
T. Maria-Silvia
Keyword(s):  
Major Depression ◽  
Depressive Disorder ◽  
Positive Affect ◽  
Psychiatric Patients ◽  
Normal Families ◽  
Control Group ◽  
World Population ◽  
Dsm Iv ◽  
The Individual ◽  
Family Functionality

Depression is a disorder of representation and regulation of mood and emotion; it affects 5% of world population, in a year. Unlike normal loss and sadness feelings, major depression is persistant and it interferes significantly with thoughts, behaviour, emotions, activity and health of the individual. If untreated, depression can lead to suicide. Using family therapy in treating psychiatric patients is a must due to the significance that a family holds in individual and society life.Objective:Assesing family functionality in families with a member diagnosed according to DSM IV TR with depressive disorder; depression intensity was assesed with HDRS.Methods:A sample of 3o families (71 members); FFS assesses the most important and consistent five functioning areas: positive affect, comunication, conflicts, worries and rituals.Results:Values obtained in each of the 40 questions of the scale can give information on variables affecting the increase or decrease in subscales values. Positive affect 35,07, communication 37, conflicts 15,11, worries 40,77, rituals 45,03. The reuslts were compared to those obtained by assessin normal families from a control group of 132 families (323 members).Conclusions:Differences were noticed. Values obtained in our study represent the standard of functioning of families with a depressed member.

Sign in / Sign up

Export Citation Format

Share Document