Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT)

The quality of patient education materials is an important issue for health educators, clinicians, and community health workers. We describe a challenge achieving reliable scores between coders when using the Patient Educational Materials Assessment Tool (PEMAT) to evaluate farmworker health materials in spring 2020. Four coders were unable to achieve reliability after three attempts at coding calibration. Further investigation identified improvements to the PEMAT codebook and evidence of the difficulty of achieving traditional interrater reliability in the form of Krippendorff’s alpha. Our solution was to use multiple raters and average ratings to achieve an acceptable score with an intraclass correlation coefficient. Practitioners using the PEMAT to evaluate materials should consider averaging the scores of multiple raters as PEMAT results otherwise may be highly sensitive to who is doing the rating. Not doing so may inadvertently result in the use of suboptimal patient education materials.

Download Full-text

CODEM Instrument

GeroPsych ◽

10.1024/1662-9647/a000100 ◽

2014 ◽

Vol 27 (1) ◽

pp. 23-31 ◽

Cited By ~ 4

Author(s):

Anne Kuemmel (This author contributed eq ◽

Julia Haberstroh (This author contributed ◽

Johannes Pantel

Keyword(s):

Convergent Validity ◽

Interrater Reliability ◽

Discriminant Validity ◽

Assessment Tool ◽

Intraclass Correlation ◽

Well Being ◽

Communication Behavior ◽

People With Dementia ◽

Pearson's R ◽

Pearson’S R

Communication and communication behaviors in situational contexts are essential conditions for well-being and quality of life in people with dementia. Measuring methods, however, are limited. The CODEM instrument, a standardized observational communication behavior assessment tool, was developed and evaluated on the basis of the current state of research in dementia care and social-communicative behavior. Initially, interrater reliability was examined by means of videoratings (N = 10 people with dementia). Thereupon, six caregivers in six German nursing homes observed 69 residents suffering from dementia and used CODEM to rate their communication behavior. The interrater reliability of CODEM was excellent (mean κ = .79; intraclass correlation = .91). Statistical analysis indicated that CODEM had excellent internal consistency (Cronbach’s α = .95). CODEM also showed excellent convergent validity (Pearson’s R = .88) as well as discriminant validity (Pearson’s R = .63). Confirmatory factor analysis verified the two-factor solution of verbal/content aspects and nonverbal/relationship aspects. With regard to the severity of the disease, the content and relational aspects of communication exhibited different trends. CODEM proved to be a reliable, valid, and sensitive assessment tool for examining communication behavior in the field of dementia. CODEM also provides researchers a feasible examination tool for measuring effects of psychosocial intervention studies that strive to improve communication behavior and well-being in dementia.

Download Full-text

Reliability and Validity Analyses of the Youth Level of Service/Case Management Inventory

Criminal Justice and Behavior ◽

10.1177/0093854804274373 ◽

2005 ◽

Vol 32 (3) ◽

pp. 329-344 ◽

Cited By ~ 103

Author(s):

Fred Schmidt ◽

Robert D. Hoge ◽

Lezlie Gomes

Keyword(s):

Juvenile Offenders ◽

Case Management ◽

Interrater Reliability ◽

Assessment Tool ◽

Reliability And Validity ◽

Level Of Service ◽

Risk Level ◽

Criminogenic Needs ◽

Health Assessments ◽

Males And Females

The Youth Level of Service/Case Management Inventory (YLS/CMI) is a structured assessment tool designed to facilitate the effective intervention and rehabilitation of juvenile offenders by assessing each youth’s risk level and criminogenic needs. The present study examined the YLS/CMI’s reliability and validity in a sample of 107 juvenile offenders who were court-referred for mental health assessments. Results demonstrated the YLS/CMI’s internal consistency and interrater reliability. Moreover, the instrument’s predictive validity was substantiated on a number of recidivism measures for both males and females. Limitations of the current findings are discussed.

Download Full-text

Understandability, actionability and suitability of educational videos on dietary management for hypertension

Health Education Journal ◽

10.1177/00178969211072275 ◽

2022 ◽

pp. 001789692110722

Author(s):

Mohd Ramadan Ab Hamid ◽

Nur Dalia Binti Mohd Yusof ◽

Siti Sabariah Buhari

Keyword(s):

Patient Education ◽

Healthcare Professionals ◽

Assessment Tool ◽

Dietary Management ◽

Educational Video ◽

Lifestyle Modifications ◽

Educational Tools ◽

Suitability Assessment ◽

Educational Videos ◽

Convenience Sampling

Background: Educational video is a productive means to advocate lifestyle modifications such as changes in the dietary routine. This study assesses the understandability, actionability and suitability of newly developed educational videos aimed at encouraging dietary adjustments among hypertension patients. Method: In all, 183 participants were recruited via convenience sampling and rated the understandability, actionability and suitability of the videos using the Patient Education Materials Assessment Tool for Audio-Visual material (PEMAT-A/V) and the Suitability Assessment of Materials (SAM) scale. Results: Eleven videos were developed from five main topics related to the dietary management of hypertension. Participants agreed that all videos were highly understandable, actionable and suitable for use as educational tools, with scores of more than 85%, 89% and 80%, respectively. Conclusion/Implications: Overall, the newly developed videos gained high scores for understandability, actionability and suitability. This finding reflects positive acceptance of the videos among various healthcare professionals and patients with hypertension.

Download Full-text

Development and Interrater Reliability of a Street Food Stand Assessment Tool

Journal of Nutrition Education and Behavior ◽

10.1016/j.jneb.2021.06.015 ◽

2021 ◽

Author(s):

Jose B. Rosales Chávez ◽

Megan Jehn ◽

Rebecca E. Lee ◽

Punam Ohri-Vachaspati ◽

Meg Bruening

Keyword(s):

Interrater Reliability ◽

Assessment Tool ◽

Street Food

Download Full-text

Patient education materials assessment tool for laryngectomy health information

Head & Neck ◽

10.1002/hed.24891 ◽

2017 ◽

Vol 39 (11) ◽

pp. 2256-2263 ◽

Cited By ~ 7

Author(s):

Kevin Wong ◽

Amir Gilad ◽

Michael B. Cohen ◽

Diana N. Kirke ◽

Scharukh M. Jalisi

Keyword(s):

Patient Education ◽

Health Information ◽

Assessment Tool

Download Full-text

Development of a Diabetes Learning Needs Assessment Tool to Promote an Individualized Predischarge Patient Education Plan

The Journal of Continuing Education in Nursing ◽

10.3928/00220124-20151020-12 ◽

2015 ◽

Vol 46 (11) ◽

pp. 484-486

Author(s):

Ann Marie Hasse ◽

Audrey Davis ◽

Tina Glowatz

Keyword(s):

Patient Education ◽

Needs Assessment ◽

Assessment Tool ◽

Learning Needs ◽

Education Plan

Download Full-text

Quality Assessment of Online Patient Information for Cosmetic Botulinum Toxin

Aesthetic Surgery Journal ◽

10.1093/asj/sjaa168 ◽

2020 ◽

Vol 40 (11) ◽

pp. NP636-NP642 ◽

Cited By ~ 1

Author(s):

Eric Barbarite ◽

David Shaye ◽

Samuel Oyer ◽

Linda N Lee

Keyword(s):

Botulinum Toxin ◽

Patient Education ◽

Online Education ◽

Grade Level ◽

Assessment Tool ◽

Botulinum Toxin Injection ◽

Reading Level ◽

Accuracy Score ◽

Flesch Reading Ease

Abstract Background In an era of widespread Internet access, patients increasingly look online for health information. Given the frequency with which cosmetic botulinum toxin injection is performed, there is a need to provide patients with high-quality information about this procedure. Objectives The aim of this study was to examine the quality of printed online education materials (POEMs) about cosmetic botulinum toxin. Methods An Internet search was performed to identify 32 websites of various authorship types. Materials were evaluated for accuracy and inclusion of key content points. Readability was measured by Flesch Reading Ease and Flesch-Kincaid Grade Level. Understandability and actionability were assessed with the Patient Education Materials Assessment Tool for Printed Materials. The effect of authorship was measured by undertaking analysis of variance between groups. Results The mean [standard deviation] accuracy score among all POEMs was 4.2 [0.7], which represents an accuracy of 76% to 99%. Mean comprehensiveness was 47.0% [16.4%]. Mean Flesch-Kincaid Grade Level and Flesch Reading Ease scores were 10.7 [2.1] and 47.9 [10.0], respectively. Mean understandability and actionability were 62.8% [18.8%] and 36.2% [26.5%], respectively. There were no significant differences between accuracy (P > 0.2), comprehensiveness (P > 0.5), readability (P > 0.1), understandability (P > 0.3), or actionability (P > 0.2) by authorship. Conclusions There is wide variability in the quality of cosmetic botulinum toxin POEMs regardless of authorship type. The majority of materials are written above the recommended reading level and fail to include important content points. It is critical that providers take an active role in the evaluation and endorsement of online patient education materials.

Download Full-text

DIRECTED ACYCLIC GRAPH-BASED PATIENT EDUCATION SYSTEM FOR CARDIOVASCULAR PATIENTS IN TAIWAN

Journal of Mechanics in Medicine and Biology ◽

10.1142/s021951941640011x ◽

2016 ◽

Vol 16 (01) ◽

pp. 1640011

Author(s):

JHEN-LI HUANG ◽

CHIEN-MING CHEN ◽

TUN-WEN PAI ◽

MIN-HUI LIU ◽

CHAO-HUNG WANG

Keyword(s):

Cardiovascular Disease ◽

Patient Education ◽

Education System ◽

Directed Acyclic Graph ◽

Medical Information ◽

Assessment Tool ◽

Acyclic Graph ◽

Video Clips ◽

Cardiovascular Patients ◽

Number Of Patients

Background: Cardiovascular disease is the leading global cause of death and the number of patients is expected to grow yearly. Pertinent patient education materials provide adequate medical information to cardiovascular patients to change their behaviors, improve their health status, and reduce medical costs. However, systematic and patient-oriented education materials for cardiovascular disease are currently not available to the public, especially in Chinese. Methods: A directed acyclic graph (DAG) database structure is proposed for storing all created low-demand medical illustrations and video clips. Further, all designed medical media elements are evaluated according to the Patient Education Materials Assessment Tool (PEMAT) to assure the quality of the medical information content. Each media element is automatically annotated by calculating representative keywords; relationships between different media elements are constructed according to keyword associations. Results: A total of 222 video clips and 45 printable medical documents in Chinese were created in the proposed patient education system. This contains fundamental knowledge of diseases, causes of diseases, medical and surgical treatments, and a prognosis of the medical care for the majority of types of cardiovascular disease. It is the first constructive and intelligent patient education system for cardiovascular patients in Taiwan.

Download Full-text