Practical Advice Regarding the Reliability of the Patient Educational Materials Assessment Tool for Health Educators

The quality of patient education materials is an important issue for health educators, clinicians, and community health workers. We describe a challenge achieving reliable scores between coders when using the Patient Educational Materials Assessment Tool (PEMAT) to evaluate farmworker health materials in spring 2020. Four coders were unable to achieve reliability after three attempts at coding calibration. Further investigation identified improvements to the PEMAT codebook and evidence of the difficulty of achieving traditional interrater reliability in the form of Krippendorff’s alpha. Our solution was to use multiple raters and average ratings to achieve an acceptable score with an intraclass correlation coefficient. Practitioners using the PEMAT to evaluate materials should consider averaging the scores of multiple raters as PEMAT results otherwise may be highly sensitive to who is doing the rating. Not doing so may inadvertently result in the use of suboptimal patient education materials.

Download Full-text

CODEM Instrument

GeroPsych ◽

10.1024/1662-9647/a000100 ◽

2014 ◽

Vol 27 (1) ◽

pp. 23-31 ◽

Cited By ~ 4

Author(s):

Anne Kuemmel (This author contributed eq ◽

Julia Haberstroh (This author contributed ◽

Johannes Pantel

Keyword(s):

Convergent Validity ◽

Interrater Reliability ◽

Discriminant Validity ◽

Assessment Tool ◽

Intraclass Correlation ◽

Well Being ◽

Communication Behavior ◽

People With Dementia ◽

Pearson's R ◽

Pearson’S R

Communication and communication behaviors in situational contexts are essential conditions for well-being and quality of life in people with dementia. Measuring methods, however, are limited. The CODEM instrument, a standardized observational communication behavior assessment tool, was developed and evaluated on the basis of the current state of research in dementia care and social-communicative behavior. Initially, interrater reliability was examined by means of videoratings (N = 10 people with dementia). Thereupon, six caregivers in six German nursing homes observed 69 residents suffering from dementia and used CODEM to rate their communication behavior. The interrater reliability of CODEM was excellent (mean κ = .79; intraclass correlation = .91). Statistical analysis indicated that CODEM had excellent internal consistency (Cronbach’s α = .95). CODEM also showed excellent convergent validity (Pearson’s R = .88) as well as discriminant validity (Pearson’s R = .63). Confirmatory factor analysis verified the two-factor solution of verbal/content aspects and nonverbal/relationship aspects. With regard to the severity of the disease, the content and relational aspects of communication exhibited different trends. CODEM proved to be a reliable, valid, and sensitive assessment tool for examining communication behavior in the field of dementia. CODEM also provides researchers a feasible examination tool for measuring effects of psychosocial intervention studies that strive to improve communication behavior and well-being in dementia.

Download Full-text

Quality Assessment of Online Patient Information for Cosmetic Botulinum Toxin

Aesthetic Surgery Journal ◽

10.1093/asj/sjaa168 ◽

2020 ◽

Vol 40 (11) ◽

pp. NP636-NP642 ◽

Cited By ~ 1

Author(s):

Eric Barbarite ◽

David Shaye ◽

Samuel Oyer ◽

Linda N Lee

Keyword(s):

Botulinum Toxin ◽

Patient Education ◽

Online Education ◽

Grade Level ◽

Assessment Tool ◽

Botulinum Toxin Injection ◽

Reading Level ◽

Accuracy Score ◽

Flesch Reading Ease

Abstract Background In an era of widespread Internet access, patients increasingly look online for health information. Given the frequency with which cosmetic botulinum toxin injection is performed, there is a need to provide patients with high-quality information about this procedure. Objectives The aim of this study was to examine the quality of printed online education materials (POEMs) about cosmetic botulinum toxin. Methods An Internet search was performed to identify 32 websites of various authorship types. Materials were evaluated for accuracy and inclusion of key content points. Readability was measured by Flesch Reading Ease and Flesch-Kincaid Grade Level. Understandability and actionability were assessed with the Patient Education Materials Assessment Tool for Printed Materials. The effect of authorship was measured by undertaking analysis of variance between groups. Results The mean [standard deviation] accuracy score among all POEMs was 4.2 [0.7], which represents an accuracy of 76% to 99%. Mean comprehensiveness was 47.0% [16.4%]. Mean Flesch-Kincaid Grade Level and Flesch Reading Ease scores were 10.7 [2.1] and 47.9 [10.0], respectively. Mean understandability and actionability were 62.8% [18.8%] and 36.2% [26.5%], respectively. There were no significant differences between accuracy (P > 0.2), comprehensiveness (P > 0.5), readability (P > 0.1), understandability (P > 0.3), or actionability (P > 0.2) by authorship. Conclusions There is wide variability in the quality of cosmetic botulinum toxin POEMs regardless of authorship type. The majority of materials are written above the recommended reading level and fail to include important content points. It is critical that providers take an active role in the evaluation and endorsement of online patient education materials.

Download Full-text

Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT)

Patient Education and Counseling ◽

10.1016/j.pec.2017.09.003 ◽

2018 ◽

Vol 101 (3) ◽

pp. 490-496 ◽

Cited By ~ 5

Author(s):

Julia Vishnevetsky ◽

Chasity Burrows Walters ◽

Kay See Tan

Keyword(s):

Patient Education ◽

Interrater Reliability ◽

Assessment Tool

Download Full-text

Ghent Developmental Balance Test: A New Tool to Evaluate Balance Performance in Toddlers and Preschool Children

Physical Therapy ◽

10.2522/ptj.20110265 ◽

2012 ◽

Vol 92 (6) ◽

pp. 841-852 ◽

Cited By ~ 9

Author(s):

Alexandra De Kegel ◽

Tina Baetens ◽

Wim Peersman ◽

Leen Maes ◽

Ingeborg Dhooge ◽

...

Keyword(s):

Interrater Reliability ◽

Assessment Tool ◽

Intraclass Correlation ◽

Correlation Coefficients ◽

Control Group ◽

Clinical Group ◽

Balance Performance ◽

Balance Test ◽

Movement Assessment ◽

Z Scores

Background Balance is a fundamental component of movement. Early identification of balance problems is important to plan early intervention. The Ghent Developmental Balance Test (GDBT) is a new assessment tool designed to monitor balance from the initiation of independent walking to 5 years of age. Objective The purpose of this study was to establish the psychometric characteristics of the GDBT. Methods To evaluate test-retest reliability, 144 children were tested twice on the GDBT by the same examiner, and to evaluate interrater reliability, videotaped GDBT sessions of 22 children were rated by 3 different raters. To evaluate the known-group validity of GDBT scores, z scores on the GDBT were compared between a clinical group (n=20) and a matched control group (n=20). Concurrent validity of GDBT scores with the subscale standardized scores of the Movement Assessment Battery for Children–Second Edition (M-ABC-2), the Peabody Developmental Motor Scales–Second Edition (PDMS-2), and the balance subscale of the Bruininks-Oseretsky Test–Second Edition (BOT-2) was evaluated in a combined group of the 20 children from the clinical group and 74 children who were developing typically. Results Test-retest and interrater reliability were excellent for the GDBT total scores, with intraclass correlation coefficients of .99 and .98, standard error of measurement values of 0.21 and 0.78, and small minimal detectable differences of 0.58 and 2.08, respectively. The GDBT was able to distinguish between the clinical group and the control group (t38=5.456, P<.001). Pearson correlations between the z scores on GDBT and the standardized scores of specific balance subscales of the M-ABC-2, PDMS-2, and BOT-2 were moderate to high, whereas correlations with subscales measuring constructs other than balance were low. Conclusions The GDBT is a reliable and valid clinical assessment tool for the evaluation of balance in toddlers and preschool-aged children.

Download Full-text

Research Implementing the Flinders Model of Self-management Support with Aboriginal People who have Diabetes: Findings from a Pilot Study

Australian Journal of Primary Health ◽

10.1071/py08009 ◽

2008 ◽

Vol 14 (1) ◽

pp. 66 ◽

Cited By ~ 38

Author(s):

Malcolm W. Battersby ◽

Jackie Ah Kit ◽

Colleen Prideaux ◽

Peter W. Harvey ◽

James P. Collins ◽

...

Keyword(s):

Quality Of Life ◽

Assessment Tool ◽

Aboriginal People ◽

Health Workers ◽

South Australia ◽

Management Program ◽

Self Management ◽

Management Support ◽

Structured Interviews

A pilot program for Aboriginal people with diabetes on Eyre Peninsula, South Australia, aimed to test the acceptability and impact of using the Flinders model of self-management care planing to improve patient self-management. A community development approach was used to conduct a twelve-month demonstration project. Aboriginal health workers (AHWs) conducted patient-centred, self-management assessment and care planning. Impacts were measured by patient-completed diabetes self-management assessment tool, goal achievement, quality of life and clinical measures at baseline and 12 months. Impact and acceptability were also assessed by semi-structured interviews and focus groups of AHWs. Sixty Aboriginal people with type 2 diabetes stated their main problems as family and social dysfunction, access to services, nutrition and exercise. Problems improved by 12% and goals by 26%, while quality of life scores showed no significant change. Self-management scores improved in five of six domains. Mean HbA1c reduced from 8.74-8.09 and mean blood pressure was unchanged. AHWs found the process acceptable and appropriate for them and their patients. It was concluded that a diabetes self-management program provided by AHWs is acceptable, improves self-management and is seen to be useful by Aboriginal communities. Barriers include lack of preventative health services, social problems and time pressure on staff. Enablers include community concern regarding the prevalence and mortality associated with diabetes.

Download Full-text

Evaluating the performance of South African primary care: a cross-sectional descriptive survey

South African Family Practice ◽

10.4102/safp.v61i3.4981 ◽

2019 ◽

Vol 61 (3) ◽

pp. 77

Author(s):

Graham Bresick ◽

Klaus B. Von Pressentin ◽

Robert Mash

Keyword(s):

South Africa ◽

Primary Care ◽

South African ◽

Assessment Tool ◽

Health Workers ◽

Cross Sectional Survey ◽

Cross Sectional ◽

Care Assessment ◽

First Contact

Introduction: In 2018 governments reaffirmed their commitment to implementing primary health care (PHC) in the Astana Declaration. South Africa has introduced a number of health reforms to strengthen PHC and enable universal health coverage (UHC). UHC requires access to quality primary care and progress needs to be measured. This study aimed to evaluate the quality of South African primary care using the Primary Care Assessment Tool (PCAT).Methods: A descriptive cross-sectional survey used data derived from a previous analytical observational study. Data from 413 patients, 136 health workers and 55 managers were analysed from 30 community health centres across four provinces of South Africa. Scores were obtained for 10 key domains and an overall primary care score. Scores were compared in terms of respondents, provinces and monthly headcount.Results: Patients rated first contact accessibility, ongoing care and community orientation as the poorest performing elements ( 50% scoring as ‘acceptable to good’); first contact utilisation, informational coordination and family-centredness as weaker elements ( 66% scoring as ‘acceptable to good’); and comprehensiveness, coordination, cultural competency and availability of the PHC team as stronger aspects of primary care (≥ 66% or more scoring as ‘acceptable or good’). Managers and providers were generally much more positive about the performance of PHC.Conclusion: Gaps exist between PHC users’ experience of care and what PHC staff believe they provide. Priorities to strengthen South African primary care include improving access, informational and relational continuity of care, and ensuring the implementation of community-orientated primary care. The PCAT is a useful tool to measure quality of primary care and progress with UHC.

Download Full-text

Grounding the Connection Between Psyche and Soma: Creating a Reliable Observation Tool for Grounding Assessment in an Adult Population

Frontiers in Psychology ◽

10.3389/fpsyg.2021.621958 ◽

2021 ◽

Vol 12 ◽

Author(s):

Einat Shuper Engelhard ◽

Michal Pitluk ◽

Michal Elboim-Gabyzon

Keyword(s):

Assessment Tool ◽

Emotional Experience ◽

Adult Population ◽

Intraclass Correlation ◽

The Body ◽

Dance Movement ◽

Theoretical Understanding ◽

Foot Placement ◽

Therapeutic Goals

The concept of grounding is accepted and common among dance movement therapists and body psychotherapists. It expresses a stable physical and emotional presence – “supported by the ground.” The assumption is that embodied emotional knowledge is expressed through the manner of physical holding and in the emotional experience in the world. However, along with the clinical use of the term, an empirical tool for examining grounding is lacking. The goal of the study was to examine the reliability and validity of an observation tool for assessing the quality of grounding, the Grounding Assessment Tool (GAT), which was created for the present study on the basis of theory, research, and clinical knowledge in the field. Forty three adult participants (age, M = 28.2 years, SD = 8.54) were recruited for an experimental and controlled session, the session included guided movement for approximately 10 min. The movement was recorded on video. The quality of the movement was rated by two raters and was scored using the GAT. The study findings indicated that the GAT is a reliable and valid tool – with good internal consistency (α = 0.850) and high interrater and intrarater reliability (Kendall’s ’range from 0.789 to 0.973 and intraclass correlation coefficient range from 0.967 to 1.00, respectively). The exploratory factor analysis showed that four factors are involved in the assessment of grounding quality: fluid and rhythmic movement, emotional expression in movement, pattern of foot placement, and lack of stability and weightiness. The results of this study expand the theoretical understanding of the concept of grounding. They contribute to the understanding of the benefits of body focus, dance and movement in psychotherapy and to validating body psychotherapy and dance movement therapy (DMT). The existence of a reliable and valid tool is essential for assessment and diagnostic processes, for formulating therapeutic goals focused on the body, and for examining their effectiveness.

Download Full-text

Evaluating the quality of educational materials about schizophrenia

Psychiatric Bulletin ◽

10.1192/pb.26.3.96 ◽

2002 ◽

Vol 26 (3) ◽

pp. 96-98 ◽

Cited By ~ 2

Author(s):

C. Currie ◽

P. Di Mambro ◽

A. Joice ◽

R. Mcglip ◽

M. O'Neill ◽

...

Keyword(s):

Mental Health Professionals ◽

Health Professionals ◽

Interrater Reliability ◽

Poor Quality ◽

Clinical Implications ◽

Educational Materials ◽

Educational Package ◽

Rating Instrument ◽

Materials Used

Aims and MethodTo identify and evaluate the reliability and quality of educational materials provided to individuals with schizophrenia and their carers. Materials used by mental health professionals working in community and in-patient settings were collated. Two independent raters used the ‘Discern’ questionnaire to assess the publications.ResultsFifteen documents were identified, but only 11 were suitable for evaluation. Interrater reliability of ratings using the Discern tool was highly significant. No educational package scored maximum marks, but four scored in the good quality range. About a fifth of the materials in widespread use were assessed as of poor quality.Clinical ImplicationsThe use of a rating instrument to assess the quality of educational publications appears to be a reliable and acceptable way to identify the strengths and weaknesses of widely available materials. Staff can use these data to assess the quality of their preferred materials against other publications and make an informed selection.

Download Full-text

Search Trends and Quality of Online Resources Regarding Thyroidectomy

Otolaryngology ◽

10.1177/0194599820969154 ◽

2020 ◽

pp. 019459982096915

Author(s):

Lena W. Chen ◽

Vandra Chatrice Harris ◽

Justin Lee Jia ◽

Deborah Xingchun Xie ◽

Ralph Patrick Tufano ◽

...

Keyword(s):

Grade Level ◽

Interrater Reliability ◽

Assessment Tool ◽

Neck Surgery ◽

Reading Level ◽

Online Resources ◽

Cross Sectional ◽

Flesch Reading Ease ◽

Google Search

Objective Thyroidectomy is one of the most common procedures performed in head and neck surgery. The quality of online resources for thyroidectomy is unknown. We aim to evaluate search trends and online resource quality regarding thyroidectomy. Study Design Cross-sectional analysis. Setting Websites appearing on Google search. Methods The first 30 Google websites for thyroidectomy were reviewed, excluding research, video, and restricted sites. Search patterns were obtained with Google Trends. Quality was measured by readability (Flesch Reading Ease and Flesch-Kinkaid Grade Level), understandability and actionability (Patient Education Materials Assessment Tool), and clinical practice guideline (CPG) compatibility. Fleiss kappa interrater reliability analysis was performed for 2 raters. Results Twenty-one sites were evaluated. Search popularity for thyroidectomy has increased since 2004. Median reading ease was 42.2 (range, 15.4-62.7) on a scale from 1 to 100, with 100 indicating maximum readability. Median reading grade level was 12 (range, 7-16). Thyroidectomy resources were poorly understandable (median, 66%; range, 21%-88%) and actionable (median, 10%; range, 0%-60%). Median CPG compatibility was 4 out of 5 (range, 0-5). Interrater reliability ranged from substantial to moderate for understandability (0.78), actionability (0.57), and CPG compatibility (0.58), with P < .05 for all results. Conclusion Online resources about thyroidectomy vary in quality and reliability and are written at grade levels above the average reading level of the public. Providers should be aware of existing resources and work to create education resources that meet universal health literacy guidelines. The framework provided in this article may also serve as a guide and provide tangible steps that providers can take to help patients access care.

Download Full-text

Colonoscopy videos on YouTube: Are they a good source of patient education?

Endoscopy International Open ◽

10.1055/a-1122-8566 ◽

2020 ◽

Vol 08 (05) ◽

pp. E598-E606

Author(s):

Dhruvil Radadiya ◽

Alexei Gonzalez-Estrada ◽

Jorge Emilio Lira-Vera ◽

Katia Lizarrga-Torres ◽

Shayan Sinha Mahapatra ◽

...

Keyword(s):

Patient Education ◽

Medical Information ◽

Intraclass Correlation ◽

Healthcare Providers ◽

Quality Score ◽

Medical Community ◽

Video Sharing ◽

Professional Societies ◽

Global Quality

Abstract Background and study aims Colonoscopy is an effective tool to prevent colorectal cancer. Social media has emerged as a source of medical information for patients.YouTube (a video sharing website) is the most popular video informative source. Therefore, we aimed to assess the educational quality of colonoscopy videos available on YouTube. Methods We performed a YouTube search using the keyword “colonoscopy” yielded 429 videos, of which 255 met the inclusion criteria. Colonoscopy Data Quality Score (C-DQS) was created to rate the quality of the videos (–10 to +40 points) based on a colonoscopy education video available on the Ameican Society of Gastrointestinal Endoscopy (ASGE) website. Each video was scored by six blinded reviewers independently using C-DQS. The Global Quality Score (GQS) was used for score validation. The intraclass correlation coefficient (ICC) was used to assess the similarity of the scores among reviewers. Results Professional societies had the highest number of videos (44.3 %). Videos from professional societies (6.94) and media (6.87) had significantly higher mean C-DQS compared to those from alternative medicine providers (1.19), companies (1.16), and patients (2.60) (P < 0.05). Mean C-DQS score of videos from healthcare providers (4.40) was not statistically different than other sources. There was a high degree of agreement among reviewers for the videos from all sources (ICC = 0.934; P < 0.001). Discussion YouTube videos are a poor source of information on colonoscopy. Professional societies and media are better sources of quality information for patient education on colonoscopy. The medical community may need to engage actively in enriching the quality of educational material available on YouTube.

Download Full-text