Measuring the Quality of Clinical Skills Mobile Apps for Student Learning: Systematic Search, Analysis, and Comparison of Two Measurement Scales

Background Mobile apps are widely used in health professions, which increases the need for simple methods to determine the quality of apps. In particular, teachers need the ability to curate high-quality mobile apps for student learning. Objective This study aims to systematically search for and evaluate the quality of clinical skills mobile apps as learning tools. The quality of apps meeting the specified criteria was evaluated using two measures—the widely used Mobile App Rating Scale (MARS), which measures general app quality, and the Mobile App Rubric for Learning (MARuL), a recently developed instrument that measures the value of apps for student learning—to assess whether MARuL is more effective than MARS in identifying high-quality apps for learning. Methods Two mobile app stores were systematically searched using clinical skills terms commonly found in medical education and apps meeting the criteria identified using an approach based on PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. A total of 9 apps were identified during the screening process. The apps were rated independently by 2 reviewers using MARS and MARuL. Results The intraclass correlation coefficients (ICCs) for the 2 raters using MARS and MARuL were the same (MARS ICC [two-way]=0.68; P<.001 and MARuL ICC [two-way]=0.68; P<.001). Of the 9 apps, Geeky Medics-OSCE revision (MARS Android=3.74; MARS iOS=3.68; MARuL Android=75; and MARuL iOS=73) and OSCE PASS: Medical Revision (MARS Android=3.79; MARS iOS=3.71; MARuL Android=69; and MARuL iOS=73) scored highly on both measures of app quality and for both Android and iOS. Both measures also showed agreement for the lowest rated app, Patient Education Institute (MARS Android=2.21; MARS iOS=2.11; MARuL Android=18; and MARuL iOS=21.5), which had the lowest scores in all categories except information (MARS) and professional (MARuL) in both operating systems. MARS and MARuL were both able to differentiate between the highest and lowest quality apps; however, MARuL was better able to differentiate apps based on teaching and learning quality. Conclusions This systematic search and rating of clinical skills apps for learning found that the quality of apps was highly variable. However, 2 apps—Geeky Medics-OSCE revision and OSCE PASS: Medical Revision—rated highly for both versions and with both quality measures. MARS and MARuL showed similar abilities to differentiate the quality of the 9 apps. However, MARuL’s incorporation of teaching and learning elements as part of a multidimensional measure of quality may make it more appropriate for use with apps focused on teaching and learning, whereas MARS’s more general rating of quality may be more appropriate for health apps targeting a general health audience. Ratings of the 9 apps by both measures also highlighted the variable quality of clinical skills mobile apps for learning.

Download Full-text

Measuring the Quality of Clinical Skills Mobile Apps for Student Learning: Systematic Search, Analysis, and Comparison of Two Measurement Scales (Preprint)

10.2196/preprints.25377 ◽

2020 ◽

Author(s):

Tehmina Gladman ◽

Grace Tylee ◽

Steve Gallagher ◽

Jonathan Mair ◽

Rebecca Grainger

Keyword(s):

Student Learning ◽

Teaching And Learning ◽

Rating Scale ◽

Clinical Skills ◽

Mobile Apps ◽

Mobile App ◽

Systematic Search ◽

Measurement Scales ◽

High Quality

BACKGROUND Mobile apps are widely used in health professions, which increases the need for simple methods to determine the quality of apps. In particular, teachers need the ability to curate high-quality mobile apps for student learning. OBJECTIVE This study aims to systematically search for and evaluate the quality of clinical skills mobile apps as learning tools. The quality of apps meeting the specified criteria was evaluated using two measures—the widely used Mobile App Rating Scale (MARS), which measures general app quality, and the Mobile App Rubric for Learning (MARuL), a recently developed instrument that measures the value of apps for student learning—to assess whether MARuL is more effective than MARS in identifying high-quality apps for learning. METHODS Two mobile app stores were systematically searched using clinical skills terms commonly found in medical education and apps meeting the criteria identified using an approach based on PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. A total of 9 apps were identified during the screening process. The apps were rated independently by 2 reviewers using MARS and MARuL. RESULTS The intraclass correlation coefficients (ICCs) for the 2 raters using MARS and MARuL were the same (MARS ICC [two-way]=0.68; P<.001 and MARuL ICC [two-way]=0.68; P<.001). Of the 9 apps, Geeky Medics-OSCE revision (MARS Android=3.74; MARS iOS=3.68; MARuL Android=75; and MARuL iOS=73) and OSCE PASS: Medical Revision (MARS Android=3.79; MARS iOS=3.71; MARuL Android=69; and MARuL iOS=73) scored highly on both measures of app quality and for both Android and iOS. Both measures also showed agreement for the lowest rated app, Patient Education Institute (MARS Android=2.21; MARS iOS=2.11; MARuL Android=18; and MARuL iOS=21.5), which had the lowest scores in all categories except information (MARS) and professional (MARuL) in both operating systems. MARS and MARuL were both able to differentiate between the highest and lowest quality apps; however, MARuL was better able to differentiate apps based on teaching and learning quality. CONCLUSIONS This systematic search and rating of clinical skills apps for learning found that the quality of apps was highly variable. However, 2 apps—Geeky Medics-OSCE revision and OSCE PASS: Medical Revision—rated highly for both versions and with both quality measures. MARS and MARuL showed similar abilities to differentiate the quality of the 9 apps. However, MARuL’s incorporation of teaching and learning elements as part of a multidimensional measure of quality may make it more appropriate for use with apps focused on teaching and learning, whereas MARS’s more general rating of quality may be more appropriate for health apps targeting a general health audience. Ratings of the 9 apps by both measures also highlighted the variable quality of clinical skills mobile apps for learning. CLINICALTRIAL

Download Full-text

Measuring the Quality of Mobile Apps for the Management of Pain: Systematic Search and Evaluation Using the Mobile App Rating Scale

JMIR mhealth and uhealth ◽

10.2196/10718 ◽

2018 ◽

Vol 6 (10) ◽

pp. e10718 ◽

Cited By ~ 34

Author(s):

Alejandro Salazar ◽

Helena de Sola ◽

Inmaculada Failde ◽

Jose Antonio Moral-Munoz

Keyword(s):

Rating Scale ◽

Mobile Apps ◽

Mobile App ◽

Systematic Search

Download Full-text

Quality of Mobile Apps for Care Partners of People with Alzheimer’s Disease and Related Dementias: Mobile App Rating Scale Evaluation (Preprint)

10.2196/preprints.33863 ◽

2021 ◽

Author(s):

Nicole E Werner ◽

Janetta C Brown ◽

Priya Loganathar ◽

Richard J Holden

Keyword(s):

Rating Scale ◽

Mobile Apps ◽

Mobile App ◽

Future Research ◽

App Development ◽

Care Partners ◽

Acceptable Quality ◽

The Us ◽

Google Play

BACKGROUND The over 11 million care partners in the US who provide care to people living with Alzheimer’s disease and related dementias (ADRD) cite persistent and pervasive unmet needs related to all aspects of their caregiving role. The proliferation of mobile applications (apps) for care partners has potential to meet the care partners’ needs, but the quality of apps is unknown. OBJECTIVE The present study aimed to 1) evaluate the quality of publicly available apps for care partners of people living with ADRD and 2) identify design features of low- and high-quality apps to guide future research and app development. METHODS We searched the US Apple and Google Play app stores with the criteria that the app needed to be 1) available in US Google play or Apple app stores, 2) directly accessible to users “out of the box”, 3) primarily intended for use by an informal (family, friend) caregiver or caregivers of a person with dementia. The included apps were then evaluated using the Mobile App Rating Scale (MARS), which includes descriptive app classification and rating using 23 items across five dimensions: engagement, functionality, aesthetics, information, and subjective quality. Next, we computed descriptive statistics for each rating. To identify recommendations for future research and app development, we categorized rater comments on the score driving factors for each item and what the app could have done to improve the score for that item. RESULTS We evaluated 17 apps (41% iOS only, 12% Android only, 47% both iOS and Android). We found that on average, the apps are of minimally acceptable quality. Although we identified apps above and below minimally acceptable quality, many apps had broken features and were rated as below acceptable for engagement and information. CONCLUSIONS Minimally acceptable quality is likely insufficient to meet care partner needs. Future research should establish minimum quality standards across dimensions for mobile apps for care partners. The design features of high-quality apps we identified in this research can provide the foundation for benchmarking those standards.

Download Full-text

Mobile apps for detecting falsified and substandard drugs: A systematic review

PLoS ONE ◽

10.1371/journal.pone.0246061 ◽

2021 ◽

Vol 16 (2) ◽

pp. e0246061

Author(s):

Agustín Ciapponi ◽

Manuel Donato ◽

A. Metin Gülmezoglu ◽

Tomás Alconada ◽

Ariel Bardach

Keyword(s):

Systematic Review ◽

Rating Scale ◽

Mobile Apps ◽

Mobile App ◽

Economic Consequences ◽

Google Scholar ◽

Medical Database ◽

Bar Code ◽

Medical Databases

The use of substandard and counterfeit medicines (SCM) leads to significant health and economic consequences, like treatment failure, rise of antimicrobial resistance, extra expenditures of individuals or households and serious adverse drug reactions including death. Our objective was to systematically search, identify and compare relevant available mobile applications (apps) for smartphones and tablets, which use could potentially affect clinical and public health outcomes. We carried out a systematic review of the literature in January 2020, including major medical databases, and app stores. We used the validated Mobile App Rating Scale (MARS) to assess the quality of apps, (1 worst score, 3 acceptable score, and 5 best score). We planned to evaluate the accuracy of the mobile apps to detect SCM. We retrieved 335 references through medical databases and 42 from Apple, Google stores and Google Scholar. We finally included two studies of the medical database, 25 apps (eight from the App Store, eight from Google Play, eight from both stores, and one from Google Scholar), and 16 websites. We only found one report on the accuracy of a mobile apps detecting SCMs. Most apps use the imprint, color or shape for pill identification, and only a few offer pill detection through photographs or bar code. The MARS mean score for the apps was 3.17 (acceptable), with a maximum of 4.9 and a minimum of 1.1. The ‘functionality’ dimension resulted in the highest mean score (3.4), while the ‘engagement’ and ‘information’ dimensions showed the lowest one (3.0). In conclusion, we found a remarkable evidence gap about the accuracy of mobile apps in detecting SCMs. However, mobile apps could potentially be useful to screen for SCM by assessing the physical characteristics of pills, although this should still be assessed in properly designed research studies.

Download Full-text

Analysis of Secure Apps for Daily Clinical Use by German Orthopedic Surgeons: Searching for the "Needle in a Haystack" (Preprint)

10.2196/preprints.17085 ◽

2019 ◽

Author(s):

Florian Dittrich ◽

Sascha Beck ◽

Anna Katharina Harren ◽

Felix Reinecke ◽

Sebastian Serong ◽

...

Keyword(s):

Rating Scale ◽

Mobile App ◽

High Quality ◽

Clinical Routine ◽

Health Apps ◽

App Store ◽

Orthopedic Surgeons ◽

Future Challenges ◽

Mobile Health Apps

BACKGROUND It is undeniable that appropriate smartphone apps offer enormous opportunities for dealing with future challenges in orthopedic surgery and public health, in general. However, it is still unclear how the apps currently available in the two major app stores can be used in daily clinical routine by German orthopedic surgeons. OBJECTIVE This study aimed to gain evidence regarding the quantity and quality of apps available in the two major app stores and their suitability for use by orthopedic surgeons in Germany. METHODS We conducted a systematic, keyword-based app store screening to obtain evidence concerning the quantity and quality of commercially available apps. Apps that met the inclusion criteria were evaluated using the app synopsis–checklist for users and the German Mobile App Rating Scale for secure use, trustworthiness, and quality. RESULTS The investigation revealed serious shortcomings regarding legal and medical aspects. Furthermore, most apps turned out to be useless and unsuitable for the clinical field of application (4242/4249, 99.84%). Finally, 7 trustworthy and high-quality apps (7/4249, 0.16%) offering secure usage in the daily clinical routine of orthopedists were identified. These apps mainly focused on education (5/7). None of them were CE (Conformité Européenne) certified. Moreover, there are no studies providing evidence that these apps have any positive use whatsoever. CONCLUSIONS The data obtained in our study suggest that the number of trustworthy and high-quality apps on offer is extremely low. Nowadays, finding appropriate apps in the fast-moving, complex, dynamic, and rudimentarily controlled app stores is most challenging. Promising approaches, for example, systematic app store screenings, app-rating developments, reviews or app libraries, and the creation of consistent standards have been established. However, further efforts are necessary to ensure that these innovative mobile health apps not only provide the correct information but are also safe to use in daily clinical practice.

Download Full-text

Assessing Quality Factors and User Preference of Drug Reference Apps in Taiwan: Systematic Search and Evaluation (Preprint)

10.2196/preprints.31195 ◽

2021 ◽

Author(s):

Yu-Chun Chen ◽

Wei-Wei Liao ◽

Mei-Chin Su ◽

Yen-Hsi Lin

Keyword(s):

Regression Model ◽

Rating Scale ◽

Drug Information ◽

Mobile App ◽

Influential Factors ◽

Systematic Search ◽

Visual Appeal ◽

User Ratings ◽

Drug Reference

BACKGROUND Drug reference apps play various distinct and vital roles through the medication use process. A drug reference app that carried out comprehensive localized drug information would greatly improve the efficiency and quality of work for physicians, nurses, pharmacists, and patients. OBJECTIVE This current study aimed to describe a systematic and stepwise process to identify drug reference apps with localized drug information in Taiwan. Moreover, we assessed the quality of these apps by using a reliable quality assessment tool and further analyzing the influential factors for user ratings. METHODS A two-step algorithm (KESS) consisting of keyword growing and systematic search was proposed. Apps were divided into two groups: higher user ratings and lower user ratings. Seven independent reviewers were trained to evaluate these apps using Mobile App Rating Scale (MARS). A logistic regression model was fitted, and average marginal effects (AME) were calculated to identify the effects of factors for higher user ratings. A p-value< 0.05 was considered statistically significant. RESULTS A total of 23 drug reference apps in Taiwan had been identified and analyzed. Ten apps had higher user star ratings (>=4 stars), and 13 apps had lower user star ratings ( < 4 stars). These drug reference apps had acceptable quality with an average MARS score of 3.23. Apps with higher user star ratings had higher MARS scores than the lowers (engagement (2.70 v.s. 2.50, P= .005), functionality (3.85 v.s. 3.49, P= .003), aesthetics (3.39 v.s. 2.98, P < .001), and information (3.55 v.s. 3.25, P= .005)). The regression model showed five influential factors for higher user ratings (navigation, AME, 13.15%; performance, AME, 11.03%; visual appeal, AME, 10.87%; credibility, AME, 10.67%; quantity of information, AME, 10.42%). CONCLUSIONS The proposed KESS algorithm could be a valuable and unbiased framework for systematic search for app. While the higher engagement, more functionality, better aesthetics, and more information associated with higher user ratings, there are five most influential factors, navigation, performance, visual appeal, credibility, and quantity of information among the four elements.

Download Full-text

Diet-Related Mobile Apps to Promote Healthy Eating and Proper Nutrition: A Content Analysis and Quality Assessment

International Journal of Environmental Research and Public Health ◽

10.3390/ijerph18073496 ◽

2021 ◽

Vol 18 (7) ◽

pp. 3496

Author(s):

Jihye Choi ◽

Chongwook Chung ◽

Hyekyung Woo

Keyword(s):

Quality Assessment ◽

Mobile Applications ◽

Rating Scale ◽

Mobile Apps ◽

Mobile App ◽

Regression Analyses ◽

Eligibility Criteria ◽

Selection Of

Dietary mobile applications (apps) continue to hold promise for facilitating a healthy diet and managing nutrition. However, few studies have objectively evaluated the content and quality of such apps in Korea. The present study assessed the content and quality of dietary mobile apps using the Mobile App Rating Scale (MARS). We selected 29 dietary apps based on keywords and eligibility criteria for inclusion in the analyses. We conducted regression analyses to examine the association between app content and MARS scores. Most of the apps featured a tracking tool, while few featured rewards or follow-up management. Our quality assessment revealed that the top-rated apps have distinct levels of quality in terms of MARS scores. The regression analyses showed that the ways in which the apps provide information and motivate the users are statistically significant predictors of app quality. Our findings may facilitate the selection of dietary apps in Korea and provide guidelines for app developers regarding potential improvements in terms of content and quality.

Download Full-text

Mobile Apps for Speech-Language Therapy in Adults With Communication Disorders: Review of Content and Quality

JMIR mhealth and uhealth ◽

10.2196/18858 ◽

2020 ◽

Vol 8 (10) ◽

pp. e18858

Author(s):

Atiyeh Vaezipour ◽

Jessica Campbell ◽

Deborah Theodoros ◽

Trevor Russell

Keyword(s):

Rating Scale ◽

Mobile Apps ◽

Communication Disorders ◽

Mobile App ◽

Evidence Based ◽

Language Therapy ◽

Eligibility Criteria ◽

App Store ◽

Google Play

Background Worldwide, more than 75% of people with acquired brain injury (ABI) experience communication disorders. Communication disorders are impairments in the ability to communicate effectively, that is, sending, receiving, processing, and comprehending verbal and nonverbal concepts and symbols. Such disorders may have enduring impacts on employment, social participation, and quality of life. Technology-enabled interventions such as mobile apps have the potential to increase the reach of speech-language therapy to treat communication disorders. However, ensuring that apps are evidence-based and of high quality is critical for facilitating safe and effective treatment for adults with communication disorders. Objective The aim of this review is to identify mobile apps that are currently widely available to adults with communication disorders for speech-language therapy and to assess their content and quality using the validated Mobile App Rating Scale (MARS). Methods Google Play Store, Apple App Store, and webpages were searched to identify mobile apps for speech-language therapy. Apps were included in the review if they were designed for the treatment of adult communication disorders after ABI, were in English, and were either free or for purchase. Certified speech-language pathologists used the MARS to assess the quality of the apps. Results From a total of 2680 apps identified from Google Play Store, Apple App Store, and web searches, 2.61% (70/2680) apps met the eligibility criteria for inclusion. Overall, 61% (43/70) were available for download on the iPhone Operating System (iOS) platform, 20% (14/70) on the Android platform, and 19% (13/70) on both iOS and Android platforms. A content analysis of the apps revealed 43 apps for language, 17 apps for speech, 8 apps for cognitive communication, 6 apps for voice, and 5 apps for oromotor function or numeracy. The overall MARS mean score was 3.7 out of 5, SD 0.6, ranging between 2.1 and 4.5, with functionality being the highest-scored subscale (4.3, SD 0.6), followed by aesthetics (3.8, SD 0.8), information (3.4, SD 0.6), and engagement (3.3, SD 0.6). The top 5 apps were Naming Therapy (4.6/5), Speech Flipbook Standard (4.6/5), Number Therapy (4.5/5), Answering Therapy, and Constant Therapy (4.4/5). Conclusions To our knowledge, this is the first study to systematically identify and evaluate a broad range of mobile apps for speech-language therapy for adults with communication disorders after sustaining ABI. We found a lack of interactive and engaging elements in the apps, a critical factor in sustaining self-managed speech-language therapy. More evidence-based apps with a focus on human factors, user experience, and a patient-led design approach are required to enhance effectiveness and long-term use.

Download Full-text

Quality, Functionality, and Features of Chinese Mobile Apps for Diabetes Self-Management: Systematic Search and Evaluation of Mobile Apps

JMIR mhealth and uhealth ◽

10.2196/14836 ◽

2020 ◽

Vol 8 (4) ◽

pp. e14836 ◽

Cited By ~ 2

Author(s):

Enying Gong ◽

Zongmuyu Zhang ◽

Xurui Jin ◽

Yishan Liu ◽

Lumin Zhong ◽

...

Keyword(s):

Blood Glucose ◽

Behavior Change ◽

Rating Scales ◽

Mobile Apps ◽

Mobile App ◽

Systematic Search ◽

Self Management ◽

Average Score ◽

Inclusion Criteria

Background The emergence and advancement of mobile technologies offer a promising opportunity for people with diabetes to improve their self-management. Despite the proliferation of mobile apps, few studies have evaluated the apps that are available to the millions of people with diabetes in China. Objective This study aimed to conduct a systematic search of Chinese mobile apps for diabetes self-management and to evaluate their quality, functionality, and features by using validated rating scales. Methods A systematic search was conducted to identify Chinese apps for diabetes self-management in the four most popular Chinese language mobile app stores. Apps were included if they were designed for diabetes self-management and contained at least one of the following components: blood glucose management, dietary and physical activity management, medication taking, and prevention of diabetes-related comorbidities. Apps were excluded if they were unrelated to health, not in Chinese, or the targeted users are health care professionals. Apps meeting the identified inclusion criteria were downloaded and evaluated by a team of 5 raters. The quality, functionalities, and features of these apps were assessed by using the Mobile App Rating Scale (MARS), the IMS Institute for Healthcare Informatics Functionality score, and a checklist of self-management activities developed based on the Chinese diabetes self-management guideline, respectively. Results Among 2072 apps searched, 199 were eligible based on the inclusion criteria, and 67 apps were successfully downloaded for rating. These 67 apps had an average MARS score of 3.42 out of 5, and 76% (51/67) of the apps achieved an acceptable quality (MARS score >3.0). The scores for the four subdomains of MARS were 3.97 for functionality, 3.45 for aesthetics, 3.21 for information, and 3.07 for engagement. On average, reviewed apps applied five out of the 19 examined behavior change techniques, whereas the average score on the subjective quality for the potential impact on behavior change is 3 out of 5. In addition, the average score on IMS functionality was 6 out of 11. Functionalities in collecting, recording, and displaying data were mostly presented in the reviewed apps. Most of the apps were multifeatured with monitoring blood glucose and tracking lifestyle behaviors as common features, but some key self-management activities recommended by clinical guidelines, such as stress and emotional management, were rarely presented in these apps. Conclusions The general quality of the reviewed apps for diabetes self-management is suboptimal, although the potential for improvement is significant. More attention needs to be paid to the engagement and information quality of these apps through co-design with researchers, public health practitioners, and consumers. There is also a need to promote the awareness of the public on the benefit and potential risks of utilizing health apps for self-management.

Download Full-text

Mobile Apps for Blood Pressure Monitoring: Systematic Search in App Stores and Content Analysis (Preprint)

10.2196/preprints.9888 ◽

2018 ◽

Author(s):

Hussein Jamaladin ◽

Tom H van de Belt ◽

Lianda CH Luijpers ◽

Falco R de Graaff ◽

Sebastian JH Bredie ◽

...

Keyword(s):

Blood Pressure ◽

Health Care Providers ◽

Rating Scale ◽

Mobile Apps ◽

Blood Pressure Monitoring ◽

Mobile App ◽

Self Management ◽

Care Providers ◽

Pressure Monitoring

BACKGROUND Using a mobile app for self-management could make it easier for patients to get insight into their blood pressure patterns. However, little is known about the availability, quality, and features of mobile apps targeting blood pressure. OBJECTIVE The goal of the research was to determine the availability, functionality, and quality of mobile apps that could be used for blood pressure monitoring purposes. METHODS A systematic app search was performed based on the standards for systematic reviews. We searched the Dutch official app stores for Android and iOS platforms using predefined keywords and included all English and Dutch mobile apps targeting blood pressure. Two independent assessors determined eligibility and quality of the apps using the 5-point Mobile App Rating Scale (MARS). Quality scores of the apps with and without 17 a priori selected characteristics were compared using independent sample t tests. RESULTS A total of 184 apps (104 Android, 80 iOS) met the inclusion criteria. The mean overall MARS score was 2.63 (95% CI 2.55-2.71) for Android and 2.64 (95% CI 2.56-2.71) for iOS. The apps Bloeddruk (4.1) and AMICOMED BP (3.6) had the highest quality scores on the Android and iOS platforms, respectively. Of the app characteristics recorded, only pricing, in-app advertisements, and local data storage were not associated with the quality scores. In only 3.8% (7/184) of the apps, involvement of medical experts in its development was mentioned, whereas none of the apps was formally evaluated with results published in a peer-reviewed journal. CONCLUSIONS This study provides an overview of the best apps currently available in the app stores and important key features for self-management that can be used by health care providers and patients with hypertension to identify a suitable app targeting blood pressure monitoring. However, the majority of the apps targeting blood pressure monitoring were of poor quality. Therefore, it is important to involve medical experts in the developmental stage of health-related mobile apps to improve the quality of these apps.

Download Full-text