Evaluating the quality and safety of health-related apps and e-tools: Adapting the Mobile App Rating Scale and developing a quality assurance protocol

BACKGROUND There is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed. OBJECTIVE The main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders. METHODS In order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability. RESULTS Only a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (α=0.78) and “Privacy” (α=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29. CONCLUSIONS This study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

Download Full-text

Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides

JMIR mhealth and uhealth ◽

10.2196/26471 ◽

2021 ◽

Vol 9 (4) ◽

pp. e26471

Author(s):

Jordi Miró ◽

Pere Llorens-Vernet

Keyword(s):

Health Care ◽

Health Care Professionals ◽

Interrater Reliability ◽

Rating Scale ◽

Mobile App ◽

App Development ◽

Different Types ◽

Health Related ◽

Medical Category

Background There is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed. Objective The main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders. Methods In order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability. Results Only a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (α=0.78) and “Privacy” (α=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29. Conclusions This study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

Download Full-text

The Association Between Personality and Homebound Status in Older Adults: Results From the NHATS

Innovation in Aging ◽

10.1093/geroni/igaa057.751 ◽

2020 ◽

Vol 4 (Supplement_1) ◽

pp. 233-233

Author(s):

Xiaocao Sun ◽

Minhui Liu ◽

Christina E Miyawaki ◽

Yuxiao Li ◽

Tianxue Hou ◽

...

Keyword(s):

Older Adults ◽

Personality Traits ◽

Personality Assessment ◽

Rating Scale ◽

Community Dwelling ◽

Cross Sectional ◽

Health Related ◽

Community Dwelling Older Adults ◽

Ordinal Logistic Regression Model ◽

Hispanic White

Abstract Personality is associated with predictors of homebound status like frailty, incident falls, and depression. It has been rarely investigated whether personality predicts homebound status among older adults. Using the combining cross-sectional data of the Year 2013 and Year 2014 data from the National Health and Aging Trends Study (NHATS), this study examined the association between personality traits and homebound status in a sample of community-dwelling older adults aged 65 years and older (N=2,788). Homebound status (non-homebound, semi-homebound, and homebound) was determined by the frequency, difficulty, and help of outdoor mobility. Personality traits, including conscientiousness, agreeableness, openness, extraversion, and neuroticism were assessed using the 10-item Midlife Development Inventory on a rating scale from 1 (not at all) to 4 (a lot). Each personality trait was included as a predictor in an ordinal logistic regression model to examine its association with homebound status after adjusting demographic and health-related covariates. The sample was on average 79±7.53 years old, non-Hispanic White (72.0%), female (58.6%), living alone (35.4%) or with spouse/partner only (37.4%). Seventy-four percent, 18%, and 8% of participants were non-homebound, semi-homebound, and homebound, respectively. Homebound participants tended to be less-educated older females. The average scores of conscientiousness, agreeableness, openness, extraversion, and neuroticism were 3.19±0.75, 3.57±0.56, 2.81±0.83, 3.13±0.75, and 2.22±0.86, respectively. Among these five personality traits, high conscientiousness (OR=1.34, p<0.001) and extraversion (OR=1.16, p=.03) were associated with a reduced likelihood of being homebound. These findings provided a basis for potential personality assessment to identify and protect individuals with high homebound risk.

Download Full-text

Moving beyond Limitations: Designing the Helpdys App for Children with Dyslexia in Rural Areas

Sustainability ◽

10.3390/su13137081 ◽

2021 ◽

Vol 13 (13) ◽

pp. 7081

Author(s):

Andres Larco ◽

Jorge Carrillo ◽

Nelson Chicaiza ◽

Cesar Yanez ◽

Sergio Luján-Mora

Keyword(s):

Rural Communities ◽

Rural Areas ◽

Rating Scale ◽

Language Disorder ◽

Educational Games ◽

Mobile App ◽

Literacy Skills ◽

Full Potential ◽

Positive Contribution ◽

Self Confidence

Dyslexia is a relatively common language disorder which is generally ignored in rural communities. It hinders children’s learning processes and, in some cases, is the cause of dropouts or violence in schools. The present work strives to create a web and mobile app as a preliminary step towards the diagnosis and treatment of dyslexic children. Apps providing didactic educational games and activities improve literacy skills for students with reading disabilities. The current work incorporates user experience and prototyping to fulfill app requirements. The authors evaluated the apps with the Mobile App Rating Scale (MARS) tool to assess engagement, functionality, aesthetics, and information. The app’s improvements were immediately implemented and tested in the “Escuela Linea Equinoccial” (Ecuador) school, proving its utility for future use in the education system. The app can be a valuable tool for children with dyslexia to progress successfully through school, raising their self-confidence and, thereby, helping them reach their full potential as adults able to make a positive contribution to society.

Download Full-text

The validity of mobile applications to facilitate patient care provided to cancer patients: opportunities and limitations

Recent Patents on Anti-Cancer Drug Discovery ◽

10.2174/1574892816666210728122304 ◽

2021 ◽

Vol 16 ◽

Author(s):

Haneen Amawi ◽

Sayer Alazzam ◽

Tasnim Alzanati ◽

Neveen Altamimi ◽

Alaa Hammad ◽

...

Keyword(s):

Health Services ◽

Patient Care ◽

Health Care Services ◽

Rating Scale ◽

Assessment Tools ◽

Free Access ◽

User Engagement ◽

Care Services ◽

Health Related ◽

Detailed Assessment

Background: The use of health-related applications (apps) on smartphones has become widespread. This is especially of value during the ongoing SAR-COV-2 pandemic, where the accessibility for health care services has been greatly limited. Patients with free access to apps can obtain information to improve their understanding and management of health issues. Currently, there are cancer-related apps available on iPhones and androids. However, there are no guidelines to control these apps and ensure their quality. Furthermore, these apps may significantly modify the patients’ perception and knowledge toward drug-related health services. Objective: The aim of this study was to assess the convenience, quality, safety and efficacy of apps for cancer patient care. Methods: The study was conducted by searching all apps related to cancer care on both Google Play Store and Apple iTunes Store. A detailed assessment was then performed using the mobile application rating scale (MARS) and risk assessment tools. Results: The results indicated that on a scale from 1-5, 47% of the apps were rated ≥ 4. The MARS assessment of the apps indicated an overall quality rating of 3.38 ± 0.9 (mean ± SD). The visual appeal of the app was found to have a significant effect on app functionality and user engagement. The potential benefits of these apps come with challenges and limitations. Patents related to smartphone applications targeting patients were also discussed. Conclusion: We recommend a greater emphasis toward producing evidence-based apps. These apps should be rigorously tested, evaluated and updated by experts, particularly clinical pharmacists. Also, these may alter patient attitudes toward services provided by physicians and pharmacists. Finally, these apps should not replace in-person interactive health services.

Download Full-text

Initiating Collaboration among Organ Transplant Professionals through Web Portals and Mobile Applications

OJIN: The Online Journal of Issues in Nursing ◽

10.3912/ojin.vol18no02ppt03 ◽

2013 ◽

Vol 18 (2) ◽

Author(s):

Susan Alexander ◽

Haley Hoy ◽

Manil Maskey ◽

Helen Conover ◽

John Gamble ◽

...

Keyword(s):

Mobile Applications ◽

Healthcare Providers ◽

Organ Transplant ◽

Pilot Project ◽

Mobile App ◽

Lessons Learned ◽

Ease Of Use ◽

Future Directions ◽

Efficient Access ◽

Health Related

The knowledge base for healthcare providers working in the field of organ transplantation has grown exponentially. However, the field has no centralized ‘space’ dedicated to efficient access and sharing of information.The ease of use and portability of mobile applications (apps) make them ideal for subspecialists working in complex healthcare environments. In this article, the authors review the literature related to healthcare technology; describe the development of health-related technology; present their mobile app pilot project assessing the effects of a collaborative, mobile app based on a freely available content manage framework; and report their findings. They conclude by sharing both lessons learned while completing this project and future directions.

Download Full-text

MOBILE PHONE DEPENDENCE AND HEALTH-RELATED LIFESTYLE OF UNIVERSITY STUDENTS

Social Behavior and Personality An International Journal ◽

10.2224/sbp.2006.34.10.1277 ◽

2006 ◽

Vol 34 (10) ◽

pp. 1277-1284 ◽

Cited By ~ 87

Author(s):

Masahiro Toda ◽

Kazuyuki Monden ◽

Kazuki Kubo ◽

Kanehisa Morimoto

Keyword(s):

University Students ◽

Mobile Phone ◽

Significant Relationship ◽

Rating Scale ◽

Healthy Lifestyle ◽

Smoking Habits ◽

Health Practice ◽

Mobile Phone Use ◽

Self Rating ◽

Health Related

This study investigated the associations between the intensity of mobile phone use and health-related lifestyle. For 275 university students, we evaluated health-related lifestyle using the Health Practice Index (HPI; Hagihara & Morimoto, 1991; Kusaka, Kondou, & Morimoto, 1992) and analyzed responses to a questionnaire (MPDQ; Toda, Monden, Kubo, & Morimoto, 2004) designed, with a self-rating scale, to gauge mobile phone dependence. For males, there was a significant relationship between smoking habits and mobile phone dependence. We also found that male respondents with low HPI scores were significantly higher for mobile phone dependence. These findings suggest that, particularly for males, the intensity of mobile phone use may be related to healthy lifestyle.

Download Full-text

The Effects of Gamification and Oral Self-Care on Oral Hygiene in Children: Systematic Search in App Stores and Evaluation of Apps (Preprint)

10.2196/preprints.16365 ◽

2019 ◽

Author(s):

Nino Fijačko ◽

Lucija Gosak ◽

Leona Cilar ◽

Alenka Novšak ◽

Ruth Masterson Creber ◽

...

Keyword(s):

Public Health ◽

Behavior Change ◽

Oral Hygiene ◽

Rating Scale ◽

Self Care ◽

Public Health Problem ◽

Mobile App ◽

Evidence Based ◽

Behavior Change Techniques ◽

Evidence Based Dentistry

BACKGROUND Poor oral hygiene is a great public health problem worldwide. Oral health care education is a public health priority as the maintenance of oral hygiene is integral to overall health. Maintaining optimal oral hygiene among children is challenging and can be supported by using relevant motivational approaches. OBJECTIVE The primary aim of this study was to identify mobile smartphone apps that include gamification features focused on motivating children to learn, perform, and maintain optimal oral hygiene. METHODS We searched six online app stores using four search terms (“oral hygiene game,” “oral hygiene gamification,” “oral hygiene brush game,” and “oral hygiene brush gamification”). We identified gamification features, identified whether apps were consistent with evidence-based dentistry, performed a quality appraisal with the Mobile App Rating Scale user version (uMARS), and quantified behavior scores (Behavior Change score, uMARS score, and Coventry, Aberdeen, and London-Refined [CALO-RE] score) using three different instruments that measure behavior change. RESULTS Of 612 potentially relevant apps included in the analysis, 17 met the inclusion criteria. On average, apps included 6.87 (SD 4.18) out of 31 possible gamification features. The most frequently used gamification features were time pressure (16/17, 94%), virtual characters (14/17, 82%), and fantasy (13/17, 76%). The most common oral hygiene evidence-based recommendation was brushing time (2-3 minutes), which was identified in 94% (16/17) of apps. The overall mean uMARS score for app quality was high (4.30, SD 0.36), with good mean subjective quality (3.79, SD 0.71) and perceived impact (3.58, SD 0.44). Sufficient behavior change techniques based on three taxonomies were detected in each app. CONCLUSIONS The majority of the analyzed oral hygiene apps included gamification features and behavior change techniques to perform and maintain oral hygiene in children. Overall, the apps contained some educational content consistent with evidence-based dentistry and high-quality background for oral self-care in children; however, there is scope for improvement.

Download Full-text

Smartphone Apps Targeting Physical Activity in People With Rheumatoid Arthritis: Systematic Quality Appraisal and Content Analysis

JMIR mhealth and uhealth ◽

10.2196/18495 ◽

2020 ◽

Vol 8 (7) ◽

pp. e18495 ◽

Cited By ~ 1

Author(s):

Lindsay M Bearne ◽

Mandeep Sekhon ◽

Rebecca Grainger ◽

Anthony La ◽

Mehrdad Shamali ◽

...

Keyword(s):

Rheumatoid Arthritis ◽

Physical Activity ◽

Rating Scale ◽

Mobile App ◽

Information Support ◽

World Health ◽

Smartphone Apps ◽

High Quality ◽

App Store ◽

Google Play

Background Rheumatoid arthritis (RA) is a disabling, inflammatory joint condition affecting 0.5%-1% of the global population. Physical activity (PA) and exercise are recommended for people with RA, but uptake and adherence tend to be low. Smartphone apps could assist people with RA to achieve PA recommendations. However, it is not known whether high quality, evidence-informed PA apps that include behavior change techniques (BCTs) previously identified as effective for PA adherence are available for people with RA. Objective This study aims to systematically identify apps that include goals to facilitate PA for adults with RA and assess app quality and content for the inclusion of relevant BCTs against recommendations for cardiorespiratory, resistance, flexibility, and neuromotor PA and exercise. Methods A systematic search of the Apple App Store and Google Play Store in the United Kingdom was conducted to identify English language apps that promote PA for adults with RA. Two researchers independently assessed app quality (mobile app rating scale [MARS]; range 0-5) and content (BCT Taxonomy version 1, World Health Organization, the American College of Sports Medicine, and the European League against Rheumatism recommendations for PA). The completeness of reporting of PA prescription was evaluated using a modified version of the Consensus on Exercise Reporting Template (CERT; range 0-14). Results A total of 14,047 apps were identified. Following deduplication, 2737 apps were screened for eligibility; 6 apps were downloaded (2 on the Apple App Store and 4 on the Google Play Store), yielding 4 unique apps. App quality varied (MARS score 2.25-4.17). Only 1 app was congruent with all aspects of the PA recommendations. All apps completely or partially recommended flexibility and resistance exercises, 3 apps completely or partially advised some form of neuromotor exercise, but only 2 offered full or partial guidance on cardiorespiratory exercise. Completeness of exercise reporting was mixed (CERT scores 7-14 points) and 3-7 BCTs were identified. Two BCTs were common to all apps (information about health consequences and instruction on how to perform behavior). Higher quality apps included a greater number of BCTs and were more closely aligned to PA guidance. No published trials evaluating the effect of the included apps were identified. Conclusions This review identifies 4 PA apps of mixed quality and content for use by people with RA. Higher quality apps were more closely aligned to PA guidance and included a greater number of BCTs. One high-quality app (Rheumatoid Arthritis Information Support and Education) included 7 BCTs and was fully aligned with PA and exercise guidance. The effect of apps on PA adherence should be established before implementation.

Download Full-text

The Arabic Version of the Mobile App Rating Scale: Development and Validation Study (Preprint)

10.2196/preprints.16956 ◽

2019 ◽

Author(s):

Marco Bardus ◽

Nathalie Awada ◽

Lilian A Ghandour ◽

Elie-Jacques Fares ◽

Tarek Gherbal ◽

...

Keyword(s):

Weight Management ◽

Information Quality ◽

Rating Scale ◽

Arab World ◽

Intraclass Correlation ◽

Correlation Coefficients ◽

Mobile App ◽

Health Apps ◽

Highly Correlated

BACKGROUND With thousands of health apps in app stores globally, it is crucial to systemically and thoroughly evaluate the quality of these apps due to their potential influence on health decisions and outcomes. The Mobile App Rating Scale (MARS) is the only currently available tool that provides a comprehensive, multidimensional evaluation of app quality, which has been used to compare medical apps from American and European app stores in various areas, available in English, Italian, Spanish, and German. However, this tool is not available in Arabic. OBJECTIVE This study aimed to translate and adapt MARS to Arabic and validate the tool with a sample of health apps aimed at managing or preventing obesity and associated disorders. METHODS We followed a well-established and defined “universalist” process of cross-cultural adaptation using a mixed methods approach. Early translations of the tool, accompanied by confirmation of the contents by two rounds of separate discussions, were included and culminated in a final version, which was then back-translated into English. Two trained researchers piloted the MARS in Arabic (MARS-Ar) with a sample of 10 weight management apps obtained from Google Play and the App Store. Interrater reliability was established using intraclass correlation coefficients (ICCs). After reliability was ascertained, the two researchers independently evaluated a set of additional 56 apps. RESULTS MARS-Ar was highly aligned with the original English version. The ICCs for MARS-Ar (0.836, 95% CI 0.817-0.853) and MARS English (0.838, 95% CI 0.819-0.855) were good. The MARS-Ar subscales were highly correlated with the original counterparts (P<.001). The lowest correlation was observed in the area of usability (r=0.685), followed by aesthetics (r=0.827), information quality (r=0.854), engagement (r=0.894), and total app quality (r=0.897). Subjective quality was also highly correlated (r=0.820). CONCLUSIONS MARS-Ar is a valid instrument to assess app quality among trained Arabic-speaking users of health and fitness apps. Researchers and public health professionals in the Arab world can use the overall MARS score and its subscales to reliably evaluate the quality of weight management apps. Further research is necessary to test the MARS-Ar on apps addressing various health issues, such as attention or anxiety prevention, or sexual and reproductive health.

Download Full-text