scholarly journals Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides

10.2196/26471 ◽  
2021 ◽  
Vol 9 (4) ◽  
pp. e26471
Author(s):  
Jordi Miró ◽  
Pere Llorens-Vernet
Keyword(s):  
Health Care ◽  
Health Care Professionals ◽  
Interrater Reliability ◽  
Rating Scale ◽  
Mobile App ◽  
App Development ◽  
Different Types ◽  
Health Related ◽  
Medical Category

Background There is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed. Objective The main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders. Methods In order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability. Results Only a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (α=0.78) and “Privacy” (α=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29. Conclusions This study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

scholarly journals Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides (Preprint)

2020 ◽  
Author(s):  
Jordi Miró ◽  
Pere Llorens-Vernet
Keyword(s):  
Health Care ◽  
Health Care Professionals ◽  
Interrater Reliability ◽  
Rating Scale ◽  
Mobile App ◽  
App Development ◽  
Different Types ◽  
Health Related ◽  
Medical Category

BACKGROUND There is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed. OBJECTIVE The main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders. METHODS In order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability. RESULTS Only a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (<i>α</i>=0.78) and “Privacy” (<i>α</i>=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: <i>α</i>=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29. CONCLUSIONS This study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

scholarly journals The Mobile App Development and Assessment Guide (MAG): Delphi-Based Validity Study (Preprint)

2020 ◽  
Author(s):  
Pere Llorens-Vernet ◽  
Jordi Miró
Keyword(s):  
Health Care ◽  
Mobile Health ◽  
Health Care Professionals ◽  
Online Survey ◽  
End Users ◽  
Mobile App ◽  
Health Apps ◽  
App Development ◽  
Mobile Health Apps

BACKGROUND In recent years, there has been an exponential growth of mobile health (mHealth)–related apps. This has occurred in a somewhat unsupervised manner. Therefore, having a set of criteria that could be used by all stakeholders to guide the development process and the assessment of the quality of the apps is of most importance. OBJECTIVE The aim of this paper is to study the validity of the Mobile App Development and Assessment Guide (MAG), a guide recently created to help stakeholders develop and assess mobile health apps. METHODS To conduct a validation process of the MAG, we used the Delphi method to reach a consensus among participating stakeholders. We identified 158 potential participants: 45 patients as potential end users, 41 health care professionals, and 72 developers. We sent participants an online survey and asked them to rate how important they considered each item in the guide to be on a scale from 0 to 10. Two rounds were enough to reach consensus. RESULTS In the first round, almost one-third (n=42) of those invited participated, and half of those (n=24) also participated in the second round. Most items in the guide were found to be important to a quality mHealth-related app; a total of 48 criteria were established as important. “Privacy,” “security,” and “usability” were the categories that included most of the important criteria. CONCLUSIONS The data supports the validity of the MAG. In addition, the findings identified the criteria that stakeholders consider to be most important. The MAG will help advance the field by providing developers, health care professionals, and end users with a valid guide so that they can develop and identify mHealth-related apps that are of quality.

scholarly journals The Mobile App Development and Assessment Guide (MAG): Delphi-Based Validity Study

10.2196/17760 ◽  
2020 ◽  
Vol 8 (7) ◽  
pp. e17760 ◽  
Author(s):  
Pere Llorens-Vernet ◽  
Jordi Miró
Keyword(s):  
Health Care ◽  
Mobile Health ◽  
Health Care Professionals ◽  
Online Survey ◽  
End Users ◽  
Mobile App ◽  
Health Apps ◽  
App Development ◽  
Mobile Health Apps

Background In recent years, there has been an exponential growth of mobile health (mHealth)–related apps. This has occurred in a somewhat unsupervised manner. Therefore, having a set of criteria that could be used by all stakeholders to guide the development process and the assessment of the quality of the apps is of most importance. Objective The aim of this paper is to study the validity of the Mobile App Development and Assessment Guide (MAG), a guide recently created to help stakeholders develop and assess mobile health apps. Methods To conduct a validation process of the MAG, we used the Delphi method to reach a consensus among participating stakeholders. We identified 158 potential participants: 45 patients as potential end users, 41 health care professionals, and 72 developers. We sent participants an online survey and asked them to rate how important they considered each item in the guide to be on a scale from 0 to 10. Two rounds were enough to reach consensus. Results In the first round, almost one-third (n=42) of those invited participated, and half of those (n=24) also participated in the second round. Most items in the guide were found to be important to a quality mHealth-related app; a total of 48 criteria were established as important. “Privacy,” “security,” and “usability” were the categories that included most of the important criteria. Conclusions The data supports the validity of the MAG. In addition, the findings identified the criteria that stakeholders consider to be most important. The MAG will help advance the field by providing developers, health care professionals, and end users with a valid guide so that they can develop and identify mHealth-related apps that are of quality.

scholarly journals Assessing Apps for Patients with Genitourinary Tumors Using the Mobile Application Rating Scale (MARS): Systematic Search in App Stores and Content Analysis (Preprint)

2019 ◽  
Author(s):  
Miguel Ángel Amor-García ◽  
Roberto Collado-Borrell ◽  
Vicente Escudero-Vilaplana ◽  
Alejandra Melgarejo-Ortuño ◽  
Ana Herranz-Alonso ◽  
...  
Keyword(s):  
Prostate Cancer ◽  
Health Care ◽  
Health Professionals ◽  
Health Care Professionals ◽  
Mobile Application ◽  
Rating Scale ◽  
Cross Sectional ◽  
Cancer Symptoms ◽  
Genitourinary Cancers

BACKGROUND The large number of available cancer apps and their impact on the population necessitates a transparent, objective, and comprehensive evaluation by app experts, health care professionals, and users. To date, there have been no analyses or classifications of apps for patients with genitourinary cancers, which are among the most prevalent types of cancer. OBJECTIVE The objective of our study was to analyze the quality of apps for patients diagnosed with genitourinary cancers using the Mobile Application Rating Scale (MARS) and identify high-quality apps. METHODS We performed an observational cross-sectional descriptive study of all smartphone apps for patients diagnosed with genitourinary cancers available on iOS and Android platforms. In July 2019, we searched for all available apps for patients with genitourinary cancers (bladder, prostate, cervical, uterine, endometrial, kidney, testicular, and vulvar) or their caregivers. Apps were downloaded and evaluated, and the general characteristics were entered into a database. The evaluation was performed by 2 independent researchers using the MARS questionnaire, which rates 23 evaluation criteria clustered in 5 domains (Engagement, Functionality, Esthetics, Information, and Subjective Quality) on a scale from 1 to 5. RESULTS In total, 46 apps were analyzed. Of these, 31 (67%) were available on Android, 6 (13%) on iOS, and 9 (20%) on both platforms. The apps were free in 89% of cases (41/46), and 61% (28/46) had been updated in the previous year. The apps were intended for prostate cancer in 30% of cases (14/46) and cervical cancer in 17% (8/46). The apps were mainly informative (63%, 29/46), preventive (24%, 11/46), and diagnostic (13%, 6/46). Only 7/46 apps (15%) were developed by health care organizations. The mean MARS score for the overall quality of the 46 apps was 2.98 (SD 0.77), with a maximum of 4.63 and a minimum of 1.95. Functionality scores were quite similar for most of the apps, with the greatest differences in Engagement and Esthetics, which showed acceptable scores in one-third of the apps. The 5 apps with the highest MARS score were the following: “Bladder cancer manager,” “Kidney cancer manager,” “My prostate cancer manager,” “Target Ovarian Cancer Symptoms Diary,” and “My Cancer Coach.” We observed statistically significant differences in the MARS score between the operating systems and the developer types (<i>P</i>&lt;.001 and <i>P</i>=.01, respectively), but not for cost (<i>P</i>=.62). CONCLUSIONS MARS is a helpful methodology to decide which apps can be prescribed to patients and to identify which features should be addressed to improve these tools. Most of the apps designed for patients with genitourinary cancers only try to provide data about the disease, without coherent interactivity. The participation of health professionals in the development of these apps is low; nevertheless, we observed that both the participation of health professionals and regular updates were correlated with quality.

scholarly journals Quality of Mobile Apps for Care Partners of People with Alzheimer’s Disease and Related Dementias: Mobile App Rating Scale Evaluation (Preprint)

2021 ◽  
Author(s):  
Nicole E Werner ◽  
Janetta C Brown ◽  
Priya Loganathar ◽  
Richard J Holden
Keyword(s):  
Rating Scale ◽  
Mobile Apps ◽  
Mobile App ◽  
Future Research ◽  
App Development ◽  
Care Partners ◽  
Acceptable Quality ◽  
The Us ◽  
Google Play

BACKGROUND The over 11 million care partners in the US who provide care to people living with Alzheimer’s disease and related dementias (ADRD) cite persistent and pervasive unmet needs related to all aspects of their caregiving role. The proliferation of mobile applications (apps) for care partners has potential to meet the care partners’ needs, but the quality of apps is unknown. OBJECTIVE The present study aimed to 1) evaluate the quality of publicly available apps for care partners of people living with ADRD and 2) identify design features of low- and high-quality apps to guide future research and app development. METHODS We searched the US Apple and Google Play app stores with the criteria that the app needed to be 1) available in US Google play or Apple app stores, 2) directly accessible to users “out of the box”, 3) primarily intended for use by an informal (family, friend) caregiver or caregivers of a person with dementia. The included apps were then evaluated using the Mobile App Rating Scale (MARS), which includes descriptive app classification and rating using 23 items across five dimensions: engagement, functionality, aesthetics, information, and subjective quality. Next, we computed descriptive statistics for each rating. To identify recommendations for future research and app development, we categorized rater comments on the score driving factors for each item and what the app could have done to improve the score for that item. RESULTS We evaluated 17 apps (41% iOS only, 12% Android only, 47% both iOS and Android). We found that on average, the apps are of minimally acceptable quality. Although we identified apps above and below minimally acceptable quality, many apps had broken features and were rated as below acceptable for engagement and information. CONCLUSIONS Minimally acceptable quality is likely insufficient to meet care partner needs. Future research should establish minimum quality standards across dimensions for mobile apps for care partners. The design features of high-quality apps we identified in this research can provide the foundation for benchmarking those standards.

scholarly journals Quality of sleep and health-related quality of life among health care professionals treating patients with coronavirus disease-19

2020 ◽  
pp. 002076402094280 ◽  
Author(s):  
Jelena Stojanov ◽  
Marina Malobabic ◽  
Goran Stanojevic ◽  
Milos Stevic ◽  
Vuk Milosevic ◽  
...  
Keyword(s):  
Quality Of Life ◽  
Health Care ◽  
Mental Status ◽  
Health Care Professionals ◽  
Short Form ◽  
Health Related Quality ◽  
Quality Of Sleep ◽  
Related Quality ◽  
Health Related

Background: Health care professionals exposed to coronavirus disease 2019 (COVID-19) are facing high levels of stress. Aim: The aim was to evaluate the quality of sleep (QoS) and health-related quality of life (HRQoL), among health care professionals treating patients with COVID-19, as well as quantifying the magnitude of symptoms of depression and levels of anxiety. Methods: We included 201 health care professionals in a cross-sectional, web-based study by applying 7-item Generalized Anxiety Disorder (GAD-7) Scale, Zung Self-rating Depression Scale, 36-item Health Survey of the Medical Outcomes Study Short Form (SF36), Pittsburgh Sleep Quality Index (PSQI) and additional survey constructed for the purpose of the study. Results: Health care workers who treated COVID-19 patients were more afraid of becoming infected or of transmitting the infection to a family member with a significantly low self-assessment of their mental status. Poor QoS and HRQoL correlated with high health anxiety and severe depressive symptoms and several demographic characteristics. Multiple linear regression analysis showed that higher scores on GAD-7 (beta = .71, p < .01) and lower scores on mental health (MH) subscale on SF36 questionnaire (beta = –.69; p < .01) were independent predictors of the higher PSQI score (adjusted R2 = .61, p < .01 for overall model). Higher scores on GAD-7 (beta = .68, p < .01) and worse self-perceived mental status (beta = .25; p < .05) were independent predictors of the lower SF36 scores (adjusted R2 = .73, p < .01 for overall model). Conclusion: The major MH burden of health care professionals treating infected patients during the COVID-19 pandemic indicates that they need psychological support.

scholarly journals Transition from pandemic to infodemic: an analysis of Turkish-language COVID-19 YouTube videos

2021 ◽  
Author(s):  
Fatih Cakmak ◽  
Seda Ozkan ◽  
Afsin Ipekci ◽  
Altug Kanbakan ◽  
Turker Demirtakan ◽  
...  
Keyword(s):  
Health Care ◽  
Health Care Professionals ◽  
Medical Information ◽  
Video Content ◽  
Global Quality ◽  
Content Index ◽  
Health Related ◽  
Youtube Videos ◽  
Turkish Language

Background: YouTube can be a powerful educational tool for the dissemination of health information. However, if uploaded health-related videos are inaccurate, it can mislead, create confusion and generate panic. Aims: This study aimed to determine the success of the most-watched Turkish-language COVID-19 YouTube videos regarding information and guidance on the disease for the public. The secondary aim of this study was to evaluate the accuracy and quality of such video content. Methods: The study was conducted during May 2020 and analysed 133 videos. The length of the videos, the number of likes and dislikes, comments and views, how long they have been on YouTube, Medical Information and Content Index (MICI) Score, mDISCERN scores, global quality scores, and the source and target audiences of the videos were all determined. Results: The average MICI Scores of videos was 2.48±3.74 and the global quality scores was 1.27±0.64. When MICI Scores were compared between video sources, the scores of academic hospitals and government videos were significantly higher. The global quality scores of videos from news agencies and independent users was significantly lower (P < 0.001). Among the targeted videos, only the global quality scores of the videos made for health-care workers were found to be significantly higher. Conclusion: Health-care professionals should upload more videos to improve the quality of health-related video content available on YouTube. Accompanied by evidence-based information, the issues of diagnosis, ways of transmission, prevention and treatment of diseases should be emphasized.

scholarly journals Content and Quality of Infant Feeding Smartphone Apps: Five-Year Update on a Systematic Search and Evaluation

10.2196/17300 ◽  
2020 ◽  
Vol 8 (5) ◽  
pp. e17300 ◽  
Author(s):  
Heilok Cheng ◽  
Alison Tutt ◽  
Catherine Llewellyn ◽  
Donna Size ◽  
Jennifer Jones ◽  
...  
Keyword(s):  
Health Care ◽  
Health Information ◽  
Infant Feeding ◽  
Rating Scale ◽  
Mobile App ◽  
Formula Feeding ◽  
Evidence Based ◽  
Smartphone Apps ◽  
Low Coverage

Background Parents use apps to access information on child health, but there are no standards for providing evidence-based advice, support, and information. Well-developed apps that promote appropriate infant feeding and play can support healthy growth and development. A 2015 systematic assessment of smartphone apps in Australia about infant feeding and play found that most apps had minimal information, with poor readability and app quality. Objective This study aimed to systematically evaluate the information and quality of smartphone apps providing information on breastfeeding, formula feeding, introducing solids, or infant play for consumers. Methods The Google Play store and Apple App Store were searched for free and paid Android and iPhone Operating System (iOS) apps using keywords for infant feeding, breastfeeding, formula feeding, and tummy time. The apps were evaluated between September 2018 and January 2019 for information content based on Australian guidelines, app quality using the 5-point Mobile App Rating Scale, readability, and suitability of health information. Results A total of 2196 unique apps were found and screened. Overall, 47 apps were evaluated, totaling 59 evaluations for apps across both the Android and iOS platforms. In all, 11 apps had affiliations to universities and health services as app developers, writers, or editors. Furthermore, 33 apps were commercially developed. The information contained within the apps was poor: 64% (38/59) of the evaluations found no or low coverage of information found in the Australian guidelines on infant feeding and activity, and 53% (31/59) of the evaluations found incomplete or incorrect information with regard to the depth of information provided. Subjective app assessment by health care practitioners on whether they would use, purchase, or recommend the app ranged from poor to acceptable (median 2.50). Objective assessment of the apps’ engagement, functionality, aesthetics, and information was scored as acceptable (median 3.63). The median readability score for the apps was at the American Grade 8 reading level. The suitability of health information was rated superior or adequate for content, reading demand, layout, and interaction with the readers. Conclusions The quality of smartphone apps on infant feeding and activity was moderate based on the objective measurements of engagement, functionality, aesthetics, and information from a reliable source. The overall quality of information on infant feeding and activity was poor, indicated by low coverage of topics and incomplete or partially complete information. The key areas for improvement involved providing evidence-based information consistent with the Australian National Health and Medical Research Council’s Infant Feeding Guidelines. Apps supported and developed by health care professionals with adequate health service funding can ensure that parents are provided with credible and reliable resources.

scholarly journals Content and Quality of Infant Feeding Smartphone Apps: Five-Year Update on a Systematic Search and Evaluation (Preprint)

2019 ◽  
Author(s):  
Heilok Cheng ◽  
Alison Tutt ◽  
Catherine Llewellyn ◽  
Donna Size ◽  
Jennifer Jones ◽  
...  
Keyword(s):  
Health Care ◽  
Health Information ◽  
Infant Feeding ◽  
Rating Scale ◽  
Mobile App ◽  
Formula Feeding ◽  
Evidence Based ◽  
Smartphone Apps ◽  
Low Coverage

BACKGROUND Parents use apps to access information on child health, but there are no standards for providing evidence-based advice, support, and information. Well-developed apps that promote appropriate infant feeding and play can support healthy growth and development. A 2015 systematic assessment of smartphone apps in Australia about infant feeding and play found that most apps had minimal information, with poor readability and app quality. OBJECTIVE This study aimed to systematically evaluate the information and quality of smartphone apps providing information on breastfeeding, formula feeding, introducing solids, or infant play for consumers. METHODS The Google Play store and Apple App Store were searched for free and paid Android and iPhone Operating System (iOS) apps using keywords for infant feeding, breastfeeding, formula feeding, and tummy time. The apps were evaluated between September 2018 and January 2019 for information content based on Australian guidelines, app quality using the 5-point Mobile App Rating Scale, readability, and suitability of health information. RESULTS A total of 2196 unique apps were found and screened. Overall, 47 apps were evaluated, totaling 59 evaluations for apps across both the Android and iOS platforms. In all, 11 apps had affiliations to universities and health services as app developers, writers, or editors. Furthermore, 33 apps were commercially developed. The information contained within the apps was poor: 64% (38/59) of the evaluations found no or low coverage of information found in the Australian guidelines on infant feeding and activity, and 53% (31/59) of the evaluations found incomplete or incorrect information with regard to the depth of information provided. Subjective app assessment by health care practitioners on whether they would use, purchase, or recommend the app ranged from poor to acceptable (median 2.50). Objective assessment of the apps’ engagement, functionality, aesthetics, and information was scored as acceptable (median 3.63). The median readability score for the apps was at the American Grade 8 reading level. The suitability of health information was rated superior or adequate for content, reading demand, layout, and interaction with the readers. CONCLUSIONS The quality of smartphone apps on infant feeding and activity was moderate based on the objective measurements of engagement, functionality, aesthetics, and information from a reliable source. The overall quality of information on infant feeding and activity was poor, indicated by low coverage of topics and incomplete or partially complete information. The key areas for improvement involved providing evidence-based information consistent with the Australian National Health and Medical Research Council’s Infant Feeding Guidelines. Apps supported and developed by health care professionals with adequate health service funding can ensure that parents are provided with credible and reliable resources.

Sign in / Sign up

Export Citation Format

Share Document