Is It Time to Go Positive? Assessing the Positively Worded System Usability Scale (SUS)

2020 ◽  
pp. 001872081988155 ◽  
Author(s):  
Philip Kortum ◽  
Claudia Ziegler Acemyan ◽  
Frederick L. Oswald
Keyword(s):  
Cognitive Load ◽  
Reliability And Validity ◽  
Subjective Assessment ◽  
Spoken Language ◽  
Viable Alternative ◽  
Psychometric Methods ◽  
System Usability Scale ◽  
System Usability ◽  
Subjective Usability ◽  
Larger Sample

Objective: The goal of the research presented in this paper was to determine if the positively worded System Usability Scale (SUS) can be used in place of the positively and negatively worded standard SUS instrument for the subjective assessment of usability, and whether the results found here replicate those of Sauro and Lewis. Background: Sauro and Lewis’ previous study found no evidence that responses to SUS items differed across the standard SUS and the modified, positively worded version of the SUS when participants assessed websites. This study replicates and extends this work by examining a large number of different systems with larger sample sizes to add to the generalizability of previous findings. Methods: So that participants could retrospectively assess 20 products, the standard SUS was administered to 268 participants and the positive SUS to 698 participants. SUS scores were computed and the data analyzed using psychometric methods to explore how the two versions of the SUS differed. Results: The standard and positive versions of the SUS yielded similar SUS scores. In addition, both versions of the scale demonstrated evidence in support of reliability and validity. Conclusion: Either version of the SUS can be used with confidence to measure subjective usability. Furthermore, the scores generated from both versions of the SUS can be directly compared. Applications: In situations where cognitive load, participants’ spoken language, or item consistency with other surveys being given may be a factor, the positive SUS is a viable alternative to the standard SUS.

The Effect of Cognitive Fatigue on Subjective Usability Scores

2017 ◽  
Vol 61 (1) ◽  
pp. 1461-1465 ◽  
Author(s):  
Ian Robertson ◽  
Philip Kortum
Keyword(s):  
Subjective Assessment ◽  
The Other ◽  
Cognitive Fatigue ◽  
Usability Assessment ◽  
System Usability Scale ◽  
Main Effect ◽  
System Usability ◽  
Subjective Usability ◽  
The Impact ◽  
True Measure

Cognitive fatigue is a known factor in errors and major accidents. What is unknown is the impact that cognitive fatigue might have on the subjective assessment of usability. If users are regularly cognitively depleted when using a given system, and fatigue makes a difference in their usability assessment, then a true measure of usability would be derived only by testing users in that fatigued state. In this study, forty-three participants voted using twelve prototype paper voting ballots. Half of the ballots were classified as being of low usability and the other half as having high usability. These ballots were randomly assigned and participants completed six ballots before a fatigue manipulation and six after the manipulation. Each ballot was rated by the user with the System Usability Scale (SUS), and pre- and post-fatigue means were compared. No significant main effect was found for fatigue and no interaction was present, although the data suggest the effect may have been masked due to the high variance in the usability of the systems assessed.

scholarly journals The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study (Preprint)

2018 ◽  
Author(s):  
Leming Zhou ◽  
Jie Bao ◽  
I Made Agus Setiawan ◽  
Andi Saptono ◽  
Bambang Parmanto
Keyword(s):  
Reliability And Validity ◽  
Correlation Coefficients ◽  
Internal Consistency Reliability ◽  
Mobile App ◽  
Target User ◽  
System Usability Scale ◽  
System Usability ◽  
Study Participants ◽  
Development And Validation ◽  
Mhealth Apps

BACKGROUND After a mobile health (mHealth) app is created, an important step is to evaluate the usability of the app before it is released to the public. There are multiple ways of conducting a usability study, one of which is collecting target users’ feedback with a usability questionnaire. Different groups have used different questionnaires for mHealth app usability evaluation: The commonly used questionnaires are the System Usability Scale (SUS) and Post-Study System Usability Questionnaire (PSSUQ). However, the SUS and PSSUQ were not designed to evaluate the usability of mHealth apps. Self-written questionnaires are also commonly used for evaluation of mHealth app usability but they have not been validated. OBJECTIVE The goal of this project was to develop and validate a new mHealth app usability questionnaire. METHODS An mHealth app usability questionnaire (MAUQ) was designed by the research team based on a number of existing questionnaires used in previous mobile app usability studies, especially the well-validated questionnaires. MAUQ, SUS, and PSSUQ were then used to evaluate the usability of two mHealth apps: an interactive mHealth app and a standalone mHealth app. The reliability and validity of the new questionnaire were evaluated. The correlation coefficients among MAUQ, SUS, and PSSUQ were calculated. RESULTS In this study, 128 study participants provided responses to the questionnaire statements. Psychometric analysis indicated that the MAUQ has three subscales and their internal consistency reliability is high. The relevant subscales correlated well with the subscales of the PSSUQ. The overall scale also strongly correlated with the PSSUQ and SUS. Four versions of the MAUQ were created in relation to the type of app (interactive or standalone) and target user of the app (patient or provider). A website has been created to make it convenient for mHealth app developers to use this new questionnaire in order to assess the usability of their mHealth apps. CONCLUSIONS The newly created mHealth app usability questionnaire—MAUQ—has the reliability and validity required to assess mHealth app usability.

The Relationship Between User Mouse-based Performance And Subjective Usability Assessments

2016 ◽  
Vol 60 (1) ◽  
pp. 1174-1178
Author(s):  
Philip Kortum ◽  
Claudia Ziegler Acemyan
Keyword(s):  
Performance Metrics ◽  
User Performance ◽  
Success Rates ◽  
Website Usability ◽  
Objective Metrics ◽  
System Usability Scale ◽  
System Usability ◽  
Effectiveness And Efficiency ◽  
Subjective Usability ◽  
The Relationship

Objective metrics, such as effectiveness and efficiency, are often considered to be the best website usability measurements. User performance metrics that can be collected remotely, such as mouse clicks and the distance the mouse has traveled show particular promise. However, no studies have demonstrated a direct relationship between subjective usability measures and these two objective user performance metrics. In this paper, thirty participants completed five different tasks of varying difficulty on a commercial website. Mouse clicks, the distance the mouse moved, success rates, and System Usability Scale (SUS) scores were collected for each task. Results showed that participants made fewer mouse clicks on tasks at which they were successful than on tasks they failed. Participants moved the mouse over twice as far on failed tasks as compared to successful tasks. The correlations between SUS scores and the two mouse-based measurements were remarkably strong.

Extraneous Factors in Usability Testing: Evidence of Decision Fatigue During Sequential Usability Judgments

2018 ◽  
Vol 62 (1) ◽  
pp. 1409-1413 ◽  
Author(s):  
Ian Robertson ◽  
Philip Kortum
Keyword(s):  
Usability Testing ◽  
Methodological Issues ◽  
Cognitive Fatigue ◽  
System Usability Scale ◽  
System Usability ◽  
Experimental Effect ◽  
Subjective Usability

Previous work has examined if cognitive fatigue caused subjects to negatively evaluate products on subjective usability measures but failed to find an effect. The authors suggested that methodological issues may have caused variance, which masked any experimental effect. This study tested that claim by eliminating the extra source of variance. Twenty-six participants voted using six prototype paper voting ballots. The ballots were randomly assigned, with three used before a fatigue manipulation and three after. All the ballots were of a similar usability profile as determined in previous studies. Each ballot was used and evaluated by the subjects using the System Usability Scale (SUS) and the pre-and post-fatigue means were compared, with no significant differences found. Although the reduction in variance did not affect the results, it was discovered that sequential usability judgments may result in decision fatigue. Subjective usability judgments may become increasingly negative with each additional judgment. Evidence from other experiments support this trend and further research may be warranted. Implications for usability researchers and practitioners are discussed.

Usability Assessments of Mobile Applications as a Function of Geographic Location

2019 ◽  
Vol 11 (1) ◽  
pp. 1-15
Author(s):  
Philip Kortum ◽  
Claudia Ziegler Acemyan
Keyword(s):  
United States ◽  
Mobile Applications ◽  
Geographic Location ◽  
The United States ◽  
Geographic Locations ◽  
Usability Tests ◽  
System Usability Scale ◽  
System Usability ◽  
Perceived Usability ◽  
Subjective Usability

Researchers recently demonstrated that subjective usability assessments of common products do not vary across geographic locations. That study did not directly address mobile applications, which are some of the most ubiquitous and geographically diverse systems in use in the United States today. To address this shortcoming, this article examined whether or not geographic location impacts the perceived usability of mobile applications by having 2,590 participants from different regions of the United States rate the usability of several mobile applications using the System Usability Scale. There was a lack of evidence to support statistically significant differences in usability scores across geographic locations for 95% of the mobile applications evaluated. This suggests that system usability assessments for mobile products do not differ across US locations. These findings further reinforce the idea that participants from any locale can be recruited for usability tests as long as all other critical demographic criteria are met.

scholarly journals Analisis Usability Sistem Komputerisasi Haji Terpadu Palembang Menggunakan Metode System Usability Scale (SUS)

2021 ◽  
Vol 2 (3) ◽  
pp. 125-137
Author(s):  
Abdurrahman Abdurrahman ◽  
Maria Ulfa
Keyword(s):  
Information System ◽  
Data Processing ◽  
Reliability And Validity ◽  
Administrative Staff ◽  
Validity And Reliability ◽  
Good Reliability ◽  
System Usability Scale ◽  
System Usability ◽  
Usability Analysis ◽  
Computerized System

The Palembang Integrated Hajj Computerized System which is currently used as an information system for Hajj and Umrah pilgrims, because the entire data processing process for the purpose of making Hajj documents such as passports, departure and return flights, banking and biodata of prospective pilgrims refers to the integrated computer system. Both internally and externally there has never been an evaluation, either at the time of design, manufacture or implementation. This research was conducted with the title "Usability Analysis of the Palembang Integrated Hajj Computerized System Using the System Usability Scale (SUS) Method". Aims to measure the level of usefulness of the Palembang Siskohat Application and test the validity and reliability using SPSS 24. System Usability Scale (SUS) With 16 questions as a benchmark for assessing the application. 30 respondents were selected among the administrative staff of the Hajj and Umrah pilgrims in Palembang. The results of the usability test showed the value of Learnibilty 4.68, Efficiency 4.46, Memorabilty 4.64, Error 3.34 Satisfaction 3.74, the average usability value was expressed by Very good reliability and validity on each variable indicate that the questionnaire for each variable has a fairly good level of reliability.

A Pilot Study on Extending the SUS Survey: Early Results

2021 ◽  
Vol 65 (1) ◽  
pp. 447-451
Author(s):  
Samantha B. Harper ◽  
Stephen L. Dorton
Keyword(s):  
Pilot Study ◽  
User Feedback ◽  
Popular Method ◽  
Early Results ◽  
System Usability Scale ◽  
System Usability ◽  
Subjective Usability

The System Usability Scale (SUS) is a popular method to measure the subjective usability of a system, due largely to the simplicity and rapidity of both collecting and analyzing data. A drawback is that the SUS generates a single unidimensional usability score from 0-100. Several researchers have amassed larger datasets across multiple projects to allow for analysis on additional methods to glean insights from the SUS survey. Along these lines, we investigate the practical value of extending the SUS survey with additional items such as open text responses, and test underlying assumptions of how SUS results are interpreted. We found that while a lower SUS score does generally correlate to a stronger desire to modify the system, people generally want to make modifications to a system regardless of its usability. Further, we found that the amount of user feedback related to modifications to a system provided predicted subjective usability ratings.

Measuring the Usability of Home Healthcare Devices Using Retrospective Measures

2017 ◽  
Vol 61 (1) ◽  
pp. 1281-1285 ◽  
Author(s):  
Meiyuzi Gao ◽  
Philip Kortum
Keyword(s):  
Medical Devices ◽  
Large Scale ◽  
Usability Testing ◽  
Home Healthcare ◽  
Home Setting ◽  
Retrospective Assessment ◽  
System Usability Scale ◽  
System Usability ◽  
Usability Metrics ◽  
Subjective Usability

As the trend of home-delivered healthcare grows, the number of healthcare devices being utilized in the home setting also increases greatly, but the usability of all these devices has not been systematically examined. While traditional in-lab usability testing is too time-consuming to make broad assessments of a large number of home healthcare devices, retrospective evaluation techniques using subjective usability surveys may be more promising because of their time and resource efficiency. However, the equivalence of data collected with these two techniques is not well understood for medical devices. This study collected usability data for multiple healthcare devices from 74 subjects in laboratory usability testing and 395 subjects using a retrospective assessment survey. Measures of overall subjective usability (measured using the System Usability Scale) and the ISO usability metrics of effectiveness, efficiency, and satisfaction were captured in both methods. Results show a consistency in satisfaction and overall subjective usability scores between the retrospective survey and laboratory testing, while measures of effectiveness and efficiency differed. These results suggest that retrospective surveys are capable of capturing the overall usability of a product by using the System Usability Scale metric. Such a finding should increase the confidence of researchers and practitioners who wish to use retrospective measures as a quick and economical technique in the large-scale assessment of medical devices. However, researchers should be cautious about measuring individual ISO usability metrics with retrospective surveys, as there are pronounced differences between lab-based usability testing and retrospective assessments.

Pengujian Usability Website Pondok Pesantren Qodratullah Menggunakan System Usability Scale

2020 ◽  
Vol 6 (1) ◽  
pp. 34-43
Author(s):  
Fitri Purwaningtias ◽  
Usman Ependi
Keyword(s):  
Evaluation Process ◽  
Boarding School ◽  
Educational Institutions ◽  
Prospective Students ◽  
A Value ◽  
System Usability Scale ◽  
System Usability ◽  
Grade Scale ◽  
Final Score ◽  
Group B

Website saat ini telah digunakan diberbagai jenis instansi termasuk instansi pendidikan seperti Pondok Pesantren Qodratullah. Saat ini website Pondok Pesantren Qodratullah menjadi tulang punggung dalam penyebaran informasi terkain pondok pesantren kepada wali santri, alumni, calon santri dan masyarakat luas. Mengingat pentingnya website bagi Pondok Pesantren Qodratullah maka perlu untuk dilakukan evaluasi apakah informasi yang diberikan dan website yang ada telah memiliki nilai kebergunaan bagi pengguna atau tidak. Untuk itu di dalam penelitian ini dilakukan evaluasi untuk melihat perspektif pengguna terhadap website. Prose evaluasi dilakukan dengan system usability scale dengan sepuluh instrumen sebagai pernyataan evaluasi. Hasil evaluasi menunjukkan bahwa website Pondok Pesantren Qodratullah mendapatkan nilai akhir 88. Nilai 88 berarti website Pondok Pesantren Qodratullah mendapatkan adjective rating yang excellence, grade scale tergolong kelompok B dan tingkat acceptability termasuk acceptable. The website is currently used in various types of institutions including educational institutions such as Qodratullah Islamic Boarding School. Currently the Qodratullah Islamic Boarding School website is the backbone in the dissemination of information about Islamic boarding schools to the guardians of students, alumni, prospective students and the wider community. Considering the importance of the website for the Qodratullah Islamic Boarding School, it is necessary to evaluate whether the information provided, and the existing website have a useful value for the user or not. For this reason, in this study an evaluation was conducted to see the user's perspective on the website. The evaluation process is carried out with a system usability scale with ten instruments as evaluation statements. Evaluation results show that the Qodratullah Islamic Boarding School website gets a final score of 88. A value of 88 means that the Qodratullah Islamic Boarding School website gets an adjective rating that excellence, grade scale belongs to group B and the level of acceptability is acceptable

Sign in / Sign up

Export Citation Format

Share Document