scholarly journals How Teams Communicate about the Quality of ML Models: A Case Study at an International Technology Company

2021 ◽  
Vol 5 (GROUP) ◽  
pp. 1-24
Author(s):  
Jumana Almahmoud ◽  
Robert DeLine ◽  
Steven M. Drucker
Keyword(s):  
User Experience ◽  
General Term ◽  
Mitigation Strategies ◽  
Mixed Methods Study ◽  
Team Members ◽  
Software Products ◽  
Crucial Component ◽  
Technical Issues

Machine learning (ML) has become a crucial component in software products, either as part of the user experience or used internally by software teams. Prior studies have explored how ML is affecting development team roles beyond data scientists, including user experience designers, program managers, developers and operations engineers. However, there has been little investigation of how team members in different roles on the team communicate about ML, in particular about the quality of models. We use the general term quality to look beyond technical issues of model evaluation, such as accuracy and overfitting, to any issue affecting whether a model is suitable for use, including ethical, engineering, operations, and legal considerations. What challenges do teams face in discussing the quality of ML models? What work practices mitigate those challenges? To address these questions, we conducted a mixed-methods study at a large software company, first interviewing15 employees in a variety of roles, then surveying 168 employees to broaden our understanding. We found several challenges, including a mismatch between user-focused and model-focused notions of performance, misunderstandings about the capabilities and limitations of evolving ML technology, and difficulties in understanding concerns beyond one's own role. We found several mitigation strategies, including the use of demos during discussions to keep the team customer-focused.

scholarly journals A Study of Telemedical Applications Based on User Experience

2015 ◽  
Vol 43 (1) ◽  
pp. 195-210
Author(s):  
Joanna Rybka ◽  
Małgorzata Plechawska-Wójcik ◽  
Tomasz Tokarek
Keyword(s):  
User Experience ◽  
Evaluation Criteria ◽  
Ease Of Use ◽  
Easy Access ◽  
End User ◽  
Software Products ◽  
Usability Problems ◽  
Expert Analysis

Abstract The popularity of telemedical applications has been increasing noticeably in recent years. Easy access to a variety of software products makes contemporary users concentrate not only on functionalities but also on the design of the Graphical User Interface (GUI) – its usefulness, ease of use, and intuitiveness. The objective of this study was to evaluate the GUI quality of selected telemedical applications dedicated to depressed patients and doctors providing medical care for this group of patients. Another aim of the paper was to propose and check assessment criteria applied to the evaluation experiment. To achieve this objective, the case study was based on two methods: expert analysis and cognitive walkthrough. The expert analysis study was conducted on three groups of users: user experience specialists, patients, and clinicians. Although the expert analysis method is usually dedicated to GUI design specialists, in this study, we also applied it to patient and physician groups. The results showed that there were no statistical differences between assessments carried out by those three groups of users. Applied testing criteria were revised with Nielsen’s heuristics. The proposed criteria helped to uncover many usability problems in several different areas during user tests. Further studies might be performed in order to strengthen accuracy and for completion of the proposed expert analysis evaluation criteria. However, our results show that the criteria we used seem to be robust enough to apply to both expert and end-user evaluations. Furthermore, multiple evaluation based on two different methods presented with better results, allowing not only problem identification but also verification.

Case Study

2011 ◽  
pp. 215-236
Author(s):  
Judith Kessler
Keyword(s):  
Success Factors ◽  
Collaborative Writing ◽  
Online Help ◽  
Team Members ◽  
Early Adopters ◽  
Software Products ◽  
Collaborative Processes ◽  
Information Models ◽  
Efficiency Increase

The Technical Publications team at Sybase, Inc. maintains many thousands of pages of user documentation and online help topics for a diverse set of software products. Writing teams work in nine locations around the globe; a given project often involves writers from multiple locations. To achieve greater efficiency, increase opportunities for reuse, and improve user experience, the department is moving to the Darwin Information Typing Architecture (DITA) from a variety of source formats. Early adopters realized the need for more detailed information models for several types of content than required by the DITA standard. This chapter discusses why models are a critical component to successful collaborative writing, especially for topic-oriented content. It then describes the collaborative processes and tools by which Sybase® Technical Publications team members propose, evaluate, develop, test, and enforce new content models, challenges encountered, and key success factors.

scholarly journals Applying CMMI Best Practices to Improve Processes

2019 ◽  
Vol 292 ◽  
pp. 01065
Author(s):  
Sussy Bayona-Oré ◽  
Josselyn Chamilco ◽  
Dayvis Perez
Keyword(s):  
Customer Satisfaction ◽  
Best Practices ◽  
Educational Institution ◽  
Verification And Validation ◽  
Competitive Market ◽  
Software Products ◽  
Management Processes ◽  
Requirements Verification

Today more and more organizations are demanding quality software products. To cover this demand, companies that provide computer solutions focus on the quality of their processes to generate value in a competitive market. This article presents the case study of an educational institution that decides to use the best practices of CMMI to improve the areas of management processes of requirements, verification and validation. The results of the improvement process show that the practices were improved, the defects decreased and the customer satisfaction was improved.

A Process of Multidisciplinary Team Communication to Individualize Stroke Rehabilitation of an 84-Year-Old Stroke Patient

2016 ◽  
Vol 17 (2) ◽  
pp. 97-104 ◽  
Author(s):  
Fukumi Hiragami ◽  
Shogo Hiragami ◽  
Yasuo Suzuki
Keyword(s):  
Multidisciplinary Team ◽  
Treatment Options ◽  
Communication Process ◽  
Knowledge Assessment ◽  
Team Members ◽  
Mobility Impairments ◽  
Patient Families ◽  
Initial Meeting

Previously, we have used a multidisciplinary team (MDT) approach to individualize rehabilitation of very old stroke patients as a means to establish intervention points for addressing impaired activities of daily living (ADL). However, this previous study was limited because of a lack in describing the communication process over time. This case study characterized the MDT communication process in the rehabilitation of an 84-year-old patient over the course of 15 weeks. The MDT consisted of 3 nurses, 1 doctor, 6 therapists, and the patient/families. Meetings (15 minutes each) were held at 4, 6, 8, and 15 weeks following the patient’s admission. To individualize the rehabilitation, the communication process involved gaining knowledge about ADL impairments, sharing assessments, providing treatment options, and reflecting on desired treatment outcomes—a process termed KATR. The knowledge, assessment, treatment, and reflection (KATR) process established intervention points focusing on specific ADL impairments. The team members focused the interventions on the impaired ADL identified in the KATR process, and individualized rehabilitation was generated from the MDT information-sharing knowledge. In the initial meeting (Week 4), intervention points derived from the KATR process focused on rehabilitation of self-care impairments. These impairments improved by Week 15. By the last meeting, the MDT intervention points focused on mobility impairments. Having an organized communication process (i.e., KATR) facilitates individualization of rehabilitation without lengthy and frequent MDT meetings and enhances the quality of rehabilitation after a stroke.

scholarly journals SOFTWARE PRODUCT AND PROCESS ASSESSMENT THROUGH PROFILE-BASED EVALUATION

2003 ◽  
Vol 13 (05) ◽  
pp. 495-512 ◽  
Author(s):  
MAURIZIO MORISIO ◽  
IOANNIS STAMELOS ◽  
ALEXIS TSOUKIAS
Keyword(s):  
Evaluation Model ◽  
Process Assessment ◽  
Process Maturity ◽  
Single View ◽  
Software Products ◽  
Software Product ◽  
Product And Process ◽  
Definition Of

Software entities (software products or processes) are characterized by many attributes, each one in its turn can be measured by one or more measures. In several cases the software entities have to be evaluated as a whole, thus raising the problem of aggregating measures to give an overall, single view on the software entity. This paper presents a method to aggregate measures, which works by comparing the product/process with predefined, ideal entities, or profiles. Profiles are defined starting from ranges of values on measures of attributes. The method is based on two main phases, namely definition of the evaluation model and application of the evaluation model. It is presented through a simplified case study that deals with evaluating the level of quality of an asset to decide whether to accept it in a reuse repository. A plausible way of how the method could be applied to process maturity assessment is also provided. The advantages of the method are that it allows using ordinal scales, while it deals explicitly with preferences expressed, implicitly or explicitly, by the evaluator.

scholarly journals An exploratory study of users’ experiences with e-participation: a case study of Malaysia

2019 ◽  
Vol 15 (3) ◽  
pp. 1138
Author(s):  
Nasrah Hassan Basri ◽  
Wan Adilah Wan Adnan ◽  
Hanif Baharin
Keyword(s):  
User Experience ◽  
Exploratory Study ◽  
Think Aloud ◽  
New Approach ◽  
The Core ◽  
Core Elements ◽  
Technical Issues ◽  
Government Website

The need to encourage citizens’ adoption of e-participation services has prompted an expanding enthusiasm for the evaluation of e-participation websites. The achievement of e-participation websites depends intensely on how well it is perceived by the users. E-participation is a relatively new approach, so it is important to evaluate it deliberately, in order to comprehend it better, obtain more knowledge about it and distinguish the preferences and advantages it offers, and not to overlook its disadvantages so that we can improve it. Concerning evaluation, many researchers have concentrated on infrastructure and technical issues without giving more attention to user experience as one of the core elements impacting the success or failure of e-participation websites. This paper evaluated e-participation module that located in Malaysia e-government website by utilizing think-aloud method.  Think-aloud was being utilized to gauge the perceptions of citizens in their use of e-participation.

scholarly journals Hold-Up Problems in International Gas Trade: A Case Study

Energies ◽  
2021 ◽  
Vol 14 (16) ◽  
pp. 4984
Author(s):  
Guych Nuryyev ◽  
Tomasz Korol ◽  
Ilia Tetin
Keyword(s):  
Natural Gas ◽  
Political Change ◽  
Mitigation Strategies ◽  
Supply Disruptions ◽  
Technical Issues ◽  
Number Of Parties ◽  
Gas Supply ◽  
Over Time

The infrastructure required for international natural gas trade is considerable, which often leads to hold-up problems and supply disruptions. This study discusses disruptions of gas supply from Algeria, Indonesia, Russia, and Turkmenistan since the early 1980s. The novelty of this study is its focus on the issues related to transit countries, which are rarely considered in the literature. The results of the study classify supply disruptions into six types, show the evolution of supply disruptions over time, and discuss mitigation strategies. The six types of disruptions include political change, price demands, debts, technical issues, transit fees, theft of gas. The evolution of the disruptions shows that the issues related to transit countries have become more frequent in the last two decades. Mitigation strategies tailored to transit countries include using an international organisation, designing contracts with price mechanisms that might reduce the possibility of disputes and reducing the number of parties involved in the trade.

scholarly journals Modeling User Psychological Experience and Case Study in Online E-learning

2015 ◽  
Vol 10 (6) ◽  
pp. 53 ◽  
Author(s):  
Xiyuan Wu ◽  
Min Liu ◽  
Qinghua Zheng ◽  
Yunqiang Zhang ◽  
Haifei Li
Keyword(s):  
User Experience ◽  
Subjective Experience ◽  
Analytic Hierarchy ◽  
Psychological Experience ◽  
E Learning ◽  
User Psychological ◽  
Hierarchy Process ◽  
Interview Questionnaire

In the post WWW era, the research of e-learning focuses on facilitating intelligent and proactive services for learners. The quality of user experience determines whether e-learning services would be accepted by learners. However, many researchers traditionally focus on the effectiveness of computer systems or the accuracy of algorithms themselves rather than on user-centric psychological experience. How to model and evaluate user experience taking into account user psychological and cognitive properties are challenging research topics. There are some traditional methods typically proposed to evaluate users’ psychological experience, such as interview, questionnaire etc. They are qualitative and easy to conduct but need more time and resource. And they are liable to subjective views. Based on user web log data, the current paper presents a quantitative approach of modeling user psychological experience in the context of intelligent e-learning. The properties and elements, which affect user experience, are analyzed and quantified. The holistic user experience is quantified through the fusion of analytic hierarchy process (AHP) and Delphi methods. A case study, at a university in China, is conducted for diagnosing whether the result of the proposed approach can be uniform with user subjective experience, and indicates that the proposed approach is effective and complements existing user experience research in intelligent e-learning.

scholarly journals User experience evaluation method based on online product reviews

2021 ◽  
pp. 1-15
Author(s):  
Fangmin Cheng ◽  
Suihuai Yu ◽  
Shengfeng Qin ◽  
Jianjie Chu ◽  
Jian Chen
Keyword(s):  
New Product Development ◽  
User Experience ◽  
Evaluation Method ◽  
Information Source ◽  
New Product ◽  
Product Reviews ◽  
Vacuum Cleaner ◽  
Online Product Reviews

Evaluating the quality of the user experience (UX) of existing products is important for new product development. Conventional UX evaluation methods, such as questionnaire, have the disadvantages of the great subjective influence of investigators and limited number of participants. Meanwhile, online product reviews on e-commerce platforms express user evaluations of product UX. Because the reviews objectively reflect the user opinions and contain a large amount of data, they have potential as an information source for UX evaluation. In this context, this study explores how to evaluate product UX through using online product reviews. A pilot study is conducted to define the key elements of a review. Then, a systematic method of product UX evaluation based on reviews is proposed. The method includes three parts: extraction of key elements, integration of key elements, and quantitative evaluation based on rough number. The effectiveness of the proposed method is demonstrated by a case study using reviews of a wireless vacuum cleaner. Based on the proposed method, designers can objectively evaluate the UX quality of existing products and obtain detailed suggestions for product improvement.

scholarly journals A Data-Driven Approach for Video Game Playability Analysis Based on Players’ Reviews

Information ◽  
2021 ◽  
Vol 12 (3) ◽  
pp. 129
Author(s):  
Xiaozhou Li ◽  
Zheying Zhang ◽  
Kostas Stefanidis
Keyword(s):  
Video Games ◽  
Video Game ◽  
Binary Classification ◽  
Game Studies ◽  
Data Driven ◽  
Software Products ◽  
Data Driven Approach ◽  
Definition Of

Playability is a key concept in game studies defining the overall quality of video games. Although its definition and frameworks are widely studied, methods to analyze and evaluate the playability of video games are still limited. Using heuristics for playability evaluation has long been the mainstream with its usefulness in detecting playability issues during game development well acknowledged. However, such a method falls short in evaluating the overall playability of video games as published software products and understanding the genuine needs of players. Thus, this paper proposes an approach to analyze the playability of video games by mining a large number of players’ opinions from their reviews. Guided by the game-as-system definition of playability, the approach is a data mining pipeline where sentiment analysis, binary classification, multi-label text classification, and topic modeling are sequentially performed. We also conducted a case study on a particular video game product with its 99,993 player reviews on the Steam platform. The results show that such a review-data-driven method can effectively evaluate the perceived quality of video games and enumerate their merits and defects in terms of playability.

Sign in / Sign up

Export Citation Format

Share Document