A Work-Centered Approach to System User-Evaluation

Author(s):  
Emilie M. Roth ◽  
Ann M. Bisantz ◽  
Xiaomei Wang ◽  
Tracy Kim ◽  
Aaron Z. Hettinger

New systems are often based on optimistic assumptions of how they will improve human performance. In the cognitive engineering tradition, these assumed benefits are regarded as hypotheses that need to be tested. An important element of a system user evaluation is to determine whether the hypothesized benefits are realized. Evaluation may also uncover unsupported aspects of performance or unanticipated side-effects of introducing the new technology that need to be addressed. We present a work-centered approach to user evaluation intended to meet these objectives, focusing specifically on design of tailored user-feedback questionnaires ( work-centered questionnaires) that are intended to be diagnostic of how specific system elements do, or do not, support work. We summarize two recent evaluation studies we have conducted that illustrate our approach and the diagnostic power of work-centered questionnaires. We discuss how the goals and approach of a work-centered evaluation differ from more traditional approaches to usability evaluation that emphasize the use of standardized questionnaires and broad assessments of usability.

2017 ◽  
Author(s):  
Zhemin Zhou ◽  
Nina Luhmann ◽  
Nabil-Fareed Alikhan ◽  
Christopher Quince ◽  
Mark Achtman

AbstractExploring the genetic diversity of microbes within the environment through metagenomic sequencing first requires classifying these reads into taxonomic groups. Current methods compare these sequencing data with existing biased and limited reference databases. Several recent evaluation studies demonstrate that current methods either lack sufficient sensitivity for species-level assignments or suffer from false positives, overestimating the number of species in the metagenome. Both are especially problematic for the identification of low-abundance microbial species, e. g. detecting pathogens in ancient metagenomic samples. We present a new method, SPARSE, which improves taxonomic assignments of metagenomic reads. SPARSE balances existing biased reference databases by grouping reference genomes into similarity-based hierarchical clusters, implemented as an efficient incremental data structure. SPARSE assigns reads to these clusters using a probabilistic model, which specifically penalizes non-specific mappings of reads from unknown sources and hence reduces false-positive assignments. Our evaluation on simulated datasets from two recent evaluation studies demonstrated the improved precision of SPARSE in comparison to other methods for species-level classification. In a third simulation, our method successfully differentiated multiple co-existing Escherichia coli strains from the same sample. In real archaeological datasets, SPARSE identified ancient pathogens with ≤ 0.02% abundance, consistent with published findings that required additional sequencing data. In these datasets, other methods either missed targeted pathogens or reported non-existent ones. SPARSE and all evaluation scripts are available at https://github.com/zheminzhou/SPARSE.


2005 ◽  
Vol 187 (S48) ◽  
pp. s91-s97 ◽  
Author(s):  
Jane Edwards ◽  
Meredith G. Harris ◽  
Swagata Bapat

BackgroundProviding specialised services to individuals experiencing first-episode psychosis (FEP) is a relatively new endeavour.AimsTo overview developing services for newly diagnosed cases of FEP and the context in which they develop.MethodThis paper describes five model multi-element FEP programmes, outlines recent evaluation studies of FEP services, discusses current evidence gaps relating to the evaluation of complex interventions and specific interventions for FEP and illustrates attempts to examine aspects of clinical work practised at the Early Psychosis Prevention and Intervention Centre (EPPIC) in Melbourne, Australia.ResultsConsiderable progress has been made in terms of influencing practice in the assessment and treatment of early psychosis.ConclusionsThere is need for quality clinical and research efforts to inform and accelerate progress in this burgeoning field.


1983 ◽  
Vol 27 (8) ◽  
pp. 704-708
Author(s):  
Woodrow Barfield ◽  
Gavriel Salvendy ◽  
Sheue-Ling Hwang

This paper deals with those areas of human factors research which relate to industrial machine actuator devices. Power press actuators are emphasized for example purposes, but the material could be generalizable to other industrial machines. The actuator control is shown to be a critical part of the industrial man-machine system. If the designed actuator device does not adequately match human performance capabilities with machine performance capabilities for the system in use, then safety problems can ensue including injury, liability for injury, or damage to equipment. The introduction of new technology to the design of actuator devices is delineated and recommendations for future research are discussed.


Author(s):  
Gunilla Bradley

Creating the global village very much attach to a macroergonomic approach to work life issues theoretically, empirically and in action. The underlying paradigm in macroergonomics is very much the same in various countries and may only differ regarding terminology. In this symposium we will enjoy the experiences from the North American pioneers and one Japanese pioneer. The world around us is being transformed during the past few years. Technical development, primarily within the information and communication area, is both a prerequisite and a driving force for the internationalization and globalization of economy, trade, science and culture. At the same time on another level the development of new technology, advanced and widespread use of computer technology and telecommunication and the convergence of these technologies have a profound impact on organizational and quality aspects of our lives today both at work and at home. New application areas within information industry are emerging in the integration of communications, computer and media. The span of control for human intervention might seem to shrink when we take use of more integrated technical solutions. On the other hand the structures in work life built during the industrialized era seem to break down and new organizational structures characterized by networking and virtual organizations, are offering new forms for human influence and interventions. Education and training have a key role for taking advantage of the opportunities and foreseeing the risks. Recent discussions about the role of ergonomists have centered around our role as change agents who are also trained as scientists. This scientist/practitioner dual identity can cause difficulties in implementing ergonomics. The purpose of this panel is to present macroergonomic strategies for creating change. Panelists will present structural, strategic, and participatory macroergonomic approaches. These approaches provide a larger framework from which we can understand our role, our research, ergonomic knowledge, and how to implement meaningful change in organizations. Without this overarching framework, ergonomics remains an interesting but impractical means for example in the field of transferring technology. Moreover these implementation issues confront people everyday in the expansion of organizations and technology across an ever shrinking global village. With or without ergonomic input, people and organizations are making changes every day that profoundly influence human performance and behavior. There are many examples that these change efforts fail. The papers will argue that these failures are rooted in the technical, structural and process focus of these changes, without considering the human. The macroergonomic strategies to be presented suggest routes on a larger road map to create meaningful change. This is even more important in the near future, when goals of sustainable environments are set. When capital and technology as well as people become increasingly and rapidly mobile at the global level, the strength of the local competence and know-how environment will become of greater significance for the development and prosperity of the individual countries and for the living conditions of their citizens. Competence to cope with the rapid pace of change will become ever more essential. Global villages are being created in the countryside around the world, with the use of information technology. Electronic marketplaces are used to strengthen small enterprises, to create new professional roles, and to strengthen the citizen's role. Adapting a perspective of macroergonomics is crucial. Guiding principles for a national information technology investment are formulated both in USA and Europe where also questions are brought up regarding information support for innovative industry, distance education, decentralized work and distributed civic information.


2004 ◽  
Vol 13 (1-2) ◽  
pp. 69-107 ◽  
Author(s):  
Martin Lefebvre ◽  
Marc Furstenau

Abstract In this essay we consider some of the effects of digital film editing technology on editing. Through an analysis of this technology, as well as DVD technology, we examine the impact these new interfaces have on the film experience. In addition, a study of the effects of compositing—understood here as digital montage par excellence—permits us to dig deeper into its impact on traditional approaches to film style, as well as to question the anxiety stemming from the new technology, given that many critics today argue that digitization has obliterated film’s indexicality.


Author(s):  
Christine M. O'Keefe ◽  
Ross Stewart Sparks ◽  
Damien McAullay ◽  
Bronwyn Loong

A remote analysis system addresses the challenge of enabling the use of confidential or private data while maintaining standards of confidentiality and privacy. Traditional approaches typically involve reducing the risk of disclosure by modifying or confidentialising data before releasing it to users. In contrast, a remote analysis system enables users to submit statistical queries and receive output without direct access to the data. A remote analysis system may involve confidentialisation of the underlying data itself or the system outputs, or both. In this paper we discuss the implementation of a remote analysis system enabling survival analysis. In this system the underlying data are not confidentialised, although for some analyses a random sample of the data is used, and the system outputs are modified to protect confidentiality and privacy. We describe confidentiality objectives for the system outputs, and describe measures for achieving them. To illustrate the effect of the methods, we provide a comprehensive example comparing confidentialised output with traditional output for a range of common survival analyses. We believe that the confidentialised output of the remote analysis system for survival analysis as described in this paper is still useful for survival analysis in some situations, provided the user understands the confidentialisation process and its potential impact. If the remote analysis system user requires more detailed information such as outlier values, event times and/or and standard errors, then they would need to apply for access to the underlying data.


Author(s):  
Benjamin James Knox ◽  
Ricardo G Lugo ◽  
Kirsi Helkala ◽  
Stefan Sütterlin

Governance of cyberpower from a military perspective are focused on the efforts to control and influence events occurring in cyberspace. For the Norwegian Defence, this means educating cyber engineers, responsible for governing cyberpower effects, beyond technical skills and competencies. To match the complexity of modern warfighting necessitates adaptive high-order thinking skills. Building on earlier cognitive engineering and human factors research in cyber defence this article suggests how Slow Education has the potential to improve cognitive performance among cyber cadets. Slow techniques were applied to 37 cyber cadets during a three-year bachelor programme at the Norwegian Defence Cyber Academy. The quantitative data for this study was gathered during a two-week Cyber Defence Exercise. Combining and applying a novel pedagogic method with psychological techniques suggests reflective pondering, self-regulation and metacognition as being associated with cognitive agility. This study helps develop and make metrics available that are suitable to evaluate human performance in cyber defence.


Author(s):  
Christopher W. Myers

An important goal of training systems research is the ability to train teams to criterion while simultaneously minimizing training resources. One promising approach is to develop synthetic agents that act as full-fledged members of a team. Five experts will highlight successes, failures, and continuing challenges associated with the development, validation, and deployment of synthetic agents as full-fledged teammates. The panel will provide an intimate look “under the hood” of synthetic agents, describe what each has found useful for developing a synthetic teammate that “plays well with others,” and discuss the key roadblocks that must be overcome for the further inclusion of synthetic teammates within human training systems. The lessons learned from these panelists will be of value to those interested in cognitive engineering and human performance modeling.


Sign in / Sign up

Export Citation Format

Share Document