Trends in Human Factors Evaluation of Work Support Systems

2000 ◽  
Vol 44 (37) ◽  
pp. 561-564 ◽  
Author(s):  
Jens Rasmussen
Keyword(s):  
Deep Structure ◽  
Empirical Evaluation ◽  
User Involvement ◽  
Internal Constraints ◽  
Work Systems ◽  
Domain Experts ◽  
Work Support ◽  
Intimate Knowledge ◽  
Design Deficiencies ◽  
Interface Matching

Society is becoming increasingly integrated, effects of disturbances and design deficiencies propagate rapidly and widely, and work is becoming increasingly dynamic. Furthermore, organizations have to face an increasingly competitive environment, where success very likely is granted those operating at the border of established practice. This raises new requirements for the design and evaluation of work systems. For discretionary tasks in a dynamic work environment, a map of the deep structure of the work content is more effective than a route instruction. Rather than an interface matching the mental models of the users, the interface should make visible the affordances and the internal constraints of the workspace, including the boundaries of acceptable conditions. That is, the interface should induce a proper mental model. Design and evaluation then cannot be based only on user involvement and empirical evaluation tests. In addition, intimate knowledge of the deep structure of the workspace is necessary and domain experts must be involved in an analytical evaluation, which is outlined.

Designing for an Uncertain Future

2011 ◽  
pp. 249-264
Author(s):  
Enid Mumford
Keyword(s):  
User Involvement ◽  
Social Needs ◽  
Work Environments ◽  
Design Development ◽  
Work Systems ◽  
Human Needs ◽  
The People ◽  
Physical Environments ◽  
Work Community ◽  
Uncertain Future

The philosophy of this book is that problem solving and the management of change will be facilitated by participation. By participation is meant that all those affected by change, or their representatives, will be able to play some part in its definition, in agreeing strategies for its implementation and in evaluating its success. Most of the case studies discussed have been concerned with the positive aspects of change involving systems redesign. The stimulus here was usually the introduction of a new technical system. Early projects used participation as a means for assisting the introduction of a specific new system into a single or small number of departments. Later ones were larger and had more dramatic effects. All employees would now be involved, either from a particular function or from the company as a whole and, on occasion, from its environment. My objective has been to try and assist the creation of work systems which are both efficient and meet human needs for acceptable, stimulating and satisfying work environments. As well as user involvement, these systems will aim to have certain other desirable characteristics, namely, suitability, flexibility, complementarity and sustainability. Suitability requires them to provide a good fit with the technical and social needs of the work situation. In other words they do a good job in production and human terms. They are also flexible and able to cope easily with subsequent technical and organizational change. They complement existing systems and connect easily with these, and because they have these characteristics, they are sustainable and endure into the future. Their complementarity feature will also extend into the external environment so that industrial and clerical systems will mesh easily with environmental systems and create harmonious work, community and physical environments. But, most important, they are also democratic. All except the first enabled the people who would work with or be affected by them to have a role in their design, development and implementation.

scholarly journals Acquiring Integer Programs from Data

2019 ◽  
Author(s):  
Mohit Kumar ◽  
Stefano Teso ◽  
Luc De Raedt
Keyword(s):  
Operations Research ◽  
Feasible Solution ◽  
Empirical Evaluation ◽  
Integer Program ◽  
Combinatorial Problems ◽  
Benchmark Problems ◽  
Domain Experts ◽  
Integer Programs ◽  
The Core ◽  
Main Challenge

Integer programming (IP) is widely used within operations research to model and solve complex combinatorial problems such as personnel rostering and assignment problems. Modelling such problems is difficult for non-experts and expensive when hiring domain experts to perform the modelling. For many tasks, however, examples of working solutions are readily available. We propose ARNOLD, an approach that partially automates the modelling step by learning an integer program from example solutions. Contrary to existing alternatives, ARNOLD natively handles multi-dimensional quantities and non-linear operations, which are at the core of IP problems, and it only requires examples of feasible solution. The main challenge is to efficiently explore the space of possible programs. Our approach pairs a general-to-specific traversal strategy with a nested lexicographic ordering in order to prune large portions of the space of candidate constraints while avoiding visiting the same candidate multiple times. Our empirical evaluation shows that ARNOLD can acquire models for a number of realistic benchmark problems

Empirical Evaluation of Test Driven Modeling

2013 ◽  
Vol 4 (2) ◽  
pp. 23-43 ◽  
Author(s):  
Stefan Zugal ◽  
Cornelia Haisjackl ◽  
Jakob Pinggera ◽  
Barbara Weber
Keyword(s):  
Process Model ◽  
Empirical Evaluation ◽  
Process Models ◽  
Test Cases ◽  
Domain Experts ◽  
Model Builder ◽  
Study Results ◽  
System Analyst ◽  
High Degree ◽  
The Impact

Declarative approaches to process modeling are regarded well suited for highly volatile environments as they provide a high degree of flexibility. However, problems in understanding and maintaining declarative process models impede their usage. To compensate for these shortcomings, Test Driven Modeling (TDM) has been proposed. This paper reports on an empirical investigation in which TDM is viewed from two different angles. First, the impact of TDM on communication is explored in a case study. Results indicate that domain experts are inclined to use test cases for communicating with the model builder (system analyst) and prefer them over the process model. The second part of the investigation, a controlled experiment, investigates the impact of TDM on process model maintenance. Data gathered in this experiment indicates that the adoption of test cases significantly lowers cognitive load and increases the perceived quality of changes.

scholarly journals Exploring the factors associated with expert systems success

1997 ◽  
Vol 4 (1) ◽  
pp. 8-36 ◽  
Author(s):  
Tor Guimaraes ◽  
Youngohc Yoon ◽  
Quinton O’Neal
Keyword(s):  
Expert Systems ◽  
User Satisfaction ◽  
Systems Approach ◽  
User Involvement ◽  
Project Managers ◽  
User Characteristics ◽  
End User ◽  
Implementation Success ◽  
Domain Experts ◽  
People Skills

As the widespread use and company dependency on Expert Systems increase, so does the need to assess their value and to ensure implementation success. This study identified and empirically tested eight major variables proposed in the literature as determinants of ES success, in this case measured in terms of user satisfaction. IBM's Corporate Manufacturing Expert Systems Project Center collected information from 69 project managers to support the study. The results clearly support the hypothesized relationships and suggest the need for ES project managers to pay special attention to these determinants of ES implementation success. ES success is directly related to the quality of developers and of the ES shells used, end-user characteristics and degree of user involvement in ES development, as each has been defined in this study. For exploratory purposes, the component items for each of these major variables were correlated with the components of user satisfaction. Based on the results, several recommendations are proposed for ES project managers to enhance the likelihood of project success, including: adding problem difficulty as a criterion for ES application selection; increasing ES developer training to improve their people skills, ability to model and to use a systems approach in solving business problems; shaping end-user attitudes and expectations regarding ES; improving the selection of domain experts; more thoroughly understanding the ES impact on end-user jobs; restricting the acquisition of ES shells based on a proposed set of criteria; and ensuring a proper match of ES development techniques and tools to the business problem at hand.

scholarly journals Conventional displays of structures in data compared with interactive projection-based clustering (IPBC)

2021 ◽  
Author(s):  
Michael C. Thrun ◽  
Felix Pape ◽  
Alfred Ultsch
Keyword(s):  
Visual Analytics ◽  
Clustering Algorithms ◽  
Empirical Evaluation ◽  
Similar Data ◽  
Qualitative Comparison ◽  
Domain Experts ◽  
Human In The Loop ◽  
3D Displays ◽  
Benchmark Datasets ◽  
Data Points

AbstractClustering is an important task in knowledge discovery with the goal to identify structures of similar data points in a dataset. Here, the focus lies on methods that use a human-in-the-loop, i.e., incorporate user decisions into the clustering process through 2D and 3D displays of the structures in the data. Some of these interactive approaches fall into the category of visual analytics and emphasize the power of such displays to identify the structures interactively in various types of datasets or to verify the results of clustering algorithms. This work presents a new method called interactive projection-based clustering (IPBC). IPBC is an open-source and parameter-free method using a human-in-the-loop for an interactive 2.5D display and identification of structures in data based on the user’s choice of a dimensionality reduction method. The IPBC approach is systematically compared with accessible visual analytics methods for the display and identification of cluster structures using twelve clustering benchmark datasets and one additional natural dataset. Qualitative comparison of 2D, 2.5D and 3D displays of structures and empirical evaluation of the identified cluster structures show that IPBC outperforms comparable methods. Additionally, IPBC assists in identifying structures previously unknown to domain experts in an application.

Requirements for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

2021 ◽  
Author(s):  
Gary Klein ◽  
Robert Hoffman ◽  
Shane T. Mueller ◽  
William Clancey
Keyword(s):  
Design Research ◽  
Empirical Evaluation ◽  
Laboratory Research ◽  
Investment Performance ◽  
Work Systems ◽  
Measurement Science ◽  
Series Of Experiments ◽  
Online Experimentation ◽  
Bench Testing ◽  
Work Method

The development of AI systems represents a significant investment. But to realize the promise of that investment, performance assessment is necessary. Empirical evaluation of Human-AI work systems must adduce convincing empirical evidence that the work method and its AI technology are learnable, usable, and useful. The theme to this Report is the notion that AI assessment must be effective but must also be efficient. Bench testing of a prototype of an AI system cannot require extensive series of experiments with complex designs. Thus, the empirical requirements that are presented in this Report involve escaping some of the constraints that are imposed in traditional laboratory research. Also, there is a recognition of new constraints that are unique to AI evaluation contexts. Empirical requirements are presented covering study design, research methods, statistical analyses, and online experimentation. The 15 requirements presented in this Report should be applicable to all research intended to evaluate the effectivity of AI systems.

Sign in / Sign up

Export Citation Format

Share Document