Requirements for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

2021 ◽  
Author(s):  
Gary Klein ◽  
Robert Hoffman ◽  
Shane T. Mueller ◽  
William Clancey
Keyword(s):  
Design Research ◽  
Empirical Evaluation ◽  
Laboratory Research ◽  
Investment Performance ◽  
Work Systems ◽  
Measurement Science ◽  
Series Of Experiments ◽  
Online Experimentation ◽  
Bench Testing ◽  
Work Method

The development of AI systems represents a significant investment. But to realize the promise of that investment, performance assessment is necessary. Empirical evaluation of Human-AI work systems must adduce convincing empirical evidence that the work method and its AI technology are learnable, usable, and useful. The theme to this Report is the notion that AI assessment must be effective but must also be efficient. Bench testing of a prototype of an AI system cannot require extensive series of experiments with complex designs. Thus, the empirical requirements that are presented in this Report involve escaping some of the constraints that are imposed in traditional laboratory research. Also, there is a recognition of new constraints that are unique to AI evaluation contexts. Empirical requirements are presented covering study design, research methods, statistical analyses, and online experimentation. The 15 requirements presented in this Report should be applicable to all research intended to evaluate the effectivity of AI systems.

scholarly journals Recommendations for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

2021 ◽  
Author(s):  
Robert Hoffman ◽  
Gary Klein ◽  
William Clancey ◽  
Shane T. Mueller
Keyword(s):  
Research Methods ◽  
Study Design ◽  
Design Research ◽  
Statistical Analyses ◽  
Work Systems ◽  
Empirical Assessment ◽  
Measurement Science ◽  
Technical Details ◽  
Online Experimentation

This Report is a companion to the Report titled "Requirements for the Evaluation of Human-AI Work Systems." Whereas that Report focused on the minimum necessary empirical requirements for the assessment of AI systems, this Report provides additional recommendations and technical details to assist the developers of AI systems. Recommendations are presented covering study design, research methods, measurement, statistical analyses, and online experimentation. This guidance should be applicable to all research intended to evaluate the effectivity of AI systems.

Autonomous Charging for an Underwater Robotic Fish by Direct Contact

2016 ◽  
Author(s):  
Paul Phamduy ◽  
Jayhwan Cheong ◽  
Maurizio Porfiri
Keyword(s):  
Autonomous Navigation ◽  
Direct Contact ◽  
Autonomous Underwater Vehicles ◽  
Robotic Fish ◽  
Laboratory Research ◽  
Free Swimming ◽  
Charging Station ◽  
Battery Capacity ◽  
Series Of Experiments ◽  
Field Deployment

The operation of autonomous underwater vehicles is often hindered by their battery capacity, limiting the duration of its use. Here, we propose an integrated solution for autonomous charging of a robotic fish via direct contact through a novel claw mechanism for docking guidance. To assist the robotic fish in the docking process, the system incorporates a charging station designed with form-fit claws. A controller is designed to monitor the battery level of the robotic fish during free swimming and coordinate the docking process with respect to the maneuvers of both the robot and form-fit claws. Upon recognizing a low battery level, the controller commands the robotic fish to begin the docking process, and video feedback from an overhead camera is used to inform the autonomous navigation toward the charging station. After reaching a battery level threshold, the robotic fish is then released back in the water and returns to free swimming until the battery is discharged again. Through a series of experiments, we demonstrate the possibility of prolonged operation, consisting of repeated cycles of autonomous charging. Our proposed charging method enables prolonged autonomous swimming with minimal human supervision, opening the door for new, transformative applications of robotic fish in laboratory research and field deployment.

scholarly journals Identifikasi Siklamat Pada Kuah Dadar Gulung Yang Dijual Di Kawasan Pelabuhan Rambang Kota Palangka Raya

2017 ◽  
Vol 3 (1) ◽  
pp. 37-53
Author(s):  
Nurul Qamariah ◽  
Karmila Karmila
Keyword(s):  
Data Collection ◽  
Experimental Research ◽  
Positive Control ◽  
Laboratory Research ◽  
Artificial Sweetener ◽  
The Third ◽  
Port Area ◽  
Series Of Experiments ◽  
Made In

The purpose of this study was to find out whether the rolled omelet sauce sold in the Rambang Port Area in Palangka Raya City contained artificial sweetener of cyclamate. This research is a type of experimental research by conducting a series of experiments which then obtained data from the results of the laboratory research and analyzed descriptively. The research sample was a rolled omelet sauce sold in the Rambang Port Area in Palangka Raya City. Data collection techniques using saturated sampling is by taking all the population as a sample because the population is relatively small, then the data is made in table form. Sampling was conducted on 19 May 2016 at 14.50 WIB for the first sample, at 15.00 WIB for the second sample and at 15.15 WIB for the third sample. From the identification results of artificial cyclamate sweeteners in roll omelet namely in sample 1, sample 2 and sample 3, after being tested there were no white deposits as happened in the positive control. So it can be concluded that the sample was not detected containing cyclamate.

Effects of Occlusion on Pigeons' Visual Object Recognition

Perception ◽  
10.1068/p3441 ◽  
2002 ◽  
Vol 31 (11) ◽  
pp. 1299-1312 ◽  
Author(s):  
Norma T DiPietro ◽  
Edward A Wasserman ◽  
Michael E Young
Keyword(s):  
Object Recognition ◽  
Laboratory Research ◽  
Visual Object ◽  
Visual Object Recognition ◽  
Occluded Objects ◽  
Novel Objects ◽  
Series Of Experiments ◽  
Different Response ◽  
Partially Occluded ◽  
Do So

Casual observation suggests that pigeons and other animals can recognize occluded objects; yet laboratory research has thus far failed to show that pigeons can do so. In a series of experiments, we investigated pigeons' ability to ‘name’ shaded, textured stimuli by associating each with a different response. After first learning to recognize four unoccluded objects, pigeons had to recognize the objects when they were partially occluded by another surface or when they were placed on top of another surface; in each case, recognition was weak. Following training with the unoccluded stimuli and with the stimuli placed on top of the occluder, pigeons' recognition of occluded objects dramatically improved. Pigeons' improved recognition of occluded objects was not limited to the trained objects but transferred to novel objects as well. Evidently, the recognition of occluded objects requires pigeons to learn to discriminate the object from the occluder; once this discrimination is mastered, occluded objects can be better recognized.

scholarly journals Developing Hot Mathematics Task with Indonesian Heritage as Context to Assess Mathematical Literacy of Students in Primary School

2020 ◽  
Vol 1 (8) ◽  
pp. 69
Author(s):  
Wuli Oktiningrum ◽  
Dyah Ayu Pramoda Wardhani
Keyword(s):  
Primary School ◽  
Formative Evaluation ◽  
Design Research ◽  
Empirical Evaluation ◽  
Potential Effect ◽  
Mathematical Literacy ◽  
School Students ◽  
Field Test Result ◽  
Is Design ◽  
Test Result

The goal of this research produces a set of HOT mathematics tasks with Indonesian heritage as a context that is valid and practical to assess the mathematical literacy of students in primary school. The participants of the study were 120 students in primary school. The data was collected through documentation, questionnaire, test result, and interviews. This is design research using a type of development research with formative evaluation. The mathematics task not only reviewed by 7 experts, who assess the context, content, and language of prototype but also beyond empirical evaluation of validation and reliability testing. The field test result showed that the HOT mathematics task has a potential effect on the mathematical literacy of primary school students.

scholarly journals Laboratory research of vibration impact parameters on the coal seam through host rocks

2019 ◽  
Vol 109 ◽  
pp. 00069
Author(s):  
Oleksandr Petukh ◽  
Kostiantyn Sofiiskyi ◽  
Dmytro Zhytlonok
Keyword(s):  
Rock Mass ◽  
Coal Seam ◽  
Laboratory Research ◽  
Hydraulic Systems ◽  
Working Face ◽  
Fundamental Possibility ◽  
Vibration Impact ◽  
Basic Parameters ◽  
Bench Testing ◽  
Host Rocks

On the basis of mechanized roof supports, the hydraulic systems of vibrators operating as compact vibrators for controlling the condition of the coal and rock mass through host rocks are selected and refined. Bench testing of vibrator systems operating in continuous and pulsed modes with automatic control were held. The basic parameters of the systems of vibrators are established, and the fundamental possibility of their work in the working face of gas-dangerous mines was proved.

scholarly journals A technique for evaluating and improving the semantic transparency of modeling language notations

2021 ◽  
Author(s):  
Dominik Bork ◽  
Ben Roelens
Keyword(s):  
Design Research ◽  
Empirical Evaluation ◽  
Modeling Language ◽  
Modeling Languages ◽  
Semantic Transparency ◽  
Language Engineering ◽  
Domain Specific ◽  
Domain Specific Modeling ◽  
Easy Integration ◽  
Iterative Evaluation

AbstractThe notation of a modeling language is of paramount importance for its efficient use and the correct comprehension of created models. A graphical notation, especially for domain-specific modeling languages, should therefore be aligned to the knowledge, beliefs, and expectations of the targeted model users. One quality attributed to notations is their semantic transparency, indicating the extent to which a notation intuitively suggests its meaning to untrained users. Method engineers should thus aim at semantic transparency for realizing intuitively understandable notations. However, notation design is often treated poorly—if at all—in method engineering methodologies. This paper proposes a technique that, based on iterative evaluation and improvement tasks, steers the notation toward semantic transparency. The approach can be efficiently applied to arbitrary modeling languages and allows easy integration into existing modeling language engineering methodologies. We show the feasibility of the technique by reporting on two cycles of Action Design Research including the evaluation and improvement of the semantic transparency of the Process-Goal Alignment modeling language notation. An empirical evaluation comparing the new notation against the initial one shows the effectiveness of the technique.

Trends in Human Factors Evaluation of Work Support Systems

2000 ◽  
Vol 44 (37) ◽  
pp. 561-564 ◽  
Author(s):  
Jens Rasmussen
Keyword(s):  
Deep Structure ◽  
Empirical Evaluation ◽  
User Involvement ◽  
Internal Constraints ◽  
Work Systems ◽  
Domain Experts ◽  
Work Support ◽  
Intimate Knowledge ◽  
Design Deficiencies ◽  
Interface Matching

Society is becoming increasingly integrated, effects of disturbances and design deficiencies propagate rapidly and widely, and work is becoming increasingly dynamic. Furthermore, organizations have to face an increasingly competitive environment, where success very likely is granted those operating at the border of established practice. This raises new requirements for the design and evaluation of work systems. For discretionary tasks in a dynamic work environment, a map of the deep structure of the work content is more effective than a route instruction. Rather than an interface matching the mental models of the users, the interface should make visible the affordances and the internal constraints of the workspace, including the boundaries of acceptable conditions. That is, the interface should induce a proper mental model. Design and evaluation then cannot be based only on user involvement and empirical evaluation tests. In addition, intimate knowledge of the deep structure of the workspace is necessary and domain experts must be involved in an analytical evaluation, which is outlined.

X-ray microanalysis of thin specimens in the transmission electron microscope at voltages up to 1000 Kv.

1978 ◽  
Vol 36 (1) ◽  
pp. 540-541
Author(s):  
G. Cliff ◽  
M.J. Nasir ◽  
G.W. Lorimer ◽  
N. Ridley
Keyword(s):  
Electron Microscope ◽  
Transmission Electron Microscope ◽  
Weight Fraction ◽  
Present Series ◽  
Constant Independent ◽  
X Ray ◽  
Transmission Electron ◽  
Series Of Experiments ◽  
Image Position ◽  
X Ray Absorption

In a specimen which is transmission thin to 100 kV electrons - a sample in which X-ray absorption is so insignificant that it can be neglected and where fluorescence effects can generally be ignored (1,2) - a ratio of characteristic X-ray intensities, I1/I2 can be converted into a weight fraction ratio, C1/C2, using the equationwhere k12 is, at a given voltage, a constant independent of composition or thickness, k12 values can be determined experimentally from thin standards (3) or calculated (4,6). Both experimental and calculated k12 values have been obtained for K(11<Z>19),kα(Z>19) and some Lα radiation (3,6) at 100 kV. The object of the present series of experiments was to experimentally determine k12 values at voltages between 200 and 1000 kV and to compare these with calculated values.The experiments were carried out on an AEI-EM7 HVEM fitted with an energy dispersive X-ray detector.

Sign in / Sign up

Export Citation Format

Share Document