Teaming With a Synthetic Teammate: Insights into Human-Autonomy Teaming

2017 ◽  
Vol 60 (2) ◽  
pp. 262-273 ◽  
Author(s):  
Nathan J. McNeese ◽  
Mustafa Demir ◽  
Nancy J. Cooke ◽  
Christopher Myers
Keyword(s):  
Team Performance ◽  
Situation Awareness ◽  
Autonomous Agents ◽  
Unmanned Aerial System ◽  
Extensive Literature ◽  
Processing Efficiency ◽  
Verbal Behaviors ◽  
Human Autonomy ◽  
Team Player ◽  
Team Situation Awareness

Objective Three different team configurations are compared with the goal of better understanding human-autonomy teaming (HAT). Background Although an extensive literature on human-automation interaction exists, much less is known about HAT in which humans and autonomous agents interact as coordinated units. Further research must be conducted to better understand how all-human teams compare to HAT. Methods In an unmanned aerial system (UAS) context, a comparison was made among three types of three-member teams: (1) synthetic teams in which the pilot role is assigned to a synthetic teammate, (2) control teams in which the pilot was an inexperienced human, and (3) experimenter teams in which an experimenter served as an experienced pilot. Ten of each type of team participated. Measures of team performance, target processing efficiency, team situation awareness, and team verbal behaviors were analyzed. Results Synthetic teams performed as well at the mission level as control (all human) teams but processed targets less efficiently. Experimenter teams performed better across all other measures compared to control and synthetic teams. Conclusion Though there is potential for a synthetic agent to function as a full-fledged teammate, further advances in autonomy are needed to improve team-level dynamics in HAT teams. Application This research contributes to our understanding of how to make autonomy a good team player.

Team Situation Awareness in Human-Autonomy Teaming: A Systems Level Approach

2018 ◽  
Vol 62 (1) ◽  
pp. 149-149 ◽  
Author(s):  
David A. Grimm ◽  
Mustafa Demir ◽  
Jamie C. Gorman ◽  
Nancy J. Cooke
Keyword(s):  
Team Performance ◽  
Situation Awareness ◽  
Information Entropy ◽  
Timely Manner ◽  
Team Members ◽  
Dynamic Task ◽  
Remotely Piloted Aircraft ◽  
Human Autonomy ◽  
Team Situation Awareness ◽  
Communication Control

Project overview. The current study focuses on analyzing team flexibility by measuring entropy (where higher values correspond to system reorganization and lower values correspond to more stable system organization) across all-human teams and Human-Autonomy Teams (HAT). We analyzed teams in the context of a fully-fledged synthetic agent that acts as a pilot for a three-agent Remotely Piloted Aircraft System (RPAS) ground crew. The synthetic agent must be able to communicate and coordinate with human teammates in a constructive and timely manner to be effective. This study involved three heterogeneous team members who had to take photographs of target waypoints and communicate via a text-based communication system. The three team members’ roles were: 1) navigator provides information about flight plan with speed and altitude restrictions at each waypoint; 2) pilot adjusts altitude and airspeed to control the Remotely Piloted Aircraft (RPA), and negotiates with the photographer about the current altitude and airspeed to take good photos for the targets; and 3) photographer screens camera settings, and sends feedback to other team members regarding the target photograph status. The three conditions differed based on the manipulation of the pilot role: 1) Synthetic – the pilot was the synthetic agent, 2) Control – the pilot was a randomly assigned participant, and 3) Experimenter – the pilot was a well-trained experimenter who focused on sending and receiving information in a timely manner. The goal of this study is to examine how overall RPAS flexibility across HATs and all-human teams are associated with Team Situation Awareness (TSA). Method. There were 30 teams (10-teams per condition): control teams consisted of three participants randomly assigned to each role; synthetic and experimenter teams included two participants randomly assigned to the navigator and photographer roles. The experiment took place over five 40-minute missions, and the goal was to take as many “good” photos of ground targets as possible while avoiding alarms and rule violations. We obtained several measures, including mission and target level team performance scores, team process measures (situation awareness, process ratings, communication and coordination), and other measures (teamwork knowledge, workload, and demographics). We first estimated amount of system reorganization of the RPAS via an information entropy measure, i.e., the number of arrangements the system occupied over a given period of time (Shannon & Weaver, 1975). Based on information entropy, we defined four layers to represent the RPAS (Gorman, Demir, Cooke, & Grimm, In Review): 1) communications - the chat-based communication among team members; 2) vehicle - the RPA itself, e.g., speed, altitude; 3) control - interface between the RPA and the user; and system - the overall activity of the sub-layers. Then, we looked at the relationship between layers and TSA, which was based on successfully overcoming and completing ad hoc embedded target waypoints. Results and conclusion. Overall, the experimenter teams adapted to more roadblocks than the synthetic teams, who were equivalent to control teams (Demir, McNeese, & Cooke, 2016). The findings indicate that: 1) synthetic teams demonstrated rigid systems level activity, which consisted of less reorganization of communication, control and vehicle layers as conditions changed, which also resulted in less adaptation to roadblocks; 2) control teams demonstrated less communication reorganization, but more control and vehicle reorganization, which also resulted in less adaptation to roadblocks; and 3) experimenter teams demonstrated more reorganization across communication, control and vehicle layers, which resulted in better adaptation to roadblocks. Thus, the ability of a system to reorganize across human and technical layers as situations change is needed to adapt to novel conditions of team performance in a dynamic task

scholarly journals Enhancing Transparency in Human-autonomy Teaming via the Option-centric Rationale Display

2019 ◽  
Vol 63 (1) ◽  
pp. 166-167
Author(s):  
Ruikun Luo ◽  
Na Du ◽  
Kevin Y. Huang ◽  
X. Jessie Yang
Keyword(s):  
Decision Making ◽  
Human Factors ◽  
Team Performance ◽  
Autonomous Agents ◽  
Design Science ◽  
Autonomous Systems ◽  
Design Rationale ◽  
Automated Vehicles ◽  
Human Autonomy ◽  
Intelligent Assistant

Human-autonomy teaming is a major emphasis in the ongoing transformation of future work space wherein human agents and autonomous agents are expected to work as a team. While the increasing complexity in algorithms empowers autonomous systems, one major concern arises from the human factors perspective: Human agents have difficulty deciphering autonomy-generated solutions and increasingly perceive autonomy as a mysterious black box. The lack of transparency could lead to the lack of trust in autonomy and sub-optimal team performance (Chen and Barnes, 2014; Endsley, 2017; Lyons and Havig, 2014; de Visser et al., 2018; Yang et al., 2017). In response to this concern, researchers have investigated ways to enhance autonomy transparency. Existing human factors research on autonomy transparency has largely concentrated on conveying automation reliability or likelihood/(un)certainty information (Beller et al., 2013; McGuirl and Sarter, 2006; Wang et al., 2009; Neyedli et al., 2011). Providing explanations of automation’s behaviors is another way to increase transparency, which leads to higher performance and trust (Dzindolet et al., 2003; Mercado et al., 2016). Specifically, in the context of automated vehicles, studies have showed that informing the drivers of the reasons for the action of automated vehicles decreased drivers’ anxiety, increased their sense of control, preference and acceptance (Koo et al., 2014, 2016; Forster et al., 2017). However, the studies mentioned above largely focused on conveying simple likelihood information or used hand-drafted explanations, with only few exceptions (e.g.(Mercado et al., 2016)). Further research is needed to examine potential design structures of transparency autonomy. In the present study, we wish to propose an option-centric explanation approach, inspired by the research on design rationale. Design rationale is an area of design science focusing on the “representation for explicitly documenting the reasoning and argumentation that make sense of a specific artifact (MacLean et al., 1991)”. The theoretical underpinning for design rationale is that for designers what is important is not just the specific artifact itself but its other possibilities – why an artifact is designed in a particular way compared to how it might otherwise be. We aim to evaluate the effectiveness of the option-centric explanation approach on trust, dependence and team performance. We conducted a human-in-the-loop experiment with 34 participants (Age: Mean = 23.7 years, SD = 2.88 years). We developed a simulated game Treasure Hunter, where participants and an intelligent assistant worked together to uncover a map for treasures. The intelligent assistant’s ability, intent and decision-making rationale was conveyed in the option-centric rationale display. The experiment used a between-subject design with an independent variable – whether the option-centric rationale explanation was provided. The participants were randomly assigned to either of the two explanation conditions. Participants’ trust to the intelligent assistant, confidence of accomplishing the experiment without the intelligent assistant, and workload for the whole session were collected, as well as their scores for each map. The results showed that by conveying the intelligent assistant’s ability, intent and decision-making rationale in the option-centric rationale display, participants had higher task performance. With the display of all the options, participants had a better understanding and overview of the system. Therefore, they could utilize the intelligent assistant more appropriately and earned a higher score. It is notable that every participant only played 10 maps during the whole session. The advantages of option-centric rationale display might be more apparent if more rounds are played in the experiment session. Although not significant at the .05 level, there seems to be a trend suggesting lower levels of workload when the rationale explanation displayed. Our study contributes to the study of human-autonomy teaming by considering the important role of explanation display. It can help human operators build appropriate trust and improve the human-autonomy team performance.

scholarly journals Team Coordination of Team Situation Awareness in Human-Autonomy Teaming

2019 ◽  
Vol 63 (1) ◽  
pp. 146-147 ◽  
Author(s):  
Mustafa Demir ◽  
Nathan J. McNeese ◽  
Manrong She ◽  
Nancy J. Cooke
Keyword(s):  
Situation Awareness ◽  
Task Environment ◽  
Team Cognition ◽  
Team Members ◽  
Team Interaction ◽  
Remotely Piloted Aircraft ◽  
Human Autonomy ◽  
Team Situation Awareness ◽  
The Right ◽  
Pushing And Pulling

Project Overview Team Situation Awareness (TSA), which is a part of team cognition, is a critical factor that influences team effectiveness. It can be defined as getting the right information from the right person within the right amount of time, in order to overcome an unexpected event (Gorman, Cooke, Pederson, Connor, & DeJoode, 2005). TSA is developed and maintained through team interactions, allowing for the measurement of TSA based on team interaction (Cooke & Gorman, 2009). In the current study, a specific measure, Coordinated Awareness of Situation by Teams (CAST) is used (Cooke & Gorman, 2009). CAST evaluates the effectiveness and efficiency of team interaction under “roadblock” scenarios (Gorman, Cooke, & Winner, 2006). These roadblocks represent novel situations in the task and require effective team communication and coordination. Team members must assess the situation according to their own specialized role and/or resources and coordinate with other team members to overcome each separate roadblock. In this task, effective communication refers to team anticipation. That is, each team member needs to anticipate each other’s needs by pushing information rather than pulling information during the task (Demir, McNeese, & Cooke, 2017). In this study, we examined how pushing and pulling information, and CAST were associated with Team Situation Awareness (TSA) in both Human-Autonomy (HAT) and all-human teams in simulated Remotely Piloted Aircraft System (RPAS) task environment. In this research, we integrated the synthetic agent to the Cognitive Engineering Research on Team Tasks Remotely Piloted Aircraft Systems - Synthetic Task Environment (CERTT-RPAS-STE) which was designed to be both a flexible research platform and a realistic task environment with a view to researching team performance and interaction-based measures of team cognition. In the simulated CERTT testbed, there are three heterogeneous teammates who need to take good photos of each target waypoint by communicating via text-chat: (1) the navigator who creates a dynamic flight plan and provides information about the waypoints, the RPA’s airspeed, and altitude restrictions to the pilot; (2) the pilot, who controls the RPA’s heading, altitude, and airspeed, and negotiates with the photographer in order to take a good photo; and (3) the photographer, who monitors sensor equipment in order to take photographs of target waypoints and sends feedback to the other team members about the quality of the photo. This project aimed to understand how team behaviors and team performance differed between HATs and all-human teams in RPAS operations: (1) the synthetic condition—the pilot role was given to the synthetic teammate, which was an ACT-R based cognitive model (which had a limited interaction ability, see Ball et al., 2010; Demir et al., 2015); (2) the control condition—the pilot was a randomly selected human participant, just like the other two participants; and (3) the experimenter condition—one of the experimenters served as an expert pilot. Experimenter condition utilized a Wizard of Oz paradigm in which a trained experimenter (located in a separate room) used a script to imitate a synthetic teammate and communicated with participants in limited communication behaviors but pushing and pulling information in a timely manner (robust coordination). Method There were 30 teams (10 for each condition): control teams consisted of three participants randomly assigned to each role; synthetic and experimenter teams included two participants randomly assigned to the navigator and photographer roles. The experiment took place over five 40-minute missions, and the goal was to take as many “good” photos of ground targets as possible while avoiding alarms and rule violations. During each mission, teams were presented with “roadblocks” by the introduction of a new, ad hoc target waypoint. We collected several measures, but we focused on: the proportion of roadblocks overcome per mission as an outcome measure of TSA; the CAST which is a coordination sequence of team interaction across the team members (i.e. which team members share with team members their experience during the roadblock); and verbal behaviors such as pushing and pulling information. Results and discussion In this team task, effective teamwork involves anticipating the needs of teammates, which in turn means pushing information before it is requested. However, in addition to anticipation, effective coordination is also needed during roadblocks. HATs demonstrated significantly lower levels of CAST than all-human teams. These results indicate that HATs’ lack of anticipation and coordination resulted in poorer TSA performance. These findings help HATs to grow its coordination and communication methodologies. Finally, future studies might examine the relationships highlighted in this study via nonlinear measures in terms of team stability and flexibility based on their communication and coordination patterns during the novel events. HAT is here to stay but improvements to human-machine interactions must continue if we are to improve team effectiveness.

Team situation awareness within the context of human-autonomy teaming

2017 ◽  
Vol 46 ◽  
pp. 3-12 ◽  
Author(s):  
Mustafa Demir ◽  
Nathan J. McNeese ◽  
Nancy J. Cooke
Keyword(s):  
Situation Awareness ◽  
Human Autonomy ◽  
Team Situation Awareness

The Synthetic Teammate as a Team Player in Command-and-Control Teams

2016 ◽  
Vol 60 (1) ◽  
pp. 116-116 ◽  
Author(s):  
Mustafa Demir ◽  
Nathan J. McNeese ◽  
Nancy J. Cooke ◽  
Christopher Myers
Keyword(s):  
Team Performance ◽  
Cognitive Model ◽  
Unmanned Aircraft ◽  
Recent Effort ◽  
Interaction Patterns ◽  
Team Process ◽  
Processing Efficiency ◽  
Specific Sequence ◽  
Team Members ◽  
Team Player

Project overview. The current project is part of a larger effort that focuses on Human-Automation Teaming (HAT) interaction in the context of the development, integration, and validation of a computational cognitive model that acts as a full-fledged synthetic teammate for a three-agent Unmanned Aircraft System (UAS) ground control crew. Our most recent effort looked at team process and team performance within the HAT. In order to be considered a team player, the synthetic teammate must be able to communicate and coordinate with its human teammates and do so in a subtle manner (Demir et al., 2016). In this task, there were three different and interdependent team members: 1) Air Vehicle Operator (AVO) – controls the UAS’s heading, altitude, and airspeed; 2) Data Exploitation, Mission Planning, and Communications (DEMPC) – provides a dynamic flight plan as well as speed and altitude restrictions; and 3) Payload Operator (PLO) – monitors sensor equipment, negotiates with the AVO, and takes photographs of target waypoints. The communication within a three-agent UAS team occurred over a text-based communications system. In this research, there were three conditions which are differentiated by the AVO role: 1) the Synthetic - the synthetic teammate was assigned the AVO role; 2) the Control - the AVO was an inexperienced human participant; 3) the Experimenter - the AVO was one of the experimenters who was experienced with the task. The experimenter AVO asked questions of other team members to ensure timely and adaptive passing of information at target waypoints. In this current study, the coordination among the team members occurs at each target waypoint and requires a specific sequence of information passing for an optimum team performance (Cooke, Gorman, Duran, & Taylor, 2007): the information is provided by the DEMPC about the upcoming target waypoint to the AVO. After that, the PLO and the AVO negotiate regarding an appropriate altitude and airspeed for the target waypoints about required camera settings. Finally, the PLO sends feedback to other team members about the status of the target photo. Method. Activities during this period included conducting an experiment to: 1) evaluate the synthetic teammate’s performance, and the HAT team performance in comparison to all human teams, 2) understand how team process differs between all human and human-synthetic teams and how this impacts performance, and 3) compare the human-synthetic teams and all human control teams to a team with a pilot that is experienced in pushing and pulling information across the team. For this experiment, participants were randomly assigned for the duration of the experiment. Within each of the five missions, teams were told to obtain as many “good” photos as possible while avoiding alarms and rule violations in less than 40 minutes. The overall focus of this paper is: team process that is comprised of eight verbal behaviors associated with team effectiveness; team performance that is a combination of mission variables, including the rate of successful target photographs, time spent in alarm and warning states (for each individual), and the critical waypoint acquisition rate; and target processing efficiency took into account the time spent inside a target waypoint to get a good photo. Results and discussion. In general, findings indicate that synthetic AVOs perform more poorly than control AVOs in terms of team performance. Synthetic teams perform as well at the mission level as control teams. However, in terms of target processing efficiency, synthetic teams perform poorer than control teams. In terms of team process, synthetic teams demonstrate interaction patterns corresponding to more pulling of information than pushing with little change over time. In summary, these results indicate that there is a strong potential for using synthetic team member as a teammate in real world tasks and for training.

Situation Awareness in Team Performance: Implications for Measurement and Training

1995 ◽  
Vol 37 (1) ◽  
pp. 123-136 ◽  
Author(s):  
Eduardo Salas ◽  
Carolyn Prince ◽  
David P. Baker ◽  
Lisa Shrestha
Keyword(s):  
Team Performance ◽  
Situation Awareness ◽  
Important Variable ◽  
Team Situation Awareness ◽  
Research Questions ◽  
Performance Research ◽  
And Training

Situation awareness has long been recognized as an important variable in aviation performance. Research to date has focused on identifying characteristics of situation awareness for individuals, not on the behaviors and processes associated with team situation awareness. The purpose of this review is to delineate and identify characteristics of team situation awareness. In addition, implications are discussed and research questions are outlined that target the measurement and training of situation awareness in teams.

Exploring Measures of Team Knowledge

1998 ◽  
Vol 42 (3) ◽  
pp. 215-219 ◽  
Author(s):  
Nancy J. Cooke ◽  
Rene'e Stout ◽  
Krisela Rivera ◽  
Eduardo Salas
Keyword(s):  
Team Performance ◽  
Situation Awareness ◽  
Relevant Information ◽  
Team Cognition ◽  
Unique Role ◽  
Team Members ◽  
Team Situation Awareness ◽  
Pathfinder Network ◽  
Knowledge Similarity ◽  
Share Information

Team cognition is more than the aggregate cognition of team members. It is an emerging feature, worthy of study in its own right. In this paper we investigate potential metrics of team knowledge in the context of a broader exploratory study on measures of team knowledge, performance, and situation awareness. Team members assumed different roles in a three-person synthetic task in which they were presented with unique role-relevant information. Successful accomplishment of team objectives required team members to share information. The focus of this paper is on one of several measures collected which required judgments of pairwise relatedness ratings for mission-relevant terms. These data were submitted to Pathfinder network scaling and used to derive three metrics of team knowledge: knowledge accuracy, interpositional knowledge, and knowledge similarity. The metrics revealed different perspectives on team knowledge and were generally predictive of team performance and team situation awareness.

Improving Teams' Interpositional Knowledge Through Cross Training

2000 ◽  
Vol 44 (11) ◽  
pp. 390-393 ◽  
Author(s):  
Nancy J. Cooke ◽  
Janis A. Cannon-Bowers ◽  
Preston A. Kiekel ◽  
Krisela Rivera ◽  
Rene'e J. Stout ◽  
...  
Keyword(s):  
Team Performance ◽  
Situation Awareness ◽  
Training Condition ◽  
Team Process ◽  
Team Members ◽  
Weak Support ◽  
Cross Training ◽  
Team Situation Awareness ◽  
Training Team ◽  
Traditional Manner

Recent investigations of team training have demonstrated advantages of cross training team members in the positions of other team members. Such benefits have been attributed to increases in interpositional knowledge. In an attempt to reduce the time demands of cross training, a conceptual cross-training condition that targeted teamwork knowledge was compared to traditional full cross-training and two control conditions. Three-person teams were assigned to a training condition and participated in two synthetic helicopter missions. Outcomes, team process behaviors, team situation awareness, taskwork knowledge, and teamwork knowledge were measured. Results indicated weak support for the benefits of full cross-training on team performance, yet minimal support for conceptual cross-training. Further, teams cross-trained in the traditional manner acquired more teamwork and taskwork interpositional knowledge than teams in any other condition. Both types of interpositional knowledge were correlated with team performance.

Assessing Team Situation Awareness in the Field: A Comparison of Measurement Approaches

2002 ◽  
Vol 46 (24) ◽  
pp. 1959-1963
Author(s):  
Gwendolyn E. Campbell ◽  
James A. Pharmer
Keyword(s):  
Field Study ◽  
Team Performance ◽  
Situation Awareness ◽  
The Other ◽  
Other Hand ◽  
Team Situation Awareness ◽  
Strong Performance ◽  
The Impact ◽  
Different Levels ◽  
Performance Pattern

In this paper we describe a field study conducted with trained Navy teams that was designed to assess the impact of an advanced watchstation on team performance. One of our primary hypotheses was that the advanced display technologies incorporated in this watchstation would support superior situation awareness (SA) within the Navy teams, when compared to the SA supported by their current watchstations. We attempted to use traditional probing techniques to assess team SA, but encountered several difficulties. On the other hand, by assessing the frequency of different classes of behaviors across objects with different levels of tactical significance, we were able to find a strong performance pattern that clearly supported our hypothesis. After explaining our approach and results, we briefly discuss implications for other efforts attempting to assess SA in field settings.

Sign in / Sign up

Export Citation Format

Share Document