Analysis of Communication, Team Situational Awareness, and Feedback in a Three-Person Intelligent Team Tutoring System

This research assessed how the performance and team skills of three-person teams working with an Intelligent Team Tutoring System (ITTS) on a virtual military surveillance task were affected by feedback privacy, participant role, task experience, prior team experience, and teammate familiarity. Previous work in Intelligent Tutoring Systems (ITSs) has focused on outcomes for task skill training for individual learners. As research extends into intelligent tutoring for teams, both task skills and team skills are necessary for good team performance. This work includes a brief review of previous research on ITTSs, feedback, teams, and teamwork, including the recounting of two categories of a framework of teamwork performance, Communication and Cognition, which are relevant to the present study. This research examines the effects of an intelligent agent, as well as features of the team, its members, and the task being undertaken, on team communication (measured by relevant key-presses) and team situation awareness (as measured by scores on a quiz). Thirty-seven teams of three participants, each at their own computer running a multiplayer surveillance simulation, were given just-in-time private (individually delivered) or public (team-delivered) performance feedback during four 5-min trials. In the fourth trial, two of the three participants switched roles. Feedback type, teamwork experience, and teammate familiarity had no statistically significant effect on communication or team situation awareness. However, higher levels of role experience and task experience showed significant and medium-sized effects on communication performance. Results, based on performance data and structured interview responses, also revealed areas of improvement in future feedback design and a potential benchmark for feedback frequency in an action-oriented serious game-based ITTS. Among the conclusions are six design objectives for future ITTSs, establishing a foundation for future research on designing effective ITTSs that train interpersonal skills to nascent teams.

The Effect of Feedback Type on Perception of Performance

In an increasingly connected world, it’s essential to have efficient teams. Training is a common method used to maintain or improve team performance. However, implementing high quality team training can be costly and require a significant amount of time and effort. An Intelligent Tutoring System (ITS) designed for a team, or an Intelligent Team Tutoring System (ITTS), can provide a solution that would reduce the cost, time, and effort required to implement high quality training. Few studies have examined the influence that feedback delivered by an ITTS has on an individual’s perception of their performance and their team’s performance. This within-subjects study, in which 117 participants (39 teams) completed a virtual shopping mall task, addresses this gap. Results indicate that user interface (UI) designers should display either Individual feedback or Team feedback, not both, to give users a correct perception of their performance and their team’s performance.

Feedback Design Considerations for Intelligent Team Tutoring Systems

Challenges arise when developing a computer-based Intelligent Team Tutoring System (ITTS) that attempts to deliver feedback to teams as effectively as a human tutor. The purpose of this current work is to outline elements of feedback that should be considered when designing feedback for an ITTS. The authors present the results of a study that consisted of 32 participants grouped into 16 teams of two. Each team conducted a surveillance task where they received individual or team feedback. Feedback content was written using either the bald (direct feedback; no need for interpretation) or off-record (general feedback; interpretation needed) etiquette strategy. The results showed that feedback delivered using the bald etiquette strategy positively correlated with improved performance. The results also showed that team level feedback positively correlated with more accurate self-assessment among participants. This suggests that in an ITTS, direct feedback can lead to better performance, and that feedback provided at the team level can help to align self-interpretation of performance with actual task performance.

The Future of Adaptive Tutoring: Wrangling Complexity across Domains, Applications, and Platforms

The purpose of this panel is to discuss current directions in research and design of adaptive tutoring, and the need for a method to uniformly describe tutors within this growing field. Discussions will focus on the increasing complexity of individual tutors, as well as how tutors could be categorized through identification of relevant, constituent parts. A standardized taxonomy would provide the foundation for establishing a quantifiable metric of complexity, which could then be used to compare vastly distinct tutors to one another. Applications of such a metric also include evaluating tutor effectiveness with respect to learning outcomes, comparing capabilities / usability of different adaptive tutor authoring tools, and providing more accurate estimates of the time required to develop an hour of tutoring. Individual elements of tutoring to be discussed within the context of this framework include team tutoring, psychomotor tutoring, multi-platform architectures, personalized tutoring, and authoring complexity.

I'Ve Got Synthers, Who Could Ask for Anything More?

Synthetic agents can enrich the environment for practice of complex skills such as airborne command and control. Even in a robust simulation, however, synthetic teammates and opponents are not sufficient to train students. We identify several additional requirements for training: a well-focused curriculum, automated performance measures that drive feedback, strategies for managing real-time feedback and off-line instruction, and learning aids that provide instructional scaffolding for novice users. We describe methods of fulfilling these requirements and recommend future paths of research in the new domain of Intelligent Team Tutoring Systems.