Evolving Multimodal Robot Behavior via Many Stepping Stones with the Combinatorial Multi-Objective Evolutionary Algorithm

An important challenge in reinforcement learning is to solve multimodal problems, where agents have to act in qualitatively different ways depending on the circumstances. Because multimodal problems are often too difficult to solve directly, it is often helpful to define a curriculum, which is an ordered set of sub-tasks that can serve as the stepping stones for solving the overall problem. Unfortunately, choosing an effective ordering for these subtasks is difficult, and a poor ordering can reduce the performance of the learning process. Here, we provide a thorough introduction and investigation of the Combinatorial Multi-Objective Evolutionary Algorithm (CMOEA), which allows all combinations of subtasks to be explored simultaneously. We compare CMOEA against three algorithms that can similarly optimize on multiple subtasks simultaneously: NSGA-II, NSGA-III and ϵ-Lexicase Selection. The algorithms are tested on a function-optimization problem with two subtasks, a simulated multimodal robot locomotion problem with six subtasks and a simulated robot maze navigation problem where a hundred random mazes are treated as subtasks. On these problems, CMOEA either outperforms or is competitive with the controls. As a separate contribution, we show that adding a linear combination over all objectives can improve the ability of the control algorithms to solve these multimodal problems. Lastly, we show that CMOEA can leverage auxiliary objectives more effectively than the controls on the multimodal locomotion task. In general, our experiments suggest that CMOEA is a promising algorithm for solving multimodal problems.

Robustly Learning Composable Options in Deep Reinforcement Learning

Hierarchical reinforcement learning (HRL) is only effective for long-horizon problems when high-level skills can be reliably sequentially executed. Unfortunately, learning reliably composable skills is difficult, because all the components of every skill are constantly changing during learning. We propose three methods for improving the composability of learned skills: representing skill initiation regions using a combination of pessimistic and optimistic classifiers; learning re-targetable policies that are robust to non-stationary subgoal regions; and learning robust option policies using model-based RL. We test these improvements on four sparse-reward maze navigation tasks involving a simulated quadrupedal robot. Each method successively improves the robustness of a baseline skill discovery method, substantially outperforming state-of-the-art flat and hierarchical methods.

Implementation of NAO Robot Maze Navigation Based on Computer Vision and Collaborative Learning

Maze navigation using one or more robots has become a recurring challenge in scientific literature and real life practice, with fleets having to find faster and better ways to navigate environments such as a travel hub (e.g. airports) or to evacuate a disaster zone. Many methods have been used to solve this issue, including the implementation of a variety of sensors and other signal receiving systems. Most interestingly, camera-based techniques have increasingly become more popular in this kind of applications, given their robustness and scalability. In this paper, we have implemented an end-to-end strategy to address this scenario, allowing a robot to solve a maze in an autonomous way, by using computer vision and path planning. In addition, this robot shares the generated knowledge to another by means of communication protocols, having to adapt its mechanical characteristics to be able to solve the same challenge. The paper presents experimental validation of the four components of this solution, namely camera calibration, maze mapping, path planning and robot communication. Finally, we present the integration and functionality of these methods applied in a pair of NAO robots.

Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps

Cognitive maps are mental representations of spatial and conceptual relationships in an environment, and are critical for flexible behavior. To form these abstract maps, the hippocampus has to learn to separate or merge aliased observations appropriately in different contexts in a manner that enables generalization and efficient planning. Here we propose a specific higher-order graph structure, clone-structured cognitive graph (CSCG), which forms clones of an observation for different contexts as a representation that addresses these problems. CSCGs can be learned efficiently using a probabilistic sequence model that is inherently robust to uncertainty. We show that CSCGs can explain a variety of cognitive map phenomena such as discovering spatial relations from aliased sensations, transitive inference between disjoint episodes, and formation of transferable schemas. Learning different clones for different contexts explains the emergence of splitter cells observed in maze navigation and event-specific responses in lap-running experiments. Moreover, learning and inference dynamics of CSCGs offer a coherent explanation for disparate place cell remapping phenomena. By lifting aliased observations into a hidden space, CSCGs reveal latent modularity useful for hierarchical abstraction and planning. Altogether, CSCG provides a simple unifying framework for understanding hippocampal function, and could be a pathway for forming relational abstractions in artificial intelligence.

Evaluation of Multimodal Tongue Drive System by People with Tetraplegia for Computer Access

The multimodal Tongue Drive System (mTDS) is an assistive technology for people with tetraplegia that provides an alternative method to interact with a computer by combining tongue control, head gesture, and speech. This multimodality is designed to facilitate the completion of complex computer tasks (e.g. drag-and-drop) that cannot be easily performed by existing uni-modal assistive technologies. Previous studies with able-bodied participants showed promising performance of the mTDS on complex tasks when compared to other input methods such as keyboard and mouse. In this three-session pilot study, the primary objective is to show the feasibility of using mTDS to facilitate human-computer interactions by asking fourteen participants with tetraplegia to complete five computer access tasks with increased level of complexity: maze navigation, center-out tapping, playing bubble shooter and peg solitaire, and sending an email. Speed and accuracy are quantified by key metrics that are found to be generally increasing from the first to third session, indicating the potential existence of a learning phase that could result in improved performance over time.

Evaluation of Multimodal Tongue Drive System by People with Tetraplegia for Computer Access

The multimodal Tongue Drive System (mTDS) is an assistive technology for people with tetraplegia that provides an alternative method to interact with a computer by combining tongue control, head gesture, and speech. This multimodality is designed to facilitate the completion of complex computer tasks (e.g. drag-and-drop) that cannot be easily performed by existing uni-modal assistive technologies. Previous studies with able-bodied participants showed promising performance of the mTDS on complex tasks when compared to other input methods such as keyboard and mouse. In this three-session pilot study, the primary objective is to show the feasibility of using mTDS to facilitate human-computer interactions by asking fourteen participants with tetraplegia to complete five computer access tasks with increased level of complexity: maze navigation, center-out tapping, playing bubble shooter and peg solitaire, and sending an email. Speed and accuracy are quantified by key metrics that are found to be generally increasing from the first to third session, indicating the potential existence of a learning phase that could result in improved performance over time.

Investigating the Effect of Sensory Concurrency on Learning Haptic Spatiotemporal Signals

A new generation of multimodal interfaces and interactions is emerging. Drawing on the principles of Sensory Substitution and Augmentation Devices (SSADs), these new interfaces offer the potential for rich, immersive human-computer interactions, but are difficult to design well, and take time to master, creating significant barriers towards wider adoption. Following a review of the literature surrounding existing SSADs, their metrics for success and their growing influence on interface design in Human Computer Interaction, we present a medium term (4-day) study comparing the effectiveness of various combinations of visual and haptic feedback (sensory concurrencies) in preparing users to perform a virtual maze navigation task using haptic feedback alone. Participants navigated 12 mazes in each of 3 separate sessions under a specific combination of visual and haptic feedback, before performing the same task using the haptic feedback alone. Visual sensory deprivation was shown to be inferior to visual & haptic concurrency in enabling haptic signal comprehension, while a new hybridized condition combining reduced visual feedback with the haptic signal was shown to be superior. Potential explanations for the effectiveness of the hybrid mechanism are explored, and the scope and implications of its generalization to new sensory interfaces is presented.