Partially Observable Markov Decision Processes and Robotics

Planning under uncertainty is critical to robotics. The partially observable Markov decision process (POMDP) is a mathematical framework for such planning problems. POMDPs are powerful because of their careful quantification of the nondeterministic effects of actions and the partial observability of the states. But for the same reason, they are notorious for their high computational complexity and have been deemed impractical for robotics. However, over the past two decades, the development of sampling-based approximate solvers has led to tremendous advances in POMDP-solving capabilities. Although these solvers do not generate the optimal solution, they can compute good POMDP solutions that significantly improve the robustness of robotics systems within reasonable computational resources, thereby making POMDPs practical for many realistic robotics problems. This article presents a review of POMDPs, emphasizing computational issues that have hindered their practicality in robotics and ideas in sampling-based solvers that have alleviated such difficulties, together with lessons learned from applying POMDPs to physical robots. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Increasingly Intelligent Micromachines

Intelligent micromachines, with dimensions ranging from a few millimeters down to hundreds of nanometers, are miniature systems capable of performing specific tasks autonomously at small scales. Enhancing the intelligence of micromachines to tackle the uncertainty and variability in complex microenvironments has applications in minimally invasive medicine, bioengineering, water cleaning, analytical chemistry, and more. Over the past decade, significant progress has been made in the construction of intelligent micromachines, evolving from simple micromachines to soft, compound, reconfigurable, encodable, multifunctional, and integrated micromachines, as well as from individual to multiagent, multiscale, hierarchical, self-organizing, and swarm micromachines. The field leverages two important trends in robotics research—the miniaturization and intelligentization of machines—but a compelling combination of these two features has yet to be realized. The core technologies required to make such tiny machines intelligent include information media, transduction, processing, exchange, and energy supply, but embedding all of these functions into a system at the micro- or nanoscale is challenging. This article offers a comprehensive introduction to the state-of-the-art technologies used to create intelligence for micromachines and provides insight into the construction of next-generation intelligent micromachines that can adapt to diverse scenarios for use in emerging fields. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Behavior Trees in Robot Control Systems

In this article, we provide a control-theoretic perspective on the research area of behavior trees in robotics. The key idea underlying behavior trees is to make use of modularity, hierarchies, and feedback in order to handle the complexity of a versatile robot control system. Modularity is a well-known tool to handle software complexity by enabling the development, debugging, and extension of separate modules without detailed knowledge of the entire system. A hierarchy of such modules is natural, since robot tasks can often be decomposed into a hierarchy of subtasks. Finally, feedback control is a fundamental tool for handling uncertainties and disturbances in any low-level control system, but in order to enable feedback control on the higher level, where one module decides what submodule to execute, information regarding the progress and applicability of each submodule needs to be shared in the module interfaces. We describe how these three concepts can be used in theoretical analysis, practical design, and extensions and combinations with other ideas from control theory and robotics. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Observer Design for Nonlinear Systems with Equivariance

Equivariance is a common and natural property of many nonlinear control systems, especially those associated with models of mechatronic and navigation systems. Such systems admit a symmetry, associated with the equivariance, that provides structure enabling the design of robust and high-performance observers. A key insight is to pose the observer state to lie in the symmetry group rather than on the system state space. This allows one to define a global intrinsic equivariant error but poses a challenge in defining internal dynamics for the observer. By choosing an equivariant lift of the system dynamics for the observer internal model, we show that the error dynamics have a particularly nice form. Applying the methodology of extended Kalman filtering to the equivariant error state yields a filter we term the equivariant filter. The geometry of the state-space manifold appears naturally as a curvature modification to the classical Riccati equation for extended Kalman filtering. The equivariant filter exploits the symmetry and respects the geometry of an equivariant system model, and thus yields high-performance, robust filters for a wide range of mechatronic and navigation systems. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Magnetic Micro- and Nanoagents for Monitoring Enzymatic Activity In Vivo

Enzymes are appealing diagnostic targets because of their centrality in human health and disease. Continuous efforts spanning several decades have yielded methods for magnetically detecting the interactions of enzymes with exogenous molecular substrates. Nevertheless, measuring enzymatic activity in vivo remains challenging due to background noise, insufficient selectivity, and overlapping enzymatic functions. Magnetic micro- and nanoagents are poised to help overcome these issues by offering possible advantages such as site-selective sampling, modular architectures, new forms of magnetic detection, and favorable biocompatibility. Here, we review relevant control and detection strategies and consider examples of magnetic enzyme detection demonstrated with micro- or nanorobotic systems. Most cases have focused on proteolytic enzymes, leaving ample opportunity to expand to other classes of enzymes. Enzyme-responsive magnetic micro- and nanoagents hold promise for lowering barriers of translation and enabling preemptive, point-of-care medical applications. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Turbulence and Control of Wind Farms

The dynamics of the turbulent atmospheric boundary layer play a fundamental role in wind farm energy production, governing the velocity field that enters the farm as well as the turbulent mixing that regenerates energy for extraction at downstream rows. Understanding the dynamic interactions among turbines, wind farms, and the atmospheric boundary layer can therefore be beneficial in improving the efficiency of wind farm control approaches. Anticipated increases in the sizes of new wind farms to meet renewable energy targets will increase the importance of exploiting this understanding to advance wind farm control capabilities. This review discusses approaches for modeling and estimation of the wind farm flow field that have exploited such knowledge in closed-loop control, to varying degrees. We focus on power tracking as an example application that will be of critical importance as wind farms transition into their anticipated role as major suppliers of electricity. The discussion highlights the benefits of including the dynamics of the flow field in control and points to critical shortcomings of the current approaches. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Probabilistic Model Checking and Autonomy

The design and control of autonomous systems that operate in uncertain or adversarial environments can be facilitated by formal modeling and analysis. Probabilistic model checking is a technique to automatically verify, for a given temporal logic specification, that a system model satisfies the specification, as well as to synthesize an optimal strategy for its control. This method has recently been extended to multiagent systems that exhibit competitive or cooperative behavior modeled via stochastic games and synthesis of equilibria strategies. In this article, we provide an overview of probabilistic model checking, focusing on models supported by the PRISM and PRISM-games model checkers. This overview includes fully observable and partially observable Markov decision processes, as well as turn-based and concurrent stochastic games, together with associated probabilistic temporal logics. We demonstrate the applicability of the framework through illustrative examples from autonomous systems. Finally, we highlight research challenges and suggest directions for future work in this area. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Analysis and Control of Autonomous Mobility-on-Demand Systems

Challenged by urbanization and increasing travel needs, existing transportation systems need new mobility paradigms. In this article, we present the emerging concept of autonomous mobility-on-demand, whereby centrally orchestrated fleets of autonomous vehicles provide mobility service to customers. We provide a comprehensive review of methods and tools to model and solve problems related to autonomous mobility-on-demand systems. Specifically, we first identify problem settings for their analysis and control, from both operational and planning perspectives. We then review modeling aspects, including transportation networks, transportation demand, congestion, operational constraints, and interactions with existing infrastructure. Thereafter, we provide a systematic analysis of existing solution methods and performance metrics, highlighting trends and trade-offs. Finally, we present various directions for further research. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Energy-Aware Controllability of Complex Networks

Understanding the fundamental principles and limitations of controlling complex networks is of paramount importance across natural, social, and engineering sciences. The classic notion of controllability does not capture the effort needed to control dynamical networks, and quantitative measures of controllability have been proposed to remedy this problem. This article presents an introductory overview of the practical (i.e., energy-related) aspects of controlling networks governed by linear dynamics. First, we introduce a class of energy-aware controllability metrics and discuss their properties. Then, we establish bounds on these metrics, which allow us to understand how the structure of the network impacts the control energy. Finally, we examine the problem of optimally selecting a set of control nodes so as to minimize the control effort, and compare the performance of some simple strategies to approximately solve this problem. Throughout the article, we include examples of structured and random networks to illustrate our results. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Stimuli-Responsive Polymers for Soft Robotics

This article reviews recent progress in the use of stimuli-responsive polymers for soft robotics. First, we introduce different types of representative stimuli-responsive polymers, which include liquid crystal polymers and elastomers, hydrogels, shape memory polymers, magnetic elastomers, electroactive polymers, and thermal expansion actuators. We focus on the mechanisms of actuation and the evaluation of performance and discuss strategies for improvements. We then present examples of soft robotic applications based on stimuli-responsive polymers for bending, grasping, walking, swimming, flying, and sensing control. Finally, we discuss current opportunities and challenges of stimuli-responsive soft robots for future study. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 5 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.