Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

<div>Recently, barrier function-based safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a model-based safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier-based control policy structure that can guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic learning algorithm is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify the offline deployment performance and the online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.</div>

Constraint-Based Sound-Motion Objects in Music Performance

The aim of this paper is to present principles of constraint-based sound-motion objects in music performance. Sound-motion objects are multimodal fragments of combined sound and sound-producing body motion, usually in the duration range of just a few seconds, and conceived, produced, and perceived as intrinsically coherent units. Sound-motion objects have a privileged role as building blocks in music because of their duration, coherence, and salient features and emerge from combined instrumental, biomechanical, and motor control constraints at work in performance. Exploring these constraints and the crucial role of the sound-motion objects can enhance our understanding of generative processes in music and have practical applications in performance, improvisation, and composition.

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

<div>Recently, barrier function-based safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a model-based safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier-based control policy structure that can guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic learning algorithm is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify the offline deployment performance and the online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.</div>

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

<div>Recently, barrier function-based safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a model-based safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier-based control policy structure that can guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic learning algorithm is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify the offline deployment performance and the online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.</div>

Safe Learning Reference Governor: Theory and Application to Fuel Truck Rollover Avoidance

This paper proposes a learning reference governor (LRG) approach to enforce state and control constraints in systems for which an accurate model is unavailable; and this approach enables the reference governor to gradually improve command tracking performance through learning while enforcing the constraints during learning and after learning is completed. The learning can be performed either on a black-box type model of the system or directly on the hardware. After introducing the LRG algorithm and outlining its theoretical properties, this paper investigates LRG application to fuel truck (tank truck) rollover avoidance. Through simulations based on a fuel truck model that accounts for liquid fuel sloshing effects, we show that the proposed LRG can effectively protect fuel trucks from rollover accidents under various operating conditions.

Protective and Exacerbating Cognition and Attribution Factors From the Cognitive Discrepancy Theory of Loneliness

According to the cognitive discrepancy theory, although the discrepancy between actual and desired social resources may result in loneliness, Perlman and Peplau (1998) suggested that cognitive processing and attributional style also impact the interpretation of social information. Previous empirical research investigating predictors of loneliness have not assessed a wide range of cognition and attribution factors, so this study filled this gap by examining how protective (optimism, sense of mastery, and purpose in life) and exacerbating (depression, control constraints, negative self-perceptions of aging (SPA), and experiences of age-based discrimination) factors influence and moderate the experience of loneliness cross-sectionally and longitudinally using a sample of 3,345 Americans aged 50 years and older from the 2008 and 2012 waves of the Health and Retirement Study. Optimism (βs = -.15, -.13), mastery (βs = -.08, -.07), purpose in life (βs = -.19, -.18), depression (βs = .22,.14), control constraints (βs = .18, .17), negative SPA (βs = .13, .14), and experiences of ageism (βs = .07, .06) were significantly related to loneliness cross-sectionally and longitudinally, respectively. Optimism buffered the negative impact of poor functional social resources (e.g., low social support) on loneliness cross-sectionally while control constraints, negative SPA, and experiencing ageism exacerbated the relationship between low functional social resources and loneliness cross-sectionally. None of the protective or exacerbating factors modulated the relationship between functional social resources and loneliness longitudinally. These findings have important implications for the development of interventions that target loneliness. Targeting maladaptive cognitions may be particularly effective in reducing loneliness.

Algorithm for Finding Feedback in a Problem with Constraints for One Class of Nonlinear Control Systems

For a continuous nonlinear control system on a finite time interval with control constraints, where the right-hand side of the dynamics equations is linear in control and linearizable in the vicinity of the zero equilibrium position, we consider the construction of a feedback according to the Kalman algorithm. For this, the solution of an auxiliary optimal control problem with a quadratic functional is used by analogy with the SDRE approach.Since this approach is used in the literature to find suboptimal synthesis in optimal control problems with a quadratic functional with formally linear systems, where all coefficient matrices in differential equations and criteria can contain state variables, then on a finite time interval it becomes necessary to solve a complicated matrix differential Riccati equations, with state-dependent coefficient matrices. This circumstance, due to the nonlinearity of the system, in comparison with the Kalman algorithm for linear-quadratic problems, significantly increases the number of calculations for obtaining the coefficients of the gain matrix in the feedback and for obtaining synthesis with a given accuracy. The proposed synthesis construction algorithm is constructed using the extension principle proposed by V. F. Krotov and developed by V. I. Gurman and allows not only to expand the scope of the SDRE approach to nonlinear control problems with control constraints in the form of closed inequalities, but also to propose a more efficient computational algorithm for finding the matrix of feedback gains in control problems on a finite interval. The article establishes the correctness of the application of the extension principle by introducing analogs of the Lagrange multipliers, depending on the state and time, and also derives a formula for the suboptimal value of the quality criterion. The presented theoretical results are illustrated by calculating suboptimal feedbacks in the problems of managing three-sector economic systems.