Configurable Environments in Reinforcement Learning: An Overview

Reinforcement Learning (RL) has emerged as an effective approach to address a variety of complex control tasks. In a typical RL problem, an agent interacts with the environment by perceiving observations and performing actions, with the ultimate goal of maximizing the cumulative reward. In the traditional formulation, the environment is assumed to be a fixed entity that cannot be externally controlled. However, there exist several real-world scenarios in which the environment offers the opportunity to configure some of its parameters, with diverse effects on the agent’s learning process. In this contribution, we provide an overview of the main aspects of environment configurability. We start by introducing the formalism of the Configurable Markov Decision Processes (Conf-MDPs) and we illustrate the solutions concepts. Then, we revise the algorithms for solving the learning problem in Conf-MDPs. Finally, we present two applications of Conf-MDPs: policy space identification and control frequency adaptation.

Scheduling PID Attitude and Position Control Frequencies for Time-Optimal Quadrotor Waypoint Tracking under Unknown External Disturbances

Recently, the use of quadrotors has increased in numerous applications, such as agriculture, rescue, transportation, inspection, and localization. Time-optimal quadrotor waypoint tracking is defined as controlling quadrotors to follow the given waypoints as quickly as possible. Although PID control is widely used for quadrotor control, it is not adaptable to environmental changes, such as various trajectories and dynamic external disturbances. In this work, we discover that adjusting PID control frequencies is necessary for adapting to environmental changes by showing that the optimal control frequencies can be different for different environments. Therefore, we suggest a method to schedule the PID position and attitude control frequencies for time-optimal quadrotor waypoint tracking. The method includes (1) a Control Frequency Agent (CFA) that finds the best control frequencies in various environments, (2) a Quadrotor Future Predictor (QFP) that predicts the next state of a quadrotor, and (3) combining the CFA and QFP for time-optimal quadrotor waypoint tracking under unknown external disturbances. The experimental results prove the effectiveness of the proposed method by showing that it reduces the travel time of a quadrotor for waypoint tracking.

LLC DC-DC Converter Performances Improvement for Bidirectional Electric Vehicle Charger Application

Electric Vehicle (EV) bidirectional charger technology is growing in importance. It defines the fact of returning the electricity stored in the batteries of EV to Grid (V2G), to Home (V2H), to Load (V2L), or in one word V2X mode. The EV onboard charger is divided into two parts: AC-DC and DC-DC converters. The isolated bidirectional DC-DC LLC resonant converter is used to improve the charger efficiency within both battery power and voltage ranges. It is controlled by varying the switching frequency based on a small signal modeling approach using the gain transfer function inversion method. The dimensions of the DC-DC LLC converter directly affect the charger cost. Moreover, they cause an important control frequency saturation zone, especially in V2X mode, where the switching frequency is out of the feasibility zone. The new challenge in this paper is to design an optimization strategy to minimize the LLC converter cost and improve the control frequency feasibility zone, for a wide variation of battery voltage and converter power, in the charging (G2V) and discharging (V2X) modes simultaneously. For our best knowledge, this optimization problem, in the case of a bidirectional (G2V and V2X) charger, is not yet considered in the literature. An optimal design that considers the control stability equations in the optimization algorithm is elaborated. The obtained results show a significant converter cost decrease and important expansion of control frequency feasibility zones. A comparative study between initial and optimized values, in G2V and V2X modes, is generated according to the converter efficiency.

Comparison of vacuum assisted closure therapy with conventional dressing in the management of necrotizing fasciitis wound

Background: The purpose of our study was to compare the effect of vacuum assisted closure (VAC) therapy and conventional dressings in patients with open wounds due to necrotizing fasciitis (NF) on the basis of healing rate, infection control, frequency of dressing and pain score.Methods: The study evaluated 50 patients admitted with NF requiring surgery over a period of 18 months. The patients were randomized to two groups. In group A patients, the wounds were managed with conventional dressings and in group B patients, negative pressure wound therapy (NPWT) was applied. Serial assessment of both groups was done for four weeks. The parameters including size of wound, wound bed, granulation tissue formation, color, amount and odor of exudate, edema, frequency of dressing, re-debridement and pain were monitored and analysed.Results: In our study, patients with NF wounds who underwent VAC therapy had earlier granulation tissue formation, resolution of infection and readiness for skin grafting. The frequency of dressing, requirement of re-debridement, resolution of edema, odor, skin maceration, inflammation around wound and pain significantly reduced in group B (VAC) when compared to conventional dressing group.Conclusions: When compared to the conventional dressing on NF wound, application of VAC helped in early appearance of granulation tissue, significant reduction of inflammation, wound odor, exudate, need for re-debridement, frequency of dressing and pain. Thus, VAC dressing can be considered as a better option in the management of NF wounds.

10-Hz tACS Over The Prefrontal Cortex Improves Phonemic Fluency

Verbal fluency is an important indicator of human verbal ability. Methods to improve fluency is an interesting issue necessitating investigation. To do this, the current study required participants to randomly receive transcranial alternating current stimulation (tACS) at 10 Hz, 40 Hz (control frequency), and sham stimulation over the prefrontal cortex before a phonemic fluency task. It was found that 10-Hz tACS significantly improved phonemic fluency relative to sham stimulation. This result demonstrates the modulatory effect of 10-Hz tACS on language ability.

Quasi-Boundary Method for Design Consideration of Resonant DC-DC Converters

The paper presents a new unified method for designing the two basic types of serial resonant DC-DC converters (series and with a parallel-loaded capacitor). The method is based on the use of a simplified generalized approach for the analysis of this type of power electronic devices. It is characterized by the fact that regardless of the specific operating mode, with a control frequency below or above the resonant of the series resonant circuit to determine the relationship between input and output voltage or current, it is assumed that the converter operates in resonance. In this sense, it is called a quasi-boundary method for analysis and design. The use of the quasi-boundary method significantly accelerates and facilitates the design of the considered power circuits, which is very useful for the purposes of engineering practice, as well as in the training of power electronics. The applicability of this approach is proven by the methods of mathematical modeling and computer simulations.

European and Indian Grid Codes for Utility Scale Hybrid Power Plants

The aim of this paper is to review and compare present European and Indian grid code requirements imposed to hybrid power plants (HPPs) combining wind, solar and storage technologies. Since there are no grid codes specifically for HPPs, the paper will review grid codes for the power plant based on individual renewable technology in the HPP. European grid codes specifies ranges for parameters inside which each national transmission system operators (TSO) has to specify the set of national parameters (Danish specifications in this paper). The comparisons are performed with respect to fault-ride-through capability, frequency and voltage operation ranges, active power control/frequency support as well as reactive power control/voltage support.

Mapping the vocal circuitry of Alston's singing mouse with pseudorabies virus

Vocalizations, like many social displays, are often elaborate, rhythmically structured behaviors that are modulated by a complex combination of cues. Vocal motor patterns require close coordination of neural circuits governing the muscles of the larynx, jaw, and respiratory system. In the elaborate vocalization of Alstons singing mouse (Scotinomys teguina), for example, each note of its rapid, frequency-modulated trill is accompanied by equally rapid modulation of breath and gape. To elucidate the neural circuitry underlying this behavior, we introduced the polysynaptic retrograde neuronal tracer pseudorabies virus (PRV) into the cricothyroid and digastricus muscles, which control frequency modulation and jaw opening respectively. Each virus singly labels ipsilateral motoneurons (nucleus ambiguous for cricothyroid, and motor trigeminal nucleus for digastricus). We find that the two isogenic viruses heavily and bilaterally co-label neurons in the gigantocellular reticular formation, a putative central pattern generator. The viruses also show strong co-labeling in compartments of the midbrain including the ventrolateral periaqueductal grey and the parabrachial nucleus, two structures strongly implicated in vocalizations. In the forebrain, regions important to social cognition and energy balance both exhibit extensive co-labeling. This includes the paraventricular and arcuate nuclei of the hypothalamus, the lateral hypothalamus, preoptic area, extended amygdala, central amygdala, and the bed nucleus of the stria terminalis. Finally, we find doubly labeled neurons in M1 motor cortex previously described as laryngeal, as well as in the prelimbic cortex, which indicate these cortical regions play a role in vocal production. Although we observe some novel patterns of double-labelling, the progress of both viruses is broadly consistent with vertebrate-general patterns of vocal circuitry, as well as with circuit models derived from primate literature.

Hybrid Parallel Compliance Allows Robots to Operate With Sensorimotor Delays and Low Control Frequencies

Animals locomote robustly and agile, albeit significant sensorimotor delays of their nervous system and the harsh loading conditions resulting from repeated, high-frequent impacts. The engineered sensorimotor control in legged robots is implemented with high control frequencies, often in the kilohertz range. Consequently, robot sensors and actuators can be polled within a few milliseconds. However, especially at harsh impacts with unknown touch-down timing, controllers of legged robots can become unstable, while animals are seemingly not affected. We examine this discrepancy and suggest and implement a hybrid system consisting of a parallel compliant leg joint with varying amounts of passive stiffness and a virtual leg length controller. We present systematic experiments both in computer simulation and robot hardware. Our system shows previously unseen robustness, in the presence of sensorimotor delays up to 60 ms, or control frequencies as low as 20 Hz, for a drop landing task from 1.3 leg lengths high and with a compliance ratio (fraction of physical stiffness of the sum of virtual and physical stiffness) of 0.7. In computer simulations, we report successful drop-landings from 3.8 leg lengths (1.2 m) for a 2 kg quadruped robot with 100 Hz control frequency and a sensorimotor delay of 35 ms.