Configurable Environments in Reinforcement Learning: An Overview

Reinforcement Learning (RL) has emerged as an effective approach to address a variety of complex control tasks. In a typical RL problem, an agent interacts with the environment by perceiving observations and performing actions, with the ultimate goal of maximizing the cumulative reward. In the traditional formulation, the environment is assumed to be a fixed entity that cannot be externally controlled. However, there exist several real-world scenarios in which the environment offers the opportunity to configure some of its parameters, with diverse effects on the agent’s learning process. In this contribution, we provide an overview of the main aspects of environment configurability. We start by introducing the formalism of the Configurable Markov Decision Processes (Conf-MDPs) and we illustrate the solutions concepts. Then, we revise the algorithms for solving the learning problem in Conf-MDPs. Finally, we present two applications of Conf-MDPs: policy space identification and control frequency adaptation.

Application Specific Integrated Circuits for High Resolution X and Gamma Ray Semiconductor Detectors

The increasing demand for performance improvements in radiation detectors, driven by cutting-edge research in nuclear physics, astrophysics and medical imaging, is causing not only a proliferation in the variety of the radiation sensors, but also a growing necessity of tailored solutions for the front-end readout electronics. Within this work, novel solutions for application specific integrated circuits (ASICs) adopted in high-resolution X and $$\upgamma $$ γ ray spectroscopy applications are studied. In the first part of this work, an ultra-low noise charge sensitive amplifier (CSA) is presented, with specific focus on sub-microsecond filtering, addressing the growing interest in high-luminosity experiments. The CSA demonstrated excellent results with Silicon Drift Detectors (SDDs), and with room temperature Cadmium-Telluride (CdTe) detectors, recording a state-of-the-art noise performance. The integration of the CSA within two full-custom radiation detection instruments realized for the ELETTRA (Trieste, Italy) and SESAME (Allan, Jordan) synchrotrons is also presented. In the second part of this work, an ASIC constellation designed for X-Gamma imaging spectrometer (XGIS) onboard of the THESEUS space mission is described. The presented readout ASIC has a highly customized distributed architecture, and integrates a complete on-chip signal filtering, acquisition and digitization with an ultra-low power consumption.

Deep Learning in Multi-step Forecasting of Chaotic Dynamics

The prediction of chaotic dynamical systems’ future evolution is widely debated and represents a hot topic in the context of nonlinear time series analysis. Recent advances in the field proved that machine learning techniques, and in particular artificial neural networks, are well suited to deal with this problem. The current state-of-the-art primarily focuses on noise-free time series, an ideal situation that never occurs in real-world applications. This chapter provides a comprehensive analysis that aims at bridging the gap between the deterministic dynamics generated by archetypal chaotic systems, and the real-world time series. We also deeply explore the importance of different typologies of noise, namely observation and structural noise. Artificial intelligence techniques turned out to provide robust predictions, and potentially represent an effective and flexible alternative to the traditional physically-based approach for real-world applications. Besides the accuracy of the forecasting, the domain-adaptation analysis attested the high generalization capability of the neural predictors across a relatively heterogeneous spatial domain.

Beyond the Traditional Analyses and Resource Management in Real-Time Systems

The difficulties in estimating the Worst-Case Execution Time (WCET) of applications make the use of modern computing architectures limited in real-time systems. Critical embedded systems require the tasks of hard real-time applications to meet their deadlines, and formal proofs on the validity of this condition are usually required by certification authorities. In the last decade, researchers proposed the use of probabilistic measurement-based methods to estimate the WCET instead of traditional static methods. In this chapter, we summarize recent theoretical and quantitative results on the use of probabilistic approaches to estimate the WCET presented in the PhD thesis of the author, including possible exploitation scenarios, open challenges, and future directions.

Optimal Management and Control of Smart Thermal-Energy Grids

This work deals with the development of novel algorithms and methodologies for the optimal management and control of thermal and electrical energy units operating in a networked configuration. The aim of the work is to foster the creation of a smart thermal-energy grid (smart-TEG), by providing supporting tools for the modeling of subsystems and their optimal control and coordination. A hierarchical scheme is proposed to optimally address the management and control issues of the smart-TEG. Different methods are adopted to deal with the features of the specific generation units involved, e.g., multi-rate MPC approaches, or linear parameter-varying strategies for managing subsystem nonlinearity. An advanced scheme based on ensemble model is also conceived for a network of homogeneous units operating in parallel. Moreover, a distributed optimization algorithm for the high-level unit commitment problem is proposed to provide a robust, flexible and scalable scheme.

Modeling of GIDL–Assisted Erase in 3–D NAND Flash Memory Arrays and Its Employment in NOR Flash–Based Spiking Neural Networks

Since the very first introduction of three-dimensional (3–D) vertical-channel (VC) NAND Flash memory arrays, gate-induced drain leakage (GIDL) current has been suggested as a solution to increase the string channel potential to trigger the erase operation. Thanks to that erase scheme, the memory array can be built directly on the top of a $$n^+$$ n + plate, without requiring any p-doped region to contact the string channel and therefore allowing to simplify the manufacturing process and increase the array integration density. For those reasons, the understanding of the physical phenomena occurring in the string when GIDL is triggered is important for the proper design of the cell structure and of the voltage waveforms adopted during erase. Even though a detailed comprehension of the GIDL phenomenology can be achieved by means of technology computer-aided design (TCAD) simulations, they are usually time and resource consuming, especially when realistic string structures with many word-lines (WLs) are considered. In this chapter, an analysis of the GIDL-assisted erase in 3–D VC nand memory arrays is presented. First, the evolution of the string potential and GIDL current during erase is investigated by means of TCAD simulations; then, a compact model able to reproduce both the string dynamics and the threshold voltage transients with reduced computational effort is presented. The developed compact model is proven to be a valuable tool for the optimization of the array performance during erase assisted by GIDL. Then, the idea of taking advantage of GIDL for the erase operation is exported to the context of spiking neural networks (SNNs) based on NOR Flash memory arrays, which require operational schemes that allow single-cell selectivity during both cell program and cell erase. To overcome the block erase typical of nor Flash memory arrays based on Fowler-Nordheim tunneling, a new erase scheme that triggers GIDL in the NOR Flash cell and exploits hot-hole injection (HHI) at its drain side to accomplish the erase operation is presented. Using that scheme, spike-timing dependent plasticity (STDP) is implemented in a mainstream NOR Flash array and array learning is successfully demonstrated in a prototype SNN. The achieved results represent an important step for the development of large-scale neuromorphic systems based on mature and reliable memory technologies.

Model, Integrate, Search... Repeat: A Sound Approach to Building Integrated Repositories of Genomic Data

A wealth of public data repositories is available to drive genomics and clinical research. However, there is no agreement among the various data formats and models; in the common practice, data sources are accessed one by one, learning their specific descriptions with tedious efforts. In this context, the integration of genomic data and of their describing metadata becomes—at the same time—an important, difficult, and well-recognized challenge. In this chapter, after overviewing the most important human genomic data players, we propose a conceptual model of metadata and an extended architecture for integrating datasets, retrieved from a variety of data sources, based upon a structured transformation process; we then describe a user-friendly search system providing access to the resulting consolidated repository, enriched by a multi-ontology knowledge base. Inspired by our work on genomic data integration, during the COVID-19 pandemic outbreak we successfully re-applied the previously proposed model-build-search paradigm, building on the analogies among the human and viral genomics domains. The availability of conceptual models, related databases, and search systems for both humans and viruses will provide important opportunities for research, especially if virus data will be connected to its host, provider of genomic and phenotype information.

Sensor-Assisted Cooperative Localization and Communication in Multi-agent Networks

This brief highlights research advances on cooperative techniques for localization and communication. These two macro trends are investigated in the general context of mobile multi-agent networks for situational awareness applications, where time-varying agents of unknown locations are asked to fulfill positioning and information sharing tasks. Cooperative localization is conceived for both active and passive agents, i.e., targets to be detected and localized, and it is analyzed in vehicular and maritime environments. Communication is investigated for vehicular scenarios, where vehicles are requested to share massive data in the perspective development of connected and automated mobility systems. Both research areas rely on the integration of heterogeneous sensors and communication. Specifically, it is studied how to improve localization by exploring communication techniques as well as how to enhance communication performances by extracting information from perception sensors. The dynamic environment of multi-agent systems calls for robust, flexible and adaptive techniques, capable of profitably fuse different types of information, and the outcomes of these researches show how a statistical approach based on cooperation guarantees higher resilience, reliability and confidence.

Computational Inference of DNA Folding Principles: From Data Management to Machine Learning

DNA is the molecular basis of life and would total about three meters if linearly untangled. To fit in the cell nucleus at the micrometer scale, DNA has, therefore, to fold itself into several layers of hierarchical structures, which are thought to be associated with functional compartmentalization of genomic features like genes and their regulatory elements. For this reason, understanding the mechanisms of genome folding is a major biological research problem. Studying chromatin conformation requires high computational resources and complex data analyses pipelines. In this chapter, we first present the PyGMQL software for interactive and scalable data exploration for genomic data. PyGMQL allows the user to inspect genomic datasets and design complex analysis pipelines. The software presents itself as a easy-to-use Python library and interacts seamlessly with other data analysis packages. We then use the software for the study of chromatin conformation data. We focus on the epigenetic determinants of Topologically Associating Domains (TADs), which are region of high self chromatin interaction. The results of this study highlight the existence of a “grammar of genome folding” which dictates the formation of TADs and boundaries, which is based on the CTCF insulator protein. Finally we focus on the relationship between chromatin conformation and gene expression, designing a graph representation learning model for the prediction of gene co-expression from gene topological features obtained from chromatin conformation data. We demonstrate a correlation between chromatin topology and co-expression, shedding a new light on this debated topic and providing a novel computational framework for the study of co-expression networks.

Low-Noise Mixed-Signal Electronics for Closed-Loop Control of Complex Photonic Circuits

An increasing research effort is being carried out to profit from the advantages of photonics not only in long-range telecommunications but also at short distances, to implement board-to-board or chip-to-chip interconnections. In this context, Silicon Photonics emerged as a promising technology, allowing to integrate optical devices in a small silicon chip. However, the integration density made possible by Silicon Photonics revealed the difficulty of operating complex optical architectures in an open-loop way, due to their high sensitivity to fabrication parameters and temperature variations. In this chapter, a low-noise mixed-signal electronic platform implementing feedback control of complex optical architectures is presented. The system exploits the ContactLess Integrated Photonic Probe, a non-invasive detector that senses light in silicon waveguides by measuring their electrical conductance. The CLIPP readout resolution has been maximized thanks to the design of a low-noise multichannel ASIC, achieving an accuracy better than −35 dBm in light monitoring. The feedback loop to stabilize the behaviour of photonic circuits is then closed in the digital domain by a custom mixed-signal electronic platform. Experimental demonstrations of optical communications at high data-rate confirm the effectiveness of the proposed approach.