Actor-Critic Traction Control Based on Reinforcement Learning with Open-Loop Training

The use of actor-critic algorithms can improve the controllers currently implemented in automotive applications. This method combines reinforcement learning (RL) and neural networks to achieve the possibility of controlling nonlinear systems with real-time capabilities. Actor-critic algorithms were already applied with success in different controllers including autonomous driving, antilock braking system (ABS), and electronic stability control (ESC). However, in the current researches, virtual environments are implemented for the training process instead of using real plants to obtain the datasets. This limitation is given by trial and error methods implemented for the training process, which generates considerable risks in case the controller directly acts on the real plant. In this way, the present research proposes and evaluates an open-loop training process, which permits the data acquisition without the control interaction and an open-loop training of the neural networks. The performance of the trained controllers is evaluated by a design of experiments (DOE) to understand how it is affected by the generated dataset. The results present a successful application of open-loop training architecture. The controller can maintain the slip ratio under adequate levels during maneuvers on different floors, including grounds that are not applied during the training process. The actor neural network is also able to identify the different floors and change the acceleration profile according to the characteristics of each ground.

Download Full-text

Diversity oriented Deep Reinforcement Learning for targeted molecule generation

Journal of Cheminformatics ◽

10.1186/s13321-021-00498-z ◽

2021 ◽

Vol 13 (1) ◽

Author(s):

Tiago Pereira ◽

Maryam Abbasi ◽

Bernardete Ribeiro ◽

Joel P. Arrais

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Reinforcement Learning ◽

Deep Neural Networks ◽

Chemical Space ◽

Biological Properties ◽

Training Process ◽

Training Strategy ◽

Inhibitory Power ◽

Exploratory Strategy

AbstractIn this work, we explore the potential of deep learning to streamline the process of identifying new potential drugs through the computational generation of molecules with interesting biological properties. Two deep neural networks compose our targeted generation framework: the Generator, which is trained to learn the building rules of valid molecules employing SMILES strings notation, and the Predictor which evaluates the newly generated compounds by predicting their affinity for the desired target. Then, the Generator is optimized through Reinforcement Learning to produce molecules with bespoken properties. The innovation of this approach is the exploratory strategy applied during the reinforcement training process that seeks to add novelty to the generated compounds. This training strategy employs two Generators interchangeably to sample new SMILES: the initially trained model that will remain fixed and a copy of the previous one that will be updated during the training to uncover the most promising molecules. The evolution of the reward assigned by the Predictor determines how often each one is employed to select the next token of the molecule. This strategy establishes a compromise between the need to acquire more information about the chemical space and the need to sample new molecules, with the experience gained so far. To demonstrate the effectiveness of the method, the Generator is trained to design molecules with an optimized coefficient of partition and also high inhibitory power against the Adenosine $$A_{2A}$$ A 2 A and $$\kappa$$ κ opioid receptors. The results reveal that the model can effectively adjust the newly generated molecules towards the wanted direction. More importantly, it was possible to find promising sets of unique and diverse molecules, which was the main purpose of the newly implemented strategy.

Download Full-text

A Task of Miniature Mobile Robot Learning for Obstacle Avoidance through Neural Networks

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.151.498 ◽

2012 ◽

Vol 151 ◽

pp. 498-502

Author(s):

Jin Xue Zhang ◽

Hai Zhu Pan

Keyword(s):

Neural Networks ◽

Reinforcement Learning ◽

Mobile Robot ◽

Obstacle Avoidance ◽

Robot Learning ◽

Q Learning ◽

Robot Systems ◽

The Neural Networks ◽

Behavior Based

This paper is concerned with Q-learning , a very popular algorithm for reinforcement learning ,for obstacle avoidance through neural networks. The principle tells that the focus always must be on both ecological nice tasks and behaviours when designing on robot. Many robot systems have used behavior-based systems since the 1980’s.In this paper, the Khepera robot is trained through the proposed algorithm of Q-learning using the neural networks for the task of obstacle avoidance. In experiments with real and simulated robots, the neural networks approach can be used to make it possible for Q-learning to handle changes in the environment.

Download Full-text

Neural Network Based Transfer Learning of Manipulator Inverse Displacement Analysis

Volume 10: 44th Mechanisms and Robotics Conference (MR) ◽

10.1115/detc2020-22508 ◽

2020 ◽

Author(s):

Houcheng Tang ◽

Leila Notash

Keyword(s):

Neural Network ◽

Neural Networks ◽

Transfer Learning ◽

Learning Approach ◽

Training Process ◽

Displacement Analysis ◽

Performance Targets ◽

The Neural Networks ◽

Random Initialization ◽

And Performance

Abstract In this paper, a neural network based transfer learning approach of inverse displacement analysis of robot manipulators is studied. Neural networks with different structures are applied utilizing data from different configurations of a manipulator for training purposes. Then the transfer learning was conducted between manipulators with different geometric layouts. The training is performed on both the neural networks with pretrained initial parameters and the neural networks with random initialization. To investigate the rate of convergence of data fitting comprehensively, different values of performance targets are defined. The computing epochs and performance measures are compared. It is presented that, depending on the structure of neural network, the proposed transfer learning can accelerate the training process and achieve higher accuracy. For different datasets, the transfer learning approach improves their performance differently.

Download Full-text

A Study on CNN-Based Detection of Psyllids in Sticky Traps Using Multiple Image Data Sources

AI ◽

10.3390/ai1020013 ◽

2020 ◽

Vol 1 (2) ◽

pp. 198-208

Author(s):

Jayme Garcia Arnal Barbedo ◽

Guilherme Barros Castro

Keyword(s):

Neural Networks ◽

Image Data ◽

Single Image ◽

Training Process ◽

Sticky Traps ◽

Multiple Image ◽

The Neural Networks ◽

Using Data ◽

Diverse Context ◽

Learning Architectures

Deep learning architectures like Convolutional Neural Networks (CNNs) are quickly becoming the standard for detecting and counting objects in digital images. However, most of the experiments found in the literature train and test the neural networks using data from a single image source, making it difficult to infer how the trained models would perform under a more diverse context. The objective of this study was to assess the robustness of models trained using data from a varying number of sources. Nine different devices were used to acquire images of yellow sticky traps containing psyllids and a wide variety of other objects, with each model being trained and tested using different data combinations. The results from the experiments were used to draw several conclusions about how the training process should be conducted and how the robustness of the trained models is influenced by data quantity and variety.

Download Full-text

Unmanned Aerial Vehicle Pitch Control under Delay Using Deep Reinforcement Learning with Continuous Action in Wind Tunnel Test

Aerospace ◽

10.3390/aerospace8090258 ◽

2021 ◽

Vol 8 (9) ◽

pp. 258

Author(s):

Daichi Wada ◽

Sergio A. Araujo-Estrada ◽

Shane Windsor

Keyword(s):

Neural Networks ◽

Reinforcement Learning ◽

Wind Tunnel ◽

Real World ◽

Time Delays ◽

Wind Tunnel Test ◽

The Real ◽

Pitch Control ◽

Controller Performance ◽

The Neural Networks

Nonlinear flight controllers for fixed-wing unmanned aerial vehicles (UAVs) can potentially be developed using deep reinforcement learning. However, there is often a reality gap between the simulation models used to train these controllers and the real world. This study experimentally investigated the application of deep reinforcement learning to the pitch control of a UAV in wind tunnel tests, with a particular focus of investigating the effect of time delays on flight controller performance. Multiple neural networks were trained in simulation with different assumed time delays and then wind tunnel tested. The neural networks trained with shorter delays tended to be susceptible to delay in the real tests and produce fluctuating behaviour. The neural networks trained with longer delays behaved more conservatively and did not produce oscillations but suffered steady state errors under some conditions due to unmodeled frictional effects. These results highlight the importance of performing physical experiments to validate controller performance and how the training approach used with reinforcement learning needs to be robust to reality gaps between simulation and the real world.

Download Full-text

A Collision Avoidance Method Based on Deep Reinforcement Learning

Robotics ◽

10.3390/robotics10020073 ◽

2021 ◽

Vol 10 (2) ◽

pp. 73

Author(s):

Shumin Feng ◽

Bijo Sebastian ◽

Pinhas Ben-Tzvi

Keyword(s):

Reinforcement Learning ◽

Collision Avoidance ◽

Real World ◽

Future Research ◽

Training Process ◽

Compact Spaces ◽

Unknown Environment ◽

The Neural Networks ◽

Collision Avoidance Problems ◽

Window Approach

This paper set out to investigate the usefulness of solving collision avoidance problems with the help of deep reinforcement learning in an unknown environment, especially in compact spaces, such as a narrow corridor. This research aims to determine whether a deep reinforcement learning-based collision avoidance method is superior to the traditional methods, such as potential field-based methods and dynamic window approach. Besides, the proposed obstacle avoidance method was developed as one of the capabilities to enable each robot in a novel robotic system, namely the Self-reconfigurable and Transformable Omni-Directional Robotic Modules (STORM), to navigate intelligently and safely in an unknown environment. A well-conceived hardware and software architecture with features that enable further expansion and parallel development designed for the ongoing STORM projects is also presented in this work. A virtual STORM module with skid-steer kinematics was simulated in Gazebo to reduce the gap between the simulations and the real-world implementations. Moreover, comparisons among multiple training runs of the neural networks with different parameters related to balance the exploitation and exploration during the training process, as well as tests and experiments conducted in both simulation and real-world, are presented in detail. Directions for future research are also provided in the paper.

Download Full-text

DeepWeave: Accelerating Job Completion Time with Deep Reinforcement Learning-based Coflow Scheduling

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/458 ◽

2020 ◽

Author(s):

Penghao Sun ◽

Zehua Guo ◽

Junchao Wang ◽

Junfei Li ◽

Julong Lan ◽

...

Keyword(s):

Neural Networks ◽

Reinforcement Learning ◽

Distributed Computing ◽

Completion Time ◽

Heuristic Algorithms ◽

Expert Knowledge ◽

Real Life ◽

Processing Efficiency ◽

Multi Stage ◽

The Neural Networks

To improve the processing efficiency of jobs in distributed computing, the concept of coflow is proposed. A coflow is a collection of flows that are semantically correlated in a multi-stage computation task. A job consists of multiple coflows and can be usually formulated as a Directed-Acyclic Graph (DAG). A proper scheduling of coflows can significantly reduce the completion time of jobs in distributed computing. However, this scheduling problem is proved to be NP-hard. Different from existing schemes that use hand-crafted heuristic algorithms to solve this problem, in this paper, we propose a Deep Reinforcement Learning (DRL) framework named DeepWeave to generate coflow scheduling policies. To improve the inter-coflow scheduling ability in the job DAG, DeepWeave employs a Graph Neural Network (GNN) to process the DAG information. DeepWeave learns from the history workload trace to train the neural networks of the DRL agent and encodes the scheduling policy in the neural networks, which make coflow scheduling decisions without expert knowledge or a pre-assumed model. The proposed scheme is evaluated with a simulator using real-life traces. Simulation results show that DeepWeave completes jobs at least 1.7X faster than the state-of-the-art solutions.

Download Full-text

The Compact Support Neural Network

Sensors ◽

10.3390/s21248494 ◽

2021 ◽

Vol 21 (24) ◽

pp. 8494

Author(s):

Adrian Barbu ◽

Hongyu Mou

Keyword(s):

Neural Network ◽

Neural Networks ◽

Compact Support ◽

Shape Parameter ◽

Autonomous Driving ◽

Activation Function ◽

Training Data ◽

Benchmark Datasets ◽

The Neural Networks ◽

Experimental Findings

Neural networks are popular and useful in many fields, but they have the problem of giving high confidence responses for examples that are away from the training data. This makes the neural networks very confident in their prediction while making gross mistakes, thus limiting their reliability for safety-critical applications such as autonomous driving and space exploration, etc. This paper introduces a novel neuron generalization that has the standard dot-product-based neuron and the radial basis function (RBF) neuron as two extreme cases of a shape parameter. Using a rectified linear unit (ReLU) as the activation function results in a novel neuron that has compact support, which means its output is zero outside a bounded domain. To address the difficulties in training the proposed neural network, it introduces a novel training method that takes a pretrained standard neural network that is fine-tuned while gradually increasing the shape parameter to the desired value. The theoretical findings of the paper are bound on the gradient of the proposed neuron and proof that a neural network with such neurons has the universal approximation property. This means that the network can approximate any continuous and integrable function with an arbitrary degree of accuracy. The experimental findings on standard benchmark datasets show that the proposed approach has smaller test errors than the state-of-the-art competing methods and outperforms the competing methods in detecting out-of-distribution samples on two out of three datasets.

Download Full-text

A Decision-Making Method for Connected Autonomous Driving Based on Reinforcement Learning

10.4271/2020-01-5154 ◽

2020 ◽

Keyword(s):

Decision Making ◽

Reinforcement Learning ◽

Learning Strategy ◽

Ride Comfort ◽

Autonomous Driving ◽

Training Process ◽

Training Strategy ◽

Reward Function ◽

Decision Making System ◽

Model Training

At present, with the development of Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), the decision-making for automated vehicle based on connected environment conditions has attracted more attentions. Reliability, efficiency and generalization performance are the basic requirements for the vehicle decision-making system. Therefore, this paper proposed a decision-making method for connected autonomous driving based on Wasserstein Generative Adversarial Nets-Deep Deterministic Policy Gradient (WGAIL-DDPG) algorithm. In which, the key components for reinforcement learning (RL) model, reward function, is designed from the aspect of vehicle serviceability, such as safety, ride comfort and handling stability. To reduce the complexity of the proposed model, an imitation learning strategy is introduced to improve the RL training process. Meanwhile, the model training strategy based on cloud computing effectively solves the problem of insufficient computing resources of the vehicle-mounted system. Test results show that the proposed method can improve the efficiency for RL training process with reliable decision making performance and reveals excellent generalization capability.

Download Full-text

Tax Fraud Detection through Neural Networks: An Application Using a Sample of Personal Income Taxpayers

Future Internet ◽

10.3390/fi11040086 ◽

2019 ◽

Vol 11 (4) ◽

pp. 86 ◽

Cited By ~ 2

Author(s):

César Pérez López ◽

María Delgado Rodríguez ◽

Sonia de Lucas Santos

Keyword(s):

Neural Network ◽

Neural Networks ◽

Income Tax ◽

Fraud Detection ◽

Personal Income ◽

Personal Income Tax ◽

Tax Returns ◽

The Neural Networks ◽

Tax Fraud ◽

Efficiency Rate

The goal of the present research is to contribute to the detection of tax fraud concerning personal income tax returns (IRPF, in Spanish) filed in Spain, through the use of Machine Learning advanced predictive tools, by applying Multilayer Perceptron neural network (MLP) models. The possibilities springing from these techniques have been applied to a broad range of personal income return data supplied by the Institute of Fiscal Studies (IEF). The use of the neural networks enabled taxpayer segmentation as well as calculation of the probability concerning an individual taxpayer’s propensity to attempt to evade taxes. The results showed that the selected model has an efficiency rate of 84.3%, implying an improvement in relation to other models utilized in tax fraud detection. The proposal can be generalized to quantify an individual’s propensity to commit fraud with regards to other kinds of taxes. These models will support tax offices to help them arrive at the best decisions regarding action plans to combat tax fraud.

Download Full-text