scholarly journals Identification of Animal Behavioral Strategies by Inverse Reinforcement Learning

2017 ◽  
Author(s):  
Shoichiro Yamaguchi ◽  
Honda Naoki ◽  
Muneki Ikeda ◽  
Yuki Tsukada ◽  
Shunji Nakano ◽  
...  
Keyword(s):  
Decision Making ◽  
Time Series ◽  
Reinforcement Learning ◽  
Time Series Data ◽  
Series Data ◽  
Behavioral Strategy ◽  
Behavioral Strategies ◽  
Inverse Reinforcement Learning ◽  
C Elegans ◽  
Temporal Derivative

AbstractAnimals are able to reach a desired state in an environment by controlling various behavioral patterns. Identification of the behavioral strategy used for this control is important for understanding animals’ decision-making and is fundamental to dissect information processing done by the nervous system. However, methods for quantifying such behavioral strategies have not been fully established. In this study, we developed an inverse reinforcement-learning (IRL) framework to identify an animal’s behavioral strategy from behavioral time-series data. As a particular target, we applied this framework to C. elegans thermotactic behavior; after cultivation at a constant temperature with or without food, the fed and starved worms prefer and avoid from the cultivation temperature on a thermal gradient, respectively. Our IRL approach revealed that the fed worms used both absolute and temporal derivative of temperature and that their strategy comprised mixture of two strategies: directed migration (DM) and isothermal migration (IM). The DM is a strategy that the worms efficiently reach to specific temperature, which explained thermotactic behaviors of the fed worms. The IM is a strategy that the worms track along a constant temperature, which reflects isothermal tracking well observed in previous studies. We also showed the neural basis underlying the strategies, by applying our method to thermosensory neuron-deficient worms. In contrast to fed animals, the strategy of starved animals indicated that they escaped the cultivation temperature using only absolute, but not temporal derivative of temperature. Thus, our IRL-based approach is capable of identifying animal strategies from behavioral time-series data and will be applicable to wide range of behavioral studies, including decision-making of other organisms.Author SummaryUnderstanding animal decision-making has been a fundamental problem in neuroscience and behavioral ecology. Many studies analyze actions that represent decision-making in behavioral tasks, in which rewards are artificially designed with specific objectives. However, it is impossible to extend this artificially designed experiment to a natural environment, because in a natural environment, the rewards for freely-behaving animals cannot be clearly defined. To this end, we must reverse the current paradigm so that rewards are identified from behavioral data. Here, we propose a new reverse-engineering approach (inverse reinforcement learning) that can estimate a behavioral strategy from time-series data of freely-behaving animals. By applying this technique with thermotaxis in C. elegans, we successfully identified the reward-based behavioral strategy.

scholarly journals Analyzing Sensor-Based Individual and Population Behavior Patterns via Inverse Reinforcement Learning

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5207
Author(s):  
Beiyu Lin ◽  
Diane J. Cook
Keyword(s):  
Time Series ◽  
Reinforcement Learning ◽  
Smart Home ◽  
Time Series Data ◽  
Population Level ◽  
Sensor Data ◽  
Series Data ◽  
Inverse Reinforcement Learning ◽  
Daily Routines ◽  
Health Diagnosis

Digital markers of behavior can be continuously created, in everyday settings, using time series data collected by ambient sensors. The goal of this work was to perform individual- and population-level behavior analysis from such time series sensor data. In this paper, we introduce a novel algorithm—Resident Relative Entropy-Inverse Reinforcement Learning (RRE-IRL)—to perform an analysis of a single smart home resident or a group of residents, using inverse reinforcement learning. By employing this method, we learnt an individual’s behavioral routine preferences. We then analyzed daily routines for an individual and for eight smart home residents grouped by health diagnoses. We observed that the behavioral routine preferences changed over time. Specifically, the probability that the observed behavior was the same at the beginning of data collection as it was at the end (months later) was lower for residents experiencing cognitive decline than for cognitively healthy residents. When comparing aggregated behavior between groups of residents from the two diagnosis groups, the behavioral difference was even greater. Furthermore, the behavior preferences were used by a random forest classifier to predict a resident’s cognitive health diagnosis, with an accuracy of 0.84.

scholarly journals Functional connectomics from neural dynamics: probabilistic graphical models for neuronal network of Caenorhabditis elegans

2018 ◽  
Vol 373 (1758) ◽  
pp. 20170377 ◽  
Author(s):  
Hexuan Liu ◽  
Jimin Kim ◽  
Eli Shlizerman
Keyword(s):  
Time Series ◽  
Nervous System ◽  
Caenorhabditis Elegans ◽  
Graphical Models ◽  
Neuronal Network ◽  
Graphical Model ◽  
Time Series Data ◽  
Series Data ◽  
Neuronal Responses ◽  
C Elegans

We propose an approach to represent neuronal network dynamics as a probabilistic graphical model (PGM). To construct the PGM, we collect time series of neuronal responses produced by the neuronal network and use singular value decomposition to obtain a low-dimensional projection of the time-series data. We then extract dominant patterns from the projections to get pairwise dependency information and create a graphical model for the full network. The outcome model is a functional connectome that captures how stimuli propagate through the network and thus represents causal dependencies between neurons and stimuli. We apply our methodology to a model of the Caenorhabditis elegans somatic nervous system to validate and show an example of our approach. The structure and dynamics of the C. elegans nervous system are well studied and a model that generates neuronal responses is available. The resulting PGM enables us to obtain and verify underlying neuronal pathways for known behavioural scenarios and detect possible pathways for novel scenarios. This article is part of a discussion meeting issue ‘Connectome to behaviour: modelling C. elegans at cellular resolution’.

scholarly journals Ship-Collision Avoidance Decision-Making Learning of Unmanned Surface Vehicles with Automatic Identification System Data Based on Encoder—Decoder Automatic-Response Neural Networks

2020 ◽  
Vol 8 (10) ◽  
pp. 754
Author(s):  
Miao Gao ◽  
Guo-You Shi
Keyword(s):  
Neural Networks ◽  
Decision Making ◽  
Time Series ◽  
Collision Avoidance ◽  
Time Series Data ◽  
Series Data ◽  
Automatic Identification ◽  
Identification System ◽  
Automatic Response ◽  
Ship Collision

Intelligent unmanned surface vehicle (USV) collision avoidance is a complex inference problem based on current navigation status. This requires simultaneous processing of the input sequences and generation of the response sequences. The automatic identification system (AIS) encounter data mainly include the time-series data of two AIS sets, which exhibit a one-to-one mapping relation. Herein, an encoder–decoder automatic-response neural network is designed and implemented based on the sequence-to-sequence (Seq2Seq) structure to simultaneously process the two AIS encounter trajectory sequences. Furthermore, this model is combined with the bidirectional long short-term memory recurrent neural networks (Bi-LSTM RNN) to obtain a network framework for processing the time-series data to obtain ship-collision avoidance decisions based on big data. The encoder–decoder neural networks were trained based on the AIS data obtained in 2018 from Zhoushan Port to achieve ship collision avoidance decision-making learning. The results indicated that the encoder–decoder neural networks can be used to effectively formulate the sequence of the collision avoidance decision of the USV. Thus, this study significantly contributes to the increased efficiency and safety of maritime transportation. The proposed method can potentially be applied to the USV technology and intelligent collision-avoidance systems.

scholarly journals Analysis of a long-term Forest Accountancy Network to Support Management and Controlling

2018 ◽  
Vol 3 (1) ◽  
pp. 155-162 ◽  
Author(s):  
Markus Dög ◽  
Johannes Wildberg ◽  
Bernhard Möhring
Keyword(s):  
Decision Making ◽  
Time Series ◽  
Time Series Data ◽  
Management Strategies ◽  
Series Data ◽  
Forest Economics ◽  
Private Forest ◽  
Forest Landowners ◽  
Forest Enterprises

Abstract Multifunctional forestry in Germany is characterized by long production periods and complex biological-technical processes. Private forest enterprises are complex systems which are closely interwoven with the economic environment. To ensure their economic success, forest landowners need to take the economic development into consideration and adapt their management strategies. Management accounting is an important source for information needed to fulfil main tasks of accounting that help to manage forest enterprises: ‘description’, ‘explanation’ and ‘decision making’. To get general data, long time series data, taken from Forest Accountancy Networks (FAN), can be analysed. For more than 45 years, data from the FAN Westfalen-Lippe in Germany has been collected and analysed by the department of Forest Economics and Forest Management at the University of Göttingen. The long-term development and adaptation strategies of defined groups of private forest enterprises can be illustrated using this data. These valuable time series can support decision-making processes for private forest landowners and provide tools for forest policy. The data shows that private forest enterprises, with spruce as the dominating tree species, have performed above average in terms of operating revenues and profit margins, but are also more susceptible to calamities resulting in higher involuntary timber harvests.

scholarly journals Prediction of Interbank Offered Rate Based on Time Series Model

2021 ◽  
Vol 251 ◽  
pp. 01014
Author(s):  
Ding Huang ◽  
Ming Zhong ◽  
Xupeng Shi
Keyword(s):  
Decision Making ◽  
Time Series ◽  
Time Series Data ◽  
Time Series Model ◽  
Grey Model ◽  
Actual Data ◽  
Series Data ◽  
Bank Management ◽  
Working Day ◽  
The Impact

This paper studies the prediction of interbank offered rate changes in each working day. Using the actual data of each working day of China’s interbank offered rate from 2007 to 2019, this paper sets up ARIMA, Prophet, grey model and MTGNN to study and verify the time series data, and make a comparison between these models. The limitation of this paper is that it does not consider the impact of macroeconomic characteristics but only considers the predict changes in time series. The results of this paper are expected to be helpful for bank management and interbank transaction decision making.

scholarly journals A hidden Markov model analysis of subject-specific seizure time-series data as a potential aid to clinical decision making

F1000Research ◽  
2016 ◽  
Vol 5 ◽  
pp. 2592
Author(s):  
Martin D. King ◽  
Suresh Pujar ◽  
Rod C. Scott
Keyword(s):  
Decision Making ◽  
Time Series ◽  
Sojourn Time ◽  
Clinical Decision Making ◽  
Time Series Data ◽  
Seizure Activity ◽  
Hidden Markov ◽  
Clinical Decision ◽  
Series Data ◽  
Subject Specific

Background The seizure-count time series data acquired from three children with refractory epilepsy were used in a statistical modelling analysis designed to provide an explanation for the marked variation in seizure frequency that often occurs over time (over-dispersed Poisson behaviour). This was motivated by an expectation that a better understanding of the spontaneous shifts in seizure-activity that are observed in some cases should reduce the risk of over-treatment caused by inappropriate changes in medication. Methods The analyses were performed using Poisson hidden Markov models (HMMs), both Bayesian and non-Bayesian, implemented using Markov chain Monte Carlo and the expectation-maximisation algorithm, respectively. A defining feature of the models, as applied to epilepsy, is the assumed existence of two or more pathological states, with state-specific Poisson rates, and random transitions between the states. Posterior predictive simulation was used to assess the validity of the Bayesian HMMs. Results The results are presented in the form of state transition probability and Poisson rate estimates (i.e., the primary HMM parameters), together with information derived from these primary parameters. State-specific mean-duration (sojourn time) estimates and sojourn-time complementary cumulative probability distributions are the main focus. HMM analyses are presented for three children that differed markedly in their seizure behaviour. The first is characterised by an extreme seizure count on one occasion; the second underwent a spontaneous decrease in seizure activity during the observation period; the third seizure-count time trajectory is characterised by a gradual change in mean seizure activity. We show that, despite their considerable differences, each of the observed seizure-count trajectories can be treated adequately using an HMM. Conclusions The study demonstrates that clinically relevant information can be obtained using HM modelling in three cases with markedly different seizure behaviour. The resulting subject-specific statistics provide useful clinical insights which should aid those engaged in clinical decision making.

Decision Making Strategy Based on Time Series Data of Voting Behavior

2015 ◽  
pp. 229-241
Author(s):  
Shogo Higuchi ◽  
Ryohei Orihara ◽  
Yuichi Sei ◽  
Yasuyuki Tahara ◽  
Akihiko Ohsuga
Keyword(s):  
Decision Making ◽  
Time Series ◽  
Voting Behavior ◽  
Time Series Data ◽  
Series Data

The Entrepreneurial Decision: Economic Theory and Empirical Evidence

1995 ◽  
Vol 19 (4) ◽  
pp. 67-79 ◽  
Author(s):  
Joseph G. Eisenhauer
Keyword(s):  
Decision Making ◽  
Economic Theory ◽  
Time Series ◽  
Empirical Evidence ◽  
Time Series Data ◽  
Economic Thought ◽  
Series Data ◽  
Entrepreneurial Decision ◽  
The U.S

This paper Integrates three major traditions of economic thought into a model of entrepreneurial decision making. Several testable hypotheses are formulated, and the model Is estimated using a 33-year sample of aggregate time-series data for the U.S.

Sign in / Sign up

Export Citation Format

Share Document