Controlling Agents by Constrained Policy Updates

Learning the optimal behavior is the ultimate goal in reinforcement learning. This can be achieved by many different approaches, the most successful of them are policy gradient methods. However, they can suffer from undesirably large updates of policies, leading to poor performance. In recent years there has been a clear trend toward designing more reliable algorithms. This paper addresses to examine different restriction strategies applied to the widely used Proximal Policy Optimization (PPO-Clip) technique. We also question whether the analyzed methods are able to adapt not only to low-dimensional tasks but also to complex, high-dimensional problems in control and robotic domains. The analysis of the learned behavior shows that these methods can lead to better performance compared to the original PPO-Clip algorithm, moreover, they are also able to achieve complex behavior and policies in high-dimensional environments.

Dopamine, behavior, and addiction

Addictive drugs are habit-forming. Addiction is a learned behavior; repeated exposure to addictive drugs can stamp in learning. Dopamine-depleted or dopamine-deleted animals have only unlearned reflexes; they lack learned seeking and learned avoidance. Burst-firing of dopamine neurons enables learning—long-term potentiation (LTP)—of search and avoidance responses. It sets the stage for learning that occurs between glutamatergic sensory inputs and GABAergic motor-related outputs of the striatum; this learning establishes the ability to search and avoid. Independent of burst-firing, the rate of single-spiking—or “pacemaker firing”—of dopaminergic neurons mediates motivational arousal. Motivational arousal increases during need states and its level determines the responsiveness of the animal to established predictive stimuli. Addictive drugs, while usually not serving as an external stimulus, have varying abilities to activate the dopamine system; the comparative abilities of different addictive drugs to facilitate LTP is something that might be studied in the future.

Learning and planning with logical automata

We introduce a method to learn policies from expert demonstrations that are interpretable and manipulable. We achieve interpretability by modeling the interactions between high-level actions as an automaton with connections to formal logic. We achieve manipulability by integrating this automaton into planning via Logical Value Iteration, so that changes to the automaton have predictable effects on the learned behavior. These qualities allow a human user to first understand what the model has learned, and then either correct the learned behavior or zero-shot generalize to new, similar tasks. Our inference method requires only low-level trajectories and a description of the environment in order to learn high-level rules. We achieve this by using a deep Bayesian nonparametric hierarchical model. We test our model on several domains of interest and also show results for a real-world implementation on a mobile robotic arm platform for lunchbox-packing and cabinet-opening tasks.

Decoding Human Behaviour in Relation to Capital: An Analysis of Maugham’s The Ant and The Grasshopper in Light of Huxley’s ‘Selected Snobberies’

This study aims to show the fictional and philosophical engagement of Aldous Huxley and Somerset Maugham in unveiling human behavior in relation to capital. Huxley in his sarcastic essay Selected Snobberies has described the nature, utility, types and sources of snobbish attitude in people. Most often snobbery stems out from an individual’s socio-economic situation and his consumerist nature. In the short story The Ant and the Grasshopper, Somerset Maugham has deconstructed the age old story of Aesop that is universally used worldwide to teach children the basic morality and work ethics. He reveals the peculiar desire of human beings to indulge in consumption in contrast with learned behavior of self-denial. This study focuses on the degenerative tendency that is outgrown in human nature through the analysis of George Ramsay from Maugham’s The Ant and the Grasshopper. In addition, this study analyses the changing nature of the idealistic tenets pertaining to the changing mode of time and situation. The binary existence of ethical tenets and the allurement of the consumerist world leads to question the value of its palpability, its effect on making people happy or snobbish. Now the fundamental question is how far a human being is capable of learning self-denial. Considering the reality of truth as not one and universal but multifaceted as Chakraborty (2020) claims, both Huxley and Maugham in these two literary pieces are interestingly inquisitive of the modernist ethics and redefine the means of success.

Learning in advance? Interspecific recognition ability in male Eurasian blackcaps

Interspecific aggression is common in bird communities. It is thought to be a consequence of learned behavior or misdirected intraspecific aggression. In the former case birds acquire the ability to identify individuals of other species by social learning of species-specific traits during interactions with heterospecifics in a shared habitat. I conducted a two-choice song playback experiment to investigate the role of associating signals from two sensory modalities, auditory and visual, in shaping the agonistic response of male Eurasian Blackcaps (Sylvia atricapilla) to simulated male Garden Warbler (S. borin) intruders. I measured focal males’ response to playbacks to the song of a Blackcap or a Garden Warbler while giving them a choice between stuffed male models of both species presented in their territories. The experiments were carried out in early spring, before the arrival of Garden Warblers from Africa. I found that male Blackcaps were able to associate species-specific songs with species-specific plumage types. The ability to associate signals characterized not only experienced after-second-year old males, but also second-year old males that had not had the opportunity to defend breeding territories against heterospecifics. This suggests that second-year old male Blackcaps acquire the ability to associate Garden Warbler song with plumage before they use this skill when defending territory during their first breeding episode. Male Blackcaps do not mistake male Garden Warblers for males of their species, but rather learn these associations during the first months of life. However, the possibility that these associations are innate cannot be excluded.

Genetic Depletion of BDNF Impairs Extinction Learning of a Spatial Appetitive Task in the Presence or Absence of the Acquisition Context

Brain derived neurotropic factor (BDNF) supports neuronal survival, growth, and differentiation and is involved in forms of hippocampus-dependent and independent learning, as well as hippocampus-dependent learning. Extinction learning comprises active inhibition of no-longer relevant learned information, in conjunction with a decreased response of a previously learned behavior. It is highly dependent on context, and evidence exists that it requires hippocampal activation. The participation of BDNF in memory processing is experience-dependent. For example, BDNF has been associated with synaptic plasticity needed for spatial learning, and it is involved in acquisition and extinction learning of fear conditioning. However, little is known about its role in spatial appetitive extinction learning. In this study, we evaluated to what extent BDNF contributes to spatial appetitive extinction learning in the presence (ABA) or absence (AAA) of exposure to the acquisition context. Daily training, of BDNF+/−-mice or their wildtype (WT) littermates, to reach acquisition criterion in a T-maze, resulted in a similar performance outcome. However, extinction learning was delayed in the AAA, and impaired in the ABA-paradigm compared to performance in WT littermates. Trial-by-trial learning analysis indicated differences in the integration of the context into extinction learning by BDNF+/−-mice compared to WT littermates. Taken together, these results support an important role for BDNF in processes that relate to information updating and retrieval that in turn are crucial for effective extinction learning.