How the value of the environment controls persistence in visual search

Classic foraging theory predicts that humans and animals aim to gain maximum reward per unit time. However, in standard instrumental conditioning tasks individuals adopt an apparently suboptimal strategy: they respond slowly when the expected value is low. This reward-related bias is often explained as reduced motivation in response to low rewards. Here we present evidence this behavior is associated with a complementary increased motivation to search the environment for alternatives. We trained monkeys to search for reward-related visual targets in environments with different values. We found that the reward-related bias scaled with environment value, was consistent with persistent searching after the target was already found, and was associated with increased exploratory gaze to objects in the environment. A novel computational model of foraging suggests that this search strategy could be adaptive in naturalistic settings where both environments and the objects within them provide partial information about hidden, uncertain rewards.

Unconscious Knowledge Produces Adaptive Decisions via Conscious Judgments in a Novel Instrumental Conditioning Paradigm

The demonstration of unconscious instrumental conditioning (i.e., unconsciously learning to choose stimuli that lead to rewards) is central for the tenet that unconscious learning supports human adaptation. Recent studies, using reliable subliminal conditioning procedures, have found evidence against unconscious instrumental conditioning. The present preregistered study proposes an alternative paradigm, in which unconscious processing is stimulated not by the subliminal exposure of the predictive (conditioned) stimuli, but by employing predictive regularities that are complex and difficult to detect consciously. Participants (N = 211) were exposed to letter strings that, unknown to them, were built from two complex artificial grammars: an “rewarded’’ or a “non-rewarded” grammar. On each trial, participants memorized a string, and subsequently had to discriminate the memorized string from a distractor. Correct discriminations were rewarded only when the identified string followed the rewarded grammar, but not when it followed the non-rewarded grammar. In a subsequent test phase, participants were presented with new strings from the rewarded and from the unrewarded grammar. Their task was now to directly choose the strings from the rewarded grammar, in order to collect more rewards. Employing a trial-by-trial awareness measure widely used in implicit learning, we found that participants accurately choose novel strings from the rewarded grammar when they had no conscious knowledge of the grammar. The awareness measure also showed that participants were accurate only when the unconsciously learned grammar led to conscious judgments. The present study provides an alternative to subliminal conditioning paradigms and shows evidence for unconscious instrumental conditioning.

The value of confidence: Confidence prediction errors drive value-based learning in the absence of external feedback

Reinforcement learning algorithms have a long-standing success story in explaining the dynamics of instrumental conditioning in humans and other species. While normative reinforcement learning models are critically dependent on external feedback, recent findings in the field of perceptual learning point to a crucial role of internally-generated reinforcement signals based on subjective confidence, when external feedback is not available. Here, we investigated the existence of such confidence-based learning signals in a key domain of reinforcement-based learning: instrumental conditioning. We conducted a value-based decision making experiment which included phases with and without external feedback and in which participants reported their confidence in addition to choices. Behaviorally, we found signatures of self-reinforcement in phases without feedback, reflected in an increase of subjective confidence and choice consistency. To clarify the mechanistic role of confidence in value-based learning, we compared a family of confidence-based learning models with more standard models predicting either no change in value estimates or a devaluation over time when no external reward is provided. We found that confidence-based models indeed outperformed these reference models, whereby the learning signal of the winning model was based on the prediction error between current confidence and a stimulus-unspecific average of previous confidence levels. Interestingly, individuals with more volatile reward-based value updates in the presence of feedback also showed more volatile confidence-based value updates when feedback was not available. Together, our results provide evidence that confidence-based learning signals affect instrumentally learned subjective values in the absence of external feedback.

A Modified Two-Way Active Avoidance Test for Combined Contextual and Auditory Instrumental Conditioning

Available two-way active avoidance paradigms do not provide contextual testing, likely due to challenges in performing repetitive trials of context exposure. To incorporate contextual conditioning in the two-way shuttle box, we contextually modified one of the chambers of a standard two-chamber rat shuttle box with visual cues consisting of objects and black and white stripe patterns. During the 5 training days, electrical foot shocks were delivered every 10 s in the contextually modified chamber but were signaled by a tone in the plain chamber. Shuttling between chambers prevented an incoming foot shock (avoidance) or aborted an ongoing one (escape). During contextual retention testing, rats were allowed to freely roam in the box. During auditory retention testing, visual cues were removed, and tone-signaled shocks were delivered in both chambers. Avoidance gradually replaced escape or freezing behaviors reaching 80% on the last training day in both chambers. Rats spent twice more time in the plain chamber during contextual retention testing and had 90% avoidance rates during auditory retention testing. Our modified test successfully assesses both auditory and contextual two-way active avoidance. By efficiently expanding its array of outcomes, our novel test will complement standard two-way active avoidance in mechanistic studies and will improve its applications in translational research.

Pavlovian and Instrumental Conditioning

Conditioning is the change in the response to a stimulus either because of the relation of that stimulus to other stimuli (Pavlovian conditioning), or because of the relation between the response and other stimuli (instrumental conditioning). These relations are formulated in terms of differences in conditional probability known as contingencies. Pavlovian contingencies refer to the difference in the conditional probability of one stimulus (the outcome, or O) given the presence vs. the absence of another stimulus (the conditioned stimulus, or CS). A conditioned response (CR) may be strengthened by a positive Pavlovian contingency (excitatory conditioning) or it may be weakened by a negative Pavlovian contingency (inhibitory conditioning). CRs are anticipatory or modified responses to the O, so their topography depends on the nature of the O (appetitive vs. aversive); the proximity between and congruency of O and CS; prior experience with the CS, O, and their contingency; the magnitude of their contingency; and the characteristics of other stimuli in the environment. Instrumental contingencies refer to the relation between one stimulus (the discriminative stimulus, or SD), a response (or operant, R), and the outcome of that response (O). The nature of the O and of its contingency with the R determines whether the O strengthens or weakens the R: Os that introduce an appetitive stimulus (positive reinforcement) or remove an aversive stimulus (negative reinforcement) strengthen the R. Positive reinforcement is typically arranged on a subset of one or more Rs following a set of rules known as a schedule of reinforcement. The probability that an R is reinforced may depend on the number of Rs (ratio schedules) or the amount of time (interval schedules) since the last reinforcer. The topography and strength of instrumental Rs depend on variables that are analogous to those that affect Pavlovian CRs: the amount and nature of prior experience with the O; the proximity, congruency, and contingency of R and O; and characteristics of other stimuli in past and present environments. Contemporary quantitative models of Pavlovian and instrumental conditioning recognize the importance of contextual stimuli that compete for cognitive and behavioral resources, constraining and shaping the expression of target responses. These models have guided the bulk of recent empirical research and conceptual developments, leading to a progressively unified view of learning and motivation processes. Along the way, Pavlovian and instrumental research have demonstrated their utility in addressing a broad range of consequential societal problems.

Interspecific two-dimensional visual discrimination of faces in horses (Equus caballus)

In social animals, recognizing conspecifics and distinguishing them from other animal species is certainly important. We hypothesize, as demonstrated in other species of ungulates, that horses are able to discriminate between the faces of conspecifics and the faces of other domestic species (cattle, sheep, donkeys and pigs). Our hypothesis was tested by studying inter-and intra-specific visual discrimination abilities in horses through a two-way instrumental conditioning task (discrimination and reversal learning), using two-dimensional images of faces as discriminative stimuli and food as a positive reward. Our results indicate that 8 out of 10 horses were able to distinguish between two-dimensional images of the faces of horses and images showing the faces of other species. A similar performance was obtained in the reversal task. The horses’ ability to learn by discrimination is therefore comparable to other ungulates. Horses also showed the ability to learn a reversal task. However, these results were obtained regardless of the images the tested horses were exposed to. We therefore conclude that horses can discriminate between two dimensional images of conspecifics and two dimensional images of different species, however in our study, they were not able to make further subcategories within each of the two categories. Despite the fact that two dimensional images of animals could be treated differently from two dimensional images of non-social stimuli, our results beg the question as to whether a two-dimensional image can replace the real animal in cognitive tests.

A global framework for a systemic view of brain modeling

The brain is a complex system, due to the heterogeneity of its structure, the diversity of the functions in which it participates and to its reciprocal relationships with the body and the environment. A systemic description of the brain is presented here, as a contribution to developing a brain theory and as a general framework where specific models in computational neuroscience can be integrated and associated with global information flows and cognitive functions. In an enactive view, this framework integrates the fundamental organization of the brain in sensorimotor loops with the internal and the external worlds, answering four fundamental questions (what, why, where and how). Our survival-oriented definition of behavior gives a prominent role to pavlovian and instrumental conditioning, augmented during phylogeny by the specific contribution of other kinds of learning, related to semantic memory in the posterior cortex, episodic memory in the hippocampus and working memory in the frontal cortex. This framework highlights that responses can be prepared in different ways, from pavlovian reflexes and habitual behavior to deliberations for goal-directed planning and reasoning, and explains that these different kinds of responses coexist, collaborate and compete for the control of behavior. It also lays emphasis on the fact that cognition can be described as a dynamical system of interacting memories, some acting to provide information to others, to replace them when they are not efficient enough, or to help for their improvement. Describing the brain as an architecture of learning systems has also strong implications in Machine Learning. Our biologically informed view of pavlovian and instrumental conditioning can be very precious to revisit classical Reinforcement Learning and provide a basis to ensure really autonomous learning.