Uncertainty avoidance versus conditioned reinforcement: exploring paradoxical choice in rats

Paying a cost to reduce uncertainty can be adaptive, because better informed decision-makers can align their preferences to opportunities. However, birds and mammals display an appetite for information that they cannot use to functionally alter behaviour or its outcomes. We explore two putative motivational mechanisms for this paradoxical behaviour. The information hypothesis proposes that reducing uncertainty is reinforcing per se, consistent with the concept of curiosity: a motivation to know, in the absence of instrumental benefits. In contrast, the conditioned reinforcement hypothesis sees information-seeking as a consequence of asymmetries in secondarily acquired reinforcement: responding increments caused by post-choice stimuli announcing positive outcomes (S+) exceed decrements caused by stimuli signalling absence of reward (S−). We contrast these hypotheses experimentally. Rats chose between two equally profitable options delivering food probabilistically after a fixed delay. In the informative option (Info), the outcome (food/no food) was signalled immediately after choice, whereas in the non-informative option (NoInfo) outcomes were uncertain until the delay lapsed. Subjects preferred (Info) when (1) outcomes were signalled by salient auditory cues, (2) only the absence of reward was signalled, and (3) only reward was signalled, though acquisition was slower when rewards were not explicitly signalled. Our results show that a salient good news signal is not required as a conditioned reinforcer to generate paradoxical preferences. Terminal preferences support the information hypothesis but the slower acquisition of (Info) preference when S+ is not present is consistent with the conditioning account. We conclude that both uncertainty reduction and conditioned reinforcement influence choice.

From Knowing to Feeling

Visual and auditory processes represent sensory information, but do not evaluate its importance for survival or success. Interactions between perceptual/cognitive and evaluative reinforcement/emotional/motivational mechanisms accomplish this. Cognitive-emotional resonances support conscious feelings, knowing their source, and controlling motivation and responses to acquire valued goals. Also explained is how emotions may affect behavior without being conscious, and how learning adaptively times actions to achieve desired goals. Breakdowns in cognitive-emotional resonances can cause symptoms of mental disorders such as depression, autism, schizophrenia, and ADHD, including explanations of how affective meanings fail to organize behavior when this happens. Historic trends in the understanding of cognition and emotion are summarized, including work of Chomsky and Skinner. Brain circuits of conditioned reinforcer learning and incentive motivational learning are modeled, including the inverted-U in conditioning as a function of interstimulus interval, secondary conditioning, and attentional blocking and unblocking. How humans and animals act as minimal adaptive predictors is explained using the CogEM model’s interactions between sensory cortices, amygdala, and orbitofrontal cortex. Cognitive-emotional properties solve phylogenetically ancient Synchronization and Persistence Problems using circuits that are conserved between mollusks and humans. Avalanche command circuits for learning arbitrary sequences of sensory-motor acts, dating back to crustacea, increase their sensitivity to environmental feedback as they morph over phylogeny into mammalian cognitive and emotional circuits. Antagonistic rebounds drive affective extinction. READ circuits model how life-long learning occurs without associative saturation or passive forgetting. Affective memories of opponent emotions like fear vs. relief can then persist until they are disconfirmed by environmental feedback.

Devaluation of a conditioned reinforcer requires its reexposure

A change in motivational state does not guarantee a change in operant behaviour. Only after an organism has had contact with an outcome while in a relevant motivational state does behaviour change, a phenomenon called incentive learning. While ample evidence indicates that this is true for primary reinforcers, it has not been established for conditioned reinforcers. We performed an experiment with rats where lever-presses were reinforced by presentations of an audiovisual stimulus that had previously preceded food delivery; in the critical experimental groups, the audiovisual stimulus was then paired a single time with a strong electric shock. Some animals were reexposed to the audiovisual stimulus. Lever-presses yielding no outcomes were recorded in a subsequent test. Animals that had been reexposed to the audiovisual stimulus after the aversive training responded less than did those that had not received reexposure. Indeed, those animals that were not reexposed did not differ from a control group that received no aversive conditioning of the audiovisual stimulus. Moreover, these results were not mediated by a change in the food’s reinforcement value, but instead reflect a change in behaviour with respect to the conditioned reinforcer itself. These are the first data to indicate that the affective value of conditioned stimuli, like that of unconditioned ones, is established when the organism comes into contact with them.

What’s in a Click? The Efficacy of Conditioned Reinforcement in Applied Animal Training: A Systematic Review and Meta-Analysis

A conditioned reinforcer is a stimulus that acquired its effectiveness to increase and maintain a target behavior on the basis of the individual’s history—e.g., pairings with other reinforcers. This systematic review synthesized findings on conditioned reinforcement in the applied animal training field. Thirty-four studies were included in the review and six studies were eligible for a meta-analysis on the effectiveness of behavioral interventions that implemented conditioned reinforcement (e.g., clicks, spoken word, or whistles paired with food). The majority of studies investigated conditioned reinforcement with dogs (47%, n = 16) and horses (30%, n = 10) implementing click–food pairings. All other species (cats, cattle, fish, goats, and monkeys) were equally distributed across types of conditioned (e.g., clicker or spoken word) and unconditioned reinforcers (e.g., food, water, or tactile). A meta-analysis on the effectiveness of conditioned reinforcement in behavioral interventions found a medium summary effect size (Tau-U 0.77; CI95% = [0.53, 0.89]), when comparing baseline, where no training was done, and treatment levels. Moderators of conditioned reinforcement effectiveness were species (e.g., horses) and research design (e.g., multiple-baseline designs). The small number of intervention-focused studies available limits the present findings and highlights the need for more systematic research into the effectiveness of conditioned reinforcement across species.

Excitatory second-order conditioning using a backward first-order conditioned stimulus: A challenge for prediction error reduction

Prével and colleagues reported excitatory learning with a backward conditioned stimulus (CS) in a conditioned reinforcement preparation. Their results add to existing evidence of backward CSs sometimes being excitatory and were viewed as challenging the view that learning is driven by prediction error reduction, which assumes that only predictive (i.e., forward) relationships are learned. The results instead were consistent with the assumptions of both Miller’s Temporal Coding Hypothesis and Wagner’s Sometimes Opponent Processes (SOP) model. The present experiment extended the conditioned reinforcement preparation developed by Prével et al. to a backward second-order conditioning preparation, with the aim of discriminating between these two accounts. We tested whether a second-order CS can serve as an effective conditioned reinforcer, even when the first-order CS with which it was paired is a backward CS that elicits no responding. Evidence of conditioned reinforcement was found, despite no conditioned response (CR) being elicited by the first-order backward CS. The evidence of second-order conditioning in the absence of excitatory conditioning to the first-order CS is interpreted as a challenge to SOP. In contrast, the present results are consistent with the Temporal Coding Hypothesis and constitute a conceptual replication in humans of previous reports of excitatory second-order conditioning in rodents with a backward CS. The proposal is made that learning is driven by “discrepancy” with prior experience as opposed to “ prediction error.”

Token Economy: A Systematic Review of Procedural Descriptions

The token economy is a well-established and widely used behavioral intervention. A token economy is comprised of six procedural components: the target response(s), a token that functions as a conditioned reinforcer, backup reinforcers, and three interconnected schedules of reinforcement. Despite decades of applied research, the extent to which the procedures of a token economy are described in complete and replicable detail has not been evaluated. Given the inherent complexity of a token economy, an analysis of the procedural descriptions may benefit future token economy research and practice. Articles published between 2000 and 2015 that included implementation of a token economy within an applied setting were identified and reviewed with a focus on evaluating the thoroughness of procedural descriptions. The results show that token economy components are regularly omitted or described in vague terms. Of the articles included in this analysis, only 19% (18 of 96 articles reviewed) included replicable and complete descriptions of all primary components. Missing or vague component descriptions could negatively affect future research or applied practice. Recommendations are provided to improve component descriptions.