Flexible Path Planning Through Vicarious Trial and Error

Flexible planning is necessary for reaching goals and adapting when conditions change. We introduce a biologically plausible path planning model that learns its environment, rapidly adapts to change, and plans efficient routes to goals. Unlike prior models of hippocampal replay, our model addresses the decision-making process when faced with uncertainty. We tested the model in simulations of human and rodent navigation in mazes. Like the human and rat, the model was able to generate novel shortcuts, and take detours when familiar routes were blocked. Similar to rodent hippocampus recordings, the neural activity of the model resembles neural correlates of Vicarious Trial and Error (VTE) during early learning or during uncertain conditions. Similar to rodent studies, after learning, the neural activity resembles forward replay or preplay predicting a future route, and VTE activity decreases. We suggest that VTE, in addition to weighing possible outcomes, is a way in which an organism may gather information for future use.

A Machine Learning Approach for Detecting Vicarious Trial and Error Behaviors

Vicarious trial and error behaviors (VTEs) indicate periods of indecision during decision-making, and have been proposed as a behavioral marker of deliberation. In order to understand the neural underpinnings of these putative bridges between behavior and neural dynamics, researchers need the ability to readily distinguish VTEs from non-VTEs. Here we utilize a small set of trajectory-based features and standard machine learning classifiers to identify VTEs from non-VTEs for rats performing a spatial delayed alternation task (SDA) on an elevated plus maze. We also show that previously reported features of the hippocampal field potential oscillation can be used in the same types of classifiers to separate VTEs from non-VTEs with above chance performance. However, we caution that the modest classifier success using hippocampal population dynamics does not identify many trials where VTEs occur, and show that combining oscillation-based features with trajectory-based features does not improve classifier performance compared to trajectory-based features alone. Overall, we propose a standard set of features useful for trajectory-based VTE classification in binary decision tasks, and support previous suggestions that VTEs are supported by a network including, but likely extending beyond, the hippocampus.

Vicarious Trial-and-Error Is Enhanced During Deliberation in Human Virtual Navigation in a Translational Foraging Task

Foraging tasks provide valuable insights into decision-making as animals decide how to allocate limited resources (such as time). In rodents, vicarious trial-and-error (back and forth movements), or VTE, is an important behavioral measure of deliberation which is enhanced early in learning and when animals are presented with difficult decisions. Using new translational versions of a rodent foraging task (the “Movie Row” and “Candy Row”), humans navigated a virtual maze presented on standard computers to obtain rewards (either short videos or candy) offered after a variable delay. Decision latencies were longer when participants were presented with difficult offers, overrode their preferences, and when they accepted an offer after rejecting a previous offer. In these situations, humans showed VTE-like behavior, where they were more likely to pause and/or reorient one or more times before making a decision. Behavior on these tasks replicated previous results from the rodent foraging task (“Restaurant Row”) and a human version lacking a navigation component (“Web-Surf”) and revealed some species differences. Compared to survey measures of delay-discounting, willingness to wait for rewards in the foraging task was not related to willingness to wait for hypothetical rewards. And, smoking status (use of cigarettes or e-cigarettes) was associated with stronger discounting of hypothetical future rewards, but was not well-related to performance on the foraging tasks. In contrast, individuals with overweight or obese BMI (≥25) did not show stronger delay-discounting, but individuals with BMI ≥ 25, and especially females, showed reduced sensitivity to sunk-costs (where their decisions were less sensitive to irrecoverable investments of effort) and less deliberation when presented with difficult offers. These data indicate that VTE is a behavioral index of deliberation in humans, and further support the Movie and Candy Row as translational tools to study decision-making in humans with the potential to provide novel insights about decision-making that are relevant to public health.

A machine learning approach for detecting vicarious trial and error behaviors

Vicarious trial and error behaviors (VTEs) indicate periods of indecision during decision-making, and have been proposed as a behavioral marker of deliberation. In order to understand the neural underpinnings of these putative bridges between behavior and neural dynamics, researchers need the ability to readily distinguish VTEs from non-VTEs. Here we utilize a small set of trajectory-based features and standard machine learning classifiers to identify VTEs from non-VTEs for rats performing a spatial delayed alternation task (SDA) on an elevated plus maze. We also show that previously reported features of the hippocampal field potential oscillation can be used in the same types of classifiers to separate VTEs from non-VTEs with above chance performance. However, we caution that the modest classifier success using hippocampal population dynamics is not sufficient for identifying trials where VTEs occur, and show that combining oscillation-based features with trajectory-based features degrades classifier performance compared to trajectory-based features alone. Overall, we propose a standard set of features useful for trajectory-based VTE classification and support previous suggestions that VTEs are supported by a network including, but likely extending beyond, the hippocampus.

The Missing Link Between Memory and Reinforcement Learning

Reinforcement learning systems usually assume that a value function is defined over all states (or state-action pairs) that can immediately give the value of a particular state or action. These values are used by a selection mechanism to decide which action to take. In contrast, when humans and animals make decisions, they collect evidence for different alternatives over time and take action only when sufficient evidence has been accumulated. We have previously developed a model of memory processing that includes semantic, episodic and working memory in a comprehensive architecture. Here, we describe how this memory mechanism can support decision making when the alternatives cannot be evaluated based on immediate sensory information alone. Instead we first imagine, and then evaluate a possible future that will result from choosing one of the alternatives. Here we present an extended model that can be used as a model for decision making that depends on accumulating evidence over time, whether that information comes from the sequential attention to different sensory properties or from internal simulation of the consequences of making a particular choice. We show how the new model explains both simple immediate choices, choices that depend on multiple sensory factors and complicated selections between alternatives that require forward looking simulations based on episodic and semantic memory structures. In this framework, vicarious trial and error is explained as an internal simulation that accumulates evidence for a particular choice. We argue that a system like this forms the “missing link” between more traditional ideas of semantic and episodic memory, and the associative nature of reinforcement learning.

Vicarious trial-and-error is associated with the use of place-strategies in human virtual navigation

Studies of decision-making in rodents have demonstrated that vicarious trial-and-error (VTE) is an important behavioral index of deliberation, when animals search through and evaluate the available options before making a decision. In rodents, VTE is enhanced during the use of hippocampally-dependent place strategies, which may represent a type of model-based behavior. While some evidence exists for VTE-like behaviors in humans during navigation, it is unknown if VTE in humans is specifically associated place-strategies, as would be predicted for model-based behaviors. To address this gap, humans were tested in navigation tasks in symmetrical environments, which allowed for the use of probe trials to assess navigation strategies (place or response) or impose them directly. The use of place strategies (on probe trials and place-training) was associated with increases in measures of VTE (reorientations and pausing) especially at high-cost decision points, similar to results from rodent studies. In contrast, response-strategies were associated with the development of efficient, stereotyped trajectories (consistent with model-free learning). These results support the identification of place- and response-strategies in human navigation with model-based and model-free learning, respectively, and demonstrate that VTE is specifically related to the use of place-strategies.

Anticipation and Utility

This chapter discusses the processes of anticipation and valuation underlying neural networks. It first considers the concept of delay and temporality before introducing a value system. It is accepted that the nervous system knows how to manage time. But is it able to generate anticipatory representations of the future? Since the early 1950s, experimental psychologists have observed that when a rat is involved in an experimental task involving choices, it moves its head alternately in the direction of the various possible options. They called this behaviour ‘vicarious trial and error’ because they postulated that it corresponded to behavioural evidence of the cognitive processes of anticipatory deliberation. Anticipating is one thing, but economists argue that in order to be able to choose, one must attribute utility to the different options. This utility is subjective and only reflects the subject's preference for the option, but rationality can only be defined if the different options are evaluated according to preference. This chapter then looks at the role of dopaminergic neurons in stimulus-response association processes, exploring the properties of the responses of dopaminergic neurons during learning and decision-making processes.

Vicarious trial-and-error is enhanced during deliberation in human virtual navigation in a translational neuroeconomic task

Foraging tasks can provide valuable insights into decision-making, as animals choose how to allocate limited resources (such as time). In the “Restaurant Row” task, rodents move between several sites to obtain food rewards available after a variable delay, while in a translational version (the “Web-Surf”) lacking the navigation component, humans are offered short videos. Both tasks have provided novel insights into decision-making and have been applied to addiction vulnerability and the impact of drug exposure on decision-making. We tested new tasks (the “Movie Row” and “Candy Row”) which use virtual navigation to determine if the behavioral correlates of human decision-making are more broadly similar to those of rodents, and explored the relationship of task performance to smoking and obesity. Humans navigated a virtual maze presented on standard computers to obtain rewards (either short videos or candy) available after a variable delay. Behavior on the tasks replicated previous results for the Restaurant Row and Web-Surf. In conditions promoting deliberation, decision latency was elevated along with measures of vicarious trial-and-error (VTE), supporting VTE as a shared behavioral index of deliberation across species. Smoking status was not well-related to performance, while high BMI (> 25) individuals showed reduced sensitivity to a sunk-costs measure and stronger sensitivity to offer value for offers below the preferred delay. These data support the Movie and Candy Row as translational tools to study decision-making during foraging in humans, providing convergent results with a rodent navigation task and demonstrating the potential to provide novel insights relevant to public health.

Internally Organized Activity During Offline Brain States

A prime example of internally organized patterns is observed during sleep. The best studied of these is the sharp wave ripple in the hippocampus. Neuronal sequences during ripple events reach back to the past to replay snippets of waking experience at times when the brain is disengaged from the outside world. This process may consolidate episodic memories and stitch together discontiguous experiences, thereby giving rise to creative thoughts. In addition, neuronal assembly sequences during ripples also act as internalized, vicarious, trial-and-error mechanisms that can assist with subconscious optimization of future plans. Because the same neuronal substrate can perform both retrospective and prospective operations, it is not clear whether the traditional separation of postdiction (i.e., memory) from prediction (i.e., planning) is justified.