Many decisions involve a choice between exploring unknown opportunities and exploiting well-known options. Work across a variety of domains, from animal foraging to human decision making, has suggested that animals solve such ``explore-exploit dilemmas'' with a mixture of two strategies: one driven by information seeking (directed exploration) and the other by behavioral variability (random exploration). Here we propose a unifying account in which these two strategies emerge from a kind of stochastic planning, known in the machine learning literature as Deep Exploration. In this model, the explore-exploit decision is made by stochastic simulation of plausible futures that are deep, in that they extend far into the future, and narrow, in that the number of possible futures they consider is small. By applying Deep Exploration to a simple explore-exploit task we show theoretically how directed and random exploration can emerge in these settings. Moreover, we show that Deep Exploration implies a tradeoff between directed and random exploration that is mediated by the number of simulations, or samples --- with more samples leading to increased directed exploration and decreased random exploration at the expense of greater time taken to respond. By measuring human behavior on the same simple task, we show that this reaction-time-mediated tradeoff exists in human behavior both between and within participants. We therefore suggest that Deep Exploration is a unifying account of explore-exploit behavior in humans.
Dual learning processes underlying human decision-making in reversal learning tasks: functional significance and evidence from the model fit to human behavior