Agents that can make better use of computation, experience, time, and memory can solve a greater range of problems more effectively. A crucial ingredient for managing such finite resources is intelligently chosen abstract representations. But, how do abstractions facilitate problem solving under limited resources? What makes an abstraction useful? To answer such questions, we review several trends in recent reinforcement-learning research that provide insight into how abstractions interact with learning and decision making. During learning, abstraction can guide exploration and generalization as well as facilitate efficient tradeoffs---e.g., time spent learning versus the quality of a solution. During computation, good abstractions provide simplified models for computation while also preserving relevant information about decision-theoretic quantities. These features of abstraction are not only key for scaling up artificial problem solving, but can also shed light on what pressures shape the use of abstract representations in humans and other organisms.
Accessing relevant information during problem solving: Time constraints on search in the problem space
