Network-Based Approximate Linear Programming for Discrete Optimization

Several prescriptive tasks in business and engineering as well as prediction in machine learning entail the solution of challenging discrete optimization problems. We recast the typical optimization formulation of these problems as high-dimensional dynamic programs and approach their approximation via linear programming. We develop tractable approximate linear programs with supporting theory by bringing together tools from state-space aggregations, networks, and perfect graphs (i.e., graph completions). We embed these models in a simple branch-and-bound scheme to solve applications in marketing analytics and the maintenance of energy or city-owned assets. We find that the resulting technique substantially outperforms a state-of-the-art commercial solver as well as aggregation-heuristics in terms of both solution quality and time. Our results motivate further consideration of networks and graph theory in approximate linear programming for solving deterministic and stochastic discrete optimization problems.

Logistic Markov Decision Processes

User modeling in advertising and recommendation has typically focused on myopic predictors of user responses. In this work, we consider the long-term decision problem associated with user interaction. We propose a concise specification of long-term interaction dynamics by combining factored dynamic Bayesian networks with logistic predictors of user responses, allowing state-of-the-art prediction models to be seamlessly extended. We show how to solve such models at scale by providing a constraint generation approach for approximate linear programming that overcomes the variable coupling and non-linearity induced by the logistic regression predictor. The efficacy of the approach is demonstrated on advertising domains with up to 2^54 states and 2^39 actions.

Constrained Bayesian Reinforcement Learning via Approximate Linear Programming

In this paper, we consider the safe learning scenario where we need to restrict the exploratory behavior of a reinforcement learning agent. Specifically, we treat the problem as a form of Bayesian reinforcement learning in an environment that is modeled as a constrained MDP (CMDP) where the cost function penalizes undesirable situations. We propose a model-based Bayesian reinforcement learning (BRL) algorithm for such an environment, eliciting risk-sensitive exploration in a principled way. Our algorithm efficiently solves the constrained BRL problem by approximate linear programming, and generates a finite state controller in an off-line manner. We provide theoretical guarantees and demonstrate empirically that our approach outperforms the state of the art.