Warren B. Powell,December 7, 2009
The first step in solving a stochastic optimization problem is providing a mathematical model. How
the problem is modeled can impact the solution strategy. In this chapter, we provide a exible
modeling framework that uses a classic control-theoretic framework, avoiding devices such as one-
step transition matrices. We describe the five fundamental elements of any stochastic, dynamic
program. Different notational conventions are introduced, and the types of policies that can be used
to guide decisions are described in detail. This discussion puts approximate dynamic programming in
the context of a variety of other algorithmic strategies by using the modeling framework to describe
a wide range of policies. A brief discussion of model-free programming is also provided.