1. Introduction
1.1 The dynamic programming and reinforcement learning problem
1.2 Approximation in dynamic programming and reinforcement learning
1.3 About this book
2. An introduction to dynamic programming and reinforcement learning
2.1 Introduction
2.2 Markov decision processes
2.2.1 Deterministic setting
2.2.2 Stochastic setting
2.3 Value iteration
2.3.1 Model-based value iteration
2.3.2 Model-free value iteration and the need for exploration
2.4 Policy iteration
2.4.1 Model-based policy iteration
2.4.2 Model-free policy iteration
2.5 Policy search
2.6 Summary and discussion
3. Dynamic programming and reinforcement learning in large and continuous spaces
3.1 Introduction
3.2 The need for approximation in large and continuous spaces
3.3 Approximation architectures
3.3.1 Parametric approximation
3.3.2 Nonparametric approximation
3.3.3 Comparison of parametric and nonparametric approximation
3.3.4 Remarks
3.4 Approximate value iteration
3.4.1 Model-based value iteration with parametric approximation
3.4.2 Model-free value iteration with parametric approximation
3.4.3 Value iteration with nonparametric approximation
3.4.4 Convergence and the role of nonexpansive approximation
3.4.5 Example: Approximate Q-iteration for a DC motor
3.5 Approximate policy iteration
3.5.1 Value iteration-like algorithms for approximate policy
evaluation
3.5.2 Model-free policy evaluation with linearly parameterized approximation
3.5.3 Policy evaluation with nonparametric approximation
3.5.4 Model-based approximate policy evaluation with rollouts
3.5.5 Policy improvement and approximate policy iteration
3.5.6 Theoretical guarantees
3.5.7 Example: Least-squares policy iteration for a DC motor
3.6 Finding value function approximators automatically
3.6.1 Basis function optimization
3.6.2 Basis function construction
3.6.3 Remarks
3.7 Approximate policy search
3.7.1 Policy gradient and actor-critic algorithms
3.7.2 Gradient-free policy search
3.7.3 Example: Gradient-free policy search for a DC motor
3.8 Comparison of approximate value iteration, policy iteration, and policy search
3.9 Summary and discussion
4. Approximate value iteration with a fuzzy representation
4.1 Introduction
4.2 Fuzzy Q-iteration
4.2.1 Approximation and projection mappings of fuzzy Q-iteration
4.2.2 Synchronous and asynchronous fuzzy Q-iteration
4.3 Analysis of fuzzy Q-iteration
4.3.1 Convergence
4.3.2 Consistency
4.3.3 Computational complexity
4.4 Optimizing the membership functions
4.4.1 A general approach to membership function optimization
4.4.2 Cross-entropy optimization
4.4.3 Fuzzy Q-iteration with cross-entropy optimization of the membership functions
4.5 Experimental study
4.5.1 DC motor: Convergence and consistency study
4.5.2 Two-link manipulator: Effects of action interpolation, and comparison with fitted Q-iteration
4.5.3 Inverted pendulum: Real-time control
4.5.4 Car on the hill: Effects of membership function optimization
4.6 Summary and discussion
5. Approximate policy iteration for online learning and continuous-action control
5.1 Introduction
5.2 A recapitulation of least-squares policy iteration
5.3 Online least-squares policy iteration
5.4 Online LSPI with prior knowledge
5.4.1 Online LSPI with policy approximation
5.4.2 Online LSPI with monotonic policies
5.5 LSPI with continuous-action, polynomial approximation
5.6 Experimental study
5.6.1 Online LSPI for the inverted pendulum
5.6.2 Online LSPI for the two-link manipulator
5.6.3 Online LSPI with prior knowledge for the DC motor
5.6.4 LSPI with continuous-action approximation for the inverted pendulum
5.7 Summary and discussion
6. Approximate policy search with cross-entropy optimization of basis functions
6.1 Introduction
6.2 Cross-entropy optimization
6.3 Cross-entropy policy search
6.3.1 General approach
6.3.2 Cross-entropy policy search with radial basis functions
6.4 Experimental study
6.4.1 Discrete-time double integrator
6.4.2 Bicycle balancing
6.4.3 Structured treatment interruptions for HIV infection control
6.5 Summary and discussion
Appendix A. Extremely randomized trees
A.1 Structure of the approximator
A.2 Building and using a tree
Appendix B. The cross-entropy method
B.1 Rare-event simulation using the cross-entropy method
B.2 Cross-entropy optimization
Symbols and abbreviations
Bibliography
List of algorithms
Index
· · · · · · (收起)