StochasticProcesses, Estimation, and Control
1 Probability Theory 1
1.1 Probability Theory as a Set of Outcomes
1.2 Set Theory .
1.3 Probability Space and the Probability Measure
1.4 Algebras of Sets and Probability Space
1.5 Key Concepts in Probability Theory .
1.6 Exercises .
2 Random Variables and Stochastic Processes 25
2.1 Random Variables . . .
2.2 Probability Distribution Function .
2.3 Probability Density Function
2.4 Probabilistic Concepts Applied to Random Variables .
2.5 Functions of a Random Variable .
2.6 Expectations and Moments of a Random Variable
2.7 Characteristic Functions
2.8 Conditional Expectations and Conditional Probabilities .
2.9 Stochastic Processes .
2.10 Gauss–Markov Processes .
2.11 Nonlinear Stochastic Difference Equations .
2.12 Exercises .
3 Conditional Expectations and Discrete-Time Kalman Filtering 81
3.1 Minimum Variance Estimation
3.2 Conditional Estimate of a Gaussian Random Vector with Additive Gaussian
Noise
3.2.1 Simplification of the Argument of the Exponential
3.2.2 Simplification of the Coefficient of the Exponential
3.2.3 Processing Measurements Sequentially
3.2.4 Statistical Independence of the Error and the Estimate .
3.3 Maximum Likelihood Estimation .
3.4 The Discrete-Time Kalman Filter: Conditional Mean Estimator .
3.5 “Tuning” a Kalman Filter .
3.6 Discrete-Time Nonlinear Filtering
3.6.1 Dynamic Propagation .
3.6.2 Measurement Update .
3.7 Exercises .
4 Least Squares, the Orthogonal ProjectionLemma, and Discrete-TimeKalman
Filtering 119
4.1 Linear Least Squares .
4.2 The Orthogonal Projection Lemma .
4.3 Extensions of Least Squares Theory
4.4 Nonlinear Least Squares: Newton–Gauss Iteration
4.5 Deriving the Kalman Filter via the Orthogonal Projection Lemma .
4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5 Stochastic Processes and Stochastic Calculus 153
5.1 RandomWalk and Brownian Motion . . . . . . . . . . . . . . . . . . . 153
5.2 Mean-Square Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
5.3 Wiener Integrals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
5.4 Itô Integrals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.5 Second-Order Itô Integrals . . . . . . . . . . . . . . . . . . . . . . . . . 178
5.6 Stochastic Differential Equations and Exponentials . . . . . . . . . . . . 180
5.7 The Itô Stochastic Differential . . . . . . . . . . . . . . . . . . . . . . . 182
5.8 Continuous-Time Gauss–Markov Processes . . . . . . . . . . . . . . . 186
5.9 Propagation of the Probability Density Function . . . . . . . . . . . . . 190
5.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
6 Continuous-Time Gauss–Markov Systems: Continuous-Time Kalman Filter,
Stationarity, Power Spectral Density, and theWiener Filter 197
6.1 The Continuous-Time Kalman Filter (Kalman–Bucy Filter) . . . . . . . 197
6.2 Properties of the Continuous-Time Riccati Equation . . . . . . . . . . . 202
6.3 Stationarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
6.4 Power Spectral Densities . . . . . . . . . . . . . . . . . . . . . . . . . 207
6.4.1 Fourier Transforms . . . . . . . . . . . . . . . . . . . . . . . . 207
6.4.2 Fourier Analysis Applied to Random Processes . . . . . . . . . . 208
6.4.3 Ergodic Random Processes . . . . . . . . . . . . . . . . . . . . 214
6.5 Continuous-Time Linear Systems Driven by Stationary Signals . . . . . 215
6.6 Discrete-Time Linear Systems Driven by Stationary Random
Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
6.7 The Steady-State Kalman Filter: TheWiener Filter . . . . . . . . . . . . 223
6.7.1 TheWiener Filtering Problem Statement . . . . . . . . . . . . . 223
6.7.2 Solving theWiener–Hopf Equation . . . . . . . . . . . . . . . . 225
6.7.3 Noncausal Filter . . . . . . . . . . . . . . . . . . . . . . . . . . 226
6.7.4 The Causal Filter . . . . . . . . . . . . . . . . . . . . . . . . . . 228
6.7.5 Wiener Filtering by Orthogonal Projections . . . . . . . . . . . . 233
6.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
7 The Extended Kalman Filter 241
7.1 Linearized Kalman Filtering . . . . . . . . . . . . . . . . . . . . . . . . 241
7.1.1 Continuous-Time Theory . . . . . . . . . . . . . . . . . . . . . 241
7.1.2 Discrete-Time Version . . . . . . . . . . . . . . . . . . . . . . . 243
7.2 The Extended Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . 244
7.3 The Iterative Extended Kalman Filter . . . . . . . . . . . . . . . . . . . 245
7.4 Filter Divergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
7.4.1 What is Divergence? . . . . . . . . . . . . . . . . . . . . . . . . 249
7.4.2 The Role of Process NoiseWeighting in the Steady
State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
7.4.3 An Analysis of Divergence . . . . . . . . . . . . . . . . . . . . 252
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
8 A Selection of Results from Estimation Theory 263
8.1 Continuous-Time Colored-Noise Filter . . . . . . . . . . . . . . . . . . 263
8.2 Optimal Smoothing and Filtering in Continuous Time . . . . . . . . . . 267
8.3 Discrete-Time Smoothing and Maximum Likelihood Estimation . . . . . 271
8.4 Linear Exponential Gaussian Estimation . . . . . . . . . . . . . . . . . 273
8.4.1 The LEG Estimator and Sherman’s Theorem . . . . . . . . . . . 273
8.4.2 Statistical Properties of the LEG Estimator and the Kalman Filter 275
8.5 Estimation with State-Dependent Noise . . . . . . . . . . . . . . . . . . 277
8.5.1 General Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 277
8.5.2 Application to Phase-Lock Loops . . . . . . . . . . . . . . . . . 279
8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
9 Stochastic Control and the Linear Quadratic Gaussian Control Problem 289
9.1 Dynamic Programming: An Illustration . . . . . . . . . . . . . . . . . . 289
9.2 Stochastic Dynamical System . . . . . . . . . . . . . . . . . . . . . . . 291
9.2.1 Stochastic Control Problem with Perfect Observation . . . . . . 292
9.3 Dynamic Programming Algorithm . . . . . . . . . . . . . . . . . . . . 292
9.4 Stochastic LQ Problems with Perfect Information . . . . . . . . . . . . 295
9.4.1 Application of the Dynamic Programming Algorithm . . . . . . 295
9.5 Dynamic Programming with Partial Information . . . . . . . . . . . . . 297
9.5.1 Sufficient Statistics . . . . . . . . . . . . . . . . . . . . . . . . . 299
9.6 The Discrete-Time LQG Problem with Partial Information . . . . . . . . 300
9.6.1 The Discrete-Time LQG Solution . . . . . . . . . . . . . . . . . 300
9.6.2 Insights into the Partial Information, Discrete-Time LQG
Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
9.6.3 Stability Properties of the LQG Controller with Partial
Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
9.7 The Continuous-Time LQG Problem . . . . . . . . . . . . . . . . . . . 305
9.7.1 Dynamic Programming for Continuous-Time Markov
Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
9.7.2 The LQG Problem with Complete Information . . . . . . . . . . 307
9.7.3 LQ Problem with State- and Control-Dependent Noise . . . . . . 309
9.7.4 The LQG Problem with Partial Information . . . . . . . . . . . . 310
9.8 Stationary Optimal Control . . . . . . . . . . . . . . . . . . . . . . . . 317
9.8.1 General Conditions . . . . . . . . . . . . . . . . . . . . . . . . 317
9.8.2 The Stationary LQG Controller . . . . . . . . . . . . . . . . . . 320
9.9 LQG Control with Loop Transfer Recovery . . . . . . . . . . . . . . . . 321
9.9.1 The Guaranteed Gain Margins of LQ Optimal
Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
9.9.2 Deriving the LQG/LTR Controller . . . . . . . . . . . . . . . . 326
9.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
10 Linear Exponential Gaussian Control and Estimation 335
10.1 Discrete-Time LEG Control . . . . . . . . . . . . . . . . . . . . . . . . 335
10.1.1 Formulation of the LEG Problem . . . . . . . . . . . . . . . . . 335
10.1.2 Solution Methodology and Properties of the LEG
Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
10.1.3 LEG Controller Solution . . . . . . . . . . . . . . . . . . . . . . 341
10.1.4 The LEG Estimator . . . . . . . . . . . . . . . . . . . . . . . . 351
10.2 Terminal Guidance: A Special Continuous-Time LEG Problem . . . . . 355
10.3 Continuous-Time LEG Control . . . . . . . . . . . . . . . . . . . . . . 362
10.4 LEG Controllers and H∞ . . . . . . . . . . . . . . . . . . . . . . . . . 364
10.4.1 The LEG Controller and Its Relationship with the
Disturbance Attenuation Problem . . . . . . . . . . . . . . . . . 365
10.4.2 The Time-Invariant LEG Estimator Transformed into the H∞
Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
10.4.3 The H∞ Measure and the H∞ Robustness Bound . . . . . . . . . 368
10.4.4 The Time-Invariant, Infinite-Time LEG Controller and Its Relationship
with H∞ . . . . . . . . . . . . . . . . . . . . . . . . . 369
10.4.5 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
10.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
Appendix A. Proof of Lemma 10.1 . . . . . . . . . . . . . . . . . . . . . . . 373
Appendix B. Proof of Lemma 10.2 . . . . . . . . . . . . . . . . . . . . . . . 374