Stochastic Processes
and Models
David Stirzaker
St John’s College, Oxford
Contents
Preface ix
1 Probability and random variables 1
1.1 Probability 1
1.2 Conditional probability and independence 4
1.3 Random variables 6
1.4 Random vectors 12
1.5 Transformations of random variables 16
1.6 Expectation and moments 22
1.7 Conditioning 27
1.8 Generating functions 33
1.9 Multivariate normal 37
2 Introduction to stochastic processes 45
2.1 Preamble 45
2.2 Essential examples; random walks 49
2.3 The long run 56
2.4 Martingales 63
2.5 Poisson processes 71
2.6 Renewals 76
2.7 Branching processes 87
2.8 Miscellaneous models 94
2.9 Some technical details 101
3 Markov chains 107
3.1 The Markov property; examples 107
3.2 Structure and n-step probabilities 116
3.3 First-step analysis and hitting times 121
3.4 The Markov property revisited 127
3.5 Classes and decomposition 132
3.6 Stationary distribution: the long run 135
3.7 Reversible chains 147
vi Contents
3.8 Simulation and Monte Carlo 151
3.9 Applications 157
4 Markov chains in continuous time 169
4.1 Introduction and examples 169
4.2 Forward and backward equations 176
4.3 Birth processes: explosions and minimality 186
4.4 Recurrence and transience 191
4.5 Hitting and visiting 194
4.6 Stationary distributions and the long run 196
4.7 Reversibility 202
4.8 Queues 205
4.9 Miscellaneous models 209
5 Diffusions 219
5.1 Introduction: Brownian motion 219
5.2 The Wiener process 228
5.3 Reflection principle; first-passage times 235
5.4 Functions of diffusions 242
5.5 Martingale methods 250
5.6 Stochastic calculus: introduction 256
5.7 The stochastic integral 261
5.8 Itô’s formula 267
5.9 Processes in space 273
6 Hints and solutions for starred exercises and problems 297
Further reading 323
Index 325