Daniela Calvetti Erkki Somersalo

Introduction to Bayesian

Scientific Computing

Ten Lectures on Subjective Computing
c 2007 Springer Science+Business Media, LLC
1 Inverse problems and subjective computing . . . . . . . . . . . . . . . . 1
1.1 What do we talk about when we talk about random variables? 2
1.2 Through the formal theory, lightly . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 How normal is it to be normal? . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2 Basic problem of statistical inference . . . . . . . . . . . . . . . . . . . . . . 21
2.1 On averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2 Maximum Likelihood, as frequentists like it . . . . . . . . . . . . . . . . . 31
3 The praise of ignorance: randomness as lack of information 39
3.1 Construction of Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 Enter, Subject: Construction of Priors . . . . . . . . . . . . . . . . . . . . . . 48
3.3 Posterior Densities as Solutions of Statistical Inverse Problems 55
4 Basic problem in numerical linear algebra . . . . . . . . . . . . . . . . . 61
4.1 What is a solution? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2 Direct linear system solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.3 Iterative linear system solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.4 Ill-conditioning and errors in the data . . . . . . . . . . . . . . . . . . . . . . 77
5 Sampling: first encounter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.1 Sampling from Gaussian distributions . . . . . . . . . . . . . . . . . . . . . . 92
5.2 Random draws from non-Gaussian densities . . . . . . . . . . . . . . . . . 99
5.3 Rejection sampling: prelude to Metropolis-Hastings . . . . . . . . . . 102
6 Statistically inspired preconditioners. . . . . . . . . . . . . . . . . . . . . . . 107
6.1 Priorconditioners: specially chosen preconditioners . . . . . . . . . . . 108
6.2 Sample-based preconditioners and PCA model reduction . . . . . 118
XIV Contents
7 Conditional Gaussian densities and predictive envelopes . . . 127
7.1 Gaussian conditional densities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.2 Interpolation, splines and conditional densities . . . . . . . . . . . . . . 134
7.3 Envelopes, white swans and dark matter . . . . . . . . . . . . . . . . . . . 144
8 More applications of the Gaussian conditioning . . . . . . . . . . . . 147
8.1 Linear inverse problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
8.2 Aristotelian boundary conditions . . . . . . . . . . . . . . . . . . . . . . . . . . 151
9 Sampling: the real thing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
9.1 Metropolis–Hastings algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
10 Wrapping up: hypermodels, dynamic priorconditioners
and Bayesian learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10.1 MAP estimation or marginalization? . . . . . . . . . . . . . . . . . . . . . . . 189
10.2 Bayesian hypermodels and priorconditioners . . . . . . . . . . . . . . . . 193
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197