Parameter Estimation and Inverse Problems

Richard C. Aster, Brian Borchers, and Clifford H. Thurber

Copyright © 2005, Elsevier Inc. All rights reserved.






Contents


Preface xi
1 INTRODUCTION 1
1.1 Classification of Inverse Problems 1
1.2 Examples of Parameter Estimation Problems 4
1.3 Examples of Inverse Problems 7
1.4 Why Inverse Problems Are Hard 11
1.5 Exercises 14
1.6 Notes and Further Reading 14
2 LINEAR REGRESSION 15
2.1 Introduction to Linear Regression 15
2.2 Statistical Aspects of Least Squares 17
2.3 Unknown Measurement Standard Deviations 26
2.4 L1 Regression 30
2.5 Monte Carlo Error Propagation 35
2.6 Exercises 36
2.7 Notes and Further Reading 40
3 DISCRETIZING CONTINUOUS INVERSE
PROBLEMS 41
3.1 Integral Equations 41
3.2 Quadrature Methods 41
v
vi Contents
3.3 Expansion in Terms of Representers 46
3.4 Expansion in Terms of Orthonormal Basis Functions 47
3.5 The Method of Backus and Gilbert 48
3.6 Exercises 52
3.7 Notes and Further Reading 54
4 RANK DEFICIENCYAND ILL-CONDITIONING 55
4.1 The SVD and the Generalized Inverse 55
4.2 Covariance and Resolution of the Generalized Inverse Solution 62
4.3 Instability of the Generalized Inverse Solution 64
4.4 An Example of a Rank-Deficient Problem 67
4.5 Discrete Ill-Posed Problems 73
4.6 Exercises 85
4.7 Notes and Further Reading 87
5 TIKHONOV REGULARIZATION 89
5.1 Selecting a Good Solution 89
5.2 SVD Implementation of Tikhonov Regularization 91
5.3 Resolution, Bias, and Uncertainty in the Tikhonov Solution 95
5.4 Higher-Order Tikhonov Regularization 98
5.5 Resolution in Higher-Order Tikhonov Regularization 103
5.6 The TGSVD Method 105
5.7 Generalized Cross Validation 106
5.8 Error Bounds 109
5.9 Exercises 114
5.10 Notes and Further Reading 117
6 ITERATIVE METHODS 119
6.1 Introduction 119
6.2 Iterative Methods for Tomography Problems 120
6.3 The Conjugate Gradient Method 126
6.4 The CGLS Method 131
Contents vii
6.5 Exercises 135
6.6 Notes and Further Reading 136
7 ADDITIONALREGULARIZATIONTECHNIQUES 139
7.1 Using Bounds as Constraints 139
7.2 Maximum Entropy Regularization 143
7.3 Total Variation 146
7.4 Exercises 151
7.5 Notes and Further Reading 152
8 FOURIER TECHNIQUES 153
8.1 Linear Systems in the Time and Frequency Domains 153
8.2 Deconvolution from a Fourier Perspective 158
8.3 Linear Systems in Discrete Time 161
8.4 Water Level Regularization 164
8.5 Exercises 168
8.6 Notes and Further Reading 170
9 NONLINEAR REGRESSION 171
9.1 Newton’s Method 171
9.2 The Gauss–Newton and Levenberg–Marquardt Methods 174
9.3 Statistical Aspects 177
9.4 Implementation Issues 181
9.5 Exercises 186
9.6 Notes and Further Reading 189
10 NONLINEAR INVERSE PROBLEMS 191
10.1 Regularizing Nonlinear Least Squares Problems 191
10.2 Occam’s Inversion 195
10.3 Exercises 199
10.4 Notes and Further Reading 199
viii Contents
11 BAYESIAN METHODS 201
11.1 Review of the Classical Approach 201
11.2 The Bayesian Approach 202
11.3 The Multivariate Normal Case 207
11.4 Maximum Entropy Methods 212
11.5 Epilogue 214
11.6 Exercises 216
11.7 Notes and Further Reading 217
A REVIEW OF LINEAR ALGEBRA 219
A.1 Systems of Linear Equations 219
A.2 Matrix and Vector Algebra 222
A.3 Linear Independence 228
A.4 Subspaces of Rn 229
A.5 Orthogonality and the Dot Product 233
A.6 Eigenvalues and Eigenvectors 237
A.7 Vector and Matrix Norms 240
A.8 The Condition Number of a Linear System 242
A.9 The QR Factorization 244
A.10 Linear Algebra in Spaces of Functions 245
A.11 Exercises 247
A.12 Notes and Further Reading 249
B REVIEW OF PROBABILITYAND STATISTICS 251
B.1 Probability and Random Variables 251
B.2 Expected Value and Variance 257
B.3 Joint Distributions 258
B.4 Conditional Probability 262
B.5 The Multivariate Normal Distribution 264
B.6 The Central Limit Theorem 265
B.7 Testing for Normality 265
Contents ix
B.8 Estimating Means and Confidence Intervals 267
B.9 Hypothesis Tests 269
B.10 Exercises 271
B.11 Notes and Further Reading 272
C REVIEW OF VECTOR CALCULUS 273
C.1 The Gradient, Hessian, and Jacobian 273
C.2 Taylor’s Theorem 275
C.3 Lagrange Multipliers 276
C.4 Exercises 278
C.5 Notes and Further Reading 280
D GLOSSARY OF NOTATION 281
Bibliography 283
Index 291

This textbook evolved from a course in geophysical inverse methods taught during the past
decade at New Mexico Tech, first by Rick Aster and, for the past 5 years, jointly between
Rick Aster and Brian Borchers. The audience for the course has included a broad range
of first- or second-year graduate students (and occasionally advanced undergraduates) from
geophysics, hydrology, mathematics, astronomy, and other disciplines. Cliff Thurber joined
this collaboration during the past 3 years and has taught a similar course at the University of
Wisconsin-Madison.
Our principal goal for this text is to promote fundamental understanding of parameter estimation
and inverse problem philosophy and methodology, specifically regarding such key
issues as uncertainty, ill–posedness, regularization, bias, and resolution. We emphasize theoretical
points with illustrative examples, and MATLAB codes that implement these examples
are provided on a companion CD. Throughout the examples and exercises, a CD icon indicates
that there is additional material on the CD. Exercises include a mix of programming and
theoretical problems. Please refer to the very last page of the book for more information on
the contents of the CD as well as a link to a companionWeb site.
This book has necessarily had to distill a tremendous body of mathematics and science
going back to (at least) Newton and Gauss. We hope that it will find a broad audience of
students and professionals interested in the general problem of estimating physical models
from data. Because this is an introductory text surveying a very broad field, we have not been
able to go into great depth. However, each chapter has a “Notes and Further Reading” section
to help guide the reader to further exploration of specific topics. Where appropriate, we have
also directly referenced research contributions to the field.
Some advanced topics have been deliberately omitted from the book because of space
limitations and/or because we expect that many readers would not be sufficiently familiar
with the required mathematics. For example, readers with a strong mathematical background
may be surprised that we consider inverse problems with discrete data and discretized models.
By doing this we avoid much of the technical complexity of functional analysis. Some advanced
applications and topics that we have omitted include inverse scattering problems, seismic
diffraction tomography, wavelets, data assimilation, and expectation maximization (EM)
methods.