Preface xv
Preface to the First Edition xvii
1. An Introduction to Set Theory 1
1.1. The Concept of a Set, 1
1.2. Set Operations, 2
1.3. Relations and Functions, 4
1.4. Finite, Countable, and Uncountable Sets, 6
1.5. Bounded Sets, 9
1.6. Some Basic Topological Concepts, 10
1.7. Examples in Probability and Statistics, 13
Further Reading and Annotated Bibliography, 15
Exercises, 17
2. Basic Concepts in Linear Algebra 21
2.1. Vector Spaces and Subspaces, 21
2.2. Linear Transformations, 25
2.3. Matrices and Determinants, 27
2.3.1. Basic Operations on Matrices, 28
2.3.2. The Rank of a Matrix, 33
2.3.3. The Inverse of a Matrix, 34
2.3.4. Generalized Inverse of a Matrix, 36
2.3.5. Eigenvalues and Eigenvectors of a Matrix, 36
2.3.6. Some Special Matrices, 38
2.3.7. The Diagonalization of a Matrix, 38
2.3.8. Quadratic Forms, 39

2.3.9. The Simultaneous Diagonalization
of Matrices, 40
2.3.10. Bounds on Eigenvalues, 41
2.4. Applications of Matrices in Statistics, 43
2.4.1. The Analysis of the Balanced Mixed Model, 43
2.4.2. The Singular-Value Decomposition, 45
2.4.3. Extrema of Quadratic Forms, 48
2.4.4. The Parameterization of Orthogonal
Matrices, 49
Further Reading and Annotated Bibliography, 50
Exercises, 53
3. Limits and Continuity of Functions 57
3.1. Limits of a Function, 57
3.2. Some Properties Associated with Limits of Functions, 63
3.3. The o, O Notation, 65
3.4. Continuous Functions, 66
3.4.1. Some Properties of Continuous Functions, 71
3.4.2. Lipschitz Continuous Functions, 75
3.5. Inverse Functions, 76
3.6. Convex Functions, 79
3.7. Continuous and Convex Functions in Statistics, 82
Further Reading and Annotated Bibliography, 87
Exercises, 88
4. Differentiation 93
4.1. The Derivative of a Function, 93
4.2. The Mean Value Theorem, 99
4.3. Taylor’s Theorem, 108
4.4. Maxima and Minima of a Function, 112
4.4.1. A Sufficient Condition for a Local Optimum, 114
4.5. Applications in Statistics, 115
4.5.1 Functions of Random Variables, 116
4.5.2. Approximating Response Functions, 121
4.5.3. The Poisson Process, 122
4.5.4. Minimizing the Sum of Absolute Deviations, 124
Further Reading and Annotated Bibliography, 125
Exercises, 127

5. Infinite Sequences and Series 132
5.1. Infinite Sequences, 132
5.1.1. The Cauchy Criterion, 137
5.2. Infinite Series, 140
5.2.1. Tests of Convergence for Series
of Positive Terms, 144
5.2.2. Series of Positive and Negative Terms, 158
5.2.3. Rearrangement of Series, 159
5.2.4. Multiplication of Series, 162
5.3. Sequences and Series of Functions, 165
5.3.1. Properties of Uniformly Convergent Sequences
and Series, 169
5.4. Power Series, 174
5.5. Sequences and Series of Matrices, 178
5.6. Applications in Statistics, 182
5.6.1. Moments of a Discrete Distribution, 182
5.6.2. Moment and Probability Generating
Functions, 186
5.6.3. Some Limit Theorems, 191
5.6.3.1. The Weak Law of Large Numbers
ŽKhinchine’s Theorem., 192
5.6.3.2. The Strong Law of Large Numbers
ŽKolmogorov’s Theorem., 192
5.6.3.3. The Continuity Theorem for Probability
Generating Functions, 192
5.6.4. Power Series and Logarithmic Series
Distributions, 193
5.6.5. Poisson Approximation to Power Series
Distributions, 194
5.6.6. A Ridge Regression Application, 195
Further Reading and Annotated Bibliography, 197
Exercises, 199
6. Integration 205
6.1. Some Basic Definitions, 205
6.2. The Existence of the Riemann Integral, 206
6.3. Some Classes of Functions That Are Riemann
Integrable, 210
6.3.1. Functions of Bounded Variation, 212

6.4. Properties of the Riemann Integral, 215
6.4.1. Change of Variables in Riemann Integration, 219
6.5. Improper Riemann Integrals, 220
6.5.1. Improper Riemann Integrals of the Second
Kind, 225
6.6. Convergence of a Sequence of Riemann Integrals, 227
6.7. Some Fundamental Inequalities, 229
6.7.1. The Cauchy
Schwarz Inequality, 229
6.7.2. H¨older’s Inequality, 230
6.7.3. Minkowski’s Inequality, 232
6.7.4. Jensen’s Inequality, 233
6.8. Riemann
Stieltjes Integral, 234
6.9. Applications in Statistics, 239
6.9.1. The Existence of the First Negative Moment of a
Continuous Distribution, 242
6.9.2. Transformation of Continuous Random
Variables, 246
6.9.3. The Riemann
Stieltjes Representation of the
Expected Value, 249
6.9.4. Chebyshev’s Inequality, 251
Further Reading and Annotated Bibliography, 252
Exercises, 253
7. Multidimensional Calculus 261
7.1. Some Basic Definitions, 261
7.2. Limits of a Multivariable Function, 262
7.3. Continuity of a Multivariable Function, 264
7.4. Derivatives of a Multivariable Function, 267
7.4.1. The Total Derivative, 270
7.4.2. Directional Derivatives, 273
7.4.3. Differentiation of Composite Functions, 276
7.5. Taylor’s Theorem for a Multivariable Function, 277
7.6. Inverse and Implicit Function Theorems, 280
7.7. Optima of a Multivariable Function, 283
7.8. The Method of Lagrange Multipliers, 288
7.9. The Riemann Integral of a Multivariable Function, 293
7.9.1. The Riemann Integral on Cells, 294
7.9.2. Iterated Riemann Integrals on Cells, 295
7.9.3. Integration over General Sets, 297
7.9.4. Change of Variables in n-Tuple Riemann
Integrals, 299

7.10. Differentiation under the Integral Sign, 301
7.11. Applications in Statistics, 304
7.11.1. Transformations of Random Vectors, 305
7.11.2. Maximum Likelihood Estimation, 308
7.11.3. Comparison of Two Unbiased
Estimators, 310
7.11.4. Best Linear Unbiased Estimation, 311
7.11.5. Optimal Choice of Sample Sizes in Stratified
Sampling, 313
Further Reading and Annotated Bibliography, 315
Exercises, 316
8. Optimization in Statistics 327
8.1. The Gradient Methods, 329
8.1.1. The Method of Steepest Descent, 329
8.1.2. The Newton
Raphson Method, 331
8.1.3. The Davidon
Fletcher
Powell Method, 331
8.2. The Direct Search Methods, 332
8.2.1. The Nelder
Mead Simplex Method, 332
8.2.2. Price’s Controlled Random Search
Procedure, 336
8.2.3. The Generalized Simulated Annealing
Method, 338
8.3. Optimization Techniques in Response Surface
Methodology, 339
8.3.1. The Method of Steepest Ascent, 340
8.3.2. The Method of Ridge Analysis, 343
8.3.3. Modified Ridge Analysis, 350
8.4. Response Surface Designs, 355
8.4.1. First-Order Designs, 356
8.4.2. Second-Order Designs, 358
8.4.3. Variance and Bias Design Criteria, 359
8.5. Alphabetic Optimality of Designs, 362
8.6. Designs for Nonlinear Models, 367
8.7. Multiresponse Optimization, 370
8.8. Maximum Likelihood Estimation and the
EM Algorithm, 372
8.8.1. The EM Algorithm, 375
8.9. Minimum Norm Quadratic Unbiased Estimation of
Variance Components, 378

8.10. Scheff´e’s Confidence Intervals, 382
8.10.1. The Relation of Scheff´e’s Confidence Intervals
to the F-Test, 385
Further Reading and Annotated Bibliography, 391
Exercises, 395
9. Approximation of Functions 403
9.1. Weierstrass Approximation, 403
9.2. Approximation by Polynomial Interpolation, 410
9.2.1. The Accuracy of Lagrange Interpolation, 413
9.2.2. A Combination of Interpolation and
Approximation, 417
9.3 Approximation by Spline Functions, 418
9.3.1. Properties of Spline Functions, 418
9.3.2. Error Bounds for Spline Approximation, 421
9.4. Applications in Statistics, 422
9.4.1. Approximate Linearization of Nonlinear Models
by Lagrange Interpolation, 422
9.4.2. Splines in Statistics, 428
9.4.2.1. The Use of Cubic Splines in
Regression, 428
9.4.2.2. Designs for Fitting Spline Models, 430
9.4.2.3. Other Applications of Splines in
Statistics, 431
Further Reading and Annotated Bibliography, 432
Exercises, 434
10. Orthogonal Polynomials 437
10.1. Introduction, 437
10.2. Legendre Polynomials, 440
10.2.1. Expansion of a Function Using Legendre
Polynomials, 442
10.3. Jacobi Polynomials, 443
10.4. Chebyshev Polynomials, 444
10.4.1. Chebyshev Polynomials of the First Kind, 444
10.4.2. Chebyshev Polynomials of the Second Kind, 445
10.5. Hermite Polynomials, 447
10.6. Laguerre Polynomials, 451
10.7. Least-Squares Approximation with Orthogonal
Polynomials, 453

10.8. Orthogonal Polynomials Defined on a Finite Set, 455
10.9. Applications in Statistics, 456
10.9.1. Applications of Hermite Polynomials, 456
10.9.1.1. Approximation of Density Functions
and Quantiles of Distributions, 456
10.9.1.2. Approximation of a Normal
Integral, 460
10.9.1.3. Estimation of Unknown
Densities, 461
10.9.2. Applications of Jacobi and Laguerre
Polynomials, 462
10.9.3. Calculation of Hypergeometric Probabilities
Using Discrete Chebyshev Polynomials, 462
Further Reading and Annotated Bibliography, 464
Exercises, 466
11. Fourier Series 471
11.1. Introduction, 471
11.2. Convergence of Fourier Series, 475
11.3. Differentiation and Integration of Fourier Series, 483
11.4. The Fourier Integral, 488
11.5. Approximation of Functions by Trigonometric
Polynomials, 495
11.5.1. Parseval’s Theorem, 496
11.6. The Fourier Transform, 497
11.6.1. Fourier Transform of a Convolution, 499
11.7. Applications in Statistics, 500
11.7.1 Applications in Time Series, 500
11.7.2. Representation of Probability Distributions, 501
11.7.3. Regression Modeling, 504
11.7.4. The Characteristic Function, 505
11.7.4.1. Some Properties of Characteristic
Functions, 510
Further Reading and Annotated Bibliography, 510
Exercises, 512
12. Approximation of Integrals 517
12.1. The Trapezoidal Method, 517
12.1.1. Accuracy of the Approximation, 518
12.2. Simpson’s Method, 521
12.3. Newton
Cotes Methods, 523

12.4. Gaussian Quadrature, 524
12.5. Approximation over an Infinite Interval, 528
12.6. The Method of Laplace, 531
12.7. Multiple Integrals, 533
12.8. The Monte Carlo Method, 535
12.8.1. Variation Reduction, 537
12.8.2. Integrals in Higher Dimensions, 540
12.9. Applications in Statistics, 541
12.9.1. The Gauss
Hermite Quadrature, 542
12.9.2. Minimum Mean Squared Error
Quadrature, 543
12.9.3. Moments of a Ratio of Quadratic Forms, 546
12.9.4. Laplace’s Approximation in Bayesian
Statistics, 548
12.9.5. Other Methods of Approximating Integrals
in Statistics, 549
Further Reading and Annotated Bibliography, 550
Exercises, 552
Appendix. Solutions to Selected Exercises 557
Chapter 1, 557
Chapter 2, 560
Chapter 3, 565
Chapter 4, 570
Chapter 5, 577
Chapter 6, 590
Chapter 7, 600
Chapter 8, 613
Chapter 9, 622
Chapter 10, 627
Chapter 11, 635
Chapter 12, 644
General Bibliography 652
Index 665