NITIS MUKHOPADHYAY
University of Connecticut
Storrs, Connecticut


Contents
Preface v
Acknowledgments xi
1 Notions of Probability 1
1.1 Introduction 1
1.2 About Sets 3
1.3 Axiomatic Development of Probability 6
1.4 The Conditional Probability and Independent Events 9
1.4.1 Calculus of Probability 12
1.4.2 Bayes.s Theorem 14
1.4.3 Selected Counting Rules 16
1.5 Discrete Random Variables 18
1.5.1 Probability Mass and Distribution Functions 19
1.6 Continuous Random Variables 23
1.6.1 Probability Density and Distribution Functions 23
1.6.2 The Median of a Distribution 28
1.6.3 Selected Reviews from Mathematics 28
1.7 Some Standard Probability Distributions 32
1.7.1 Discrete Distributions 33
1.7.2 Continuous Distributions 37
1.8 Exercises and Complements 50
2 Expectations of Functions of Random Variables 65
2.1 Introduction 65
2.2 Expectation and Variance 65
2.2.1 The Bernoulli Distribution 71
2.2.2 The Binomial Distribution 72
2.2.3 The Poisson Distribution 73
2.2.4 The Uniform Distribution 73
2.2.5 The Normal Distribution 73
2.2.6 The Laplace Distribution 76
2.2.7 The Gamma Distribution 76
2.3 The Moments and Moment Generating Function 77
2.3.1 The Binomial Distribution 80
2.3.2 The Poisson Distribution 81
2.3.3 The Normal Distribution 82
xiv Contents
2.3.4 The Gamma Distribution 84
2.4 Determination of a Distribution via MGF 86
2.5 The Probability Generating Function 88
2.6 Exercises and Complements 89
3 Multivariate Random Variables 99
3.1 Introduction 99
3.2 Discrete Distributions 100
3.2.1 The Joint, Marginal and Conditional Distributions 101
3.2.2 The Multinomial Distribution 103
3.3 Continuous Distributions 107
3.3.1 The Joint, Marginal and Conditional Distributions 107
3.3.2 Three and Higher Dimensions 115
3.4 Covariances and Correlation Coefficients 119
3.4.1 The Multinomial Case 124
3.5 Independence of Random Variables 125
3.6 The Bivariate Normal Distribution 131
3.7 Correlation Coefficient and Independence 139
3.8 The Exponential Family of Distributions 141
3.8.1 One-parameter Situation 141
3.8.2 Multi-parameter Situation 144
3.9 Some Standard Probability Inequalities 145
3.9.1 Markov and Bernstein-Chernoff Inequalities 145
3.9.2 Tchebysheff.s Inequality 148
3.9.3 Cauchy-Schwarz and Covariance Inequalities 149
3.9.4 Jensen.s and Lyapunov.s Inequalities 152
3.9.5 H.lder.s Inequality 156
3.9.6 Bonferroni Inequality 157
3.9.7 Central Absolute Moment Inequality 158
3.10 Exercises and Complements 159
4 Functions of Random Variables and Sampling
Distribution 177
4.1 Introduction 177
4.2 Using Distribution Functions 179
4.2.1 Discrete Cases 179
4.2.2 Continuous Cases 181
4.2.3 The Order Statistics 182
4.2.4 The Convolution 185
4.2.5 The Sampling Distribution 187
4.3 Using the Moment Generating Function 190
4.4 A General Approach with Transformations 192
4.4.1 Several Variable Situations 195
4.5 Special Sampling Distributions 206
Contents xv
4.5.1 The Student.s t Distribution 207
4.5.2 The F Distribution 209
4.5.3 The Beta Distribution 211
4.6 Special Continuous Multivariate Distributions 212
4.6.1 The Normal Distribution 212
4.6.2 The t Distribution 218
4.6.3 The F Distribution 219
4.7 Importance of Independence in Sampling Distributions 220
4.7.1 Reproductivity of Normal Distributions 220
4.7.2 Reproductivity of Chi-square Distributions 221
4.7.3 The Student.s t Distribution 223
4.7.4 The F Distribution 223
4.8 Selected Review in Matrices and Vectors 224
4.9 Exercises and Complements 227
5 Concepts of Stochastic Convergence 241
5.1 Introduction 241
5.2 Convergence in Probability 242
5.3 Convergence in Distribution 253
5.3.1 Combination of the Modes of Convergence 256
5.3.2 The Central Limit Theorems 257
5.4 Convergence of Chi-square, t, and F Distributions 264
5.4.1 The Chi-square Distribution 264
5.4.2 The Student.s t Distribution 264
5.4.3 The F Distribution 265
5.4.4 Convergence of the PDF and Percentage Points 265
5.5 Exercises and Complements 270
6 Sufficiency, Completeness, and Ancillarity 281
6.1 Introduction 281
6.2 Sufficiency 282
6.2.1 The Conditional Distribution Approach 284
6.2.2 The Neyman Factorization Theorem 288
6.3 Minimal Sufficiency 294
6.3.1 The Lehmann-Scheffé Approach 295
6.4 Information 300
6.4.1 One-parameter Situation 301
6.4.2 Multi-parameter Situation 304
6.5 Ancillarity 309
6.5.1 The Location, Scale, and Location-Scale Families 314
6.5.2 Its Role in the Recovery of Information 316
6.6 Completeness 318
6.6.1 Complete Sufficient Statistics 320
6.6.2 Basu.s Theorem 324
6.7 Exercises and Complements 327
xvi Contents
7 Point Estimation 341
7.1 Introduction 341
7.2 Finding Estimators 342
7.2.1 The Method of Moments 342
7.2.2 The Method of Maximum Likelihood 344
7.3 Criteria to Compare Estimators 351
7.3.1 Unbiasedness, Variance and Mean Squared Error 351
7.3.2 Best Unbiased and Linear Unbiased Estimators 354
7.4 Improved Unbiased Estimator via Sufficiency 358
7.4.1 The Rao-Blackwell Theorem 358
7.5 Uniformly Minimum Variance Unbiased Estimator 365
7.5.1 The Cramér-Rao Inequality and UMVUE 366
7.5.2 The Lehmann-Scheffé Theorems and UMVUE 371
7.5.3 A Generalization of the Cramér-Rao Inequality 374
7.5.4 Evaluation of Conditional Expectations 375
7.6 Unbiased Estimation Under Incompleteness 377
7.6.1 Does the Rao-Blackwell Theorem Lead
to UMVUE? 377
7.7 Consistent Estimators 380
7.8 Exercises and Complements 382
8 Tests of Hypotheses 395
8.1 Introduction 395
8.2 Error Probabilities and the Power Function 396
8.2.1 The Concept of a Best Test 399
8.3 Simple Null Versus Simple Alternative Hypotheses 401
8.3.1 Most Powerful Test via the Neyman-Pearson
Lemma 401
8.3.2 Applications: No Parameters Are Involved 413
8.3.3 Applications: Observations Are Non-IID 416
8.4 One-Sided Composite Alternative Hypothesis 417
8.4.1 UMP Test via the Neyman-Pearson Lemma 417
8.4.2 Monotone Likelihood Ratio Property 420
8.4.3 UMP Test via MLR Property 422
8.5 Simple Null Versus Two-Sided Alternative Hypotheses 425
8.5.1 An Example Where UMP Test Does Not Exist 425
8.5.2 An Example Where UMP Test Exists 426
8.5.3 Unbiased and UMP Unbiased Tests 428
8.6 Exercises and Complements 429
9 Confidence Interval Estimation 441
9.1 Introduction 441
9.2 One-Sample Problems 443
9.2.1 Inversion of a Test Procedure 444
Contents xvii
9.2.2 The Pivotal Approach 446
9.2.3 The Interpretation of a Confidence Coefficient 451
9.2.4 Ideas of Accuracy Measures 452
9.2.5 Using Confidence Intervals in the Tests
of Hypothesis 455
9.3 Two-Sample Problems 456
9.3.1 Comparing the Location Parameters 456
9.3.2 Comparing the Scale Parameters 460
9.4 Multiple Comparisons 463
9.4.1 Estimating a Multivariate Normal Mean Vector 463
9.4.2 Comparing the Means 465
9.4.3 Comparing the Variances 467
9.5 Exercises and Complements 469
10 Bayesian Methods 477
10.1 Introduction 477
10.2 Prior and Posterior Distributions 479
10.3 The Conjugate Priors 481
10.4 Point Estimation 485
10.5 Credible Intervals 488
10.5.1 Highest Posterior Density 489
10.5.2 Contrasting with the Confidence Intervals 492
10.6 Tests of Hypotheses 493
10.7 Examples with Non-Conjugate Priors 494
10.8 Exercises and Complements 497
11 Likelihood Ratio and Other Tests 507
11.1 Introduction 507
11.2 One-Sample Problems 508
11.2.1 LR Test for the Mean 509
11.2.2 LR Test for the Variance 512
11.3 Two-Sample Problems 515
11.3.1 Comparing the Means 515
11.3.2 Comparing the Variances 519
11.4 Bivariate Normal Observations 522
11.4.1 Comparing the Means: The Paired Difference
t Method 522
11.4.2 LR Test for the Correlation Coefficient 525
11.4.3 Tests for the Variances 528
11.5 Exercises and Complements 529
12 Large-Sample Inference 539
12.1 Introduction 539
12.2 The Maximum Likelihood Estimation 539
12.3 Confidence Intervals and Tests of Hypothesis 542
xviii Contents
12.3.1 The Distribution-Free Population Mean 543
12.3.2 The Binomial Proportion 548
12.3.3 The Poisson Mean 553
12.4 The Variance Stabilizing Transformations 555
12.4.1 The Binomial Proportion 556
12.4.2 The Poisson Mean 559
12.4.3 The Correlation Coefficient 560
12.5 Exercises and Complements 563
13 Sample Size Determination: Two-Stage
Procedures 569
13.1 Introduction 569
13.2 The Fixed-Width Confidence Interval 573
13.2.1 Stein.s Sampling Methodology 573
13.2.2 Some Interesting Properties 574
13.3 The Bounded Risk Point Estimation 579
13.3.1 The Sampling Methodology 581
13.3.2 Some Interesting Properties 582
13.4 Exercises and Complements 584
14 Appendix 591
14.1 Abbreviations and Notation 591
14.2 A Celebration of Statistics: Selected Biographical Notes 593
14.3 Selected Statistical Tables 621
14.3.1 The Standard Normal Distribution Function 621
14.3.2 Percentage Points of the Chi-Square Distribution 626
14.3.3 Percentage Points of the Student.s t Distribution 628
14.3.4 Percentage Points of the F Distribution 630
References 633
Index 649