本书为荆炳义教授在兰州大学数学与统计学院暑期班的讲义，共分两部分，其目录如下

part one
Contents
1 Probability Models 2
1.1 Exponential family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.1 De¯nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Canonical form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3 Natural parameter space is a convex set . . . . . . . . . . . . . . . . . . . . 4
1.1.4 Exponential family of full rank . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.5 Exponential family is closed under independent sum, marginal and condi-
tional operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.6 T(X) is also in another exponential family . . . . . . . . . . . . . . . . . . . 5
1.1.7 A useful property . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.8 Examples of exponential family . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Location-scale family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Principles of Data Reduction 7
2.1 Su±ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Minimal Su±ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.1 Some simple rules to ¯nd minimal su±cient statistics . . . . . . . . . . . . 11
2.2.2 Minimal Su±ciency for Exponential Family of Full Rank . . . . . . . . . . . 13
2.2.3 Minimal Su±ciency for Location Family . . . . . . . . . . . . . . . . . . . . 15
2.3 Complete Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.1 Full rank exponential family is complete . . . . . . . . . . . . . . . . . . . . 17
2.3.2 Examples of Complete Families . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.3 Examples of Non-Complete Families . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Ancillary Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5 Relationship between Minimal Su±cient, Complete and Ancillary Statistics . . . . 20
2.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3 Unbiased Estimation 24
3.1 Criteria of estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 Do unbiased estimators always exist? . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3 Characterization of a UMVUE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.4 Existence of UMVUE among unbiased estimators . . . . . . . . . . . . . . . . . . . 27
3.5 Searching UMVUE's by su±ciency and completeness . . . . . . . . . . . . . . . . . 32
3.6 Some examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.6.1 Method 1: Solving equations. . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.6.2 Method 2: Conditioning by using Rao-Blackwell theorem. . . . . . . . . . . 36
3.6.3 Method 3: Using Characterization Theorem (when C-S stat is not available) 38
3.7 Failure of UMVUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.8 Procedures to ¯nd a UMVUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.9 Information inequality (or Cramer-Rao lower bound) . . . . . . . . . . . . . . . . . 41
3.9.1 Information inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.9.2 Some properties of the Fisher information matrix . . . . . . . . . . . . . . . 43
3.9.3 Some examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.9.4 A necessary and su±cient condition for reaching Cramer-Rao lower bound . 46
3.9.5 Information inequality in exponential family . . . . . . . . . . . . . . . . . . 48
3.9.6 Information inequality in location-scale family . . . . . . . . . . . . . . . . 50
3.9.7 Regularity conditions must be checked . . . . . . . . . . . . . . . . . . . . . 51
3.9.8 Relative E±ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.10 Some comparisons about the UMVUE and MLE's . . . . . . . . . . . . . . . . . . 54
3.10.1 Di±culties with UMVUE's . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4 The Method of Maximum Likelihood 55
4.1 Maximum Likelihood Estimate (MLE) . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2 Invariance property of MLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2.1 g(¢) is a 1-to-1 mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2.2 g(¢) is not a 1-to-1 mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3 Some examples of MLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.4 MLE's in exponential families . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.5 Numerical solution to likelihood equations . . . . . . . . . . . . . . . . . . . . . . . 63
4.6 \Problems" with MLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.6.1 MLE may not be unique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.6.2 MLE may not be asymptotically normal . . . . . . . . . . . . . . . . . . . . 65
4.6.3 MLE may be inadmissible . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.6.4 MLE can be inconsistent . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5 Asymptotically E±cient Estimation 69
5.1 E±ciency vs. Super-E±ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.1.1 Review: Cramer-Rao lower bound for ¯nite sample . . . . . . . . . . . . . . 69
5.1.2 Asymptotic Cramer-Rao lower bound . . . . . . . . . . . . . . . . . . . . . 69
5.2 De¯nition of Asymptotic E±ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3 MLE is Asymptotic E±cient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3.1 Approach I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3.2 Approach II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4 Some further examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.4.1 Some comparisons about the UMVUE and MLE's . . . . . . . . . . . . . . 77

part two
Contents
1 Introduction and Basic Concepts 4
1.1 Hypotheses and Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Type I and II Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Non-randomized tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Some examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4.1 Normal example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4.2 Binomial example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 The uniformly most powerful (UMP) tests 11
2.1 De¯nitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 UMP tests for simple hypothesis: Neyman-Pearson Lemma . . . . . . . . . 11
2.3 The relationship between the power and the size . . . . . . . . . . . . . . . 15
2.4 Some examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.1 Binomial example. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.2 Normal example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.3 Testing Normal Against Double Exponential. . . . . . . . . . . . . . 18
2.5 The Neyman-Pearson Lemma in terms of su±cient statistics . . . . . . . . 19
3 UMP tests For One-Sided Hypotheses 20
3.1 One-sided hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2 Monotone Likelihood Ratio (MLR) . . . . . . . . . . . . . . . . . . . . . . 20
3.3 Some examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3.1 Examples when MLR exists . . . . . . . . . . . . . . . . . . . . . . 26
3.3.2 A UMP test may exist even if an MLR does not . . . . . . . . . . . 28
3.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4 UMP tests for two-sided hypotheses 31
4.1 Two-sided hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2 An example where a UMP test does not exist . . . . . . . . . . . . . . . . 31
4.3 Generalized N-P Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3.1 Revision on N-P Lemma . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3.2 Generalized N-P Lemma (GNP Lemma) . . . . . . . . . . . . . . . 33
4.4 UMP tests for one-parameter exponential families. . . . . . . . . . . . . . . 35
4.4.1 Uniqueness of UMP tests . . . . . . . . . . . . . . . . . . . . . . . . 39
4.5 UMP tests for totally positive families. . . . . . . . . . . . . . . . . . . . . 43
4.6 Some examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.6.1 Normal example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.6.2 A non-regular example¤. (Optional) . . . . . . . . . . . . . . . . . 46
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.8 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5 Unbiased Tests. 48
5.1 De¯nitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.2 UMPU for One-parameter exponential family . . . . . . . . . . . . . . . . 50
5.2.1 Case I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.2.2 Case II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.2.3 A lemma used in the proof of the last theorem. . . . . . . . . . . . 54
5.2.4 Some examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.3 UMPU tests for multiparameter exponential families . . . . . . . . . . . . 58
5.3.1 Complete and Boundedly Complete Statistics . . . . . . . . . . . . 58
5.3.2 Similarity and Neyman Structure . . . . . . . . . . . . . . . . . . . 61
5.3.3 UMPU tests for multiparameter exponential families . . . . . . . . 62
5.3.4 UMPU tests for linear combinations of parameters in multiparam-
eter exponential families . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3.5 Power calculation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3.6 Some examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6 Unbiased tests for special families (e.g., Normal and Gamma families) 72
6.1 Ancillary Statistics and Basu's Theorem . . . . . . . . . . . . . . . . . . . 72
6.2 UMPU tests for multi-parameter exponential families . . . . . . . . . . . . 76
6.2.1 Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.2.2 UMPU tests for special families . . . . . . . . . . . . . . . . . . . . 77
6.2.3 Some basic facts about bivariate normal distribution . . . . . . . . 80
6.2.4 Application 1: one-sample problem. . . . . . . . . . . . . . . . . . . 82
6.2.5 Application 2: two-sample problem. . . . . . . . . . . . . . . . . . . 85
6.2.6 Application 3: Testing for independence in the bivariate normal
family. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.2.7 Application 4: Regression. . . . . . . . . . . . . . . . . . . . . . . . 93
6.2.8 Application 5: Non-normal example. . . . . . . . . . . . . . . . . . 93
6.3 The LSE in linear models . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.4.1 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7 Hypothesis Testing by Likelihood Methods 96
7.1 Likelihood Ratio Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.1.1 De¯nition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.2 LRT with no nuisance parameters. . . . . . . . . . . . . . . . . . . . . . . 98
7.2.1 Some examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.2.2 LRT in one-parameter exponential family. . . . . . . . . . . . . . . 100
7.2.3 LRT with non-exponential family (with no nuisance parameters) . . 102
7.3 Equivalence of LRT and Neyman-Pearson test when both exist . . . . . . . 103
7.4 LRT with nuisance parameters . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.4.1 Examples with multiparameter exponential families . . . . . . . . . 104
7.5 Bad performance of LRT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.6 Asymptotic Â2 approximation of the LRT. . . . . . . . . . . . . . . . . . . 113
7.6.1 Why asymptotics? . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.6.2 Review: Asymptotic properties of MLE . . . . . . . . . . . . . . . . 116
7.6.3 Formulation of the problem. . . . . . . . . . . . . . . . . . . . . . . 117
7.6.4 Asymptotic Â2 approximation to LRT . . . . . . . . . . . . . . . . . 118
7.7 Wald's and Rao's tests and their relation with LRT . . . . . . . . . . . . . 123
7.8 LRT,Wald's and Rao's tests in independent but non-identically distributed
r.v. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.9 Â2-tests for multinominal distribution . . . . . . . . . . . . . . . . . . . . . 125
7.9.1 Prelimanaries with the multinominal distribution . . . . . . . . . . 125
7.9.2 Tests for multinomial distribution . . . . . . . . . . . . . . . . . . . 126
7.9.3 Application: Goodness of ¯t tests . . . . . . . . . . . . . . . . . . . 131
7.10 Test of independence in contingency tables . . . . . . . . . . . . . . . . . . 134
7.11 Some general comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137