Contents

Preface vii

Contentsxiii

List of Figures xix

List of Tables xxvii

1 Introduction 1

1.1 Outline 3

1.2 A note on programming 5

1.3 Symbols used throughout the book 6

2 Probability Theory and Classical Statistics 9

2.1 Rules of probability 9

2.2 Probability distributions in general 12

2.2.1 Important quantities in distributions 17

2.2.2 Multivariate distributions 19

2.2.3 Marginal and conditional distributions 23

2.3 Some important distributions in social science 25

2.3.1 The binomial distribution 25

2.3.2 The multinomial distribution 27

2.3.3 The Poisson distribution 28

2.3.4 The normal distribution 29

2.3.5 The multivariate normal distribution30

2.3.6 t and multivariate t distributions 33

2.4 Classical statistics in social science33

2.5 Maximum likelihood estimation 35

2.5.1 Constructing a likelihood function 36

2.5.2 Maximizing a likelihood function 38

2.5.4 A normal likelihood example 41

2.6 Conclusions 44

2.7 Exercises 44

2.7.1 Probability exercises 44

2.7.2 Classical inference exercises 45

3 Basics of Bayesian Statistics 47

3.1 Bayes’ Theorem for point probabilities 47

3.2 Bayes’ Theorem applied to probability distributions 50

3.2.1 Proportionality 51

3.3 Bayes’ Theorem with distributions: A voting example 53

3.3.1 Specification of a prior: The beta distribution 54

3.3.2 An alternative model for the polling data: A gamma

prior/ Poisson likelihood approach 60

3.4 A normal prior–normal likelihood example with σ2 known 62

3.4.1 Extending the normal distribution example 65

3.5 Some useful prior distributions 68

3.5.1 The Dirichlet distribution 69

3.5.2 The inverse gamma distribution 69

3.5.3 Wishart and inverse Wishart distributions 70

3.6 Criticism against Bayesian statistics 70

3.7 Conclusions 73

3.8 Exercises 74

4 Modern Model Estimation Part 1: Gibbs Sampling 77

4.1 What Bayesians want and why 77

4.2 The logic of sampling from posterior densities 78

4.3 Two basic sampling methods 80

4.3.1 The inversion method of sampling 81

4.3.2 The rejection method of sampling 84

4.4 Introduction to MCMC sampling 88

4.4.1 Generic Gibbs sampling 88

4.4.2 Gibbs sampling example using the inversion method 89

4.4.3 Example repeated using rejection sampling 93

4.4.4 Gibbs sampling from a real bivariate density 96

4.4.5 Reversing the process: Sampling the parameters given the data100

4.5 Conclusions 103

4.6 Exercises 105

5 Modern Model Estimation Part 2: Metroplis–Hastings

Sampling 107

5.1 A generic MH algorithm108

5.2 Example: MH sampling when conditional densities are difficult to derive 115

5.3 Example: MH sampling for a conditional density with an unknown form 118

5.4 Extending the bivariate normal example: The full multiparameter model 121

5.4.1 The conditionals for μx and μy 122

5.4.2 The conditionals for σ2x, σ2y, and ρ 123

5.4.3 The complete MH algorithm 124

5.4.4 A matrix approach to the bivariate normal distribution problem126

5.5 Conclusions 128

5.6 Exercises 129

6 Evaluating Markov Chain Monte Carlo Algorithms and

Model Fit 131

6.1 Why evaluate MCMC algorithm performance? 132

6.2 Some common problems and solutions132

6.3 Recognizing poor performance 135

6.3.1 Trace plots 135

6.3.2 Acceptance rates of MH algorithms 141

6.3.3 Autocorrelation of parameters 146

6.3.4 “ˆR” and other calculations 147

6.4 Evaluating model fit 153

6.4.1 Residual analysis 154

6.4.2 Posterior predictive distributions 155

6.5 Formal comparison and combining models 159

6.5.1 Bayes factors 159

6.5.2 Bayesian model averaging 161

6.6 Conclusions 163

6.7 Exercises 163

7 The Linear Regression Model 165

7.1 Development of the linear regression model 165

7.2 Sampling from the posterior distribution for the model

parameters 168

7.2.1 Sampling with an MH algorithm 168

7.2.2 Sampling the model parameters using Gibbs sampling 169

7.3 Example: Are people in the South “nicer” than others? 174

7.3.1 Results and comparison of the algorithms 175

7.3.2 Model evaluation 178

7.4 Incorporating missing data 182

7.4.1 Types of missingness 182

7.4.2 A generic Bayesian approach when data are MAR: The “niceness” example revisited 186

7.5 Conclusions 191

7.6 Exercises 192

8 Generalized Linear Models 193

8.1 The dichotomous probit model 195

8.1.1 Model development and parameter interpretation 195

8.1.2 Sampling from the posterior distribution for the model parameters 198

8.1.3 Simulating from truncated normal distributions 200

8.1.4 Dichotomous probit model example: Black–white

differences in mortality 206

8.2 The ordinal probit model 217

8.2.1 Model development and parameter interpretation 218

8.2.2 Sampling from the posterior distribution for the parameters 220

8.2.3 Ordinal probit model example: Black–white differences in health 223

8.3 Conclusions 228

8.4 Exercises 229

9 Introduction to Hierarchical Models 231

9.1 Hierarchical models in general 232

9.1.1 The voting example redux 233

9.2 Hierarchical linear regression models 240

9.2.1 Random effects: The random intercept model 241

9.2.2 Random effects: The random coefficient model 251

9.2.3 Growth models256

9.3 A note on fixed versus random effects models and other

terminology264

9.4 Conclusions 268

9.5 Exercises 269

10 Introduction to Multivariate Regression Models 271

10.1 Multivariate linear regression 271

10.1.1 Model development 271

10.1.2 Implementing the algorithm 275

10.2 Multivariate probit models 277

10.2.1 Model development 278

10.2.2 Step 2: Simulating draws from truncated multivariate normal distributions 283

10.2.3 Step 3: Simulation of thresholds in the multivariate probit model 289

10.2.4 Step 5: Simulating the error covariance matrix 295

10.2.5 Implementing the algorithm 297

10.3.1 Model specification and simulation 307

10.3.2 Life table generation and other posterior inferences 310

10.4 Conclusions 315

10.5 Exercises 317

11 Conclusion 319

A Background Mathematics 323

A.1 Summary of calculus 323

A.1.1 Limits 323

A.1.2 Differential calculus324

A.1.3 Integral calculus326

A.1.4 Finding a general rule for a derivative 329

A.2 Summary of matrix algebra 330

A.2.1 Matrix notation 330

A.2.2 Matrix operations 331

A.3 Exercises 335

A.3.1 Calculus exercises 335

A.3.2 Matrix algebra exercises 335

B The Central Limit Theorem, Confidence Intervals, and

Hypothesis Tests 337

B.1 A simulation study 337

B.2 Classical inference 338

B.2.1 Hypothesis testing 339

B.2.2 Confidence intervals 342

B.2.3 Some final notes344

References 345