目录如下：

Contents
Preface ...................................................... v
1 Topic-Focused Introduction to R and Data sets Used ..... 1
1.1 Resources and Software for the Imputation of Missing Values ................................................ 1
1.1.1 Amelia ....................................... 2
1.1.2 mi ........................................... 3
1.1.3 mice (and BaBooN) ............................ 4
1.1.4 missMDA ..................................... 4
1.1.5 missForest and missRanger...................... 4
1.1.6 robCompositions............................... 4
1.1.7 VIM ......................................... 5
1.2 The Statistics Environment R........................... 5
1.3 Simple Calculations in R ............................... 6
1.4 Installation of R and Updates ........................... 7
1.5 Help ................................................. 8
1.6 The R Workspace and the Working Directory ............. 10
1.7 Data Types........................................... 10
1.8 Generic Functions, Methods, and Classes ................. 17
1.9 A Note on Functions with the Same Name in Different Packages ............................................. 20
1.10 Basic Data Manipulation with the dplyr Package ........... 21
1.10.1 Pipes......................................... 22
1.10.2 dplyr—tibbles ................................. 23
1.10.3 dplyr—Selection of Rows ....................... 24
1.10.4 dplyr—Order.................................. 25
1.10.5 dplyr—Selection of Columns .................... 26
1.10.6 dplyr—Uniqueness ............................. 27
1.10.7 dplyr—Creating Variables....................... 28 xv
xvi Contents
1.10.8 dplyr—Grouping and Summary Statistics
.........
28
1.10.9 dplyr—Window Functions
......................
29
1.11 Data Manipulation with the data.table Package
...........
30
1.11.1 data.table—Variable Construction
................
31
1.11.2 data.table—Indexing/Subsetting
..................
31
1.11.3 data.table—Keys
..............................
33
1.11.4 data.table—Fast Subsetting
.....................
33
1.11.5 data.table—Calculations in Groups
..............
35
1.12 Data Sets
............................................
35
1.12.1 Census Data from UCI
.........................
36
1.12.2 Airquality
.....................................
37
1.12.3 Breast Cancer
.................................
37
1.12.4 Brittleness Index
...............................
38
1.12.5 Kola C-horizon Data
...........................
39
1.12.6 Colic Horse Data
..............................
39
1.12.7 New York Collission Data
.......................
40
1.12.8 Diabetes
.......................................
41
1.12.9 Austrian EU-SILC Data
........................
42
1.12.10 Food Consumption
.............................
45
1.12.11 Pulp Lignin
...................................
46
1.12.12 Structural Business Statistics Data
...............
46
1.12.13 Mammal Sleep Data
...........................
47
1.12.14 West Pacific Tropical Atmosphere Ocean Data
.....
49
1.12.15 Wine Tasting and Price of Wines
................
49
1.12.16 Further Data Sets
..............................
50
References
................................................
52
2 Distribution, Pre-analysis of Missing Values and Data Quality
..................................................
55
2.1 Introduction
..........................................
55
2.2 How Does Missing Data Arise?
..........................
56
2.2.1 Surveys in Official Statistics and Surveys Obtained with Questionaires
....................
56
2.2.2 Comment on Structural Zeros and Non-applicable Questions in a Questionaire
.......
58
2.2.3 Missing Values from Measuring Experiments
......
59
2.2.4 Censored Values
...............................
60
2.2.5 Monotone Missingness
..........................
60
2.3 Missing Value Mechanisms
..............................
62
2.3.1 Missing at Random (MAR)
.....................
63
2.3.2 Missing at Completely Random (MCAR)
.........
64
2.3.3 Missing Not at Random (MNAR)
................
64
2.3.4 Example
......................................
65
2.3.5 Summary on MCAR, MAR, and MNAR
..........
65
Contents xvii
2.4 Limitations for the Detection of the Missing Value Mechanisms
..........................................
66
2.5 Kinds of Attributes
....................................
69
2.5.1 Binary and Nominal Variables and Related Distances
.....................................
69
2.5.2 Ordered Categorical Variables
...................
70
2.5.3 Count Variables, Continuous Variables, Semi-continuous Variables, and Related Distances
.....................................
70
2.5.4 The Gower Distance
............................
72
2.6 Data Quality and Consistency of Data
...................
73
2.6.1 Outliers
......................................
73
2.6.2 Rule-Based Approaches for Checking the Consistency of Data
............................
80
2.6.3 Localization of Inconsistencies and Errors
.........
80
References
................................................
83
3 Detection of the Missing Values Mechanism with Tests and Models
..............................................
89
3.1 Introduction
..........................................
89
3.2 A Simple t-Test for MCAR
.............................
90
3.3 Non-parametric Version
................................
93
3.4 Extension to the Multiple Case
..........................
93
3.5 Little’s Test on MCAR and Extensions
...................
96
3.6 Further Tests
.........................................
102
References
................................................
105
4 Visualization of Missing Values
..........................
107
4.1 Motivation
...........................................
107
4.1.1 Why to Apply Visualization Methods
............
108
4.1.2 Software
.......................................
111
4.2 Rough Summaries and the Aggregation Plot
..............
112
4.3 Histogram, Barplot, Spinogram, and Spine Plot
...........
119
4.4 Parallel Boxplots
......................................
125
4.5 Scatterplots
...........................................
127
4.6 Scatterplot Matrices
...................................
130
4.7 Scatterplot Faceting
....................................
131
4.8 Parallel Coordinate Plot
................................
133
4.9 Matrix Plot
...........................................
137
4.10 Visualizing Missing Values in Multivariate Categorical Data with Mosaic Plots
.................................
141
4.11 Missing Values in Maps
................................
142
4.12 Studying Dropouts in Longitudinal Cohort Studies
........
144
4.13 Summary
.............................................
146
References
................................................
147
xviii Contents
5 General Considerations on Univariate Methods: Single and Multiple Imputation
.................................
151
5.1 A Few Words on Listwise Deletion; Deletion of Observation with Missing Values
.........................
151
5.1.1 Bias in Complete Case Analysis
.................
152
5.2 Univariate Imputation Methods
.........................
155
5.2.1 Imputation with the Mean
......................
155
5.2.2 Bias for Mean Imputation
.......................
157
5.2.3 Univariate Mean Imputation for Non-continuous Variables
.....................................
160
5.3 What Is Single Imputation?
..............................
161
5.4 What Is Multiple Imputation?
...........................
161
5.4.1 General Steps in Multiple Imputation
............
162
5.4.2 Benefits
......................................
164
5.4.3 Requirements
.................................
164
5.4.4 How Many Imputations?
........................
164
5.5 Single Imputation Versus Multiple Imputation
............
165
5.5.1 Why Standard Errors in Single Imputation Are Not Quite Correct?
............................
167
5.6 Pooling of Multiple Imputation Results
...................
172
5.7 General Concepts to Allow for Randomness of Imputations
..........................................
178
5.8 Joint Modeling, Distribution Fitting, and Copulas
.........
184
5.9 The EM Algorithm and Fully Conditional Modeling in the Multivariate Case
..................................
187
References
................................................
190
6 Deductive Imputation and Outlier Replacement
.........
193
6.1 Correction of Errors, But Where?
.......................
194
6.2 Correct Typos in Continuous Variables
...................
195
6.3 Adjusting Values After Imputation to Match Restrictions
...
198
6.4 Imputing Outliers and Erroneous Values
..................
200
References
................................................
204
7 Imputation Without a Formal Statistical Model
.........
207
7.1 Hot-Deck Imputation
..................................
208
7.1.1 Random Hot-Deck
.............................
209
7.1.2 Random or Sequential Hot-Deck with Constraints
....................................
211
7.1.3 Sequential Hot-Deck
...........................
212
7.1.4 An Application of Hot-Deck
.....................
213
7.1.5 Computational Time Revisited
..................
214
7.1.6 Some Remarks on Hot-Deck Imputation
..........
215
Contents xix
7.2 k Nearest Neighbor Methods
............................
216
7.2.1 Distance Calculation Within kNN and Weighting
.................................
217
7.2.2 Illustration of kNN Imputation
..................
219
7.2.3 Application of kNN
............................
220
7.2.4 Weighting the Variables
.........................
221
7.2.5 Random k Nearest Neighbor Imputation
..........
224
7.3 Covariance-Based Methods
.............................
225
7.3.1 Principal Component Analysis
...................
228
7.3.2 Imputation with Principal Component Analysis
......................................
230
7.3.3 Multiple Imputation with PCA Imputation
........
231
7.3.4 A Simple Example to Compare the Approaches
....
232
References
................................................
233
8 Model-Based Methods
...................................
237
8.1 Linear Regression
.....................................
238
8.2 Robust Linear Regression
...............................
238
8.2.1 Regression M Estimator
........................
240
8.2.2 Weight Functions
...............................
241
8.2.3 Regression S Estimator
.........................
243
8.2.4 MM-Estimator
................................
244
8.3 Robustness in Regression Imputation
....................
244
8.3.1 Why Not Use Expected Values from a Model?
.....
245
8.4 Making Regression-Based Imputation Methods Fit for Multiple Regression
....................................
247
8.4.1 (Normal) Noise Added to Predicted Values
........
248
8.4.2 Bootstrap Residuals and Addition to the Predicted Values of the Missings
.................
250
8.4.3 Additional Consideration of Model Uncertainty Through Bootstrapping
..........................
251
8.4.4 Additional Consideration of Model Uncertainty Through Bayesian Regression
....................
253
8.5 Predictive Mean Matching (PMM)
.......................
257
8.6 Weighted Donor Selection with Midastouch
................
261
8.7 EM-Based Imputation and Extensions to Non-continuous Variables
.............................................
263
8.8 Robust Stepwise Sequential EM-Based Imputation with IRMI
................................................
265
8.9 Enhancement of IRMI: Robust Multiple Imputation Using imputeRobust
...................................
269
References
.................................................
271
xx Contents
9 Nonlinear Methods
......................................
273
9.1 Tree-based Methods Using Random Forests
...............
274
9.1.1 Imputation Using Random Forests
...............
276
9.2 Tree-Based Methods Using XGBoost
.....................
279
9.2.1 Imputation Using XGBoost
.....................
280
9.3 Generalized Additive Models
............................
283
9.3.1 Splines
.......................................
284
9.3.2 Piecewise Polynomials
..........................
286
9.3.3 GAM’s
........................................
291
9.3.4 Imputation with GAMs Using Thin Plate Regression Splines
.............................
298
9.3.5 Imputation with GAMs for Location, Scale, and Shape
........................................
302
9.4 Artificial Neural Network–Based Methods
................
304
9.4.1 Fully Conditional Modelling with Artificial Neural Networks
...............................
305
9.4.2 Artificial Neural Networks to Impute Missing Values
........................................
308
9.4.3 Extension to Impute Rounded Zeros in Compositional Data
............................
315
9.4.4 Imputation Using GAN (GAIN)
.................
318
References
................................................
319
10 Methods for Compositional Data
........................
325
10.1 What are Compositional Data?
..........................
325
10.1.1 Negative Bias
.................................
326
10.1.2 The Simplex Sample Space
......................
327
10.1.3 Absolute or Relative Information
................
328
10.1.4 Log-ratio Coordinate Representation
.............
329
10.1.5 Requirements of a Compositional Analysis
........
330
10.1.6 Are Outliers Also Relevant for Compositional Data?
.........................................
331
10.2 Different Types of Missing Information
...................
332
10.3 Imputation of Missing Values
...........................
332
10.3.1 k-nearest neighbor imputation
...................
333
10.3.2 Iterative Model-Based Imputation
................
335
10.3.3 Using the R-package robCompositions for Imputing Missing Values
........................
337
10.3.4 Comments
.....................................
341
10.4 Imputation of Rounded and Count Zeros
..................
341
10.4.1 Imputation of Rounded Zeros
...................
343
10.4.2 Model-Based Replacement of Rounded Zeros
......
345
10.4.3 An Artificial Neural Network Approach to Impute Rounded Zeros
.........................
347
10.4.4 Rounded Zeros in High-Dimensional Data
.........
347
Contents xxi
10.4.5 Application in R
...............................
349
10.4.6 Count Zeros
...................................
350
10.5 Compositional Approach for Structural Zeros
..............
351
10.5.1 Avoiding Zeros by Amalgamation
................
352
10.5.2 Imputation of Structural Zeros as Auxiliary Step
...
353
10.5.3 Application in R
...............................
354
References
................................................
359
11 Evaluation of the Quality of Imputation
.................
363
11.1 Visual Inspection of Imputed Values
.....................
363
11.1.1 Aggregation Plots of Missings and Imputed Values
........................................
364
11.1.2 Comparing Complete Cases with Imputed Data
....
367
11.1.3 An Example of a Univariate Plot: Histogram with Imputed Values
...........................
369
11.1.4 An Example of a Multiple Plot: Marginplot with Imputed Missings
..............................
370
11.2 Parallel Coordinate Plot for Imputed Values
..............
377
11.3 Biplot Analysis of Imputed Data
........................
379
11.4 Insepection of Multiple Imputed Data Sets
................
384
11.5 Tours
................................................
386
11.6 How to Apply Evaluation Measures
......................
388
11.7 Evaluation Measures: Precision
..........................
389
11.7.1 Mean Absolute Percentage Error (MAPE)
........
389
11.7.2 Normalized Root Mean Squared Error (NRMSE)
.....................................
390
11.7.3 Considering Continuous and Semi-Continuous Variables
.....................................
390
11.7.4 Percentage of Falsely Classified Entries
............
391
11.7.5 Illustration of Precision Measures
.................
391
11.7.6 Compositional Error Deviation
..................
392
11.8 Evaluation Measures: Correlation-Based Measures
.........
393
11.9 Evaluation Measures Based on Estimators
................
394
11.9.1 Bias, Variance and Mean Squared Error of an Estimator
.....................................
394
11.9.2 Coverage Rate
.................................
402
11.10Evaluation Based on Prediction Performance
..............
405
11.11Final Comments
.......................................
406
References
................................................
406
12 Simulation of Data for Simulation Studies
...............
409
12.1 Introduction
..........................................
409
12.2 Type of Simulation
....................................
410
12.2.1 Insertion of Missing Values in Real Data
...........
411
12.2.2 Model-Based Simulation
........................
415
12.2.3 Design-Based Simulation
........................
417
xxii Contents
12.3 Inserting Missing Values
................................
418
12.3.1 Simulating MCAR Data (Model-Based)
...........
418
12.3.2 Creating Patterns of Missingness
..................
421
12.3.3 Simulating MAR Data
..........................
422
12.3.4 Simulating MNAR Data
........................
427
12.4 Simulating Data Using Covariances (Model-Based)
........
427
12.5 Simulating an Additional Variable
.......................
429
12.6 Simulating (High-Dimensional) Data with a Latent Model
................................................
430
12.7 A Design-Based Simulation Study Based on a Complex Survey from a Finite Population
.........................
434
12.7.1 Setup of the Structure
..........................
438
12.7.2 Simulation of Additional Variables
...............
439
12.7.3 Example in R
..................................
441
12.8 Prediction Performance Simulation
......................
446
12.9 Final Comments
........................................
451
References
................................................
456
Index
........................................................
459