Contents
Preface vii
Acknowledgments ix
List of Tables xix
List of Figures xxiii
1 Introduction 1
1.1 Overview . . . . . . .  . . . . . . . . . . . . . . . . . 1
1.2 Multivariate Models and Methods . . . . . . . . . .. . . . 1
1.3 Scope of the Book . . . . . .  . . . . . . . . . . . . . . 3
2 Vectors and Matrices 7
2.1 Introduction . . . . . . . . . . .. . . . . . . . . . . . 7
2.2 Vectors, Vector Spaces, and Vector Subspaces .. . 7
a. Vectors . . . . . .. . . . . . . . . . . . . . . . 7
b. Vector Spaces . . . . . . . . . . . . . . . .. . . 8
c. Vector Subspaces . . . . . . . . . . . . . . . . . 9
2.3 Bases, Vector Norms, and the Algebra of Vector Spaces . . 12
a. Bases . . . . . . . . . . . .. . . . . . . . . . . . . 13
b. Lengths, Distances, and Angles . . .  . . . . . .. 13
c. Gram-Schmidt Orthogonalization Process . . . . . 15
d. Orthogonal Spaces . . . . . . . . . . .  . . . . .. 17
e. Vector Inequalities, Vector Norms, and Statistical Distance . . 21
xii Contents
2.4 BasicMatrixOperations . . . . . . . . . . . . . . . . 25
a. Equality, Addition, and Multiplication of Matrices .  . 26
b. Matrix Transposition . . . . . . . . . . . . . . . . . . . . 28
c. Some Special Matrices . . . . . . . . . . . . . . . . . . 29
d. Trace and the Euclidean Matrix Norm . . . . . . 30
e. Kronecker and Hadamard Products . . . . . . . . 32
f. DirectSums . . . . . . . . . . . . . . . . . . . .. . . . . . . 35
g. The Vec(•) and Vech(•)Operators . . . . . . . . . . 35
2.5 Rank, Inverse, and Determinant . . . . . . . . . 41
a. Rank and Inverse . . . . . . . . . . . . . . . . . . . . . 41
b. Generalized Inverses . . . . . . . . . . . . .. . . . . . 47
c. Determinants . . . . . . . . . . . . . . . . . . . . . . . . 50
2.6 Systems of Equations, Transformations, and Quadratic Forms . 55
a. Systems ofEquations . . . . . . . . . . . . . . . . . . 55
b. Linear Transformations . . . . . . . . . . . . . . . . 61
c. ProjectionTransformations . . . . . . . . . . . 63
d. Eigenvalues andEigenvectors . . . .. . . . . . . 67
e. MatrixNorms . . . . . . . . . . . . . . . .. . . . . . 71
f. Quadratic Forms and Extrema . . .. . . . . . 72
g. Generalized Projectors . . . . . . .. . . . . . 73
2.7 Limits andAsymptotics . . . . . . . . . . . . 76
3 Multivariate Distributions and the Linear Model 79
3.1 Introduction . . . . . . . . . . . . . . . . . . . .. . . . 79
3.2 Random Vectors and Matrices . . .  . . . . . 79
3.3 TheMultivariateNormal (MVN)Distribution .  . . . 84
a. Properties of the Multivariate Normal Distribution . . 86
b. Estimating μ and _ . . . . . . . . . . . .. . . . . . 88
c. TheMatrixNormalDistribution . . . .. . . . . 90
3.4 The Chi-Square and Wishart Distributions . .. 93
a. Chi-Square Distribution . . . . . . . . . . . . 93
b. TheWishartDistribution . . . . . . . . . . . . 96
3.5 OtherMultivariateDistributions . . . .. . . 99
a. TheUnivariate t andFDistributions . . . . . . 99
b. Hotelling’s T 2Distribution . . . . . . . . . . 99
c. TheBetaDistribution . . . . . . . .  . . . . 101
d. Multivariate t, F, and χ2Distributions . 104
3.6 The General Linear Model . . . .. . . . . . 106
a. Regression, ANOVA, and ANCOVA Models . .. . 107
b. Multivariate Regression, MANOVA, and MANCOVA Models . . 110
c. The Seemingly Unrelated Regression (SUR) Model . 114
d. The General MANOVA Model (GMANOVA) . . . . . 115
3.7 Evaluating Normality . . . . . . . . .. . . 118
3.8 Tests of Covariance Matrices . .  . . . . . 133
a. Tests of Covariance Matrices . . . . . .. . 133
Contents xiii
b. Equality of Covariance Matrices . . .. . . 133
c. Testing for a Specific Covariance Matrix . . . 137
d. Testing for Compound Symmetry . . . . . . 138
e. Tests of Sphericity . . . . . . . . . . . 139
f. Tests of Independence . . . . .. . . . . 143
g. Tests for Linear Structure . . . . . . . 145
3.9 Tests ofLocation . . . . . .  . . . 149
a. Two-Sample Case, _1 = _2 = _ . .  . . . 149
b. Two-Sample Case, _1 _= _2 . . .. . . . . 156
c. Two-Sample Case, Nonnormality . . . . 160
d. ProfileAnalysis,OneGroup . . . . . . . . 160
e. Profile Analysis, Two Groups . .  . . . 165
f. Profile Analysis, _1 _= _2 . . . . .. . 175
3.10 UnivariateProfileAnalysis . . . . . . . 181
a. UnivariateOne-GroupProfileAnalysis . . . . . . . 182
b. UnivariateTwo-GroupProfileAnalysis . . . . . . . 182
3.11 PowerCalculations . . . . . . . . . .. . . .. . 182
4 Multivariate Regression Models 185
4.1 Introduction . . . . . . . . . . .. . . . . . . 185
4.2 MultivariateRegression . . .  . . . . . . . . 186
a. Multiple Linear Regression . . .  . . . . . . . . 186
b. Multivariate Regression Estimation and Testing Hypotheses . 187
c. Multivariate Influence Measures . . . . . . . . . 193
d. Measures of Association, Variable Selection and Lack-of-Fit Tests  197
e. Simultaneous Confidence Sets for a New Observation ynew
and the Elements of B . . . . . . . . . . . . . . . 204
f. Random X Matrix and Model Validation: Mean Squared Error
ofPrediction inMultivariateRegression . . .  . 206
g. Exogeniety in Regression . . . . .. . . . . . . . . 211
4.3 MultivariateRegressionExample . . .. . . . . . 212
4.4 One-WayMANOVAandMANCOVA . .. . . . 218
a. One-WayMANOVA . . . . . .. . . . . 218
b. One-WayMANCOVA . . . . . . . . . . . 225
c. Simultaneous Test Procedures (STP) for One-WayMANOVA
/MANCOVA . . . . . . . .. . . . . . 230
4.5 One-WayMANOVA/MANCOVAExamples . . . . . 234
a. MANOVA(Example 4.5.1) . . . . .. . . . . . . . . 234
b. MANCOVA(Example 4.5.2) . . . . . . . . . . . . 239
4.6 MANOVA/MANCOVA with Unequal _i or Nonnormal Data .. 245
4.7 One-Way MANOVA with Unequal _i Example . . 246
4.8 Two-WayMANOVA/MANCOVA . . . . . . 246
a. Two-WayMANOVAwithInteraction . . .. . . 246
b. AdditiveTwo-WayMANOVA . .. . . . . 252
c. Two-WayMANCOVA . . . . . . . . . . 256
xiv Contents
d. Tests of Nonadditivity . . . . . . . . 256
4.9 Two-WayMANOVA/MANCOVAExample .. . . 257
a. Two-WayMANOVA(Example 4.9.1) . . . . . 257
b. Two-WayMANCOVA(Example 4.9.2) .. . . 261
4.10 Nonorthogonal Two-Way MANOVA Designs . .. . . 264
a. Nonorthogonal Two-Way MANOVA Designs with andWithout
EmptyCells, and Interaction . . .. . . . 265
b. AdditiveTwo-WayMANOVADesignsWithEmptyCells .. 268
4.11 Unbalance, Nonorthogonal Designs Example . . . 270
4.12 Higher Ordered Fixed Effect, Nested and Other Designs .. . 273
4.13 ComplexDesignExamples . . . . . . . . 276
a. NestedDesign(Example 4.13.1) . . . . . 276
b. Latin Square Design (Example 4.13.2) . . . . . . 279
4.14 Repeated Measurement Designs . . .. . . . . . . 282
a. One-Way Repeated Measures Design . . .. . . . . . 282
b. Extended Linear Hypotheses . . . . . .. . . . 286
4.15 Repeated Measurements and Extended Linear Hypotheses Example  294
a. Repeated Measures (Example 4.15.1) . . .. . . . 294
b. Extended Linear Hypotheses (Example 4.15.2) . . .. . 298
4.16 Robustness and Power Analysis for MR Models . . .. . . 301
4.17 PowerCalculations—Power.sas . . . . . . .. . . . . 304
4.18 Testing for Mean Differences with Unequal Covariance Matrices 307
5 Seemingly Unrelated Regression Models 311
5.1 Introduction . . . . . . . . . . . . . . . . 311
5.2 The SUR Model . . . . . . . . .. . . . . . . 312
a. Estimation and Hypothesis Testing . .. . . . 312
b. Prediction . . . . . . . .. . . . . . . 314
5.3 Seeming Unrelated Regression Example . .. 316
5.4 The CGMANOVA Model .. . . . . . . . . . 318
5.5 CGMANOVAExample . . . .. . . . . . 319
5.6 The GMANOVA Model . . .. . . . . . 320
a. Overview . . . . . . . . .. . . . 320
b. Estimation and Hypothesis Testing . .. . . . . . 321
c. Test ofFit . . . . . . . . . . . . 324
d. Subsets of Covariates . . . . . . 324
e. GMANOVAvsSUR . . . . .. . . 326
f. MissingData . . . . . . . . . .. . . 326
5.7 GMANOVAExample . . . . . . . . . . . . 327
a. OneGroupDesign (Example 5.7.1) . .. . . . 328
b. TwoGroupDesign (Example 5.7.2) . . . .. . . . 330
5.8 Tests of Nonadditivity . . . . . . . . . . . 333
5.9 Testing for Nonadditivity Example . .. . . . . . . . . 335
5.10 Lack ofFitTest . . . . . . . . . . . . .. . . . . . 337
5.11 SumofProfileDesigns . . . . . . . . . . . . . 338
Contents xv
5.12 The Multivariate SUR (MSUR) Model . .. . . . . 339
5.13 SumofProfileExample . . . . . . . . . . . . . 341
5.14 Testing Model Specification in SUR Models  . . . . 344
5.15 Miscellanea . . . . . . . . . . . . . . . . . . . 348
6 Multivariate Random and Mixed Models 351
6.1 Introduction . . . . . . . . . . . . . . 351
6.2 Random Coefficient Regression Models . . . . . . 352
a. Model Specification . . . . . .. . . . . 352
b. Estimating theParameters . . .. . . . . . 353
c. Hypothesis Testing . . . . . . . . . . . . . 355
6.3 Univariate General Linear Mixed Models . .. . . 357
a. Model Specification . . . . .. . . . . . . . 357
b. Covariance Structures and Model Fit . . . . . . 359
c. Model Checking . . . . . . . . . . . 361
d. Balanced Variance Component Experimental Design Models .. 366
e. Multilevel Hierarchical Models . . . 367
f. Prediction . . . . . . . . . . . . . . . . 368
6.4 Mixed Model Examples . . . . . . . . 369
a. Random Coefficient Regression (Example 6.4.1) . . . 371
b. Generalized Randomized Block Design (Example 6.4.2) .. 376
c. Repeated Measurements (Example 6.4.3) . .. . 380
d. HLM Model (Example 6.4.4) . . . . . .. . . . 381
6.5 Mixed Multivariate Models . . .. . . . . 385
a. Model Specification . . . . . . . . .. . 386
b. Hypothesis Testing . . . . . . . . . . . . . 388
c. Evaluating Expected Mean Square . . . . . . . 391
d. Estimating theMean . . . .. . . 392
e. Repeated Measurements Model . . . .. . 392
6.6 Balanced Mixed Multivariate Models Examples . . . 394
a. Two-wayMixedMANOVA . . . . . . . . 395
b. Multivariate Split-Plot Design . . . . . . . . 395
6.7 Double Multivariate Model (DMM) . . . . . . 400
6.8 Double Multivariate Model Examples . . . . 403
a. Double Multivariate MANOVA (Example 6.8.1) . .. . 404
b. Split-Plot Design (Example 6.8.2) . . .. . 407
6.9 Multivariate Hierarchical Linear Models . . . . . . 415
6.10 Tests of Means with Unequal Covariance Matrices  . . . 417
7 Discriminant and Classification Analysis 419
7.1 Introduction . . . . . . . . 419
7.2 TwoGroupDiscriminationandClassification . .. . 420
a. Fisher’s Linear Discriminant Function . .. . . 421
b. Testing Discriminant Function Coefficients . .. . 422
c. ClassificationRules . . . .. . . . 424
xvi Contents
d. EvaluatingClassificationRules . . . . 427
7.3 Two Group Discriminant Analysis Example . . 429
a. Egyptian Skull Data (Example 7.3.1) .. . . . 429
b. BrainSize (Example 7.3.2) . . . . . . . . . 432
7.4 Multiple Group Discrimination and Classification . .. 434
a. Fisher’s Linear Discriminant Function . . . . 434
b. Testing Discriminant Functions for Significance . . . 435
c. VariableSelection . . . . . . . . . 437
d. ClassificationRules . . . .. . 438
e. LogisticDiscrimination andOtherTopics . .. 439
7.5 Multiple Group Discriminant Analysis Example . . . . 440
8 Principal Component, Canonical Correlation, and Exploratory
Factor Analysis 445
8.1 Introduction . . . . . . . . . . . . . . . . . . 445
8.2 Principal Component Analysis . . . .. . . . 445
a. Population Model for PCA . . . . . . . . . 446
b. Number of Components and Component Structure . . . . . 449
c. Principal Components with Covariates . . . . . . 453
d. SamplePCA . . . . . . . . . . . . . . . . 455
e. Plotting Components . . .. . . . . 458
f. Additional Comments . . . . . . . . . . 458
g. Outlier Detection . . . . . . . . . . . . . 458
8.3 Principal Component Analysis Examples . . . . . 460
a. TestBattery (Example 8.3.1) . . . . . . . . . 460
b. SemanticDifferentialRatings (Example 8.3.2) .. . . 461
c. Performance Assessment Program (Example 8.3.3) . . . . 465
8.4 Statistical Tests in Principal Component Analysis . . . . 468
a. Tests Using the Covariance Matrix . . . 468
b. TestsUsing aCorrelationMatrix . .. . 472
8.5 Regression on Principal Components . .. . . 474
a. GMANOVA Model . . . . . . . 475
b. The PCA Model . . . . . . . . . 475
8.6 Multivariate Regression on Principal Components Example . . 476
8.7 Canonical Correlation Analysis . .. . 477
a. Population Model for CCA . . . .. . 477
b. SampleCCA . . . . . .. . . 482
c. Tests of Significance . . . .. . . 483
d. Association and Redundancy . .. . . . . . . . . . . . 485
e. Partial, Part and Bipartial Canonical Correlation . .. 487
f. PredictiveValidityinMultivariateRegression usingCCA . 490
g. Variable Selection and Generalized Constrained CCA . . 491
8.8 Canonical Correlation Analysis Examples .. . 492
a. RohwerCCA(Example 8.8.1) . . . .. . . . 492
b. Partial andPartCCA (Example 8.8.2) .. . . 494
Contents xvii
8.9 ExploratoryFactorAnalysis .. . 496
a. Population Model for EFA . . . . . . . 497
b. Estimating Model Parameters . .. 502
c. Determining Model Fit . . . . . . . . . . . 506
d. FactorRotation . . . . . .. . . . . . . . 507
e. Estimating Factor Scores . . .. . . . . . . 509
f. Additional Comments . . .. . . . . 510
8.10 ExploratoryFactorAnalysisExamples . . . . . 511
a. Performance Assessment Program (PAP—Example 8.10.1) 511
b. DiVesta andWalls (Example 8.10.2) . 512
c. Shin (Example 8.10.3) . .. . . . . 512
9 Cluster Analysis and Multidimensional Scaling 515
9.1 Introduction . . . . . . . . . . . . . . 515
9.2 ProximityMeasures . . . . . . . 516
a. DissimilarityMeasures .. . . . . . . . . 516
b. SimilarityMeasures . . . . . . . . . . . . 519
c. ClusteringVariables . .. . . . . . . . 522
9.3 ClusterAnalysis . . .. . . . . . . . . 522
a. Agglomerative Hierarchical Clustering Methods . 523
b. Nonhierarchical Clustering Methods . . . . 530
c. Number ofClusters . . . . .. . . . 531
d. Additional Comments . . . . . . . . 533
9.4 ClusterAnalysisExamples . . . .. . . 533
a. ProteinConsumption (Example 9.4.1) . . . . . 534
b. Nonhierarchical Method (Example 9.4.2) . . . . 536
c. Teacher Perception (Example 9.4.3) .. . . 538
d. Cedar Project (Example 9.4.4) . .. . . . 541
9.5 Multidimensional Scaling . . . . .. . . . . . 541
a. ClassicalMetricScaling . . . . . . . . . . 542
b. NonmetricScaling . . . . . . . . . . 544
c. Additional Comments . . . . . . . . . 547
9.6 Multidimensional Scaling Examples . . . . 548
a. ClassicalMetricScaling (Example 9.6.1) .. 549
b. Teacher Perception (Example 9.6.2) .. . 550
c. Nation (Example 9.6.3) . . . . . 553
10 Structural Equation Models 557
10.1 Introduction . . . . . . . . . . . . 557
10.2 Path Diagrams, Basic Notation, and the General Approach . . 558
10.3 Confirmatory Factor Analysis . . . . . . 567
10.4 Confirmatory Factor Analysis Examples . .. . 575
a. Performance Assessment 3 - Factor Model (Example 10.4.1) 575
b. Performance Assessment 5-Factor Model (Example 10.4.2) 578
10.5 PathAnalysis . . . . . . . . . . . . . . . 580
xviii Contents
10.6 PathAnalysisExamples . . . . . . 586
a. Community Structure and Industrial Conflict (Example 10.6.1) 586
b. Nonrecursive Model (Example 10.6.2) . . 590
10.7 StructuralEquationswithManifest andLatentVariables .. 594
10.8 StructuralEquationswithManifest andLatentVariablesExample. 595
10.9 LongitudinalAnalysiswithLatentVariables . . . . . 600
10.10 Exogeniety in Structural Equation Models . .. . . . 604
Appendix 609
References 625
Author Index 667
Subject Index 675