Contents

1 Basics 5

1.1 Trace and Determinants 5

1.2 The Special Case 2x2 5

2 Derivatives 7

2.1 Derivatives of a Determinant 7

2.2 Derivatives of an Inverse 8

2.3 Derivatives of Eigenvalues 9

2.4 Derivatives of Matrices, Vectors and Scalar Forms 9

2.5 Derivatives of Traces 11

2.6 Derivatives of vector norms 13

2.7 Derivatives of matrix norms 13

2.8 Derivatives of Structured Matrices 14

3 Inverses 16

3.1 Basic 16

3.2 Exact Relations 17

3.3 Implication on Inverses 19

3.4 Approximations 19

3.5 Generalized Inverse 20

3.6 Pseudo Inverse 20

4 Complex Matrices 23

4.1 Complex Derivatives 23

4.2 Higher order and non-linear derivatives 26

4.3 Inverse of complex sum 26

5 Solutions and Decompositions 27

5.1 Solutions to linear equations 27

5.2 Eigenvalues and Eigenvectors 29

5.3 Singular Value Decomposition 30

5.4 Triangular Decomposition 32

5.5 LU decomposition 32

5.6 LDM decomposition 32

5.7 LDL decompositions 32

6 Statistics and Probability 33

6.1 Definition of Moments 33

6.2 Expectation of Linear Combinations 34

7 Multivariate Distributions 36

7.1 Cauchy 36

7.2 Dirichlet 36

7.3 Normal 36

7.4 Normal-Inverse Gamma 36

7.5 Gaussian 36

7.6 Multinomial 36

7.7 Student’s t 36

7.8 Wishart 37

7.9 Wishart, Inverse 38

8 Gaussians 39

8.1 Basics 39

8.2 Moments 41

8.3 Miscellaneous 43

8.4 Mixture of Gaussians 44

9 Special Matrices 45

9.1 Block matrices 45

9.2 Discrete Fourier Transform Matrix, The 46

9.3 Hermitian Matrices and skew-Hermitian 47

9.4 Idempotent Matrices 48

9.5 Orthogonal matrices 48

9.6 Positive Definite and Semi-definite Matrices 50

9.7 Singleentry Matrix, The 51

9.8 Symmetric, Skew-symmetric/Antisymmetric 53

9.9 Toeplitz Matrices 54

9.10 Transition matrices 55

9.11 Units, Permutation and Shift 56

9.12 Vandermonde Matrices 57

10 Functions and Operators 58

10.1 Functions and Series 58

10.2 Kronecker and Vec Operator 59

10.3 Vector Norms 61

10.4 Matrix Norms 61

10.5 Rank 62

10.6 Integral Involving Dirac Delta Functions 62

10.7 Miscellaneous 63

A One-dimensional Results 64

A.1 Gaussian 64

A.2 One Dimensional Mixture of Gaussians 65

B Proofs and Details 67