The Matrix Cookbook
                                                  Kaare Brandt Petersen
                                                 Michael Syskind Pedersen
                                               Version: January 5, 2005
                            What is this? These pages are a collection of facts (identities, approxima-
                            tions, inequalities, relations, ...) about matrices and matters relating to them.
                            It is collected in this form for the convenience of anyone who wants a quick
                            desktop reference .
                            Disclaimer: Theidentities, approximations and relations presented here were
                            obviously not invented but collected, borrowed and copied from a large amount
                            of sources. These sources include similar but shorter notes found on the internet
                            and appendices in books - see the references for a full list.
                            Errors: Very likely there are errors, typos, and mistakes for which we apolo-
                            gize and would be grateful to receive corrections at kbp@imm.dtu.dk.
                            Its ongoing:  The project of keeping a large repository of relations involving
                            matrices is naturally ongoing and the version will be apparent from the date in
                            the header.
                            Suggestions: Your suggestion for additional content or elaboration of some
                            topics is most welcome at kbp@imm.dtu.dk.
                            Acknowledgements: We would like to thank the following for discussions,
                            proofreading, extensive corrections and suggestions: Esben Hoegh-Rasmussen
                            and Vasile Sima.
                            Keywords: Matrix algebra, matrix relations, matrix identities, derivative of
                            determinant, derivative of inverse matrix, di®erentiate a matrix.
                                                              1
                                  CONTENTS                                                                 CONTENTS
                                  Contents
                                  1 Basics                                                                              5
                                      1.1   Trace and Determinants . . . . . . . . . . . . . . . . . . . . . . .        5
                                      1.2   The Special Case 2x2 . . . . . . . . . . . . . . . . . . . . . . . . .      6
                                  2 Derivatives                                                                         7
                                      2.1   Derivatives of a Determinant . . . . . . . . . . . . . . . . . . . .        7
                                      2.2   Derivatives of an Inverse . . . . . . . . . . . . . . . . . . . . . . .     8
                                      2.3   Derivatives of Matrices, Vectors and Scalar Forms         . . . . . . . .   9
                                      2.4   Derivatives of Traces . . . . . . . . . . . . . . . . . . . . . . . . .    11
                                      2.5   Derivatives of Structured Matrices . . . . . . . . . . . . . . . . .       12
                                  3 Inverses                                                                           14
                                      3.1   Exact Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . .    14
                                      3.2   Implication on Inverses . . . . . . . . . . . . . . . . . . . . . . . .    14
                                      3.3   Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . .     15
                                      3.4   Generalized Inverse . . . . . . . . . . . . . . . . . . . . . . . . . .    15
                                      3.5   Pseudo Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . .     15
                                  4 Complex Matrices                                                                   17
                                      4.1   Complex Derivatives . . . . . . . . . . . . . . . . . . . . . . . . .      17
                                  5 Decompositions                                                                     20
                                      5.1   Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . .       20
                                      5.2   Singular Value Decomposition . . . . . . . . . . . . . . . . . . . .       20
                                      5.3   Triangular Decomposition . . . . . . . . . . . . . . . . . . . . . .       21
                                  6 General Statistics and Probability                                                 22
                                      6.1   Moments of any distribution . . . . . . . . . . . . . . . . . . . . .      22
                                      6.2   Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . .     22
                                  7 Gaussians                                                                          24
                                      7.1   One Dimensional . . . . . . . . . . . . . . . . . . . . . . . . . . .      24
                                      7.2   Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   25
                                      7.3   Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .      27
                                      7.4   Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . .    29
                                      7.5   One Dimensional Mixture of Gaussians . . . . . . . . . . . . . . .         29
                                      7.6   Mixture of Gaussians . . . . . . . . . . . . . . . . . . . . . . . . .     30
                                  8 Miscellaneous                                                                      31
                                      8.1   Functions and Series . . . . . . . . . . . . . . . . . . . . . . . . .     31
                                      8.2   Indices, Entries and Vectors . . . . . . . . . . . . . . . . . . . . .     32
                                      8.3   Solutions to Systems of Equations       . . . . . . . . . . . . . . . . .  35
                                      8.4   Block matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . .     36
                                      8.5   Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . .     37
                                      8.6   Positive De¯nite and Semi-de¯nite Matrices . . . . . . . . . . . .         38
                                      Petersen & Pedersen, The Matrix Cookbook (Version: January 5, 2005), Page 2
                                  CONTENTS                                                                CONTENTS
                                      8.7  Integral Involving Dirac Delta Functions . . . . . . . . . . . . . .      39
                                      8.8  Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   40
                                  A Proofs and Details                                                               41
                                     Petersen & Pedersen, The Matrix Cookbook (Version: January 5, 2005), Page 3
                           CONTENTS                                                 CONTENTS
                           Notation and Nomenclature
                                   A     Matrix
                                  A      Matrix indexed for some purpose
                                    ij
                                  Ai     Matrix indexed for some purpose
                                    ij
                                  A      Matrix indexed for some purpose
                                  An     Matrix indexed for some purpose or
                                         The n.th power of a square matrix
                                  A−1    The inverse matrix of the matrix A
                                    +
                                  A      The pseudo inverse matrix of the matrix A
                                  A1/2   The square root of a matrix (if unique), not elementwise
                                 (A)ij   The (i,j).th entry of the matrix A
                                  Aij    The (i,j).th entry of the matrix A
                                   a     Vector
                                   ai    Vector indexed for some purpose
                                   ai    The i.th element of the vector a
                                   a     Scalar
                                  ℜz     Real part of a scalar
                                  ℜz     Real part of a vector
                                  ℜZ     Real part of a matrix
                                  ℑz     Imaginary part of a scalar
                                  ℑz     Imaginary part of a vector
                                  ℑZ     Imaginary part of a matrix
                                 det(A)  Determinant of A
                                  ||A||  Matrix norm (subscript if any denotes what norm)
                                  AT     Transposed matrix
                                  A∗     Complex conjugated matrix
                                  AH     Transposed and complex conjugated matrix
                                 A◦B Hadamard(elementwise) product
                                 A⊗B Kronecker product
                                   0     The null matrix. Zero in all entries.
                                   I     The identity matrix
                                    ij
                                  J      The single-entry matrix, 1 at (i,j) and zero elsewhere
                                   §     Apositive de¯nite matrix
                                   ¤     Adiagonal matrix
                              Petersen & Pedersen, The Matrix Cookbook (Version: January 5, 2005), Page 4