Matrix Algebra: Exercises and Solutions

Springer Science+Business Media, LLC

David A. Harville

Matrix Algebra: Exercises and Solutions

" Springer

David A. Harville Mathematical Sciences Department IBM TJ. Watson Research Center Yorktown Heights, NY 10598-0218 USA

Library ofCongress Cataloging-in-Publieation Data Harville, David A.

Matrix algebra: exereises and solutions / David A. Harville. p. em.

lncludes bibliographieal referenees and index. ISBN 978-0-387-95318-2 ISBN 978-1-4613-0181-3 (eBook) DOI 10.1007/978-1-4613-0181-3

1. Matrices-Problems, exereises, etc. QAI88 .H38 2001 519.9'434--iic21

Printed on acid-free paper.

I. Title.

2001032838

© 2001 Springer Science+Business Media New York Originally published by Springer-Verlag New York, Ine in 2001 AII rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher Springer Science+Business Media, LLC, except for brief excerpts inconnection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden, The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the former are not especially identified, is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.

Production managed by Yong-Soon Hwang; manufacturing supervised by Jeffrey Taub. Photocomposed copy prepared from the author's LaTeX file.

9 8 765 432 I

ISBN 978-0-387-95318-2

Preface

This book comprises well over three-hundred exercises in matrix algebra and their solutions. The exercises are taken from my earlier book Matrix Algebra From a Statistician's Perspective. They have been restated (as necessary) to make them comprehensible independently of their source. To further insure that the restated exercises have this stand-alone property, I have included in the front matter a section on terminology and another on notation. These sections provide definitions, descriptions, comments, or explanatory material pertaining to certain terms and notational symbols and conventions from Matrix Algebra From a Statistician's Perspective that may be unfamiliar to a nonreader of that book or that may differ in generality or other respects from those to which he/she is accustomed. For example, the section on terminology includes an entry for scalar and one for matrix. These are standard terms, but their use herein (and in Matrix Algebra From a Statistician's Perspective) is restricted to real numbers and to rectangular arrays of real numbers, whereas in various other presentations, a scalar may be a complex number or more generally a member of a field, and a matrix may be a rectangular array of such entities.

It is my intention that Matrix Algebra: Exercises and Solutions serve not only as a "solution manual" for the readers of Matrix Algebra From a Statistician's Perspective, but also as a resource for anyone with an interest in matrix algebra (including teachers and students of the subject) who may have a need for exercises accompanied by solutions. The early chapters of this volume contain a relatively small number of exercises-in fact, Chapter 7 contains only one exercise and Chapter 3 only two. This is because the corresponding chapters of Matrix Alge­bra From a Statistician's Perspective cover relatively standard material, to which many readers will have had previous exposure, and/or are relatively short. It is

vi Preface

the final ten chapters that contain the vast majority of the exercises. The topics of many of these chapters are ones that may not be covered extensively (if at all) in more standard presentations or that may be covered from a different perspec­tive. Consequently, the overlap between the exercises from Matrix Algebra From a Statistician's Perspective (and contained herein) and those available from other sources is relatively small.

A considerable number of the exercises consist of verifying or deriving results supplementary to those included in the primary coverage of Matrix Algebra From a Statistician's Perspective. Thus, their solutions provide what are in effect proofs. For many of these results, including some of considerable relevance and interest in statistics and related disciplines, proofs have heretofore only been available (if at all) through relatively high-level books or through journal articles.

The exercises are arranged in 22 chapters and within each chapter, are numbered successively (starting with 1). The arrangement, the numbering, and the chapter titles match those in Matrix Algebra From a Statistician's Perspective. An exercise from a different chapter is identified by a number obtained by inserting the chapter number (and a decimal point) in front of the exercise number.

A considerable effort was expended in designing the exercises to insure an appropriate level of difficulty-the book Matrix Algebra From a Statistician's Perspective is essentially a self-contained treatise on matrix algebra, however it is aimed at a reader who has had at least some previous exposure to the subject (of the kind that might be attained in an introductory course on matrix or linear algebra). This effort included breaking some of the more difficult exercises into relatively palatable parts and/or providing judicious hints.

The solutions presented herein are ones that should be comprehensible to those with exposure to the material presented in the corresponding chapter of Matrix Algebra From a Statistician's Perspective (and possibly to that presented in one or more earlier chapters). When deemed helpful in comprehending a solution, references are included to the appropriate results in Matrix Algebra From a Statis­tician's Perspective-unless otherwise indicated a reference to a chapter, section, or subsection or to a numbered result (theorem, lemma, corollary, "equation", etc.) pertains to a chapter, section, or subsection or to a numbered result in Matrix Algebra From a Statistician's Perspective (and is made by following the same con­ventions as in the corresponding chapter of Matrix Algebra From a Statistician's Perspective). What constitutes a "legitimate" solution to an exercise depends of course on what one takes to be "given". If additional results are regarded as given, then additional, possibly shorter solutions may become possible.

The ordering of topics in Matrix Algebra From a Statistician's Perspective is somewhat nonstandard. In particular, the topic of eigenvalues and eigenvectors is deferred until Chapter 21, which is the next-to-Iast chapter. Among the key results on that topic is the existence of something called the spectral decomposition. This result if included among those regarded as given, could be used to devise alternative solutions for a number of the exercises in the chapters preceding Chapter 21. However, its use comes at a "price"; the existence of the spectral decomposition can only be established by resort to mathematics considerably deeper than those

Preface vii

underlying the results of Chapters 1-20 in Matrix Algebra From a Statistician's Perspective.

I am indebted to Emmanuel Yashchin for his support and encouragement in the preparation of the manuscript for Matrix Algebra: Exercises and Solutions. I am also indebted to Lorraine Menna, who entered much of the manuscript in IhTPC, and to Barbara White, who participated in the latter stages of the entry process. Finally, I wish to thank John Kimmel, who has been my editor at Springer-Verlag, for his help and advice.

Contents

Preface v

Some Notation ........................................................................................... xi

Some Terminology

1 Matrices

2 Submatrices and Partitioned Matrices

3 Linear Dependence and Independence

4 Linear Spaces: Rowand Column Spaces

5 Trace of a (Square) Matrix

6 Geometrical Considerations

7 Linear Systems: Consistency and Compatibility ............................ ..

8 Inverse Matrices

9 Generalized Inverses

10 Idempotent Matrices

xvii

7

11

13

19

21

27

29

35

49

x Contents

11 Linear Systems: Solutions ........... ...................................................... 55

12 Projections and Projection Matrices 63

13 Determinants ...................................................................................... 69

14 Linear, Bilinear, and Quadratic Forms 79

15 Matrix Differentiation 113

16 Kronecker Products and the Vec and Vech Operators 139

17 Intersections and Sums of Subspaces 161

18 Sums (and Differences) of Matrices 179

19 Minimization of a Second-Degree Polynomial (in n Variables) Subject to Linear Constraints ........................................................... 209

20 The Moore-Penrose Inverse 221

21 Eigenvalues and Eigenvectors 231

22 Linear Transformations 251

References 265

Index 267

Some Notation

{xd A row or (depending on the context) column vector whose ith element is Xi

{aij} A matrix whose ijth element is aij (and whose dimensions are arbitrary or may be inferred from the context)

A' The transpose of a matrix A

AP The pth (for a positive integer p) power of a square matrix A; i.e., the matrix product AA ... A defined recursively by setting A 0 = I and taking Ak = AAk- 1 (k = 1, ... , p)

C(A) Column space of a matrix A

R(A) Row space of a matrix A

R mxn The linear space comprising all m x n matrices

R n The linear space R nx I comprising all n-dimensional column vectors or (depending on the context) the linear space R I xn comprising all n­

dimensional row vectors

speS) Span of a finite set S of matrices; Sp({AI, ... , Ad), which represents the span of the set {AI, ... , Ad comprising the k matrices AI, ... , Ab is generally abbreviated to Sp(AI, ... , Ak)

C Writing SeT (or T ~ S) indicates that a set S is a (not necessarily proper) subset of a set T

dim (V) Dimension of a linear space V

rank A The rank of a matrix A

rank T The rank of a linear transformation T

xii Some Notation

tr(A) The trace of a (square) matrix A

o The scalar zero (or, depending on the context, the zero transformation from one linear space into another)

o A null matrix (whose dimensions are arbitrary or may be inferred from the context)

I An identity transformation

I An identity matrix (whose order is arbitrary or may be inferred from the context)

In An identity matrix of order n

A • B Inner product of a pair of matrices A and B (or if so indicated, quasi-inner product of the pair A and B)

II A II Norm of a matrix A (or, in the case of a quasi-inner product, the quasi norm of A)

8(A, B) Distance between two matrices A and B T- I

A-I

A­N(A)

N(T)

J..

J..w

Px

Px,w

The inverse of an invertible transformation T

The inverse of an invertible matrix A

An arbitrary generalized inverse of a matrix A

NulI space of a matrix A

Null space of a linear transformation T

A symbol for "is orthogonal to"

A symbol used to indicate (by writing x J..w y, x J..w U, orU J..w V) that 2 vectors x and y, a vector x and a subspace U, or 2 subspaces U and V are orthogonal with respect to a symmetric nonnegative definite matrix W

The matrix X(X'X)-X' [which is invariant to the choice of the general­ized inverse (X'X)-]

The matrix X(X'WX) -X'W [which if W is symmetric and positive def­inite, is invariant to the choice of the generalized inverse (X'WX) -]

The orthogonal complement of a subspace U of a linear space V

The orthogonal (with respect to the usual inner product) complement of the column space C(X) of an n x p matrix X [when C(X) is regarded as a subspace of R"]

The orthogonal complement of the column space C (X) of an n x p matrix X when, for an Ii x n symmetric positive definite matrix W, the inner product is taken to be the bilinear form x'Wy [and when C(X) is regarded as a subspace of R"]

A function whose value an(il, il;"'; ill, ill) for any two (not neces­sarily different) permutations of the first n positive integers is the num­ber of negative pairs among the G) pairs that can be formed from the i IiI, ... , in ill th elements of an n x Ii matrix

A function whose value ¢n (i I, ... , ill) for any sequence of n distinct

Some Notation xiii

IAI

det(A)

adj(A)

J

Jmn

A®B

vecA

vechA

Kmn

Gn

Hn

Df(e)

integers ii, ... , in is PI + ... + Pn-l, where (for k = 1, ... , n - 1) Pk represents the number of integers in the subsequence ik+ 1, ... , in that are smaller than h The determinant of a square matrix A - with regard to partitioned ma-

(All . .. Ale) All Ale

trices,: : may be abbreviated to

Arl Are Arl Are

The determinant of a square matrix A

The adjoint matrix of a square matrix A

A matrix, all of whose elements equal one (and whose dimensions are arbitrary or may be inferred from the context)

An m x n matrix, all of whose mn elements equal one

The Kronecker product of a matrix A and a matrix B - this notation extends in an obvious way to the Kronecker product of 3 or more matrices

The vec of a matrix A

The vech of a (square) matrix A

The mn x mn vec-permutation (or commutation) matrix

The n2 x n(n + 1)/2 duplication matrix

An arbitrary left inverse of Gn, so that Hn is any n(n + 1)/2 x n2

matrix such that HnGn = lor equivalently such that, for every n x n symmetric matrix A, vech A = Hn vec A - one choice for Hn is Hn = (G~Gn)-lG~

The jth (first-order) partial derivative of a function f, with domain S in nmxl, at an interior point e of S - the function whose value at a point cis Dj/(e) is represented by the symbol Dj/

The jth partial derivative of a function f of an m x 1 vector x =

(Xl, ... , Xm)' - an alternative notation to D j f or D j f (x)

The 1 x m vector [Dd(e), ... , Dm(e)] (where f is a function with domain S in nm x 1 and where c is an interior point of S) - similarly, Df is the 1 x m vector (Dlf, ... , Dmf)

The m x 1 vector (af /aXl, ... , af/axm)' of partial derivatives of a func­

tion f of an m x 1 vector x = (Xl, ... , xm )' - an alternative [to (D f)' or (D f (x) )'] notation for the gradient vector

The 1 x m vector (af/axl, ... , af/axm) of partial derivatives of a func­

tion f of an m x 1 vector x = (Xl, ... , xm)' - equals (af/ax)' and is an alternative notation to D f or D f (x)

D~ fee) The ijth second-order partial derivative of a function f, with domain S

in nm xl, at an interior point e of S - the function whose value at a point e is D~f(e) is represented by the symbol D~f

xiv Some Notation

~ aXiaXj An alternative [to Drjf or DrJ(x)] notation for the ijth (second-order)

partial derivative of a function 1 of an m x 1 vector x = (XI, ... , xm )'­

this notation extends in a straightforward way to third- and higher-order partial derivatives

HI The Hessian matrix of a function 1 - accordingly, Hf(c) represents the value of Hf at an interior point c of the domain of f

Djf The p x 1 vector (Dj iJ, ... , D J!p)', whose ith element is the jth partial derivative D j Ii of the ith element fi of a p x 1 vector f = (fl, ... , I p )'

of functions, each of whose domain is a set S in nmxl_ similarly, Djf(c} = [DjiJ (c), ... , DJ!p(c}]', where c is an interior point of S

g~ The p x q matrix whose stth element is the partial derivative afsr/axj

of the stth element of a p x q matrix F = {fsd of functions of a vector x = (XI, ... ,xm )' of m variables

a2F aXi ax j The p x q matrix whose stth element is the second-order partial derivative

a2fsr/aXiaXj of the stth elementofa p x q matrix F = {fsd offunctions of a vector x = (XI, ... ,xm )' of m variables-this notation extends in a straightforward way to a p x q matrix whose s tth element is one of the third- or higher-order partial derivatives of the stth element of F

Df The Jacobian matrix (Dlf, ... , Dmf) of a vector f = (fl,"" fp)' of functions, each of whose domain is a set S in nm x I - similarly, Df(c} = [Dlf(c), ... , Dmf(c}], where c is an interior point of S

:: An alternative [to Df or Df(x}] notation for the Jacobian matrix of a vector

f = (iJ, ... , fp)' of functions of an m x I vector x = (XI, ... ,xm )' -

af/ax' is the p x m matrix whose ijth element is ali/axj

ar' ax An alternative [to (Df)' or (Df(x»'] notation for the gradient (matrix)

of a vector f = (fJ, ... , fp)' of functions of an m x 1 vector x = (XI, ... , xm)' - ar' lax is the m x p matrix whose jith element is af;/axj

aaaf ax The derivative of a function f of an m x n matrix X of mn "independent" variables or (depending on the context) of an n x n symmetric matrix X - the matrix af/aX' is identical to (af/aX)'

u n V The intersection of 2 sets U and V of matrices-this notation extends in an obvious way to the intersection of 3 or more sets

U U V The union of 2 sets U and V of matrices (of the same dimensions )-this notation extends in an obvious way to the union of 3 or more sets

U + V The sum of 2 nonempty sets U and V of matrices (of the same dimen­sions)-this notation extends in an obvious way to the sum of 3 or more nonempty sets

U EI1 V The direct sum of 2 (essentially disjoint) linear spaces U and V in nm xn

Some Notation xv

- writing U EEl V (rather than U + V) serves to emphasize, or (in the absence of any previous indication) imply, that U and V are essentially disjoint and hence that their sum is a direct sum

A + The Moore-Penrose inverse of a matrix A (kT) The scalar multiple of a scalar k and a transformation T from a linear

space V into a linear space W; in the absence of any ambiguity, the parentheses may be dropped, that is, kT may be written in place of (kT)

(T + S) The sum of two transformations T and S from a linear space V into a linear space W; in the absence of any ambiguity, the parentheses may be dropped, that is, T + S may be written in place of (T + S) - this notation extends in an obvious way to the sum of three or more transformations

(T S) The product of a transformation T from a linear space V into a linear space W and a transformation S from a linear space U into V; in the absence of any ambiguity, the parentheses may be dropped, that is, T S may be written in place of (T S) - this notation extends in an obvious way to the product of three or more transformations

L B A transformation defined for any (nonempty) linearly independent set B of matrices (of the same dimensions), say the matrices YI, Y2, ... , Yn :

it is the transformation from nnxl onto the linear space W = sp(B) that assigns to each vector x = (Xl, X2, ... , xn)' in nnxl the matrix XIYI +X2Y2 + ... +xnYn in W.

Some Terminology

adjoint matrix The adjoint matrix of an n x n matrix A = {aij} is the transpose of the cofactor matrix of A (or equivalently is the n x n matrix whose ijth element is the cofactor of a ji).

algebraic mUltiplicity The characteristic polynomial, say p(o), of an n x n matrix A has a unique (aside from the order of the factors) representation of the form

p(A) = (_l)np .. - Ad Y1 ••• (A - Ak)Ykq(A) (-00 < A < 00),

where {AI, ... , Ak} is the spectrum of A (comprising the distinct scalars that are eigenvalues of A), YI, ... , Yk are (strictly) positive integers, and q is a polynomial (of degree n - L~=l yd that has no real roots; for i = I, ... , k, Yi is referred to as the algebraic multiplicity of the eigenvalue Ai .

basis A basis for a linear space V is a finite set of linearly independent matrices in V that spans V.

basis (natural) The natural basis for R mxn comprises the mn matrices UII, U21, ... , Uml, ... , Uln, U2n, ... , Umn ' where (fori = 1, ... , m andj = 1, '" n) Uij is the m x n matrix whose ijth element equals I and whose remaining mn - 1 elements equal 0; the natural (or usual) basis for the linear space of all n x n symmetric matrices comprises the n(n + 1)/2 matrices Uil' Uil' ... , U~l' ... , Wi' W+I,i' ... , U~i' ... , U~n' where (for i = I, ... , n) uti is the n x n matrix whose ith diagonal element equals 1 and whose remaining n2 - 1 elements equal 0 and (for j < i = 1, ... , n) utj is the n x n matrix whose ijth and jith elements equal I and whose remaining n2 - 2 elements equal O.

bilinear form A bilinear form in an m x I vector x = (XI, ... , xm)' and an n x 1 vector y = (YI, ... , Yn)' is a function of x and y (defined for x E R m and

xviii Some Terminology

Y E nn) that, for some m x n matrix A = {aij} (called the matrix of the bilinear form), is expressible as x'Ay = Li,j aijXiYj - the bilinear form is said to be symmetric if m = n and x' Ay = y' Ax for all x and all y or equivalently if the matrix A is symmetric,

block-diagonal A partitioned matrix of the form (A~1 A~2 ~ ) (all

o 0 Arr of whose off-diagonal blocks are null matrices) is said to be block-diagonal and may be expressed in abbreviated notation as diag(AIl, A22, ... , Arr).

o A22 ... A2r (

All AJ2 ... Air)

block-triangular A partitioned matrix of the form: . . . : or

o 0 Arr

(!~: A~2 ~ ) is respectively upper or lower block-triangular-

Arl Ar2 Arr to indicate that a partitioned matrix is upper or lower block-triangular (with­out specifying which), the matrix is referred to simply as block-triangular.

characteristic polynomial (and equation) Corresponding to any n x n matrix A is its characteristic polynomial, say p(o), defined (for -00 < ).. < (0) by p()..) = IA - All, and its characteristic equation p()..) = 0 obtained by setting its characteristic polynomial equal to 0; p()..) is a polynomial in).. of degree n and hence is of the form p()..) = Co +CI)..+· .. +Cn_l)..n-1 +cn)..n,

where the coefficients Co, C I, ... , C n -I, C n depend on the elements of A.

Cholesky decomposition The Cholesky decomposition of a symmetric positive definite matrix, say A, is the unique decomposition of the form A = T'T, where T is an upper triangular matrix with positive diagonal elements. More generally, the Cholesky decomposition of an n x n symmetric nonnegative definite matrix, say A, of rank r is the unique decomposition of the form A = T'T, where T is an n x n upper triangular matrix with r positive diagonal elements and n - r null rows.

cofactor (and minor) The cofactor and minor of the ijth element, say aij, of an n x n matrix A are defined in terms of the (n - 1) x (n - 1) sub matrix, say Aij, of A obtained by striking out the ith row and jth column (i.e., the row and column containing aij): the minor of aij is IAij I, and the cofactor is the "signed" minor ( _1)i + j IAij /.

cofactor matrix The cofactor matrix (or matrix of cofactors) of an n x n matrix A = {aij} is the n x n matrix whose ijth element is the cofactor of aij.

column space The column space of an m x n matrix A is the set whose elements consist of all m-dimensional column vectors that are expressible as linear

Some Tenninology XIX

combinations of the n columns of A.

commute Two n x n matrices A and B are said to commute if AB = BA.

commute in pairs n x n matrices, say AI, ... , Ak, are said to commute in pairs if AsAi = AiAs for s > i = 1, ... , k.

consistent A linear system is said to be consistent if it has one or more solutions.

continuous A function f, with domain S in nm x I , is continuous at an interior point c of S if limx~c f(x) = fCc).

continuously differentiable A function f , with domain S in nm x I , is contin­uously differentiable at an interior point c of S if DJ!(c), D2f(c), ... , Dmf(c) exist and are continuous at every point x in some neighborhood of c - a vector or matrix of functions is continuously differentiable at c if all of its elements are continuously differentiable at c.

derivative of a function of a matrix The derivative of a function f of an m x n matrix X = {xij} of mn "independent" variables is the m x n matrix whose ijth element is the partial derivative af/aXij of f with respect to Xij when f is regarded as a function of an mn-dimensional column vector x formed from X by rearranging its elements; the derivative of a function f of an n x n symmetric (but otherwise unrestricted) matrix of variables is the n x n (symmetric) matrix whose ijth element is the partial derivative af/axij or af/aXji of f with respect to Xi} or Xji when f is regarded as a function of an n (n + 1) /2-dimensional column vector x formed from any set of n (n + 1) /2 nonredundant elements of X.

determinant The determinant of an n x n matrix A = {aij} is (by definition) the (scalar-valued) quantity L (_I)<i>n(jl , ... ,}n)alh ... anjn' or equivalently

the quantity L (_I)<i>n(il , ... ,in)aill ... ainn, where iJ, ... , jn or ii, ... , in is a permutation of the first n positive integers and the summation is over all such permutations.

diagonalization An n x n matrix, say A, is said to be diagonalizable if there exists an n x n nonsingular matrix Q such that Q-I AQ is diagonal, in which case Q is said to diagonalize A (or A is said to be diagonalized by Q); a matrix that can be diagonalized by an orthogonal matrix is said to be orthogonally diagonalizable.

diagonalization (simultaneous) k matrices, say A I, ... , Ak, of dimensions n x n, are said to be simultaneously diagonalizable if all k of them can be diag­onalized by the same matrix, that is, if there exists an n x n nonsingular matrix Q such that Q-I Al Q, ... , Q-I AkQ are all diagonal, in which case Q is said to simultaneously diagonalize AI, ... , Ak (or AI, ... , Ak are said to be simultaneously diagonalized by Q).

dimension (of a linear space) The dimension of a linear space V is the number of matrices in a basis for V.

dimension (of a row or column vector) A row or column vector having n ele­ments is said to be of dimension n.

xx Some Tenninology

dimensions (of a matrix) A matrix having m rows and n columns is said to be of dimensions m x n.

direct sum If 2 linear spaces in nm xn are essentially disjoint, their sum is said to be a direct sum.

distance The distance between two matrices A and B in a linear space V is IIA-B II.

dual transformation Corresponding to any linear transformation T from an n­dimensional linear space V into an m-dimensionallinear space W is a linear transformation from W into V called the dual transformation: denoting by X 0 Z the inner product of an arbitrary pair of matrices X and Z in V and by U * Y the inner product of an arbitrary pair of matrices U and Y in W, the dual transformation is the (unique) linear transformation, say S, from W into V such that (for every matrix X in V and every matrix Y in W) XoS(Y) = T(X)*Y; further,forallYinW,S(Y) = 'LJ=I [Y*T(Xj)]Xj, where XI, X2, ... , Xn are any matrices that form an orthonormal basis for V.

duplication matrix The n2 x 11(11 + 1)/2 duplication matrix is the matrix, denoted by the symbol Gn , such that, for every n x n symmetric matrix A, vec A = GnvechA.

eigenspace The eigenspace of an eigenvalue, say A, of an n x n matrix A is the linear space N (A - AI) - with the exception of the n x I null vector, every member of this space is an eigenvector (of A) corresponding to A.

eigenvalues and eigenvectors An eigenvalue of an n x 11 matrix A is (by defini­tion) a scalar (real number), say A, for which there exists an 11 x I vector, say x, such that Ax = AX, or equivalently such that (A - AI)x = 0; any such vector x is referred to as an eigenvector (of A) and is said to belong to (or correspond to) the eigenvalue A - eigenvalues (and eigenvectors), as defined herein, are restricted to real numbers (and vectors of real numbers).

eigenvalues (not necessarily distinct) The characteristic polynomial, say p(o), of an n x n matrix A is expressible as

(-00 < A < (0),

where dl, d2,.'" dm are not-necessarily-distinct scalars and q(o) is a poly­nomial (of degree n - m) that has no real roots; dl, d2, ... , dm are referred to as the not-necessarily-distinct eigenvalues of A or (at the possible risk of con­fusion) simply as the eigenvalues of A - ifthe spectrum of A has k members, say AI, ... , Ab with algebraic multiplicities of YI, ... , Yk, respectively, then m = 'L7=1 Yi, and (for i = I, ... , k) Yi of the m not-necessarily-distinct eigenvalues equal Ai .

essentially disjoint Two subspaces, say U and V, of nmxn are (by definition) essentially disjoint if U n V = {OJ, i.e., if the only matrix they have in common is the (m x n) null matrix-note that every subspace of nm xn

contains the (m x n) null matrix, so that no two subspaces can be entirely disjoint.

Some Terminology XXI

full column rank An m x n matrix A is said to have full column rank if rank(A) =n.

full row rank An m x n matrix A is said to have full row rank ifrank(A) = m.

generalized eigenvalue problem The generalized eigenvalue problem consists of finding, for a symmetric matrix A and a symmetric positive definite matrix B, the roots of the polynomial IA - ABI (i.e., the solutions for A to the equation IA - ABI = 0).

generalized inverse A generalized inverse of an m x n matrix A is any n x m matrix G such that AGA = A - if A is nonsingular, its only generalized inverse is A-I; otherwise, it has infinitely many generalized inverses.

geometric multiplicity The geometric multiplicity of an eigenvalue, say A, of an n x n matrix A is (by definition) dim[N(A - AI)] (i.e., the dimension of the eigenspace of A).

gradient (or gradient matrix) The gradient of a vector f = (h, ... , Ip)' of functions, each of whose domain is a set in nm xl, is the m x p matrix [(DI!)', ... , (Dip)'], whose jith element is Ddi-the gradient off is the transpose of the Jacobian matrix of f.

gradient vector The gradient vector of a function I, with domain in nmx1 , is the m-dimensional column vector (D f)', whose jth element is the partial derivative D j I of I

Hessian matrix The Hessian matrix of a function I, with domain in nm xl, is the m x m matrix whose ijth element is the ijth partial derivative D;j I of I

homogeneous linear system A linear system (in a matrix X) of the fonn AX = 0; i.e., a linear system whose right side is a null matrix.

idempotent A (square) matrix A is idempotent if A2 = A.

identity transformation An identity transfonnation is a transfonnation from a linear space V onto V defined by T (X) = X.

indefinite A square (symmetric or nonsymmetric) matrix or a quadratic fonn is (by definition) indefinite if it is neither nonnegative definite nor non positive definite-thus, an n x n matrix A and the quadratic fonn x' Ax (in an n x 1 vector x) are indefinite if x' Ax < ° for some x and x' Ax > ° for some (other) x.

inner product The inner product A • B of an arbitrary pair of matrices A and B in a linear space V is the value assigned to A and B by a designated function having the following 4 properties: (1) A· B = B· A; (2) A· A :::: 0, with equality holding if and only if A = 0; (3) (kA) • B = k(A· B) (where k is an arbitrary scalar); (4) (A + B) • C = (A· C) + (B· C) (where C is an arbitrary matrix in V)-the quasi-inner product A· B is defined in the same way as the inner product except that Property (2) is replaced by the weaker property (2') A· A :::: 0, with equality holding if A = O.

inner product (usual) The usual inner product of a pair of matrices A and B in a linear space is tr(A'B) (which in the special case of a pair of column vectors

xxii Some Tenninology

a and b reduces to a'b).

interior point A matrix, say X, in a set S of m x n matrices is an interior point of S if there exists a neighborhood, say N, of X such that N C S.

intersection The intersection of 2 sets, say U and V, of m x n matrices is the set comprising all matrices that are contained in both U and V; more generally, the intersection of k sets, say UI, ... ,Uk. of m x n matrices is the set comprising all matrices that are contained in every one of UI, ... , Uk.

invariant subspace A subspace U of the linear space R n x I is said to be invariant relative to an n x n matrix A if, for every vector x in U, the vector Ax is also in U; a subspace U of an n-dimensional linear space V is said to be invariant relative to a linear transformation T from V into V if T (U) C U, that is, if the image T (U) of U is a subspace of U itself.

inverse (matrix) A matrix B that is both a right and left inverse of a matrix A (so that AB = I and BA = I) is called an inverse of A.

inverse (transformation) The inverse of an invertible transformation T from a linear space V into a linear space W is the transformation from W into V that assigns to each matrix Y in W the (unique) matrix X (in V) such that T(X) = Y.

invertible (matrix) A matrix that has an inverse is said to be invertible-a matrix is invertible if and only if it is nonsingular.

invertible (transformation) A transformation from a linear space V into a linear space W is (by definition) invertible if it is both 1-1 and onto.

involutory A (square) matrix A is involutory if A 2 = I, i.e., if it is invertible and is its own inverse.

isomorphic If there exists a 1-1 linear transformation, say T, from a linear space V onto a linear space W, then V and W are said to be isomorphic, and T is said to be an isomorphism of V onto W.

Jacobian matrix The Jacobian matrix of a p-dimensional vector f = (II, ... , Ip)' of functions, each of whose domain is a set in R m x I, is the p x m matrix (Dlf, ... , Dmf), whose ijth element is D j Ii - in the special case where p = m, the determinant of this matrix is referred to as the Jacobian (or Jacobian determinant) of f.

Kronecker product The Kronecker product of two matrices, sayan m x n matrix A = {aij} and a p x q matrix B, is the mp x nq matrix

(:~~: :~~: am:IB am:2B

:~:) amnB

obtained by replacing each element aij of A with the p x q matrix aijB­the Kronecker-product operation is associative [for any 3 matrices A, B, and C, A ® (B ® C) = (A ® B) ® C], so that the notion of a Kronecker product extends in an unambiguous way to 3 or more matrices.

Some Tenninology xxiii

k times continuously differentiable A function j, with domain S in nmxI , is k times continuously differentiable at an interior point c of S if it and all of its first- through (k - 1)th-order partial derivatives are continuously dif­ferentiable at c or, equivalently, if all of the first- through kth-order partial derivatives of j exist and are continuous at every point in some neighborhood of c - a vector or matrix of functions is k times continuously differentiable at c if all of its elements are k times continuously differentiable at c.

LDU decomposition An LDU decomposition of a square matrix, say A, is a decomposition of the form A = LDU, where L is a unit lower triangular matrix, D a diagonal matrix, and U an upper triangular matrix.

least squares generalized inverse A generalized inverse, say G, of an m x n matrix A is said to be a least squares generalized inverse (of A) if (AG)' = AG; or, equivalently, an n x m matrix is a least squares generalized inverse of A if it satisfies Moore-Penrose Conditions (1) and (3).

left inverse A left inverse of an m x n matrix A is an n x m matrix L such that LA = In - a matrix has a left inverse if and only if it has full column rank.

linear dependence or independence A nonempty (but finite) set of matrices (of the same dimensions), say A I, A2, ... , Ak, is (by definition) linearly depen-dent if there exist scalars Xl, X2, ... ,Xk, not all 0, such that E~=I XiAi = 0; otherwise (if no such scalars exist), the set is linearly independent-by con­vention, the empty set is linearly independent.

linear space The use of this term is confined (herein) to sets of matrices (all of which have the same dimensions). A nonempty set, say V, is called a linear space if: (1) for every matrix A in V and every matrix B in V, the sum A + B is in V; and (2) for every matrix A in V and every scalar k, the product kA is in V.

linear system A linear system is (for some positive integers m, n, and p) a set of mp simultaneous equations expressible in nonmatrix form as EJ=I aijXjk

= bik (i = i, ... , m; k = 1, ... , p), or in matrix form as AX = B, where A = {aij} is an m x n matrix comprising the "coefficients", X = {Xjk} is an n x p matrix comprising the "unknowns", and B = {bik} is an m x p matrix comprising the "right (hand) sides"-A is referred to as the coefficient matrix and B as the right side of AX = B; and to emphasize that X comprises the unknowns, AX = B is referred to as a linear system in X.

linear transformation A transformation, say T, from a linear space V (of m x n matrices) into a linear space W (of p x q matrices) is said to be linear if it satisfies the following two conditions: (1) for all X and Z in V, T(X + Z) = T(X) + T(Z); and (2) for every scalar c and for all X in V, T(cX) = c T (X) - in the special case where W = n, it is customary to refer to a linear transformation from V into W as a linear functional on V.

matrix The use of the term matrix is confined (herein) to real matrices, i.e., to rectangular arrarys of real numbers.

matrix representation The matrix representation of a linear transformation from

XXIV Some Terminology

an n-dimensional linear space V, with a basis B comprising matrices V I, V 2, ... , V n, into a linear space W, with a basis C comprising matrices WI, W2 , ••• , W m, is the m x 11 matrix A = {aij} whose jth column is (for j = 1, 2, ... , n) uniquely determined by the equality

T(Vj) = aljWI + a2jW2 + ... + amjWm ;

this matrix (which depends on the choice of B and C) is such that if x = {x j }

is the 11 x I vector that comprises the coordinates of a matrix V (in V) in terms of the basis B (i.e., V = L j x j V j), then the m x I vector y = {Yi} given by the formula y = Ax comprises the coordinates of T (V) in terms of the basis C [i.e., T (V) = Li )'i WiJ.

minimum norm generalized inverse A generalized inverse, say G, of an m x 11

matrix A is said to be a minimum norm generalized inverse (of A) if (GA)' = GA; or, equivalently, an 11 x m matrix is a minimum norm generalized inverse of A if it satisfies Moore-Penrose Conditions (1) and (4).

Moore-Penrose inverse (and conditions) Corresponding to any m x 11 matrix A, there is a unique 11 x m matrix, say G, such that (1) AGA = A (i.e., G is a generalized inverse of A), (2) GAG = G (i.e., A is a generalized inverse of G), (3) (AG)' = AG (i.e., AG is symmetric), and (4) (GA)' = GA (i.e., GA is symmetric). This matrix is called the Moore-Penrose inverse (or pseudoinverse) of A, and the four conditions that (in combination) define this matrix are referred to as Moore-Penrose (or Penrose) Conditions (1) -(4).

negative definite An n x 11 (symmetric or nonsymmetric) matrix A and the quad­ratic form x' Ax (in an 11 x I vector x) are (by definition) negative definite if -x' Ax is a positive definite quadratic form (or equivalently if -A is a pos­itive definite matrix)-thus, A and x' Ax are negative definite if x' Ax < 0 for every nonnull x in nn.

negative or positive pair Any pair of elements of an 11 x n matrix A = {aij) that do not lie either in the same row orthe same column, say aij and ai'j' (where i' :j::. i and j' :j::. j) is (by definition) either a negative pair or a positive pair: it is a negative pair if one of the elements is located above and to the right of the other, or equivalently if either i' > i and j' < j or ;' < ; and j' > j; otherwise (if one of the elements is located above and to the left of the other, or equivalently if either;' > ; and j' > j or;' < ; and j' < j), it is a positive pair-note that whether a pair of elements is a negative pair or a positive pair is completely determined by the elements' relative locations and has nothing to do with whether the numerical values of the elements are positive or negative.

negative semidefinite An 11 x 11 (symmetric or nonsymmetric) matrix A and the quadratic form x' Ax (in an 11 x I vector x) are (by definition) negative semidefinite if -x' Ax is a positive semidefinite quadratic form (or equiva­lently if -A is a positive semidefinite matrix)-thus, A and x' Ax are nega­tive semidefinite if they are non positive definite but not negative definite, or equivalently if x' Ax ::::: 0 for every x in nn with equality holding for some

Some Tenninology xxv

nonnull x.

neighborhood A neighborhood of an m x n matrix C is a set of the general form {X E nmxn: IIX-CII < r}, wherer is a positive number called the radius of the neighborhood (and where the norm is the usual norm).

nonhomogeneous linear system A linear system whose right side (which is a column vector or more generally a matrix) is nonnull.

nonnegative definite An n x n (symmetric or nonsymmetric) matrix A and the quadratic form x' Ax (in an n x 1 vector x) are (by definition) nonnegative definite if x' Ax :::: 0 for every x in nn.

nonpositive definite An n x n (symmetric or nonsymmetric) matrix A and the quadratic form x' Ax (in an n x 1 vector x) are (by definition) nonpositive definite if -x' Ax is a nonnegative definite quadratic form (or equivalently if -A is a nonnegative definite matrix)-thus, A and x' Ax are nonpositive definite if x' Ax :::: 0 for every x in nn.

nonnull matrix A matrix having 1 or more nonzero elements.

nonsingular A matrix is nonsingular if it has both full row rank and full column rank or equivalently if it is square and its rank equals its order.

norm The norm of a matrix A in a linear space V is (A· A)1/2-the use of this term is limited herein to norms defined in terms of an inner product; in the case of a quasi-inner product, (A· A) 1/2 is referred to as the quasi norm.

normal equations A linear system (or the equations comprising the linear system) of the form X/Xb = X' y (in a p x 1 vector b), where X is an n x p matrix and y an n x 1 vector.

null matrix A matrix all of whose elements are o. null space (of a matrix) The null space of an m x n matrix A is the solution

space of the homogeneous linear system Ax = 0 (in an n-dimensional column vector x), or equivalently is the set {x E nnxl : Ax = O} .

null space (of a transformation) The null space-also known as the kernel- of a linear transformation T from a linear space V into a linear space W is the set (X E V : T(X) = O}, which is a subspace of V.

one to one A transformation T from a set V into a set W is said to be 1-1 (one to one) if each member of the range of T is the image of only one member of V.

onto A transformation T from a set V into a set W is said to be onto if T (V) = W (i.e., if the range of T is all of W), in which case T may be referred to as a transformation from V onto W.

open set A set S of m x n matrices is an open set if every matrix in S is an interior point of S.

order A (square) matrix of dimensions n x n is said to be of order n.

orthogonal complement The orthogonal complement of a subspace U of a linear space V is the set comprising all matrices in V that are orthogonal to U -note that the orthogonal complement of U depends on V as well as U (and

xxvi Some Terminology

also on the choice of inner product).

orthogonality of a matrix and a subspace A matrix Y in a linear space V is or­thogonal to a subspace U (of V) if Y is orthogonal to every matrix in U.

orthogonality of two subspaces A subspace U of a linear space V is orthogonal to a subspace W (of V) if every matrix in U is orthogonal to every matrix inW.

orthogonality with respect to a matrix For any n x n symmetric nonnegative definite matrix W, two n x 1 vectors, say x and y, are said to be orthogonal with respect to W if x'Wy = 0; an n x I vector, say x, and a subspace, say U, of nn x I are said to be orthogonal with respect to W if x'Wy = 0 for every y in U; and two subspaces, say U and V, of nnx I are said to be orthogonal with respect to W if x'Wy = 0 for every x in U and every y in V.

orthogonal matrix A (square) matrix A is orthogonal if A' A = AA' = I. orthogonal set A finite set of matrices in a linear space V is orthogonal if the

inner product of every pair of matrices in the set equals O.

orthonormal set A finite set of matrices in a linear space V is orthonormal if it is orthogonal and if the norm of every matrix in the set equals 1.

(

All A12 . .. Ale) A21 A22 ... A2e

partitioned matrix A partitioned matrix, say : : ' is a ma-

Arl Ar2 Are trix that has (for some positive integers r and c) been subdivided into rc sub­matrices Aij (i = 1,2, ... , r; j = 1,2, ... , c), called blocks, by implicitly superimposing on the matrix r - 1 horizontal lines and c -1 vertical lines (so that all of the blocks in the same "row" of blocks have the same number of rows and all of those in the same "column" of blocks have the same number of columns )-in the special case where c = r, the blocks A II, A22, ... , Arr

are referred to as the diagonal blocks (and the other blocks are referred to as the off-diagonal blocks).

permutation matrix An n x n permutation matrix is a matrix that is obtainable from the n x n identity matrix by permuting its columns; i.e., a matrix of the form (Ukl ' Ukz, ... , Ukn ), where U I, U2, ... , Un are respectively the first, second, ... , nth columns of In and where kl, k2, ... , kn is a permutation of the first n positive integers.

positive definite An n x n (symmetric or nonsymmetric) matrix A and the quad­ratic form x' Ax (in an n x 1 vector x) are (by definition) positive definite if x' Ax > 0 for every nonnull x in nn.

positive semidefinite An n x n (symmetric or nonsymmetric) matrix A and the quadratic form x' Ax (in an n x I vectorx) are (by definition) positive semidef­inite if they are nonnegative definite but not positive definite, or equivalently if x' Ax ::: 0 for every x in nn with equality holding for some nonnull x.

Some Terminology xxvii

principal submatrix A submatrix of a square matrix is a principal submatrix if it can be obtained by striking out the same rows as columns (so that the ith row is struck out whenever the ith column is struck out, and vice versa); the r x r (principal) submatrix of an n x n matrix obtained by striking out its last n - r rows and columns is referred to as a leading principal submatrix (r = I, ... , n).

product (of transformations) The product (or composition) of a transformation, say T, from a linear space V into a linear space Wand a transformation, say S, from a linear space U into V is the transformation from U into W that assigns to each matrix X in U the matrix T[S(X)] (in W)-the definition of the term product (or composition) extends in a straightforward way to three or more transformations.

projection (orthogonal) The projection-also known as the orthogonal projec­tion-of a matrix Y in a linear space V on a subspace U (of V) is the unique matrix, say Z, in U such that Y - Z is orthogonal to U; in the special case where (for some positive integer n and for some symmetric positive definite matrix W) V = nnxl and the inner product is the bilinear form x'Wy, the projection of y (an n x 1 vector) on U is referred to as the projection of y on U with respect to W - this terminology can be extended to a symmetric nonnegative definite matrix W by defining a projection of yon U with respect to W to be any vector z in U such that (y - z) -Lw U.

projection along a subspace For a linear space V of m x n matrices and for subspaces U and W such that U EB W = V (essentially disjoint subspaces whose sum is V), the projection of a matrix in V, say the matrix Y, on U along W is (by definition) the (unique) matrix Z in U such that Y - Z E W.

projection matrix (orthogonal) The projection matrix-also known as the or­thogonal projection matrix-for a subspaceU ofnnx1 is the unique (n x n) matrix, say A, such that, for every n x 1 vector y, Ay is the projection (with respect to the usual inner product) of yon U - simply saying that a matrix is a projection matrix means that there is some subspace of nnx 1 for which it is the projection matrix.

projection matrix (general orthogonal) The (orthogonal) projection matrix for a subspace U of nnx 1 with respect to an n x n symmetric positive definite matrix W is the unique (n x n) matrix, say A, such that, for every n x 1 vector y, Ay is the projection of y on U with respect to W - simply saying that a matrix is a projection matrix with respect to W means that there is some subspace of nnxl for which it is the projection matrix with respect to W- more generally, a projection matrix for U with respect to an n x n symmetric nonnegative definite matrix W is an (n x n) matrix, say A, such that, for every n x 1 vector y, Ay is a projection of y on U with respect to W.

projection matrix for one subspace along another For subspaces U and W (of nn xl) such that U EB W = nn x 1 (essentially disjoint subspaces whose sum is nnx 1), the projection matrix for U along W is the (unique) n x n matrix,

xxviii Some Terminology

say A, such that for every n x I vector y, Ay is the projection of yon U along W.

QR decomposition The QR decomposition of a matrix offull column rank, sayan m x k matrix A of rank k, is the unique decomposition of the form A = QR, where Q is an m x k matrix whose columns are orthonormal (with respect to the usual inner product) and R is a k x k upper triangular matrix with positive diagonal elements.

quadratic form A quadratic form in an n x I vector x = (XI, ... , xn)' is a function of x (defined for x E Rn) that, for some n x n matrix A = {aij}, is expressible as x' Ax = Li.j aijXiX j - the matrix A is called the matrix of the quadratic form and, unless n = I or the choice for A is restricted (e.g., to symmetric matrices), is nonunique.

range The range of a transformation T from a set V into a set W is the set T (V) (i.e., the image of the domain of T)-in the special case of a linear transformation from a linear space V into a linear space W, the range T(V) of T is a linear space and is referred to as the range space of T.

rank (of a linear transformation) The rank of a linear transformation T from a linear space V into a linear space W is (by definition) the dimension dim[T(V)] of the range space T(V) of T.

rank (of a matrix) The rank of a matrix A is the dimension of C(A) or equiva­lently of R(A).

rank additivity Two matrices A and B (of the same size) are said to be rank additive if rank(A + B) = rank(A) + rank(B); more generally, k ma­trices AI, A2 .... , Ak (of the same size) are said to be rank additive if rank(L7=1 Ai) = L7=1 rank(Ai) (i.e., if the rank of their sum equals the sum of their ranks).

reflexive generalized inverse A generalized inverse, say G, of an m x n matrix A is said to be reflexive if GAG = G; or, equivalently, an n x m matrix is a reflexive generalized inverse of A if it satisfies Moore-Penrose Conditions (1) and (2).

restriction If T is a linear transformation from a linear space V into a linear space Wand if U is a subspace of V, then the transformation, say R, from U into W defined by R(X) = T(X) (which assigns to each matrix in U the same matrix in W assigned by T) is called the restriction of T to U.

right inverse A right inverse of an m x n matrix A is an n x m matrix R such that AR = 1m - a matrix has a right inverse if and only if it has full row rank.

row space The row space of an m x n matrix A is the set whose elements consist of all n-dimensional row vectors that are expressible as linear combinations of the m rows of A.

scalar The term scalar is (herein) used interchangeably with real number.

scalar mUltiple (of a transformation) The scalar multiple of a scalar k and a transformation, say T, from a linear space V into a linear space W is the

Some Tenninology xxix

transformation from V into W that assigns to each matrix X in V the matrix kT (X) (in W).

Schur complement In connection with a partitioned matrix A of the form A =

(~ ~) or A = (~ ~), the matrix Q = W - VT-U is referred to

as the Schur complement of T in A relative to T- or (especially in a case where Q is invariant to the choice of the generalized inverse T-) simply as the Schur complement of T in A or (in the absence of any ambiguity) even more simply as the Schur complement of T.

second-degree polynomial A second-degree polynomial in an n x 1 vector x = (Xl, ... , xn)' is a function, say f(x), of x that is defined for all x in nn

and that, for some scalar c, some n x I vector b = {bi}, and some n x n

matrix V = {vi}}, is expressible as f(x) = c - 2b'x +x'Vx, or in nonmatrix notation as f(x) = c - 2 I:7=1 biXi + I:7=1 I:J=1 VijXiXj - in the special case where c = 0 and V = 0, f(x) = -2b'x, which is a linear form (in x), and in the special case where c = 0 and b = 0, f(x) = x'vx, which is a quadratic form (in x).

similar An n x n matrix B is said to be similar to an n x n matrix A if there exists an n x n nonsingular matrix C such that B = C- I AC or, equivalently, such that CB = AC - if B is similar to A, then A is similar to B.

singular A square matrix is singular if its rank is less than its order.

singular value decomposition An m x n matrix A of rank r is expressible as

A = p(~l ~)Q' = PIDIQ'I = t siPiq; = t ajUj ,

where Q = (ql, ... , qn) is an n x n orthogonal matrix and Dl = diag(sl,

... , sr) an r x r diagonal matrix such that Q' A' AQ = (~i ~), where

Sl, ... , Sr are (strictly) positive, where QI = (ql, ... , qr)' PI = (PI, ... , Pr) = AQIDjl, and, for any m x (m - r) matrix P2 such that ~P2 = 0, P = (PI, P2), where ai, ... ,ak are the distinct values represented among SI, ... ,Sr, and where (for j = 1, ... ,k)Uj = I:{i:si=aj}Piq;; any of these four representations may be referred to as the singular value decom­position of A, and Sl, ... , Sr are referred to as the singular values of A -SI, ... , Sr are the positive square roots of the nonzero eigenvalues of A' A (or equivalently AA'), ql' ... , qn are eigenvectors of A' A, and the columns of P are eigenvectors of AA'.

skew-symmetric An n x n matrix, say A = {aij}, is (by definition) skew-sym­metric if A' = -A; that is, if aji = -aij for all i and j (or equivalently if au =Ofori = l, ... ,nandaji = -aij for j =l=i = l, ... ,n).

solution A matrix, say X*, is said to be a solution to a linear system AX = B (in X) ifAX* = B.

solution set or space The collection of all solutions to a linear system AX = B (in X) is called the solution set of the linear system; in the special case of

xxx Some Tenninology

a homogeneous linear system AX = 0, the solution set may be called the solution space.

span The span of a finite set of matrices (having the same dimensions) is defined as follows: the span of a finite nonempty set {A I, ... , Ak J is the set consisting of all matrices that are expressible as linear combinations of AI, ... , Ak. and the span of the empty set is the set (OJ, whose only element is the null matrix. And, a finite set S of matrices in a linear space V is said to span V ifsp(S) = V.

spectral decomposition An n x n symmetric matrix A is expressible as

n k

A = QDQ' = L diqiq; = L AjE) , i=1 )=1

where dl, ... , dn are the not-necessarily-distinct eigenvalues of A, ql' ... , ~ are orthonormal eigenvectors corresponding to dl, ... , dn , respectively, Q = (ql' ... ' ~), D = diag(dl, ... , dn ), V'I, ... , Ad is the spectrum of A, and (for j = 1, ... , k) Ej = Eli :dj=).j} qiq;; any of these three representations may be referred to as the spectral decomposition of A.

spectrum The spectrum of an n x n matrix A is the set whose members are the distinct (different) scalars that are eigenvalues of A.

subspace A subspace of a linear space V is a subset of V that is itself a linear space.

sum (of sets) The sum of 2 nonempty sets, say U and V, of m x n matrices is the set {A + B : A E U, B E VJ comprising every (m x n) matrix that is expressible as the sum of a matrix in U and a matrix in V; more generally, the sum of k sets, say UI, ... , Uk. of m x n matrices is the set (E7=IAi : Al eUI, ... ,Ak eUd·

sum (oftransformations) The sum oftwo transformations, say T and S, from a linear space V into a linear space W is the transformation from V into W that assigns to each matrix X in V the matrix T(X) + S(X) (in W)-since the addition of transformations is associative, the definition of the term sum extends in an unambiguous way to three or more transformations.

symmetric A matrix, say A, is symmetric if A' = A, or equivalently if it is square and (for every i and j) its ijth element equals its jith element.

trace The trace of a (square) matrix is the sum of its diagonal elements.

transformation A transformation (also known as a function, operator, map, or mapping), say T, from a set V, called the domain, into a set W is a corre­spondence that assigns to each member X of V a unique member of W; the member of W assigned to X is denoted by the symbol T (X) and is referred to as the image of X, and, for any subset U of V, the set of all members of W that are the images of one or more members of U is denoted by the symbol T (U) and is referred to as the image of U - V and W consist of scalars, row or column vectors, matrices, or other "objects".

transpose The transpose of an m x n matrix A is the n x m matrix whose ijth

Some Tenninology xxxi

element is the jith element of A.

union The union of 2 sets, say U and V, of m x n matrices is the set comprising all matrices that belong to either or both of U and V; more generally, the union of k sets, say UI, ... ,Uk, of m x n matrices comprises all matrices that belong to at least one of UI, ... , Uk.

unit (upper or lower) triangular matrix A unit triangular matrix is a triangular matrix all of whose diagonal elements equal one.

U'DU decomposition A U'DU decomposition of a symmetric matrix, say A, is a decomposition of the form A = U'DU, where U is a unit upper triangular matrix and D is a diagonal matrix.

Vandermonde matrix A Vandermonde matrix is a matrix of the general form

(

1 Xl Xf ... X~_l) 1 2 n-l

X2 X2 ••. X2 : :: : (where Xl, X2, ..• , xn are arbitrary scalars)

1 X x2 x n- I n n n vee The vee of an m x n matrix A = (aI, a2, ... , an) is the mn-dimensional

(column) vee,,,, (:) obtained by succe"ively stacking the firs~ second,

... , nth columns of A one under the other.

vech The vech of an n x n matrix A = {aij} is the n(n + 1)j2-dimensional

(column) vee'or (1} where (fo, i = 1.2 •...• n).; = (aii. ai+l.i.···.

and is the subvector of the i th column of A obtained by striking out its first i-I elements.

vee-permutation matrix The mn x mn vee-permutation matrix is the unique permutation matrix, denoted by the symbol Kmn , such that, for every m x n matrix A, vec(A') = Kmn vee (A) - the vee-permutation matrix is also known as the commutation matrix.

zero transformation The linear transformation from a linear space V into a linear space W that assigns to every matrix in V the null matrix (in W) is called the zero transformation.