Universitext

Universitext

Series Editors: Sheldon Axler San Francisco State University Vincenzo Capasso Università degli Studi di Milano Carles Casacuberta Universitat de Barcelona Angus J. MacIntyre Queen Mary, University of London Kenneth Ribet University of California, Berkeley Claude Sabbah CNRS, École Polytechnique Endre Süli University of Oxford Wojbor A. Woyczynski Case Western Reserve University Universitext is a series of textbooks that presents material from a wide variety of mathematical disciplines at master’s level and beyond. The books, often well class-tested by their author, may have an informal, personal even experimental approach to their subject matter. Some of the most successful and established books in the series have evolved through several editions, always following the evolution of teaching curricula, to very polished texts. Thus as research topics trickle down into graduate-level teaching, first textbooks written for new, cutting-edge courses may make their way into Universitext.

For further volumes: http://www.springer.com/series/223

Fuzhen Zhang

Matrix TheoryBasic Results and Techniques

Second Edition

Linear Park, Davie, Florida, USA

Division of Math, Science, and Technology

USA

ISBN 978-1-4614-1098-0 e-IS BN 978-1 -4614-1099-7 DOI 10.1007/978-1-4614-1099-7 Springer New York Dordrecht Heidelberg London

Mathematics Subject Classification (2010): 15-xx, 47-xx

Library of Congress Control Number: 2011935372

Fuzhen Zhang

Nova Southeastern UniversityFort Lauderdale, FL 33314

zhang@nova.edu

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.springer.com)

© Springer Science+Business Media, LLC 2011 All rights reserved. This work may not be translated or copied in whole or in part without the writtenpermission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use inconnection with any form of information storage and retrieval, electronic adaptation, computer software,or by similar or dissimilar methodology now known or hereafter developed is forbidden.The use in this publication of trade names, trademarks, service marks, and similar terms, even if theyare not identified as such, is not to be taken as an expression of opinion as to whether or not they aresubject to proprietary rights.

To Cheng, Sunny, Andrew, and Alan

Preface to the Second Edition

The first edition of this book appeared a decade ago. This is a revisedexpanded version. My goal has remained the same: to provide a text fora second course in matrix theory and linear algebra accessible to advancedundergraduate and beginning graduate students. Through the course, stu-dents learn, practice, and master basic matrix results and techniques (ormatrix kung fu) that are useful for applications in various fields such asmathematics, statistics, physics, computer science, and engineering, etc.

Major changes for the new edition are: eliminated errors, typos, andmistakes found in the first edition; expanded with topics such as matrixfunctions, nonnegative matrices, and (unitarily invariant) matrix norms;included more than 1000 exercise problems; rearranged some material fromthe previous version to form a new chapter, Chapter 4, which now containsnumerical ranges and radii, matrix norms, and special operations such asthe Kronecker and Hadamard products and compound matrices; and addeda new chapter, Chapter 10, “Majorization and Matrix Inequalities”, whichpresents a variety of inequalities on the eigenvalues and singular values ofmatrices and unitarily invariant norms.

I am thankful to many mathematicians who have sent me their com-ments on the first edition of the book or reviewed the manuscript of thisedition: Liangjun Bai, Jane Day, Farid O. Farid, Takayuki Furuta, GeoffreyGoodson, Roger Horn, Zejun Huang, Minghua Lin, Dennis Merino, GeorgeP.H. Styan, Gotz Trenkler, Qingwen Wang, Yimin Wei, Changqing Xu, HuYang, Xingzhi Zhan, Xiaodong Zhang, and Xiuping Zhang. I also thankFarquhar College of Arts and Sciences at Nova Southeastern University forproviding released time for me to work on this project.

Readers are welcome to communicate with me via e-mail.

vii

Fuzhen ZhangFort Lauderdale

May 23, 2011zhang@nova.edu

www.nova.edu/˜zhang

Preface

It has been my goal to write a concise book that contains fundamen-tal ideas, results, and techniques in linear algebra and (mainly) in matrixtheory which are accessible to general readers with an elementary linearalgebra background. I hope this book serves the purpose.

Having been studied for more than a century, linear algebra is of centralimportance to all fields of mathematics. Matrix theory is widely used ina variety of areas including applied math, computer science, economics,engineering, operations research, statistics, and others.

Modern work in matrix theory is not confined to either linear or alge-braic techniques. The subject has a great deal of interaction with combina-torics, group theory, graph theory, operator theory, and other mathematicaldisciplines. Matrix theory is still one of the richest branches of mathematics;some intriguing problems in the field were long standing, such as the Vander Waerden conjecture (1926–1980), and some, such as the permanental-dominance conjecture (since 1966), are still open.

This book contains eight chapters covering various topics from sim-ilarity and special types of matrices to Schur complements and matrixnormality. Each chapter focuses on the results, techniques, and methodsthat are beautiful, interesting, and representative, followed by carefully se-lected problems. Many theorems are given different proofs. The materialis treated primarily by matrix approaches and reflects the author’s tastes.

The book can be used as a text or a supplement for a linear algebraor matrix theory class or seminar. A one-semester course may consist ofthe first four chapters plus any other chapter(s) or section(s). The onlyprerequisites are a decent background in elementary linear algebra andcalculus (continuity, derivative, and compactness in a few places). Thebook can also serve as a reference for researchers and instructors.

The author has benefited from numerous books and journals, includingThe American Mathematical Monthly, Linear Algebra and Its Applications,Linear and Multilinear Algebra, and the International Linear Algebra Soci-ety (ILAS) Bulletin Image. This book would not exist without the earlierworks of a great number of authors (see the References).

I am grateful to the following professors for many valuable suggestionsand input and for carefully reading the manuscript so that many errorshave been eliminated from the earlier version of the book:

Professor R.B. Bapat (Indian Statistical Institute),Professor L. Elsner (University of Bielefeld),Professor R.A. Horn (University of Utah),

ix

x Preface

Professor T.-G. Lei (National Natural Science Foundation of China),Professor J.-S. Li (University of Science and Technology of China),Professor R.-C. Li (University of Kentucky),Professor Z.-S. Li (Georgia State University),Professor D. Simon (Nova Southeastern University),Professor G.P.H. Styan (McGill University),Professor B.-Y. Wang (Beijing Normal University), andProfessor X.-P. Zhang (Beijing Normal University).

F. ZhangFt. LauderdaleMarch 5, 1999

zhang@nova.eduwww.nova.edu/˜zhang

Contents

Preface to the Second Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixFrequently Used Notation and Terminology . . . . . . . . . . . . . . . . . . . .xvFrequently Used Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii

1 Elementary Linear Algebra Review 1

1.1 Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Matrices and Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.3 Linear Transformations and Eigenvalues . . . . . . . . . . . . . . . . . . . . 171.4 Inner Product Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2 Partitioned Matrices, Rank, and Eigenvalues 35

2.1 Elementary Operations of Partitioned Matrices . . . . . . . . . . . . . 352.2 The Determinant and Inverse of Partitioned Matrices . . . . . . .422.3 The Rank of Product and Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512.4 The Eigenvalues of AB and BA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572.5 The Continuity Argument and Matrix Functions . . . . . . . . . . . 622.6 Localization of Eigenvalues: The Gersgorin Theorem . . . . . . . 67

3 Matrix Polynomials and Canonical Forms 73

3.1 Commuting Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 733.2 Matrix Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .793.3 Annihilating Polynomials of Matrices . . . . . . . . . . . . . . . . . . . . . . . 873.4 Jordan Canonical Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 933.5 The Matrices AT , A, A∗, ATA, A∗A, and AA . . . . . . . . . . . . 102

4 Numerical Ranges, Matrix Norms, and Special Operations 107

4.1 Numerical Range and Radius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1074.2 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1134.3 The Kronecker and Hadamard Products . . . . . . . . . . . . . . . . . . .1174.4 Compound Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

xi

xii Contents

5 Special Types of Matrices 125

5.1 Idempotence, Nilpotence, Involution, and Projections . . . . . .1255.2 Tridiagonal Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1335.3 Circulant Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1385.4 Vandermonde Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1435.5 Hadamard Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1505.6 Permutation and Doubly Stochastic Matrices . . . . . . . . . . . . . .1555.7 Nonnegative Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

6 Unitary Matrices and Contractions 171

6.1 Properties of Unitary Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1716.2 Real Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1776.3 Metric Space and Contractions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1826.4 Contractions and Unitary Matrices . . . . . . . . . . . . . . . . . . . . . . . . 1886.5 The Unitary Similarity of Real Matrices . . . . . . . . . . . . . . . . . . . 1926.6 A Trace Inequality of Unitary Matrices . . . . . . . . . . . . . . . . . . . .195

7 Positive Semidefinite Matrices 199

7.1 Positive Semidefinite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1997.2 A Pair of Positive Semidefinite Matrices . . . . . . . . . . . . . . . . . . . 2077.3 Partitioned Positive Semidefinite Matrices . . . . . . . . . . . . . . . . . 2177.4 Schur Complements and Determinant Inequalities . . . . . . . . . 2277.5 The Kronecker and Hadamard Products

of Positive Semidefinite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 2347.6 Schur Complements and the Hadamard Product . . . . . . . . . . .2407.7 The Wielandt and Kantorovich Inequalities . . . . . . . . . . . . . . . 245

8 Hermitian Matrices 253

8.1 Hermitian Matrices and Their Inertias . . . . . . . . . . . . . . . . . . . . 2538.2 The Product of Hermitian Matrices . . . . . . . . . . . . . . . . . . . . . . . 2608.3 The Min-Max Theorem and Interlacing Theorem . . . . . . . . . . 2668.4 Eigenvalue and Singular Value Inequalities . . . . . . . . . . . . . . . . 2748.5 Eigenvalues of Hermitian matrices A, B, and A+B . . . . . . . 2818.6 A Triangle Inequality for the Matrix (A∗A)1/2 . . . . . . . . . . . . 287

9 Normal Matrices 293

9.1 Equivalent Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2939.2 Normal Matrices with Zero and One Entries . . . . . . . . . . . . . . .3069.3 Normality and Cauchy–Schwarz–Type Inequality . . . . . . . . . . 3129.4 Normal Matrix Perturbation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319

Contents xiii

10 Majorization and Matrix Inequalities 325

10.1 Basic Properties of Majorization . . . . . . . . . . . . . . . . . . . . . . . . 32510.2 Majorization and Stochastic Matrices . . . . . . . . . . . . . . . . . . . 33410.3 Majorization and Convex Functions . . . . . . . . . . . . . . . . . . . . . 34010.4 Majorization of Diagonal Entries, Eigenvalues,

and Singular Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34910.5 Majorization for Matrix Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . 35610.6 Majorization for Matrix Product . . . . . . . . . . . . . . . . . . . . . . . . 36310.7 Majorization and Unitarily Invariant Norms . . . . . . . . . . . . 372

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395

Frequently Used Notation and Terminology

dimV , 3 dimension of vector space VMn, 8 n× n (i.e., n-square) matrices with complex entriesA = (aij), 8 matrix A with (i, j)-entry aijI, 9 identity matrixAT , 9 transpose of matrix AA, 9 conjugate of matrix A

A∗, 9 conjugate transpose of matrix A, i.e., A∗ = AT

A−1, 13 inverse of matrix Arank (A), 11 rank of matrix AtrA, 21 trace of matrix AdetA, 12 determinant of matrix A|A|, 12, 83, 164 determinant for a block matrix A or (A∗A)1/2 or (|aij |)(u, v), 27 inner product of vectors u and v∥ · ∥, 28, 113 norm of a vector or a matrixKer(A), 17 kernel or null space of A, i.e., Ker(A) = {x : Ax = 0}Im(A), 17 image space of A, i.e., Im(A) = {Ax}ρ(A), 109 spectral radius of matrix Aσmax(A), 109 largest singular value (spectral norm) of matrix Aλmax(A), 124 largest eigenvalue of matrix AA ≥ 0, 81 A is positive semidefinite (or all aij ≥ 0 in Section 5.7)A ≥ B, 81 A−B is positive semidefinite (or aij ≥ bij in Section 5.7)A ◦B, 117 Hadamard (entrywise) product of matrices A and BA⊗B, 117 Kronecker (tensor) product of matrices A and B

x ≺w y, 326 weak majorization, i.e., all∑k

i=1 x↓i ≤

∑ki=1 y

↓i hold

x ≺wlog y, 344 weak log-majorization, i.e., all∏k

i=1 x↓i ≤

∏ki=1 y

↓i hold

An n× n matrix A is said to be

upper-triangular if all entries below the main diagonal are zerodiagonalizable if P−1AP is diagonal for some invertible matrix Psimilar to B if P−1AP = B for some invertible matrix Punitarily similar to B if U∗AU = B for some unitary matrix Uunitary if AA∗ = A∗A = I, i.e., A−1 = A∗

positive semidefinite if x∗Ax ≥ 0 for all vectors x ∈ Cn

Hermitian if A = A∗

normal if A∗A = AA∗

λ ∈ C is an eigenvalue of A ∈ Mn if Ax = λx for some nonzero x ∈ Cn.

xv

Frequently Used Theorems

• Cauchy–Schwarz inequality: Let V be an inner product space overa number field (R or C). Then for all vectors x and y in V

|(x, y)|2 ≤ (x, x)(y, y).

Equality holds if and only if x and y are linearly dependent.

• Theorem on the eigenvalues of AB and BA: Let A and B be m×nand n ×m complex matrices, respectively. Then AB and BA have thesame nonzero eigenvalues, counting multiplicity. As a consequence,

tr(AB) = tr(BA).

• Schur triangularization theorem: For any n-square matrix A, thereexists an n-square unitary matrix U such that U∗AU is upper-triangular.

• Jordan decomposition theorem: For any n-square matrix A, thereexists an n-square invertible complex matrix P such that

A = P−1(J1 ⊕ J2 ⊕ · · · ⊕ Jk)P,

where each Ji, i = 1, 2, . . . , k, is a Jordan block.

• Spectral decomposition theorem: Let A be an n-square normalmatrix with eigenvalues λ1, λ2, . . . , λn. Then there exists an n-squareunitary matrix U such that

A = U∗ diag(λ1, λ2, . . . , λn)U.

In particular, if A is positive semidefinite, then all λi ≥ 0; if A is Her-mitian, then all λi are real; and if A is unitary, then all |λi| = 1.

• Singular value decomposition theorem: Let A be anm×n complexmatrix with rank r. Then there exist an m-square unitary matrix U andan n-square unitary matrix V such that

A = UDV,

where D is the m×n matrix with (i, i)-entries being the singular valuesof A, i = 1, 2, . . . , r, and other entries 0. If m = n, then D is diagonal.

xvii