This page intentionally left blank

Practical Extrapolation MethodsAn important problem that arises in many scientic and engineering applications isthat of approximating limits of innite sequences {Am}. In most cases, these sequencesconverge very slowly. Thus, to approximate their limits with reasonable accuracy, onemust compute a large number of the terms of {Am}, and this is generally costly. Theselimits can be approximated economically and with high accuracy by applying suitableextrapolation (or convergence acceleration)methods to a small number of terms of {Am}.

This book is concerned with the coherent treatment, including derivation, analysis, andapplications, of the most useful scalar extrapolation methods. The methods it discussesare geared toward problems that commonly arise in scientic and engineering disci-plines. It differs from existing books on the subject in that it concentrates on the mostpowerful nonlinear methods, presents in-depth treatments of them, and shows whichmethods are most effective for different classes of practical nontrivial problems; it alsoshows how to ne-tune these methods to obtain best numerical results.

This book is intended to serve as a state-of-the-art reference on the theory and practice ofextrapolation methods. It should be of interest to mathematicians interested in the theoryof the relevant methods and serve applied scientists and engineers as a practical guidefor applying speed-up methods in the solution of difcult computational problems.

Avram Sidi is Professor of Numerical Analysis in the Computer Science Departmentat the TechnionIsrael Institute of Technology and holds the Technion AdministrationChair in Computer Science. He has published extensively in various areas of numericalanalysis and approximation theory and in journals such asMathematics of Computation,SIAM Review, SIAM Journal on Numerical Analysis, Journal of Approximation The-ory, Journal of Computational and Applied Mathematics, Numerische Mathematik, andJournal of Scientic Computing. Professor Sidis work has involved the development ofnovel methods, their detailed mathematical analysis, design of efcient algorithms fortheir implementation, and their application to difcult practical problems. His methodsand algorithms are successfully used in various scientic and engineering disciplines.

CAMBRIDGE MONOGRAPHS ONAPPLIED AND COMPUTATIONALMATHEMATICS

Series EditorsP. G. CIARLET, A. ISERLES, R. V. KOHN, M. H. WRIGHT

10 Practical Extrapolation Methods

The Cambridge Monographs on Applied and Computational Mathematics reect thecrucial role of mathematical and computational techniques in contemporary science.The series presents expositions on all aspects of applicable and numerical mathematics,with an emphasis on new developments in this fast-moving area of research.

State-of-the-art methods and algorithms as well as modern mathematical descriptionsof physical and mechanical ideas are presented in a manner suited to graduate researchstudents and professionals alike. Sound pedagogical presentation is a prerequisite. It isintended that books in the series will serve to inform a new generation of researchers.

Also in this series:A Practical Guide to Pseudospectral Methods, Bengt FornbergDynamical Systems and Numerical Analysis, A. M. Stuart and A. R. HumphriesLevel Set Methods and Fast Marching Methods, J. A. SethianThe Numerical Solution of Integral Equations of the Second Kind,Kendall E. AtkinsonOrthogonal Rational Functions, Adhemar Bultheel, Pablo Gonzalez-Vera,Erik Hendriksen, and Olav NjastadTheory of Composites, Graeme W. MiltonGeometry and Topology for Mesh Generation, Herbert EdelsbrunnerSchwarzChristoffel Mapping, Tobin A. Driscoll and Lloyd N. Trefethen

Practical Extrapolation MethodsTheory and Applications

AVRAM SIDITechnionIsrael Institute of Technology

Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, So Paulo

Cambridge University PressThe Edinburgh Building, Cambridge , United Kingdom

First published in print format

ISBN-13 978-0-521-66159-1 hardback

ISBN-13 978-0-511-06862-1 eBook (EBL)

Cambridge University Press 2003

2003

Information on this title: www.cambridge.org/9780521661591

This book is in copyright. Subject to statutory exception and to the provision ofrelevant collective licensing agreements, no reproduction of any part may take placewithout the written permission of Cambridge University Press.

ISBN-10 0-511-06862-X eBook (EBL)

ISBN-10 0-521-66159-5 hardback

Cambridge University Press has no responsibility for the persistence or accuracy ofs for external or third-party internet websites referred to in this book, and does notguarantee that any content on such websites is, or will remain, accurate or appropriate.

Published in the United States by Cambridge University Press, New York

www.cambridge.org

Contents

Preface page xix

Introduction 10.1 Why ExtrapolationConvergence Acceleration? 10.2 Antilimits Versus Limits 40.3 General Algebraic Properties of Extrapolation Methods 5

0.3.1 Linear Summability Methods and theSilvermanToeplitz Theorem 7

0.4 Remarks on Algorithms for Extrapolation Methods 80.5 Remarks on Convergence and Stability of Extrapolation Methods 10

0.5.1 Remarks on Study of Convergence 100.5.2 Remarks on Study of Stability 100.5.3 Further Remarks 13

0.6 Remark on Iterated Forms of Extrapolation Methods 140.7 Relevant Issues in Extrapolation 15

I The Richardson Extrapolation Process and Its Generalizations 19

1 The Richardson Extrapolation Process 211.1 Introduction and Background 211.2 The Idea of Richardson Extrapolation 271.3 A Recursive Algorithm for the Richardson Extrapolation Process 281.4 Algebraic Properties of the Richardson Extrapolation Process 29

1.4.1 A Related Set of Polynomials 291.4.2 AnEquivalentAlternativeDenitionofRichardsonExtrapolation 31

1.5 Convergence Analysis of the Richardson Extrapolation Process 331.5.1 Convergence of Columns 331.5.2 Convergence of Diagonals 34

1.6 Stability Analysis of the Richardson Extrapolation Process 371.7 A Numerical Example: Richardson Extrapolation on

the Zeta Function Series 381.8 The Richardson Extrapolation as a Summability Method 39

1.8.1 Regularity of Column Sequences 39

vii

viii Contents

1.8.2 Regularity of Diagonal Sequences 401.9 The Richardson Extrapolation Process for Innite Sequences 41

2 Additional Topics in Richardson Extrapolation 422.1 Richardson Extrapolation with Near Geometric and Harmonic {yl} 422.2 Polynomial Richardson Extrapolation 432.3 Application to Numerical Differentiation 482.4 Application to Numerical Quadrature: Romberg Integration 522.5 Rational Extrapolation 55

3 First Generalization of the Richardson Extrapolation Process 573.1 Introduction 573.2 Algebraic Properties 593.3 Recursive Algorithms for A( j)n 61

3.3.1 The FS-Algorithm 623.3.2 The E-Algorithm 64

3.4 Numerical Assessment of Stability 663.5 Analysis of Column Sequences 67

3.5.1 Convergence of the ( j)ni 683.5.2 Convergence and Stability of the A( j)n 693.5.3 Convergence of the k 723.5.4 Conditioning of the System (3.1.4) 733.5.5 Conditioning of (3.1.4) for the Richardson Extrapolation Process

on Diagonal Sequences 743.6 Further Results for Column Sequences 763.7 Further Remarks on (3.1.4): Related Convergence

Acceleration Methods 773.8 Epilogue: What Is the E-Algorithm? What Is It Not? 79

4 GREP: FurtherGeneralization of theRichardsonExtrapolation Process 814.1 The Set F(m) 814.2 Denition of the Extrapolation Method GREP 854.3 General Remarks on F(m) and GREP 874.4 A Convergence Theory for GREP 88

4.4.1 Study of Process I 894.4.2 Study of Process II 904.4.3 Further Remarks on Convergence Theory 914.4.4 Remarks on Convergence of the ki 924.4.5 Knowing the Minimal m Pays 92

4.5 Remarks on Stability of GREP 924.6 Extensions of GREP 93

5 The D-Transformation: A GREP for Innite-Range Integrals 955.1 The Class B(m) and Related Asymptotic Expansions 95

5.1.1 Description of the Class A( ) 96

Contents ix

5.1.2 Description of the Class B(m) 985.1.3 Asymptotic Expansion of F(x) When f (x) B(m) 1005.1.4 Remarks on the Asymptotic Expansion of F(x) and

a Simplication 1035.2 Denition of the D(m)-Transformation 103

5.2.1 Kernel of the D(m)-Transformation 1055.3 A Simplication of the D(m)-Transformation:

The sD(m)-Transformation 1055.4 How to Determine m 106

5.4.1 By Trial and Error 1065.4.2 Upper Bounds on m 107

5.5 Numerical Examples 1115.6 Proof of Theorem 5.1.12 1125.7 Characterization and Integral Properties of Functions in B(1) 117

5.7.1 Integral Properties of Functions in A( ) 1175.7.2 A Characterization Theorem for Functions in B(1) 1185.7.3 Asymptotic Expansion of F(x) When f (x) B(1) 118

6 The d-Transformation: A GREP for Innite Series and Sequences 1216.1 The Class b(m) and Related Asymptotic Expansions 121

6.1.1 Description of the Class A( )0 1226.1.2 Description of the Class b(m) 1236.1.3 Asymptotic Expansion of An When {an} b(m) 1266.1.4 Remarks on the Asymptotic Expansion of An and

a Simplication 1296.2 Denition of the d (m)-Transformation 130

6.2.1 Kernel of the d (m)-Transformation 1316.2.2 The d (m)-Transformation for Innite Sequences 1316.2.3 The Factorial d (m)-Transformation 132

6.3 Special Cases with m = 1 1326.3.1 The d (1)-Transformation 1326.3.2 The Levin L-Transformation 1336.3.3 The Sidi S-Transformation 133

6.4 How to Determine m 1336.4.1 By Trial and Error 1346.4.2 Upper Bounds on m 134

6.5 Numerical Examples 1376.6 A Further Class of Sequences in b(m): The Class b(m) 140

6.6.1 The Function Class A(,m)0 and Its Summation Properties 1406.6.2 The Sequence Class b(m) and a Characterization Theorem 1426.6.3 Asymptotic Expansion of An When {an} b(m) 1456.6.4 The d (m)-Transformation 1476.6.5 Does {an} b(m) Imply {an} b(m)? A Heuristic Approach 148

6.7 Summary of Properties of Sequences in b(1) 150

x Contents

6.8 A General Family of Sequences in b(m) 1526.8.1 Sums of Logarithmic Sequences 1536.8.2 Sums of Linear Sequences 1556.8.3 Mixed Sequences 157

7 Recursive Algorithms for GREP 1587.1 Introduction 1587.2 The W-Algorithm for GREP(1) 159

7.2.1 A Special Case: (y) = yr 1637.3 The W(m)-Algorithm for GREP(m) 164

7.3.1 Ordering of the k(t)t i 1647.3.2 Technical Preliminaries 1667.3.3 Putting It All Together 1677.3.4 Simplications for the Cases m = 1 and m = 2 171

7.4 Implementation of the d (m)-Transformation by the W(m)-Algorithm 1727.5 The EW-Algorithm for an Extended GREP(1) 173

8 Analytic Study of GREP(1): Slowly Varying A(y) F(1) 1768.1 Introduction and Error Formula for A( j)n 1768.2 Examples of Slowly Varying a(t) 1808.3 Slowly Varying (t) with Arbitrary tl 1818.4 Slowly Varying (t) with liml(tl+1/tl) = 1 185

8.4.1 Process I with (t) = tH (t) and Complex 1858.4.2 Process II with (t) = t 1898.4.3 Process II with (t) = tH (t) and Real 190

8.5 Slowly Varying (t) with liml(tl+1/tl) = (0, 1) 1938.6 Slowly Varying (t) with Real and tl+1/tl (0, 1) 196

8.6.1 The Case (t) = t 1968.6.2 The Case (t) = tH (t) with Real and tl+1/tl (0, 1) 199

9 Analytic Study of GREP(1): Quickly Varying A(y) F(1) 2039.1 Introduction 2039.2 Examples of Quickly Varying a(t) 2049.3 Analysis of Process I 2059.4 Analysis of Process II 2079.5 Can {tl} Satisfy (9.1.4) and (9.4.1) Simultaneously? 210

10 Efcient Use of GREP(1): Applications to the D(1)-, d (1)-,and d (m)-Transformations 212

10.1 Introduction 21210.2 Slowly Varying a(t) 212

10.2.1 Treatment of the Choice liml(tl+1/tl) = 1 21210.2.2 Treatment of the Choice 0 < tl+1/tl < 1 215

10.3 Quickly Varying a(t) 216

Contents xi

11 Reduction of the D-Transformation for Oscillatory Innite-RangeIntegrals: The D-, D-, W -, and mW -Transformations 218

11.1 Reduction of GREP for Oscillatory A(y) 21811.1.1 Review of the W-Algorithm for Innite-Range Integrals 219

11.2 The D-Transformation 22011.2.1 Direct Reduction of the D-Transformation 22011.2.2 Reduction of the sD-Transformation 221

11.3 Application of the D-Transformation to Fourier Transforms 22211.4 Application of the D-Transformation to Hankel Transforms 22311.5 The D-Transformation 22411.6 Application of the D-Transformation to Hankel Transforms 22511.7 Application of the D-Transformation to Integrals of Products of

Bessel Functions 22611.8 The W - and mW -Transformations for Very Oscillatory Integrals 227

11.8.1 Description of the Class B 22711.8.2 The W - and mW -Transformations 22811.8.3 Application of the mW -Transformation to Fourier,

Inverse Laplace, and Hankel Transforms 23111.8.4 Further Variants of the mW -Transformation 23411.8.5 Further Applications of the mW -Transformation 235

11.9 Convergence and Stability 23511.10 Extension to Products of Oscillatory Functions 236

12 Acceleration of Convergence of Power Series by the d-Transformation:Rational d-Approximants 238

12.1 Introduction 23812.2 The d-Transformation on Power Series 23812.3 Rational Approximations from the d-Transformation 240

12.3.1 Rational d-Approximants 24012.3.2 Closed-Form Expressions for m = 1 241

12.4 Algebraic Properties of Rational d-Approximants 24212.4.1 Pade-like Properties 24212.4.2 Recursive Computation by the W(m)-Algorithm 244

12.5 Prediction Properties of Rational d-Approximants 24512.6 Approximation of Singular Points 24612.7 Efcient Application of the d-Transformation to Power Series with APS 24712.8 The d-Transformation on Factorial Series 24912.9 Numerical Examples 250

13 Acceleration of Convergence of Fourier andGeneralized Fourier Seriesby the d-Transformation: The Complex Series Approach with APS 253

13.1 Introduction 25313.2 Introducing Functions of Second Kind 254

13.2.1 General Background 25413.2.2 Complex Series Approach 254

xii Contents

13.2.3 Justication of the Complex Series Approach and APS 25513.3 Examples of Generalized Fourier Series 256

13.3.1 Chebyshev Series 25613.3.2 Nonclassical Fourier Series 25713.3.3 FourierLegendre Series 25713.3.4 FourierBessel Series 258

13.4 Convergence and Stability when {bn} b(1) 25913.5 Direct Approach 25913.6 Extension of the Complex Series Approach 26013.7 TheH-Transformation 261

14 Special Topics in Richardson Extrapolation 26314.1 Conuence in Richardson Extrapolation 263

14.1.1 The Extrapolation Process and the SGRom-Algorithm 26314.1.2 Treatment of Column Sequences 26514.1.3 Treatment of Diagonal Sequences 26614.1.4 A Further Problem 267

14.2 Computation of Derivatives of Limits and Antilimits 26814.2.1 Derivative of the Richardson Extrapolation Process 26914.2.2 ddGREP

(1): Derivative of GREP(1) 274

II Sequence Transformations 277

15 The Euler Transformation, Aitken 2-Process, andLubkin W -Transformation 279

15.1 Introduction 27915.2 The EulerKnopp (E, q) Method 279

15.2.1 Derivation of the Method 27915.2.2 Analytic Properties 28115.2.3 A Recursive Algorithm 283

15.3 The Aitken 2-Process 28315.3.1 General Discussion of the 2-Process 28315.3.2 Iterated 2-Process 28615.3.3 Two Applications of the Iterated 2-Process 28715.3.4 A Modied 2-Process for Logarithmic Sequences 289

15.4 The Lubkin W -Transformation 29015.5 Stability of the Iterated 2-Process and Lubkin Transformation 293

15.5.1 Stability of the Iterated 2-Process 29315.5.2 Stability of the Iterated Lubkin Transformation 294

15.6 Practical Remarks 29515.7 Further Convergence Results 295

16 The Shanks Transformation 29716.1 Derivation of the Shanks Transformation 29716.2 Algorithms for the Shanks Transformation 301

Contents xiii

16.3 Error Formulas 30216.4 Analysis of Column Sequences When Am A +

k=1 k

mk 303

16.4.1 Extensions 30916.4.2 Application to Numerical Quadrature 312

16.5 Analysis of Column Sequences When {Am} b(1) 31316.5.1 Linear Sequences 31316.5.2 Logarithmic Sequences 31616.5.3 Factorial Sequences 316

16.6 The Shanks Transformation on Totally Monotonic andTotally Oscillating Sequences 31716.6.1 Totally Monotonic Sequences 31716.6.2 The Shanks Transformation on Totally Monotonic Sequences 31816.6.3 Totally Oscillating Sequences 32016.6.4 The Shanks Transformation on Totally Oscillating Sequences 321

16.7 Modications of the -Algorithm 321

17 The Pade Table 32317.1 Introduction 32317.2 Algebraic Structure of the Pade Table 32517.3 Pade Approximants for Some Hypergeometric Functions 32717.4 Identities in the Pade Table 32917.5 Computation of the Pade Table 33017.6 Connection with Continued Fractions 332

17.6.1 Denition and Algebraic Properties of Continued Fractions 33217.6.2 Regular C-Fractions and the Pade Table 333

17.7 Pade Approximants and Exponential Interpolation 33617.8 Convergence of Pade Approximants from Meromorphic Functions 338

17.8.1 de Montessuss Theorem and Extensions 33817.8.2 Generalized Koenigs Theorem and Extensions 341

17.9 Convergence of Pade Approximants from Moment Series 34217.10 Convergence of Pade Approximants from Polya Frequency Series 34517.11 Convergence of Pade Approximants from Entire Functions 346

18 Generalizations of Pade Approximants 34818.1 Introduction 34818.2 Pade-Type Approximants 34818.3 Vanden BroeckSchwartz Approximations 35018.4 Multipoint Pade Approximants 351

18.4.1 Two-Point Pade Approximants 35318.5 HermitePade Approximants 354

18.5.1 Algebraic HermitePade Approximants 35518.5.2 Differential HermitePade Approximants 355

18.6 Pade Approximants from Orthogonal Polynomial Expansions 35618.6.1 Linear (Cross-Multiplied) Approximations 35618.6.2 Nonlinear (Properly Expanded) Approximations 358

xiv Contents

18.7 BakerGammel Approximants 36018.8 PadeBorel Approximants 362

19 The Levin L- and Sidi S-Transformations 36319.1 Introduction 36319.2 The Levin L-Transformation 363

19.2.1 Derivation of the L-Transformation 36319.2.2 Algebraic Properties 36519.2.3 Convergence and Stability 367

19.3 The Sidi S-Transformation 36919.4 A Note on Factorially Divergent Sequences 371

20 The Wynn - and Brezinski -Algorithms 37520.1 The Wynn -Algorithm and Generalizations 375

20.1.1 The Wynn -Algorithm 37520.1.2 Modications of the -Algorithm 376

20.2 The Brezinski -Algorithm 37920.2.1 Convergence and Stability of the -Algorithm 38020.2.2 A Further Convergence Result 383

21 The G-Transformation and Its Generalizations 38421.1 The G-Transformation 38421.2 The Higher-Order G-Transformation 38521.3 Algorithms for the Higher-Order G-Transformation 386

21.3.1 The rs-Algorithm 38621.3.2 The FS/qd-Algorithm 38721.3.3 Operation Counts of the rs- and FS/qd-Algorithms 389

22 The Transformations of Overholt and Wimp 39022.1 The Transformation of Overholt 39022.2 The Transformation of Wimp 39122.3 Convergence and Stability 392

22.3.1 Analysis of Overholts Method 39222.3.2 Analysis of Wimps Method 394

23 Conuent Transformations 39623.1 Conuent Forms of Extrapolation Processes 396

23.1.1 Derivation of Conuent Forms 39623.1.2 Convergence Analysis of a Special Case 399

23.2 Conuent Forms of Sequence Transformations 40123.2.1 Conuent -Algorithm 40123.2.2 Conuent Form of the Higher-Order G-Transformation 40223.2.3 Conuent -Algorithm 40323.2.4 Conuent Overholt Method 404

Contents xv

23.3 Conuent D(m)-Transformation 40523.3.1 Application to the D(1)-Transformation and Fourier Integrals 405

24 Formal Theory of Sequence Transformations 40724.1 Introduction 40724.2 Regularity and Acceleration 408

24.2.1 Linearly Convergent Sequences 40924.2.2 Logarithmically Convergent Sequences 410

24.3 Concluding Remarks 411

III Further Applications 413

25 Further Applications of Extrapolation Methods andSequence Transformations 415

25.1 Extrapolation Methods in Multidimensional Numerical Quadrature 41525.1.1 By GREP and d-Transformation 41525.1.2 Use of Variable Transformations 418

25.2 ExtrapolationofNumerical Solutions ofOrdinaryDifferential Equations 42125.3 Romberg-Type Quadrature Formulas for Periodic Singular and

Weakly Singular Fredholm Integral Equations 42225.3.1 Description of Periodic Integral Equations 42225.3.2 Corrected Quadrature Formulas 42325.3.3 Extrapolated Quadrature Formulas 42525.3.4 Further Developments 427

25.4 Derivation of Numerical Schemes for Time-Dependent Problemsfrom Rational Approximations 428

25.5 Derivation of Numerical Quadrature Formulas fromSequence Transformations 430

25.6 Computation of Integral Transforms with Oscillatory Kernels 43325.6.1 Via Numerical Quadrature Followed by Extrapolation 43325.6.2 Via Extrapolation Followed by Numerical Quadrature 43425.6.3 Via Extrapolation in a Parameter 435

25.7 Computation of Inverse Laplace Transforms 43725.7.1 Inversion by Extrapolation of the Bromwich Integral 43725.7.2 Gaussian-Type Quadrature Formulas for the Bromwich Integral 43725.7.3 Inversion via Rational Approximations 43925.7.4 Inversion via the Discrete Fourier Transform 440

25.8 Simple Problems Associated with {an} b(1) 44125.9 AccelerationofConvergenceof Innite SerieswithSpecial SignPatterns 442

25.9.1 Extensions 44325.10 Acceleration of Convergence of Rearrangements of Innite Series 443

25.10.1 Extensions 44425.11 Acceleration of Convergence of Innite Products 444

25.11.1 Extensions 445

xvi Contents

25.12 Computation of Innite Multiple Integrals and Series 44625.12.1 Sequential D-Transformation for s-D Integrals 44725.12.2 Sequential d-Transformation for s-D Series 447

25.13 A Hybrid Method: The RichardsonShanks Transformation 44925.13.1 An Application 450

25.14 Application of Extrapolation Methods to Ill-Posed Problems 45325.15 Logarithmically Convergent Fixed-Point Iteration Sequences 454

IV Appendices 457

A Review of Basic Asymptotics 459A.1 The O , o, and Symbols 459A.2 Asymptotic Expansions 460A.3 Operations with Asymptotic Expansions 461

B The Laplace Transform and Watsons Lemma 463B.1 The Laplace Transform 463B.2 Watsons Lemma 463

C The Gamma Function 465

D Bernoulli Numbers and Polynomials and the EulerMaclaurin Formula 467D.1 Bernoulli Numbers and Polynomials 467D.2 The EulerMaclaurin Formula 468

D.2.1 The EulerMaclaurin Formula for Sums 468D.2.2 The EulerMaclaurin Formula for Integrals 469

D.3 Applications of EulerMaclaurin Expansions 470D.3.1 Application to Harmonic Numbers 470D.3.2 Application to Cauchy Principal Value Integrals 471

D.4 A Further Development 471D.5 The EulerMaclaurin Formula for Integrals with Endpoint Singularities 472

D.5.1 Algebraic Singularity at One Endpoint 472D.5.2 Algebraic-Logarithmic Singularity at One Endpoint 473D.5.3 Algebraic-Logarithmic Singularities at Both Endpoints 474

D.6 Application to Singular Periodic Integrands 475

E The Riemann Zeta Function and the Generalized Zeta Function 477E.1 Some Properties of (z) 477E.2 Asymptotic Expansion of

n1k=0(k + )z 477

F Some Highlights of Polynomial Approximation Theory 480F.1 Best Polynomial Approximations 480F.2 Chebyshev Polynomials and Expansions 480

G A Compendium of Sequence Transformations 483

Contents xvii

H Efcient Application of Sequence Transformations: Summary 488

I FORTRAN 77 Program for the d (m)-Transformation 493I.1 General Description 493I.2 The Code 494

Bibliography 501

Index 515

Preface

An important problem that arises in many scientic and engineering applications is thatof nding or approximating limits of innite sequences {Am}. The elements Am of suchsequences can show up in the form of partial sums of innite series, approximations fromxed-point iterations of linear and nonlinear systems of equations, numerical quadratureapproximations to nite- or innite-range integrals, whether simple or multiple, etc. Inmost applications, these sequences converge very slowly, and this makes their directuse to approximate limits an expensive proposition. There are important applications inwhich they may even diverge. In such cases, the direct use of the Am to approximatetheir so-called antilimits would be impossible. (Antilimits can be interpreted in appro-priate ways depending on the nature of {Am}. In some cases they correspond to analyticcontinuation in some parameter, for example.)

An effective remedy for these problems is via application of extrapolation methods(or convergence acceleration methods) to the given sequences. (In the context of innitesequences, extrapolation methods are also referred to as sequence transformations.)Loosely speaking, an extrapolation method takes a nite and hopefully small number ofthe Am and processes them in some way. A good method is generally nonlinear in theAm and takes into account, either explicitly or implicitly, their asymptotic behavior asm in a clever fashion.

The importance of extrapolation methods as effective computational tools has longbeen recognized. Indeed, the Richardson extrapolation and the Aitken 2-process, twopopular representatives, are discussed in some detail in almost all modern textbooks onnumerical analysis, and Pade approximants have become an integral part of approxi-mation theory. During the last thirty years a few books were written on the subject andvarious comparative studies were done in relation to some important subclasses of con-vergence acceleration problems, pointing to the most effective methods. Finally, sincethe 1970s, international conferences partly dedicated to extrapolationmethods have beenheld on a regular basis.

The main purpose of this book is to present a unied account of the existing literatureon nonlinear extrapolation methods for scalar sequences that is as comprehensive andup-to-date as possible. In this account, I include much of the literature that deals withmethods of practical importance whose effectiveness has been amply veried in varioussurveys and comparative studies. Inevitably, the contents reect my personal interestsand taste. Therefore, I apologize to those colleagues whose work has not been covered.

xix

xx Preface

I have left out completely the important subject of extrapolation methods for vectorsequences, even though I have been actively involved in this subject for the last twentyyears. I regret this, especially in view of the fact that vector extrapolation methods havehad numerous successful applications in the solution of large-scale nonlinear as well aslinear problems. I believe only a fully dedicated book would do justice to them.

The Introduction gives an overview of convergence acceleration within the frame-work of innite sequences. It includes a discussion of the concept of antilimit throughexamples. Following that, it discusses, both in general terms and by example, the devel-opment of methods, their analysis, and accompanying algorithms. It also gives a detaileddiscussion of stability in extrapolation. A proper understanding of this subject is veryhelpful in devising effective strategies for extrapolation methods in situations of inherentinstability. The reader is advised to study this part of the Introduction carefully.

Following the Introduction, the book is divided into three main parts:

(i) Part I dealswith theRichardson extrapolation and its generalizations. It also preparessome of the background material and techniques relevant to Part II. Chapters 1 and2 give a complete treatment of the Richardson extrapolation process (REP) thatis only partly described in previous works. Following that, Chapter 3 gives a rstgeneralization of REP with some amount of theory. The rest of Part I is devotedto the generalized Richardson extrapolation process (GREP) of Sidi, the LevinSidi D-transformation for innite-range integrals, the LevinSidi d-transformationfor innite series, Sidis variants of the D-transformation for oscillatory innite-range integrals, the SidiLevin rational d-approximants, and efcient summationof power series and (generalized) Fourier series by the d-transformation. (Twoimportant topics covered in connection with these transformations are the classof functions denoted B(m) and the class of sequences denoted b(m). Both of theseclasses are more comprehensive than those considered in other works.) Efcientimplementation of GREP is the subject of a chapter that includes the W-algorithmof Sidi and the W(m)-algorithm of Ford and Sidi. Also, there are two chapters thatprovide a detailed convergence and stability analysis of GREP(1), a prototype ofGREP, whose results point to effective strategies for applying these methods indifferent situations. The strategies denoted arithmetic progression sampling (APS)and geometric progression sampling (GPS) are especially useful in this connection.

(ii) Part II is devoted to development and analysis of a number of effective sequencetransformations. I begin with three classical methods that are important historicallyand that have been quite successful in many problems: the Euler transformation (theonly linear method included in this book), the Aitken 2-process, and the Lubkintransformation. I next give an extended treatment of theShanks transformation alongwith Pade approximants and their generalizations, continued fractions, and the qd-algorithm. Finally, I treat other importantmethods such as theG-transformation, theWynn -algorithm and its modications, the Brezinski -algorithm, the Levin L-and Sidi S-transformations, and the methods of Overholt and of Wimp. I also givethe conuent forms of some of these methods. In this treatment, I include knownresults on the application of these transformations to so-called linear and logarithmicsequences and quite a few new results, including some pertaining to stability. I use

Preface xxi

the latter to draw conclusions on how to apply sequence transformations moreeffectively in different situations. In this respect APS of Part I turns out to be aneffective strategy when applying the methods of Part II to linear sequences.

(iii) Part III comprises a single chapter that provides a number of applications to problemsin numerical analysis that are not considered in Parts I and II.

Following these, I have also included as Part IV a sequence of appendices that providea lot of useful information recalled throughout the book. These appendices cover brieyseveral important subjects, such as asymptotic expansions, EulerMaclaurin expansions,the Riemann Zeta function, and polynomial approximation theory, to name a few. Inparticular, Appendix D puts together, for the rst time, quite a few EulerMaclaurinexpansions of importance in applications. Appendices G and H include a summary of theextrapolation methods covered in the book and important tips on when and how to applythem. Appendix I contains a FORTRAN 77 code that implements the d-transformationand that is applied to a number of nontrivial examples. The user can run this code as is.

The readermay bewonderingwhy I separated the topic of REP and its generalizations,especially GREP, from that of sequence transformations and treated it rst. There aretwo major reasons for this decision. First, the former is more general, covers more casesof interest, and allows for more exibility in a natural way. For most practical purposes,the methods of Part I have a larger scope and achieve higher accuracy than the sequencetransformations studied in Part II. Next, some of the conclusions from the theory ofconvergence and stability of extrapolation methods of Part I turn out to be valid for thesequence transformations of Part II and help to improve their performance substantially.

Now, the treatment of REP and its generalizations concerns the problem of ndingthe limit (or antilimit) as y 0+ of a function A(y) known for y (0, b], b > 0. Herey may be a continuous or discrete variable. Such functions arise in many applications ina natural way. The analysis provided for them suggests to which problems they shouldbe applied, how they should be applied for best possible outcome in nite-precisionarithmetic, and what type of convergence and accuracy one should expect. All this alsohas an impact on how sequence transformations should be applied to innite sequences{Am} that are not directly related to a function A(y). In such a case, we can alwaysdraw the analogy A(y) Am for some function A(y) with y m1, and view therelevant sequence transformations as extrapolation methods. When treated this way, itbecomes easier to see that APS improves the performance of these transformations onlinear sequences.

The present book differs from already existing works in various ways:

1. The classes of (scalar) sequences treated in it include many of those that arise inscientic and engineering applications. They are more comprehensive than thoseconsidered in previous books and include the latter as well.

2. Divergent sequences (such as those that arise from divergent innite series or inte-grals, for example) are treated on an equal footing with convergent ones. Suitableinterpretations of their antilimits are provided. It is shown rigorously, at least in somecases, that extrapolation methods can be applied to such sequences, with no changes,to produce excellent approximations to their antilimits.

xxii Preface

3. A thorough asymptotic analysis of tails of innite series and innite-range integralsis given. Based on this analysis, detailed convergence studies for the differentmethodsare provided. Considerable effort is made to obtain the best possible results under thesmallest number of realistic conditions. These results either come in the form ofbona de asymptotic expansions of the errors or they provide tight upper bounds onerrors. They are accompanied by complete proofs in most cases. The proofs chosenfor presentation are generally those that can be applied in more than one situation andhence deserve special attention. When proofs are not provided, the reader is referredto the relevant papers and books.

4. The stability issue is formalized and given full attention for the rst time. Conclusionsare drawn from stability analyses to devise effective strategies that enable one toobtain the best possible accuracy in nite-precision arithmetic, also in situationswhere instabilities are built in. Immediate applications of this are to the summationof so-called very oscillatory innite-range integrals, logarithmically convergent (ordivergent) series, power series, Fourier series, etc., where most methods have notdone well in the past.

I hope the book will serve as a reference for researchers in the area of extrapolationmethods and for scientists and engineers in different computational disciplines and as atextbook for students interested in the subject. I have kept the mathematical backgroundneeded to cope with the material to a minimum. Most of what the reader needs andis not covered by the standard academic curricula is summarized in the appendices. Iwould like to emphasize here that the subject of asymptotics is of utmost importance toany treatment of extrapolation methods. It would be impossible to appreciate the beautyof the subject of extrapolation the development of the methods and their mathemat-ical analyses without it. It is certainly impossible to produce any honest theory ofextrapolation methods without it. I urge the reader to acquire a good understanding ofasymptotics before everything else. The essentials of this subject are briey discussedin Appendix A.

I wish to express my gratitude to my friends and colleagues David Levin, William F.Ford, Doron S. Lubinsky, Moshe Israeli, and Marius Ungarish for the interest they tookin this book and for their criticisms and comments in different stages of the writing. Iam also indebted to Yvonne Sagie and Hadas Heier for their expert typing of part of thebook.

Lastly, I owe a debt of gratitude to my wife Carmella for her constant encouragementand support during the last six years while this book was being written. Without herunderstanding of, and endless patience for, my long absences from home this bookwould not have seen daylight. I dedicate this book to her with love.

Avram SidiTechnion, HaifaDecember 2001

Introduction

0.1 Why ExtrapolationConvergence Acceleration?Inmany problems of scientic computing, one is facedwith the task of nding or approx-imating limits of innite sequences. Such sequences may arise in different disciplinesand contexts and in various ways. In most cases of practical interest, the sequencesin question converge to their limits very slowly. This may cause their direct use forapproximating their limits to become computationally expensive or impossible.

There are other cases in which these sequences may even diverge. In such a case, weare left with the question of whether the divergent sequence represents anything, and ifso, what it represents. Although in some cases the elements of a divergent sequence canbe used as approximations to the quantity it represents subject to certain conditions, inmost other cases it is meaningless to make direct use of the sequence elements for thispurpose.

Let us consider two very common examples:(i) Summation of innite series: This is a problem that arises in many scientic dis-

ciplines, such as applied mathematics, theoretical physics, and theoretical chemistry. Inthis problem, the sequences in question are those of partial sums. In some cases, the termsak of a series

k=0 ak may be known analytically. In other cases, these terms may be

generated numerically, but the process of generating more and more terms may becomevery costly. In both situations, if the series converges very slowly, the task of obtaininggood approximations to its sum only from its partial sums An =

nk=0 ak, n = 0, 1, . . . ,

may thus become very expensive as it necessitates a very large number of the terms ak .In yet other cases, only a nite number of the terms ak , say a0, a1, . . . , aN , may beknown. In such a situation, the accuracy of the best available approximation to the sumof

k=0 ak is normally that of the partial sum AN and thus cannot be improved further.If the series diverges, then its partial sums have only limited direct use. Divergent seriesarise naturally in different elds, perturbation analysis in theoretical physics being one ofthem. Divergent power series arise in the solution of homogeneous ordinary differentialequations around irregular singular points.

(ii) Iterative solution of linear and nonlinear systems of equations: This problemoccurs very commonly in applied mathematics and different branches of engineering.When continuum problems are solved by methods such as nite differences and niteelements, large and sparse systems of linear and/or nonlinear equations are obtained. A

1

2 Introduction

very attractive way of solving these systems is by iterative methods. The sequences inquestion for this case are those of the iteration vectors that have a large dimension ingeneral. In most cases, these sequences converge very slowly. If the cost of computingone iteration vector is very high, then obtaining a good approximation to the solution ofa given system of equations may also become very high.

The problems of slow convergence or even divergence of sequences can be overcomeunder suitable conditions by applying extrapolation methods (equivalently, convergenceacceleration methods or sequence transformations) to the given sequences. When ap-propriate, an extrapolation method produces from a given sequence {An} a new sequence{ An} that converges to the formers limit more quickly when this limit exists. In casethe limit of {An} does not exist, the new sequence { An} produced by the extrapolationmethod either diverges more slowly than {An} or converges to some quantity called theantilimit of {An} that has a useful meaning and interpretation in most applications. Wenote at this point that the precise meaning of the antilimit may vary depending on thetype of the divergent sequence, and that several possibilities exist. In the next section,we shall demonstrate through examples how antilimits may arise and what exactly theymay be.

Concerning divergent sequences, there are three important messages that we wouldlike to get across in this book: (i) Divergent sequences can be interpreted appropriately inmany cases of interest, and useful antilimits for them can be dened. (ii) Extrapolationmethods can be used to produce good approximations to the relevant antilimits in anefcient manner. (iii) Divergent sequences can be treated on an equal footing with con-vergent ones, both computationally and theoretically, and this is what we do throughoutthis book. (However, everywhere-divergent innite power series, that is, those with zeroradius of convergence, are not included in the theoretical treatment generally.)

It must be emphasized that each An is determined from only a nite number of theAm . This is a basic requirement that extrapolation methods must satisfy. Obviously, anextrapolation method that requires knowledge of all the Am for determining a given Anis of no practical value.

We now pause to illustrate the somewhat abstract discussion presented above withthe Aitken 2-process that is one of the classic examples of extrapolation methods.This method was rst described in Aitken [2], and it can be found in almost every bookon numerical analysis. See, for example, Henrici [130], Ralston and Rabinowitz [235],Stoer and Bulirsch [326], and Atkinson [13].

Example 0.1.1 Let the sequence {An} be such that

An = A + an + rn with rn = bn + o(min{1, ||n}) as n , (0.1.1)

where A, a, b, , and are in general complex scalars, and

a, b = 0, , = 0, 1, and || > ||. (0.1.2)

As a result, rn bn = o(n) as n . If || < 1, then limn An = A. If || 1,then limn An does not exist, A being the antilimit of {An} in this case. Consider nowthe Aitken 2-process, which is an extrapolation method that, when applied to {An},

0.1 Why ExtrapolationConvergence Acceleration? 3

produces a sequence { An} with

An =An An+2 A2n+1

An 2An+1 + An+2 =

An An+1An An+1

1 1An An+1

, (0.1.3)

where Am = Am+1 Am, m 0. To see how An behaves for n , we substitute(0.1.1) in (0.1.3). Taking into account the fact that rn+1 rn as n , after somesimple algebra it can be shown that

| An A| |rn| = O(n) = o(n) as n , (0.1.4)

for some positive constant that is independent of n. Obviously, when limn rn = 0,the sequence { An} converges to A whether {An} converges or not. [If rn = 0 for n N ,then An = A for n N aswell, as implied by (0.1.4). In fact, the formula for An in (0.1.3)is obtained by requiring that An = Awhen rn = 0 for all large n, and it is the solution forA of the equations Am = A + am, m = n, n + 1, n + 2.] Also, in case {An} converges,{ An} convergesmore quickly and to limn An = A, because An A an as n from (0.1.1) and (0.1.2). Thus, the rate of convergence of {An} is enhanced by the factor

| An A||An A| = O(|/|

n) = o(1) as n . (0.1.5)

A more detailed analysis of An A yields the result

An A b ( )2

( 1)2 n as n , (0.1.6)

that is more rened than (0.1.4) and asymptotically best possible as well. It is clearfrom (0.1.6) that, when the sequence {rn} does not converge to 0, which happens when|| 1, both {An} and { An} diverge, but { An} diverges more slowly than {An}.

In view of this example and the discussion that preceded it, we now introduce theconcepts of convergence acceleration and acceleration factor.

Denition 0.1.2 Let {An} be a sequence of in general complex scalars, and let { An} bethe sequence generated by applying the extrapolation method ExtM to {An}, An beingdetermined from Am, 0 m Ln , for some integer Ln, n = 0, 1, . . . . Assume thatlimn An = A for some A and that, if limn An exists, it is equal to this A. We shallsay that { An} converges more quickly than {An} if

limn

| An A||ALn A|

= 0, (0.1.7)

whether limn An exists or not. When (0.1.7) holds we shall also say that theextrapolation method ExtM accelerates the convergence of {An}. The ratio Rn =| An A|/|ALn A| is called the acceleration factor of An .

4 Introduction

The ratios Rn measure the extent of the acceleration induced by the extrapolationmethod ExtM on {An}. Indeed, from | An A| = Rn|ALn A|, it is obvious that Rnis the factor by which the acceleration process reduces |ALn A| in generating An .Obviously, a good extrapolation method is one whose acceleration factors tend to zeroquickly as n .

In case {An} is a sequence of vectors in some general vector space, the precedingdenition is still valid, provided we replace |ALn A| and | An A| everywhere with ALn A and An A, respectively, where is the norm in the vector spaceunder consideration.

0.2 Antilimits Versus LimitsBefore going on, we would like to dwell on the concept of antilimit that we mentionedbriey above. This concept can best be explained by examples to which we now turn.These examples do not exhaust all the possibilities for antilimits by any means. We shallencounter more later in this book.

Example 0.2.1 Let An, n = 0, 1, 2, . . . , be the partial sums of the power seriesk=0 akz

k, that is, An =

nk=0 akz

k, n = 0, 1, . . . . If the radius of convergence of this series is nite and positive, then limn An exists for |z| < and is a functionf (z) that is analytic for |z| < . Of course,k=0 akzk diverges for |z| > . If f (z) canbe continued analytically to |z| = and |z| > , then the analytic continuation of f (z)is the antilimit of {An} for |z| .

As an illustration, let us pick a0 = 0 and ak = 1/k, k = 1, 2, . . . , so that = 1and limn An = log(1 z) for |z| 1, z = 1. The principal branch of log(1 z) thatis analytic for all complex z [1,+) serves as the antilimit of {An} in case |z| > 1but z [1,+).

Example 0.2.2 Let An, n = 0, 1, 2, . . . , be the partial sums of the Fourier seriesk= ake

ikx ; that is, An =n

k=n akeikx , n = 0, 1, 2, . . . , and assume thatC1|k|

|ak | C2|k| for all large |k| and some positive constantsC1 andC2 and for some 0,so that limn An does not exist. This Fourier series represents a 2 -periodic general-ized function; see Lighthill [167]. If, for x in some interval I of [0, 2 ], this generalizedfunction coincides with an ordinary function f (x), then f (x) is the antilimit of {An} forx I . (Recall that limn An , in general, exists when < 0 and an is monotonic in n.It exists unconditionally when < 1.)

As an illustration, let us pick a0 = 0 and ak = 1, k = 1,2, . . . . Then the se-ries

k= ake

ikx represents the generalized function 1+ 2m= (x 2m ),where (z) is the Dirac delta function. This generalized function coincides with the or-dinary function f (x) = 1 in the interval (0, 2 ), and f (x) serves as the antilimit of{An} for n when x (0, 2 ).

Example 0.2.3 Let 0 < x0 < x1 < x2 < , limn xn = , s = 0 and real, and letAn be dened as An =

xn0 g(t)eist dt, n = 0, 1, 2, . . . , where C1t |g(t)| C2t

for all large t and some positive constants C1 and C2 and for some 0, so that

0.3 General Algebraic Properties of Extrapolation Methods 5

limn An does not exist. In many such cases, the antilimit of {An} is the Abel sum ofthe divergent integral

0 g(t)eist dt (see, e.g., Hardy [123]) that is dened by

lim0+0 e

t g(t)eist dt . [Recall that 0 g(t)eist dt exists and limn An =0 g(t)eist dt , in general, when < 0 and g(t) is monotonic in t for large t . This istrue unconditionally when < 1.]

As an illustration, let us pick g(t) = t1/2. Then the Abel sum of the divergent integral0 t

1/2eist dt is ei3/4/(2s3/2), and it serves as the antilimit of {An}.

Example 0.2.4 Let {hn} be a sequence in (0, 1) satisfying h0 > h1 > h2 > , andlimn hn = 0, and dene An =

1hn x

g(x) dx, n = 0, 1, 2, . . . , where g(x) is con-tinuously differentiable on [0, 1] a sufcient number of times with g(0) = 0 and is in general complex and 1 but = 1,2, . . . . Under these conditionslimn An does not exist. The antilimit of {An} in this case is the Hadamard nite partof the divergent integral

10 x

g(x) dx (see Davis and Rabinowitz [63]) that is given bythe expression

m1

i=0

1 + i + 1

g(i)(0)i!

+ 1

0x[

g(x)m1

i=0

g(i)(0)i!

xi]

dx

with m > 1 so that the integral in this expression exists as an ordinary integral.[Recall that 10 xg(x) dx exists and limn An =

10 x

g(x) dx for > 1.]As an illustration, let us pick g(x) = (1+ x)1 and = 3/2. Then the Hadamard

nite part of 10 x

3/2(1+ x)1 dx is 2 /2, and it serves as the antilimit of {An}.Note that limn An = + but the associated antilimit is negative.

Example 0.2.5 Let s be the solution to the nonsingular linear system of equations(I T )x = c, and let {xn} be dened by the iterative scheme xn+1 = T xn + c, n =0, 1, 2, . . . , with x0 given. Let (T ) denote the spectral radius of T . If (T ) > 1, then{xn} diverges in general. The antilimit of {xn} in this case is the solution s itself. [Recallthat limn xn exists and is equal to s when (T ) < 1.]

As should become clear from these examples, the antilimit may have different mean-ings depending on the nature of the sequence {An}. Thus, it does not seem to be possibleto dene antilimits in a unique way, and we do not attempt to do this. It appears, though,that studying the asymptotic behavior of An for n is very helpful in determiningthe meaning of the relevant antilimit. We hope that what the antilimit of a given diver-gent sequence is will become more apparent as we proceed to the study of extrapolationmethods.

0.3 General Algebraic Properties of Extrapolation MethodsWe saw in Section 0.1 that an extrapolation method operates on a given sequence {An}to produce a new sequence { An}. That is, it acts as a mapping from {An} to { An}. In allcases of interest, this mapping has the general form

An = n(A0, A1, . . . , ALn ), (0.3.1)

6 Introduction

where Ln is some nite positive integer. (As mentioned earlier, methods for whichLn = are of no use, because they require knowledge of all the Am to obtain An withnite n.) In addition, for most extrapolation methods there holds

An =Kn

i=0ni Ai , (0.3.2)

where Kn are some nonnegative integers and the ni are some scalars that satisfy

Kn

i=0ni = 1 (0.3.3)

for each n. (This is the case for all of the extrapolation methods we consider in thiswork.) A consequence of (0.3.2) and (0.3.3) is that such extrapolation methods act assummability methods for the sequence {An}.

When the ni are independent of the Am , the approximation An is linear in the Am , thusthe extrapolationmethod that generates { An} becomes a linear summability method. Thatis to say, this extrapolation method can be applied to every sequence {An} with the sameni . Both numerical experience and the different known convergence analyses suggestthat linear methods are of limited scope and not as effective as nonlinear methods.

As the subject of linear summability methods is very well-developed and is treated indifferent books, we are not going to dwell on it in this book; see, for example, the booksby Knopp [152], Hardy [123], and Powell and Shah [231]. We only give the denition oflinear summability methods at the end of this section and recall the SilvermanToeplitztheorem, which is one of the fundamental results on linear summability methods. Laterin this work, we also discuss the Euler transformation that has been used in differentpractical situations and that is probably the most successful linear summability method.

When the ni depend on the Am , the approximation An is nonlinear in the Am . Thisimplies that if Cm = Am + Bm, m = 0, 1, 2, . . . , for some constants and , and{ An}, { Bn}, and { Cn} are obtained by applying a given nonlinear extrapolation method to{An}, {Bn}, and {Cn}, respectively, then Cn = An + Bn, n = 0, 1, 2, . . . , in general.(Equality prevails for all n when the extrapolation method is linear.) Despite this fact,most nonlinear extrapolation methods enjoy a sort of linearity property that can be de-scribed as follows: Let = 0 and be arbitrary constants and considerCm = Am + ,m = 0, 1, 2, . . . . Then

Cn = An + , n = 0, 1, 2, . . . . (0.3.4)

In other words, {Cn} = {An} + implies { Cn} = { An} + . This is called the quasi-linearity property and is a useful property that we want every extrapolation methodto have. (All extrapolation methods treated in this book are quasi-linear.) A sufcientcondition for this to hold is given in Proposition 0.3.1.

Proposition 0.3.1 Let a nonlinear extrapolation method be such that the sequence { An}that it produces from {An} satises (0.3.2) with (0.3.3). Then the sequence { Cn} thatit produces from {Cn = An + } for arbitrary constants = 0 and satises the

0.3 General Algebraic Properties of Extrapolation Methods 7

quasi-linearity property in (0.3.4) if the ni in (0.3.2) depend on the Am through theAm = Am+1 Am only and are homogeneous in the Am of degree 0.

Remark. We recall that a function f (x1, . . . , xp) is homogeneous of degree r if, forevery = 0, f (x1, . . . , xp) = r f (x1, . . . , xp).

Proof. We begin by rewriting (0.3.2) in the form An =Kn

i=0 ni ({Am})Ai . Similarly, wehave Cn =

Kni=0 ni ({Cm})Ci . From (0.3.1) and the conditions imposed on the ni , there

exist functions Dni ({um}) for whichni ({Am}) = Dni ({Am}) and ni ({Cm}) = Dni ({Cm}), (0.3.5)

where the functions Dni satisfy for all = 0Dni ({um}) = Dni ({um}). (0.3.6)

This and the fact that {Cm} = {Am} imply thatni ({Cm}) = Dni ({Cm}) = Dni ({Am}) = ni ({Am}). (0.3.7)

From (0.3.2) and (0.3.7) we have, therefore,

Cn =Kn

i=0ni ({Am})(Ai + ) = An +

Kn

i=0ni ({Am}). (0.3.8)

The result now follows by invoking (0.3.3).

Example 0.3.2 Consider the Aitken 2-process that was given by (0.1.3) in Exam-ple 0.1.1. We can reexpress An in the form

An = n,n An + n,n+1An+1, (0.3.9)with

n,n = An+1An+1 An , n,n+1 =

AnAn+1 An . (0.3.10)

Thus, ni = 0 for 0 i n 1. It is easy to see that the ni satisfy the conditions ofProposition 0.3.1 so that the 2-process has the quasi-linearity property described in(0.3.4). Note also that for this method Ln = n + 2 in (0.3.1) and Kn = n + 1 in (0.3.2).

0.3.1 Linear Summability Methods and the SilvermanToeplitz TheoremWe now go back briey to linear summability methods. Consider the innite matrix

M =

00 01 02 10 11 12 20 21 22 .

.

.

.

.

.

.

.

.

, (0.3.11)

8 Introduction

where ni are some xed scalars. The linear summability method associated with M isthe linear mapping that transforms an arbitrary sequence {An} to another sequence {An}through

An =

i=0ni Ai , n = 0, 1, 2, . . . . (0.3.12)

This method is regular if limn An = A implies limn An = A. The SilvermanToeplitz theorem that we state next gives necessary and sufcient conditions for a linearsummability method to be regular. For proofs of this fundamental result see, for example,the books by Hardy [123] and Powell and Shah [231].

Theorem 0.3.3 (SilvermanToeplitz theorem). The summability method associated withthe matrix M in (0.3.11) is regular if and only if the following three conditions arefullled simultaneously:(i) limn

i=0 ni = 1.

(ii) limn ni = 0, i = 0, 1, 2, . . . .(iii) supn

i=0 |ni |

0.4 Remarks on Algorithms for Extrapolation Methods 9

by those extrapolation methods with large qn , in general. This means that we actuallywant to solve large systems of equations, which may be a computationally expensiveproposition. In such cases, the development of good algorithms becomes especiallyimportant. The next example helps make this point clear.

Example 0.4.1 The Shanks [264] transformation of order k is an extrapolation method,which, when applied to a sequence {An}, produces the sequence { An = ek(An)}, whereek(An) satises the nonlinear system of equations

Ar = ek(An)+k

i=1i

ri , n r n + 2k, (0.4.2)

where i and i are additional (auxiliary) 2k unknowns. Provided this system has asolution with i = 0 and i = 0, 1 and i = j if i = j , then ek(An) can be shown tosatisfy the linear system

Ar = ek(An)+k

i=1iAr+i1, n r n + k, (0.4.3)

where i are additional (auxiliary) k unknowns. Here Am = Am+1 Am, m =0, 1, . . . , as before. [In any case, we can start with (0.4.3) as the denition of ek(An).]Now, this linear system can be solved using Cramers rule, giving us ek(An) as the ratioof two (k + 1) (k + 1) determinants in the form

ek(An) =

An An+1 An+kAn An+1 An+k.

.

.

.

.

.

.

.

.

An+k1 An+k An+2k1

1 1 1An An+1 An+k.

.

.

.

.

.

.

.

.

An+k1 An+k An+2k1

. (0.4.4)

We can use this determinantal representation to compute ek(An), but this would be veryexpensive for large k and thus would constitute a bad algorithm. A better algorithm isone that solves the linear system in (0.4.3) by Gaussian elimination. But this algorithmtoo becomes costly for large k. The -algorithm of Wynn [368], on the other hand, isvery efcient as it produces all of the ek(An), 0 n + 2k N , that are dened byA0, A1, . . . , AN in only O(N 2) operations. It reads

(n)1 = 0, (n)0 = An, n = 0, 1, . . . ,

(n)k+1 = (n+1)k1 +

1(n+1)k (n)k

, n, k = 0, 1, . . . , (0.4.5)

and we have

ek(An) = (n)2k , n, k = 0, 1, . . . . (0.4.6)

10 Introduction

Incidentally, An in (0.1.3) produced by the Aitken 2-process is nothing but e1(An).[Note that the Shanks transformations are quasi-linear extrapolation methods. This canbe seen either from the equations in (0.4.2), or from those in (0.4.3), or from the deter-minantal representation of ek(An) in (0.4.4), or even from the -algorithm itself.]

Finally, there are extrapolation methods in the literature that are dened exclusivelyby recursive algorithms from the start. The -algorithm of Brezinski [32] is such anextrapolation method, and it is dened by recursion relations very similar to those of the-algorithm.

0.5 Remarks on Convergence and Stability of Extrapolation MethodsThe analysis of convergence and stability is the most important subject in the theory ofextrapolation methods. It is also the richest in terms of the variety of results that existand still can be obtained for different extrapolation methods and sequences. Thus, it isimpossible to make any specic remarks about convergence and stability at this stage.We can, however, make several remarks on the approach to these topics that we take inthis book. We start with the topic of convergence analysis.

0.5.1 Remarks on Study of ConvergenceThe rst stage in the convergence analysis of extrapolation methods is formulation ofconditions that we impose on the {An}. In this book, we deal with sequences that arisein common applications. Therefore, we emphasize mainly conditions that are relevantto these applications. Also, we keep the number of the conditions imposed on the {An}to a minimum as this leads to mathematically more valuable and elegant results. Thenext stage is analysis of the errors An A under these conditions. This analysis maylead to different types of results depending on the complexity of the situation. In somecases, we are able to give a full asymptotic expansion of An A for n ; in othercases, we obtain only the most dominant term of this expansion. In yet other cases, weobtain a realistic upper bound on | An A| from which powerful convergence resultscan be obtained. An important feature of our approach is that we are not content onlywith showing that the sequence { An} converges more quickly than {An}, that is, thatconvergence acceleration takes place in accordance with Denition 0.1.2, but insteadwe aim at obtaining the precise asymptotic behavior of the corresponding accelerationfactor or a good upper bound for it.

0.5.2 Remarks on Study of StabilityWe now turn to the topic of stability in extrapolation. Unlike convergence, this topic maynot be common knowledge, so we start with some rather general remarks on what wemean by stability and how we analyze it. Our discussion here is based on those of Sidi[272], [300], [305], and is recalled in relevant places throughout the book.

When we compute the sequence { An} in nite-precision arithmetic, we obtain a se-quence { An} that is different from { An}, the exact transformed sequence. This, of course,

0.5 Remarks on Convergence and Stability of Extrapolation Methods 11

is caused mainly by errors (roundoff errors and errors of other kinds as well) in the An .Naturally, we would like to know by how much An differs from An , that is, we wantto be able to estimate | An An|. This is important also since knowledge of | An An|assists in assessing the cumulative error | An A| in An . To see this, we start with

| An An| | An A|

| An A| | An An| + | An A|. (0.5.1)

Next, let us assume that limn | An A| = 0. Then (0.5.1) implies that | An A| | An An| for all sufciently large n, because | An An| remains nonzero.

Wehave observed numerically that, formany extrapolationmethods that satisfy (0.3.2)with (0.3.3), | An An| can be estimated by the product n(n), where

n =Kn

i=0|ni | 1 and (n) = max{|i | : ni = 0}, (0.5.2)

and, for each i , i is the error in Ai . The idea behind this is that the ni and hence n donot change appreciably with small errors in the Ai . Thus, if Ai + i are the computed Ai ,then An , the computed An , is very nearly given by

Kni=0 ni (Ai + i ) = An +

Kni=0 nii .

As a result,

| An An|

Kn

i=0nii

n(n). (0.5.3)

The meaning of this is that the quantity n [that always satises n 1 by (0.3.3)]controls the propagation of errors in {An} into { An}, in the sense that the absolutecomputational error | An An| is practically the maximum of the absolute errors in theAi , 0 i Kn , magnied by the factor n . Thus, combining (0.5.1) and (0.5.3), weobtain

| An A| n(n) + | An A| (0.5.4)

for the absolute errors, and

| An A||A| n

(n)

|A| +| An A||A| , provided A = 0, (0.5.5)

for the relative errors.The implication of (0.5.4) is that, practically speaking, the cumulative error | An A| is

at least of the order of the corresponding theoretical error | An A| but it may be as largeas n

(n) if this quantity dominates. [Note that because limn | An A| = 0, n(n)will dominate | An A| for sufciently large n.] The approximate inequality in (0.5.5)reveals even more in case {An} is convergent and hence An , A, and the Ai , 0 i Kn ,are all of the same order of magnitude and the latter are known correctly to r signi-cant decimal digits and n is of order 10s, s < r . Then (n)/|A| will be of order 10r ,and, therefore, | An A|/|A| will be of order 10sr for sufciently large n. This impliesthat An will have approximately r s correct signicant decimal digits when n is suf-ciently large. When s r , however, An may be totally inaccurate in the sense that it

12 Introduction

may be completely different from An . In other words, in such cases n is also a measureof the loss of relative accuracy in the computed An .

One conclusion that can be drawn from this discussion is that it is possible to achievesufcient accuracy in An by increasing r , that is, by computing the An with high accuracy.This can be accomplished on a computer by doubling the precision of the oating-pointarithmetic used for computing the An .

When applying an extrapolation method to a convergent sequence {An} numerically,we would like to be able to compute the sequence { An} without | An An| becomingunbounded for increasing n. In view of this and the discussion of the previous paragraphs,we now give a formal denition of stability.

Denition 0.5.1 If an extrapolation method that generates from {An} the sequence { An}satises (0.3.2) with (0.3.3), then we say that it is stable provided supn n

0.5 Remarks on Convergence and Stability of Extrapolation Methods 13

are large when is too close to 1 in the complex plane. It is not difcult to see thatthey can be reduced simultaneously in such a situation if the 2-process is applied toa subsequence {An}, where {2, 3, . . . }, since, for even small , is farther awayfrom 1 than is.

We continue our discussion of | An A| assuming now that the Ai have been computedwith relative errors not exceeding . In other words, i = i Ai and |i | for all i .(This is the case when the Ai have been computed to maximum accuracy that is possiblein nite-precision arithmetic with rounding unit u; we have = u in this situation.) Then(0.5.4) becomes

| An A| n In({As})+ | An A|, In({As}) max{|Ai | : ni = 0}. (0.5.8)Obviously, when {An} converges, or diverges but is bounded, the term In({As}) re-mains bounded as n . In this case, it follows from (0.5.8) that, provided nremains bounded, | An A| remains bounded as well. It should be noted, however,that when {An} diverges and is unbounded, In({As}) is unbounded as n , whichcauses the right-hand side of (0.5.8) to become unbounded as n , even whenn is bounded. In such cases, | An A| becomes unbounded, as we have observed inall our numerical experiments. The hope in such cases is that the convergence rateof the exact transformed sequence { An} is much greater than the divergence rate of{An} so that sufcient accuracy is achieved by An before In({As}) has grown toomuch.

We also note that, in case {An} is divergent and the Ai have been computed withrelative errors not exceeding , numerical stability can be assessed more accurately byreplacing (0.5.3) and (0.5.4) by

| An An| Kn

i=0|ni | |Ai | (0.5.9)

and

| An A| Kn

i=0|ni | |Ai | + | An A|, (0.5.10)

respectively. Again, when the Ai have been computed to maximum accuracy that ispossible in nite-precision arithmetic with rounding unit u, we have = u in (0.5.9) and(0.5.10). [Of course, (0.5.9) and (0.5.10) are valid when {An} converges too.]

0.5.3 Further RemarksFinally, in connection with the studies of convergence and stability, we have foundit very useful to relate the given innite sequences {Am} to some suitable functionsA(y), where y may be a discrete or continuous variable. These relations take the formAm = A(ym), m = 0, 1, . . . , for some positive sequences {ym} that tend to 0, suchthat limm Am = limy0+ A(y) when limm Am exists. In some cases, a sequence{Am} is derived directly from a known function A(y) exactly as described. The sequences

14 Introduction

discussed in Examples 0.2.3 and 0.2.4 are of this type. In certain other cases, we can showthe existence of a suitable function A(y) that is associated with a given sequence {Am}even though {Am} is not provided by a relation of the form Am = A(ym), m = 0, 1, . . . ,a priori.

This kind of an approach is obviously of greater generality than that dealing withinnite sequences alone. First, for a given sequence {Am}, the related function A(y) mayhave certain asymptotic properties for y 0+ that can be very helpful in deciding whatkind of an extrapolation method to use for accelerating the convergence of {Am}. Next,in case A(y) is known a priori, we can choose {ym} such that (i) the convergence ofthe derived sequence {Am = A(ym)} will be easier to accelerate by some extrapolationmethod, and (ii) this extrapolation method will also enjoy good stability properties. Fi-nally, the function A(y), in contrast to the sequence {Am}, may possess certain analyticproperties in addition to its asymptotic properties for y 0+. The analytic propertiesmay pertain, for example, to smoothness and differentiability in some interval (0, b]that contains {ym}. By taking these properties into account, we are able to enlarge con-siderably the scope of the theoretical convergence and stability studies of extrapolationmethods. We are also able to obtain powerful and realistic results on the behavior of thesequences { An}.

We shall use this approach to extrapolation methods in many places throughout thisbook, starting as early as Chapter 1.

Historically, those convergence acceleration methods associated with functions A(y)and derived from them have been called extrapolation methods, whereas those thatapply to innite sequences and that are derived directly from them have been calledsequence transformations. In this book, we also make this distinction, at least as faras the order of presentation is concerned. Thus, we devote Part I of the book to theRichardson extrapolation process and its various generalizations and Part II to sequencetransformations.

0.6 Remark on Iterated Forms of Extrapolation MethodsOne of the ways to apply extrapolation methods is by simply iterating them. Let us againconsider an arbitrary extrapolation method ExtM. The iteration of ExtM is performedas follows: We rst apply ExtM to {C (n)0 = An} to obtain the sequence {C (n)1 }. We nextapply it to {C (n)s }n=0 to obtain {C (n)s+1}n=0, s = 1, 2, . . . . Let us organize the C (n)s in atwo-dimensional array as in Table 0.6.1. In general, columns of this table converge, eachcolumn converging at least as quickly as the one preceding it. Diagonals converge aswell, and they converge much quicker than the columns.

It is thus obvious that every extrapolation method can be iterated. For example, wecan iterate ek , the Shanks transformation of order k discussed in Example 0.4.1, exactlyas explained here. In this book, we consider in detail the iteration of only two classicmethods: The 2-process, which we have discussed briey in Example 0.1.1, and theLubkin transformation. Both of thesemethods and their iterations are considered in detailin Chapter 15. We do not consider the iterated forms of the other methods discussed inthis book, mainly because, generally speaking, their performance is not better than that

0.7 Relevant Issues in Extrapolation 15

Table 0.6.1: Iteration of an extrapolation method ExtM.Each column is obtained by applying ExtM to the

preceding column

C (0)0C (1)0 C

(0)1

C (2)0 C(1)1 C

(0)2

C (3)0 C(2)1 C

(1)2 C

(0)3

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

provided by the straightforward application of the correspondingmethods; in some casesthey even behave in undesired peculiar ways. In addition, their analysis can be done veryeasily once the techniques ofChapter 15 (on the convergence and stability of their columnsequences) are understood.

0.7 Relevant Issues in ExtrapolationWe close this chapter by giving a list of issues we believe are relevant to both the theoryand the practice of extrapolation methods. Inevitably, some of these issues are moreimportant than others and deserve more attention.

The rst issue we need to discuss is that of development and design of extrapolationmethods. Being the start of everything, this is a very important phase. It is best to embarkon the project of development with certain classes of sequences {An} in mind. Once wehave developed a method that works well on those sequences for which it was designed,we can always apply it to other sequences as well. In some cases, this may even resultin success. The Aitken 2-process may serve as a good example to illustrate this pointin a simple way.

Example 0.7.1 Let A, , a, and b be in general complex scalars and let the sequence{An} be such that

An = A + n[ans + bns1 + O(ns2)] as n ; = 0, 1, a = 0, and s = 0.

(0.7.1)Even though the 2-process was designed to accelerate the convergence of sequences{An} whose members satisfy (0.1.1), it turns out that it accelerates the convergence ofsequences {An} that satisfy (0.7.1) as well. Substituting (0.7.1) in (0.1.3), it can be shownafter tedious algebraic manipulations that

An A Knns2 as n ; K = as(

1)2, (0.7.2)

as opposed to An A anns as n . Thus, for || < 1 both {An} and { An} con-verge, and { An} converges more quickly.

16 Introduction

Naturally, if we have to make a choice about which extrapolation methods to employregularly as users, wewould probably prefer thosemethods that are effective acceleratorsfor more than one class of sequences. Finally, we would like to remark that, as the designof extrapolation methods is based on the asymptotic properties of An for n inmany important cases, there is great value to analyzing An asymptotically for n .In addition, this analysis also produces the conditions on {An} that we need for theconvergence study, as discussed in the second paragraph of Section 0.5. In many casesof interest, this study may turn out to be very nontrivial and challenging.

The next issue is the design of good algorithms for implementing extrapolation meth-ods. Recall that there may be more than one algorithm for implementing a given method.As mentioned before, a good algorithm is one that requires a small number of operationsand little storage. In addition, it should be as stable as possible numerically. Needless tosay, from the point of view of a user, we should always prefer the most efcient algo-rithm for a given method. Note that the development of algorithms, because it relies onalgebraic manipulations only, is not in general as important and illuminating as eitherthe development of extrapolation methods or the analysis of these methods, an issue wediscuss below, or the asymptotic study of {An}, which precedes all this. After all, thequality of the sequence { An} is determined exclusively by the extrapolation method thatgenerates it and not by whichever algorithm is used for implementing this extrapolationmethod. Algorithms are only numerical means by which we obtain the { An} alreadyuniquely determined by the extrapolation method. For these reasons, we reduce ourtreatment of algorithms to a minimum.

Some of the literature deals with so-called kernels of extrapolation methods. Thekernel of an extrapolation method is the class of sequences {An} for which An = A forall n, where A is the limit or antilimit of {An}. Unfortunately, there is very little one canlearn about the convergence behavior and stability of the method in question by lookingat its kernel. For this reason, we have left out the treatment of kernels almost completely.We have considered in passing only those kernels that are obtained in a trivial way.

Given an extrapolation method and a class of sequences to which it is applied, aswe mentioned in the preceding section, the most important and challenging issues arethose of convergence and stability. Proposing realistic and useful analyses for conver-gence and stability has always been a very difcult task because most of the practicalextrapolation methods are highly nonlinear. As a result, very few papers have dealt withconvergence and stability, where we believe more efforts should be spent. A variety ofmathematical tools are needed for this task, asymptotic analysis being the most crucialof them.

As for the user, we believe that he should be at least familiar with the existing conver-gence and stability results, as these may be very helpful in deciding which extrapolationmethod should be applied, to which problem, how it should be applied, and how itshould be tuned for good stability properties. In addition, having at least some ideaabout the asymptotic behavior of the elements of the sequence whose convergence isbeing accelerated is of great assistance in efcient application of extrapolation methods.

We would like to make one last remark about extrapolation methods as opposedto algorithms by which methods are implemented. In recent years, there has been an

0.7 Relevant Issues in Extrapolation 17

unfortunate confusion of terminology, in thatmanypapers use the concepts ofmethod andalgorithm interchangeably. The result of this confusion has been that effectiveness dueto extrapolation methods has been incorrectly assigned to extrapolation algorithms. Asnoted earlier, the sequences { An} are uniquely dened and their properties are determinedonly by extrapolation methods and not by algorithms that implement the methods. Inthis book, we are very careful to avoid this confusion by distinguishing betweenmethodsand algorithms.

Part IThe Richardson Extrapolation Process

and Its Generalizations

1The Richardson Extrapolation Process

1.1 Introduction and BackgroundIn many problems of practical interest, a given innite sequence {An} can be relatedto a function A(y) that is known, and hence is computable, for 0 < y b with someb > 0, the variable y being continuous or discrete. This relation takes the form An =A(yn), n = 0, 1, . . . , for some monotonically decreasing sequence {yn} (0, b] thatsatises limn yn = 0. Thus, in case limy0+ A(y) = A, limn An = A as well.Consequently, computing limn An amounts to computing limy0+ A(y) in such acase, and this is precisely what we want to do.

Again, in many cases of interest, the function A(y) may have a well-dened expansionfor y 0+ whose form is known. For example and this is the case we treat in thischapter A(y) may satisfy for some positive integer s

A(y) = A +s

k=1k yk + O(ys+1 ) as y 0+, (1.1.1)

where k = 0, k = 1, 2, . . . , s + 1, and 1 < 2 < < s+1, and where k areconstants independent of y. Obviously, 1 > 0 guarantees that limy0+ A(y) = A.When limy0+ A(y) does not exist, A is the antilimit of A(y) for y 0+, and in thiscase i 0 at least for i = 1. If (1.1.1) is valid for all s = 1, 2, 3, . . . , and 1 0 so that limy0+ A(y) = A.Then A can be approximated byA(y)with sufciently small values of y, the error in this approximationbeing A(y) A =O(y1 ) as y 0+ by (1.1.1). If 1 is sufciently large, A(y) can approximate A welleven for values of y that are not too small. If this is not the case, however, thenwemayhaveto compute A(y) for very small values of y to obtain reasonably good approximations

21

22 1 The Richardson Extrapolation Process

to A. Unfortunately, this straightforward idea of reducing y to very small values is notalways applicable. In most cases of interest, computing A(y) for very small values ofy either is very costly or suffers from loss of signicance in nite-precision arithmetic.The deeper idea of the Richardson extrapolation, on the other hand, is to somehoweliminate the y1 term from the expansion in (1.1.1) and to obtain a new approximationA1(y) to A whose error is A1(y) A = O(y2 ) as y 0+. Obviously, A1(y) will be abetter approximation to A than A(y) for small y since 2 > 1. In addition, if 2 issufciently large, then we expect A1(y) to approximate A well also for values of y thatare not too small, independently of the size of 1. At this point, we mention only thatthe Richardson extrapolation is achieved by taking an appropriate weighted averageof A(y) and A(y) for some (0, 1). We give the precise details of this procedure inthe next section.

From (1.1.1), it is clear that A(y) A = O(y1 ) as y 0+, whether1 > 0 or not.Thus, the function A1(y) that results from the Richardson extrapolation can be a usefulapproximation to A for small values of y also when1 0, provided2 > 0. That isto say, limy0+ A1(y) = A provided 2 > 0 whether limy0+ A(y) exists or not. Thisis an additional fundamental and useful feature of the Richardson extrapolation.

In the following examples, we show how functions A(y) exactly of the form we havedescribed here come about naturally. In these examples, we treat the classic problemsof computing by the method of Archimedes, numerical differentiation by differences,numerical integration by the trapezoidal rule, summation of an innite series that isused in dening the Riemann Zeta function, and the Hadamard nite parts of divergentintegrals.

Example 1.1.1 The Method of Archimedes for Computing The method of Arch-imedes for computing consists of approximating the area of the unit disk (that is nothingbut ) by the area of an inscribed or circumscribing regular polygon. If this polygon isinscribed in the unit disk and has n sides, then its area is simply Sn = (n/2) sin(2/n).Obviously, Sn has the (convergent) series expansion

Sn = + 12

i=1

(1)i (2)2i+1(2i + 1)! n

2i , (1.1.3)

and the sequence {Sn} is monotonically increasing and has as its limit.If the polygon circumscribes the unit disk and has n sides, then its area is Sn =

n tan(/n), and Sn has the (convergent) series expansion

Sn = +

i=1

(1)i4i+1(4i+1 1)2i+1B2i+2(2i + 2)! n

2i , (1.1.4)

where Bk are the Bernoulli numbers (see Appendix D), and the sequence {Sn} this timeis monotonically decreasing and has as its limit.

As the expansions given in (1.1.3) and (1.1.4) are also asymptotic as n , Sn inboth cases is analogous to the function A(y). This analogy is as follows: Sn A(y),n1 y, k = 2k, k = 1, 2, . . . , and A. The variable y is discrete and assumesthe values 1/3, 1/4, . . . .

1.1 Introduction and Background 23

Finally, the subsequences {S2m } and {S32m } can be computed recursively withouthaving to know , their computation involving only square roots. (See Example 2.2.2 inChapter 2.)

Example 1.1.2 Numerical Differentiation by Differences Let f (x) be continuouslydifferentiable at x = x0, and assume that f (x0), the rst derivative of f (x) at x0, isneeded. Assume further that the only thing available to us is f (x) , or a procedure thatcomputes f (x), for all values of x in a neighborhood of x0.

If f (x) is known in the neighborhood [x0 a, x0 + a] for some a > 0, then f (x0)can be approximated by the centered difference 0(h) that is given by

0(h) = f (x0 + h) f (x0 h)2h , 0 < h a. (1.1.5)

Note that h here is a continuous variable. Obviously, limh0 0(h) = f (x0). The ac-curacy of 0(h) is quite low, however. When f C3[x0 a, x0 + a], there exists (h) [x0 h, x0 + h], for which the error in 0(h) satises

0(h) f (x0) = f( (h))3!

h2 = O(h2) as h 0. (1.1.6)

When the function f (x) is continuously differentiable a number of times, the error0(h) f (x0) can be expanded in powers of h2. For f C2s+3[x0 a, x0 + a], thereexists (h) [x0 h, x0 + h], for which we have

0(h) = f (x0)+s

k=1

f (2k+1)(x0)(2k + 1)! h

2k + Rs(h), (1.1.7)

where

Rs(h) = f(2s+3)( (h))(2s + 3)! h

2s+2 = O(h2s+2) as h 0. (1.1.8)

The proof of (1.1.7) and (1.1.8) can be achieved by expanding f (x0 h) in a Taylorseries about x0 with remainder.

The difference 0(h) is thus seen to be analogous to the function A(y). This analogyis as follows: 0(h) A(y), h y, k = 2k, k = 1, 2, . . . , and f (x0) A.

When f C[x0 a, x0 + a], the expansion in (1.1.7) holds for all s = 0, 1, . . . .As a result, we can replace it by the genuine asymptotic expansion

0(h) f (x0)+

k=1

f (2k+1)(x0)(2k + 1)! h

2k as h 0, (1.1.9)

whether the innite series on the right-hand side of (1.1.9) converges or not.As is known, in nite-precision arithmetic, the computation of 0(h) for very small

values of h is dominated by roundoff. The reason for this is that as h 0 both f (x0 + h)and f (x0 h) tend to f (x0), which causes the difference f (x0 + h) f (x0 h) tohave fewer and fewer correct signicant digits. Thus, it is meaningless to carry out thecomputation of 0(h) beyond a certain threshold value of h.

24 1 The Richardson Extrapolation Process

Example 1.1.3 Numerical Quadrature by Trapezoidal Rule Let f (x) be dened on[0, 1], and assume that I [ f ] = 10 f (x) dx is to be computed by numerical quadrature.One of the simplest numerical quadrature formulas is the trapezoidal rule. Let T (h) bethe trapezoidal rule approximation to I [ f ], with h = 1/n, n being a positive integer.Then, T (h) is given by

T (h) = h[12f (0)+

n1

j=1f ( jh)+ 1

2f (1)

]

. (1.1.10)

Note that h for this problem is a discrete variable that takes on the values 1, 1/2, 1/3, . . . .It iswell known thatT (h) tends to I [ f ] ash 0 (orn ),whenever f (x) isRiemannintegrable on [0, 1]. When f C2[0, 1], there exists (h) [0, 1], for which the errorin T (h) satises

T (h) I [ f ] = f( (h))12

h2 = O(h2) as h 0. (1.1.11)

When the integrand f (x) is continuously differentiable a number of times, the errorT (h) I [ f ] can be expanded in powers of h2. For f C2s+2[0, 1], there exists (h) [0, 1], for which

T (h) = I [ f ]+s

k=1

B2k(2k)!

[ f (2k1)(1) f (2k1)(0)] h2k + Rs(h), (1.1.12)

where

Rs(h) = B2s+2(2s + 2)! f(2s+2)( (h))h2s+2 = O(h2s+2) as h 0. (1.1.13)

Here Bp are the Bernoulli numbers as before. The expansion in (1.1.12) with (1.1.13) isknown as the EulerMaclaurin expansion (see Appendix D) and its proof can be foundin many books on numerical analysis.

The approximation T (h) is analogous to the function A(y) in the following sense:T (h) A(y), h y, k = 2k, k = 1, 2, . . . , and I [ f ] A.Again, for f C2s+2[0, 1], an expansion that is identical in form to (1.1.12) with

(1.1.13) exists for the midpoint rule approximation M(h), where

M(h) = hn

j=1f ( jh 12h). (1.1.14)

This expansion is

M(h) = I [ f ]+s

k=1

B2k( 12 )(2k)!

[ f (2k1)(1) f (2k1)(0)] h2k + Rs(h), (1.1.15)

where, again for some (h) [0, 1],

Rs(h) =B2s+2( 12 )(2s + 2)! f

(2s+2)( (h))h2s+2 = O(h2s+2) as h 0. (1.1.16)

Here Bp(x) is the Bernoulli polynomial of degree p and B2k( 12 ) = (1 212k)B2k,k = 1, 2, . . . .

1.1 Introduction and Background 25

When f C[0, 1], both expansions in (1.1.12) and (1.1.15) hold for all s =0, 1, . . . . As a result, we can replace both by genuine asymptotic expansions of theform

Q(h) I [ f ]+

k=1ckh2k as h 0, (1.1.17)

where Q(h) stands for T (h) or M(h), and ck is the coefcient of h2k in (1.1.12) or(1.1.15). Generally, when f (x) is not analytic in [0, 1], or even when it is analytic therebut is not entire, the innite series

k=1 ckh2k in (1.1.17) diverges very strongly.

Finally, by h = 1/n, the computation of Q(h) for very small values of h involves alarge number of integrand evaluations and hence is very costly.

Example 1.1.4 Summation of the Riemann Zeta Function Series Let An =nm=1 m

z, n = 1, 2, . . . . When z > 1, limn An = (z), where (z) is the

Riemann Zeta function. For z 1, on the other hand, limn An does not exist.Actually, the innite series

m=1 m

z is taken as the denition of (z) for z > 1.With this denition, (z) is an analytic function of z for z > 1. Furthermore, it can becontinued analytically to the whole z-plane with the exception of the point z = 1, whereit has a simple pole with residue 1.

For all z = 1, i.e., whether limn An exists or not, we have the well-known asymp-totic expansion (see Appendix E)

An (z)+ 11 z

i=0(1)i

(1 zi

)

Binzi+1 as n , (1.1.18)

where Bi are the Bernoulli numbers as before and(a

i)are the binomial coefcients. We

also recall that B3 = B5 = B7 = = 0, and that the rest of the Bi are nonzero.The partial sum An is thus analogous to the function A(y) in the following

sense: An A(y), n1 y, 1 = z 1, 2 = z, k = z + 2k 5, k = 3, 4, . . . , and (z) A provided z = m + 1, m = 0, 1, 2, . . . . Thus, (z) is the limit of {An}whenz > 1, and its antilimit otherwise, provided z = m + 1, m = 0, 1, 2, . . . .Obviously,the variable y is now discrete and takes on the values 1, 1/2, 1/3, . . . .

Note also that the innite series on the right-hand side of (1.1.18) is strongly divergent.

Example 1.1.5 Numerical Integration of Periodic Singular Functions Let us nowconsider the integral I [ f ] = 10 f (x) dx , where f (x) is a 1-periodic function thatis innitely differentiable on (,) except at the points t + k, k = 0,1,2, . . . , where it has logarithmic singularities, and can be written in the form f (x) =g(x)