Adaptive Multigrid for Lattice QCD

James BrannickOctober 22, 2007

Penn Statebrannick@psu.edu

Presentation plan

• Intro to Lattice Quantum Chromodynamics

• Review of Multigrid Basics

• (Adaptive) MG for QCD

• Numerical Experiments

Participants in the MG-QCD project

A. Bessen (Columbia University)

A. Brandt (UCLA, WIS)

J. Brannick (PSU)

M. Brezina (CU Boulder)

R. Brower (BU)

M. Clark (BU)

R. Falgout (CASC-LLNL)

A. Frommer (Wuppertal)

K. Kahl (Wuppertal)

C. Ketelson (CU Boulder)

D. Keyes (Columbia University) O. Livne (Univ. of Utah)I. Livshits (Ball State)S. MacLachlan (TU Delft)T. Manteuffel (CU Boulder)S. McCormick (CU Boulder)V. Nistor (PSU)K. Osterlee (TU Delft)J. Osborne (ANL) C. Rebbi (BU)J. Ruge (CU Boulder) P. Varanas (LLNL)

Forces in Standard Model: SU(N)

e l e c t r o n

p r o t o n

n e u t r o n

quarks

-

+

+

Atoms: MaxwellN=1(charge)

Nuclei: WeakN=2 (isospin)

Sub nuclear: StrongN=3 (color)Standard Model: U(1) £ SU(2) £ SU(3)

QCD path integralAnti-quark quark

Gauge

Dirac Operator Generalized Curl (Maxwell)

The Dirac PDE (for Quarks)

x = (x1 ,x2 ,x3 ,x4 ) (space,time)

4 x 4 sparse spin matrices: 4 non-zero entries 1,-1, i, -i

3x3 color gauge matrices

Wilson (1974): discretization on a hypercubic lattice

U(x,x+) = exp[i hg A(x)]

Discrete Dirac operator on hypercubic lattice

x x+

Dimension:1,2,…,d

Colora,b = 1,2,3

Spini,j = 1,2,3,4

x1 axis

x 2 a

xis

Spin projection Operator

Wilson fermion matrix: M• Typical lattice size : 32Typical lattice size : 323 3 хх 24 24

• Dimension of Dimension of M then then 161633 хх 24 24 хх 2 2 хх 3 3 хх 4 4 ≈ ≈ 101066 хх 10 1066 !!!!

• Need Need M-1(U), Tr[M-1(U)], Det[M(U)]

• Together account for dominant cost more than 80% of the overall flops! Different types of algorithms used for different fermionic actions: Krylov methods typical, as M* = M

M(U)

• Goal of our NSF PetaApps project is inversion on Goal of our NSF PetaApps project is inversion on 2562564 4 lattice lattice - overall - overall simulation at this resolution requires sustained Petaflop years!simulation at this resolution requires sustained Petaflop years!

/

2-d “toy” problem: Schwinger Model

• Space time is 2-d

• Gauge links: U(x,x+) = exp[i e A(x)], U(1) theory

• Dirac fields have 2 spins (not 4)

• Operator is quaternionic (Pauli) matrix involving1, 2

Spectrum of Schwinger matrix & “critical slowing down’’

mass gap = .1mass gap = .1 mass gap = .01mass gap = .01 mass gap = .001mass gap = .001

Stationary Linear Iterations

• Consider solving the linear system using a SLI:

• Let ek=u - uk be the error, and note that rk=Aek. The error

propagation operator is then

, with rk=f - Auk

Multigrid V-cycle

Multigrid solvers are optimal (O(N) operations), and hence have good scaling potential

MG uses a sequence of coarse-grid problems to accelerate the solution of the original problem

smoothing

Fine Grid

Smaller Coarse Grid

restriction

prolongation(interpolation)

Why does standard MG work?

• Low mode is constant -- key to MG success is smooth error, e.g., Laplacian

• Constant exactly preserved on coarse level

• All near zero modes also preserved!

Lattice QCD MG• Gauge field and hence low modes not

geometrically smooth (locally oscillatory)

Geometric MG completely fails,Geometric MG completely fails,preserving low modes for gauge fields requires adaptivity

Real Part Imaginary PartLowest mode of M, ¯ = 6

Lattice QCD & (A)MG

Lattice QCD and MG have a long and painful history (15+ yrs.)

* PTMG (Lauwers et al, 93)* PTMG (Lauwers et al, 93)* Renormilazation MG (de Forcrand et al, 91)* Renormilazation MG (de Forcrand et al, 91)* Projective MG (Brower et al, 91)* Projective MG (Brower et al, 91)* Many others ..* Many others ..

All failed for non-smooth fields in the All failed for non-smooth fields in the m m !! m mcrcr limit, failed because did not limit, failed because did not

consider why MG works in first place!consider why MG works in first place!

Diamond: JacobiDiamond: JacobiCircle: CGCircle: CGSquare: V-cycleSquare: V-cycleStar: W-cycleStar: W-cycle

Adaptive Smooth Aggregation AMG

Adaptive Smoothed Aggregation (SA) Multigrid, Brezina, Falgout, MacLachlan,

Manteuffel, McCormick, and Ruge, SIAM Review, 2006.

Adaptive Smoothed Aggregation Multigrid for Lattice QCD, Brannick, Brezina, Keyes, Livne, Livshits, MacLachlan, Manteuffel, McCormick, and Ruge, Zikatanov,J. Comp. Physics, 2006.

Adaptively use (s)low modes to define the evolving V-cycleAdaptively use (s)low modes to define the evolving V-cycle

Initial Algorithm SetupInitial Algorithm Setup 1. Relax on 1. Relax on Ax = 0Ax = 0, with random initial guess, with random initial guess (resulting error is repesentative of (s)low modes).(resulting error is repesentative of (s)low modes). 2. Cut vector into pieces over aggregates (blocks).2. Cut vector into pieces over aggregates (blocks). 3. Define the prolongator so that 3. Define the prolongator so that x = Px .x = Px . 4. Smooth the vector using simple Richardson 4. Smooth the vector using simple Richardson iteration: iteration: P = (I - P = (I - ¸̧ A)P A)P, , ¸̧ choosen to minimize choosen to minimize condition condition numbernumber of coarse scale operator: of coarse scale operator: A = ((I-A = ((I-¸̧ A)P)A(I- A)P)A(I-¸̧ A)P. A)P.**

cc

cc

General setupGeneral setup adaptive process repeated with current adaptive process repeated with current solver to find additional vectors as neededsolver to find additional vectors as needed

• Adaptive SA designed for problem without underlying geometry• Uses algebraic defintion of “strength of connection” to define aggregates• QCD defined on regular lattice with unitary connections and so use regular

geometric blocking strategy (i.e., 4 x n x n )

• Maintains simple regular geometry on coarse scales, allowing for perfect load balancing and minimal comm. • Original algorithm requires HPD operator and thus we solve M M

Adaptive Smooth Aggregation MG

ddss cc

**

Results for M M

• Schwinger model, 2-d with U(1) background on128 x 128 lattice with ¯ = 6,10, Q = 0,4,

mass gap = 0.001 - 0.5• Use 4 x 4 ( x 2 ) blocking and 3 levels with 8 vectors• Under-relaxed MinRes smoother (Bank and Douglas)• Compare MG-PCG with CG

**

Results…¯ = 6 ¯ = 10

• Critical slowing down eliminated!Critical slowing down eliminated!• No dependence on No dependence on ¯ • Dramatic improvement over CGDramatic improvement over CG

Adaptive Multigrid for QCD, Brannick, Brower, Clark, Osborne, and Rebbi,

Phys. Rev. Letters, sub. 2007.

Adaptive Multigrid for Wilson Fermions, Brannick, Brower, Clark, Osborne,

and Rebbi, Proc. of Science: Lattice 07, sub. 2007.

Results: MG for M¯ = 6, N = 128

• Huge reduction of flopsHuge reduction of flops• Fewer vectors neededFewer vectors needed• Develop. of code for full 4-d QCD underwayDevelop. of code for full 4-d QCD underway

Adaptive Multigrid for the non-hermitian Wilson operator, Brannick, Brower, Clark,

Osborne, and Rebbi, in preparation.

Possible future collaborations

Current projects (with J. Xu & L. Zikatanov) Current projects (with J. Xu & L. Zikatanov)

• Radiation transport Radiation transport • ElectromagneticsElectromagnetics• Oil resevoir simulationsOil resevoir simulations• Fuel cell dynamicsFuel cell dynamics• Stochastic matricesStochastic matrices• Lattice field theoriesLattice field theories