gtc – gyrokinetic toroidal code training course: part i ihor holod

GTC – Gyrokinetic Toroidal CodeTraining course: Part I

Ihor HolodUniversity of California, Irvine

March 5-7, 2011 GTC Training Course - PKU 2

Outline

• Introduction• Gyrokinetic approach• GK equations of motion. GK Poisson’s equation. Delta-f method• Fluid-kinetic hybrid electron model: continuity equation,

Ampere’s law, drift-kinetic equation• Tokamak geometry. Magnetic coordinates• Particle-in-cell approach. General structure of GTC. Grid.

Parallel computing.• Setup and loading


Introduction

• Gryrokinetic simulations – important tool to study turbulence in magnetically confined plasma• Main target – low frequency microinstabilities• Source of free energy is needed to drive instabilities in plasma

• Pressure gradient of ions, electrons or fast (energetic) particles• Ion temperature gradient mode (ITG)• Trapped electron mode (TEM)• Electron-temperature gradient mode (ETG)• Energetic particle mode (EPM)


Gyrokinetic approach

1~~~~

Ff

BB

Te

• Main advantage: gyroaveraging reduces dimensionality of particle dynamics from 6 to 5 [Hahm et al. PoF’88]• Gyrokinetic ordering:

• Frequency smaller than gyrofrequency• Small fluctuation amplitude

1~sk 1~||

kk

• Additionally

• Magnetic field inhomogeneity is small: 1~ln BB


Charged particle motion in constant uniform magnetic filed


Guiding center motion in the inhomogeneous magnetic filed

dvdtd vbX

0||ˆ 0

dtd

gcd vvv

0

2|| b̂

v

vc

• Magnetic drift velocity

• Magnetic curvature drift

• Magnetic gradient drift

00ˆ B

mg

bv

00|| ˆ B

mdtdv

b

|||| ,,,,, vv, GYingGyroaverag

GC XXvx


• Gyrocenter equations of motion [Brizard and Hahm, RMP‘07]:

Gyrocenter equations of motion

Edvdtd vvbX

0||ˆ

tA

mcZZB

Bmdtdv

||0

0

*|| 1 B

BbBBBB

0||0

0*0

* ˆvB

0

0ˆ

Bc

E

bv

• Modified magnetic field:

• ExB drift velocity:

• Magnetic field perturbation:

0||b̂BB A


Gyrokinetic Vlasov-Maxwell’s system of equations

• Gyrokinetic Poisson’s equation

eiii

ii ennZT

nZ 4~4 2

• Gyrokinetic Ampere’s law

iiiee unZuenc

A ||||||2 4

0||

||

vfvf

tf X

• Gyrokinetic Vlasov equation

• Charge density and current

fvddvmBZZnu

fddvmBZZn

||||0

||

||0


Particle-in-cell method

Monte-Carlo sampling of phase space volume

• Phase space is randomly sampled by Lagrangian “markers”• Number of markers is much smaller than # of physical particles• Dynamics of markers is governed by the same equations of motion as for physical particles• Fluid moments, like density and current, are calculated on Eulerian grid

Fluid moments and potentialsare calculated on stationary grid


Delta-f method• Discrete particle noise – statistical error from MC sampling• Noise can be reduced by increasing # of markers, hence increasing CPU time• Delta-f method to reduce noise:

Perturbed part of the distribution function is considered as additional dynamical quantity associated with particle, and is governed by gyrokinetic equation Fluid moments are calculated integrating contributions from each individual particle

Edvdtd vvbX

0||ˆ

tA

mcZZB

Bmdtdv

||0

0

*|| 1 B

||

0||

0

*

00

00

||ln11ln1vf

mtA

cBZB

Bf

Bvw

dtdw

E BBvB


Fluid-kinetic hybrid electron model - I

• Perturbed diamagnetic drift velocity

PPmn ee

||00

*ˆ1 bv

0lnˆ0*0

0

00

0

||000

Bn

BnB

Bun

Btn

EEee vvvb

• Electron continuity equation

• Perturbed pressure

fmvddvmBP

fBddvmBP

2||||

0||

0||0

• Simulations of various Alfven modes


Fluid-kinetic hybrid electron model - II

• Inverse gyrokinetic Ampere’s law

iiiee unZc

Auenc ||||

2||

44

0

0

)0( ln nnn

Te e

e

eff

• Electron adiabatic response

• Faraday’s law (linear tearing mode is removed)

effEtA

c 00|||| ˆˆ1 bb

0

)0(

0

)0( ln fT

eff

e

effe

• Lowest order solution of electron DKE based on /k||v|| expansion


Fluid-kinetic hybrid electron model - III

t

A

cTev

ff

Te

ff

ff

tfw

ff

dtdw

e

eE

e

ed

eeEe

ee

||||

0

)0(

0

)0(

0

)0(

00

)0(

ln1

vv

v

• High-order electron kinetic response

0

)1()1(

nn

Te e

e

eff

• Zonal flow – nonlinearly self-generated large scale ExB flow with electrostatic potential having k||=0


Tokamak geometry

0 g I B

• Equilibrium magnetic field (covariant representation) in flux coordinates

1d

d q

• Helicity of magnetic field is determined by safety factor q (number of toroidal turns of magnetic field line per one poloidal turn)

Magnetic flux surface


Magnetic coordinates - definition• Covariant representation of the magnetic field

Ig0B

• Contravariant representation of the magnetic field q0B

• Jacobian

)()()(),(201

Iqg

BJ

OOOOg

OI

OB

0

0

2

2

20

01

0

cos1


Magnetic coordinates - example

• Parallel derivative

qJqq

q

1

0B

1J


GTC grids

• Two spatial grids• Coarse grid for equilibrium and profiles

• Field-aligned simulation mesh for fields

• Spline interpolation is used to calculate equilibrium quantities on simulation grid and particle positions


Simulation grid• Radial grid in (i=0:mpsi)

• Poloidal angle () grid with continuous indexing (ij=1:mgrid) over the whole poloidal plane

• Number of poloidal grid points mtheta(i)differs from one flux surface to another

mtheta(i)2

integerj jigrid(i)ij

ij igrid(0)=1

do i=1,mpsi

igrid(i)=igrid(i-1) + mtheta(i-1) + 1

enddo


Parallel computing• Distribution of independent tasks between different processors

• Domain decomposition: each processor simulates it’s own part of simulation domain

• Particle decomposition: each processor is responsible for it’s own set of particles

Node 0

Core 0 Core 1

Core 2 Core 3

Node 1

Core 0 Core 1

Core 2 Core 3

Supercomputer structure


MPI-Parallelization

MPI_BCAST(send_buff,buff_size,data_type,send_pe,comm,ierror)

MPI_REDUCE(send_buff,recv_buff,buff_size,data_type,operation,recv_pe,comm,ierror)

MPI_ALLREDUCE(send_buff,recv_buff,buff_size,data_type,operation,comm,ierror)

• Parallelization between nodes

• Each node has independent memory

• Inter-node communication through special commands


OpenMP-Parallelization

!$omp parallel do private(m)

do m=1,mi

…

enddo

• Parallelization between cores of one node

• Each core has access to common memory

• Used for parallelization of cycles

• Useful with modern multi-core processors (GPU)


ne A||

ue

ge1fi

nene1uiA|| ni ue1

indesA|| ZF

Dynamics

Sources

Fields

GTC structure


main.F90 CALL INITIAL(cdate,timecpu) CALL SETUP CALL LOAD CALL LOCATEI if(fload>0)CALL LOCATEF! main time loop do istep=1,mstep do irk=1,2! idiag=0: do time history diagnosis at irk=1 & istep=ndiag*N idiag=mod(irk+1,2)+mod(istep,ndiag) if(idiag==0)then if(ndata3d==1)CALL DATAOUT3D !write 3D fluid data if(track_particles==1)CALL PARTOUT !write particle data endif CALL FIELD_GRADIENT if(magnetic==1)CALL PUSHFIELD CALL PUSHI CALL SHIFTI(mimax,mi) CALL LOCATEI CALL CHARGEI if(fload>0)then CALL PUSHF CALL SHIFTF(mfmax,mf) CALL LOCATEF CALL CHARGEF endif if(magnetic==0)then CALL POISSON_SOLVER("adiabatic-electron") else CALL POISSON_SOLVER("electromagnetic") CALL FIELD_SOLVER endif


main.F90 (cont)

! push electron, sub-cycling do i=1,ncycle*irk! 1st RK step CALL PUSHE(i,1) CALL SHIFTE(memax,me)! 2nd RK step CALL PUSHE(i,2) CALL SHIFTE(memax,me)! electron-ion and electron-electron collision if(ecoll>0 .and. mod(i,ecoll)==0)then call lorentz call collisionee endif enddo CALL CHARGEE CALL POISSON_SOLVER("kinetic-electron") enddo if(idiag==0)CALL DIAGNOSIS enddo! i-i collisions if(icoll>0 .and. mod(istep,icoll)==0)call collisionii

if(mod(istep,mstep/msnap)==0 .or. istep*(1-irun)==ndiag)CALL SNAPSHOT if(mod(istep,mstep/msnap)==0)CALL RESTART_IO("write") enddo CALL FINAL(cdate,timecpu)


if(numereq==1)then ! Cyclone case psiw=0.0375 !poloidal flux at wall! q and zeff profile is parabolic: q=q1+q2*psi/psiw+q3*(psi/psiw)^2 q=(/0.82,1.1,1.0/) !cyclone case er=(/0.0,0.0,0.0/) itemp0=1.0 ! on-axis thermal ion temperature, unit=T_e0 ftemp0=2.0 ! on-axis fast ion temperature, unit=T_e0 fden0=1.0e-5 ! on-axis fast ion density, unit=n_e0! density and temperature profiles are hyperbolic: ! ne=1.0+ne1*(tanh(ne2-(psi/psiw)/ne3)-1.0) ne=(/0.205,0.30,0.4/) !Cyclone base case te=(/0.415,0.18,0.4/) ti=(/0.415,0.18,0.4/) ! ITG tf=ti ! fast ion temperature profile nf=ne ! fast ion density profile

analytic.F90

• Specifies profiles for different analytic equilibriums


! allocate memory for equilibrium spline array allocate (stpp(lsp),mipp(lsp),mapp(lsp),nepp(3,lsp),tepp(3,lsp),& nipp(3,lsp),tipp(3,lsp),zepp(3,lsp),ropp(3,lsp),erpp(3,lsp),& qpsi(3,lsp),gpsi(3,lsp),ppsi(3,lsp),rpsi(3,lsp),torpsi(3,lsp),& bsp(9,lsp,lst),xsp(9,lsp,lst),zsp(9,lsp,lst),gsp(9,lsp,lst),& rd(9,lsp,lst),nu(9,lsp,lst),dl(9,lsp,lst),spcos(3,lst),spsin(3,lst),& rgpsi(3,lsp),cpsi(3,lsp),psirg(3,lsp),psitor(3,lsp),nfpp(3,lsp),& tfpp(3,lsp))

if(mype==0)then if(numereq==0)then ! use EFIT & TRANSP data! EFIT spdata used in GTC: 2D spline b,x,z,rd; 1D spline qpsi,gpsi,torpsi call spdata(iformat,isp)! TRANSP prodata used in GTC: tipp,tepp,nepp,zeff,ropp,erpp call prodata else! Construct analytic equilibrium and profile assuming high aspect-ratio, concentric cross-section call analyticeq endif

eqdata.F90• Equilibrium and profiles are represented using 2D-spline on (psi, theta) or 1D-spline on (psi)


!$omp parallel do private(i,pdum,isp,dpx,dp2) do i=0,mpsi pdum=psimesh(i) isp=max(1,min(lsp-1,ceiling(pdum*spdpsi_inv))) dpx=pdum-spdpsi*real(isp-1) dp2=dpx*dpx

! 1D spline in psi meshni(i)=nipp(1,isp)+nipp(2,isp)*dpx+nipp(3,isp)*dp2 kapani(i)= 0.0-(nipp(2,isp)+2.0*nipp(3,isp)*dpx)/meshni(i) ... enddo

setup.F90• Read input parameters

• Initialize quantities determined on equilibrium mesh

! allocate memory allocate(phi(0:1,mgrid), gradphi(3,0:1,mgrid),...,STAT=mtest)

• Allocate fields


!iload=0: ideal MHD with nonuniform pressure profile!iload=1: Marker uniform temperature & density: n0,te0,ti0 (two-scale expansion) !iload=2: Marker non-uniform temperature & uniform density: n0,te(r),ti(r), marker weight corrected!iload=3: Physical PDF for marker with non-uniform temperature & density: n(r),te(r),ti(r)!iload>99: non-perturbative (full-f) simulation

loadi.F90• Loads ion markers

zion(nparam,mimax)nparam=6 (8 if particle tracking is on)zion(1,m) – zion(2,m) – zion(3,m) – zion(4,m) – (parallel velocity)zion(5,m) – wi (particle weight)zion(6,m) – (magnetic moment)

• Marker data array:

gtc – gyrokinetic toroidal code training course: part i ihor holod

Documents