d. scott mcrae aerospace engineering north carolina state university
DESCRIPTION
Prediction of Optical Scale Turbulence with a Typical NWP Code: Lessons Learned (?) Projected to Petascale Computing. D. Scott McRae Aerospace Engineering North Carolina State University NCAR Theme of the Year Workshop May 6, 2008. Acknowledgments. Prior funding: - PowerPoint PPT PresentationTRANSCRIPT
NC STATE UNIVERSITY
Prediction of Optical Scale Turbulence with a Prediction of Optical Scale Turbulence with a Typical NWP Code: Lessons Learned (?) Typical NWP Code: Lessons Learned (?)
Projected to Petascale ComputingProjected to Petascale Computing
D. Scott McRae
Aerospace Engineering
North Carolina State University
NCAR Theme of the Year Workshop
May 6, 2008
NC STATE UNIVERSITYAcknowledgmentsAcknowledgments
• Prior funding: – The US Army Research Laboratory, Battlefield Environments
Division, WSMR and the HELJTO monitored by Dr. David Tofsted, ARLWSMR
• Current funding:– The US Air Force Research Laboratory, Space Vehicles
Directorate, Hanscom AFB, MA through contract
FA8718-04-C-0019; monitored by Dr. George Jumper, AFRL/VSBYA
– NorthWest Research Associates, CORA division, monitored by Dr. Joe Werne
• Contributors – Xudong Xiao, H. A. Hassan, and Yih-Pin Liew, NCSU; Talat Odman, GIT; and Frank Ruggiero,AFRL
NC STATE UNIVERSITYOutlineOutline
• Current goal– Prediction of Optical and Clear Air Turbulence
by modifying existing NWP codes to increase prediction accuracy
• Approach• Examples of current work• Projection to Petascale• Analysis of solution• Numerical Issues• Concluding Remarks
NC STATE UNIVERSITYWhat is optical turbulence?What is optical turbulence?
• The term “Atmospheric Optical Turbulence” refers to fluctuations in the refractive index of air due to turbulence in the atmosphere.– Affects optical propagation by random refraction– Reduces the effective power of optical signals
• Quantitative measure of the intensity of atmospheric optical turbulence: structure parameter of refractive index, Cn
2 ,
• Integration of an accurate prediction of Cn2 along the
beam/viewing path is the primary need for many optical systems
• Accurate prediction requires well resolved dynamics and physics, whether from observation or from atmospheric models. In the latter case, physically accurate turbulence models are required for the scales unresolved by the atmospheric model
NC STATE UNIVERSITYObservation versus Clear 1 ModelObservation versus Clear 1 Model
Jumper, Beland,
2000
NC STATE UNIVERSITYApproachApproach
• Prediction usually required for sub-mesoscale domains– Radiosonde soundings
– Numerical weather prediction codes with parameterizations for
– DNS simulation– Statistical techniques using many sources
• Present approach- modify existing NWP codes to increase accuracy of prediction– LES scale Prediction Using Dynamic 3-D Adaptive Grid– Hybrid LES/RANS Turbulence Model with direct Output-
described by Hassan in a later talk
• Resolution of shear requires adaptation in all three coordinate directions
• Accuracy of prediction can be increased by including more physics of turbulence in the model
2TC
2nC
NC STATE UNIVERSITYModel ModificationsModel Modifications
• NCSU r-refinement Dynamic Solution Adaptive Grid Algorithm (DSAGA) – Resolve selected features/characteristics/properties
dynamically– Criteria selected initially for resolution– Code determines location and resolution automatically– Adapts in all three dimensions
• The NCSU k- hybrid turbulence model
– Four equations based on exact equations derived from the Navier-Stokes and modeled term by term- described by Hassan in a following talk
NC STATE UNIVERSITYResults – 2D caseResults – 2D case
x
z
-50000 0 50000 100000
5000
10000
15000
20000
25000
220 km
2 km
Geometry of the computational domainGeometry of the computational domain
22
2
ax
haz
Mesh size:
221X126
Same setup as in Ref. 14 (by Doyle etc.)
NC STATE UNIVERSITYResults – 2D caseResults – 2D case
u , m/s
z,m
10 20 30 400
5000
10000
15000
20000
25000
Inflow wind speed profile from the Grand Junction, CO, sounding for 1200 UTC 11 January 1972
NC STATE UNIVERSITYResults – 2D case(MILES)Results – 2D case(MILES)
Adaptive mesh at t=3h
x (m)
z(m
)
-20000 0 20000 40000
0
10000
20000
NC STATE UNIVERSITYResults – 2D(MILES)Results – 2D(MILES)
Potential temperature contoursPotential temperature contours
NC STATE UNIVERSITYNumerical Lidar (x= -2km)Numerical Lidar (x= -2km)
time (sec)
z(m
)
5000 10000
4000
6000
8000
10000
12000
14000
16000
18000
20000
22000
24000
w1411852
-1-4-7-10-13-16
NC STATE UNIVERSITYVelocity vectors- 3hoursVelocity vectors- 3hours
x
z
0 20000 40000
0
5000
10000
15000
20000
25000
NC STATE UNIVERSITYDetail- Velocity vectors- 3 hoursDetail- Velocity vectors- 3 hours
x
z
30000 35000 40000 45000 50000
10000
15000
NC STATE UNIVERSITY““Isolated “ vorticesIsolated “ vortices
NC STATE UNIVERSITYVelocity vector animationVelocity vector animation
NC STATE UNIVERSITYCnCn22 contours and balloon trajectory contours and balloon trajectory
0y200000
00
XY
Z
1.30E-165.50E-172.32E-179.82E-184.15E-181.75E-187.41E-193.13E-191.32E-195.60E-202.37E-201.00E-20
NC STATE UNIVERSITYComparison of zonal wind profileComparison of zonal wind profile
u (m/s)
z(m
)
-25 -20 -15 -10 -5 0 5 10 15 20 25
5000
10000
15000
20000
25000ObsStandard MM5 (Ruggiero)Adaptive
NC STATE UNIVERSITYComparison of zonal wind profileComparison of zonal wind profile
u (m/s)
z(m
)
-25 -20 -15 -10 -5 0 5 10 15 20 25
5000
10000
15000
20000
25000ObsUniformAdaptive
NC STATE UNIVERSITYComparison of meridional wind profileComparison of meridional wind profile
v (m/s)
z(m
)
-35 -30 -25 -20 -15 -10 -5 0 5 10
5000
10000
15000
20000
25000ObsStandard MM5 (Ruggiero)Adaptive
NC STATE UNIVERSITYComparison of meridional wind profileComparison of meridional wind profile
v (m/s)
z(m
)
-35 -30 -25 -20 -15 -10 -5 0 5 10
5000
10000
15000
20000
25000ObsUniformAdaptive
NC STATE UNIVERSITYComparison ofComparison of
Cn2 (m-2/3)
z(m
)
10-20 10-19 10-18 10-17 10-16 10-15
5000
10000
15000
20000
25000
ObsUniformAdaptive
2nC
1W V
NC STATE UNIVERSITYComparison of vertical spacingComparison of vertical spacing
z (m)
z(m
)
0 200 400 600 800 1000
5000
10000
15000
20000
25000
UniformAdaptive
NC STATE UNIVERSITYWeight Function Along TrajectoryWeight Function Along Trajectory
Normailzed Weight Function
z(m
)
0 0.2 0.4 0.6 0.8 1
5000
10000
15000
20000
25000
Snapshot of the Adaptive Grid: Tennessee Valley Ozone
SimulationGrid adapting to density of NOx plumes
From Dabberdt W. F. et al., “Meteorological Research Needs for Improved Air Quality Forecasting” Bulletin of the American Meteorological Society, vol. 85, no.4, pp. 563-586, April 2004.
Superior O3 Predictions with Adaptive Grid
Sumner Co., TN
Graves Co., KY
0.0
40.0
80.0
120.0
160.0
1 13 25 37 49 61 73
Time starting from 7/14/1995 (hour)
O3 C
on
cen
trat
ion
(p
pb
)
Observation 4-km Static 8-km Static Adaptive
0.0
40.0
80.0
120.0
160.0
200.0
1 13 25 37 49 61 73
Time Starting from 7/11/1995 (hour)
O3 C
on
cen
trat
ion
(p
pb
)
Observation 4-km Static 8-km Static Adaptive
NC STATE UNIVERSITYScaling Adaptive MM5 to FranklinScaling Adaptive MM5 to Franklin
Processors
Z Grid points
Vertical Spacing
Min Time Step
Wall clock
Memory
Number of points 10^6
Np N1 N2 N3 1 2 3 N z z dt3 Days GBi n
Baseline 1 109 163 121 45 15 5 80 0.26 6.67 8.82 1.9 4.2E+004 X 4 X 1 16 109 163 121 45 15 5 80 0.26 6.67 0.83 0.12 4.2E+00
16 X 16 X 1 256 109 163 121 45 15 5 80 0.26 6.67 0.05 0.01 4.2E+00
4 X 4 X 1 16 436 652 484 11 3.8 1.3 80 0.26 1.67 35.28 1.9 6.8E+01
16 X 16 X 1 256 1744 2608 1936 2.8 0.9 0.3 80 0.26 0.42 141.13 1.9 1.1E+03
4 X 4 X 4 64 436 652 484 11 3.8 1.3 320 0.065 1.67 35.28 1.9 2.7E+028 X 8 X 8 512 872 1304 968 5.6 1.9 0.6 640 0.033 0.83 70.57 1.9 2.2E+0316 X 16 X
16 4096 1744 2608 1936 2.8 0.9 0.3 1280 0.016 0.42 141.13 1.9 1.7E+04
8 X 8 X 8 512 872 1304 968 5.6 1.9 0.6 640 0.033 0.83 45.01 1.21 9.9E+0216 X 16 X
16 4096 1744 2608 1936 2.8 0.9 0.3 1280 0.016 0.42 64.47 0.87 6.4E+0316 X 16 X
16 x 4 16384 1744 2608 1936 2.8 0.9 0.3 20480 0.001 0.42 162.07 2.14 7.8E+04
Nest Grid points Nest Spacings
Scaling to higher resolution
Scaling to higher resolution for all directions, larger in nest 3.
Scaling to higher resolution for all directions
NC STATE UNIVERSITYAdaptation plus PetascaleAdaptation plus Petascale
• Adaptive meshing will move specific resolutions upwards on the previous chart
• Dynamic adaptation may provide a more efficient alternative to standard nesting– Targeted solution dependent weight function determines
automatically where resolution is needed rather than a predetermined nest structure
– Initial budget mapped uniformly unto processors– No mesh boundary errors (however, adaptation is not without
error)
• Unfortunately, present NWP codes are unlikely to scale to full use of a petascale resource due to a combination of factors
NC STATE UNIVERSITYEvaluation of MM5 ResultsEvaluation of MM5 Results
• Filtering/dissipation in code tends to reduce or eliminate structure/frequencies needed for prediction
• Solution does not converge as mesh is refined- S. Koch, NOAA
• Approximately 3-1 reduction in vertical spacing improves structure but still appears over damped. Benefit due to local mesh refinement difficult to assess– LES resolution of turbulence scales not yet achieved
(Terra Incognita – Wyngaard)• WRF-ARW shares basic MM5 approach with
updated algorithms- has 8 identifiable dissipation sources
2nC
NC STATE UNIVERSITYNumerics- MM5Numerics- MM5
• Horizontal integration scheme- time centered explicit (leapfrog)– Neutrally stable for all , for central space– MM5 filtering (Asselin, 1972) for any variable α:
Where for all conditions
– Staggered grid with averaging
1CFL
)2(ˆ 1tt1ttt 1.0
)(O 2
NC STATE UNIVERSITYNumericsNumerics– Vertical coordinate
– Where
– The turbulence model provides an alternative Eddy viscosity
NC STATE UNIVERSITYNumericsNumerics
• Vertical integration scheme- semi-implicit (Klemp and Wilhelmson, 1978)– horizontal results held constant,
With divergence damping added
NC STATE UNIVERSITYAssessment of Damping Assessment of Damping
• prediction derived from local state and spatial variation- spatial filtering is then the issue
• Asselin filter used to stabilize Leapfrog in MM5
Where for all conditionsThis can be expressed as
Assuming linear advection
2nC
)2(ˆ 1tt1ttt 1.0
42
22tt tO
ttˆ
42
22tt tO
xCFLˆ
NC STATE UNIVERSITYAssessment of Damping Assessment of Damping
Which becomes
This implies that spatial damping due to the Asselin filter remains constant relative to the mesh if and CFL remain constant as the mesh spacing is reduced
This filter contributes to mathematical non-convergence
24t1ii1i
2tt x,tO2CFLˆ
NC STATE UNIVERSITYAssessment of Damping Assessment of Damping
Divergence Damping (Skamarock and Klemp, 1992)– Essentially a modification of the normal stress term in the
diagonal of the stress tensor (“Normal Stress Damping”, McRae, 1976)
– Simplified analysis as in Durran – x momentum
where
Discretizing the first term gives
y
v
x
u
xF
x
P
t
uxu
xu
xx
tx
001.02
x
21ii1i xO2001.0
NC STATE UNIVERSITYAssessment of Damping Assessment of Damping
This is a spatial damping that remains constant relative to the mesh as the mesh spacing is reduced
This damping contributes to mathematical non-convergence
Furthermore, divergence damping-– Inserts a modified normal stress into the Euler equations
with an unscaled coefficient of the order of an eddy viscosity
– Interacts with turbulence models/parameterizations resulting in non-physical shear layers
– Biases the pseudo- incompressibility formulation
NC STATE UNIVERSITYEvaluation of MM5 ResultsEvaluation of MM5 Results
• Assessing sensitivity of convergence to individual dissipations time consuming and problematical
• Needed- A technique for assessing the net effect of all of the dissipations applied in the course of the integration– A posteriori ( forensic ) analysis of the solution
• The mesh is a band pass filter and determines the number of Fourier terms available for constructing the solution
x2
2k
L2
2
NC STATE UNIVERSITYTotal Assessment of DampinfgTotal Assessment of Dampinfg
Use Fourier analysis of the solution to ascertain, approximately, how much of the spectrum theoretically resolved by the mesh remains in the solution (ignoring aliasing)
– Sample the solution in a direction in which resolution is important
– Use discrete FFT to obtain approximate frequency versus amplitude distribution
– Infer overall filtration
?kL2
2
NC STATE UNIVERSITYSamplingSampling
x
z
-10000 0 10000 20000
0
5000
10000
15000
20000
25000
30000
NC STATE UNIVERSITYSampled FunctionSampled Function
u
z
0 10 20 30 40 50
5000
10000
15000
20000
NC STATE UNIVERSITYFFT OutputFFT Output
1/
u
-0.02 -0.01 0 0.01 0.02-2
0
2
4
6
8
10
12
x=25mx=50m
NC STATE UNIVERSITY
• Three coordinate dynamic grid adaptation in conjunction with physically based turbulence modeling results in demonstrated improvement in optical turbulence prediction
• However, damping and filtering in present NWP codes limit the improvement
• Some damping types, as used, contribute directly to mathematical non-convergence and may lead to non-physical shear layers, thereby damaging further the optical turbulence prediction
Concluding RemarksConcluding Remarks
NC STATE UNIVERSITY
• Achieving full advantage from use of Petascale computing resources will likely require major changes in the current NWP code equations, algorithms and their implementation
Concluding RemarksConcluding Remarks
NC STATE UNIVERSITY
• We are grateful for the many helpful conversations with the people of AFRL/VSBYA, ARL/WSMR, NWRA, DRI, NCSU MEAS, NCAR and others
Concluding RemarksConcluding Remarks