using xdb workflows to analyze high lift drag prediction

Applied Research Group Seeking Answers, Deploying Solutions Using XDB Workflows to Analyze High Lift Drag Prediction Workshop Simulations ARG‐13‐01

DR. EARL P.N. DUQUE

MANAGER OF APPLIED RESEARCH – INTELLIGENT LIGHT

ARG-13-01 (CASE STUDY) Page - i

Intelligent Light 301 Route 17 North, 7th Floor, Rutherford, NJ 07070, USA +1 (201) 460-4700 Copyright © 2013 Intelligent Light - All trademarks are the property of their respective owners

Table of Contents Table of Contents ........................................................................................................................................................ i

XDB Workflows for Large Scale CFD Post‐Processing ................................................................................................ 1

OVERFLOW on Kaibab (A Cray XE6) ....................................................................................................................... 4

Using FieldView on Kaibab .................................................................................................................................... 4

XDB Data reduction ............................................................................................................................................... 5

Workshop Results and FVX Examples ....................................................................................................................... 6

Extract ‐ Boundary Surface .................................................................................................................................... 6

Surface Pressure Coefficients Comparisons .......................................................................................................... 6

Multi‐plot comparisons using Gnuplot .................................................................................................................. 7

Velocity comparisons ............................................................................................................................................ 7

Surface streamlines ............................................................................................................................................... 8

Summary .................................................................................................................................................................. 10

Works Cited ............................................................................................................................................................. 10

ARG-13-01 (CASE STUDY) Page - 1


Using XDB Workflows to Analyze High Lift Drag Prediction Workshop Simulations This case study presents how the CFD flow solver, OVERFLOW2, and Intelligent Light’s CFD post-processor, FieldView, was used in a remote solving and remote post-processing environment to analyze solutions for the AIAA High Lift Drag Prediction Workshop. Using the OVERFLOW2 solver from NASA, over 62 solutions were obtained on a Cray corporate XE6 system remotely located in Minnesota. The large volumes of data were stored on the remote system where FieldView was used in batch and interactive modes with up to 16 processors in parallel. An extract workflow was applied to create FieldView surface extract databases (XDB’s), line extracts and streamlines which reduced the data by up to a factor of 275. The surface, line and streamline extracts were then transferred to a local laptop where FieldView FVX scripts, together with GNUplot were used to automatically generate comparisons between the cases and against experimental data. This case study summarizes this effort and provides examples on how to use FieldView, FVX and XDB workflows to automate the post-processing of large scale simulations.

XDB Workflows for Large Scale CFD Post‐Processing The AIAA High Lift Prediction workshop consisted of 2 cases. Case 1 was a Grid Convergence Study with four grid refinements each computed at six (6) angles of attack. The geometry for this grid system consisted of just the wing, slat, flap and fuselage. There is a total of 44 grids for each grid system which vary in size: Coarse = 29.4 million grid points, Medium= 69 million, Fine = 230.7 million, Extra Fine = 544.5 million grid points.

Figure 1: Case 1 Overset Grid System (Coarse Grid)



FieldView Version 14 and its multi-window graphics capability was used to compare many of the different cases. Figure 1 illustrates some of the details of the grid at the coarse grid level. Figure 2 provides details of the surface definition at two key locations – the wing-body junction at the rear and front of the wing. Both locations have been identified as possible regions of concern particularly for side of body separation phenomena.

Figure 2: Case 1 Grid Refinement Focused on Side of Body Fairing (SOB) & Slat and Wing Body Junction

Case 2 was a turbulence model study with four (4) turbulence model variations [low Reynolds, SA, kw-SST, Menter-Langtry Transition, High Reynolds SA], eight (8) angles of attack. The main difference between the Case 2 and the Case 1 grid is the inclusion of the slat and flap attachment hardware as highlighted in Figure 3. This overset grid system consists of 163 grids totaling 97.2 million grid points.

Figure 3: Surface Grids Highlighting the Slat and Flap Attachment Hardware and Side of Body Fairing

Case 1 and Case 2 totaled sixty-two (62) individual solutions totaling approximately 2.7 terabytes of data. Post-processing and managing this large set of data requires automation to quickly create images that compare the solutions to each other and to experimental data. However, the large size and remote location prevents moving the data from the large storage systems.



Figure 4 illustrates a typical Volume Data file-based post-processing workflow. In this workflow, the solver has written files to disk of the complete grid and solution volume. The post-processor reads the volume data and then computes the various graphics based objects such a the geometric surface, coordinate cutting planes, iso-surfaces of arbitrary scalar functions, streamlines, etc. After the post-processing objects are created, they are then further processed to render a graphical image, integrate scalar functions on the surface to yield integral functions like force and moments, and plot values on the surface such as Pressure Coefficient Distributions.

Much of the computational cost and wall time is taken up by the post-processing tool reading in the volume dataset and creating the post-processing objects. In a typical workflow, the post-processing objects are thrown away at the end of the session. In a FieldView XDB workflow, the post-processing objects are saved as an XDB file for future and repeated use.

Figure 4: Traditional Volume Data Post‐Processing

Furthermore, the XDB file can be created by executing FieldView in batch on a large HPC resource or created interactively and saved. In either method, the XDB file can then be read into FieldView where all the other post-processing actions can be performed on the XDB extracts. Since the extracts are at the fidelity of the original data, there is no loss in accuracy yet several orders of magnitude of data reduction can be obtained.

Figure 5: eXtract DataBase (XDB) Post‐Processing



As mentioned, an XDB workflow utilizes small data files. XDB’s are typically on the order of 10-100x smaller than the volume data. The data can be read back into FieldView very quickly and allows for a much more efficient workflow particularly when working with data that has been created on remote servers and on relatively slow networks.

OVERFLOW on Kaibab (A Cray XE6)

The OVERFLOW2 code version 2.0e was used for the current study [1] with all of the computations by the flow solver performed on a computer system called “Kaibab” located at Cray Inc. Kaibab is a Cray XE6 machine which at the time that most of these computuations were performed was configured with 96 IL16 nodes (96*32=3072 Processing Elements (pes)) with 2100Mhz 16-core chips, 32GB memory, 84 MC12 nodes (84*24=2016pes) with 2000Mhz 12-core chips, 32GB memory, DDN 3000 Raid and 12 OST lustre file system. OVERFLOW2 was compiled using Intel compilers (version 13) with OpenMP and MPICH2. Depending upon the load of the system and memory requirements of each of the OVERFLOW runs, the computations were performed across 128 to 1056 processors.

Using FieldView on Kaibab

The FieldView XDB surfaces of the wing, slat, flap and fuselage boundary surfaces, x-y-z Coordinate Cut Planes, velocity profile line extracts and surface constrained streamlines were all created using parallel processing with up to 16 distributed processors directly on Kaibab using the Cray “Cluster Compatibilty Mode” (CCM). “CCM is a software solution that provides the services needed to run most cluster-based independent software vendor (ISV) applications out-of-the-box with some configuration adjustments. It is built on top of the compute node root runtime environment (CNRTE), the infrastructure that provides dynamic library support in Cray systems.” [2] Within CCM, FieldView ran either in Client-Server mode for interactive sessions or in batch mode, just as if it were running on any x86-64 based cluster.

The FieldView post-processing software (versions 13.2 and the beta version of version 14) was used for this study. For interactive mode, the FieldView client was initiated on a windows based remote laptop computer 1. A Virtual Private Network (VPN) session to the Cray Inc. network was established that enabled the local laptop client to connect to either Kaibab’s head node or a master compute node as assigned by the PBS scheduler. To establish the connection, the FieldView client session was entered into “Manual Mode”. When the user selects a reader, FieldView uses a popup window to notify the user client’s IP address and the command to execute on the server in order to connect. The FieldView client listens via an open secure SSH port for a connection request by the FieldView server master process. Appendix A presents a script that the user executes on the server side which creates a PBS job submission script and automatically submits a job to the PBS queue (./scriptAA IP.Client) 2.

1 HP EliteBook 8730w, 2 cores, 8GB RAM, NVIDIA Quadro FX2700M, Bandwidth 512 KB/second 2 From David Whitaker, Cray Inc.



Once the job has begun executing on the server, monitored by “qstat”, the client side FieldView could connect.

For the batch job of FieldView, Appendix B presents an example PBS job submission script that was used to create the boundary surface extracts. The line:

ccmrun fv -batch -fvx surface_boundaries.fvx

executes the FVX script “surface_boundaries.fvx” in the CCM environment, Appendix C. This FVX script requests to read the dataset and execute in parallel on 16 processors as requested in the server_config line:

server_config = "kaibab_p4"

The server configuration file “kaibab_p4.srv” defines the setup of the parallel server session. This file may be stored in the sconfig directory of the FieldView main install ($FV_HOME/sconfig) on the server or it may be stored in a user defined location on the server if the environment variable FV_SERVER_CONFIG_DIR is defined.3 Once stored, this configuration file can be re-used by anyone who wishes to start on the same resource.

XDB Data reduction

Table 1 presents the data reduction that was attained by using an XDB workflow. For the coordinate cut planes in the span direction (y), with 7 scalars stored on those planes we see a 167.2 to 449 fold data reduction. For the Boundary Surfaces (fuselage, flap, slat and wing) and 12 scalar variables including the Pressure Coefficient and the Shear Stress Tensor there is a 98 to 275 fold reduction in data. If one considers both the Boundary Surfaces and the coordinate cut planes, the data reduction is 62 to 170 fold.

Table 1: Case 1 Data Reduction Using XDB's

3 See FieldView User’s Manual version 13.2, page 19 for more details



Workshop Results and FVX Examples

Extract ‐ Boundary Surface

In order to create the surface pressure coefficent plots, skin friction coefficient plots and the surface streamline images, boundary surfaces of the no-slip surfaces were created and saved to XDB files. For case 1, boundary surfaces for the Slat, Wing, Flap and Fuselage were created. For case 2, additional boundary surfaces for the attached hardware (Pods) were also created. Appendix C presents the FVX script that created the boundary surface XDB for case 1.

To specify the scalar quantities that are output to the XDB, an “xfn” file was used to declare that the scalars of pressure coefficient, skin friction magnitude and the skin friction vector were to be written to the XDB. The Skin Friction is not a standard derived function in FieldView’s OVERFLOW2 reader. A custom function was created using a Formula restart file (surface_boundaries.frm) to create the skin friction magnitude and vector.

Surface Pressure Coefficients Comparisons

Figure 6: Case 1, =22.4, Presuure Coefficient Composite

To avoid having to move the large volumetric grid and solutions off of the remote Cray system, all the boundary surface XDB’s were created in batch on the remote system, transferred to a local system and then further processed using FieldView and FVX scripts.

The surface pressure taps from the experiment were defined along the wing at constant y-axis coordinate locations. The flap and slat presure tap locations however, are not aligned with a coordinate axis. Instead, the committee provided planar definitions for each tap row location [3]. An FVX script was then used to create the surface pressure datafiles for plotting. To create the plotting data, the FVX script, Appendix I, loads the boundary surface XDB and then creates a “slice” of the part



in question (slat, flap or wing) with some small thickness by using a threshold function based upon the planar function defined by the committee. Since the slice consists of a set of polygons, the geometric center of each polygon is determined and then the pressure and friction data is interpolated from the polygon nodes to the polygon center. The FVX automatically sweeps through all angles of attack and grid resolutions and outputs the commmittee defined datafiles for 2-D plotting.

Multi‐plot comparisons using Gnuplot

Once the 2-D plotting data has been created, a bash script was used to generate Gnuplot script files which rendered multi-plot images for all cases, angles of attack and grid refinement levels. Figure 7 illustrates examples of multi-plot comparisons for Case 1 medium grid at an angle of attack of 18.5 degrees.

Figure 7: Case 1, High Reynolds Number, Medium Grid, =18.5o

Velocity comparisons

The interaction between the shear and boundary layers of the slat, flap and wing has a significant effect upon the performance of any multi-element wing. The ability to capture the interaction depends upon the grid resolution at the interaction, the numerics and the turbulence model utilized.

All the line plots for this case study were created in batch using FieldView’s 2-D line plot tool upon the volume data. Here, an FVX script is used to successively export 2-D line plot data from the volumetric data. The script loads the grid and solution and then creates and exports the line data to a file as specified by the committee. The exported line plot data is then transferred to a local workstation where



an FVX script is used to convert the line plot exported data into the appropriate standardized data form. Another script is then used to automatically create the multi-plot comparisons. Figure 8 illustrates the resulting multi-plot comparison Case 1 at an angle of attack of 18.5o for all grids as compared to the low Reynolds number experimental data obtained from PIV.

Figure 8: Case 1: Effect of Grid Resolution upon Velocity Profiles, 18.5º Angle of Attack

Surface streamlines

Surface streamlines visualizations are a classic method that mimics experimental oil flow visualizations. Surface streamlines were created in FieldView using the “Vortex Cores/Surface Flows Visualization Panel” and the “Surface Restricted Flow: No Slip” Feature. The setup for the vortex cores was first performed interactively with the coarse dataset in order to create a Vortex Core/Surface Flow restart. Once the vortex core restart (i.e. vcore.vtx) is created it can be re-used for all the cases (Case 1 and Case 2) to define and compute the surface flows.

FieldView was executed on Kaibab in batch mode and parallel with 16 processors to create the surface streamlines. This FVX sweeps through all the angles of attack for a given case grid refinement level and results in a FieldView particle file (i.e. “surface_streams_wing_7.fvp”). The resulting fvp file is then transferred to the local client for interactive viewing and creation of visualization scenes of interest.

-70

-60

-50

-40

-30

-20

-10

0

10

0 0.5 1 1.5 2 2.5 3

P1 WB L1

-50

-40

-30

-20

-10

0

10

20

0 0.5 1 1.5 2 2.5 3

P1 WB L2

-40

-30

-20

-10

0

10

20

30

40

0 0.5 1 1.5 2 2.5 3

P1 wC L1

-60

-50

-40

-30

-20

-10

0

10

20

0 0.5 1 1.5 2 2.5 3

P1 wD L1

-10

0

10

20

30

40

50

60

0 0.5 1 1.5 2 2.5 3

P2 wB L1

-10

0

10

20

30

40

50

60

0 0.5 1 1.5 2 2.5 3

P2 wB L2

-10

0

10

20

30

40

50

60

0 0.5 1 1.5 2 2.5 3

P2 wD L1

-20

-10

0

10

20

30

40

50

0 0.5 1 1.5 2 2.5 3

P2 wE L1

-40

-30

-20

-10

0

10

20

30

0 0.5 1 1.5 2 2.5 3

P2 wE L2

0

10

20

30

40

50

60

70

0 0.5 1 1.5 2 2.5 3

P3 wE L1

-20

-10

0

10

20

30

40

50

0 0.5 1 1.5 2 2.5 3

P3 wE L2

exp

crse

med

fine

xfine



Figure 9: Case 1 Surface Streamlines Showing Effect of Grid Resolution Upon the Side of Body Flow, =18.5o

Figure 9 presents a multi-window display of the Case 1 surface stream lines and highlights a rear view of the wing-body junction at 18.5o angle of attack. The image was created using FVX and FieldView version 14 which features the new multi-window display. The FVX automatically loads the previously saved Boundary Surfaces XDBs and surface streamline particle file for the dataset of interest. The datasets are distributed into a 2 by 2 display and then automatically annotated accordingly.

The Multi-Window display is a versatile feature for comparing many different datasets. Each dataset can be rendered in its own window. The viewpoint transforms can be linked across the windows so that each window replicates the same viewpoint as one moves the images. Each window can also display other types of data.

For example, Figure 10 shows a 3 by 2 multi-window display. The top row presents a view of the surface streamlines and friction coefficient on the boundary surface. The bottom row presents the digital photographs from the actual wind tunnel experiment. This picture allows for the quick comparison of the CFD results against each other and the experiment.



Figure 10: Case 2 ‐ Surface Streamlines an Friction Coefficient vs Experimental Data

Summary The results for the AIAA High Lift Prediction Workshop II presented here were generated using the OVERFLOW2 code (version 2e) running on a Cray XE6 with OVERFLOW2 grid and solver inputs as provided by the committee. Given the large number of cases and the size of results files, a FieldView XDB workflow was used in batch on the Cray system to post-process the volume data and reduce the size of the data. The XDB workflow created line and surface extracts that were several orders of magnitude smaller than the original volume data. Once transferred to a local workstation, the extracts were used to easily create multi-plot images and comparisons against experimental data. The results were presented at the workshop held at the AIAA Fluid Dynamics Conference, July 13-14, 2013.

Works Cited

[1] P. G. Buning, R. J. Gomez e W. I. Scallion, “CFD Approaches for Simulation of Wing‐Body Stage Separation,”

AIAA‐2004‐4838, AIAA 22nd Applied Aerodynamics Conference, Providence, RI, 2004.

[2] “Workload Management and Application Placement for the Cray Linux Environment™ ‐ S–2496–4101,” Cray

Inc, [Online]. Available: http://docs.cray.com/books/S‐2496‐4101/html‐S‐2496‐4101/chapter‐9b6qil6d‐

craigf.html. [Accessed on 2 Aug 2013].

[3] NASA Langley Research Center, “ EXPERIMENTAL DATA: Forces, Moments, Pressure Coefficients,” 04

February 2013. [Online]. Available: http://hiliftpw.larc.nasa.gov/Workshop2/model‐sketch‐



dimensions_hiliftpw2‐f11.pdf. [Accessed on 3 August 2013].

[4] “gnuplot homepage,” April 2013. [Online]. Available: http://www.gnuplot.info/. [Accessed on 2 Aug 2013].

[5] P. R. Spalart e S. R. Allmaras, “A One‐Equation Turbulence Model for Aerodynamic Flows,” AIAA Paper 92‐

0439, 1992.

[6] D. S. Chaussee e T. H. Pulliam, “Two‐ Dimensional Inlet Simulation Using a Diagonal Implicit Algorithm,” AIAA

Journal, vol. 19, n. 2, pp. 153‐160, 1981.

[7] M. L. Shur, M. K. Strelets, A. K. Travin e P. R. Spalart, “Turbulence Modeling in Rotating and Curved Channels:

Assessing the Spalart‐Shur Correction,” AIAA Journal, vol. 38, n. 5, pp. 784‐792, 2000.



AppendixA–ScripttostartaninterativePBSrequestandconnectFieldView#!/bin/csh \rm -rf /tmp/scriptAA cat > /tmp/scriptAA << EOD #!/bin/csh #PBS -q ccm_queue #PBS -lmppwidth=16 #PBS -lmppnppn=1 #PBS -N fvserver #PBS -l walltime=8:00:00 # setenv FV_HOME /home/users/n16326/bin/fv setenv CLIENT $1 cd `pwd` setenv PATH \${FV_HOME}/bin:\$PATH source ${MODULESHOME}/init/csh module load ccm setenv tmp_list \`sort ~/.crayccm/ccm_nodelist.\${PBS_JOBID} | uniq -c | awk '{ printf("%s:%s ", \$2, \$1) ; }'\` setenv proc_list \`echo \$tmp_list | tr " " ","\` setenv proc_count \`cat ~/.crayccm/ccm_nodelist.\${PBS_JOBID} | wc -l\` echo proc_count=\$proc_count echo proc_list=\$proc_list echo CLIENT=\$CLIENT ccmrun fvrunsrv -np \$proc_count -hosts \$proc_list \$CLIENT -conn_to 120 EOD qsub /tmp/scriptAA n16326@kaibab:~/bin> cat scriptAA \rm -rf /tmp/scriptAA cat > /tmp/scriptAA << EOD

AppendixB–FieldViewBatchPBSJobSubmissiononKaibab,16corep4server #!/bin/csh #PBS -q ccm_queue #PBS -lmppwidth=16 #PBS -N coarse_post #PBS -l walltime=8:00:00 # setenv FV_HOME /home/users/n16326/bin/fv setenv FV_SERVER_CONFIG_DIR $FV_HOME/sconfig # # change pwd to the directory that the job was launched from # cd $PBS_O_WORKDIR # source ${MODULESHOME}/init/csh # # Create a list of the hosts assigned to this job # rm -f machinelist cat $PBS_NODEFILE > machinelist



#- # Load the CCM environment # module load ccm # # Execute FieldView in Batch in the ccm environment and run # the fvx script “surface_boundaries.fvx # ccmrun fv -batch -fvx surface_boundaries.fvx

AppendixC–FieldViewFVXforSurfaceBoundaryCreationontheCase1Grids -------------------------------------------------------------------------------- -- Copyright (c) 2012 Intelligent Light -- -- All rights reserved. -- -- -- -- This sample FVX script is not supported by Intelligent Light -- -- and Intelligent Light provides no warranties or assurances -- -- about its fitness or merchantability. It is provided at no -- -- cost and is for demonstration purposes only. -- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- -- DATA INPUT -------------------------------------------------------------------------------- AngleOfAttack = { "7", "16", "18.5", "20", "21", "22.4"} fv_script("RESTART DATA surface_boundaries") for nAlpha=1,getn(AngleOfAttack) do local datasets_info_table = {} datasets_info_table[1] = read_dataset( { data_format = "overflow-2", server_config = "kaibab_p4", input_parameters = { q_file = { name = "./data/q."..AngleOfAttack[nAlpha], options = { format = "dp_unformatted", input_mode = "replace", coords = "3d", multi_grid = "on", iblanks = "on" } -- options }, -- q_file } -- input_parameters } ) -- read_dataset -- print_dataset_table( datasets_info_table[1] ) -------------------------------------------------------------------------------- -- FORMULAS -------------------------------------------------------------------------------- fv_script("RESTART FORMULA Friction_Coefficient.frm") -------------------------------------------------------------------------------- -- BOUNDARY SURFACES -------------------------------------------------------------------------------- local boundary_surfs={}



boundary_surfs[1] = create_boundary( { geometric_color = 4, transparency = 0, types = { "BODY", }, -- types display_type = "smooth_shading", contours = "none", show_mesh = "off", number_of_contours = 16, vector_func = "none", scalar_func = "none", visibility = "on", threshold_func = "none", line_type = "thin", dataset = 1, } ) -- boundary_surfs[1] boundary_surfs[2] = create_boundary( { geometric_color = 4, transparency = 0, types = { "FLAP", }, -- types display_type = "smooth_shading", contours = "none", show_mesh = "off", number_of_contours = 16, vector_func = "none", scalar_func = "none", visibility = "on", threshold_func = "none", line_type = "thin", dataset = 1, } ) -- boundary_surfs[2] boundary_surfs[3] = create_boundary( { geometric_color = 4, transparency = 0, types = { "SLAT", }, -- types display_type = "smooth_shading", contours = "none", show_mesh = "off", number_of_contours = 16, vector_func = "none", scalar_func = "none", visibility = "on", threshold_func = "none", line_type = "thin", dataset = 1, } ) -- boundary_surfs[3] boundary_surfs[4] = create_boundary( { geometric_color = 4,



transparency = 0, types = { "WING", }, -- types display_type = "smooth_shading", contours = "none", show_mesh = "off", number_of_contours = 16, vector_func = "none", scalar_func = "none", visibility = "on", threshold_func = "none", line_type = "thin", dataset = 1, } ) -- boundary_surfs[4] fv_script("XDB_WRITE surface_boundaries") execute("mv surface_boundaries.xdb coarse_boundaries_"..AngleOfAttack[nAlpha]..".xdb") end

using xdb workflows to analyze high lift drag prediction

Documents