Firebolt
06/27/2011 Page-1 Approved for public release; distribution is unlimited.
Firebolt - A High Fidelity
Airframe/Propulsion Integration Tool
R. H. Nichols
UAB/AEDC
931-454-3431
NDIA Physics-Based Modeling in Design &
Development for U.S. Defense Conference
November 14-17, 2011
Firebolt
06/27/2011 Page-2 Approved for public release; distribution is unlimited.
CREATE Program Overview
Firebolt
06/27/2011 Page-3 Approved for public release; distribution is unlimited.
GFLOPs:World’s
Fastest Computer
1940 1950 1960 1970 1980 1990 2000 2010 2020Year
1
106
10-3
103
109
10-6
Cores
105
Cores
108
DRAM Speed Trend
Computational Performance Trends
Firebolt
06/27/2011 Page-4 Approved for public release; distribution is unlimited.
Traditional “Design-Test-Build-Fix”
Acquisition
Firebolt
06/27/2011 Page-5 Approved for public release; distribution is unlimited.
CREATE Project Genesis
CREATE-AV CREATE-SH CREATE-RF CREATE-MG
• In 2007, DoD HPCMO proposed an initiative to develop and
deploy Computationally Based Engineering (CBE) tools that
exploit next generation HPC computer resources to improve
DoD acquisition processes for aircraft, ships, and RF
antennas.
• CREATE is a sustained 12 year development program to
develop production quality design and analysis software that
is adaptable and maintainable.
Firebolt
06/27/2011 Page-6 Approved for public release; distribution is unlimited.
CREATE-AV Software Products
• Fixed Wing A/C (Kestrel)
– Hosted at Eglin AFB
• Rotary Wing (Helios)
– Hosted at ARMDEC (AMES)
• Airframe/Propulsion Integration
(Firebolt )
– Hosted at AEDC
• Conceptual Design (DaVinci)
• Hosted at WPAFB
Firebolt
06/27/2011 Page-7 Approved for public release; distribution is unlimited.
Infrastructure
CFD
Solver
Structural
Solver
Mesh
Deformer
Fluid - Structure
Interface
Control Surface
Deflection
Aircraft
Trim
On - the - Fly
Visualizer Engine Thrust
Model Autopilot 6DOF
Store - Release
Constraints
Prescribed
Motion
Rigid - Grid
Move
Mesh
Adaptation
Integrated Force &
Moment Calculator
CFD
Solver
Autopilot
Additional
Executables
Common Scalable Infrastructure (CSI)
CFD
Solver
CSM
Solver
Mesh
Deformer
Fluid - Structure
Interface
Control Surface
Deflection
Aircraft
Trim
On - the - Fly
Visualizer Firebolt Autopilot 6DOF
Store - Release
Constraints
Prescribed
Motion
Rigid - Grid
Move
Mesh
Adaptation
Integrated Force &
Moment Calculator
CREATE Provided Executables
CFD
Solver
Autopilot
Additional
Executables
• Cross-over between aerodynamics, stability and control, structures, propulsion, store separation
• Improved performance on parallel architectures with from 104 to 106 cores (giga-flop to peta-flop)
• Event driven architecture
• CSI Infrastructure and Executive in Python, KUI Graphical User Interface in wxPython
• Components programmed in F90/95/03 and C
• All APIs and data structures for components consistent across CREATE-AV products
Kestrel Fixed Wing Virtual Aircraft
Description
Firebolt
06/27/2011 Page-8 Approved for public release; distribution is unlimited.
Provide CBE Tools for Both the Near-Term and the Long-Term
Firebolt
06/27/2011 Page-9 Approved for public release; distribution is unlimited.
CREATE-AV CFD Code Philosophy
Control and couple solvers with an
agile infrastructure
– Overset communication
– Common heavy data (pass
pointers)
Use the best solver or solvers for
the problem
– Near body solvers (structured,
unstructured, strand)
– Off body solvers (cartesian)
Use common modules
– Gas properties
– Chemical source terms
– Grid deformation
– AMR
Use common procedures
– Turbulence models
– Low Mach number
preconditioning
Firebolt
06/27/2011 Page-10 Approved for public release; distribution is unlimited.
Firebolt Module
Firebolt
06/27/2011 Page-11 Approved for public release; distribution is unlimited.
• Software product must be able to integrate high fidelity
transient/dynamic representations of the aircraft forebody-inlet-
compression systems
• Turbomachinery component must be able to compute full annulus
solutions for multiple blade row fan and compression systems for
complex inlet flow fields
• Software product should be multiphysics including combustion
chemistry and CSM capability
• Software product must be configured for multiple processors beyond
the current parallelization techniques to be able to compute solutions in
hours or days (current turbomachinery CFD calculation take weeks to
months)
• Software product must be validated with experimental results
• Software product must be easy to use with examples that provide
guidance to an inexperienced user for use in other specific applications
Firebolt Requirements
Firebolt
06/27/2011 Page-12 Approved for public release; distribution is unlimited.
Pre
ssu
re R
ati
o
Inlet Design Using CFD
and Subscale Duct Tests
Aircraft Subscale
Wind Tunnel Tests
AIP Distortion Patterns
Full Scale Distortion
Screen Engine Tests
Performance and
Operability Analysis
Without Dynamic Distortion
Flight Test Verification
1.20
1.18
1.16
1.14
1.12
1.10
1.08
1.06
1.04
1.02
1.00
0.98
0.96
0.94
0.92
0.90
Run 1603 Point 4
0.9
0
0.9
2
0.92 0.9
4
0.9
4
0.96
0.9
6
0.96
0.98
0.98
0.9
8
0.9
8
0.98
0.98
1.00
1.0
0
1.00
1.0
01.0
0
1.00
1.00
1.02
1.0
2
1.021.02
1.0
2
1.0
4
1.04
1.0
4
1.04
Corrected Airflow
FF--3535FF--3535
Current Airframe-Propulsion Integration Process
AIP
Firebolt
06/27/2011 Page-13 Approved for public release; distribution is unlimited.
Pre
ssu
re R
ati
o
Full Scale Aircraft/Fan CFD/CSM Validated
With Wind Tunnel and Engine Test Data
Transient API Analysis Performance and Operability
Analysis With Dynamic
Distortion
Flight Test Verification
Corrected Airflow
FF--3535FF--3535
Augmented Airframe-Propulsion Integration Process
Firebolt
06/27/2011 Page-14 Approved for public release; distribution is unlimited.
1
2011 2017
Co
de c
ap
ab
ilit
y
2009 2013 2015 2019
8
4
3
5 6 7
2
6) - High altitude UAV engine cooling and performance analysis capability. - Engine-bay cooling analysis capability. 7) Significantly improved physical accuracy of engine combustion
prediction. 8) Propulsion module for Kestrel and Helios – Rotating machinery
simulation (single passage, full annulus, FSI, Inlet/forebody coupling, maneuvering aircraft); Nozzle simulation (variable area capability, multispecies); Engineering model of combustion w/ architecture for future gas-phase chemistry modeling)
1) Engineering model for propulsion effects available to Kestrel and Helios.
2) Full vehicle engine inlet and compression system performance prediction structured grid capability.
3) Full vehicle engine inlet and compression system performance prediction unstructured grid capability.
4) Engine compression system (aero-elastic) blade life assessment capability.
5) Enhanced vehicle aerodynamic performance through accurate representation of nozzle geometry shape changes.
Operations and Support
Technology Development Engineering Development Product
Deployment Solutions Analysis
A B C D
Firebolt Annual Release Schedule
Firebolt
06/27/2011 Page-15 Approved for public release; distribution is unlimited.
Firebolt v1
Firebolt
06/27/2011 Page-16 Approved for public release; distribution is unlimited.
0D Engine Model Coupling
• Developed a common methodology for creating engine modules from existing 0D engine models ‒ ATEST (AEDC Developed)
‒ NPSS (NASA Developed)
‒ OEM customer decks
‒ Flat Engine (General tabular engine input for steady state or transient)
• Created coupling interface for CREATE-AV CFD codes ‒ Engine transient time step advancement controlled by the
infrastructure
‒ CFD code (kAVUS) coupling through inflow and outflow boundary conditions
‒ 0D interface data clipping and filtering
• User interface through KUI and HUI – Setup engine model inputs
– Preflight engine models
– Post process engine model output
Firebolt
06/27/2011 Page-17 Approved for public release; distribution is unlimited.
• F-16 Unstructured Grid
(17M elements)
• F110-100 Transient ATEST
Engine Model
• M=0.6 AOA=6o 20kft
• Throttle Decel/Accel
Kestrel/Firebolt Throttle Transient
M=0.6 AOA=6o 20kft M=1.4 AOA=6o 42.5kft
Firebolt
06/27/2011 Page-18 Approved for public release; distribution is unlimited.
F-16/F110-100 PLA Decel/Accel
Vorticity Isosurface Colored by Mach Centerline Pressure Distribution
Kestrel/Firebolt Throttle Transient
Inlet Spillage
Ventral Surfaces
LEX Vortex
M=0.6 AOA=6o 20kft M=1.4 AOA=6o 42.5kft
Normal
Shock
Firebolt
06/27/2011 Page-19 Approved for public release; distribution is unlimited.
Kestrel/Firebolt Throttle Transient
Firebolt
06/27/2011 Page-20 Approved for public release; distribution is unlimited.
Firebolt v2
Firebolt
06/27/2011 Page-21 Approved for public release; distribution is unlimited.
Turbomachinery CFD Challenges
• Large computational grids
– 2x106 points/blade, 20 – 100 blades/row, ~1x109 points
• Multiple time scales ‒ Time for throttle transient (10 sec.)
‒ Time for pressure wave to traverse the inlet (1x10-2 sec.)
‒ Blade revolution time (1x10-3 sec.)
‒ Blade structural response time (1x10-6 sec.)
• Complex flow physics
• Difficult to start solution process
• Stringent accuracy requirements
– Mass flow, total pressure, and total temperature to within 0.5%
• Multiphysics (CFD/CSM)
Firebolt
06/27/2011 Page-22 Approved for public release; distribution is unlimited.
Structured Grid Solver
• Built around modules from OVERFLOW 2.2
• Why OVERFLOW 2.2 ? ‒ NASA maintained code with large user and contributor base
‒ Contains SOA flux algorithms, implicit solvers, and turbulence models
‒ Overset grid capability
‒ Moving body capability
‒ Perfect gas and multispecies capability
‒ Dual parallel capability using MPI and OPENMP
‒ Run time grid decomposition for parallel load balancing
• Modifications for rotating machinery ‒ Added additional bc’s for single passage simulations
‒ Added general rotating reference plane capability
‒ Developed methodology for starting multistage simulations
‒ Developed mixing plane multistage capability
‒ Developed utility codes for assembling full annulus composite grid
Firebolt
06/27/2011 Page-23 Approved for public release; distribution is unlimited.
• Difficult to start a multi-stage full annulus solution
– Need pressure rise through machine to get required mass flow in back stages
– High specified outflow pressure can cause reverse flow on boundary
– Must keep solution between stall and choke
• Developed a starting process from uniform flow conditions
‒ Ramp up rotational velocity of rotor
‒ Use grid sequencing during initial solution transients
‒ Ramp up rotational velocity of rotor on coarse grid level
‒ Created Mach number limited specified pressure outflow BC
• Can now easily start multi-stage solutions using this procedure
CFD Starting Problems
3 Stage Military Fan with an IGV
Firebolt
06/27/2011 Page-24 Approved for public release; distribution is unlimited.
Mixing Plane Capability
• Mixing plane simulations use a single passage representation
– Use inflow and outflow boundary conditions for each passage
– Circumferentially average to remove blade wakes
• Quantities conserved on mixing plane
‒ Mass flow
‒ Total pressure
‒ Total temperature
‒ Flow angles
R1
R2
R3
S1 S2 S3
Mixing Planes
Firebolt
06/27/2011 Page-25 Approved for public release; distribution is unlimited.
Single Stage V&V Example
NASA Stage 35
• 36 high aspect ratio fan blades
• 46 stator blades
• 70%, 80%, 90%, and 100% speed
lines
• 1.8M points/passage
• Gridded 0.37 mm tip gap (0.65% tip
chord)
• Full annulus and mixing plane
simulations
• Reid, L. and Moore, R., “Design and Overall Performance of Four Highly Loaded, High Speed Inlet Stages for an Advanced High-Pressure-Ratio Core
Compressor,” NASA-TP-1337, 1978.
Firebolt
06/27/2011 Page-26 Approved for public release; distribution is unlimited.
Stage 35 Massflow Convergence
Mixing Plane Full Annulus
Firebolt
06/27/2011 Page-27 Approved for public release; distribution is unlimited.
Stage 35 Overall Performance
MP = Mixing Plane CFD
Full = Full Annulus CFD
Firebolt
06/27/2011 Page-28 Approved for public release; distribution is unlimited.
Stage 35 Component Performance
Firebolt
06/27/2011 Page-29 Approved for public release; distribution is unlimited.
Stage 35 Circumferential Averages 100% Speed 20.0 kg/s
Firebolt
06/27/2011 Page-30 Approved for public release; distribution is unlimited.
Stage 35 100% Speed Mach No.
20.892 kg/s 20.771 kg/s
19.880 kg/s 18.655 kg/s
Firebolt
06/27/2011 Page-31 Approved for public release; distribution is unlimited.
Summary
• Firebolt v1 designed to couple 0D engine models with high fidelity Navier-Stokes codes within a light weight Python infrastructure – Kestrel spring 2012
– Helios spring 2013
• Firebolt v2 designed for structured grid Navier-Stokes simulations of aircraft and engine rotating machinery – Kestrel spring 2013
– Helios spring 2013
• Software product includes training, user and developer documentation, and user support
• GUI will be provided to set up inputs and post process results
Firebolt
06/27/2011 Page-32 Approved for public release; distribution is unlimited.
References
D. E. Post, S. Arevalo, C. Atwood, P. Bell, T. D. Blacker, S. Dey, D. Fisher, D. A. Fisher, P. Genalis, J. Gorski, A. Harris, K. Hill, M. Hurwitz, R. P. Kendall, R. L. Meakin, S. Morton, E. T. Moyer, R. Strawn, D. van Veldhuizen, L. G. Votta, S. Wynn, and G. Zelinski, Journal of Physics: Conference Series Vol. 125 (2008) 012090.
Morton, S. A., McDaniel, D. R., Sears, D. R., Tillman, B., and Tuckey, T. R., “Kestrel – A fixed Wing Virtual Aircraft Product of the CREATE Program,” AIAA-2009-338.
Morton, S. A., McDaniel, D. R., Sears, D. R., Tillman, B., and Tuckey, T. R., “Kestrel v2.0 – 6DoF and Control Surface Additions to a CREATE Simulation Tool,” AIAA-2010-511, Jan. 2010.
Wissink, A. M., Sitaraman, J., Sankaran, V., and Mavriplis, D. J., “A Multi-Code Python-Based Infrastructure for Overset CFD with Adaptive Cartesian Grids,” AIAA-2008-927, Jan. 2008.
Nichols, R. H., Denny, A. G., Calahan, J. A., Savelle, S. A., and Heikkinen, B. D., “Firebolt v1.0 – Coupling of Transient and Steady Engine Performance Models with a High-fidelity Navier-Stokes Code,” AIAA-2011-3190, Jun. 2011.
Nichols, R. H., Denny, A. G., Calahan, J. A., Sirbaugh, J. R., Klepper, J. B., and Heikkinen, B. D., “Firebolt v2.0 – Structured Grid Navier-Stokes Code for Airframe/Propulsion Integration,” AIAA-2011-5970, July 2011.
Firebolt
06/27/2011 Page-33 Approved for public release; distribution is unlimited.
Acknowledgements
Material presented in this paper is a product of the CREATE-AV Element of the Computational Research and Engineering for Acquisition Tools and Environments (CREATE) Program sponsored by the U.S. Department of Defense HPC Modernization Program Office. Computational resources were also provided by HPCMP.