the future of ground testing

Air Force Materiel Command

Developing, Fielding, and Sustaining America’s Aerospace Force

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

The Future of Ground Testing

Dr. Ed KraftAEDC/CZ

Presented at the AIAA Ground Test Conference

June 29, 2010

Key Points

• The Nation’s test infrastructure is under serious distress - time to change the dialogue from reactive to proactive

• Need to quantify, articulate and improve– Effectiveness– Relevance– Value

• People and processes may be more important than facilities for increasing effectiveness and value

Current State of Affairs

• Cause and effect not understood

• Minor wounds could lead to serious consequences

• Amputation primary form of surgery

• Not a very high survival rate

Analogous to Early Civil War Medicine

We need to better quantify cause and effect of test capabilities on aeronautical system development

outcomes

Efficiency vs Effectiveness

Contractor System T&E Cost (Average $795.6M in FY 2001 $)*

*Fox et al “Test and Evaluation Costs for Aircraft and Guided Weapons” Rand Project Air Force Report., 2004

+ GAO-05-301, Assessment of Selected Major Weapon Programs, March 2005

RDT&EOverrun

EquivalentAircraft

F/A-22 $10.5B -116

F-35 $10.1B -157.8

GAO Program Reviews+

Acquisition Cycle Time Key T&E Effectiveness Parameter

• Workload – Process driven, currently ~22,000 of wind tunnel testing and 13,000 of propulsion cell testing

• q (inverse of rework) – Process driven, typically have 10 structural failures found in flight; control surface resizing

• Capacity – Budget driven, availability x staffing x throughput

Cycle Time ~ Workloadq · Capacity

50% reduction in wind tunnel costs equates to just a few tenths of a percent reduction in program costs –Reducing acquisition cycle time by a month could save more than the cost of the entire wind tunnel campaign

Kraft, Edward M. “ Integrating Computational Science and Engineering with Testing to Re-engineer the Aeronautical Development Process,” AIAA Paper 2010-0139 , January, 2010.

Kraft, Edward M. and Huber, Arthur F II “A Vision for the Future of Aeronautical Ground Testing,” The ITEA Journal of Test and Evaluation, Vol 30, No 2, June 2009.

Identifying ValueMeasures of Effectiveness and Key Leverage Points in the

Acquisition Process

0

20

40

60

80

100

0 20 40 60 80 100

% Design Drawings Completed at CDR

% G

row

th in

Dev

elop

men

t Sc

hedu

le

Technology Maturity @ Milestone B

Design Closure@ CDR

Late DefectsIOT&EPause TestRate

CycleTime

RDT&E$/Total $ AverageFleet Age

SystemsDelivered

Suitability

RDT&E Fraction is revealing metric

Top Line Economic ModelQuantifying the Fraction of Systems Delivered

Fraction of Systems Delivered

Frac

tion

of B

udge

t

RDT&E“Non-Recurring Cost”

Procurement“Recurring Cost”RDT&E

Overrun

Fraction ofSystemsLost toRDT&EOverrun

0 1

RDT&EBudget

1

Final number of systems actually delivered driven by:

•Overruns in RDT&E•Changes in the average per unit cost (APUC) during production

•Congressional or DoD dictates•Final Budget constraints

Fraction of SystemsActually Delivered

RDT&E Budget fraction amplifies the RDT&E and Procurement overruns plus Budget changes!

ProcOverrun

APUC

Fraction ofSystemsActually

Delivered

= 1-RDT&EOverrun

RDT&EBudget

1 -

ProcOverrun

+ -+ DeltaBudget

Budget +

Budget -

Comparing RDT&E and Procurement Costs to RDT&E Fraction

DoD Total Outlay for Improvements(FY08 $, Millions)

0

0.1

0.2

0.3

0.4

0.5

1950 1960 1970 1980 1990 2000 20100

20000

40000

60000

80000

100000

120000

140000

RDT&E

Procurement0.5

0.4

0.3

0.2

0.1

0

RDT&EFraction

Note: Discreet jumps in RDT&E Fraction are triggered by Procurement downturns, but sustained by relative increase in RDT&E for each Period

Correlating Time History of Decline in Capacity, Capability, and Competence

• Industry consolidation – Reduced capability, investments– Reduced IR&D– Loss of intellectual capital

• RDT&E infrastructure– Reduced investments– Staff reductions

• Loss of intellectual capital– Reduced staffs– Loss of govt expertise through

reductions in hands on experience, staffing,TSPR, etc

• Loss of capacity– Availability of infrastructure– Staffing to perform RDT&E– Throughput/Quality

All correlate with Macro Cycles and have become deeper declines with each cycle

Reduced Aerospace Capacity

1.0

0.5

0

Frac

tion

of P

eak

Out

put

70 80 90 00 10

IR&D % Industry Cash Flow (22%)

% Aerospace to Total R&D S&E (25%)

US Aircraft Companies (10)

Aircraft Production Lines (30)

MRTFB Workyears (40K)

AEDC Workyears (4K)

Trend Parameter (Peak Output)

Capacity Trend Data from Multiple Elements of the Aerospace Industry Supportive of RDT&E

Exploring Cause and Effect Simple Dynamic Model

•Simple sinusoidal Proc $ with 20 yr period , $90B±$30B

•Baseline RDT&E $ expended at 0.25 Acq $

•With perfectly balanced, infinitely elastic capacity RDT&E $ would stay at 0.25 Acquisition $

•Reduced capacity consistent with previous chart

•-15% in 70’s, •constant in 80’s @ 85%, •further reduced 25%in 90’s •constant in 00’s @ 60%

•Replicates major trends, Total RDT&E $ and RDT&E Fraction escalate after each cycle

• Quantify effectiveness and value– Send me any data / case studies supportive of analysis– Build white papers targeted at key decision makers*

• Optimize total investments in capabilities– Optimum local elements ≠ optimum system – Use policy for Major Range Test Facility Base*

• Build, validate, and apply tools to increase effectiveness– Integrate computational science and engineering*, fly the mission

testing*, DOE response surface methodology to reduce cycle time and late defects

• Invest in technical excellence– Train in DOE, CSE, integrated T&E*– Fund hands-on approaches to learn craft*– Maintain a community with critical mass

• Change the total development process– Target success at leverage points– Reduce cycle time by >50%

SummaryNow What Do We Need to Do?

Challenge

Can the TC Lead This ?

Additional Information

• Contact meEdward.Kraft@arnold.af.mil

• Additional Reading– Kraft, Edward M. “ Integrating Computational Science and Engineering with Testing to

Re-engineer the Aeronautical Development Process,” AIAA Paper 2010-0139 , January, 2010.

– Kraft, Edward M. “After 40 Years Why Hasn’t the Computer Replaced the Wind Tunnel?” The ITEA Journal of Test and Evaluation, September 2010

– Kraft, Edward M. “DOE Application to Ground Testing – Advances and Challenges” Invited Panel Presentation at the AIAA Air Force T&E Days Conference, Nashville, TN, Feb 3, 2010.

– Kraft, Edward M. “Macro-Dynamics of Acquisition,” Draft White Paper available on request

– Kraft, Edward M. and Huber, Arthur F II “A Vision for the Future of Aeronautical Ground Testing,” The ITEA Journal of Test and Evaluation, Vol 30, No 2, June 2009.

– Huber, Arthur F II, Kraft, Edward M,, Best, John T., Stich, Phillip B., Eldridge, David A., Baker, William B., and Montgomery, Peter A. “Growing Technical Excellence in the AEDC Workforce.” AIAA Paper 2009-1762, February 2009.

– Best, John, T., Kraft, Edward M., and Huber, Col Arthur F. “Revitalizing the Technical Excellence at the Arnold Engineering Development Center (AEDC) and Beyond,” AIAA 2008-16115-7, Feb 2008.