the costs of unsuitability and benefits of building in reliability, … · 2017-05-19 · sea wolf...

55
The Costs of Unsuitability and Benefits of Building in Reliability, Availability and Maintainability Dr. Ernest Seglie, Science Advisor, DOT&E [email protected]

Upload: others

Post on 31-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

The Costs of Unsuitability and Benefits of Building in

Reliability, Availability and Maintainability

Dr. Ernest Seglie, Science Advisor, DOT&[email protected]

Page 2: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds
Page 3: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Program Service ACAT Reason

F-15 TEWS USAF II Effective Not Suitable Reliability, Maintainability, Availability

V-22 Osprey Navy 1D Effective Not Suitable Reliability, Availability, Maintainability (RAM), Human Factors, BIT

Joint Direct Attack Munitions (JDAM)

USAF 1C Effective only with legacy fuses

Not Suitable Integration with delivery platforms

M2A3 Bradley Fighting Vehicle

Army 1D Effective Suitable

Joint Primary Aircraft Training System (JPATS)

USAF 1C Effective with deficiencies

Not Suitable RAM, Safety, Human Factors

Cooperative Engagement Capability (CEC)

Navy 1D Effective Suitable

Multiple Rocket Launcher System (MLRS)

Army 1C Effective Suitable

MH-60S Navy 1C Effective Not Suitable RAM, excessive administrative and logistic repair time impacted RAM

B-1B Block E Mission Upgrade Program

USAF 1D Effective Not Suitable 16% decrease in weapons release rate, reduction in accuracy of Mark 82 low drag weapons, 14% hit rate on moving targets

Sea wolf Nuclear Attack Submarine

Navy 1D Effective Suitable Several requirement thresholds were not met but overall system effective and

IOT&E ResultFY 2001

FY 2002

FY 2003

DoD IOT&E Results FY 2001 - 2003

Page 4: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

DoD IOT&E Results FY 2004, 2005

Program ACAT Reason

Evolved Sea sparrow Missile Navy Effectiveness unresolved

Suitable Testing was not adequate to determine effectiveness.

Stryker Army Effective SuitableEffective for short duration missions; not effective for all missions and profiles. Not suitable due to RAM.

Tactical Tomahawk Navy Effective Suitable

Stryker Mortar Carrier-B (MC-B) Army Effective Not Suitable RAM and safety concerns.

CH-47F Block I Army Effective Not Suitable RAM; communications system less suitable than CH-47D; did not meet Information Exchange Requirements for Block I.

F/A-22 USAF Effective Not Suitable RAM; needed more maintenance resources and spare parts; BIT

Joint Stand-Off Weapon-C Navy Not Effective Not effective against moderately hardened targets; mission planning time was excessive.

Guided-MLRS Army Effective Suitable

High Mobility Attack Rocket System (HMARS)

Army Effective Suitable

V-22 Osprey Navy Effective Suitable

EA-6B (ICAP III) Navy Effective Suitable

Service IOT&E Result FY 2004

II

1DAdvanced SEAL Delivery System (ASDS)

Navy 1D Effective with restrictions

Not suitable

1C

1D

FY 20051C

1D

1C

1C

1C

1D

II

Page 5: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Air Force IOT&E Results

RAMNot SuitableEffectiveIIUSAFF-15 TEWS

FY 2002

Technical ReasonIOT&E ResultACATServiceProgram

Program did not establish sound independent technical review processes. Software assurance and metrics not sufficiently established. Technical entrance and exit criteria not established for Developmental Test (DT) reviews and decisions.

Requirements DevelopmentSystem Integration, Test, and Verification

Technical Process

Program integrated Electronic Warfare (EW) systems with known reliability issues without performing a systemic analysis prior to design and integration.

AcceptabilityAcquisition Strategy

RAM requirements not fully defined. BIT architecture and subsystem reliability not designed into system. BIT system was a major requirement for the system.

ReasonablenessVerification

Requirements

Program focused mainly on Band 1.5 and did not address newer SAM systems; inadequate processing capability. Systemic analysis was not performed; might have captured systems integration problems and identified root causes for inadequate processing.

AllocationSufficiency

Program Planning

RationaleSE AreaIssue

SE Issues

Page 6: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

RAM; safety; human factors.Not SuitableEffective with deficiencies

1CUSAFJoint Primary Aircraft Training System (JPATS)

FY 2002

Technical ReasonIOT&E ResultACATServiceProgram

COTS mentality led to simplistic test approach (e.g., FAA cert, Contractor Qual Test approach led to insufficient DT). Multiple slips. Requirements not tracked/traced to a verification and test plan.

Requirements DevelopmentSystem Integration, Test, and Verification

Technical Process

Simple COTS approach. “Militarization” not fully defined or understood. Multiple slips: evidence of schedule-driven nature.

AcceptabilityAcquisition Strategy

No ORD Thresholds for R&M; program measured against objectives.

ReasonablenessVerification

Requirements

Acquisition Reform – pilot program. Designed as COTS program. Multiple slips: evidence of a schedule-driven nature. Requirements not fully defined and understood.

AllocationSufficiency

Program Planning

RationaleSE AreaIssue

SE Issues

Air Force IOT&E Results

Page 7: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Excessive mission planning times (Navy); system reliability; B-52 load times; container deficiencies (stacking, carrier ops).

Not SuitableEffective only with legacy fuses

1CUSAFJoint Direct Attack Munitions (JDAM)

FY 2001

ReasonIOT&E ResultACATServiceProgram

Unrealistic load times; test team load crew experience, training; test team mission planning experience/training.

System Integration, Test, and Verification

Technical Process

Acquisition Reform – pilot program. Small program office. Capability-based contracting strategy; significant SE contracted as result. Significant focus on capability (accuracy). Reliability relied heavily on “warranty.”

AcceptabilityAcquisition Strategy

B-52 load times not reflective of new complexity. Navy carrier operability (ruggedness) not adequately captured/defined. Significant focus on capability (accuracy). Reliability relied heavily on “warranty.”

Reasonableness; Design SynthesisVerification

Requirements

RationaleSE AreaIssue

Storage reliability. Significant failures related to minor quality control errors (i.e., missing sealant, kit packed with wrong covers, etc.).

Quality Control (Plant Layout)Reducibility and Production Planning

SE Issues

Air Force IOT&E Results

Page 8: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

16% decrease in weapons release rate; reduction in accuracy of Mark 82 low drag weapons; 14% hit rate on moving targets.

Suitable Not Effective1DUSAFB-1B Block E Msn Upgrade Program

FY 2003

ReasonIOT&E ResultACATServiceProgram

T&E measures not well founded in ORD/CDD.System Integration, Test, and Verification

Technical Process

Software conversion oversimplified. Significant program growth. “Program clarity” - funded program did not address numerous “known issues”; resulted in re-identification of numerous issues (situational awareness, controls and displays, reliability).

Acceptability; SufficiencyAcquisition Strategy

Validity of effectiveness measures, based on comparison with prior block (not as complex; different release mechanism; different weapons mix; key requirement met: weapons flexibility).

ReasonablenessVerification

Requirements

RationaleSE AreaIssue

SE Issues

Air Force IOT&E Results

Page 9: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

RAM; needed more maintenance resources and spare parts; BIT.

Not SuitableEffective1DUSAFF-22

FY 2003

ReasonIOT&E ResultACATServiceProgram

Program did not establish entrance/exit criteria for software development, verification, validation, and test. Software not adequately tested and fixed in the lab prior to flight test. Mission technical issues overshadow RAM issues and RAM resources diverted to technical mission issues. Program did not have a sound risk assessment program.

System Integration, Test, and Verification

Technical Process

Program did not recognize or fully fund RAM requirements and software development, especially the maintenance software portion. Labs were insufficiently supported with hardware-in-the-loop.

Acceptability; SufficiencyAcquisition Strategy

RAM requirements not fully defined for IOT&E but for a mature aircraft at 100K flight hours. RAM and BIT requirements not tracked/traced to a verification or test plan.

ReasonablenessVerification

Requirements

RationaleSE AreaIssue

SE Issues

Air Force IOT&E Results

Page 10: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Additional Costs When a Systemis Judged Unsuitable (1)

• Some programs extended their SDD and added resources to redesign, reengineer and to retest till they became suitable– V-22 extended its SDD by five years and spent ~$1B to

resolve its suitability issues. (It had a catastrophic failure in 2000)

– C-17 is likely to be another interesting case.

• When failure in OT&E delays the full production and the fielding of a new system, it may require extra cost to operate and support, and in some cases, even Service Life Extension Program (SLEP) on legacy systems.

Page 11: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Additional Costs When a Systemis Judged Unsuitable (2)

• Some programs were granted FRP and delayed RAM remedial actions as Block Upgrades– Approach requires additional cost for RDT&E and retrofit. It is also

more expensive to maintain and support several different configurations than one.

– It turns to a spiral development approach.– Identify related RAM development and retrofit costs– Estimate additional operating and support costs for extra

configurations• Some programs are fielded with known RAM shortcomings

– Extra costs for repair and maintenance or contractor logistic support when fielded at insufficient RAM level

– Possible cost to procure and operate additional units to compensate for low availability to meet desired sortie rates or ton-mile capacity

Page 12: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

LCCLCC DistributionDistribution

20 YEARS

LIFE-CYCLE COST

10%30% 60%

SYSTEMRESEARCH

ANDDEVELOPMENT

PRODUCTION

SYSTEM ACQUISITIONO&S

Page 13: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

BAConcept

RefinementSystem Development

& DemonstrationProduction &

Deployment

Systems Acquisition

C

FRP DecisionReviewLRIP/IOT&E

Design Readiness

Review

TechnologyDevelopment

(Program

Initiation)

ConceptDecision

Pre-Systems Acquisition

Operations & Support (O&S)Sustainment

65-80% of the Life Cycle Cost

Design ForSustainment

Sustain TheDesign

O&S Costs Are Determined Early In The Acquisition Phase

Life Cycle Management

Page 14: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Four Causes

• No requirements• Lack of incentives• Attention elsewhere• Poor Systems Engineering

Page 15: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

JROC Memo: 17 Aug 2006

“MATERIEL AVAILABILITY” KPP for all MDAPs and Select ACAT II and III

(KSAs):A. Materiel Reliability KSAB. Ownership Costs KSA

Page 16: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

• Single KPP:• Materiel Availability (= )

• Mandatory KSAs:• Materiel Reliability (MTBF)(= )• Ownership Cost (O&S costs associated w/materiel readiness)

• Ownership Cost provides balance; solutions cannot be availability and reliability “at any cost.”

JROC Approved* Mandatory Sustainment KPP and KSAs

Number of End Items Operational Total Population of End Items

Total Operating Hours Total Number of Failures

*JROC Approval Letter JROCM 161-06 Signed 17 Aug 06;Revised CJCS 3170 will put into Policy

Page 17: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Return on Investment

Page 18: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Estimate O&S and Initial Sparesof Different F-22 MTBMs

(Constant 2006 $B)

F-22 life cycle cost could be $5B – $7B (constant 2006) more if projected program reliability is not realized.

Reliability Level at Maturity

FOT&E Actual (1a) 0.65 $ 42B $ 7B

IOT&E Actual with Historical Growth (1b) 0.83 $ 40B $ 5B

Air Force Program Reliability Projection (1c) 1.50 $ 35B

(1) Mean Time between Maintenance. F-22 ORD established MTBM threshold at 3 hours. (1a) MTBM of 0.65 hours achieved in Follow-on Operational Test and Evaluation (FOT&E). (1b) IOT&E MTBM score 0.45 hours. F-22 will achieve MTBM of 0.825 hours at maturity (100,000 FH), if its reliability growth rate is similar to the historical rates of existing fighter aircraft programs. (1c) Air Force Program Office projects F-22 to achieve 1.5 hours MTBM at maturity. (2) O&S cost for 148 Primary Aerospace vehicle Authorization (PAA), 336 flying hours per aircraft per year for 24 years. Initial spares requirement for 182 Total Active Inventory (TAI), computed at $120M recurring flyaway cost each.(3) Baseline assumes the Air Force projected 1.5 hours MTBM at maturity. At the F-22 ORD MTBM threshold of 3 hours, the estimated life cycle cost would be $4B lower than the baseline in constant 2006 dollars.

MTBM in Hours (1)

Life Cycle Cost Difference (3)

O&S & Initial Spares (2)

Page 19: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Return of R&M Investment(Present Value 2006 $B)

Reliability Level at Maturity

FOT&E Actual 0.65 $ 30B

Air Force Program Reliability Projection 1.50 $ 25B

Potential Savings (4) $ 5B

Budgeted Investment $ 0.7B

Potential Return of Investment

7 : 1

(1) Mean Time between Maintenance. F-22 ORD established MTBM threshold at 3 hours. F-22 Follow-on Operational Test and Evaluation (FOT&E) MTBM score 0.65 hours. Air Force Program Office projects F-22 to achieve 1.5 hours MTBM at maturity.(2) O&S cost for 148 Primary Aerospace vehicle Authorization (PAA), 336 flying hours per aircraft per years for 24 years. Initial spares requirement for 182 Total Active Inventory (TAI), costed at $120M recurring flyaway cost per aircraft.(3) President Budget Submission (February 2005 and February 2006): F-22 Reliability and Maintainability Maturation Program (RAMMP). F119 engine Component Improvement Program (CIP). R&M retrofits: air vehicle RAMMP modification and F119 engine CIP modification(4) Saving will be substantially lower if F-22 does not achieve MTBM of 1.5 hours at maturity.

MTBM in Hours (1)

Savings to Investment Ratio

O&S & Initial Spares (2)

RDT&E & Retrofit (3)

Page 20: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

R&M Investment and Savings(PV 2006 $B)

--

$5.1B

$3.3B

$1.3B

2:1

5:1

7:1

0

2

4

6

8

0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6

MTBM (hour) at Maturity

R&M Investment: $0.7B

Potential Life Cycle Cost Savings

Saving to Investment Ratio

Page 21: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040

Defense System Life CyclesDefense System Life Cycles

94 yrsB-5267 yrs2.5 Ton Truck

93 yrsC-130

UH-1 69 yrsM-113 59 yrs

72 yrsAIM-9

56 yrsSSN 688

36 yrsF-1471 yrsCH-47

44 yrsHEMTT

51 yrsF-15

86 yrsKC-135

SOURCE: John F. Phillips DUSD (L)

Page 22: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Myths about Building-in Reliability, Availability and

Maintainability

Myth 1: Building-in Reliability costs money.

Page 23: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

HH-60H and MH-60S Reliability and Cost Comparison

* 0 failures observed in one year

Component MFHBRPUC

(FY07$K) Component MFHBRPUC

(FY07$K)

CPU159/A AFCS COMPUTER 582 $180

CPU133/A DIGITAL COMPUTER 1,944 $86

AUXILIARY POWER SYSTEMS 2,160 $86 ACFT POWER UNIT 10,000 $80

SECT'S 2/3/4 DRIVE SHAFT ASSY 6,480 $4

SECTIONS 2/3/4 DRIVE SHAFT ASSY 10,000 $4

CP1820/ASN150 NAV COMPUTER 434 $99

CP-2428/A DIGITAL DATA COMPUTER 2,236 $84

STABILATOR AMPLIFIER INSTALL 549 $34

AMPLIFIER INSTALLATION 1,351 $43

MLG DRAG BEAM/AXLE ASSY 10,000 $24

BEAM-AXLE ASSEMBLY 10,000 $26

FLOOR ASSEMBLY 10,000 $10 AIRCRAFT FLOOR 10,000 $20T1360( )/ALQ144(V) TRANSMITTER 582 $52

LIGHT,INFRARED TRANSMITTER 10,000 $5

HH-60H MH-60S

>

>

>

>

>

>

>

**

**

***

Page 24: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

When to Invest?

Page 25: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds
Page 26: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Return on Investmen

0.1

1

10

100

0.01 0.1 1 10 100

Investment Cost ($M

Series1

Page 27: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds
Page 28: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds
Page 29: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds
Page 30: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds
Page 31: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Summary

• How to Address Problems• Size of ROIs?• When to invest?

Page 32: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

THE END

Page 33: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds
Page 34: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

SDD Investment for Operational Suitability

• Compare and contrast SDD/EMD approaches of cases with different emphasis on RAM (F/A-18 C/D vs. E/F; F-15 vs. F-16)– Design and Engineering – Schedule, costs, test articles

• Compare and contrast distinct DT approaches taken in cases and resulting suitability rating (JSOW, JASSM)– Standards and specifications – Schedule, cost, test articles

• Contractors’ proposed fixes and associated costs for RAM issues

Page 35: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

reduce total ownership costs (R-TOC)

grew out of (PEO/SYSCOM) Commanders' conferences, the Defense Science Board, and others. The R-TOC program was established in response to longstanding concerns about the adverse impact of defense budgetary and operational trends on force structure and readiness. Rising operations and support (O&S) costs can consume higher portions of defense budget and leave even less available for modernization.

The purpose of the R-TOC program is to achieve readiness improvements in weapon systems by improving the reliability of the systems or the efficiency of the processes used to support them. New Technologies and management practices may provide significant opportunities to improve readiness and reduce ownership costs. In recent years, world-class suppliers have achieved cost reductions while making major improvements in customer support. Some DoD programs have achieved similar successes in adopting private sector improvements in logistics and supply chain management.

Page 36: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Multiple (Fuel)

Minuteman

F100/F-119

KC-135

F-15 & F-16

F-15/Multiple

F-15

C-5A/F-15

B-52

C-130

F-16

A-10

B-1B

Patriot Missle

Ground Support

H-47/H-64

CH-47 & AH-64

AH-64

UH-60

Multiple

CH-47

Weapon Systems

Page 37: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

UH-1

T700

T-58

HMMWV

H-1

F/A-18 & E2/C2

EA-6B, E-2/C-2

E2-C-2

CH-46

C-130/P-3

AH-1/H-46

F404

T-64

AV-8B

H-60

EA-6B

Multiple (Navy)

F/A-18

Page 38: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Assessing Cost and R&M

• Models to link Mean Time Between Maintenance and other R&M metrics to requirements for – Maintenance manpower– Sustaining spares– Initial spares

• F-22 example– Compare the O&S cost and initial spares requirement for a

range of reliability assumptions, following established analytical approaches for other F-22 Studies.

– Tabulate the trade-off between budgeted F-22 R&M investment (RDT&E and retrofit) and potential O&S and initial spares savings.

Page 39: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

RAM DESIGN: Reliability Design Techniques

Mission profile

Stress analysis

Worst case analysis

FMECA

Sneak circuit

Allocation

Parts selection

Derating criteria

Simplification

Design reviews

Page 40: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

• Materiel Availability (KPP*)– A Key Data Element Used In Maintenance

And Logistics Planning• Materiel Reliability (KSA*)

– Provides A Measure Of How Often The System Fails/Requires Maintenance– Another Key Data Element In Forecasting Maintenance/Logistics Needs

• Ownership Cost (KSA*)– Focused On The Sustainment Aspects Of The System– An Essential Metric For Sustainment Planning And Execution – Useful For Trend Analyses – Supports Design Improvements/Modifications

• Mean Downtime– A Measure Of How Long A System Will Be Unavailable After A Failure– Another Key Piece Used In The Maintenance/Logistics Planning Process

• Other Sustainment Outcome Metrics May Be Critical To Specific Systems, And Should Be Added As Appropriate

* Sustainment KPP & KSAs Included In Revised Draft CJCSM 3170

Proposed Life Cycle Sustainment Outcome Metrics

These 4 Life Cycle Sustainment Outcome Metrics Are UniversalAcross All Programs And Are Essential To Effective Sustainment Planning

Page 41: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Stryker Findings & Observations

• Warfighters very satisfied with Stryker performance in-theatre

• Brigade Commanders extremely happy with ICLS• High Operational Readiness Rates, but ORR is

prioritized over support costs• Op Temp in-theatre far exceeds planned usage

rates (X10, X15, X30 ?)• Operational Environment much different than

expected• Combat configurations add excessive weight to

vehicle (affecting reliability and performance)• Army did not buy Tech Data Pkg – “Prohibitively

expensive” . . . risk to government

Page 42: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Findings & Observations• Operational Readiness Rate not necessarily

consistent with traditional Ao– RAM issues can be masked by ORR

• Mission Completion vs. Subsystem Failure– Possibly leads to overestimating system reliability due to non-

reporting on individual subsystem (component) failures – Multi-mission vehicle – with subsystem failures, system can still

perform alternate missions

• Reporting Criteria Issue:– ORR vs. MTBF of individual subsystems

Page 43: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Reliability Issues• RAM requirement poorly defined in ORD

– 4.3.1.3. The Stryker (vehicle only, excluding GFE components/systems) will have a reliability of 1000 mean miles between critical failure (i.e., system aborts).

• Reliability issues and cost drivers found during DT/OT correlate well with in-theatre experience

Page 44: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Operational Environment• Field usage much harsher than planned

– e.g., higher tire pressure, roads, curbs, weight (armor, sandbags)• OMS/MP says 80% XCountry, 20% Primary Roads

– in-theater mission just the opposite . . . most missions in urban environment (police action) on paved roads

• OpTempo very high (>10X)– High OpTempo may improve reliability numbers, but beats up equipment– With low usage, seals can dry up, humidity can build up in electrical

components• Changes in mission & configuration are putting

excess stress on vehicle: armor/sandbags, over inflated tires, going over curbs– replacing 9 tires/day (>3200 tires/yr)– wheel spindles developing fatigue cracks– drive shafts breaking – prescribed tire pressure is 80 PSI, however, with slat

armor/sandbags – must maintain >95 PSI– 95 PSI is a logistics burden on operators

• Must be maintained by the soldier (CTIS system can’t do it)• Soldiers must check tire pressure more than 3 times per day to

maintain 95 PSI

Page 45: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Stryker Fleet ReadinessORR vs Strykers Fielded

As Of: 20 Feb 2007

0501001502002503003504004505005506006507007508008509009501000105011001150120012501300

1350140014501500

Jul

AugSepOctNovDecJa

n '04Feb

MarAprMayJu

neJuly

AugSepOctNovDecJa

n '05Feb

MarAprMayJu

nJul

AugSep

OctNovDecJa

n '06Feb

MarAprMayJu

nJul

AugSepOctNovDecJa

n '07Feb

Num

ber F

ield

ed

84%

86%

88%

90%

92%

94%

96%

98%

100%

ORR

Per

cent

age

Strykers Fielded ORR

1470 FieldedORR = 97%

Page 46: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Operational Readiness Rate (ORR)

• PBL Contractual Requirement: ORR > 90% – Does not include GFE (base vehicle configuration only)

• Consistently above requirement– Current ORR 97% (20 Feb 07)

• Cost-plus-fixed-fee contract motivates GDLS to meet ORR . . . .– However, contract does not incentivise controlling costs . . .

risk to government– Example – to repair cracked hyd res in power pack, whole

power pack is replaced in field

Page 47: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Cost Estimate Variation• CPM estimate - $17.19 (GAO 04-925, including labor, parts & repair)• CPM estimate - $18.78 (Stryker R-TOC Brief)• CPM estimate - $18.23 (based on M113 methodology w/Stryker adjustments)• CPM estimate - $14.53 (based on initial 4 month deployment data)• CPM estimate - $12.72 (GDLS estimate, labor, parts & repair)

• We need to understand the basis for these estimates more thoroughly (assumptions, models, configurations, limitations . . . )

Page 48: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Tactical Considerations• Slat Armor design (additional 5000 lb) is effective for many RPG

threats, but negatively impacts circumference, weight and performance of Stryker– Causes multiple problems for safe and effective operation

• Slat armor on rear ramp too heavy - greatly strains lifting equipment – Occasionally, crews must assist raising/lowering ramp

• Bolts on rear ramp break off frequently with normal use• Slat armor bends with continued ops . . . can cover escape hatches and

block rear troop door in ramp• Interferes with driver’s vision• Slat armor difficult to see at night . . . Safety hazard in urban environment• Slat Armor prohibits normal use of exterior storage racks

– Significantly impacts handling/performance in wet conditions• Excessive strain on engine, drive shafts, differentials

– Impairs off-road ops• Though not designed primarily for the urban fight (MOUT),

Stryker is well-suited for it– Unlike M-1, Stryker is “Ghostly” quiet . . . tactical advantage

• Stryker overall OIF performance significantly better than HUMVEE, BRADLEY or M-1 in this environment

Page 49: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Other Findings . . . cont.• Stryker initial deployment/fielding was extremely accelerated to

meet urgent combat need– Result was that Army was doing these things concurrently:

• Testing• Producing• Fielding• Conducting combat operations

• Immature Maintenance Procedures (many procedures have not been validated in IETMs) lead to: – “Tribal System Maintenance” from experienced crews (that new book

isn’t any good . . . . . . . . . . . this is the way it worked on the M113, so do it like this)

• With Kr support to maintain vehicles, soldier crews develop “rental car mentality” . . . – Overly dependent on Kr . . . . lack of ownership mentality– Sometimes they forget the basics (oil check)– One vehicle lost because pre-mission checks were ignored

• maybe lack feeling of ownership, maybe overly depend on Kr

Page 50: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Typical Suitability Requirements

• Logistic Support– Deployability of Forces and Equipment

• Training– Ability to Train Maintenance Personnel

• Ability to Launch Aircraft– Sortie Generation Rates (Peacetime and Wartime)

• Reliability and Maintainability (R&M)– Break Rate, Fix Rate– Mean Time Between Maintenance (MTBM)– Mean Time Between Removals (MTBR), Mean Down

Time– Mission Capable Rate and Weapon System Reliabilities

Unsuitability issues are often R&M related- high Break Rate, low Fix Rate, low MTBM, low MTBR

Page 51: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Elements Affected by R&M

• Poor R&M leads to more frequent maintenance/repair and longer time to repair/replace.– More initial spares = > higher procurement cost– More repair parts = > higher operating and support (O&S) cost– More maintenance manpower

= > higher O&S cost

• To correct the shortcoming in R&M after full rate production requires:– Operational System Development to design and test fixes

= > additional RDT&E cost– Retrofit existing production units

= > additional modernization cost

• Poor R&M lowers sortie generation rate, could necessitate additional procurement

Page 52: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

C-17 MTBM (Cumulative)

α=0.620.1

1

1000 10000 100000

FH

MTB

M MTBMMTBM (Duane)

α=0.35

Page 53: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

F-22A Mean Time Between Maintenance (MTBM)

0.01

0.10

1.00

10.00

10 100 1000 10000 100000

OT&E Flight Hours

MTBM (Cum)

MTBM (Cum fit)

MTBM (Inst f it)

MTBM (Inst)α = 0.44

Page 54: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

IMEASUREMaintenance Manpower

vs. Calibration Factor for Reliability

y = -47.5Ln(x) + 79.6R2 = 0.99

y = -43.7Ln(x) + 131.8R2 = 0.98

0

50

100

150

200

250

0 0.5 1 1.5 2 2.5Calibration Factor

Man

pow

er

F16 Data

F22 Data (13.5 hrrepair time) Log. (F16 Data)

Log. (F22 Data (13.5hr repair time) )

`

Page 55: The Costs of Unsuitability and Benefits of Building in Reliability, … · 2017-05-19 · Sea wolf Nuclear Attack Submarine Navy 1D Effective Suitable Several requirement thresholds

Estimate O&S and Initial Sparesof Different F-22 MTBMs

(Present Value 2006 $B)

F-22 life cycle cost could be $4B – $5B more in 2006 present value if projected program reliability is not realized.

Reliability Level at Maturity

MTBM in Hours (1)

FOT&E Actual (1a) 0.65 $ 30B $ 5B

IOT&E Actual with Historical Growth (1b) 0.83 $ 29B $ 4B

Air Force Program Reliability Projection (1c) 1.50 $ 25B

(1) Mean Time between Maintenance. F-22 ORD established MTBM threshold at 3 hours. (1a) MTBM of 0.65 hours achieved in Follow-on Operational Test and Evaluation (FOT&E). (1b) IOT&E MTBM score 0.45 hours. F-22 will achieve MTBM of 0.825 hours at maturity (100,000 FH), if its reliability growth rate is similar to the historical rates of existing fighter aircraft programs. (1c) Air Force Program Office projects F-22 to achieve 1.5 hours MTBM at maturity. (2) O&S cost for 148 Primary Aerospace vehicle Authorization (PAA), 336 flying hours per aircraft per year for 24 years. Initial spares requirement for 182 Total Active Inventory (TAI), computed at $120M recurring flyawy cost each.(3) Baseline assumes the Air Force projected 1.5 hours MTBM at maturity. At the F-22 ORD MTBM threshold of 3 hours, the estimated life cycle cost would be $3B lower than the baseline in 2006 present value.

O&S & Initial Spares (2)

Life Cycle Cost Difference (3)