san antonio findings

8/9/2019 San Antonio findings

1/68


2/68


3/68


4/68


5/68


6/68


7/68


8/68

Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED

MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN

ANTONIO (LPD 17)

2

(31) Summary of Interview

(32) NR 1B MPDE Photos

(33) Work Item 233-11-001, Reservation Task Request 33

(34) SAN ANTONIO Engineering Log of Nov 09

(35) NR 2A MPDE Summary of Inspection(36) NR 2A MPDE Photos

(37) MARMC Lab Rpt #2009NN03572 of 4 Dec 09

(38) MARMC Lab Rpt #2010NN00038 of 7 Jan 10

(39) MARMC Lab Rpt #2009NN03466 of 23 Nov 09

(40) MPDE Lube Oil Wetted Parts List

(41) LO Service System Description and Diagram

(42) E-mail of 10 Nov 09

(43) LO Piping Material Specification

(44) LO Pump Parts List

(45) LO System Valve Assembly

(46) LO Service System 6 Inch Valve Diagram(47) LO Service System H Valve Diagram

(48) S9233-DL-010

(49) MRC 2331 W-1R

(50) Investigator Notes (CoS)(3M Program Review)

(51) Maintenance Bucket Pictures

(52) SAN ANTONIO LO Log Aug 09

(53) SAN ANTONIO Engineering Log of Sep 09

(54) SAN ANTONIO LO Log Sep 09

(55) SAN ANTONIO LO Log Oct 09

(56) SAN ANTONIO LO Log Nov 09

(57) memo 5830 of 14 Jan 10 (Muslin Bag)

(58) Muslin Bag LO Flush Photos

(59) LO Flush Manual

(60) NSTM 233

(61) CLARUS Reports


(63) Fairbanks Morse Condition Reports

(64) memo 5830 of 14 Jan 10 (Bearing

Inspection)

(65) Bearing Damage Photos

(66) E-mail of 15 Jan 10

(67) JFMM Volume IV Chapter 4

(68) memo 5830 of 17 Dec 09 (MPDE MaterialHistory Logs)

(69) EM01 Quarterly Schedule

(70) MRC Card 2640 M-2

(71) MRC 2331 18M-12R

(72) EM02 Quarterly PMS Schedule

(73) MRC 2331 R-12W

FOR OFFICIAL USE ONLY


9/68



ANTONIO (LPD 17)

3

(74) memo 5830 of 17 Dec 09 (3M Program)

(75) MRC 2331 R-13W

(76) MRC 2331 R-14M

(77) MRC 2331 U-35

(78) MRC 2620 U-1(79) MRC 2620 U-3

(80) memo 5830 of 15 Jan 10 (LOQM)

(81) SAN ANTONIO SAMM Oil Sample DB 6 Dec 09

(82) EOCC Procedures

(83) memo 5830 of 15 Jan 10 (Eng Dept Trng)

(84) Investigator Notes (CoS) (Training Program Review)

(85) E-mail of 22 Dec 09

(86) E-mail of 29 Dec 09



(89) Summary of Interview(90) ECS PP Brief

(91) E-mail 13 Oct 06

(92) PRESINSURV NORFOLK VA 132000Z Apr 07

(93) PRESINSURV NORFOLK VA 061500Z Dec 07

(94) CASREP 07075 08099 Summary

(95) e-mail of 16 Nov 07

(96) CRO Predeployment PowerPoint of Aug 08

(97) LPD 17 Deployment Status Report 27 Nov 08 to 12 Dec

08

(98) ail of 15 Dec 08

(99) E-mail of 12 Oct 06

(100) E-mail of 22 Oct 06(101) Post FOA Support PowerPoint

(102) e-mail 5 Sep 07

(103) SAN ANTONIO TSO List


(105) E-mail 8 Jan 10

(106) NAVSEA Philadelphia Code 911 LPD 17 Class ICAS

Status Update

(107) ICAS PP Brief

(108) memo 5830 of 15 Jan 10

(109) LPD 17 Class NTSP 1 Jul 09 (Final-Approved)

(110) Letter of Promulgation for the ManagementResponsibilities

(111) LPD Training POM 12 PP Brief

(112) PSMD LPD 17 6 Jan 04

(113) PSMD LPD 17 Change One 25 Feb 05

(114) PSMD LPD 17 Change Two 9 Apr 07




10/68



ANTONIO (LPD 17)

4

(116) PEO Washington DC 301804Z Mar 09 LPD 17 Class FY09

Interim Training Plan for Main Propulsion

(117) PEO Washington DC 171810Z Oct 09 LPD 17 Class FY10

Interim Training Plan for Main Propulsion

(118) LPD 17 Class MPDE-SSDG-ECS Interim Training PP(119) E-mail of 18 Dec 09

(120) EOOW PQS NAVEDTRA 43104

(121) Engineering Collaterals and Qualifications PQS

NAVEDTRA 43704

(122) CO, USS SAN ANTONIO (LPD 17) ltr 5100 Ser 00 of 26

Mar 08


Jun 09


Sep 09

PRELIMINARY STATEMENT

1. Pursuant to enclosure (1), and in accordance with reference(a), I conducted a command investigation into the facts and

circumstances surrounding the discovery of metal shavings on the

oil side of three of four of USS SAN ANTONIO (LPD 17) Main

Propulsion Diesel Engines (MPDE) and a review of the maintenance

performance and quality control efforts by shipyard facilities,

intermediate maintenance activities, and SAN ANTONIO. Enclosure

(2), a list of relevant acronyms and abbreviations, is included

to assist in a review of the investigation.

2. Executive Summarya. Metal Shavings

(1) In October 2008, MARMC/NNSY voyage repair team

conducted significant repairs to the MPDE lube oil service

system piping onboard SAN ANTONIO due to failure of several

welded joints. The repair included a partial use of carbon

steel piping. Starting June 2009 during CNO PMA, NNSY conducted

permanent repairs of the piping that included the replacement of

approximately 80% of the existing external lube oil servicepiping with new 300 series stainless steel piping. After the

completion of the piping repairs, CLARUS using the NAVSEA

procedure conducted a flush of the piping. However, as designed

the flushing procedure was inadequate to completely remove

contaminants from the lube oil service system.



11/68



ANTONIO (LPD 17)

5

(2) In November and December 2009, NSSA Diesel

Inspectors and SAN ANTONIO personnel discovered foreign matter

in the lube oil service system for NR 1A, 1B, and 2A MPDE. The

foreign matter included 300 series stainless steel. The

probable source of the 300 series stainless steel found in NR1A, 1B, and 2A MPDE lube oil service systems was the byproducts

of the production and installation of stainless steel lube oil

service system piping. The cause of the stainless steel was

potentially the result of poor weld joint fit up during new

construction that created contaminate traps that could allow for

an intermittent release of contaminates.

(3) In November 2009, NSSA determined that depot level

maintenance was required to address excessive crankshaft main

bearing and thrust bearing wear in 1A MPDE, 1B MPDE, and 2A

MPDE. This bearing damage is the primary component of MPDEcorrective maintenance since completion of the PMA in October

2009. This damage is not related to the metal shavings found in

the MPDE lube oil service systems. Bearing failure root cause

is lube oil contamination that occurred during the Shipbuilding

process to include all repairs and maintenance prior to the PMA.

Total estimated cost of this additional MPDE corrective

maintenance is $7.5M. The operational impact is such that

repairs may preclude SAN ANTONIO executing her next scheduled

deployment.

(4) An alternate means of contamination causing bearing

damage based on technical evaluation of bearing failures on LPD

17 and LPD 21 has been proposed by NAVSEA during a recent video

teleconference with Fleet Forces Command, Second Fleet,

Commander Naval Surfaces Forces Atlantic and Expeditionary

Strike Group Two on 8 January. This hypothesis suggests that

the lube oil contaminates that are circulated back to the MPDE

sump via the self-cleaning pre-filter are continuously reduced

in physical size by the macerating action of the lube oil

service pumps until the contaminates are small enough to pass

through the filter and duplex strainers. Contaminates that pass

through the filter and strainer complex could still be large

enough to cause bearing damage. NAVSEA is considering anemergent alteration to eliminate this mechanism of contaminate

introduction into the diesel complex by filtering out

contamination discharged by the filter back to the sump.

(5) Ships Force was slow to recognize lube oil

contamination for a variety of long term issues. Command



12/68


13/68



ANTONIO (LPD 17)

7

(6) An external review to validate if Sailor/Officertraining and NAVSEA conditioned based maintenance reach backsystems were in place to support the lead ship of a new classthat had a crew that was "right sized" with these externalassumptions in place,

(7) A review of the automated systems designed to supportshipboard operations including the Engineering Control System(ECS), Integrated Condition Assessment System (ICAS) and TotalShip Training System (TSTS) was conducted to evaluate theeffectiveness of these systems to support ship's force's abilityto operate the propulsion plant per the current ship's manningdocuments,

(8) An audit by using the appropriate Technical WorkDocuments (TWD) and associated Objective Quality Evidence (OQE)

against Joint Fleet Maintenance Manual (JFMM), NAVSEA StandardItems, Specification Items, and other appropriate technicalguidance to assess Depot and MSR corrective maintenanceperformance,

(9) A review of ship's force corrective and preventivemaintenance by examining preventive maintenance scheduling andexecution,

(10) Oral interviews with the chain of command,

(11) A review of NAVSEA technical documentationdescribing how SAN ANTONIO class ships are to manned, trained and

provided external support was reviewed to determine completenessand effectiveness of execution,

(12) A briefing from NAVSEA representatives on theEngineering Control System (ECS) and Integrated Condition

Assessment System (ICAS), and

(13) A review of the syllabus for training provided bySurface Warfare Officer School (SWOS) to prospective SAN ANTONIOclass Commanding Officers, Executive Officers and ChiefEngineers.

4. The Investigating Officer has the following priorengineering/maintenance experience: USS HUNLEY (AS 31) Planning

and Estimating Officer (12 months), Repair Duty Officer (12

months), and Auxiliaries Officer (11 months); USS LEXINGTON (AVT

16) Main Engines Officer (19 months); Steam Engineering Officer

of the Watch (EOOW) Instructor and Course Coordinator (30



14/68



ANTONIO (LPD 17)

8

months); and USS MOUNT HOOD (AE 32), Chief Engineer (CHENG) (20

months).

5. The following personnel assisted in the conduct of the

investigation (prior engineering/maintenance experience listed):

a. Chief of Staff, Expeditionary

Strike Group TWO: BS Mechanical Engineering, University of

Wisconsin 1983; MA Systems Engineering, Naval Postgraduate

School 1991; Nuclear Power School 1984; Nuclear Power Training

Unit, Idaho Falls (A1W Prototype) 1985; USS LONG BEACH (CGN 9),

Number Two Engine Room Division Officer (36 months); USS

ARKANSAS (CGN 41), Main Propulsion Assistant (MPA) (16 months);

USS TRUXTUN (CGN 35), CHENG (22 months); Atlantic Fleet Nuclear

Propulsion Examination Team (NPEB), Junior Team Member (28

months); and USS EISENHOWER (CVN 69) Reactor Officer (28months).

b. N4, Expeditionary Strike Group

TWO: USS SCOTT (DDG 995), MPA (30 months); USS CHINOOK (PC 9),

CHENG (36 months); Mk 5 Special Operations Craft, Officer in

Charge/Assistant Officer in Charge (38 months); USS KEARSARGE

(LHD 3), MPA (24 months); USS WASP (LHD 1), CHENG (18 months);

and Auxiliary Floating Dry Dock Dynamic, Commanding Officer (23

months).c. N43, Naval Surface Force

Atlantic: USS NASSAU (LHA 4), Assault Officer (24 months),

Auxiliaries Officer (18 months), and Repair Officer (six

months); Afloat Training Group, Atlantic, Training Liaison

Officer (24 months); and USS OAK HILL (LSD 51), CHENG (13

months).

d. Engineering Support,

Expeditionary Strike Group TWO: OAK HILL, Electrical Officer (28

months).

e. Engineering Support,

Expeditionary Strike Group TWO: NASSAU, Electrical Officer (17months).

f. Diesel Engine Inspector

(Code 216.5) Diesel Engine Inspector, Norfolk Naval Shipyard:

USS MESA VERDE (LPD 19), Top Snipe (24 months); USS SAN ANTONIO,

Pre-Commissioning Unit (PCU) for MESA VERDE (six months); USS



15/68



ANTONIO (LPD 17)

9

NEW ORLEANS, PCU for MESA VERDE (six months); USS CARTER HALL

(LSD 50), PCU for MESA VERDE; OAK HILL, M-Division Leading Chief

Petty Officer (eight months); and HSV-X-1, Lead Engineman (12

months).

g. Maintenance and Material

Management System Chief Petty Officer, Amphibious Squadron SIX:

USS REID (FFG-30) (36 months); USS MCKEE (AS-41) (24 months);

and USS CORMORANT (MHC 57), EA01 Leading Petty Officer (48

months).

h. , Norfolk Ship Support Activity (NSSA):

Merchant Marine Officer (9yrs) U.S Coast Guard licensed

Engineering Officer on medium speed diesel powered ships; U.S.

Navy Port Engineer (12 years) DDG Class/CG Class Class

maintenance management; NSSA LSD/LPD Class Team Leader (threeyears) Managed RMC Waterfront Operations Department LSD/LPD

4/LPD 17 Class Maintenance Teams; NSSA CVN/Amphibious Ship

Division Manager (one year) - Principal Waterfront Operations

Department advisor in matters relating to CVN and amphibious

ship maintenance.

i. , Code 212.5, Nondestructive

Testing Level III, Quality Assurance Supervisor, NSSA: Retired

HTC; engineering plant operator/watch stander (MMOW) (four

years); ship/submarine maintenance, repair and quality assurance

(22 years); and contractor oversight (seven years).

j. Quality Assurance

Specialist with Code 212.5, Norfolk Naval Shipyard Engineering

and Technical Fleet Support Department: Retired HTC; NAVSEA

certified NDT Level II Inspector (22 years); and NAVSEA

certified NDT Level III Examiner (eight years).

k. Project Team Leader, Norfolk Naval

Shipyard (NNSY), USS BATAAN (LHD 5): prior experience as

Supervisory Assistant Project Superintendent: SAN ANTONIO, WASP,

USS NASSAU, and USS GEORGE WASHINGTON (CVN 73); NNSY Production

Department (30 years).

6. Expeditionary Strike

Group TWO, and Expeditionary Strike

Group TWO provided administrative support in the course of this

investigation.



16/68



ANTONIO (LPD 17)

10

7. provided legal

guidance in the course of this investigation.

FINDINGS OF FACT

Metal Shavings Discovery and Causation

Introduction

1. During a deployment to Fifth Fleet Area of Operationsbetween 9 October and 25 October 2008, SAN ANTONIO suffered

repetitive weld joint failures in the MPDE lube oil service

system piping. [Encls. 3, 4, 5)]

2. The weld joint failures were a result of poor weldtechniques. [Encls. (5, 6)]

3.The poor weld techniques during new construction causedexcessive oxidation and stainless steel globules inside the lube

oil service system piping. [Encl. (6)]

4. The poor weld joint fit up during new construction createdtraps. [Encls. (5, 6)]

5. From 2 to 23 November 2008, MARMC/NNSY voyage repair teamconducted repairs on the lube oil piping inport Bahrain. [Encls.

(3, 4)]

6. The repair included piping replacement using 304 stainlesssteel and carbon steel piping. [Encls. (3, 4)]

7. The repairs did not include a flush of the lube oil system.[Encls. (3, 4)]

8. Upon completion of repairs, the lube oil service systemserving each engine was operated in normal configuration. [Encl.

(7)]

9. Ships force continued to monitor lube oil duplex strainercontent until the baskets were clear of foreign material and

particulate. [Encl. (7)]

10. On 27 March 2009, SAN ANTONIO returned from the deployment.[Encl. (8)]



17/68



ANTONIO (LPD 17)

11

11. On 17 June 2009, SAN ANTONIO began a Chief of NavalOperations Planned Maintenance Availability (CNO PMA). [Encl.

(8)]

12. During the CNO PMA, NNSY replaced approximately 80% of theexisting external lube oil service system piping serving allfour MPDEs with new 300 series stainless steel piping. [Encls.

(9-15)]

13. After completion of the lube oil service system pipingrepairs, a flush of the lube oil system occurred. [Encl. (9)]

14. During the CNO PMA, contract personnel conducted thefollowing MPDE repairs: NR 2A1 lube oil service pump, NR 1A and

NR 1B MPDEs valve cover software replacement, NR 1B MPDE

crankcase fan oil seal, and NR 2A MPDE low pressure air lineline. [Encl. (9)]

15. On 2 October 2009, SAN ANTONIO completed a light offassessment (LOA). [Encl. (8)]

16. On 7 October 2009, SAN ANTONIO conducted sea trials.[Encl. (8)]

NR 1A MPDE Foreign Matter Discovery Timeline

17. On 29 July 2009, work started to replace NR 1A MPDEcylinder head (16 total) valve cover gaskets and software

crankcase in accordance with NSTM S9233-DL-MMM-020 (Chapter 7)

and NSTM S9233-DL-MMM-010. [Encl. (16)]

18. On 19 August 2009, CHENG, closed out NR 1A MPDElube oil sump. [Encl. (17)]

19. On 11 September 2009, work completed to replace NR 1A MPDEcylinder head (16 total) valve cover gaskets and software



20. If normal cleanliness standards are adhered to during theexecution of the work, a comprehensive lube oil service system

flush is not required. [Encl. (18, 19)]

21. Normal cleanliness standards were met prior to commencementof production work. [Encl. (20)]



18/68



ANTONIO (LPD 17)

12

22. On 7 October 2009 at 1903 while operating, NR 1A MPDEsuffered an engineering casualty: unusual noise and vibration

on NR 9 cylinder. [Encl. (21)]

23. On 7 October 2009 at 1955, a lube oil casualty sample wastaken from NR 1A MPDE. [Encl. (21)]24. On 14 October 2009, CASREP submitted on NR 1A MPDEregarding loud metallic noise. [Encl. (22)]

25. The CASREP indicated no unusual conditions were noted on NR9 cylinder. [Encl. (22)]

26. On 21 October 2009, CASCOR submitted on NR 1A MPDEregarding loud metallic noise. [Encl. (22)]

27. On 22 November 2009, Naval Surface Warfare Center CarderockDivision-Ship Systems Engineering Station Philadelphia (NSWCCD-

SSES Philly) reported abnormal wear levels in NR 1A MPDE based

on a lube oil sample submitted on 7 October 2009. [Encl. (23)]

28. Due to the abnormal wear levels, NSWCCD-SSES Phillyrecommended an inspection of NR 1A MPDE. [Encl. (23)]

29. On 25 November 2009, the Commanding Officer, SAN ANTONIO,requested Port Engineer, and , CHENG,

consult with NSWCCD-SSES Philly, for MPDE

inspection recommendations. [Encl. (23)]

30. On 25 November 2009, requestedand Norfolk Naval

Shipyard Diesel Inspectors, provide inspection recommendations.

[Encl. (23)]

31. On 25 November 2009, conducted the inspectionof the lube oil service system and an inspection of NR 1A MPDE.

[Encls. (23-25)]

32. During the inspection of NR 1A MPDE, the engine remainedstatic and was not barred over. [Encl. (25)]

33. The inspection of NR 1A MPDE included a visual inspectionof cam bearing and journals, and rod bearing and journals, and

main bearing and journals. [Encl. (25)]



19/68



ANTONIO (LPD 17)

13

34. The online strainer exhibited no particulates. [Encls.(25, 26)]

35. The off-line strainer exhibited ferrous particulate (fine)on the magnet that if accumulated would be less than 1% of themagnet cylinder surface. [Encls. (25, 26)]

36. The back-flushing filter exhibited particulate: metalshavings, paint chips, gasket material, fine debris located in

remnants of fibrous material in 20 candles of 66. [Encls. (25,

26)]

37. The cam covers accesses exhibited no debris orparticulates. [Encls. (25, 26)]

38.The dry sump exhibited no solid material. [Encl. (25)]

39. In the dry sump, there was free water standing in the oil.[Encl. (25)]

40. recommended the wet sump be pumped out andcertified gas free for further inspection. [Encl. (25)]

41. On 1 December 2009, wet sump cleaned and gas freed. [Encls.(25)]

42. During the initial visual inspection of the wet sump, thewet sump exhibited a higher than expected amount of babbitt

material. [Encl. (25)]

43. In the wet sump, there was free water standing in theresidual oil. [Encl. (25)]

44. On 15 December 2009, on-line and off-line Rocker DuplexStrainers removed by Ships Force and inspected by DEI

. [Encl. (24)]

45. On-line and Off-line Rocker Strainers exhibited free water,dirt, and paint chips (white, terracotta, and yellow). [Encl.(24)]

46. Rocker Sump opened by Ships Force and inspected by DEI[Encl. (24)]



20/68



ANTONIO (LPD 17)

14

47. Rocker Sump exhibited free water, dirt, and paint chips.[Encl. (24)]

NR 1B MPDE Foreign Matter Discovery Timeline

48. On 21 July 2009, work tasked to repair NR 1B MPDE crankcasefan oil seal. This repair had an estimated start date of 27July 2009 and was signed complete on 25 August 2009. [Encl.

(27)]

49. On 27 July 2009, work started to repair NR 1B MPDEcrankcase fan oil seal. [Encl. (27)]

50. On 29 July 2009, work started to replace NR 1B MPDE cylinderhead (16 total) valve cover gaskets and software crankcase in

accordance with NSTM S9233-DL-MMM-020 (Chapter 7) and NSTM

S9233-DL-MMM-010. [Encl. (28)]

51. On 25 August 2009, work completed on NR 1B MPDE crankcasefan oil seal. [Encl. (27)]

52. On 26 August 2009 at 1140, Chief Engineer,closed out NR 1B MPDE lube oil sump. [Encl. (17)]

53. On 11 September 2009, work completed to replace NR 1B MPDEcylinder head (16 total) valve cover gaskets and software



54. If normal cleanliness standards are adhered to during theexecution of the work, a comprehensive lube oil service system

flush is not required. [Encl. (18, 19)]

55. Work would not have commenced prior to cleanlinessstandards being met. [Encls. (29-31)]

56. On 7 October 2009 at 1010 while operating NR 1B MPDE, NR 1BMPDE suffered an engineering casualty: hot bearing on NR5 main

bearing (172 degrees F). [Encl. (21)]

57. The hot bearing was a 21 degree spread from the nexthighest bearing temperature. [Encl. (21)]

58. On 7 October 2009 at 1010, NR 5 main bearing (172 degreesF) was 21 degrees F higher than the next highest bearing

temperature. [Encl. (21)]



21/68



ANTONIO (LPD 17)

15

59. On 1 December 2009, filter removed for NR 1B MPDE due toissues with NR 1A and 2A MPDE. [Encl. (24)]

60. Lube oil results for NR 1B had been satisfactory (did notindicate need to conduct inspection). [Encl. (24)]61. Filter element removed. Set on work bench. Amount ofbabbitt material on filter and filter candles. Had fibrous

material as well. No further inspection. [Encl. (24)]

62. On 15 December 2009, on-line and off-line strainers wereremoved by ships force and inspected by DEI [Encl.

(24)]

63.On-line strainer exhibited fine ferrous particulate on themagnet, accumulated debris did not exceed the circumference of a

dime. [Encls. (24, 32)]

64. Off-line strainer exhibited fine ferrous particulate on themagnet, accumulated debris did not exceed the circumference of a

dime. [Encls. (24, 32)]

65. On 15 December 2009, on-line and off-line Rocker DuplexStrainers removed by Ships Force and inspected by DEI

[Encl. (24)]

66. On-line and Off-line Rocker Strainers exhibited dirt,gasket material, paint chips and free water. [Encls. (24, 32)]

67. Rocker Sump opened by Ships Force and inspected by DEI[Encl. (24)]

68. Rocker Sump exhibited free water, dirt, and paint chips.Sump bottom was coated with a large amount of fine bronze

material and non-ferrous matter similar to babbitt. [Encls. (24,

32)]

69. On 16 December 2009, 1B MPDE lube oil sump pumped, 600gallons of lube oil remained at the pump end of the sump. [Encl.

(24)]

70. On 17 December 2009, 1B MPDE lube oil sump opened forinspection. [Encl. (24)]



22/68



ANTONIO (LPD 17)

16

71. At the sump cover opening ferrous particulate, paint chips,dirt and carbon were immediately visualized before entering the

sump. [Encls. (24, 32)]

72. The DEI hand skimmed the oil layer at the sump coveropening; this revealed fine particulate (bronze in color), nonferrous particulate resembling babbit. [Encls. (24, 32)]

NR 2A MPDE Foreign Matter Discovery Timeline

73. On 22 July 2009, work tasked to replace NR 2A MPDE lowpressure air line serving the engine barring gear (turning

gear). [Encl. (33)]

74. On 25 July 2009, work started and completed to replace NR2A MPDE low pressure air lube oil line serving the engine

barring gear (turning gear). [Encl. (33)]

75. The low pressure air line consisted of 3/8 inch stainlesssteel tube. [Encl. (33)]

76. On 13 August 2009, , Chief Engineer, closed out NR2A MPDE lube oil sump. [Encl. (17)]

77. On 14 August 2009 at 1815, NR 2A MPDE lube oil strainercheck point complete by Engineering Duty Officer in preparation

for engine flush. [Encl. (17)]

78. On 6 October 2009 at 0750, NR 2A MPDE emergency stopped dueto NR 8 hot bearing. [Encl. (21)]

79. On 6 October 2009 at 0822, casualty sample taken for NR 2AMPDE. [Encl. (21)]

80. On 6 October 2009 at 1210 NR 2A MPDE lube oil straineropened and inspected SAT. [Encl. (21)]

81. On 6 October 2009 at 1250, NR 2A MPDE lube oil servicesystem duplex strainer tested satisfactory. [Encl. (21)]

82. On 6 October 2009 at 1315, DEI, andverified positive lube oil flow on NR 2A MPDE NR 8 bearing.

[Encl. (21)]

83. On 6 October 2009 at 1734, bearing inspection on NR 2A MPDENR8 bearing revealed no apparent damage. [Encl. (21)]



23/68



ANTONIO (LPD 17)

17

84. DEI, and the CHENG closed out NR 8 bearing.[Encl. (21)]

85. On 2 November 2009 at 1043, NR 2A MPDE stopped due to hotbearing casualty. [Encl. (34)]

86. On 2 November 2009 at 1102, casualty sample drawn on NR 2AMPDE NR 2 bearing. [Encl. (34)]

87. On 2 November 2009 at 1154, Diesel Engine Inspector, CHENG, and inspected NR 2A MPDE NR 1 and

NR 2 main bearing. [Encl. (35)]

88. SAN ANTONIO personnel and found a quarter ofan inch ferrous metal sliver and several non-ferrous particles.

[Encls. (35, 36)]

89. On 2 November 2009, NR 2A MPDE lube oil service systemtagged out for inspection of strainer and back-flushing filter.

[Encls. (35, 36)]

90. The strainer candles were dislodged. [Encl. (35)]91. The strainer exhibited ferrous particulate (fine) on themagnet that if accumulated would be not exceed the area of a

dime. [Encls. (35, 36)]

92. The back-flushing filter exhibited ferrous and non-ferrousover 75% of the candles. [Encl. (35)]

93. The ferrous particles exhibited on the back-flushing filtermeasured under one half inch when stretched. [Encls. (35, 36)]

Identification of Foreign Matter

NR 1A MPDE

94. The foreign debris removed from NR 1A MPDE sump includedorganic and metallic materials. [Encl. (37)]

95. The organic materials consisted of items similar to paint,textile floor coverings, synthetic polymer Nylon, rubber,

hydraulic oil, Kevlar, glass fibers, hair, polypropylene, and

cellulose fibers. [Encl. (37)]

96. The metallic materials consisted of material similar to a300 Series Stainless Steel alloy. [Encl. (37)]



24/68



ANTONIO (LPD 17)

18

97. Three of the metallic samples provided appear to be from amachining operation. [Encl. (37)]

98. For comparison MARMC personnel analyzed known standards of304 Series Stainless Steel, 316 Series Stainless Steel, 321Series Stainless Steel, and 347 Series Stainless Steel alloys.

[Encl. (37)]

99. MARMC personnel cannot identify the precise composition ofthe metallic materials due to possible contamination. [Encl.

(37)]

NR 1B MPDE

100. The debris removed from NR 1B MPDE are consistent withcarbon steel of undetermined origin, and 300 Series Stainless

Steel alloys from a machining operation. [Encl. (38)]

NR 2A MPDE

101. The metallic debris removed from NR 2A MPDE lube oilservice system back-flushing filter consisted of a material

similar to a 300 Series Stainless Steel alloy from a machining

peration. [Encl. (39)]o

Internal Source of the Foreign Matter

MPDE

102. There are no parts of the MPDE wetted by lube oil thatcontain stainless steel. [Encl. (40)]

LO Service System

103. Each MPDE has its own lube oil sump, lube oil servicepumps, self-cleaning filter, heat exchanger and duplex strainer.

Lube oil is drawn from the sump via the installed positive

displacement lube oil pumps. [Encl. (41)]

104. Lube oil discharged from the lube oil service pumps entersa self-cleaning filter. [Encl. (41)]

105. The self-cleaning filter filtration is 30 micron absoluteand is equipped with an external (15PSID) and internal (29PSID)bypass that ensures the MPDE is not starved for oil if a high

differential pressure condition occurs in the filter. [Encl.

(421)]

106. The external bypass valve has been recently directed byappropriate technical authority to be disabled via a blank



25/68



ANTONIO (LPD 17)

19

flange to prevent contaminates bypassing the filter. [Encl.

(41)]

107. The filter has a rotating set of internal elements thatalternately filter the lube oil when in service. [Encl. (41)]

108. When the internal elements are not in service, they areflushed with lube oil discharged from the lube oil service

pumps. [Encl. (41)]

109. The filter effluent is discharged directly back to thesump, which results in the total mass of contaminates remaining

constant in the MPDE sump. [Encl. (6, 41)]

110. The filtered lube oil discharged from the self-cleaningfilter flows through a heat exchanger, header pressureregulating valve, a duplex strainer (55 micron absolute), and

then into the diesel complex where the lube oil is ultimately

discharged into the MPDE lube oil sump. [Encl. (41)]

111. The attached transfer pump and purifier bowl assembly ofthe lube oil purifier come into contact with the lube oil.

[Encl. (42)]

112. The transfer pump has carbon steel rotors and a bronzehousing. [Encl. (42)]

113. The purifier bowl is made of a special stainless steelalloy. [Encl. (42)]

114. The paring disc is made of 316 Series Stainless Steel.[Encl. (42)]

115. If the paring disc and purifier bowl come into contact,this would cause noticeable sound and vibration. [Encl. (42)]

116. Any contact will be evident on the paring disc top andbottom surfaces. [Encl. (42)]

117. The lube oil piping system contains 304 Series StainlessSteel, 316 Series Stainless Steel alloy and Carbon Steel (from

voyage repair). [Encl. (43)]



26/68



ANTONIO (LPD 17)

20

118. MPDE lube oil service pump special washers used to fastenthe pump casing to its suction inlet bell and mechanical seal

cover assembly are made from stainless steel. [Encl. (44)]

119. MPDE lube oil service pump special washers do not come incontact with lube oil during the normal operation of the lubeoil service system. [Encl. (44)]

120. The valves in the lube oil service system do not containstainless steel. [Encls. (45-47)]

MPDE Bearings

121. The MPDE bearings consist of a steel backing, top layerin overlay, babbit, and copper. [Encl. (48)]t

Internal Cause of Foreign Matter Introduction Into the LO SystemLO Service System Filling/Strikedown Procedures

122. MRC 2331 W-1R requires the drain and refill of the MPDElube oil loop seals. [Encl. (49)]

123. During the performance of MRC 2331 W-1R, SAN ANTONIOpersonnel indicated the use of a blue plastic bucket to perform

the maintenance check. [Encls. (50, 51)]

124. The blue plastic bucket is used to perform other tasks inthe propulsion spaces. [Encls. (50, 51)]

NR 1A MPDE

125. On 18 August 2009, the lube oil quality management (LOQM)log annotated zero gallons in the NR 1A MPDE lube oil sump.

[Encl. (52)]

126. On 9 September 2009 at 1738, the prior to transfer (PTT)sample on lube oil tank 6-65-2-F was satisfactory. [Encl. (53)]

127. On 9 September 2009 at 1855, a lube oil transfer from lubeoil tank 6-65-2-F to NR 1A MPDE lube oil sump commenced. [Encl.

(53)]

128. On 9 September 2009 at 2113, a lube oil transfer from lubeoil tank 6-65-2-F to NR 1A MPDE lube oil sump of 1647 gallons

completed. [Encl. (53)]

129. On 10 September 2009, the LOQM log annotated 2000 gallonsin the NR 1A MPDE lube oil sump. [Encl. (54)]



27/68



ANTONIO (LPD 17)

21

130. On 15 September 2009, the LOQM annotated 1750 gallons inthe NR 1A MPDE lube oil sump. [Encl. (54)]

131. On 18 September 2009 at 0915, the PTT sample on lube oiltank 6-68-2-F was satisfactory. [Encl. (53)]

132. On 18 September 2009 at 1350, there was 538 gallons of oiltransferred to NR 1A MDPE lube oil sump from lube oil tank 6-68-

2-F. [Encl. (53)]

133. On 8 October 2009 at 0329, the PTT sample from lube oiltank 6-122-1-F satisfactory. [Encl. (21)]

134. On 21 October 2009 at 1340, the PTT sample on lube oiltank 6-68-2-F was satisfactory. [Encl. (21)]

135. On 22 October 2009, the LOQM log annotated 1640 gallons inthe NR 1A MPDE lube oil sump. [Encl. (55)]

136. A lube oil transfer is not documented in the engineeringlog. [Encl. (21)]

137. On 23 October 2009 at 0639, the PTT sample on lube oiltank 6-121-1-F was satisfactory. [Encl. (21)]

138. On 4 November 2009 at 1548, NR 2 40WT lube oil purifierstarted purifying NR 1A MPDE sump to 6-117-1-F IAW PTLO. [Encl.

(34)]

139. On 19 November 2009, the LOQM log annotated 1657 gallonsin the NR 1A MPDE lube oil sump. [Encl. (56)]


141. On 30 November 2009 at 0914, NR 1 40WT lube oil purifiertransfer pump started taking suction from NR 1A MPDE lube oil

sump discharging to lube oil tank 6-63-2-F IAW PTLO. [Encl.

(34)]

142. On 30 November 2009 at 1453, NR 1 40WT lube oil purifiertransfer pump secured IAW PTLO. [Encl. (34)]



28/68



ANTONIO (LPD 17)

22

143. On 30 November 2009 at 1453, a 1355 gallon transfer fromNR 1A MPDE lube oil sump to lube oil tank 6-63-2-F completed.

[Encl. (34)]

NR 1B MPDE144. On 18 August 2009, there was a 10 gallon increase in 1BMPDE sump (280 gallons to 290 gallons). No Prior to Transfer

sample taken. [Encl. (52)]



147.On 9 September 2009 at 1309, NR 1 40 WT lube oil purifiertaking suction from lube oil tank 6-68-2-F to NR 1B MPDE. [Encl.

(53)]


149. On 9 September 2009 at 1329, a lube oil transfer commencedfrom lube oil tank 6-68-2-F to NR 1B MPDE sump. [Encl. (53)]


151. On 9 September 2009 at 1729, a transfer of 2118 gallons oflube oil to NR 1B MPDE was completed. [Encl. (53)]

152. On 10 September 2009, the LOQM log annotated 2025 gallonsin NR 1B MPDE sump. [Encl. (54)]







29/68



ANTONIO (LPD 17)

23

157. On 22 October 2009, the LOQM log annotated 1940 gallons inNR 1B MPDE lube oil sump. [Encl. (55)]


159. On 23 October 2009 at 0639, the sample on lube oil tank6-121-1-F was satisfactory. [Encl. (21)]

160. On 23 October 2009, the LOQM log annotated 1769 gallons inNR 1B MPDE lube oil sump. [Encl. (55)]


NR 2A MPDE162. On 4 August 2009, the LOQM log annotated 380 gallons in NR2A MPDE lube oil sump. [Encl. (52)]


164. On 8 August 2009, the LOQM log annotated 280 gallons in NR2A MPDE lube oil sump. [Encl. (52)]


166. On 11 August 2009, the LOQM log annotated zero gallons inNR 2A MPDE lube oil sump. [Encl. (52)]



169. On 15 September 2009 at 0923, NR 2 40WT lube oil purifieraligned and started filling NR 2A MPDE. [Encl. (53)]

170. On 15 September 2009, the LOQM log annotated 1416 gallonsin NR 2A MPDE lube oil sump. [Encl. (54)]




30/68



ANTONIO (LPD 17)

24

172. On 26 September 2009, the LOQM log annotated 1657 gallonsin NR 2A MPDE lube oil sump. [Encl. (54)]


174. On 7 October 2009, the LOQM annotated 1421 gallons in NR2A MPDE lube oil sump. [Encl. (55)]


176. On 8 October 2009 at 0329, the PTT sample on lube oil tank6-122-1-F is satisfactory. [Encl. (21)]

177.On 8 October 2009 at 0340, NR 2 40WT lube oil transferpump aligned and started taking suction from 6-122-1-F and

discharging to NR 2A MPDE sump IAW PTLO. [Encl. (21)]

178. On 8 October 2009 at 0407, secured from transfer of lubeoil to NR 2A MPDE Sump IAW PTLO. [Encl. (21)]

179. On 9 October 2009, 1657 gal annotated in LOQM Logs, +237gal from previous day. [Encl. (55)]

180. The lube oil transfer is documented in the engineering logas occurring two days earlier and accounts for the 179 gallon

increase. [Encl. (21)]

181. On 16 October 2009 at 1236, a PTT sample on lube oil tank6-122-1-F was satisfactory. [Encl. (21)]

182. On 23 October 2009 at 0639, a PTT sample on lube oil tankon 6-121-1F was satisfactory. [Encl. (21)]

183. On 23 October 2009 at 0702, NR 2 40WT lube oil purifieraligned and started suction from 6-122-1-F discharging to NR 2A

MPDE sump. [Encl. (21)]

184. On 23 October 2009 at 0742, secured from transferring 40WTlube oil to NR 2A MPDE Sump. Transferred 445 gal. [Encl. (21)]



31/68



ANTONIO (LPD 17)

25


Possible External Source of the Foreign Matter

LO System Flushing Components

185. The flushing procedure involves the use of muslin bagsthat contain kevlar to catch debris. [Encls. (57, 58)]

External Cause of the Foreign Matter Introduction Into the LO

System

Lube Oil Service System Flushing Process

186. The flushing process for the LPD 17 Class is NAVSEA S9233-DM-HBK-010-007 Diesel Engine Maintenance Standard Flush LPD 17,

MPDE Lube Oil System and Engine. [Encl. (59)]

187. The flushing procedure consists of the following steps:a. The MPDE sump is drained and refilled with a high

detergent, low viscosity special diesel lube oil.

b. The installed lube oil service pumps, self-cleaning

filter element and duplex strainer elements are removed to

prevent a low flow area that could allow the accumulation of

contaminants.

c. A flush rig with pumps, heaters, filters and associated

manifolds are installed as a temporary system.

d. The pumps provide a pressure source for the lube oil,

the heaters maintain lube oil temperature/viscosity at the

required value and the manifolds aligned flushing lube oil to

the appropriate components.

e. Muslin bags are installed in the temporary filter and

are used to capture any contaminants.

f. The individual flushes are timed flushes that are

complete once the contamination found in the muslin bags meets

acceptance criteria.

g. The temporary pumps take suction from the main sump and

in series with the filter discharge clean oil into the externaland/or internal piping where the contaminants in the piping are

discharged into the sump.

h. The general sequence is that the external lube oil

piping is flushed, then the MPDE internal lube oil


32/68



ANTONIO (LPD 17)

26

piping/components are flushed followed by a combined (external

and internal) flush.

i. Upon completion of the flushes the sump is pumped down

and cleaned.

j. The LO Cooler flush can be accomplished at anytime

during the flushing evolution.

k. In addition to the temporary manifold installed to align

flushing lube oil to the external and/or internal piping, two

temporary valves were installed to bypass both pressure

regulating valves serving the lube oil system.

l. The government approved procedure fails to indicate the

required position of these valves at the start of the externalpiping flush.

[Encl. (60)]

188. NAVSEA S9233-DM-HBK-010-007 Note 2 (page 6) states themanual exceeds the requirements of NAVSEA S9086-HB-STM-010 NSTM

Chapter 233 Diesel Engines. [Encl. (59)]

189. NAVSEA S9233-DM-HBK-010-007 does not contain a requirementfor the lube oil sump to be cleaned prior to commencing the

flush. [Encl. (59)]

190. NSTM Chapter 233 paragraph 233-8.21.12.f states that priorto flushing the engine sump shall be drained and wiped clean

with lint free rags. [Encl. (60)]

191. NAVSEA S9233-DM-HBK-010-007 does not indicate location oramount of quadraplex filters required or installed during

flushing. [Encl. (59)]

192. NSTM Chapter 233, Figures 233-8-1 and 233-8-2 depictfilters installed immediately downstream of the flushing pump

and immediately upstream of the sump. [Encl. (60)]

193. NAVSEA S9233-DM-HBK-010-007 Note 18 states that allcheckpoints shall be verified, logged on enclosure (4) of the

procedure and signed by performing activity representative and

DEI before the next step can be performed. Verify with ships

force representative only if required. [Encl. (59)]



33/68



ANTONIO (LPD 17)

27

194. NAVSEA S9233-DM-HBK-010-007 Checkpoint 2 states that DEIor Ship Representative and the contractor shall inspect all

accessible engine areas to verify all foreign material and

debris has been removed. [Encl. (59)]

195. NAVSEA S9233-DM-HBK-010-007 Temporary valves (TV 1 and 2)are installed in the external piping, however the alignment of

these valves are not indicated in Table 1 (Valve Alignment).

Paragraph 3-1.13.1 requires that TVs be opened for a minimum of

5 minutes; however this occurs during the flushing process

allowing the potential for the valves to be improperly aligned

at the start of the flush. [Encl. (59)]

196. NAVSEA S9233-DM-HBK-010-007, paragraphs 1-5.1 and 1-5.4require removal of number 1 and 2 LO service pumps respectively,

prior to commencing flush. [Encl. (59)]

197. The lube oil pumps were not removed prior to the flush.[Encls. (61, 62)]

198. NAVSEA S9233-DM-HBK-010-007 does not require thecontractor to document the foreign material found during the

flush of the system. [Encl. (60)]

Damage Caused to the MPDE Bearings (to include cost)

Bearing Inspection Results

199. An inspection of the following bearings has beenconducted: NR 1A MPDE-Nr 2, 4, 6, and 8 main bearings and thrust

bearing; NR 1B MPDE-all main bearings and thrust bearing; and NR

2A MPDE-all main bearings and thrust bearing. [Encl. (63, 64)]

200. The bearing inspections revealed that all bearing surfacesshowed debris passed through. [Encl. (63-65)]

201. The bearing surfaces reflected debris similar to thefollowing items: industrial, bead blasting material, material

consistent with pipe welding process, dirt and sand, and water.

[Encl. (63-65)]

202. All main bearing backings reflected fretting at the shellback ends with oil was found on the lower bearing steel-backs

during the removal due to inadequate bearing crush. [Encl. (63-

65)]



34/68



ANTONIO (LPD 17)

28

203. NR 1A MPDE thrust bearing lower shell exhibited high heatmarks on the steel backing, which reflect a previous hot bearing

during operation. [Encl. (63-65)]

204. The date and time of the damage to NR 1A thrust bearingcannot be determined due to no ICAS records, i.e., systeminoperable or switched off. [Encl. (63, 64)]

205. NR 1B MPDE thrust bearing surfaces exhibited the passingof large and small debris. [Encl. (63-65)]

206. A 3 millimeter wide and 2 millimeter deep groove wasvisualized in the babbitt surface and embedded debris remained

in the bearing surface and journal. [Encl. (63-65)]

207.The thrust journal has grooves and cuts matching thedebris tracks in the bearing surface. [Encl. (63-65)]

208. The thrust bearing face showed wear consistent withGeisling mis-alignment/improper spacing. [Encl. (63-65)]

209. NR 2A MPDE thrust bearing surfaces exhibited the passingof small debris. [Encl. (63-65)]

210. The wear patterns were not consistent with the lowoperating hours. [Encl. (63, 64)]

211. All bearings removed and inspected meet the discardcriteria in Chapter 4 of the LPD17 Class PC2.5 STC Volume 1

Technical Manual. [Encl. (63, 64)]

212. The current cost estimate to repair NR 1A, 1B, and 2A MPDEis $7.5 million dollars. [Encl. (66)]

Maintenance Performance and Quality Control Efforts

Shipboard Engineering Programs

MPDE Material History

213. The Joint Fleet Maintenance Manual, Volume IV, Chapter 4Diesel Engines, paragraph 4.4b requires that the CommandingOfficer shall ensure that each inspection/assessment is entered

in the applicable machinery history with a brief description of

any repair before operating conditions found at the conclusion

of the assessment/inspection. [Encl. (67)]



35/68



ANTONIO (LPD 17)

29

214. The Joint Fleet Maintenance Manual, Volume IV, Chapter 4Diesel Engines, paragraph 4.5d requires that Engine Operation

Hours since Overhaul and Engine Operating Hours since

Commissioning, at a minimum, shall be kept for all MPDEs, SSDGs,

and EDGs. For Colt Pielstick Engines on LSD-41/49, LPD-17, andLHD-8 Class ships, operating hours since cylinder head

refurbishment and individual power pack change outs shall also

be kept. [Encl. (67)]

215. The Joint Fleet Maintenance Manual, Volume IV, Chapter 4Diesel Engines, Paragraph 4.6b(4) Commanding Officers shall

ensure all diesel records are readily available to include

Diesel Maintenance/history records to include all maintenance

and significant items accomplished on the engine since last

engine overhaul. [Encl. (67)]

216. SAN ANTONIO maintained material history logs usingstandard green hardback log books. [Encl. (68)]

217. The material history documented hours for each MPDE.[Encl. (68)]

218. The material history did not document inspection orssessment for each MPDE. [Encl. (68)]a

3M Program

219. On 19 September 2009, EM01 conducted PMS Check: 2331-017-49 U-11 Change Main Engine Lube Oil System Lubricating Oil.

[Encl. (69)]

220. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]

221. On 20 September 2009, EM01 conducted PMS Check: 2331-017-49 U-35 Clean and Inspect Rocker Arm Lube Oil Strainer. [Encl.

(69)]


223. On 20 September 2009, EM01 completed PMS Check: 2331-017-49 U-11 Change Main Engine Lube Oil System Lubricating Oil.

[Encl. (69)]



36/68



ANTONIO (LPD 17)

30


225. On 20 September 2009, EM01 completed PMS Check: 2331-017-49 U-35 Clean and Inspect Rocker Arm Lube Oil Strainer. [Encl.(69)]


227. On 22 September 2009, EM01 conducted PMS Check: 2640/006-98 M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl.

(69)]


229. The check requires a tag out. [Encl. (70)]230. On 1 October 2009, EM01 completed PMS Check: 2331-017-4918M-8R Measure and Record Crankshaft Deflection Readings. [Encl.

(69)]


232. On 29 October 2009, EM01 completed PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl (Tag Out # E-

1281). [Encl. (69)]


234. On 29 October 2009, EM01 conducted PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl. (69)]


236. On 12 November 2009, EM01 conducted PMS Check: 2331/017-4918M-12R Test Injector Nozzle. [Encl. (69)]




37/68



ANTONIO (LPD 17)

31

238. The check requires a tag out. [Encl. (71)]239. The 13 Week Accountability Log has N/A annotated for thetag out number. [Encl. (69)]

240. On 25 November 2009, EM01 conducted PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl. (69)]


242. On 27 October 2009, EM02 conducted PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl. (72)]


244. MRC 2331 R-12W MPDE Sump Samples is required: weekly inuse. [Encl. (73)]

245. SAN ANTONIO personnel last performed the check in March2009. [Encl. (50, 74)]

246. SAN ANTONIO personnel lined out MRC 2331 R-12W on MIP.[Encl. (50, 74)]

247. MRC 2331 R-13W MPDE Rocker Arm LO Viscosity Sample usingOTC is required: weekly in use. [Encl. (75)]

248. SAN ANTONIO personnel last performed the check in March2009. [Encl. (50, 74)]

249. SAN ANTONIO personnel lined out MRC 2331 R-13W on MIP.[Encl. (50, 74)]

250. MRC 2331 R-14M MPDE Sump Shore Lab Samples is required:Monthly in use. [Encl. (76)]

251. SAN ANTONIO personnel typically performed the check butdid not take credit for performance of the check for PMS

purposes. [Encl. (50)]

252. MRC 2331 U-35 C/I Rocker Arm LO Strainer is required: (a)>5 PSID; and (b) Change Rocker Arm LO. [Encl. (77)]



38/68



ANTONIO (LPD 17)

32

253. SAN ANTONIO personnel did not document the check for NR 2AMPDE, following change out of lube oil in August 2009, in 3M

Program. [Encl. (50)]

254. MRC 2620 U-1 C/I Duplex LO Strainer is required: (a) >15PSID; (b) LO Change; and (c) 1/watch 1st 48 hours aftermaintenance. [Encl. (78)]

255. SAN ANTONIO personnel did not perform check after lube oilchange out or after maintenance. [Encl. (50)]

256. MRC 2620 U-3 Renew LO Filter Elements is required: (a) >15PSID; (b) As directed by results of LO analysis; and (c) Change

LO. [Encl. (79)]

257.SAN ANTONIO personnel did not document the check inconjunction with a lube oil change out.[Encl. (50)]

258. Ships Force provided Memorandum of Record reportingcompletion of MRC. [Encl. (50)]

Lube Oil Quality Management Program

259. SAN ANTONIO personnel made no meaningful attempt tomonitor JOAP/NOAP monthly preventive maintenance lube oil

samples. [Encl. (81)]

260. Chain of command review of the LOQM logs were cursory andnot performed on a daily basis. [Encl. (81)]

261. Quarterly calibration of the onboard KITTEWAKE lube oilanalysis equipment was not performed. [Encl. (81)]

262. Over the period of February to May 2009, the SAMM Databaseconsistently revealed high levels of engine wear products to

include Iron (Fe) in NR 1A, 1B, and 2A MPDE lube oil. [Encl.

(80, 81)]

263. EOCC Procedure Hot Bearing in a Main Engine (MHMEB) doesnot require a visual inspection of the strainers or filter mediafor the presence of babbit which would provide an immediate

indication of bearing failure. [Encl. (82)]

264. EOCC Procedure Unusual Noise and Vibration in a MainPropulsion Diesel Engine (MNVME) does not require a visual

inspection of the strainers or filter media for the presence of



39/68



ANTONIO (LPD 17)

33

babbit which would provide an immediate indication of bearing

failure. [Encl. (82)]

Training Program

265. An audit of FLTMPS TYCOM required courses associated withlube oil systems was conducted and the following deficiencieswere noted:

a. One of fifty required Sailors have completed the LPD 17

Class Lube Oil System Familiarization Course (LPD-005)

b. Zero of two required Sailors have completed the

Propulsion Alarm and Indication Course (A-651-0047)

c. One of fifty required Sailors have completed the LPD 17

Class Fuel Oil System Familiarization Course (LPD-004)

d. Five of twenty-three required Sailors have completed the

LPD 17 Class Electrical Systems Familiarization Course (LPD-006)

e. Two of sixty-five required Sailors have completed the

LPD 17 Class Electrical Generation and Distribution Course (LPD-

007)

f. One of twenty-three required Sailors have completed the

LPD 17 Class AC and Refrigeration Course (LPD-014)

g. Two of fifty-eight required Sailors have completed the

LPD 17 Class Reverse Osmosis Familiarization Course (LPD-017)

h. Six of eighty-nine required Sailors have completed the

LPD 17 Class Compressed Air System Familiarization Course (LPD-

019)

i. Five of eighty-eight required Sailors have completed the

LPD 17 Class Engineering Control System (ECS) Course (LPD-032)

j. Two of forty-five required Sailors have completed the

LPD 17 Class Shipboard Status Monitor System Course (LPD-036)

k. Three of forty-two required Sailors have completed the

LPD 17 Class Main Propulsion Diesel Engine Operation Course

(LPD-103A)



40/68



ANTONIO (LPD 17)

34

l. Two of fifty-eight required Sailors have completed the

LPD 17 Class Ships Service Diesel Generator Operation Course

(LPD-103B)

m. One of seventy-four required Sailors have completed theLPD 17 Class Steering Gear Machinery System Operation Course

(LPD-103D)

n. Two of twenty-five required Sailors have completed the

LPD 17 Class Controllable Pitch Propeller Operation Course (LPD-

103E)

o. Two of twenty-five required Sailors have completed the

LPD 17 Class Fuel Oil Purifier Operation Course (LPD-104)

p. Two of twenty-five required Sailors have completed theLPD 17 Class MPDE LO Purifier Operation Course (LPD-105A)

q. Two of twenty-five required Sailors have completed the

LPD 17 Class Reduction Gear LO Purifier Operation Course (LPD-

105B)

r. Two of seventeen required Sailors have completed the LPD

17 Class Electrical System Operation Course (LPD-106)

s. Three of fifteen required Sailors have completed the LPD

17 Class Refrigeration Plant Operation Course (LPD-114B)

t. One of twenty-five required Sailors have completed the

LPD 17 Class Reverse Osmosis Operation Course (LPD-117)

u. Three of forty-five required Sailors have completed the

LPD 17 Class Shipboard Status Monitor Subsystem Operation Course

(LPD-136)

v. One of thirty-eight required Sailors have completed the

LPD 17 Class Steering Gear Machinery Maintenance Course (LPD-

203D)

w. Three of forty-eight required Sailors have completed the

LPD 17 Class Controllable Pitch Propeller Maintenance Course

(LPD-203E)

x. Zero of thirty-five required Sailors have completed the

LPD 17 Class Fuel Oil Purifier Maintenance Course (LPD-204)



41/68



ANTONIO (LPD 17)

35

y. One of thirty-five required Sailors have completed the

LPD 17 Class Main Reduction Gear LO Purifier Maintenance Course

(LPD-205A)

z. Two of twenty-five required Sailors have completed theLPD 17 Class Fuel Oil Purifier Operation Course (LPD-104)

[Encl. (83)]

266. Engineering Department has not completed many required NKOcourses including courses associated with ship control systems

(ECS/MCS), main propulsion and lubricating oils. [Encl. (84)]

267. Three months of training records for the period ofFebruary to December 2009 were unavailable. [Encl. (84)]

268.Despite a post deployment leave period in March/April2009, MP Division reported 100% attendance at all documented

training (typical reports stated that 25 of 25 Sailors attended

training). [Encl. (84)]

269. All MP Division training conducted since February 2009 wasdocumented as being conducted by the same second class petty

officer. [Encl. (84)]

270. The SAN ANTONIO Training Instruction requires that aSubject Matter Expert conduct the training. [Encl. (84)]

271. Since February 2009, MP Division conducted training threetimes on a system that did not exist on SAN ANTONIO: Diesel

Engine Brake System. [Encl. (84)]

272. From February to December 2009, training was not conductedon MPDE bearing high temperature casualties, loss of lube oil

pressure or loss of lube oil cooling casualties. [Encl. (84)]

273. Two lesson plans (Lubricating Oil Storage, Transfer andPurification and Diesel Engine Clutch and Brake System) were

dated December 2000. [Encl. (84)]

274. Five lesson plans (Lubricating Oil Quality Management,Diesel Main Reduction Gear, Bearing Records, Air System

Casualties and Diesel Engine Lube Oil System were dated 2005

(various months). [Encl. (84)]

275. The Diesel Lube Oil System lesson plan:



42/68



ANTONIO (LPD 17)

36

a. Did not address the self-cleaning filter or strainer

inspection criteria and potential sources/types of

contamination, or typical lube oil casualties (low level, low

pressure, loss of cooling (high temperature)

b. Had incorrect or incomplete criteria for the shift clean

and inspection of the lube oil strainers

[Encl. (84)]

276. The Lube Oil Storage, Transfer and Purification Systemlesson plan did not address required samples or appropriate lube

oil log entries in support of transfers. [Encl. (84)]

277. The Lube Oil Quality Management lesson plan did notaddress NSTM requirement to periodically test lube oil analysis

equipment, or weekly lube oil sample analysis when in operation,or the means to obtain the lube oil test results via the SAMMs

database and the requirement to maintain lube oil in

specification. [Encl. (84)]

SAN ANTONIO Class Engineering Systems

Shipboard Automated Maintenance Management System

278. Shipboard Automated Maintenance Management (SAMM) Systemprovides electronic means for ships force to track lube oil

analysis results. Ships force samples are sent to the lab.

Results are posted on the internet in the SAMMS folder. [Encl.

(85, 86)]

279. Based on oil sample results, recommendations to ship'sforce are developed by NSWC Philadelphia and forwarded to the

ship via email. [Encl. (85, 86)]

280. PREDICT is contracted by the Navy to conduct lube oilsample analysis. The lab is required to provide analysis of

samples two working days after the sample is received at the

lab. [Encl. (85, 86)]

281. Currently NSWC Philadelphia has tasked PREDICT to conductferrography analysis on samples for LPD 17 class ships. [Encl.(85, 86)]

282. Lube oil sample shipping costs are paid for by the programunder NSWC Philadelphia. [Encl. (85, 86)]



43/68



ANTONIO (LPD 17)

37

283. Based on location of the ship, and the methodology used toship samples (transfer to support ship, inport, etc) samples may

be received in the PREDICT lab out of sequence. [Encl. (85, 86)]

284. SAN ANTONIO has a stand alone version of SAMMS. Insteadof automatically sending data to SAMMS, ships force mustdownload information to a disc, and send information via the

internet. [Encl. (85, 86)]

285. With a stand alone system, ships force does not havedirect access to the SAMMS folder or lube oil analysis trend

data. [Encl. (85, 86)]

286. NSWC Philadelphia calibrates Kittiwake equipment onboardships every two years. [Encl. (85, 86)]

287. Training on SAMMS is scheduled for ships, and additionaltraining is available upon request. [Encl. (85, 86)]

288. Crew members from SAN ANTONIO received SAMMS training inApril 2008, July 2008, and August 2008 prior to the ships

deployment. [Encl. (85, 86)]

289. Ships force had the capability via their Port Engineer togain access to the monthly lube oil samples but did not attempt

to do so because of their level of knowledge of the system.

[Encl. (87-89)]

290. The current work around is that based on oil sampleresults recommendations to ship's force are developed by NSWC

Philadelphia and forwarded to the ship via email. [Encl. (87-

89)]

Engineering Control System

291. LPD 17 class Engineering Control System (ECS) is amicroprocessor-based real-time distributed data acquisition,

control, processing, and display system that:

a. provides remote monitoring and control of propulsion,electrical, auxiliary, fuel, damage control, and ballast systems

b. consists of 20 operator Multi-Function Workstations

(MFWS) and 35 Data Acquisition Units (DAUs)



44/68



ANTONIO (LPD 17)

38

c. uses the Shipboard Wide Area Network (SWAN) to

distribute data to and from ECS consoles

[Encl. (90)]

292. SWAN and ECS are commercial off the shelf systemsintegrated and maintained by Raytheon Incorporated during thenew construction process. [Encl. (90)]

293. DAU circuit boards, also called VME Remote Terminal Units(VRTU), are the interface between engineering plant sensors and

ECS and are designed to accept inputs from multiple sensors.

[Encl. (90)]

294. The DAU configuration allows interruption of multiplesensor signals if one sensor electrically fails. [Encl. (90)]

295. Troubleshooting of the DUA is complex and tedious becausethe technician must search each signal wired to the VRTU for the

failed sensor. [Encl. (90)]

296. After ship delivery in 2006, SAN ANTONIO and an outsidemaintenance community documented problematic ECS operation,

troubleshooting, and repair. [Encl. (91)]

297. PREINSURV findings during Final Contract Trials held from26-30 March 2007 included:

a. Significant deficiencies associated with the SWAN that

negatively impacted ECS operation.

b. LPD 17 Class integrated support was marginal at best.

[Encl. (92)]

298. PREINSURV findings during Material Inspection (MI)completed 26-30 November 2007 included:

a. Ships force lacked the fundamental troubleshooting tools

to attain self-sufficiency in effectively maintaining the SWAN

and ECS because of system software upgrades and installation ofalterations without supporting technical documentation

b. Technical documentation was not available because of

proprietary rights (i.e., MPDE engine controls technical

documentation)



45/68



ANTONIO (LPD 17)

39

c. Integrated logistic support necessary to troubleshoot

analog and digital signals between installed equipment and ECS

DAUs did not exist

d. At the start of the MI, ships force and NSSA technicianswere actively troubleshooting 55 out of service faults and 71

unknown faults displayed at ECS consoles

e. Optimal manning, existing level of knowledge, and lack

of technical documentation had combined to exceed available

ships force resources to operate the engineering plant

f. During underway portion the MI, an average of 20 alarms

per minute displayed by ECS during underway operations.

g. Watchstanders had become conditioned to ignore alarms.[Encl. (93)]

299. ECS maintenance after INSURV focused on correcting shorepower interface problems and restoring DAU circuit boards to

reduce the number of ECS faults in support of the February 2008

OPEVAL. [Encl. (94, 95)]

300. Significant post OPEVAL maintenance included theintermittent loss of speed and heading at the Bridge and

Steering Helm console. [Encl. (94)]

301. Technical community approach was to complete a system wideassessment of SWAN to determine root cause of signal failure.

[Encl. (96)]

302. NAVSEA PMS 317 funded an onboard SWAN subject matterexpert for the entire deployment to assist ships force with

troubleshooting/repair and defining SWAN/ECS interface problems.

[Encl. (97, 98)]

303. LPD 17 Class precommissioning crews received SWAN and ECStraining directly from Raytheon during the commissioning

process. [Encl. (91)]

304. Upon ship arrival in Norfolk, no RMC technicians weretrained in the ECS system. [Encl. (91)]

305. NAVSEA-Philadelphia Code 955 has been designated as InService Engineering Agent for ECS. [Encl. (91)]



46/68



ANTONIO (LPD 17)

40

306. Access to Raytheon for shipboard ECS technical support wasconvoluted due lack of a contract vehicle that could be used by

the Regional Maintenance Centers (RMCs) for emergent

troubleshooting and repair. Raytheon support is funded by

NAVSEA PMS 317. [Encl. (100)]

307. A post delivery support assist process was used by the RMCMaintenance Team to interface with PMS 317 and Raytheon on a

case by case basis. This process resulted in delays in system

restoration. [Encls. (101, 102)]

308. ECS reliability issues caused SAN ANTONIO to set onaverage three additional engineering watch standers during

normal operation. [Encl. (87, 88)]

309.In mid-December 2009, the Commanding Officer had approvedapproximately fifty Temporary Standing Orders (TSOs) that

authorized ships force to modify EOSS or add additional duties

to watch standers to locally monitor, operate or verify the

proper operation of equipment vice remote operation and

monitoring via the Engineering Control Systems (ECS). [Encl.

(87, 89, 90, 103)]

ICAS

310. ICAS provides a distributed set of six personal computers(PC) workstations and associated Portable Data Acquisition

(PDA), Portable Data Terminal (PDT) and Data Acquisition Set-64

(DAS-64) capable of obtaining and displaying engineering plant

equipment status data. [Encl. (104)]

311. ICAS maintains records of individual plant equipment in aship unique Configuration Data Set (CDS), for monitoring and

trend analysis functions. [Encl. (104)]

312. ICAS is a non-mission critical subsystem that only runs asa stand alone application on the unclassified side of the SWAN

network. [Encl. (104)]

313. The Channel B connection (dual hone) to SWAN modificationsfor each ICAS work station were completed during the Post

Shakedown Availability (2 April 2007 25 June 2007). [Encl.

(105)]

314. During the PSA sea trials, the Main Machinery Room NR 1ECS console and ICAS console desynched a total of three times:



47/68



ANTONIO (LPD 17)

41

ECS console desynched one time; ICAS console desynched two

times. [Encl. (105)]

315. The desynchronization of one console did not affect theother console. [Encl. (105)]

316. NAVSEA ICAS and ECS representatives were onboard duringsea trials. [Encl. (105)]

317. On 18 July 2007, NAVSEA ICAS representatives left SANANTONIO with all five ICAS workstations and one ICAS viewport

operational. [Encl. (105)]

318. In October 2007, NAVSEA ICAS representative participated in an ICMP assessment aboard SAN ANTONIO. [Encl.

(105)]

319. In October 2007, NAVSEA ICAS representativevisited SAN ANTONIO to update logsheets and install new ECS

software. [Encl. (105)]

320. On 1 March 2008, NAVSEA ICAS representativereceived email from stating ICAS was up and running,

but needed to install a new logsheet. [Encl. (105)]

321. On 7 March 2008, NAVSEA ICAS representative received email stating that during an OPEVAL: "ICAS fault

caused ECS to desync during ballast operations with a LCU in the

well delaying the evolution." [Encl. (105)]

322. In July 2008, NAVSEA ICAS representativeattempted to further assist ship via distance support (email)

but lost communication. [Encl. (105)]

323. The LPD 17 Class de-synchronization issue is caused by theShip Wide Area Networks (SWAN) handling of network

communications amongst the various ECS consoles. [Encl. (106)]

324. The de-synchronization problem was initially encounteredduring SAN ANTONIOs builder trials (prior to ICAS

installation). [Encl. (106)]

325. The recommended solution was the five ICAS workstationsbecome dual honed with a Channel B connection and network card.

[Encl. (106)]



48/68



ANTONIO (LPD 17)

42

326. The installed solution differed in that a shared ChannelB connection with the neighboring ECS console was implemented.

[Encl. (106)]

327. The five ICAS consoles thus shared a pre-existing channelB connection with a neighboring ECS console. [Encl. (106)]328. The installation of the GIG E network onboard USS NEWORLEANS (LPD 18) in Spring 2008 has further mitigated the

potential of de-synchronization since it is more robust than the

original ATM solution. [Encl. (106)]

329. NEW ORLEANS has been using ICAS since Spring 2008. [Encl.(106)]

330.Although SAN ANTONIO received the GIG E upgrade in (fall2009) she is reluctant to utilize the ICAS consoles for fear of

desynchronizing the entire control system. [Encl. (106)]

331. NAVSEA ICAS representatives believe that SAN ANTONIO haslost confidence in ICAS and will provide her training and a ship

rider during her upcoming trial in January 2010. [Encl. (106)]

332. Ships force has mitigated the risk associated with ICASbeing secured on SAN ANTONIO by setting additional watches and

manually logging and trending propulsion plant parameters.

[Encl. (88)]

333. Operating parameters and other equipment informationmonitored by ICAS is designed to be automatically sent off ship.

[Encl. (107)]

334. This interface has not been enabled on LPD 17, so shipsforce would conduct a manual push of the data off ship on a

periodic basis if ICAS was operational. [Encl. (107)]

335. Off ship subject matter experts would review the data andmake recommendations with regard to operations and condition

based maintenance. [Encl. (107)]

336. Until recently, only automated reports associated for airconditioning systems has been funded and available for ships

force although ICAS is designed to monitor the entire propulsion

plant. [Encl. (108)]



49/68



ANTONIO (LPD 17)

43

337. ICAS support would be in place to monitor MPDE performancefor LPD 19 deployment. [Encl. (5)]

Navy Training System Plan

338. The NTSP for USS San Antonio (LPD 17) Class AmphibiousTransport Dock Ship is NSTP N85-NTSP-S-30-9603B (01 Jul 09).[Encl. (109)]

339. NSTP N85-NTSP-S-30-9603B page xii states that the 01 July2009 version was prepared to document changes since the

previously approved NSTP in May 2003. [Encl. (109)]

340. NSTP N85-NTSP-S-30-9603B pages I-3 to I-4 states: LPD 17Class ships were introduced into the Fleet beginning with the

commissioning of USS SAN ANTONIO in January 2006. Despite the

relative newness of the Class, modest improvements,modernization and changes have already been made that

necessitate updating this NTSP. These ships were built with the

primary goal of reducing the total ownership cost (TOC) while

maintaining and improving readiness and reducing manning through

greater emphasis on technological enhancements, distance

learning, continuous learning systems, job/training aids,

distributed/joint learning, virtual reality, and multi-media

training. Introduction of these technologies provides the crew

and embarked forces with training tools that provide the

flexibility to adjust to meet mission objectives. The use of

electronic media has greatly improved the overall capacity for

the ship to train itself. This advance training technology

insertion, combined with implementation of advance Condition

Based Maintenance technologies and the use of Integrated

Condition Assessment System (ICAS) to eliminate work and

increase system readiness, provides the LPD 17 class with a

comprehensive program that reduces TOC and meets reduced manning

requirements. [Encl. (109)]

341. NSTP N85-NTSP-S-30-9603B page I-5 states: TSTS. TheTSTS provides the LPD 17 Class ships with centralized

coordination and management of training. It maintains the

individual and team training records including courses/eventsrequired and completed, qualifications attained, and evaluations

of training effectiveness. ICW courses can be taken from any

work station computer or tactical console with the results

interfacing directly with TSTS. TSTS also supports and records

the results of stand alone training exercises using the embedded

training features of specific systems. The Navy Integrated



50/68



ANTONIO (LPD 17)

44

Application Product Suite (NIAPS) and Navy Knowledge Online

(NKO) Afloat will replace TSTS functionality once the LPD 17

Class ICWs and IETMs are compatible with and hosted on NKO.

[Encl. (109)]

342. The Ships Electronic Classroom was never finished and iscurrently used as the ships gym. [Encl. (83, 88)]

343. The TSTS Server is located in the Learning Resource Center(LRC) but is not connected to the ships LAN or desk top

computers in the LRC. [Encl. (83, 88)]

344. The work around for TSTS is that classes that should beresident on the TSTS are available via Navy Knowledge Online

(KNO). Course completion is automatically posted to FLTMPS

database. Qualifications are manually tracked using onboarddata bases. [Encl. (83, 87, 88)]

345. Engineering Department has a low completion rate of NKOcourses. [Encl. (83)]

346. In the opinion of the Training Officer, the training department is too small (approximately four

Officers/Sailors) to track and record training. [Encl. (83)]

347. The ships training instruction was signed by the currentCommanding Officer in February 2009. [Encl. (83)]

348. The Training Officer was unfamiliar with several aspectsof the ships training instruction including lesson plan

approval, record keeping requirements and monthly training

reports. [Encl. (83)]

349. NSTP N85-NTSP-S-30-9603B page I-7 under OrganizationalLevel states: The LPD 17 maintenance strategy calls for the

maximum use of onboard O-Level maintenance capabilities through

state of the art maintenance aids such as the Module Test and

Repair capability, ICAS, ETMs, and IETMs. PM is accomplished in

accordance with the ships Planned Maintenance System (PMS)Schedule. The performance of PM maximizes the service life and

minimizes equipment down time. PM tasks include condition

directed and time directed tasks. Every effort should be made

to reduce the number of time driven tasks and make optimum use

of condition monitoring equipment instead [Encl. (109)]



51/68



ANTONIO (LPD 17)

45

350. NSTP N85-NTSP-S-30-9603B page I-7 under CorrectiveMaintenance states: CM incorporates the use of ETMs/IETMs and

interactive diagnosis to reduce MTTR. The installation of ICAS

affords shore based knowledge aboard ship, thereby reducing

trouble shooting time and maintenance-induced failures andimproving the operational efficiency of the ship. [Encl. (109)]

351. NSTP N85-NTSP-S-30-9603B Appendix A discusses a LPD 17Class Stakeholder meeting and training summit held 22 January

2008. [Encl. (109)]

352. The Stakeholders meeting identified five major systemsaboard the LPD 17 Class ships that presented critical operation,

maintenance and training concerns. These systems included the

Main Propulsion Diesel Engines (MPDE), the Ship Service Diesel

Generators (SSDG), the Engineering Control System (ECS), theShip Wide Area Network (SWAN), and the MK 46 MOD 30mm Gun

Weapon System(GWS). All of these systems except the MK 46 GWS

are commercial off the shelf systems that are unique to LPD 17

Class and for which no Equipment/System/Subsystem (E/S/S) Navy

Training System Plan has yet been developed and no formal Navy

schoolhouse/pipeline training currently exists. A major outcome

of this meeting was that TEAM SHIPS and all the Stakeholders

established, in accordance with OPNAVINST 1500.76A, a Training

Planning Process Methodology (TRPPM) Advisory Board to plan the

way ahead to determine the long term training solution and

update this LPD 17 Class NTSP. [Encl. (109)]

353. The LPD 17 Class training summit was called to addressthe action items identified during the Stakeholders Meeting.

The four main tasks assigned were to:

a. Describe the immediate training requirements for USS SAN

ANTONIO (LPD 17)

b. Describe initial training for LPD 17 Class Pre-

Commissioning Crews

c. Determine what the interim training plan for LPD 17

Class follow-on crews should be



52/68



ANTONIO (LPD 17)

46

d. Develop the process for determining the formal long term

training solution for the LPD 17 Class and brief the Way

Ahead.

[Encl. (109)