LMC Update - R2E Failures December 5 th 2012 (R2E) Mitigation Project:LMC Update December 5 th 2012 R2E SEE Update M. Brugger on behalf of the R2E Project Review follow-up: LinkLink !!! Many Thanks To Everybody !!! LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Power-Converter Riddle AC/DC Problem understood Full Mitigation (LS1) New Failures Vacuum PLC in P7 Increase in Rad-levels New Critical Area Cryo Mecos in UX45 P4 10x more radiation in 2012 Heaviest Radiation Test & Biggest Burn H4Irrad last run (11 parallel User) Spectacular final failure Introduction LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Critical areas well within predictions Operational parameters are important Other areas not to be forgottenn e.g, UX45: 2011: 2.6x : almost 2x10 7 cm -2 Radiation Levels Predictions versus Observations (scaled to exp. lumi) 1.1x x x x x x x x10 7 and Today Decrease: TCL closed Increase: Tight Collimators LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 xMasBreak Actions very efficient Mecos AMB Events (P4/8) 4 A. Suraci et al. TE-CRG-OA Better cleaning/control of the beam CV891/fb ? Cryogenics TS2/3 Period interesting LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 Difference in the periods between Technical Stops? Clear correlation Reason? (Vacuum [Scrubbing, Solenoids], Satellite Bunches, ?) Interesting behaviour also in P4 5 Cryogenics LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th H4irrad tests confirmed expected sensitivity Cryogenics UPS FIRE No Beam (4h40) Beam without Target (7h40) Destructive SEU AMB Mecos Linde Type (P4/P8) S2M X X X X X X X 2X X X XXXX Vacon X No Beam (12h10) SEU Similar to P4 events 3h502h40 B. Ivens, F. Morant, M. Pezzetti et al. LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Locations: UJ56, UJ76, UX45, US85 Failure types: Mecos AMB Profibus communications Mitigations: AMB case (UX45, UX65, US85): Relocation under study [Mecos, S2M] Profibus: Relocation from UJs during LS1 Main Impact: Important contribution to R2E downtime Cryogenics DumpDump/TBCNo DumpNo Dump/TBC 4210 Status November 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th QPS SEE Related Failures DumpDump/TBCNo DumpNo Dump/TBC Locations: UJs, RRs, DS Failure types: Digital DAQ systems for nQPS 600A protection uFIP communication Mitigations: New DAQ system under design (FPGA) Firmware upgrades BricoFip (mid-term)/NanoFip (long-term) Main Impact: caused most of the R2E dumps Status November 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th QPS SEE Related Failures R2E consolidation 2012: 27 R2E trips during stable beam >LS1: with all foreseen measures in place: 1 R. Denz, J. Steckert 2011: 3.5 dumps per fb : 1.4 dumps per fb -1 1/3 of errors in UJ14/16/56 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Locations: UJs, RRs, ARC Failure types (only confirmed ones kept in follow-up): 600A: Aux Power Supply [14] FGC [10] 120A failure [1] Mitigations: AC/DC power supply problem understood New FGClite design (on-going) Radiation tolerant power converters (started) Main Impact: Many destructive failures Power Converters DumpDump/TBCNo DumpNo Dump/ TBC Status November 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Failure Overview Dump ConfirmedDump TBCNo DumpNo Dump TBC Destructive failures Dump ConfirmedDump TBCNo DumpNo Dump TBC Preliminary! Status November 2012 (all failures) LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Total R2E related downtime ~250 (confirmed) - ~300h (incl. TBC) Mainly based on dump cases (others difficult to track) Downtime Analysis Preliminary! Status November 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 What Changes After LS1 13 Main impacted systems observed failure signatures: Equipment Upgrades New developments Relocation (UJs, US85, UX45 partly) Shielding (RRs) (Future) Commercial systems: Full relocation (except REs) No new installation in critical areas without radiation tests & developments >LS1 R2E related failures: Compatible with nominal/ultimate goal Enormous Effort during LS1 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Tunnel Equipment: DS/ARC will get critical Long-term damage to be considered (cells 8-13) Impact of 25ns operation? Future upgrades/new installations RRs & Power Converters FGClite prototype in time and to be deployed Power-part R&D started & component tests on-going Areas to be watched UX45!, (UX25), UX65, UJ/UA23, UJ/UA87 (+REs) What Remains After LS1 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Implementation has started General layout soon to be frozen Optimization (RP, CV, Tests) on-going Solutions for rail system investigated 1 st Planning exists DIRAC dismantling critical to be ready in time Aiming for 2014 Operation PS-EAirrad LMC Update - R2E Failures December 5 th 2012 R2E: Past/Present/Future ~3 dumps per fb ~12 dumps per fb /12 xMasBreak Early Relocation + Additional Shielding + Equipment Upgrades Several shielding campaigns prior Relocations on the fly + Equipment Upgrades < 0.5 dump per fb -1 R2E-Project aiming for >LS1 (nominal -> ultimate) 2012 SEE Failure Analysis -Equipment 4+ LHC Points (>100 Racks, >60 weeks of work) -Additional shielding -Critical system upgrades (QPS, FGC) 16 Status November 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th , xMasBreak and previous actions very efficient 2012 Failures: Within Chamonix predictions and the winner is: QPS + EPC (+Cryo) LS1 Mitigation Enormous relocation campaign (4+ Points) Remaining shielding Numerous system upgrades (QPS, EPC) R2E test facility in preparation Long-Term Damage & 25ns Operation -> impact on radiation levels in ARC/P4/P6 Conclusions LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th BACKUP LMC Update - R2E Failures December 5 th 2012 UJ overview High Energy Hadron Fluence [ /cm -2 ] Area /fb /fb -1 Comments UJ142.8* * x x10 6 Physics deb. UJ162.2* * x x10 6 Physics deb. UJ231.5* * x10 5 (*)1.5x10 5 (*)Injection UJ563.9* * x x10 6 Physics deb. UJ766.4* * x10 6 (*)3.4x10 6 (*)Collimation UJ873.0* * x10 5 (*)1.5x10 5 (*)Injection P1: decrease by factor of 6x, thanks to additional shielding P7: increase due to tight collimator settings P2/8: reduction because of cleaner injections in the LHC (*) = ATLAS/CMS luminosity 19 LMC Update - R2E Failures December 5 th 2012 RR overview P1/5: decrease up to a factor of 2x (1.5-2x) due to TCL.4 closure P7: asymmetry L/R (also observed in operation) High Energy Hadron Fluence [ /cm -2 ] Area /fb /fb -1 Comments RR137.2* x x x10 5 Physics deb. RR178.4* x x x10 6 Physics deb. RR531.2* x x x10 6 Physics deb. RR571.0* x x x10 6 Physics deb. RR737.7* x x10 6 (*)2.4x10 6 (*)Collimation RR771.1* x x10 6 (*)1.2x10 6 (*)Collimation (*) = ATLAS/CMS luminosity 20 LMC Update - R2E Failures December 5 th 2012 UX overview (4/6/8) P4: increase, beam-gas coupled with higher cumulative current in the machine How much can/will it go up in view of 25ns operation? High Energy Hadron Fluence [ /cm -2 ] Area /fb /fb -1 Comments UX452.6* x x10 5 (*)8.2x10 5 (*)Beam-gas UX65N/A~1.x10 6 N/A Beam-gas UX852.1* x x10 8 (**)1.6x10 8 (**)Physics deb. US854.4* x x10 7 (**)4.1x10 7 (**)Physics deb. (*) = ATLAS/CMS luminosity (**) = LHCb luminosity 21 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Leakage into DS getting important Cells 7-10: ~10Gy Cell17: ~0.1Gy Currently relatively low vacuum activity ?How will this possibly change with 25ns? ARC & DS LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Beam Loss Analysis (B1) Injection: ~7-8x10 16 Lost protons: Total: ~1x10 16 Experiments: ~3.4x10 15 Rest: ~6.6x10 15 Collimation: IR3: ~6x10 14 IR7: ~6x10 15 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Beam Loss Analysis (B1) Injection: ~7-8x10 16 Lost protons: Total: ~1x10 16 Experiments: ~3.4x10 15 Rest: ~7.4x10 15 Collimation: IR3: ~7x10 14 IR7: ~6.7x10 15 LMC Update - R2E Failures December 5 th 2012 Cryo Actions Taken CV891: automatic reset implemented into PLC (action: switch off 24 Vdc) TT89x: modification of electronics card PLC: moved into better protected zone (P4-P6-P8 caverns) 24 Vdc: longer term, to be moved into a safer protected zone and better distribution of power consumption to be done. AMB (Magnetic bearing controllers): relocated partially in P8, (2 nd unit will be done at LS1, additional shielding foreseen; at P4, will be done during LS1 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th UJs: relocated DSP: To be replaced by rad-tol version + Firmware upgrade of the uC based boards oQPS, nQPS: Firmware upgrade for DAQ systems, dipole splice supervision boards, current lead protection systems (RR) Detectors with rad-tol DAQ in cells 8-12 (nDQQDL) IPQ/IPD/IT: Rad-tol DQQDI in all RRs (200 boards already produced) 600A: Rad-tol quench detection board (prototype developed) Optional: Special rad-tol splice superivsion and BricoFIP TM QPS R2E Upgrades LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Vacuum DumpDump/TBCNo DumpNo Dump/ TBC 4300 Locations: UJ76 (all confirmed), Point 4 (TBC), Point6 (TBC) Failure types: Loss of the PLC control (soft) Power supply (destructive failures) Mitigations: Relocation from UJ76 Status November 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th Failure Statistics LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012 LMC Update - R2E Failures December 5 th 2012