TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
LS1The upgrade of
600 AENERGY EXTRACTION SYSTEMS
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
600 A EE; what, where, how ? Reliability Hardware Failures to be cured Why upgrade? Failure causes Solutions LS1 actions Resources Planning Showstoppers Non LHC activities during LS1
Overview
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Energy Extraction 600 A circuit
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
SnubberCapacitor
0.8 mF
Fuse32 A
SwitchA
SwitchB
SwitchZ
Equalizing Resistors
each
600 A Energy Extraction Facility
600 A warm cables 400mm2
600 A warm cables 400mm2
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
600A Energy Extraction Systems
202 systems installed in the LHC tunnel in corrector circuits with stored energy between 2.2 and 150 kJ
In 15 different locations; 8 x UA parallel service tunnel and 6 x RR and 1 x UJ tunnel
caverns
Systems developed in close collaboration between CERN and the Budker Institute of Nuclear Physics (BINP), Novosibirsk, Russia.
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Reliability
Systems reliability from protection point of view:No opening failures during Fast Power Abort events or tests
(> 100.000 openings since 2005)
2 independent opening mechanisms/circuits per breaker; guaranteeing 6-fold redundancy
Calculated by Antonio Vergara in December 2003: System Failure probability in 20 years of operation < 0.01% (Preventive maintenance!!)
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Reliability (2)
Systems reliability from hardware failure point of view: 6 hardware failures in 2010 1 hardware failure in 2011 3+1* hardware failures in 2012
For operation: 2 failures per year – not in shadow Mean Time To Recovery : 4 hours
(including preparation, access and travel) All failures were “fail-safe” for the s.c. circuits
reducing the availability of the system.
Changes with respect to 2010 operational year :
Preventive maintenance with help of automatic analysis and tracking software tool.
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Hardware Failures treated during LS1
5-10/year - 95% of all failures: Breaker refuses to close Loosening of holding coil mechanism – LHC re-start delayed
1/year - Breaker is physically open but indicates “CLOSED”
micro-switch not actuated / main axle – LHC re-start delayed
1/year - Breaker main contact re-bounces on micro-switch main axle – LHC re-start delayed
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Why do we upgrade? Correcting measures are relatively easy < 100 kCHF
This upgrade will improve the availability of 600 A EE systems for LHC operation by 4-8 hours per year….
Less preventive maintenance
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Failure causes
1- Loosening of holding coil mechanism Weak design of U-shape
2- Micro-switch not actuated Tolerances in dimensions of housing
3- Main axle fixation Weakness during production – not an increasing
fault
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
1- Loosening of holding coil mechanism Weak design of U-shape and dimensional tolerances
Solutions
Needs to achieve Plate Glue Action in all 606 breakers
Open all breakers (anyway foreseen for regular maintenance)Plate gluing – Once open: 2-3 minutes per breaker
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
2- Micro-switch not actuated (temporarily) Tolerances in dimensions of housing
Already tracked by automatic analysis software toolLeading to an unscheduled but still preventive maintenance
intervention (in the shadow)
On-going but to be extended with advice in form of logbook entries (MPE-MS)
Solutions cntd.
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
3- Main axle fixation A loose main axle means no current in 2 oo 3 phases in
the CB leading (> 200A circuit current) to overheating of the 3rd phase.
Weakness during production – no increasing numbers Manufacturer has improved this process for new breakers For existing breakers no solution.. But..
We can track it because of known early symptoms
Solutions cntd.
Already tracked by the same automatic analysis software tool but with very few cycles left – we can measure the System Resistance!
Leading to an unscheduled corrective or preventive maintenance intervention
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
3- Main axle fixation tracking improvement - Measurement
Solutions cntd.
Needs to achieve Improvement of existing Voltage Measurement Channel
Replacement of Voltage Divider by low impedance divider and noise reduction components (270 pcs MPE-EM)
Replacement of Measurement Board by low noise separation amplifier (270 pcs: MPE-EM)
Replacement of DQAMS by 12 bit to 24 bit ADC (MPE-EP & EM: 220 pcs)
Voltage measurement taps to be improvedHi-level SW for on-line analysis (MPE-MS)
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
3- Main axle fixation (and loose bolts) Save the EE system
in case we don’t track (or too late) it with software tools Overcurrent in one or two phases of the breakers will lead to over-
temperature on the Equalizing Resistor and eventually destroy all breakers in one system
Installation of a Thermostat on each Equalizing Resistor will trip the system early before hardware damage is done
A spare internal interlock channel is available and wired up to the power board.
Solutions cntd.
Needs to achieve Installation of Thermostats
Reprogramming of Firmware in all 404 interface cards (MPE-EP)Allow trip on over temperature of Equalizing Resistor
»Profit to limit cycles of the breakers
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
LS1 Actions
Breakers - Regular maintenance and Consolidation
Equalizing Resistors – Installation/wiring of Thermostats
Voltage measurement-channel improvement
Visual inspection of main axles – Endoscope
IST SCT HWC
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Resources
Maintenance IST Installation man-hours T/TE man-hours E/TE
per point per pointEven point - 2 UAs 100 30Odd point - 2 RRs 40 10P7 right 30 10P7 left 30 10UJ33 65 20
Total LHC approx. : 630 h 160 h
130 man-days6 man-months
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Planning (not incl. SCT and HWC)
ANYTIME – “ANYWHERE” April 2013 – April 2014 – 6 man-months –
Team of 3 persons could come 2 x 1 month
Man-power Calculations based on one team of Specialists from BINP
Perfect preparation before start of activityOne Engineer or TE as team leader plus two TechniciansWorking in a “train”
Start – to – Finish one point, including IST
But ..
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Planning SCT & HWC
SCT & HWC – 2014 SCT was presented as not needed for 600 A EE but …
latest tests show.. Yes, if we could we should 1 day per UA or RR – 2 TE or E
15 working-days - NO recourses allocated yet
HWC like SOF test PLI3.b1 – remotely from CCC – Automatic Analysis of PM
PNO.b1 – 1 day per UA or RR – 2 TE or E 15 working-days - NO recourses allocated yet
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
“Showstoppers” Preparation for LS1
Planning – Tests – Tooling – Purchasing (thermostat-study still to be finished)
A LONG WAY TO GO!!!
No reference contract with Russia If done by non-experts
More CERN staff needed for training, the actual job, follow up, debugging of collateral damage etc. Might delay re-start after LS1.
Spare Circuit Breakers: DZNVA – Visit in March 2012 - Manufacturer’s premises in very bad state and
only 20% occupied - might stop delivering (special) breakers Currently: 28 spares (13 operational – 15 to be qualified before TS4) Consumption average: 8-10 per year (including preventive changes)
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Non LHC activities during LS1 Currently low in spare systems (<1%) and
spare breakers (3-4%) In order: 20 spare systems – ATA Autumn 2013 – to be
mounted and tested in ISR test building Urgent: to order 120-150 spare breakers in order to
exclude the regular need of Russian Specialists and cover mid-term needs.
Long term strategy for 600 A EE systems: Switch partially (UA areas) to solid state breakers
R&D necessary: 2012 development of Lab version (Bi-polar)2013/2014 development of prototype – 0.5 day/week2015 Market Survey/Tender
What about the study to prevent Quench-Back on RQTF/Dthe installation of lower dump resistances in 600A circuits
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Thank you for your attention.. So far..!
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Questions before continuing with Cable
Activities?
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Planning activities GJC Preparation 600 A EE activities to start LS1:
70% Installation/Maintenance – 2013
Coordination – during the activity 50% IST – 2013
Coordination – during the activity 50% SCT – 2014
Coordination/installation/Analysis: 70% during activity HWC – 2014
PLI3.b1 – remotely from CCC – Automatic Analysis of PMActivities if automatic analysis fails
PNO.b1 – 1 day per UA or RR – 2 TE or ECoordination/installation/Analysis: 70% during activity
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
1- Loosening of holding coil mechanism Weak design of U-shape and dimensional tolerances
Vibrations causing the closing failures – if starting increasing with cycles To prevent vibrations the manufacturer proposed decreasing of gaps on the side of
the U-Shape by gluing small plates between the U-shape and the housing Several cycling tests have proven the efficiency of this method. Number of closing
failures reduces from typically 4% to less than 0.5 % during lifetime of breaker
Solutions
Needs to achieve Plate Glue Action in all 606 breakers
Open all breakers (anyway foreseen for regular maintenance) Plate gluing – Once open: 3-4 minutes per breaker Glue = Silicon paste (Dow Corning Mil. Standard) approved by breaker
manufacturer
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
Solutions
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
3- Main axle fixation tracking improvement Weakness during production
If this occurs there’s an increase from 0.5 mΩ to max 1.25 m Ω of system impedance due to decrease of contact pressure in the breaker.
By improving the existing voltage (drop) measurement, correlating it to the circuit current, can lead to an early warning, preventing the circuit to trip
An intervention can be planned in the shadow
Solutions cntd.
Needs to achieve Improvement of existing Voltage Measurement Channel
Replacement of Voltage Divider by low impedance divider and noise reduction components (MPE-EM)
Replacement of Measurement Board by low noise separation amplifier (MPE-EM)
Replacement of DQAMS by 12 bit to 24 bit ADC (MPE-EP & EM) Voltage measurement taps to be improved Hi-level SW for on-line analysis (MPE-MS)
TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE
3- Main axle fixation (and loose bolts) Overcurrent in one or two phases of the breakers will lead to over-
temperature on the Equalizing Resistor and destructing all 3 breakers in one system
Installation of a Thermostat on each Equalizing Resistor will trip the system early before hardware damage is done
A spare internal interlock channel is available and wired up to the power board.
Solutions cntd.
Needs to achieve Installation of Thermostats and Wiring up to the spare channel
Dismounting, drilling, threading of all 606 Equalizing Resistors Mounting of pre-cabled Thermostats and a connector PCB Wire installation from connector PCB to Burndy connector SK24 Reprogramming of Firmware in all 404 interface cards (MPE-EP)