summary of operational experience powering interlocks vs beam losses dependability

Powering Interlocks (PIC, WIC) + FMCMMPS Internal ReviewM. Zerlauth 17-18th June 2010

CERN

Markus ZERLAUTH (TE/MPE) LHC Machine Protection Systems - Internal review - 17-18th June 2010 2

Summary of operational experience Powering Interlocks vs Beam Losses Dependability FMCM vs mains disturbances R2E Automated testing of interlock systems Miscellaneous Conclusions

Outline

CERN


Powering Interlocks vs Beam Interlock

Beam Interlock System

PIC essential+ auxiliarycircuits

WIC

QPS(several 1000)

Power Converters~1500

AUG

UPS

Power Converters

BLMVacuumSystem

VacuumValves(~300)

Access System

BPM in IR6

AccessSafetyBlocks

RF /e-Stoppers

Magnets Doors EIS

Beam Dumping System

FMCM

CryoOK

Injection BIS

Monitorsaperture limits(some 100)

Monitors in arcs(several 1000)

CCC Operator Buttons

Transverse Feedback

Movable Devices

Beam Aperture Kickers

SMP Experiments

BCMBeam Loss

Experimental Magnets

RFSystem

FBCMLifetime

CollimationSystem

CollimatorPositions

Environmentalparameters

BTV

BTV screens

SoftwareInterlocks

LHCDevices

Timing System (PM)

SEQ viaGMT

LHCDevices

LHCDevices

Tim

ing

SafeBeamFlag

MKI

Mirrors

32 8 12

1/4 of LHC BIS user connections for powering interlocks , collecting a large inventory of interlock channels

CERN


A bit of statistics…FMCM (RMSD)

1%ATLAS1%FMCM (RD1, RQ5, RBXWTV)

2%FMCM (RD34)

1%

Operator Buttons4%

Programmed Dumps5%

Beam Excursion Redund5%

SIS5%

MKI2/MKI86%

BLM UNM9%

BLM MSK23%

LBDS 24%

PIC MSK 4%

Beam Excurison4%

PIC UNM3%

COLL_MOT3%

Activation of BIS channels with closed BPL (~850 in total)

CERN


Dump statistics and origins for 2010 run…

● This year total of 515 ‚beam dumps‘ (breaking of BPL) ● 55 dumps by PIC

– 23 AT INJECTION, 6 DURING RAMP, 16 AT FLAT TOP

– 15 tune/orbit feedback vs QPS/PC– 12 individual circuit trips (RB.A78 VS, RQX.R2, 600A correctors due

to HW failure in PC or QPS)– 11 from CRYO– 10 sector trips (QPS, nQPS)– 4 due to U_RES of Undulator– 3 electrical perturbations

● 9 dumps by FMCM after mains perturbations (see later)● 1 dump by WIC in IR4 (problem with gas monitoring circuit)

CERN


Powering Interlocks vs Beam Losses…

● For current intensities and setup redundancy to BLM system has worked very well (neglecting losses around IR6 due to abort gap population), ie

● PIC: Apart from event in S12 (RQD.A12 not at injection), all powering events caught by PIC before any losses/orbit movement occur (including SPA and FPA in a complete sector)

– Side note: Did not see a FPA in two adjacent sectors yet (e.g. AUG/UPS)

● Warm magnet interlocks: Only 1 real dump provoked @ inj due to gas monitoring circuits in IR4 being switched OFF

● FMCMs: No considerable losses (ie no BLM triggers) or orbit movements for any of the MPS tests or (frequent) network perturbations

CERN


Dependability

● Very good experience so far, industrial as well as ‘home-made‘ electronics are exceeding reliability predictions *

● No machine downtime from powering interlock systems due to component failures

● In more than 3 years of commissioning/operation of ~70 systems, only 3 (transparent) interventions on redundant power supplies

*Not taking into accound possible radiation effects (see later)

CERN


Dependability vs Interventions/Maintenance

● Current good performance is based on a very thorough hardware commissioning campaign, where all protection related features of installed HW have been tested and validated for operation

● During technical stops, interventions, etc... we exchange, upgrade, fix protection related equipment without requalifying the equipment (after exchange of power modules, QPS cards, etc..)

● Currently (for me), no clear tracability of changes to protection related systems or clear guidelines/documentation for revalidation of equipment?– Point to follow up by MP3 for powering system….

CERN


FMCMs and mains disturbances● Beams dumped upon 9 occasions by FMCM following network perturbations, ie

11-JUN-10 02.41.38 AM (400kV – all LHC) at 450GeV 26-MAY-10 09:47:54 PM (400kV – all LHC due to thunderstorm) at 3.5 TeV18-MAY-10 05.35.42 AM (400kV – all LHC due to thunderstorm) at 3.5 TeV10-MAY-10 22.48.44 PM (400kV – all LHC) at 450GeV 02-MAY-10 02.59.37.127000 AM (400kV - all LHC) at 450GeV01-MAY-10 05.50.32.127000 AM (18kV ring line, seen on dump septas) at 450GeV19-APR-10 05.14.30.396000 AM (400kV - all LHC) at 3.5 TeV07-APR-10 06.46.58.724000 AM (18kV ring line, seen on dump septas) at 450GeV03-APR-10 07.24.04.890000 AM (400kV - all LHC) at 3.5 TeV

● All trips happened at flat top (either injection or 3.5TeV) and did not result in self-trips of power converters, apart the one on 2nd of May which tripped both RD1s, RD34s and the ALICE and LHCb dipoles (+ LHC Coll and RF equipment)

● All dumps correct, as current changes exceeded specified values (decreasing thresholds will not help)

● Mains perturbations seen in all circuits, but current intensities and setup do not yet induce considerable beam movements or losses, will look different later (and if happens e.g. during ramping)

CERN


03-APR-10 07.24.04.890000 AM – Fault in 400kV S phase -40kV , 30ms

Courtesy of D.ArnoultTypical perturbation originating in 400kV (2 phases, V dip of ~15% for some 60ms)

CERN


Comparison to spec of “minimum immunity of equipment”

Based on the statistic of past network disturbances, a minimum immunity for equipment has been defined in an LHC ES ‘Main Parameters of LHC 400/230V Distribution System’

U nom= 400 / 230V ± 10 %

Typ. Variations = ± 5 %

Transients= 1200V for 0.2ms

THD = 5%

Voltage Swells = +50%, 10ms

Voltage Dips = -50%, 100ms

19-APR-10 05.14.30.396000 AM

X

07-APR-10 06.46.58.724000 AM

03-APR-10 07.24.04.890000 AM

X

01-MAY-10 05.50.32.127000 AM

XX

CERN


02-MAY-10 02.59.37.127000 AM – RBXWTV.L2 + others

∆I = 2A∆I /I = 3.4 •10E-3

∆V = 1.5V∆V /V = 5 •10E-2

FMCM:Measured excursion > 1.5Threshold : 0.6

CERN


19-APR-10 05.14.30.396000 AM – Trip of RD1s in IR1 an IR5

∆I = 0.7A∆I /I = 2 •10E-3

∆V = 20V∆V /V = 8 •10E-2

FMCM:Measured excursion > 8Threshold : 0.4

CERN


19-APR-10 05.14.30.396000 AM – Perturbation on RB.A12

∆I = 0.018A∆I /I = 3 •10E-6

∆V = 4V∆V /V = 8 •10E-1

CERN


Radiation

● Radiation might become an issue for industrial components of the PIC and WIC (PLCs are known to be sensitive)

● New R2E studies require relocation of some PLCs– UJ56, UJ14, UJ16: Relocation of interlock equipment already prepared in

2009, might have to change depending on decision for QPS/PC equipment– US85: WIC to be relocated to UA83 (in progress , before end 2010)– TI8: WIC to be relocated upstream of collimator (in progress, before end

2010)

● In-house electronics has been shown to be adaequat for expected radiation levels (e.g. in RRs). Dedicated CNGS rad test for XC95144 (will start investigation of rad tolerant version)

CERN


Automated tests for links to BIS – 1/2

SIEMENS 319 CPUMax 16 Inputs / Patch PanelMax 96 Inputs / Total

PROFIBUS

MATRIX

ESSENTIAL CIRCUITS(some 720)

ESSENTIAL + AUXILIARYCIRCUITS(some 864)

=UNMASKABLE BEAM

DUMP REQUEST

=MASKABLE BEAM DUMP REQUEST

● Whether a sc circuit failure will trigger a (maskable/unmaskable) beam dump request is defined by configuration data

● Redundant, independent paths trough PLC and CPLD/Boolean Processor

CERN


● Both (redundant) paths are now activated in PIC and WIC● As final MPS test of powering interlock system, current

configuration has been validated with automated test sequence– Unmaskable & maskable BIS input: RB, RQD, RQF, RQX, RD1-4, RQ4-RQ10,

all nc magnets– maskable BIS input: RCS, RQT%, RSD%, RSF%, RCBXH/V and RCB% (except

RCBCHS5.L8B1, RCBXH3.L5 and RCBYV5.L4B2 which all have NCs and are locked)

– no impact on the beam: RCD, RCO, ROD, ROF, RQS, RSS and 60A DOC

● Automated test sequences available for all powering interlock systems, should be performed on regular basis (and upon changes of config) to maintain dependability of systems– TODO: Discuss integration in sequencer

Automated tests for links to BIS – 2/2

CERN


Miscellaneous 1

● Since this week last SW interlock in PIC has been unmasked (CRC verification of configuration data)

● Post Operational Checks are implemented in Post Mortem for PIC and FMCM, please let us know if they should fail!

CERN


Miscellaneous 2

● Sector Access interlock to allow for short expert interventions while leaving circuits powered < 1kA has been implemented and documented for 2010/11 run

● As recommended in Chamonix, more reliable implementation is currently being studied with GS/ASE (unlikely before long shut-down)

PVSS PIC

PIC PLC1

PIC PLC2

PIC PLC36

Private Installation Network (Optical Fibre)

Global Interlock PLC

Gateway PLC

Operation & Maintenance Desk CERN External Systems

Access Mode Selection

Access/Beam Selection

C om m a n de p ou r re ti rer un V ET O +co nf ir ma t ion qu e le V E T O e st re tir é

Po s itio n s û re d e t o us l es E IS (no n a li me n té , fer m é e tc .)

Co m m a nd e p o ur e nv o yer un V E T O

+ co nf ir m at ion qu e le V ET O e st m is

M ODE A uc u n V E T O su r le gro u pe d’E I S c o rre s po n da n t

Po s itio n n o n s û re d ’a u m o ins un EI S (a lim e n té , ou v ert , etc . )

C lig n ote m e n t s ig na la nt un in ci d en t ou a no m a lie s u r le s E I Sco rr e sp o nda nts (ex intr u si on )

OF F

O N

EI S -f c irc ul an t + T ED ba s de TI 2/ TI8+ (( S EP T )

EIS -m -R F

O NO F F

R F

E IS- a T

O N OF F

AC C E ST UN NE L

POINT 1

A T LA S

E IS -a T

O N O FF

AC CE ST U NNE L

E IS -a S

O N O F F

AC CE SS ER VI C E

E IS- a Ex

O NO FF

AC CE SE XP ER I EN C E

EIS - aT

O N O FF

AC CE STU NN EL

E I S-a S

ON O F F

A C CE SSE R V I CE

E IS- a T R F

O N O F F

A C C ES RF

E IS -a T

O N O F F

A C C E ST U NNE L

AL I C E

E I S-a E x

O N OF F

A C C E SEX P E R IE NC E

P OINT 2

E IS -a S

O N O F F

A C C E SS E R V IC E

EIS - aT

O N OF F

A C C E STU NNE L

CMS

E IS -a E x

O N O F F

A C C ESEX P E R IE NC E

EIS - aS

O N O F F

A C C E SSE R V I CE

EI S -aT

O N O F F

AC C E STU NN EL

EIS -aS

O N O F F

AC CE SSE RV I CE

E IS -a T

O N O F F

A C C E ST U NNE L

EIS -aT

O N O FF

AC C E STU NN EL

L HC - B

EIS -aE x

O N O F F

AC C E SE X P ER I EN C E

EIS -aS

O N O F F

AC CE SSE RV I CE

LH C

O K O K

E IS -a

O N O F F

A C C E S

BIWD ela yRA D .De la y

EI S -m -E L

ONO FF

E le c tr o n S to .

TI8E IS -f

O F F

FA I SC EA U

O N

E IS -f

O NOF F

F AI S CE AU

TI2

E I S-f

ONO F F

F AI S C E AU

RF T E ST

Y E S N O

LOC A L

A T L A S

Y E S N O

DE L EG AT I ON

AL IC E

Y E S N O

DE L E G A T ION

CM S

Y E SN O

DE LE G A T IO N

LHC -B

Y E SN O

DE L E GA TI ON

POINT 3 POINT 4 P OINT 5 P OINT 6 P OINT 7 P OINT 8

AT LA S

CERN Control Centre

Private Terminal Network

Technical Infrastructure

Monitoring

Server 2

Server 1

Operator Console

PLC 8PLC 7PLC 6PLC 5PLC 4PLC 3PLC 2PLC 1PLC 1.8

Siemens

Station

Pa tr olGen eral Clos ed

Mo de select io n

CANCEL VALI DAT E

PM 25

PM 45

PM 65

PM 85

Po int 1

Po int 2

Po int 3

Po int 8

ULX1 5

Z one s er vice Zo ne Tu nne l Z on e exp er im ental e

Po int 4

Po int 5

Po int 6

Po int 7

US C5 5

PX24

UPX 16

PZ45

UX85

Select ALL

PM 5 6

UP23

PZ 33

UJ47

UP 55

UP63

UJ27

UX 46

UL 5 5

UJ67

UJ87

UJ 23

UJ 43

UPX5 6

UJ 63

PM 7 6

UJ 83

PM 15 PM 1 8

P P

P

UJ 14 UJ 1 6

Restr ic te d

LACS

NTP

CERN Control Centre Back-office

Access-Powering

PLC

Technical Network

Technical Network

CERN


Conclusions● So far very good experience with powering interlock systems

– Dependable and fast– Providing required redundancy to BLM system

● All MPS checks completed and systems fully operational (all redundancy, no masks,…)

● No issues to further increase intensity (few FMCM tests to be redone with > intensities, BLM red to be watched)

● R2E developments are being followed up but not a (major) concern

● Will have to implement more rigorous approach for IPOCs and automated test sequences

● Need to define (with client systems + MP3) clear maintenance/ intervention procedures and eventually define tests needed for revalidation

CERN


summary of operational experience powering interlocks vs beam losses dependability

Documents

beam dumps

blm system

wic fmcmmps internal

a78 vs

powering events

access system bpm

augupswarm magnet interlocks

dump statistics