machine protection update rüdiger schmidt mac december 2006

1MAC December 2006

• Audit of the Beam Interlock System

• Prototyping interlock systems at CNGS: LHC type Beam Interlock System and Fast Magnet Current change Monitors

• Software and machine protection – Software Interlocks– Sequencing– Post-Mortem – Critical Settings Management

• Compromise between the built-in flexibility of the MP system for the sake of optimum machine performance and efficient commissioning on one hand and the robust and the fail-proof design

Machine protection update Rüdiger Schmidt MAC December 2006

2MAC December 2006

• Safe operation of the LHC in presence of the energy stored in each beam of up to 362 MJ requires as a key system the Beam Interlock System

• Failures are detected by beam loss monitors and other beam instruments, by the quench protection system and other hardware related equipment. Beam dump requests are transmitted via the Beam Interlock System to the beam dumping system.

• There has been a review on the Machine Protection System. One recommendation was to organise more detailed reviews of the most critical systems (Beam Interlock System and Beam Dumping System).

The architecture of the LHC machine protection is recalled. The functionality of the Beam Interlock System with respect to machine protection is presented.

Main emphasis is on the safety critical aspects of the Beam Interlock System.

This event is an audit of the Beam Interlock System. This is the last option for (limited) design changes, before starting series production of the electronics.

Introduction to the Audit of the Beam Interlock System – September 2006

3MAC December 2006

• Experts: Matthias Werner (DESY); Javier Serrano (AB-CO); Yves Thurel (AB-PO); Philippe Farthouat (PH); Reiner Denz (AT); Stefan Lueders (IT)

• Members of the interlock team: Bruno Puccio; Benjamin Todd; Rudiger Schmidt; Philippe Nouchi; Markus Zerlauth; Christophe Martin; Arend Dinius; Samir Hamnache

• Collaborators giving other presentations: Jorg Wenninger; Etienne Carlier

• Guests: Bernd Dehning; Jan Uythoven; Rossano Giachino; David Belohrad

• Group Leader: Hermann Schmickler

Participants

4MAC December 2006

Organisation

• Today (18/9/2006)– presentations

– providing the experts with all required material (specifications, layout of the electronics, test results, ...)

– showing the prototypes of the electronics

• Second day 25/9/2006– asking questions

– discussions of findings

– formulating recommendations

– presenting the outcome

• The week between 18/9 and 25/9 could be used to further understand and discuss details of the interlock system, if the experts consider this to be useful.

• We might organise a demonstration of the operating system at the SPS

5MAC December 2006

• Audit NOW since..... – we have experience from CNGS (10-20% of LHC type hardware is operating)

– all electronic boards are in the final design phase

• Not the purpose to propose alternative ideas for interlocking the High Energy CERN accelerators

• The purpose is to identify critical parts, to understand the function of those parts, to validate their design or propose modifications– split the audit in two parts, and leave some time for discussions / investigations in

between

• Possible impact of the audit– many comments might not have an impact on the hardware

– small HW mods: could possibly be still implemented

– larger HW mods: maybe only for LHC, possibly via an upgrade

Some comments....

6MAC December 2006

Programme

09:00 0 Introduction to the Audit Hermann Schmickler 10

09:10 1 Machine Protection and the Beam Interlock Systems Rudiger Schmidt 15

09:30 2 Beam Interlocking of the SPS / CNGS and LHC Bruno Puccio 20

10:00 3 Realisation of the Beam Interlock System Benjamin Todd 30

10:40

11:05 4 Dependability and signal integrity Benjamin Todd 20

11:35 5 Beam Permit Loops and Optical cards Benjamin Todd 20

12:05 6Module with safety matrix: CIBM and variants (CIBG, Master CIBX, CIBV verifier)

Philippe Nouchi 30

12:45

13:45 Meeting among the experts (if required)

14:10 7 Interface to the user: CIBU and CIBD Christophe Martin 20

14:40 8 Testing the Beam Interlock System: CIBT Benjamin Todd 20

15:10 9 Beam Interlock System Hardware Implementation Arend Dinius 20

15:45

16:15 10Interfaces to the Extraction System and Injection Systems

Etienne Carlier 15

16:40 11Outlook and operational aspects of the Beam Interlock System

Rudiger Schmidt + B.Puccio 15

17:10 12 Experience from the 2006 SPS / CNGS run Jorg Wenninger 20

17:50 Executive session of the experts

Lunch

Other aspects

Coffee

Hardware of the Beam Interlock System

Machine protection and Beam Interlock System

Coffee

7MAC December 2006

• Do you consider the requirements for the Beam Interlock System adequately defined?

• Does the proposed realisation of the Beam Interlock System fulfill the requirements?

• Do you see any specific risks with the electronics?

• Do you see any failure modes that should be considered?

• Are the interfaces between the Beam Interlock Systems and the other systems clearly specified?

• Would you expect that the Beam Interlock System allows for safe operation (reliability)?

• Would you expect that the Beam Interlock System allows for efficient operation (availability)?

• Based on experience elsewhere: what is most critical and where have been surprises?

Questions to the experts

8

Beam Interlock System Audit

Report on the Audit held in September 2006

Stefan Lüders (IT/CO) on behalf of the Auditors

Reiner Denz (CERN AT/MEL), Philippe Farthouat (CERN PH/ATLAS), Stefan Lüders (CERN IT/CO), Javier Serrano (CERN

AB/CO), Yves Thurel (CERN AB/PO), Matthias Werner (DESY)

9

ScopeThis audit is supposed to verify design and implementation of the BIS:

fundamental design decisions

PCB schematics and layouts & VHDL programming

mechanics

interfaces to other systems

Particular focus put on safety relevant aspects:

safe and efficient operation of LHC

sufficiently high reliability and availability

single points of failures AND failure modes leading to blind faults

This audit does not cover

system software running on PowerPC

control aspects & methods for remote diagnostics

10

One slide on the “Why”

6 cm

81012 610124101221012

25 cm

0.1 % of the full LHC beam

81012 protons

σx/y = 1.1mm/0.6mm

11

The BIS Architecture

Three ring-type systems:• LHC Beam 1 & Beam 2• SPS

Four tree-type systems:• LHC injection (Beam 1 & 2)• SPS extraction (BA4 & BA6)

User Permit A

User Permit B

Beam Permit Info

Test

Monitor

<1200m

USERINTERFACE

ELECTRONICS

BEAMINTERLOCK

CONTROLLERELECTRONICS

true false

‘DC’ Signals

Encoded Data Frames

User Permit A

User Permit B

Beam Permit Info

USERSYSTEM

ELECTRONICS

<4m

Permit Loop Beam-1 Anti-Clockwise

Permit Loop Beam-1 Clockwise

Permit Loop Beam-2 Clockwise

Permit Loop Beam-2 Anti-Clockwise

To/FromNext/

PreviousBICs

<6000mCurrent Loops RS485 Channels Fibre Optics

1 2 3

CIBU

12

Auditor’s RecommendationsRecommendations have been

distributed to all parties involved.

Focus on major points.

Numbers refer to Audit Report.

13

General ImpressionDesign and implementation of the BIS is

sound,

complete,

straight-forward, and,

conform to requirement on high inherent level ofsafety, reliability and availability (SIL4) .

BIS as such makes a mature and solid impression.

Requirements have been adequately defined.

The present implementation fulfils completely the requirements.

14

DocumentationQuite complete set of documentation on EDMS:• incl. drawings for PCB schematics, PCB layout and VHDL code• Additional documents on Bit-Error-Rate of optical link, resistance under EMC,

detailed FMECA

1. Consistent set of up-to-date and finalized documents should be provided.

One of the main actors finishing his thesis soon:

2. Ensure that all information relevant for the project is properly retrieved.

Started: We have begun to update the existing documentation (EDMS). It will be completed in the next future.

Done: An engineer has been appointed (former PhD student).

15

Optical Loop ISevere denial-of-service due to spurious signals on optical fiber link:

ELED driver amplified power supply ripples on the “high” state after the signal inversion, transmitted by the ELED, and amplified and shaped by the PIN diode receiver circuit.

13. A solution should be found to avoid this behavior.

14.The availability of the “old-fashioned” ELED seems to present a potential problem.

Started: The Optical board is a small daughter board.

We have launched a rigorous re-study of the current Optical board (CIBO). Several solutions are under study.

We have checked with the distributor of ELEDs. These components will stay available for at least five years (confirmation received in writing).

In case that the board would have to be changed later, this would be a minor investment.

16

Optical Loop IITwo optical loops at 8.000 and 8.192 MHz:

4. More separated frequencies like 8.750 and 9.375 MHz should be used.

For hardwired CCC interlock signals, a CIBU is too distant:

15.Consistent and safe solution should be found for mitigation.The auditors prefer an additional BIC in the CCC.

Started: We have decided to change the frequencies of the 2 optical loops. We have received some quartz:37.5 MHz => 9.375 MHz permit loops

35.5 MHz => 8.875 MHz permit loops




The choice will be done from two of the above values.

Started: We intend to install a 17th BIC closed to the CCC.

17

Testing & EnvironmentCareful functional testing is essential:

Electrical tests of all PCBs

17.Power soak test duration should be justified and adjusted

17.Accelerated thermal aging test of one system

18.“Walkie-Talkie”-type or RF susceptibility test

BIS depends highly on proper electrical grounding:

19.Conductivity of unit’s enclosure and earth connection of rack should be tested after installation.

Done: We have designed new testers in order to perform electrical tests on the CIBE cards & on the CIBP cards.

Will be done

Started: We are going to carefully check the grounding and the earth connection of each BIS rack. In addition, we are going to systematically check every cable (with a dedicated cable tester).

18

CIBUs have never been specified to be radiation tolerant:

27. Radiation tolerance should be defined and verified.

27. Persons responsible for BIS users must be made aware of the situation concerning radiation tolerance.

Started: We are preparing radiation tests for the CIBU unit (foreseen in January 2007 at PSI).

Relevant numbers and threshold will be communicated after the radiation tests.

19

Components, Xilinx & VHDLRecommendations have been made on choice of components:

5. Extra power filters for the 230V mains

9. Choice of ceramic capacitors

10.Choice of bi-directional transil suppressor

Xilinx chip will block once the external clock is missing:

21.Failure modes and corresponding mitigations should be checked.

Power filters: better understand motivation from DESY colleagues for having extra filters

Done: We have modified the part lists containing large values capacitors.

Done: We have checked with the outside company that Transil from Fairchild won’t be mounted on our cards.

Not yet started, but planned

20

A number of questions came up during reviewing the VHDL code:

VHDL code review should be conducted when the final CIBM design is ready.

23.Storage of VHDL code inside a software repository (e.g. CVS)

Critical parts of the code will be written by two different programmers

A full review of the VHDL code will be done (experts available inside AB-CO)

Is being done

21

Injection Interlocks

Beam Interlock System

Powering Interlock

Systems

QPS(several 1000)

PCs

(800)

AUG

UPS

VacuumSystem

Vacuumvalves

BLMcontrollers

Access System

4 BPMs

AccessSafetyBlocks

RF Stoppers

Doors BeamStoppers

Beam Dumping System

FMCM Current

24

CryoOK

Beam Dumping System

Timing System (Post Mortem)

BLMs(several 1000)

Safe BeamFlag

Magnets

SoftwareInterlocks

LHCDevices

Interfaces ISubstantial amount of effort has been put in high reliability and

availability of the BIS:

Dependence on quite a long list of user systems and on proper functioning of the LDBS

LHC safety is only as good as its weakest element:

35.Clear procedures for testing the full BEAM_PERMIT-signal chain should be defined. Started: We are preparing documents describing the

BIS commissioning and the role of the User systems in the test of Beam_Permit chain.

We started to address full automatic testing of the beam interlock system WITH the user systems by a software sequencer….. but not as highest priority for LHC initial operation.

22

Interfaces II (Users)LBDS kicker magnet system is essential for LHC safety:

29.Similar audit should be conducted for LBDS kicker magnets and their trigger mechanism.

LHC safety relies largely on a small set of monitoring and control systems, e.g. Beam Lifetime Monitors, BLM, FMCM, Powering Interlock, Transverse Feedback System:

31.Dependencies analysis with regard to LHC safety and of an audit of the major dependencies should be conducted.

32.Procedures are mandatory to guarantee that BIS user systems obey standard safety rules.

32.Awareness discussions & training

Other systems are thinking about organising audits… not the responsibility of the interlock team

(at least partial audits have been or are planned)

Being done – long term activity

23

Interfaces II (Users)33.“Walkie-Talkie”-type or RF susceptibility test on (critical) BIS users

34.Can SOFTWARE_PERMIT sustain high level of reliability and availability of the overall BIS ? Alternatives should be evaluated.

Started: We intend to disable the so-called “Software input”. The Software Interlock System will be connected to the BIS via the standard way (i.e. the CIBU).

Will be done together with the user systems

24

Safe Beam ModeSAFE_BEAM_FLAG allows masking half

of the user inputs to the BIC:

36.No protection mechanism to prevent the exchange of the cables

Done: Before the re-initialization of the Beam_Permit loop, we check “connected CIBUs” vs. “Configuration DB” => Swapped cables will be detected.

25

The SAFE_BEAM_FLAG is distributed by the SMP / GMT:

Safe solutions for the implementation of the SLBR board should be investigated.

No documentation for the implementation and the SLBR.

38. The distribution of the SAFE_BEAM_FLAG should be consistent with reliability / availability / safety of BIS.

Fully agreed – the project is done in collaboration with another section.

The distribution of the SafeLHCParameters is being developed. A Functional specification has been written.

A local watch-dog will be implemented on the SLBR board. In case of missing data, the Safe_Beam_Flag will be forced to FALSE.

In addition, a software monitoring could be implemented for checking this (slow) process.

26

SummaryDesign and implementation of the BIS is sound, complete, straight-

forward, and, conform to requirement on high inherent level ofsafety, reliability and availability (SIL4) .

However:

To keep reliability high, functional testing on regular basis is vital.

Worried about the behavior of the optical link electronics(esp. the ELED and ELED driver circuit).

VHDL code should be reviewed separately.

Further electrical and RF susceptibility tests should be conducted.

Concern about the safety/reliability/availability of BLM and kicker system of the LBDS (separate systematic audit / review should be conducted on them).

Finalize documentation.

27MAC December 2006

Some comments – after the audit

• Not everyone was convinced that such an audit at this stage makes sense.

• All members of the interlock team fully supported it.

• The time that we (..the interlock team) invested in the preparation was a few days – not more.

• We were pretty sure that we proposed a solid system, and therefore we did not expect findings that would create great difficulties, however, if the experts would have found a major problem, we rather like to know now.

• I see many reasons for having an audit, and the (only?) reason of not having it is lack of time for the preparation (I would have been nervous without the audit - I cannot judge all detailed aspects of the system).

• The findings at the evening of the first day would have been moderately useful….. but due to the intense work of the experts (many many hours between the first day and the second day a week later), I consider the audit to be very useful for the project.

I like to thank the experts for their help

28MAC December 2006

• The beam energy extracted from SPS could damage equipment, stringent machine protection is required

• Beam interlock system, about 10-20% of LHC system, worked successfully

• Fast Magnet Current change Monitors: few (HERA) systems installed– measuring small current change of magnets within less than 1 ms– worked very well, performance better than required– no false beam dumps– upgraded system: prototype unit currently tested at DESY (M.Werner)

• Software Interlock System became operational during CNGS run

Experience from the CNGS run

MAC December 2006

FMCM

• 3 FMCMs are installed on MSE.418, MBSG.4000 and MBG dipole string.

• Tested using steep reference changes to trigger FMCM. The trigger threshold is measured using BIC history buffer + read out of current (1 ms step).

Reference

PC current

FMCM trigger 0.1% drop !

SC time (ms)

SC time (ms)

I (A)

I (A)

J.Wenninger, MPWG talk

MAC December 2006

FMCM beam testsThe tests were repeated with beam by adjusting the step and FMCM trigger to occur ~ nominal

extraction time record max. trajectory excursion – confirms current measurements.

PC FMCM trigger ROCS tol.

MSE.418 < 0.1% 0.1%

MBSG.4000 < 2.5×10-4 0.1%

MBG/MBI.816 < 2.5×10-4 0.1%

Example :

• Max. trajectory excursion due to MSE @ time of FMCM trigger.

• Trajectory interlock tolerances :

• ±4 mm BPMs no. 1 to 20.

• ±0.5 mm last 3 BPMs.

In all cases the position change @ target is < 0.5 mm

(as specified).

Triggers a BPM interlock


MAC December 2006 32

Summary

• The machine protection system for CNGS with 4 BICs and 32 clients was commissioned in ~ 50-60 hours. This time only includes the specified tests (mostly from CCC). It does not include the hardware tests to connect the clients…

• All documentations and information for operation is available on a WEB page.

• No major problems were encountered during the commissioning and the transition to high intensity operation was smooth.

• Almost all interlock systems performed very well, with special mention for the powering interlocks (current surveillance & FMCM) that were quite outstanding !

• The only problem was due to the beam position interlock in the SPS ring – this became a major issue due to the simultaneous out-gassing problems of the SPS beam dump.

I think we must consider installing a more robust (no gains) and faster system, even at the price of a reduced accuracy.

• The SafeBeamFlag was introduced for the first time with success.

• The next challenge will be operation with CNGS cycles in a super-cycle in October. This is going to be an interesting problem of settings management !

If the beam dump continues to outgas, operation may be VERYVERY delicate!


33MAC December 2006

• Post Mortem – Required for commissioning of electrical circuits in March 2007– Beam related post mortem being addressed - workshop is being organised in January 2007: http://

indico.cern.ch/conferenceDisplay.py?confId=9340

• Sequencer– Sequencer for hardware commissioning operational– Sequencer for beam commissioning in progress– Possible merge of the both sequencers

• Management of Critical settings: good progress, should become operational during first part 2007

• Role based access project started (restricting access to some applications)

• Beam instruments – no news on Fast Beam Current change Monitor, but not required for 2007

• MP Commissioning: Working Group started, reporting to LHCCWG (summary report possibly for next MAC)

News on other issues

http://indico.cern.ch/conferenceDisplay.py?confId=9340



34MAC December 2006

• The beam interlock system receives (nearly) all inputs for permitting beam operation

• Disabling on the level of the beam interlock system is in principle possible, but requires NO BEAM and ACCESS to the hardware– disabling a channel is a heavy procedure

• Some input channels are MASKABLE - only during low intensity beam– masking is a light procedure, but requires non-damaging beam in LHC

– very successfully used for CNGS

• Many input channels will be commissioned during “hardware commissioning” and “cold checkout”, to be operational for first beam

• Work ongoing: – one accelerator physicist (PhD) studying failure scenarios

– one reliability engineerr studying consequences for interlock systems

The following slides are for illustration of the flexibility, and NOT to define the final phases for the commissioning of the interlocks

Flexibility of Machine Protection during commissioning

Beam Interlock Tree maskable and unmaskable inputs


PIC essential

32

WIC

14

QPS(several 1000)

Power Converters

730

AUG

UPS

Power Converters

BLM aperture limits

16

PIC essential + auxiliary

32

VacuumSystem

34

Vacuumvalves

BLM in arcs

16

Access System

2

BPM in IR6

2

AccessSafetyBlocks

RF Stoppers

QPS(several 1000)

Power Converters

864

AUG

UPS

Magnets Doors EIS

Beam Dumping System

2

FMCM Current

24

CryoOK

CryoOK

Beam Dumping System

2


1 or 2


2

BLMs(some 100)

BLMs(several 1000)

Safe BeamFlag


CCC Buttons

2

Transverse Feedback

2

Movable Devices

Beam Aperture Kickers

2

SLP

2

Experiments

26

BCMBeamLoss

Experimental Magnets

RFSystem

4

FBCMLifetime

2

CollimationSystem

42

CollimatorPositions

Environmentalparameters

BTV

9

BTV screens

Mirrors

SoftwareInterlocks

2

LHCDevices

to be defined

preliminary

Beam Interlock TreePhase I after Hardware Commissioning


PIC essential

32

WIC

14

QPS(several 1000)

Power Converters

730

AUG

UPS

Power Converters

BLM aperture limits

16


32

VacuumSystem

34

Vacuumvalves

BLM in arcs

16

Access System

2

BPM in IR6

2

AccessSafetyBlocks

RF Stoppers

QPS(several 1000)

Power Converters

864

AUG

UPS

Magnets Doors EIS

Beam Dumping System

2

FMCM Current

24

CryoOK

CryoOK

Beam Dumping System

2


1 or 2


2

BLMs(some 100)

BLMs(several 1000)

Safe BeamFlag


CCC Buttons

2

Transverse Feedback

2

Movable Devices


2

SLP

2

Experiments

26

BCMBeamLoss


RFSystem

4

FBCMLifetime

2

CollimationSystem

42

CollimatorPositions


BTV

9

BTV screens

Mirrors

SoftwareInterlocks

2

LHCDevices

to be defined

preliminary

Beam Interlock TreePhase II aftercold checkout


PIC essential

32

WIC

14

QPS(several 1000)

Power Converters

730

AUG

UPS

Power Converters

BLM aperture limits

16


32

VacuumSystem

34

Vacuumvalves

BLM in arcs

16

Access System

2

BPM in IR6

2

AccessSafetyBlocks

RF Stoppers

QPS(several 1000)

Power Converters

864

AUG

UPS

Magnets Doors EIS

Beam Dumping System

2

FMCM Current

24

CryoOK

CryoOK

Beam Dumping System

2


1 or 2


2

BLMs(some 100)

BLMs(several 1000)

Safe BeamFlag


CCC Buttons

2

Transverse Feedback

2

Movable Devices


2

SLP

2

Experiments

26

BCMBeamLoss


RFSystem

4

FBCMLifetime

2

CollimationSystem

42

CollimatorPositions


BTV

9

BTV screens

Mirrors

SoftwareInterlocks

2

LHCDevices

to be defined

preliminary

Beam Interlock TreePhase III during very first beam


PIC essential

32

WIC

14

QPS(several 1000)

Power Converters

730

AUG

UPS

Power Converters

BLM aperture limits

16


32

VacuumSystem

34

Vacuumvalves

BLM in arcs

16

Access System

2

BPM in IR6

2

AccessSafetyBlocks

RF Stoppers

QPS(several 1000)

Power Converters

864

AUG

UPS

Magnets Doors EIS

Beam Dumping System

2

FMCM Current

24

CryoOK

CryoOK

Beam Dumping System

2


1 or 2


2

BLMs(some 100)

BLMs(several 1000)

Safe BeamFlag


CCC Buttons

2

Transverse Feedback

2

Movable Devices


2

SLP

2

Experiments

26

BCMBeamLoss


RFSystem

4

FBCMLifetime

2

CollimationSystem

42

CollimatorPositions


BTV

9

BTV screens

Mirrors

SoftwareInterlocks

2

LHCDevices

to be defined

preliminary

Beam Interlock TreePhase IV during early beam commissioning


PIC essential

32

WIC

14

QPS(several 1000)

Power Converters

730

AUG

UPS

Power Converters

BLM aperture limits

16


32

VacuumSystem

34

Vacuumvalves

BLM in arcs

16

Access System

2

BPM in IR6

2

AccessSafetyBlocks

RF Stoppers

QPS(several 1000)

Power Converters

864

AUG

UPS

Magnets Doors EIS

Beam Dumping System

2

FMCM Current

24

CryoOK

CryoOK

Beam Dumping System

2


1 or 2


2

BLMs(some 100)

BLMs(several 1000)

Safe BeamFlag


CCC Buttons

2

Transverse Feedback

2

Movable Devices


2

SLP

2

Experiments

26

BCMBeamLoss


RFSystem

4

FBCMLifetime

2

CollimationSystem

42

CollimatorPositions


BTV

9

BTV screens

Mirrors

SoftwareInterlocks

2

LHCDevices

to be defined

preliminary

MAC December 2006 40

machine protection update rüdiger schmidt mac december 2006

Documents

optical link

optical fiber link

current optical board

optical loop isevere

stefan lders cern itco

lhc injection beam

audit report

bis issound