1

LHC Timing

Requirements and implementation

J.Lewis for AB/CO/HT 21st/Sep/06

TC presentation

2

Initial observations on LHC timing

Basic-Periods don’t mean a lot Telegrams don’t mean a lot Cycle means even less No super-cycles Little or no PPM/Multiplexing The LHC timing is machine safety critical, so it must be very

simple/reliable and hardware monitored Response time to operational requests must be very rapid

<<100ms UTC is important LHC timing directly controlled by the High Level LSA sequencer

[LSEQ] in real time! It’s a Collider

3

Basic LIC & LHC Requirements

Respond to commands from the High Level LSA Sequencer [LSEQ] for LHC events in < 100ms

Provide an accurate UTC time reference Pilot the LHC Injector Chain [LIC] to fill the LHC Produce the LHC timing from external events and

tables loaded by LSEQ Distribute the safe beam parameters and flags very

reliably Minimize impact on the existing controls

infrastructure

4

Respond to commands from the High Level LSA Sequencer [LSRQ] Implement a suitable API which…

Selects Pilot/Intermediate/Nominal [PIN] LIC beams Controls LIC to deliver a set of CPS batches [1..4] to a

target RF bucket and Ring with correct characteristics Sends LHC event(s) on request from LSEQ Loads, runs N times, starts, stops and aborts

concurrent asynchronous LHC event tables on LSEQ requests

Distributes repeatedly at 1Hz some machine parameters such as MODE

5

About the LHC Timing, observations

LHC MTG is highly interactive, the intelligence is delegated almost entirely to LSEQ

Runs/Stops/Aborts several independent asynchronous concurrent event tables on LSEQ request

Sends event(s) on LSEQ request Sends event(s) such as “post-mortem” and “injection-

warning” on external hardware trigger Observe that most data on the LHC timing cable

comes from the outside world Looks much closer to the LEP than to other cycling

machines in the LHC Injector Chain [LIC] Almost completely decoupled from LIC

6

Pilot the LIC for LHC Filling

LHC Nominal LHC Nominal

Batch 1 Batch 2 Batch 3 Batch 4 Batch 1 Batch 2 Batch 3 Batch 4CPS

PSB

SPS

LHC Injection plateauxLHC1S 1.2S

EastA

text

Status

Beam

Decide on the nextRing/Bucket

RephaseSPS RF

R1B2 3 Batch NominalR1B1 2 Batch Nominal

Pre-Injection plateaux Ramp Pilots

IsoldParasiticBeamS

EastaParasitic

Beam

MDMD

Will bedriven into

sparedependingon batchnumber

LHC Filling scheme TableFilling scheme: 234 334 334 334

Bucket Order: 123 456 789 10 11 12Beam intensity: Nominal

Order: All Ring 1 then All Ring 2On error: Repeat

Retry R1B1 2Batch

Dead Line

Nothing on thisside of the deadline can dependon the status of

the beam transfer

Go for R1B2 3Batch

7

1Hz repeated information on the LHC GMT cable By information I mean data transmitted both by events and by

the LHC telegram at 1Hz The telegram is a snap shot taken each basic-period, that is

each UTC second Events arrive asynchronously and can be subscribed to The event payload contains relevant information for that event ** nHz - means distribute at n Hertz AT - means distribute at arrival time & at nHz

** Payloads carry information useful for the event. Payloads maybe set to any value by the LSEQ in an event send request.This mechanism is one way to trigger power-converter equipmentgroupings where for example the Start-Ramp would control whichequipment is actually started. This usage is between OP and the equipment specialists

8

Information on the LHC GMT cable

BTC1 & BTC2 Circulating Beam Type (per Ring) – 1HzThese values will change soon after the injection has completed and will be distributed as soon as the results of the injection are known. This value must be fully controlled through the API as it is impossible for the timing system to know whether the injection has been successful or not. It is controlled by LSEQ based on measurements taken in the LHC ring.

The beam type is a 16 bit encoding of…

The Number of circulating bunches. Range 1..2808

The Bunch intensity. Example 4 x 1011

The Beam Current The Bunch Spacing. Values 25ns or 75ns The Particle Type. Values Protons or Ions

9

Information on the LHC GMT cableRF-Injection-Parameters These parameters are distributed from the API

with very low latency, they are controlled by the LSEQ. These parameters must be established in the RF system 450ms before the first CPS batch is extracted towards the SPS. They must also remain stable for at least one second after the SPS extraction towards the SPS has taken place.

All these parameters are RESET to zero one second after the SPS Beam-Out event or whenever the LHC Mode is not “Filling”.

10

Establish RF injection parameters 450ms before first CPS extraction

7/18/2005 1

RF synchronization

PSB

Re-phase withbeam ~50ms

Re-phase withbeam ~20ms

Rephase with beam: The timedifference between the targetbucket FREV and the source

bucket FREV is measured by aTDC. The TDC value then

changes the RF frequency sothat they come in line.

This takes time to complete, anddisturbs the orbit.

RF frequency isnearly fixed. Justjump to the nextbucket. This can

only happen if theSPS is empty

FillingLHC Ring 1

LHC RFSR4

R1 Bucket fromtelegram

SPS

SPS RFBA3

CPS

CPS Sourcebucket FREV

Next SPS targetbucket FREV

Timedifferenceto digitalconvertor

VCO CPS RFRF

Batch 3 Batch 4 Batch 5

Injection Kicker

OutIn

Target BucketFREV

LHC TBF LHC TBFSPS SBF SPS SBF

SB-Frev jumps tothe next LHC TB-Frev

Target bucket revolutionfrequency

Target bucketrevolution frequency

Kicker tail is to long to allowinserting batch 4 between batch

3 and batch 5

+-

Referencefrequency

Batch number to Bucket numberconversion

Bucket = K + h/nh = 35640 RF Buckets

n = 12 BatchesK = 121 Approx (Offset)

11

Information on the LHC GMT cableRF-Parameters

BTNI Next injection Beam Type – “AT”Obviously the next injected beam type is determined by the settings in the injector chain and by nothing else. The LSEQ may request a certain type of beam to be injected, but if the requested value does not correspond to the actual beam type being provided by the injector chain, then the request can not be fulfilled and no injection can take place. This value is thus inherited from the injector chain

BKNI Next injection RF Bucket –“AT”There are 35640 RF buckets around the LHC ring. It is essential that this parameter is established before RF re-synchronization starts between the CPS and the SPS RF systems, namely 450ms before CPS extraction towards the SPS

RNGI Next injection Ring –“AT”This parameter determines the value of the SPS beam destination in the DEST group of the telegram. Various ways to do this are possible. Its an OP decision.

AT: Arrival Time & 1Hz

12

Information on the LHC GMT cable

Safe Beam Parameters

The safe beam parameters must be delivered to the users over the GMT. Failure to deliver these parameters correctly and in time may result in a beam dump **. The Beam energy is measured by independent systems and transferred to the LHC central timing where one of them is sent over the GMT. A hardware monitor then compares these energy measurements against the GMT value and removes the beam permit if they differ by a threshold value, this hardware module also has a watch dog. Similar mechanisms exist for the intensities. The SBF is calculated by both systems and compared in hardware.

** For example the Beam Energy is received by a CTRV module, with timeout, that delivers this value directly over the P2 VME bus to hardware involved in Beam Transfer.

13

14

Information on the LHC GMT cableSafe Beam Parameters

ENG1 & ENG2 Beam Energy (per Ring) –“AT”These values are sent over dedicated timing cables from two independent measurement systems for each ring (4 Values in all) both to the LHC MTG and to the hardware monitor module (Controls Interlock SBF Generator CIG). If any of these 3 values differ for a ring, or if they are not transmitted within the watch dog time out period, the CIG removes the interlocks and the BIS then may remove the Beam Permit and dump the beam.

INT1 & INT2 Beam Intensity (per Ring) –“AT”The beam intensities are measured by BI and sent over dedicated timing cable as with Energy. Any failures will cause the BIS to remove the Beam Permit Flag. The intensity will be encoded as N x 10^10 where N is the value transmitted in the INT parameters.

15

Information on the LHC GMT cableSafe Beam Parameters

BPF1 & BPF2 Beam Present flag (per Ring) – 10HzThe beam present flags are calculated by the LHC MTG from the energy and intensity values, they are transmitted over the GMT cable with a 100ms period. See SBF below

PRF1 & PRF2 Beam Permit flag (per Ring) – 10HzThe beam permit flags are simply forwarded from the BIS at 10Hz. A transition of this flag from 1 to 0 provokes a postmortem event to be sent. Failure scenarios need study

SBF1 & SBF2 Safe Beam flag (per Ring) – 10Hz

The safe beam flags are calculated in the LHC MTG. As with the BPF they are transmitted over the GMT with a 10Hz frequency at millisecond 0, 100, 200, 300 …1000. The CIG module compares its own calculated values against that on the GMT with a 100ms watchdog, any differences reset the Beam Permit interlocks.

16

Information on the LHC GMT cable

MODE LHC Machine Mode – 1Hz

The LHC machine mode described in the FS is controlled through the API by the LSEQ and will appear immediately on the LHC timing cable. See **

** Monitoring this by hardware is not a good idea as it would require VME access to the interlock generator.

17

Information on the LHC GMT cable

FNUM Fill number – 1HzThis is controlled across the API by the HLLSA and may be used to tag postmortem data. At some point in the mode sequence the LSEQ will decide to issue a new fill number which is subsequently distributed as a 16 bit value at the ready telegram frequency.

BPNM Basic-Period Number – 1HzThe basic period of the LHC machine has been chosen to be one second, and to be coincident with the Pulse Per Second. The basic-period number is reset to zero when the Mode changes to Start of pre-injection. Note also that the millisecond modulo is also zeroed at PPS time, so the LHC machine time in the run is defined by BPNM + Millisecond in BP

PTY1 & PTY2 Particle Type (per Ring) – 1HzIn the distant future it may be possible to collide Ions and Protons. If this is in fact the case, then each Ring could have a different particle type. This value is inherited by the LHC MTG from the injector chain at the time of injection. Logically it could be treated or even be part of the Beam Type.

18

Other events on the LHC GMT

Post-Mortem Hardware triggered from the BIS via the Beam Present Flag transitions

LHC Injection forewarning Hardware triggered by the SPS extraction forewarning event

Other hardware triggered events as needed Events sent on request from the LSEQ Event tables repeating under LSEQ control Ready BP and TGM

19

Implementation decisions

Safe beam parameter distribution will be hardware monitored

A FESA Class will implement the API for the LSEQ across reflective memory

The basic-period will be equated with the UTC second

Telegrams will be a snap shot of LHC parameters taken each second

Telegram values will also be events that can be subscribed to

New approach is needed for the event tables, and best implemented in new hardware design

20

LHC MTG

2.2 G-Bit / S optical link64Mb Reflective memories

CMW ServerLHC MTG

GMTLHC

Clocks:40.00 MHz GPS clock1PPS (1Hz) clockBasic period clock

Event

Tables

SafeParams

Energy/RingIntensity/RingBIS Beam permit Flags

External Events

LSA High levelSequencer

LSA Core

FESA LHC API

Slave/Master

21

Multi-tasking CTG module

Same idea as the BST master card Multiple independent asynchronous tasks

running event tables under LSEQ control, looks like a collection of CTG cards all driving the same GMT cable

Load/Unload table

Run table N times

Run table for ever

Stop table

Abort table

LHC GMT

MT-CTG

16

22

Multi-tasking CTG moduleInstruction sampleADDR ,Add ,11,Reg ,Reg ,RegSUBR ,Subtract ,12,Reg ,Reg ,RegLORR ,Or ,13,Reg ,Reg ,RegANDR ,And ,14,Reg ,Reg ,RegXORR ,XOr ,15,Reg ,Reg ,Reg

ADDV ,Add ,16,Lit ,Reg ,RegSUBV ,Subtract ,17,Lit ,Reg ,RegLORV ,Or ,18,Lit ,Reg ,RegANDV ,And ,19,Lit ,Reg ,RegXORV ,XOr ,20,Lit ,Reg ,Reg

WDOG ,Watch Dog ,25,AdrJMP ,Jump ,26,Adr BEQ ,Branch Equal ,27,Adr BNE ,Branch Not Equal ,28,Adr BLT ,Branch Less Than ,29,Adr BGT ,Branch Greater Than ,30,Adr BLE ,Branch Less Than Equal ,31,Adr BGE ,Branch Greater Than Equal ,32,Adr BCR ,Branch Carry ,33,Adr

23

Multi-tasking CTG module16 Task Control Blocks

typedef enum { ILLEGAL_OP_CODE = 0x01, ILLEGAL_VALUE = 0x02, ILLEGAL_REGISTER = 0x04, WAITING = 0x08, STOPPED = 0x10, RUNNING = 0x20,} Task_Status;

typedef enum { EQ = 0x1, /* Last instruction result was Zero */ LT = 0x2, /* Last instruction result was Less Than Zero */ GT = 0x4, /* Last instruction result was Greater Than Zero */ CR = 0x8, /* Last instruction result Set the Carry */} Processor_Status;

typedef struct { unsigned long Pc; /* Program counter */ Task_Status TaskStatus; /* Status of Task, see above */ Processor_Status ProcessorStatus; /* Processor status word */ } Task_Block;

24

CCC TIMING RACK 4SYNC

CCC TIMING RACK 5CBCM B

CCC TIMING RACK 3CBCM A

Main MTG A

LHC MTG A

Main MTG B

LHC MTG B

HP GPS Receiver

GPS Antenna

1PPS 10MHz

LHC Switch

Main Switch

Reflective memoryHUB

DSC Comunications

DSC Synchronization

96

HardwareExternal

Conditionsfrom patch pannel

1PPS

10MHz

40MHz

Sync

A1PPS40MHz

Sync

A1PPS

40MHz

Sync

A1PPS40MHz

Sync

A1PPS

40MHz

Sync

A1PPS

GMTs PSB LEI CPS ADE SPS GMTs PSB LEI CPS ADE SPS

LHC GMT

LHC GMT

16 Ext Events

16 Ext Events 16 Ext Events

16 Ext Events

LHC Beam Energy/Intensity

GMT

LHC Beam Energy/Intensity

GMT

Work-Station /Server

Timing EventGMT

to CCC

SPS Intensity SPS Intensity

Master Slave configuration

25

Putting it all together

3 Sequences [PIN] loaded in LIC CBCM Pilot Intermediate Nominal

LIC Beams for LHC require the mode to be filling to execute, otherwise they spare **

The LIC telegrams are extended to contain the Beam-Type

** This may not be the best way, anyway this is an operational decision.There are many ways Sequences and Normal/Spare can be used

26

Nominal Filling

635 2395

SPS Extraction

18 basic-periods = 21.6 Sec

~3.6 Sec CBCM-Forewarning EstablishBatches Request

Batch 2 Batch 3 Batch 4Batch 1

7.2 Sec14.4 Sec

Last possible moment TCLPFor PSB Injection Bch3 Linac

1S

LHC bunch TestBunch QualityGo/No Go

Last possible moment Bch3For CPS Extraction to D3-Line

Next LHC RF Bucket established

3030

3S

Nominal LHC injector filling

CBCM OSD

27

Simple Use Case: Fill LHC Pilot-Intermediate-Nominal [PIN]

1. SPS operator selects CBCM Sequence PINThis is done in the usual way using the Sequence Manager, this action determines the Beam-Type. All LHC beams are in spare

2. LSEQ Selects mode = FillingUntil the mode is set filling, all LHC beams are spared, probably the SPS is in economy mode (There are other possibilities)

3. LSEQ Requests PINRequest: Beam-Type, CPS-Batches, LHC Ring, RF BucketIf the LSEQ were to ask for another beam type than what the LIC is programmed for an error is returned. Also it must ask in good time else an error is returned and no beam is delivered.

4. CBCM delivers PINThe BIS is in charge of the transfer and suppresses the LHC injection if needed

28

Implications for controls Very little or no classic multiplexing Basic-Period is 1S

This means that we can get the time since the start of the pre-injection mode I.E. LHC Machine-Time by taking the BPNM and adding the millisecond modulo to it

No cycles or super-cycles For cycle stamps we can use the basic-period stamp instead as a way

to stamp acquisitions No meaning for the USER telegram group

We can have a USER group, but it would probably always contain the same value

Payloads do not contain the USER, but they are not zero, they contain corresponding values like Energy, Mode, Beam-Type, Intensity, Power Converter groups, Flags etc.

Telegrams still exist, but it’s better to subscribe to the event if you need it as soon as it changes

The LHC timing will be implemented in its own TGM network with its own set of Ctims

LHC central timing is completely driven from LSA

29

Schedule

By January 2007 Running prototype ready for testing installed No master/slave capability initially LIC Part may not be fully functional Extensive reliability tests must be done over months

Ref: LHC Slow Timing System – High Level Operational Requirements

Functional Specification[Draft] Mike et All