Operational Efficiency

Walter Venturini Delsolaro

LHC Beam Operation Workshop - Evian 7-9 December 2010

Outline• Quick run through the run• Definitions and method• Fault statistics• Dump statistics• Technical stops• Tools• Operational efficiency

“Now, what I want is, facts. Teach these boys and girls nothing but Facts. Facts alone are wanted in life. Plant nothing else, and root out everything else. You can only form the minds of reasoning animals upon Facts: nothing else will ever be of any service to them. (…) Stick to Facts, sir!”

"a little inaccuracy saves a world of explanation"

Quick run through the run• 1-30 March: ramp commissioning,

first collisions• 1-16 April: squeeze commissioning,

then Physics • May: increasing Nb and kb, Physics• Intense summer: pushing & Physics• Canicular Physics • September: resuming commissioning • Bunch harvest• Heavy November

Method and Definitions • From 1 March to 30 November: 6600 hours• From the logbook, cross checked with status

reports of coordination and Timber for the beam presence

• Grid: Setup no beam (grey), beam setup (silver), stable beams (gold), TS/HC (purple), Fault (red)

• Availability = Beam presence + Setup no beam

• Downtime=Fault + TS/HC– Additional setup time after faults credited to the

faulty system– Not straightforward (coupled faults)

• … Shake and pie it up

Initial commissioning (March pie )

beam setup42%

Stable beams1%

setup no beam10%

Technical stop/HC19%

Fault29% 53%

availability

March faults distribution

CryogenicsQPS

EL+UPSPC

LBDSInjectorsVacuum

Access systemQ, Qp Feedbacks

RFControls

BLMExperiments

CVIQCMKI

0 1 2 3 4 5 6 7 8

April is the cruellest month…?

beam setup40%

Stable beams17%

setup no

beam9%

Technical stop/HC

13%

Fault22%

65% availability

April faults distribution

CryogenicsQPS

PCEL+UPS

LBDSInjectors

Q, Qp FeedbacksOP

ControlsAccess system

MKIVacuum

NOFBLM

QuenchIT

RFCollimators

Experiments

0 1 2 3 4 5 6

What the Thunder said

beam setup41%

Stable beams15%

setup no

beam8%

Tech-nical stop/HC6%

Fault30%

64% availability

All May faults

EL+UPSCryogenics

QPSPC

InjectorsLBDSBLM

CVControls

OPCollimators

MKIExperiments

Q, Qp FeedbacksRF

BICSoftIQC

0 2 4 6 8 10 12

June, change of Tune

beam setup44%

Stable beams4%

setup no beam8%

Technical stop/HC9%

Fault36%

56% availability

June faults

Cryogenics

QPS

NOF

Controls

RF

LBDS

CV

OP

MKI

Experiments

0 2 4 6 8 10 12

Another pie…July

beam setup35%

Stable beams18%

setup no beam

7%

Techni-cal

stop/HC15%

Fault24%

61% availability

…other testimony of summer nights

QPSPC

EL+UPSCollimators

Access systemOP

InjectorsControls

RFCryogenics

BLMQ, Qp Feedbacks

IQCMKI

LBDSBIC

Experiments

0 1 2 3 4 5 6

The summers corny crownbeam setup

31%

Stable beams29%

setup no

beam6%

Technical stop/HC7%

Fault26%

66% availability

Usual suspects

QPSCryogenics

PCEL+UPS

CVInjectors

Access systemBLM

RFPIC

settingsCollimators

MKIControls

Q, Qp FeedbacksBPM

VacuumOP

0 1 2 3 4 5 6

September trains

beam setup51%

Stable beams10%

setup no beam

11%

Technical stop/HC7%

Fault21%

72% availability

September faults distribution

QPS

PC

Injectors

Access system

EL+UPS

Controls

PIC

MKI

LBDS

Q, Qp Feedbacks

0 1 2 3 4 5 6

October slices

beam setup41%

Stable beams18%

setup no

beam10%

Techni-cal

stop/HC12%

Fault19%

69% availability

October faults distribution

CryogenicsInjectors

QPSPC

EL+UPSExperiments

BLMLBDS

TimingOPBICRF

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

Heavy ending

beam setup44%

Stable beams26%

setup no

beam11%

Techni-cal

stop/HC1%

Fault19%

80% availability !

Heavy faults

QPSPC

InjectorsAccess system

NOFLBDS

CVEL+UPS

CollimatorsQ, Qp Feedbacks

RFCryogenics

ControlsExperiments

OPMKIIQC

0 1 2 3 4 5 6 7

All faults downtime distribution

0 1 2 3 4 5 6

QPSCryogenics

PCEL+UPS

InjectorsAccess sys tem

LBDSCol l imators

ControlsRFOP

BLMCV

Q, Qp FeedbacksExperiments

NOFMKI

VacuumBICPIC

Alarm-fi re IT

IQCsetti ngs

BPMTiming

SoftQuench

Equipment type Faults Qty. Availability[1] [%] MTBF [hours]

Quench heater power supplies 26 6076 99.998 1145760

Quench detection systems 19 10438 99.999 3362135

DAQ caused by radiation (SEU) 12 1624 99.997 828240

DAQ other causes than radiation 8 2532 99.999 1936980

DAQ all faults combined 20 2532 99.997 774792

EE600 6 202 99.988 206040

EE13 kA 5 32 99.939 39168

QPS wins by a neck…

R. Denz

0

10

20

30

40

50

60

70

80

90

100

25 01 22 02 22 03 19 04 17 05 14 06 12 07 09 08 06 09 04 10 01 11 29 11

Avai

labi

lity

Daily "Global LHC Cryo" weekly AVG Monthly AVG Scheduled Stops

LHC Cryo global availability

Powering tests

Perturbations: clogging sub-atm circuits-CV891-instrumentation-Shaft seals-VFD/MB-24V

Learning spring Fantastic since summer !

Results for 2010 above expectations, thanks as well to periodic technical stops

S. Claudet

Dump statistics during ramp

BLM

Coll

MP test

SIS COD/orbit

OP mistake

software

LBDS fault

Exp

controls

0 2 4 6 8 10 12

74 protection dumps

Dump statistics during squeeze

PC tripOP switch

BLMCryoQPS

BPM IP6BPM

LBDS faultBISCollOFB

OP mistake

0 2 4 6 8 10 12 14

53 protection dumps

Dump statistics: from stable beams

BLMEL pert

QPS PC trip

LBDS faultOP switch

CryoBLM (spurious)

WaterBPM IP6

Exp (spurious)Access

OFBBPM

SIS CODRF (cryo)

Coll

0 2 4 6 8 10 12 14

66 protection dumps

Technical Stops• A total of 6 were done as scheduled • First started on March 15• Pattern: 4-36-3-31-4-45-5-37-4-45-4-

40+• Naive question: is the machine

availability more or less after a TS ? • Consider the 72 hours preceding and

the 72 hours following a TS, and compare pies...

• Compare faults for the various systems

BEAM (%)STABLE Beams

Setup no beam TS-HC Access FAULT (%)

before TS 75.00 0.00 13.89 8.33 6.94 5.56after TS 55.63 0.00 14.79 0.00 8.45 30.99DELTA -19.37 0.00 0.90 -8.33 1.51 25.43before TS 70.14 48.61 9.03 0.00 11.11 22.22after TS 49.65 0.00 11.81 0.00 24.31 42.71DELTA -20.49 -48.61 2.78 0.00 13.19 20.49before TS 61.81 1.39 9.03 0.00 6.94 29.86after TS 51.39 5.56 15.28 0.00 5.56 34.72DELTA -10.42 4.17 6.25 0.00 -1.39 4.86before TS 63.19 28.47 5.56 0.00 5.56 31.94after TS 45.14 0.00 27.78 0.00 11.11 28.47DELTA -18.06 -28.47 22.22 0.00 5.56 -3.47before TS 64.58 39.58 8.33 0.69 22.22 29.17after TS 54.86 0.00 14.58 0.00 0.00 31.94DELTA -9.72 -39.58 6.25 -0.69 -22.22 2.78before TS 67.71 15.28 20.14 0.00 2.78 14.93after TS 80.56 29.86 16.67 0.00 0.00 4.17DELTA 12.85 14.58 -3.47 0.00 -2.78 -10.76

All faults: after TS- before TS

Trends of “TS messing up” effect...

0 1 2 3 4 5 6 7-15

-10

-5

0

5

10

15

20

25

30

Tools• Logbooks to get the fault attributions• Measurement and Logging DB: a lot of information,

JAVA API available to do specific searches, already used by some people (C. Roderick)

• Web-based Post Mortem Data Extraction (M. Zerlauth)

For next year: try at least to copy SPS: automatic entries in the logbook when there is a fault, for LHC it is more difficult, one has to take into account machine modes, etc. (more in Oliver’s talk). Also we need to agree on conventions.

Several people are working to applications for the Fill statistics (e.g. SUPERTABLE, and others), using the same API used by TIMBER

Wrap up • Machine availability for the run: 65%• Faults: 25% (TS 10%)• Beam presence: 56%, setup no beam: 9%• Stable beams: 15.7 % (e1)• e2=Physics/Available: 23.7%• For most of the 2010 run, e2 is not a good indicator of

operational efficiency, as it rejects all the beam commissioning time

• Last two weeks of August: e2 ~ 50%. • Max e2 = 83 % (with 10.6 hs fill time and minimum

turnaround)– With 65% machine availability and only trying to do phyics:

Max e1 = 54% , or 32% if we had the same efficiency as in August

Conclusions• 2010 run was driven by commissioning, not

physics• Machine Availability was satisfactory and steadily

increasing• Equipment performs above expectations (MTBF

etc)• Equipment groups are aware of the weak points

and are working to improve them• Less mixing of Physics and beam commissioning

in 2011• TS to be reviewed for 2011, can probably be less

frequent• More tools for statistics to be developed, also with

discussions with equipment groups • Margin to improve operational efficiency ( see

Stefano’ talk)