atlas operations 2008 getting ready / first beam / combined running & shutdown planning
DESCRIPTION
ATLAS Operations 2008 Getting ready / First beam / combined running & Shutdown planning. T. Wengler DESY seminar 19 Jan 08. The ATLAS collaboration. 37 countries 169 Institutes 2500 Authors. The ATLAS Detector. Solenoid field: 2T Toroid field: 4T. Inner Detector. Barrel. - PowerPoint PPT PresentationTRANSCRIPT
ATLAS Operations 2008Getting ready / First beam / combined running
& Shutdown planning
T. Wengler
DESY seminar
19 Jan 08
The ATLAS collaboration
2
37 countries169 Institutes2500 Authors
The ATLAS Detector
3
Solenoid field: 2TToroid field: 4T
Inner Detector
4
Barrel
The LHC complex
5
Need full LHC for any beam through ATLAS
Milestone commissioning weeks
6
Commissioning of the integrated system
2007
BarrelCalorimeters
BarrelCalorimetersBarrel Muon
M2 + EC CalorimetersEC MuonBarrel SCT, TRT
2008
M1 M2 M3 M4 M6M5
M3 + (-)SCT r/oL1 Mu/Calo
M4 + Pixel r/o(-) TRT
M5 + SCT detectors
M6 highlights 1/3
7
SCT Barrel (~1/2 SCT), stable cooling throughout M6 TRT 25% Barrel A. 10% Barrel C, top + bottom sections RPC Sect 7/8, TGC 3 Sect/side, MDT Sect 1-12 Barrel, Big Wheels, 650 chambers
Reconstructed cosmic SCT-TRT-MuonReconstructed cosmic SCT-TRT-Muon
Phi difference SCT+TRT track segment RPC
M6 highlights 2/3
8
EMB pulse
shape
L1Calo trigger seen in LArL1Calo trigger seen in LAr
Extensive testing of Calorimeters and L1Calo during M6 and after, including trigger timing studies
L1Calo trigger seen in TileL1Calo trigger seen in Tile
M6 highlights 3/3
9
Analysis of one M6 cosmics commissioning run , March 2008– The trigger is requiring TRT tracks reconstructed online within the pixel
volume (equivalent to d0 25 cm)
– Triggered events (red) end up in one stream file, non-triggered events (blue) into another one: proves trigger and streaming are working
All TRT reco tracks
Triggered TRT tracksHigh LevelTrigger
Month Date System Requirements, remarks Parallel Shifts
Week 1721/4 - 27/4
TDAQ/HLT week ID ROSs in useL1Calo ROS requested 25/4
ID standalone tests w/o CTP; BCM integration
April Week 1828/4 - 30/4
L1Calo+Calo run No TDAQ 24 hour period
May Week 1830/4 - 4/5
May 1st Muon-CSC test
Sub-systems: Transition to tdaq-01-09
Week 195/5 – 11/5
5/5 – 6/5 Muon-CSC test 7/5 – 9/5 TRT test8/5 11:00 – 9/5 11:00 TDAQ 24h
SCT likely not fully available yet (cooling) to start on Thursday
Start of SCT calibration [Round 1]Start of magnet test ~HLT algos available
May Week 20
12/5-18/5 12-13/5 for SCT+CTPCalo+L1calo+HLT start on 14th
-Timing, calo DQ, debugging, high rate, algo tests- Finish with a stable week end run
Week days: morning expert work; evening calo + central desks
WE: 16/7 calos + central desks
Week 21
19/5-25/5 19/5 CaloMuon+L1Mu+HLT starts on 20/5
-Same as above
- Finish with a stable week end run with calos
Week days: morning expert work; evening muon + central desks[SCT calibration round 2]WE: 16/7 muon + calos + central desks
Week 22
26/5-1/6 ID+DAQ+HLT -Same as above-Dedicated DAQ test after detector testing and before WE run with muons + calos
Week days: morning expert work; evening ID (Muon/calo?) + central desksWE: 16/7 ID (muon/calos) + central desks
June Week 232/6 – 8/6
Keep system in one piece, perform HLT tests 2-3/6, end 9:00 on 4/6First week of semi continuous running
No Tier-0 ! Magnet testFDR-2 10
Schedule: May-June
Milestone commissioning weeks
11
Commissioning of the integrated system
2008
Full system(-) CSC
2009
M8M7
Full system
first beam M9 ?
cosmics comb.
Magnet system
12
The magnets and their cryogenics, powering, dump etc systems have been fully commissioned to nominal field strength
Another importantmile stone
Closure of the LHC beam pipe
13
The Final Piece: Closure of the LHC beam pipe ring on 16thJune 2008 ATLAS was ready for data taking in August 2008
ACR for 2008 running
14
Trigger•Configuration•Performance
Trigger•Configuration•Performance
Level-1•L1Calo,L1Muon, CTP h/w & r/o•Overall Timing
Level-1•L1Calo,L1Muon, CTP h/w & r/o•Overall Timing
DAQ/HLT•DAQ/Dataflow•HLT/DAQ farms
DAQ/HLT•DAQ/Dataflow•HLT/DAQ farms
Muon detectors•TGC, RPC, MDT, CSC
Muon detectors•TGC, RPC, MDT, CSC
SLIMOS•CIC, safety, access•CCC/TC communication
SLIMOS•CIC, safety, access•CCC/TC communication
Run Control•Main RunControl
Run Control•Main RunControl
Shift Leader•Shift Leader tasks•LHC communications•DCS central operation
Shift Leader•Shift Leader tasks•LHC communications•DCS central operation
Data Quality•Central DQ•Online/Offline/Tier-0
Data Quality•Central DQ•Online/Offline/Tier-0
Tile Tile
Exp. Supervisor•Chair + Laptop
Exp. Supervisor•Chair + Laptop
Fwd Detectors•Lucid, ZDC (ALFA …)•Luminosity
Fwd Detectors•Lucid, ZDC (ALFA …)•Luminosity
TRT TRT
SCT SCT
Pixel Pixel
ID Gen•Group two screens intoID Gen access station(evap. cool etc.)•No separate shifter
ID Gen•Group two screens intoID Gen access station(evap. cool etc.)•No separate shifter
LAr•Two shifters with separate access
LAr•Two shifters with separate access
Going into beam op. Sep 10th
• Muon system (MDT, RPC, TGC) on at reduced HV
• LAr (-FCAL HV), Tile on
• TRT on, SCT reduced HV, Pixel off
• BCM, LUCID, MinBias Scint. (MBTS), Beam pickups (BPTX)
• L1 trigger processor, DAQ up and running, HLT available (but used for streaming only)
ATLAS was ready for first beam:
tertiarycollimators
140 m
BPTX175 m
LHC start-up scenario:• Go step-by-step, stopping beam on
collimators, re-align with centre
• Open collimator, keep going
• Last collimator before beam through ATLAS – tertiary collimators (protection of triplets)
• Splash event from these collimators for each beam shot with collimator closed
LHC beamloss monitor
Strategy for first beam / first signals
• Splash events are first– only 10 shots expected – not
enough time to set up BPTX
– Rely on small radius triggers with well defined cosmics timing (L1Calo + MBTS)
– Catch the first events!
• Start timing in Beam Pickups (our timing reference) rapidly to trigger on through-going beam– First signal seen in ATLAS
– Multi-turns!
BPTX
Trigger
Worked out nicely, with very stable data taking
RF captured beam for > 20 min on day 2!
16
The first events – did we get them?
17
Zoom into the first beam ‘splash’ activity with beam-1 closed collimator, recorded in run 87764.
Of 11 beam ‘splashes’ (solid-blue), 9 have been triggered (ATLAS events are dashed-black, the event numbers are indicated).
“First beam event seen in ATLAS” (not quite !)
Timing of the trigger inputs
18
1 BC = 25 ns
Intense programme of timing adjustments of trigger inputs
Only about 30 hours of (any kind of) beam in 2008 …
The first has been delivered to the CCC
20092009 20092009
Delivered!
LHC incident on Sept. 19th
20
See slides R. Aymar attached below for full details
Powering tests in Sect 3-4 for
5.5 TeV operation
Powering tests in Sect 3-4 for
5.5 TeV operation
Resistive zone in bus between C24 and Q24
Resistive zone in bus between C24 and Q24
In < 1 sec power converter tripped and dump started
In < 1 sec power converter tripped and dump started
Systems reacted as expected until here
Electrical arc in bus, punctured
helium enclosure
Electrical arc in bus, punctured
helium enclosure
Helium release into isolation vacuum of
cryostat
Helium release into isolation vacuum of
cryostat
Safety valves start venting helium to the
tunnel
Safety valves start venting helium to the
tunnel
Pressure rise too fast for valves,
pressure wave in isolation vacuum
Pressure rise too fast for valves,
pressure wave in isolation vacuum
Damage to super-insulation
for some magnets
Damage to super-insulation
for some magnets
Displacements where wave hits vacuum
barriers
Displacements where wave hits vacuum
barriers
Both beam pipes cut open
and contaminated
Both beam pipes cut open
and contaminated
About 60 magnets to be removed for repairs etc
About 60 magnets to be removed for repairs etc
No more beam in
2008
No more beam in
2008
Q27R3
21
R. Aymar, PECFA, 28 Nov 08
QQBI.27R3
22
R. Aymar, PECFA, 28 Nov 08
Magnet movements Sector 3-4
23
Displacements status in sector 3-4 (From Q17R3 to Q31L4)Based on measurements by TS-SU, TS-MME and AT-MCS
P3Q17 A18 B18 C18 Q18 A19 B19 C19 Q19 A20 B20 C20 Q20 A21 B21 C21 Q21
Cryostat <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2CM Longi ? ? ? ? ? ? ? ? <2 <2 <2 <2 <2 <2 <2 <2 <2CM Vert ? ? ? ? ? ? ? ? <2 <2 <2 <2 <2 <2 <2 <2 <2CM Rad ? ? ? ? ? ? ? ? <2 <2 <2 <2 <2 <2 <2 <2 <2
Q21 A22 B22 C22 Q22 A23 B23 C23 Q23 A24 B24 C24 Q24 A25 B25 C25 Q25
Cryostat <2 <2 <2 <2 -7 <2 <2 <2 -187 <2 <2 <2 <2 <2 <2 <2 <2CM Longi <2 <2 <2 <2 -20 -65 -104 -141 <2 -186 -127 -70 <2 <2 <2 <2 <2CM Vert <2 <2 <2 <2 <2 -6 -5 -4 <2 -4 -5 -5 <2 <2 <2 <2 2CM Rad <2 <2 <2 <2 <2 0/10 11/8 7/3 <2 15/3 8/13 11/3 <2 <2 <2 <2 <2
Q25 A26 B26 C26 Q26 A27 B27 C27 Q27 A28 B28 C28 Q28 A29 B29 C29 Q29
Cryostat <2 <2 <2 <2 <2 <2 <2 <2 474 -4 <2 <2 11 <2 <2 <2 <2CM Longi <2 <2 <2 <2 <2 57 108 168 -38 232 188 145 95 70 35 3 <2CM Vert 2 <2 <2 <2 <2 -5 -5 -4 -26 58/-7 -7/-5 -8/33 12 -5 <2 <2 <2CM Rad <2 <2 <2 <2 <2 2/<2 8/9 3/15 22 20/<2 <2/12 16/6 <2 <2 <2 <2 <2
Q29 A30 B30 C30 Q30 A31 B31 C31 Q31 A32 B32 C32 Q32 A33 B33 C33 Q33
Cryostat <2 <2 <2 <2 <2 <2 <2 <2 188 <2 <2 <2 5 <2 <2 <2 <2CM Longi <2 <2 <2 <2 <2 19 81 146 <2 141 102 63 10 <2 <2 <2 <2CM Vert <2 <2 <2 <2 <2 <2 -5 -4 <2 -11/-5 -6/-5 -5 3 <2 <2 <2 <2CM Rad <2 <2 <2 <2 <2 <3 3/6 10/17 <2 -3/6 6 6/<2 <2 <2 <2 <2 <2
Q33 A34 B34 C34 Q34 C34 B34 A34 Q33 C33 B33 A33 Q32 C32 B32 A32 Q31
SSS with vacuum barrier Open interconnection Disconnected P4>0 To P4, up, center Electrical interruptions Removed[mm] Values are in mm Dipole circuit (diode) XYZ Reinstalled? Not measured yet Electrically damaged IC
Cold mass displacement Buffer zones Electrical cantonsCryostat displacement Date 25/11/2008 JPh Tock
R. Aymar, PECFA, 28 Nov 08
LHC incident on Sept. 19th
• Prevention of similar incidents– Search for indicators of faulty connections in commissioning
data– Retests of all suspicious locations– Improvements on Quench Protection Systems
• Mitigation of consequences in case of similar incidents– Increase number and discharge capacity of relief valves– Reinforce external anchoring of cryostats at the locations of the
vacuum barriers
24
See slides R. Aymar attached below for full details
Shutdown intervention schedule
25
ATLAS reacted with a revised running and shutdown plan:
Running 2008
26
Oct Nov Dec41 42 44 45 46 47 48 49 50 5143
41 42 44 45 46 47 48 49 50 5143
So +To Sol. only
Sub-system (group) commissioning24/7 combined run
Stop TGC n-pentane
Start to open
Full shiftcoverage as agreed for this period
9:30 RunMeeting27. Oct
Weekly run meetings only
• SLIMOS + SL(RC)• TDAQ/Sys-admin on-call• Effort from sub-systems as
needed for ongoing activity• RPC S12-14 available for
combined runs
Run Priorities until Oct 27th
1. Collect enough data with small wheels to check efficiencies of chambers potentially damaged by overpressure before n-pentane stops. Implications:– Recording: Record all TGC triggered events
– Trigger: enable full TGC trigger coverage for some time (partial blocking of trigger with good timing for ID end-caps)
2. Collect large samples of good ID tracks with Solenoid on/off. Implications:– High recording rate of L1 trigger = small event size (HLT not fully
efficient for these conditions)
3. Collect large samples of good Muon spectrometer tracks with field off/on/off. Implications:– Same as 2.
4. Run combined system for cross-detector studies and stability– All (available) systems in readout 27
✔
✔
✔
✔
Data overview
28
High rate running
• Re-visited at the end of combined period– Stability (i.e. people not touching essential parts of their DAQ
and FE) had started to erode somewhat …– Still managed to do several high rate runs, with random triggers
at L1, filtered at HLT
• Today’s evidence suggests we can run combined with about 40 kHz L1A– No indication for bottlenecks at nominal rates downstream
• Also did test on stop-less recovery (automatic disabling of blocked read-out links) – switched on without problems, but – no systematic testing (provoked failures) during combined
running 29
After the end of the combined run
• Moved immediately into system commissioning– ID running 24/7 shifts for flat-out commissioning work into
December– Frequent L1Calo/Calo runs with and without CTP, including 2
weeks of overnight/WE runs from Nov 3rd – RPC trigger test and threshold scans– …
• Major events still in 2008– ID combined run (RPC trigger, TRT fast OR) 26th Nov – 1st Dec– New major TDAQ release beginning of December
(testing by sub-detectors starts before end of 2008)
• Test Programme– Bare-bone run transition timing (all monitoring etc turned off)– Check situation on amount of messages in MRS
30
Collection of cosmic ray data• Full cosmic rate (dominated by RPC) at L1 is ~500 Hz
– Nominal ATLAS recording rate is 200 Hz @ 1.5 MB/event– For cosmics with full system 3-13 MB/event, dominated by number
of LAr samples read out → We need the HLT for cosmics!
• Example: Rate of Pixel tracks:
31
Full L1 raterecorded (no LAr)
HLT cosmics filtering learning curve
New TRT fast-OR at L1
Pixel alignment
32
An example of using cosmic-ray data for detector alignment
The TRT as …
33
Bubble chamber
A cosmic event with
field on
Occupancy ~ 1%
A beamsplash event
Fixed target detector
Occupancies up to 30%
Transition radiation in the TRT
34
Transition radiation (TR) photons generated by radiator foils (boundary of 2 materials with different dielectric constants)
Effect starts at γ = (E/m) ≈ 103
➞ mostly for electron ID
In the TRT, photons are absorbed in chamber gas
➞ large pulse ➞ passes high threshold
γ
Turn-on of TR from cosmicsat about γ=103 as expected
Muon spectrometer
Optical alignment system• Dead channels < 1%• End-cap: 40 μm o.k.➞• Barrel: 250 μm in progress ➞
Very good correlation between RPC (trigger chambers) andMDT (precision chambers) hits
X-ray of the ATLAS cavern with cosmic muons
Elevators Access shafts
Calorimetry
36
1 MeV
Pedestal stability in LAr EM layer over a 5 months period
Tile cell uniformity measured with horizontal beam
L1Calo trigger tower (x) vs. calorimeter full readout (y),Spread will decrease with full calibration
How much of ATLAS is operational
• Inner Detector– Critical item was evaporative
cooling plant (for Pixel/SCT) after failure on May 1st 2008 (repair finished end of July 2008)
– About 2.5 % dead channels due to cooling/module problems, similar amount of dead channels in TRT
• Calorimeters– For LAr ~0.01% dead channels in
EM, and ~0.1% in HEC, plus LV power supply affecting ¼ of one endcap (under repair)
– No dead channels in FCAL
– About 1 % dead channels in Tile cal
• Muon spectrometer– Low noise and low number of dead
channels
– MDT: ~1.5%, CSC: 0.5%, TGC: 0.03%, RPC: 6% (commissioning still ongoing)
• Magnets– Toroids and Solenoid fully
operational
• Trigger and DAQ– All L1 trigger inputs and central L1
trigger fully operational
– HLT farm and algorithms, and DAQ system fully operational (will be scaled up according to need)
37
In all cases expect to recover a significant fraction during shutdown from an already small number of missing channels
Conclusions
• We had a very successful start-up– Caught the first beam events
– Managed to go through part of the single beam commissioning programme on day 1+2, with beam
– We start to be able to operate and steer ATLAS as one system
– On-the-fly L1-pre-scale changes and r/o recovery are extremely useful
• Only the CSC now missing from regular combined data taking– CSC chambers up and running, r/o
ROD issues being addressed, first tracks seen in combined runs
– Pixel off on day one for safety reasons, but has seen many cosmic tracks in combined running since
• Timing needs to be completed– As much as possible with test-
pulses & cosmics over the next months, but need beam to finalise it
– Will be in good shape for collisions
• The operation model is working– Control room is working
– Operational structure is sound
• Main work for operations– Bring down overheads (and up
efficiency)
• run start/stops
• recovery for power cuts
• DQ tools / turn-around
• …38
Additional Slides
Inner Detector
40
Calorimeter system
41
Muon system
42
Beam RF capture
43
Correct phasingCorrect reference
‘Mountain range’plot
‘Each line = one bunch
orbit
LHC RF group
Beam splash events in LAr and Tile
44
Energy in LAr EM layer 2
Average cellenergy in Tile
LHC incident on 19th Sep 2008
45
Incident during powering
The magnet circuits in the seven other sectors of the LHC had been fully commissioned to their nominal currents (corresponding to beam energy of 5.5 TeV) before the first beam injection on 10 September 2008. For the main dipole circuit, this meant a powering in stages up to a current of 9.3 kA. The dipole circuit of sector 3-4, the last one to be commissioned, had only been powered to 7 kA prior to 10 September 2008. After the successful injection and circulation of the first beams at 0.45 TeV, commissioning of this sector up to the 5.5 TeV beam energy level was resumed as planned and according to established procedures.
On 19 September 2008 morning, the current was being ramped up to 9.3 kA in the main dipole circuit at the nominal rate of 10 A/s, when at a value of 8.7 kA, a resistive zone developed in the electrical bus in the region between dipole C24 and quadrupole Q24. No resistive voltage appeared on the dipoles of the circuit, so that the quench of any magnet can be excluded as initial event. In less than 1s, when the resistive voltage had grown to 1 V and the power converter, unable to maintain the current ramp, tripped off, the energy discharge switch opened, inserting dump resistors in the circuit to produce a fast power abort. In this sequence of events, the quench detection, power converter and energy discharge systems behaved as expected.
R. Aymar, PECFA, 28 Nov 08
Summary Report on the analysis of the 19th September 2008 incident at the LHC
46
Sequence of events and consequences
Within the first second, an electrical arc developed and punctured the helium enclosure, leading to release of helium into the insulation vacuum of the cryostat.
The spring-loaded relief discs on the vacuum enclosure opened when the pressure exceeded atmospheric, thus relieving the helium to the tunnel. They were however unable to contain the pressure rise below the nominal 0.15 MPa absolute in the vacuum enclosures of subsector 23-25, thus resulting in large pressure forces acting on the vacuum barriers separating neighboring subsectors, which most probably damaged them. These forces displaced dipoles in the subsectors affected from their cold internal supports, and knocked the Short Straight Section cryostats housing the quadrupoles and vacuum barriers from their external support jacks at positions Q23, Q27 and Q31, in some locations breaking their anchors in the concrete floor of the tunnel. The displacement of the Short Straight Section cryostats also damaged the “jumper” connections to the cryogenic distribution line, but without rupture of the transverse vacuum barriers equipping these jumper connections, so that the insulation vacuum in the cryogenic line did not degrade.
R. Aymar, PECFA, 28 Nov 08
Summary Report on the analysis of the 19th September 2008 incident at the LHC
47
Inspection and diagnosticsThe number of magnets to be repaired is at maximum of 5 quadrupoles (in Short Straight Sections) and 24 dipoles, but more (42 dipoles and 15 quadrupoles) will have to be removed from the tunnel for cleaning and exchange of multilayer insulation.
Spare magnets and spare components are available in adequate types and sufficient quantities for allowing replacement of the damaged ones.
The extent of contamination to the beam vacuum pipes is not yet fully mapped, but known to be limited; in situ cleaning is being considered to keep to a minimum the number of magnets to be removed.
The plan for removing/reinstallation, transport and repair of magnets in sector 3-4 is being established and integrated with the maintenance and consolidation work to be performed during the winter shutdown.
It should be available for the next Council meeting in December.
The corresponding manpower resources have been secured.
R. Aymar, PECFA, 28 Nov 08
Summary Report on the analysis of the 19th September 2008 incident at the LHC
48
Follow-up actions (preliminary)
Two different goals, namely to prevent any other occurrence of this type of initial event, and to mitigate its consequences should it however reproduce accidentally. Precursors of the incident in sector 3-4 are being scrutinized in the electrical and calorimetric data recorded on all sectors, which remain cold, in order to spot any other problem of the same nature in the machine.
• An improvement of the quench detection system is currently tested, before being implemented.
• The relief devices on the cryostat vacuum vessels will be increased in discharge capacity and in number.
• The external anchoring of the cryostats at the locations of the vacuum barriers will be reinforced to guarantee mechanical stability.
Until now, no other interconnection resistance has been identified as above specification, but two (?) connections inside the cold masses (which have been tested successfully to 9T) have been measured higher than specified.
R. Aymar, PECFA, 28 Nov 08