6/21/2007 PHENIX - MJL 1
cosmic raymuons
Mike Leitch, LANLRun Coordinator
Run7 Report
PHENIX 2007 Collaboration Meeting
Boulder, 11-13 July 2007reaction plane
6/21/2007 PHENIX - MJL 2
Run7 Chronology• Originally hoped to start with Nov 1st cool-down
and PHENIX planned for:– AuAu – 15 wks (1.1 nb-1), pp – 10 wks (32 pb-1)
• Delayed by continuing resolution & meager FY06 level to match• But PHENIX started anyhow:
– Cosmic rays starting Jan 9th
– “Run7 is on” Feb 1st
– Start of cool-down Feb 8th
– Cryo problems Feb 24th
– Rings cold & with beam Mar 13th
– Start of Physics Mar 27th
• End of Run7 Jun 26th
• 13 wks of physics (Mar 27th – Jun 26th)
cosmics
6/21/2007 PHENIX - MJL 3
• 3/27 "Start of physics" 1st run 227016• 4/2 23:59 Change to BBCLL1(>1 tubes) 1st run after change 228040
for main MB trigger • 4/4 14:15 scaler update change 1st run after change 228040
from 5 to 30 seconds • 4/25 HBD West out 1st run after this 231210
(during maintenance day) • 5/11 20:18 last level-2 config change 1st run after 234416
(E/p > 0.35 instead of > 0.5) • 5/21 9:30 am "++" magnetic field 1st run after this started
235889 - 5/23 10:00 am
• 5/23 10:00 go to "-+" magnet config 1st run after change 236133(during maintenance day)
• 6/4 14:22 fixed PC-E channels finally about 237717 • 6/6 5pm low-energy run last run before this 237831
- 6/7 4pm (after maintenance day) 1st run after it 238022 • 6/12 major DC LV repair after 238681
West-south sectors 6,7,8 • 6/12 12:05 am (west) converter run includes runs between
238682 - 6/14 8:00 am and 238854 (inclusive)
run 238943 is after it is out • 6/26 11:00 end of run last run 240121 • 6/29 2:00 end of run party....
Important Run7 Dates and Run Numbers
6/21/2007 PHENIX - MJL 4
PHENIX Physics Goals for Run7• Increase statistical & systematic
precision of rare signals in AuAu, e.g. J/, jet correlations, etc
• Increase reach in pT, especially with PID from new TOF-West detector (pT > 8 GeV/c)– Identified particle spectra – Identified leading particles in
jets• Factor of two or more
improvement in Reaction Plane resolution - valuable to many signals– v2 for J/, - new– electrons, hadrons – extended
• Low-mass lepton pairs with the HBD
simulatio
n
Submitted to PRL
6/21/2007 PHENIX - MJL 5
TOF-W
RXNP
HBD
MPC-N
Four New Detector Subsystems for Run7
6/21/2007 PHENIX - MJL 6
Centrality
RXNP
MPC
BBC
< c
os
2(
so
uth -
n
ort
h )
>PHENIX Reaction Plane Measurements Now
from MPC, RXNP & BBC
RP
vv
expt2true
2
< c
os
2(
A+
B -
tr
ue
) >
< cos 2(A – B ) >
RXN
BBCMPC
MPC
Reaction Plane Detector
6/21/2007 PHENIX - MJL 7
TOF-West Particle Identification
6/21/2007 PHENIX - MJL 8
5.12 Billion Min-bias events
813 b-1
(~3.4 x Run4)
6/21/2007 PHENIX - MJL 9
All Run7 % Last week (6/19-26) %PHENIX
BUP
CA-D luminosity
3260 b-1 482 b-1
PHENIX ZDC luminosity
Uptime
20.46B ZDC/9.8b= 2088 b-1
64.4%3.41B ZDC/9.8b
= 348 b-172.2%
PHENIX BBC luminosityVtx cut*
6.27B BBCLL1/(0.92*6.847b)
= 995 b-1
47.7%960M BBCLL1/(0.92*6.847b)
= 152 b-1
43.7% 70%
PHENIX live luminosityLive time
5.12B BBCLL1/(0.92*6.847b)
=813 b-1
81.7%854M BBCLL1/(0.92*6.847b)
=136 b-1
89.5%
PHENIX up*live time
59.5%*81.7% 48.6% 72.2%*89.5% 64.6% 60%
Overall 24.9% 28.2%
The PHENIX Luminosity Story
*Vertex eff slightly worse after “+-” “-+” magnetic field change (cause of lower vtx eff for last week than whole run)
6/21/2007 PHENIX - MJL 10
beginningof store
endof store
from yellow beam
from blue beam
Effects of Stochastic Cooling at PHENIX
Improvement in integrated luminosity for PHENIX with Stochastic Cooling (SC)
16%
6/21/2007 PHENIX - MJL 11
Fraction of Time at Collision
6/21/2007 PHENIX - MJL 12
Luminosity/Day vs Day
6/21/2007 PHENIX - MJL 13
PHENIX Triggering & fast analysis for Run7
Level-1 Triggering just Minimum-Bias (+UPC)• 5 kHz event rate through DAQ• only small fraction of integrated luminosity missed due to rates above the DAQ limit in the 1st ½ hour or so of storesLevel-2 filtering in ATP’s producing selected, parallel sample of enriched events for fast processing• J/ ; J/ ee; high-pT level-2 triggers
• 10% of raw data size• sent to CCF in France via GRIDFTP• analyzed in France for fast results
10% of min-bias events sent to Vanderbilt for fast analysis
• check analysis/calibrations, prepare for full analysis pass• provide events needed for mixing in level-2 analysis
Counting house machines for fast HBD analysis
6/21/2007 PHENIX - MJL 14
PHENIX Data Acquisition (DAQ)
DAQ Advances:• level-2 code speedup• additional buffer box• fix SEB memory leaks• more compact HBD data format• etc
Present DAQ performance:• up to 5 khz event rate & 700 Mb/sec throughput
6/21/2007 PHENIX - MJL 15
(plot from Tom Throwe)
6/21/2007 PHENIX - MJL 16
cosmics
PHENIX Shakedown/Cosmic Ray Running(Jan & Feb)
Need Cosmic Ray background measurement for W physics• Cosmic/W estimated to be 1/1 for p>=10 GeV, but estimate is unreliable – need measurement!Also Cosmic ray in EMCal looks much like high-pT photon
6/21/2007 PHENIX - MJL 17
PRL98, 232301 (2007)
Statistical & systematic precision of new Run7 AuAu data will improve this picture – But will also benefit enormously from new dAu data that can constrain the cold nuclear matter baseline accurately
J/’s in AuAu Collisions at PHENIX
PRL96, 012304 (2006)
6/21/2007 PHENIX - MJL 18
Fro
m T
ony
Fra
wle
y, h
eavy
PW
G,
Jan
4th
Expected Results for J/ flow from Run7
6/21/2007 PHENIX - MJL 19
for ~8% of present Run7 integrated luminosity(~16,000 J/ for present luminosity sum)
J/
unlike-sign pairslike-sign pairs (randoms)
thanks to Catherine Silvestre
6/21/2007 PHENIX - MJL 20
J/ e+e-
about 15% of level-2 data
thanks to Ermias Atomssa& Raphael Granier de Cassagnac
6/21/2007 PHENIX - MJL 21
0’s & ‘sthanks to Justin Franz
from about 16% of present data
0
extended range in pT
6/21/2007 PHENIX - MJL 22
• A “hadron-blind” detector to detect and track electrons near the vertex.”
• Dalitz rejection via opening angle – Identify electrons in field free
region– Veto signal electrons with
partner• HBD: a novel detector
concept: – windowless CF4 Cherenkov
detector – 50 cm radiator length– CsI reflective photocathode – Triple GEM with pad readout
• The HBD will improve our S/Bby a factor of ~100
Hadron Blind Detector (HBD)
signal electron
Cherenkov blobs
partner positronneeded for rejection e+
e-
pair opening angle
Submitted to PRL
6/21/2007 PHENIX - MJL 23
electrons
hadrons
Hadron Blind Detector (HBD)West side of HBD taken out Apr 25th for repairs• HV trips with large stored energy damaged detector• Now being refurbished with new GEM’s & fixed HV • ½ of the East side of HBD still in
6/21/2007 PHENIX - MJL 24
PHENIX Issues & Failures• HV problems
– Slow or dead servers– SA1 screen slow, or dead– Non-standard subsystem HV control – HBD, ToF-W, ToF-E– Unclear subsystem standing orders
• LV problems– Connections or burned out LV modules
• DAQ front end– HBD event size and zero suppression– DC large packets– MPC fiber split, muTr & EmCal new formats done
• SEB memory leaks• Buffer Boxes
– Confusion about Box7 (LVL-2) & its transfers– Weird transfer impact on DAQ speed when trigger rate high– Several disk failures
• Event builder & ATP’s– Did not reach near 5khz event rate until last few weeks of run!– Only one expert, many changes– Will performance survive for Run8 and Run9??– Trigger/level-2 configuration complicated – few know how to do it– Diagnostics weak for DAQ operator
6/21/2007 PHENIX - MJL 25
PHENIX Issues & Failures - 2
• muTr HV trips & dead channels (capacitor removal in progress)• Level-2 analysis slow starting; partly because of slow DAQ level-2 start
caused by struggles to get DAQ to run fast• SA2 under utilized• Communication in English difficult for some young subsystem specialists
and shifters– recommend one person in subsystem team be fluent in English
• Never had a chance to start new mandatory shift assignment scheme because of run start uncertainties
• HBD data size– Already implemented more compact format, but will (hopefully) have
many more good channels next year (~24/6)– Drop to fewer than 12 time samples?– Reduce large tail in event size with light baffling?– More sophisticated zero-suppression scheme
• Haggerty-less operation next year – a challenge to us all
6/21/2007 PHENIX - MJL 26
Other Issues• Water cooling
– Availability of chiller & evaporator status to shift crews so they diagnose cooling problems quickly & call for help
– Coupling of magnet and electronics cooling• Air conditioning
– Humidity swings in IR, can they be minimized to alleviate HV problems?– Get all four air conditioners to run (most of the time)
• Flammable gas detectors– Many recalibrations or head replacements– New technology that is stable at levels being monitored?
• Fire Alarms & power shutdowns– HSSD smoke detectors?– Fix light bulb coupling to PHENIX power shutoff system
• Magnet trips – CMI mostly• Access to MCR logbooks (readonly) for PHENIX shift crews
– New xterm for acnlinxx access?• Communications between workers in IR and PHENIX control room for
debugging problems quickly and efficiently (old radios not allowed)?– POTS with long cords or wireless phones?
6/21/2007 PHENIX - MJL 27
HV trips 161 HBD,muTr,DC,PCserver/MF 10massive trips 9
Front end feed 60 muTr AMU& glink
Back end EB,BB,RC 34 EB,BBSEB,CorbaOncs Cleanup
Magnet trips 14 CMICooling Water,A/C 8 or more…Power Off Fire Alarms 3 HSSD smoke detectors
& Light bulbs
Rough Survey of PHENIX Problems in Last 2 weeks of Run7
6/21/2007 PHENIX - MJL 28
Concluding Remarks• Over 3 times Run4 luminosity (~813b/241b)
PLUS new capabilities from new detectors– E.g. 3-4 times better Reaction plane resolution– Full /K/p PID up to 9 GeV/c– major advances in our physics should result
• Analysis in progress already thanks to level-2 filtering & mature analysis model
• HBD a very state-of-the-art detector, but lessons learned in Run7 should allow full operation in Run8 & beyond
• Excellent stores when they could be delivered, but too many breakdowns– Stochastic cooling - a step towards the future– Low energy (9.2 GeV) running feasible
6/21/2007 PHENIX - MJL 29
Backup
6/21/2007 PHENIX - MJL 30
charged multiplicity in RXN (a.u.)
centrality selections are givenby BBC charged multiplicity
z-vertex selections by BBC[-40,40] cm
[-30, -20] cm[-5,5] cm
[20,30] cm
total[~500]
[500~1500][1500~]
RXN south charged mult. (a.u.)
RX
N n
ort
h c
ha
rge
d m
ult
. (a
.u.)
south (rad)
north (rad)
< c
os
2(
sou
th -
n
ort
h )
>
mid-central collisions
PHENIX Reaction Plane Detector (RXNP)200GeV Au+Au collisionsRUN7 - 2007/Mar
6/21/2007 PHENIX - MJL 31
Luminosity fraction captured by PHENIX DAQ saturation in red
Fraction of luminosity for late April stores (at bottom) 97.8% for 4.8 khz DAQ rate
• for ½ hr turn-on time
6 min 21 min 18 min 16 min 32 min 26 min
0 min
starts at10 amFri Apr 27th
6/21/2007 PHENIX - MJL 32
date time store cooled b-1/hrZdc
(pk)
FM
(x10-3)
5/15 2:10 8776 no 0.9804 34.97 28.04
5/30 2:10 8875 no 0.8789 32.07 27.41
5/30 23:50 8879 no 0.9746 34.85 27.97
5/19 14:47 8805 yes 1.1524 35.19 32.75
5/25 16:30 8849 yes 0.9874 30.34 32.54
6/1 8:00 8886 yes 1.0530 33.24 31.68
Average Improvement with SC:
03.016.135.081.27
57.032.32
Improvement in integrated luminosity for PHENIX with Stochastic Cooling (SC)
Store 8879(no cooling)
Store 8886(cooling on)
6/21/2007 PHENIX - MJL 33
T. Roser’s vertex shape
Differentiation of vertex into distinct peaks due to Stochastic Cooling
ZDC vertex (ZDC Wide)BBC vertex (BBC no vertex cut)BBC vertex (BBC ±30 cm vertex cut)
6/21/2007 PHENIX - MJL 34
HPSS To
Data flow
rftpexp
phenix
ccphenix
srm/dcache@RCF
srm/dcache10 TB
Online filtered PRDF (21TB)
DST, 7 MB/s, srmcpdccp???
sps disk4.1 TB
DB snapshot, scp or bbftp
bqsworkerssrmcp
srmcp
HPSS TB
(database server)
Input prdf flowOutput plan AOutput plan B
Aggregation not done
Offline filtered PRDF
6/21/2007 PHENIX - MJL 35
PRDFsRaw data filesGridFTP to VU30 MBytes/sec
11
33
44
FDT45
MB/s
22
Vanderbilt Farm1600 CPUs, 80 TBytes disk
PRDFs nanoDSTs
ReconstructionReconstruction200 jobs/cycle200 jobs/cycle
45TB and 200 CPUsAvailable for Run7 Reconstruction
4.4TB of buffer disk space
Dedicated GridFTP ServerDedicated GridFTP ServerFirebirdFirebird
18 hours/job
FDT45
MB/s
770 GBytesper cycle
nanoDSTsReco output to RCFGridFTP 23 MB/sec
RCFRHIC computing facility
6/21/2007 PHENIX - MJL 36
HBD Status • We believe we have determined the main cause
of the sparking problems with the both detectors:
• Normal GEM spark would cause Lecroy HV to trip• Mesh also trips leaving large stored energy on filter
capacitor• As GEM voltage goes to zero, large DV develops
across gap between top GEM and mesh, ultimately resulting in a large spark
• This spark induces sparks in other GEMs (massive trips) by propagation of scintillation light
There is however a danger in the “One source of all troubles” theory and we are still looking for more possible causes of the problems we had during the run
from T.Hemmick
6/21/2007 PHENIX - MJL 37
Pock Marks
• Human eye could see the holes.• Holes smaller than pock parks.• Pocks almost certainly from meshGEM
from T.Hemmick
6/21/2007 PHENIX - MJL 38
Low Energy Running at PHENIX• 9.19 GeV/nucleon• PHENIX RXNP trigger
– timed in at full energy– expect ~96% efficiency (compared to
~15% for BBC)– ZDC trigger should be very inefficient
due to large Fermi motion wrt longitudinal momentum
EvB
bufferbox
• Wednesday afternoon could run our DAQ with the “blue low-energy clock”• But Thursday morning, when beam was in the rings, we could not• Presumably due to glitches in the clock (associated with other clock events that were not there on Wednesday??)