streamlining cdf shift operations peter wilson mary convery cdf detector operations all...

Streamlining CDF Shift Operations

Peter Wilson

Mary Convery

CDF Detector Operations

All Experimenters Meeting

February 26, 2007

Peter Wilson - All Expt Meet

February 26, 2007 slide - 2

Challenge: collect high quality data with high efficiency as the pool of physicists shrinks! LHC will turn on in FY2008 Many people working on CDF also on ATLAS or CMS Expect fewer Postdocs and students in 2008/2009

However, we don’t need as many people as in 2001: Experiment is mature Systems are well understood Procedures are well defined Upgrades/changes are minimal

Effort initiated by previous Ops heads (Willis Sakumoto and Pat Lukens) to re-organize shift operations to use fewer people while maintaining or increasing operational efficiency

Running CDF thru FY2009



CDF Shift Crew - Ops Man

Operations Manager (Ops Man) - On call 24/7 Makes daily operational plan Typically experienced postdoc - 1 wk on/2 wks off for 6 months

CDF Control Room



CDF Shift Crew - SciCoScientific Coordinator (SciCo) - 8/7 x 3shifts Leader of the shift crew Responsible for implementing daily plan Communicate with MCR Summon expert help Provide food for the crew! Faculty, lab scientific, staff or senior postdoc

CDF Control Room



CDF Shift Crew - DAQ Ace

DAQ ACE - 8/7 x 3 shifts “Runs” the experiment Controls the data acquisition system

Setup run configurations Start/stop DAQ runs

Monitor DAQ status, diagnose and fix DAQ problems Run calibrations

CDF Control Room



CDF Shift Crew - Mon AceMonitoring ACE - 8/7 x 3 shifts Detector HV monitoring and control. Diagnose/fix HV problems Monitor status of low voltage power supplies Responsible for safety of Si in the event of a cooling alarm Monitor beam conditions for detector safety Twelve Students and postdocs cover DAQ & Monitoring Shifts in 10 week rotation

CDF Control Room



CDF Shift Crew - COConsumer Operator - 8/7 x 3 shifts Data quality monitoring for all detector subsystems Ten “Consumers” each receiving a few Hz (of ~100Hz) of events from L3

Detector hit occupancies and average pulse-heights in minimum bias data Trigger rates and trigger algorithm validation (compare emulation to data) DAQ performance and errors

Compare results to references Check all routine detector calibrations Any CDF physicist

CDF Control Room



CDF Shift Crew - CryoProcess System (Cryo) Operator - 12/7 x 2 shifts Solenoid and Cryogenic System operation Gas systems Cooling systems for electronics and detectors Electronics rack protection Four PPD technicians on rotation, one other tech fills in for sick leave/vacations

CDF Cryo Control Room (1st Floor)



Streamlining the Shift Crew

Consumer Operator Shifts taken remotely Pilot operation at INFN-Pisa Reduce need for travel for overseas members of CDF “OWL” shift is 7:00-15:00 in Pisa, Italy or 15:00-23:00 in

Tsukuba, Japan

Reduce the number of Aces from 2 to 1 Consolidate responsibilities with the remaining ACE Transfer some monitoring tasks to Cryo Operator SciCo trained on some tasks of monitoring Silicon detector

- Safety of detector if Ace is not in control room (eg high beam losses)

Automate tasks to allow shift crew to focus on decision making



Remote CO Effort started early in 2006:

All monitoring plots available via Web Tools for remote starting and stopping of Consumer programs CO shift room in Pisa: monitoring workstations and video conferencing CDF Pisa Video conference for SciCo-CO communication Remote CO does all of the same functions as local CO Experience shared with and gained from US CMS remote operations Team from Fermilab, Pisa, Tsukuba, UCLA and Waseda led by Kaori Maeshima

and Fabrizio Scuri First official remote shift: week of November 10

Commissioning in October 2006: simultaneous Remote and local CO Through June 2007: schedule 1week/month of remote CO from Pisa Easy to extend to other locations: planning to start shifts in Japan starting July 2007 Goal: all OWL shifts covered remotely in 2008-2009



Single ACE Operation Planning started in Summer 2006 Implementation starting mid-October

Re-arranged control room computer consoles- Added ACNET and HV control next to DAQ consoles

- DAQ and Monitor functions from one work area Automated recovery of Silicon HV crate communication failures, Ace

previously left the control room (Hockerization Auto-Hockerization) Automated generation of Hourly ACNET plots of beam losses Updated monitoring pages and documentation for Silicon Cooling and

LV PS monitoring Cryo Operator Add audible alarm for Silicon detector PS current monitoring SciCo Improved SciCO training Team from Silicon detector group, CDF Ops, CDF DAQ and

Accelerator (Ron Moore and Brian Hedricks) led by operations heads First shift with single Ace Dec 29

Followed 10 days where 2nd Ace acted only as backup



CDF Owl Shift - Feb 15

SciCo - John Yoh

CO in Pisa (Simone Donati)

Ace (Michal Kreps)



Owl Shift in Pisa - Feb 15



Efficiency High with Reduced Crew

Monthly average good runs w/ Si



Experience with Reduced Shift Crew

No significant problems with shift crew changes, biggest challenges have been substitutions for shifters who are ill! New Years weekend needed to find substitute Ace Remote CO needed to leave after 6 hours of shift

Operational efficiency has not dropped With single Ace, crew is operating even more as team Experience from three weeks of remote CO: as effective as

local CO Automation is reducing downtime:

Recovery of Si HV Hockerization took 10-15min Auto-Hockerization now takes about 4 mintues Typically about once per store



Additional Improvements Coming

In case of dangerous beam conditions, detector HV must be set to Stand-By: CDF TEVMON tool monitors losses, abort gap beam, TevRF, etc

based on ACNET variables, warns shift crew of dangerous conditions If shift crew is occupied might be delayed in responding New TEVMON based on an Open Access Client (OAC) by Ron Moore

- ACNET variables indicate danger level and provide permit for operation of detectors

- Automatically set Silicon HV to standby in dangerous conditions Final testing underway

HV problems sometimes require Ace to leave control room: Plug calorimeter HV regularly requires procedure to re-establish

communications move control PC and interface hardware from 3rd floor to control room (2nd flr)

Other HV systems need to have programs restarted or PCs to be rebooted on 1st floor working on remote access to HV PCs from control room



Some Key Members of the Team



Members of the TeamThank you to the many people who helped make this happen: Fermilab: Bill Badgett, Kurt Biery, Ray Culbertson, Steve Hahn, Brian

Hendricks, Jonathan Lewis, Kaori Maeshima, Ron Moore, JJ Schmidt, Peter Wilson

CDF Technical support team: Dervin Allen, Del Allspach, Jim Loskot, Lew Morris, Bill Noe, George Wyatt

Duke: Dean Hidas INFN Pisa: Dario Fabiani, Luciano Ristori, Fabrizio Scuri, Jose Enrique-

Garcia Tsukuba: Taichi Kubo, Koji Nakamura Waseda: Junji Naganoma, Tetsuo Arisawa Rochester: Jennifer Gimmel, Willis Sakumoto Rockefeller: Mary Convery UCLA: Charles Plager Yale: Ulrich Husemann



Summary CDF has successfully reduced the shift crew from 4 physicists

to 3. 25% of Owl shifts have only 2 at B0

Process of reducing shift crew has improved the tools for shift takers

Operational efficiency has been maintained or improved during the transition Additional improvements are coming which should result in more

efficient shift operation We are working on other improvements to increase our

operational efficiency and prepare for a smaller physicist support For example, adapting Electronic Worklists from AD for scheduling

accesses and tests

streamlining cdf shift operations peter wilson mary convery cdf detector operations all...

Documents