24/02/2011muon barrel workshop1 gianni masetti & ignacio redondo (and others) shutdown...

24/02/2011 Muon Barrel Workshop 1

Gianni Masetti & Ignacio Redondo (and others)

Shutdown interventions and operation in 2011


Outline• Operations in 2011

– New oncall support structure– Comment on Interfill Cosmics

• DocDB interlude• Attemp to summarize shutdown interventions

– Everything touched,…but MCs & chambers– Several crucial soft points reinforced.– 2011 DT is a stronger system

• Thanks to everyone involved


Operations in 2011• Special runs in first fills under discussion (only RPC in trigger for

instance) • New oncall support scheme: 1. DT DOC (Detector On Call) is our first defense line:

– 1 responsible per week– Reports in DT weekly meeting/daily run meeting– Earns central credit –points (21) – and 0.25 ManMonth of ESP for the training week?– More exposure (4 weeks min) and longer shift periods (1-2 week)– First discussion session held yesterday– Practical trainings to be organized– Population: 10/15 old DT oncall; 2/15 old DTFM; 3/15 newcomers

2. Infrastructure oncall support down to minimum (2)3. DT expert cloud


Le XV de DT (DOCs 2011) Zoltan Szillasi

Noemi Beni Mario Pelliccioni Marco Dallavalle

Daniele Fasanella Jesus Puerta Pelayo

Daniel Dominguez Vazquez Juan Pablo Fernandez

Alicia CalderonShan-Huei Chuang

Isidro González Caballero Massimo Nespolo

Paolo Bellan Antonio BrancaDaniel Teyser


DT interfill Operations

• No Heavy Stable Particle data this year• Remaining HV ON ~1 h after the fill dump was and is

not a problem• Probably cosmics

(What flavour of cosmics?)(How many?)

• Expectacion is interfill time will narrow down• System operation will need (part of) this time for

mantainance – MC/Mezzanine reconfiguration– System changes tests– Routine Test pulse runs– …


CMS DocDB: (Yet another) CMS database CMS DocDB: (Yet another) CMS database

DB documental dropbox: designed to host any kind of documents except:•Talks already included in Indico•Engineering docs (EDMS)•Official papers (CDS)

Any other document can (must?) be uploaded to DocDB

Open to any CMS member (NICE login)

Still preliminary version (some issues with permissions)

https://cms-docdb.cern.ch/cgi-bin/DocDB/DocumentDatabase

Jesus Puerta


A single «document» can contain several files

Any entry is updateable a posteriori (versioning)

Uploaded document data to be included:TitleDescriptionPrivilegesTopics/Tags (to be chosen from a predefined list)

Several tags for same document can be used. Documents can be browsed

by tags/topics!

Any document or list of docs worth being kept, feel free to give DocDB a try.

Let Jesús know about any other tag structure

Preliminary list of DT-related tags(Cristina/Alberto/Jesús)

Jesus Puerta


Readout/Control– New VME PC crate control board: Linco

• SC power cycle required vmepc reboot• Sometimes Linco Driver reload results in blocked PC • More frequent after general PC reinstallation by system

administrators• Exchange two damange fibres helps• Need long term monitoring• SC power cycle does not require to reboot PCs ! • Still have two remember to hit 2 buttons in two programs

– Thinking how to integrate this

– 10 5 DDU configuration• Minor FW upgrade• Spares!• As a subproduct, get faster configuration (30 %)

A. Triossi et al.

V. Monaco et al.


ROS FW upgradeVME FPGA cannot be remotely programmed: delicate intervention

Main changes:

GOL does not power cycle on each configuration (to minimize stress), it will only power cycle at power boot (or at demand with expert stand alone program)

Maximum Event Size register can be reseted (manually and with a TTC command):

TTC BGo Channel 12 (command 0x30) performs Event Size reset, so we can choose the frequency we want to monitor max event size (once per orbit, etc)

TTC Hard reset implemented (BGo Channel 6, command 0x18):– Reload the firmware of CEROS and ROSCTRL FPGAs (radiation SEE)– Drawbacks:

• HU &TO resynch indep registers will give error after reload• ROS configuration will be erased (if something comes as mandatory it

will have to be hardcoded in the firmware)

DAQ software has been changed to make it faster (~50 % )

Cristina Fernandez Bedoya


New MC Secondary link

• Granularity increased from half wheel to one sector (maximum achievable)

• Each sector chain completely independent (in the old board only 4 rs485 ports are available)

• FW remotely upgradable• the old board was upgraded adding some protection

circuitry, this circuitry will be integrated in the new one• Fibre connectors inside the crate, so more protected• Added an 1-wire bus for linco monitoring• More robust link to USC: rs422 via linco (fast), 1 fiber

compatible mode (same as now), 1 fiber fast mode• Power supply independent from trg/ro crate

Franco Gonella


Primary link fiber 5 sec3(new fast mode)

UXCUSC

PIC32 Ethernet Starter KitWith optical interface

Ethernetconnection

DCS PCs rack

Primary link fiber 5 sec2(compatible mode)

DCS fibers rack

DCS in0 YB+1

USB

I connector

Secondary link board prototype installed on YB+1 TRG/RO upper crate

(the old board was left in site for quickly restoring in case of problems)

newold

Franco Gonella

SC crate•the prototype seemsto work properly•10 boards (+ 5 spares) will be ready at the end of the year


Trigger

• See Session with specific talks• Janos installed new VME Control FW in DTTF crates

– Common effort: DT/DTTF, Bolonga/Padova (linco was very helpful), others,…

– Enables optoRX monitoring– Enables automatic link synchronization

• New crate tested (and removed) in sectors #1 # 2• New Barrel Sorter FW (under test)• New Wedge sorter FW (under test)• DT technical trigger (to be commisioned)


Infrastructure

• Power Supply system• HV• LV

• Gas

24/02/2011 Muon Barrel Workshop 14Alberto Benvenuti

Chambers with Problems Since January 24 HV ON

All problems but for YB+2 MB2 S03 SL2 L2 occurred at HV ON Six chambers had problems at HV ON in 2010 and were quiet afterwards. Most probably

due to A877 vagaries In addition the A876 for YB0 S09 failed to turn ON and was replaced No HV problems occurred during HV ON from STANDBY after ramp to 3.8 T


• Mapping of the smoke detectors DSS alarms was checked and several wrong paths were corrected. It was not easy but now the map is correct for all tower racks. (D. Fasanella)

•Several temperature sensors were replaced by David and Cristina, all sensors are now operational

•The grounding of the cable braids was checked and standardized at all tower racks (Franco,Barcellan):

• Braids connected to ground for FE LV cables• All MC LV cables are floating

Maintenance Report (NOT a complete list)


One major incident:Oct 5 post-mortem

• During fill T sensor in turbine rack (wheel/4) fails, triggering DSS interlock. • Unfortunately, safety system was wrongly cabled, switching off wrong

rack, bringing down wheel -2• Problem understood (and short-term fixed) before the end of the fill.

Interlock expert had to drive in (1/2h).• Final fix done and tested during last technical stop.

• ~1 h to change conf to get CMS running (without wheel -2).– Good part of it because we needed trigger to change configuration accordingly– Prompt DT response: The “right” experts where involved.– Some time lost because power cycling crates involve rebooting control PCs to

load control drivers, sometimes PCs block at load.» Upgrade control boards this Christmas shutdown so no reboot is required.

– Would all DT experts have been equally fast ?• Documentation written• Need to spread the ability to change configuration among the DT FM pool

Reminder

Zurich Run Coordination Workshop


LV VCON errors

• Several cable connectors were checked/replaced that showed anomalous temperatures at the A3009 modules

– No clear defects (L. Barcellan) were visible

• Several LV modules have been exchanged thanks to David and Daniele:

– MAO for YB0 S10, A3100 for YB0 Top Sector Collector– A3050 for YB0 MB2 S10– NOT obvious what to do with these modules since there is no

specific fault apart for repeated VCON VCC/VDD errors.• All connectors have been lubricated but those in

YB0+SCs (already done in summer 2010)

Alberto Benvenuti


30°

25°

New LV Connector failuresThe temperature of one A3100 connector powering YB+2 SC 1-7 jumped to 32° and then dropped to 27°Checking the temperature after power cycle it dropped a bit. There was another sudden jump above 30° on 18-feb and then it dropped to a normal value on 17-feb

YB0 MB4L S10 Vcon from February 1 to 20

This episode occurred all by itself.The Voltage is still higher than it should be albeit below the alarm threshold

Drops when “touching” PS Positive polarity connector(failure not inside the crate)


YB-1 MB3 S12 Vcon ErrorNew LV Connector

failuresFirst alarm 21-feb at 22:52 MC switched OFF at 0:42. Problem persists after MC power up.UX access needed to fix it.

Problems are no new in the sense that the pattern is the same as last year (positive polarity is warmer through an increase of resisitivity) It is a bit unsettling that there are still these problems after changing cables, reinforcing contacts, adding lubricant….

Maybe we are seeing effects due to the magnetic field, we could try to block the cables as was done before the connector exchange

At any rate, I suspect we have to live with this, since changing the connectors on the A3050 looks like mission impossible

Alberto Benvenuti


Gas Analysis

A rather painful restart:•Software updates blocked the data storage/acquisition•The refresh rate of the measurements was changed and the time stamp is not useful anymore, still to be fixed•O2 Analysis cells damaged by leak (will be replaced ASAP)•H2O analysis cells to be replaced with a new model (proto in YB0)•Current O2 measurements are 20 to 50 ppm lower than in 2010 probably due to analysis cells•The system is still leaking 1000 liters/hour, ongoing since July

Issues were discussed with gas group (Haider, Guida…) suggestions for a more efficient operation should be implemented ASAP

Online gas chromatograph should be operational by end of march.

Alberto Benvenuti


DT shutdown highlights• Readout/Control

– New VME PC crate control board: LincoNo need to reboot PCs after crate Power cycle

– 10->5 DDU configurationSpares!

• As a subproduct get faster configuration (30 %)– New ROS FW (delicate intervention)

FW reloaded at TTCHardReset• As a subproduct get faster configuration (50 %)

– Prototype Secondary link to MC installed • Trigger (see session with specific talks)

– Janos installed new VME Control FW in DTTF crates • Common effort: DT/DTTF, Bologna/Padova (linco was very helpful), others

Enables optoRX monitoringEnables automatic link synchronization

• Infrastructure: – Extensive DSS tests

Several mapping errors corrected– HV– LV

All connectors lubricated– Gas

VDC system installed (following talk)


The End

24/02/2011 Muon Barrel Workshop 23Franco Gonella

sec1

sec2sec3

sec4sec5

sec6

TSB backup link

sec1

sec2

sec3

sec4

sec5

sec6

Foreseen mapping

node1

node2

node3

node4

node5

node6

node7

Present mapping

Fiber1 (compatible mode)

Fiber2 (new fast mode)RS4221-Wire

RS422 VME crate Primary link 5 sec2

Primary link 5 sec3

PIC

Protection circuitry

PIC

PIC

PIC

PIC

PIC

PIC

DS1820 temp sensor

PIC

PIC

PIC

Can bus

node0

node8

node9

RS422 link 230400 bps

Optical link 230400 bps

Optical link 38400 bps

485port0

485port1

485port2

485port3

485port4

485port5

485port6

Prototype of the new Secondary Board installed on YB+1

With this mapping the board is fullycompatible with the old one

-the prototype seemsto work properly

-10 boards (+ 5 spares)will be ready at theend of the year

24/02/2011muon barrel workshop1 gianni masetti & ignacio redondo (and others) shutdown...

Documents

dt docs

old dt oncall

dt expert cloudle

lincosc power cycle

dt doc detector

power boot

interfill time

blocked pc