tutorial for shifter at the mdt calibration centers

Tutorial for shifter at the

MDT Calibration Centers

Elena Solfaroli CamillocciUniversità La Sapienza & INFN Roma

11/03/2011 Tutorial for shifter - Elena Solfaroli Camillocci 2

What has the shifter to do?For each one of the three Calibration Centers:• Check the correct running of the Local Calibration Data Splitter

(LCDS).• Monitor the transfer of the data from the Calibration Stream.• Start the Data Quality (DQ) analysis and the calibration fits for

interesting runs.• Monitor the correct processing of each operation.• Alert the expert in case of anomaly.• Report the DQ results and store the global flags in the ATLAS

database.• Validate the calibration results.

Compare and merge the results from the three sites with the help of the expert on shift.

Present the DQA results at the “Muon Daily Operations and Data Quality Meeting” at 15:10.

NEW


Before to start the shift at the MDT Calibration Centers, the shifter has to Have a GRID certificate

– installed in a browser;– DN registered in the LCDS to enable the shifter access;– installed to provide a valid proxy.

Belong to the e-group atlas-dataQuality-automatic-notifications to receive the daily list of interesting runs to be analyzed

a. Go to https://e-groups.cern.ch ;b. Search e-group name contains: “atlas-dataQuality-automatic-notifications”;c. Click on subscribe.

Book the shift on OTP http://atlas-otp.cern.ch/ Task 529188 - MUON CALIBRATION CENTER Shifter(Contact Fabrizio Petrucci to be added to the shifter list).

Check the connection to the web interfacesRome LCDS https://atlas-splitter.roma1.infn.it:9613Munich LCDS https://lcg-lrz-muon.grid.lrz-muenchen.de:9613/Michigan LCDS https://splitter.aglt2.org:9613/

Read the NEW twiki for shifters available at https://twiki.cern.ch/twiki/bin/viewauth/Atlas/AtlasMdtCalibShift

NEW


My OTP page

The night time slot is more suitable for people sitting in the US.

HOW TO BOOK THE SHIFT

3 shifters per day6-14 morning14-22 afternoon22-6 night

NEW


Let’s start a shift

Check the status of the proxy. Read previous elog messages and

the name of the expert on shift. Register your shift.


1) Go to https://atlas-splitter-test.roma1.infn.it:9613/ (version for tutorial)2) Check the proxy validity3) Click on Documentation to open the menu

https://atlas-splitter-test.roma1.infn.it:9613/

https://atlas-splitter-test.roma1.infn.it:9613/


4) Click on calibration elog


5) Read the messages left by the previous shifter.6) Check the name of the expert on shift during the week.7) Click on New and register your shift.8) Report each operation you will do as shifter.

NEW


Calibration stream

Check data transfer. Monitor dataset splitting and

ntuple production.


1. Click on General (in LCDS tools) to open the menu.2. Click on datasets


3. Compare dataset list with last 10 runs stored, seeing the database run query.

4. Check the data transfer to Calibration Center browsing the link replication status.

3

4


Runs closed at CERN have to be transferred to the Calibration Centers.


5. Check the status of the runs at the Tier2 (repeat after 2 hours). Datasets are sorted by date.

DISCOVER = dataset scanning,WAIT_FOR_DATA = dataset discovered but no data in DDM wait some

minutes, then alert the expert,SPLITTING = dataset splitting in calibration regions,SPLIT = operation concluded.

6. Check the ntuples:n_Files/5 x n_Regions(204) = n_FragmentsAll ntuples READY = production completed.(move the mouse on READY to show more information on the ntuple status)

7. Click on READY or on ntuples in the LCDS tools menu.


7. Click on READY or click on ntuples. To select a dataset, a detailed search is available in the bar on the top

(i.e. search a run number: ‘*150419*’ and select max records per page >204).

8. Check the status of each ntuple (one per calibration region)PENDING = wait for running alert the experts after many minutes,CREATING = ntuple creation is running,FAILED = failure of the ntuple creation try to recreate it, READY = ntuple created, ready for running DQA and calibration,ADVISORY_FIT = calibration job submitted,FIT_FAILED = calibration failed,FITTED = calibration completed.

(per calibration region)

*150419*

10000


9. Check the DQA StatusREADY = ntuple ready for DQA analysis,DISABLE = user disabled this ntuple for DQA analysis,ADVISORY_DQA_CREATE = DQA job submitted,DQA_CREATE_FAILED = DQA failed check the details browsing the DQA menu,DQA_CREATE_DONE = DQ analysis correctly completed.

Shifter can download the ntuple file, clicking on the floppy icon.

(per calibration region)


Data Quality Assessment

Submit a DQA job. Monitor the job processing. Provide a report.


NEW

HOW TO CHANGE FILE FOR DQ REFERENCE

ONLY IF IT IS REQUESTED BY THE EXPERT

Open the MDT DQA/DQMF menu- Click on dqa create.BEFORE TO SUBMIT THE DQ JOB1. Click on “here” to change the

reference file for the data quality assessment.

2. Choose the reference file, clicking on Set Reference.

3. Check the new reference name.

1

2

3


Open the MDT DQA/DQMF menu1. Click on dqa create.2. “Click+Ctrl” on the runs to be

analyzed (Default: last run) and click on the button Select.

3. Select (all or someone) the spectrometer regions to be analyzed.

4. Start the DQ with the button on the bottom.

Only one set of runs can be processed per time.

1

2

3

4

Runs to be analyzed are those in the daily mail notification.


1. Click on dqa histo.2. Click on a set of runs to see the details of the operation status.


3. Check the operation status for each spectrometer regionCREATING = processing;FAILED = the histogram filling failed investigate the log file and report to the expert;READY = histograms filled, waiting for histogram file merging;MERGING = merging the histogram files for each spectrometer region (BA, BC, EA, EC);MERGE_FAILED = crash in the histogram merging try to re-submit this job;MERGE_DONE = histogram filling and merging well done.

Shifter can download the histogram file. To investigate job crash, click on the status to see the job view and click on the job status to

see the log file.


1. Click on dqa merge.2. Click on a set of runs to see the details of the operation status.3. Check the status of each regions (BARRELA/C, ENDCAPA/C)

MERGING = merging the histogram files of the same spectrometer region,MERGE_FAILED = job failed try to re-submit this job,MERGE_DONE = operation completed.

4 regions(so far)


1. Click on dqa finalize.2. Click on a set of runs to see the details of the operation status.3. Check the status of each spectrometer regions (BA, BC, EA, EC)

FINALIZING = running the DQ histogram analysis;FINALIZE_FAILED = DQ histogram analysis failed investigate the log file;FINALIZE_DONE = MdtDqa analysis done, start automatically the DQMF analysis;

To investigate job crash: click on status to see the job view and click on the job status to see the log file of the DQA job. Then report the crash reason to the expert.


1. Click on dqmf2. Check the quality flag for each spectrometer regions (BA, BC, EA, EC).3. Click on the set of runs to see the details.


4. Check the status for each region (BA, BC, EA, EC)PROCESSING = running of DQMF FAILED = DQMF checks failed investigate the log file and report to the expertDONE = DQA completed

Shifter can download the dqmf output file.


5. Click on the button “Start Macro”, to produce and publish the report file.pdf in the DQA web page (at Rome http://classis01.roma1.infn.it/dqmf/dqmf.php)

NEW = waiting for the other regions,READY = click the button to submit the macro,ADVISORY_REPORT = macro submitted,PROCESSING = macro is running,FAILED = report failed try to resubmit, then alert the expert,DONE = report.pdf published.

6. Click on dqmf summary


Some histograms already in the DQMF web display, to show a global vision of the muon spectrometer.

DQA report .pdf

Endcap

Barr

el


http://classis01.roma1.infn.it/dqmf/dqmf.php

DQA report .pdf

7. Click on the balloon to browse the DQMF web display.8. Provide a report for the afternoon shifter.

Afternoon shifter has to attend the “Muon Daily Operations and Data Quality Meeting” at 15:10 and present the DQ results of those runs listed in the daily notification mail. (At the beginning the shifter will be supported by the expert on shift).


Run Number ReferenceDQA

ReferenceFIT

#events on calib stream

BA BC EA EC

DQA - Macro Occupancy percentage (IMO)

Average Efficiencies (IMO)

Overall Flags

DQMF New Dead MLs

Red chambers 1

bad ADC spectra

Red chambers 2

bad TDC spectra

Red chambers 3

dead / low eff.

FIT check <t0>

RMS (t0)

<tdrift>

RMS (tdrift)

T0 jumps

(if any)

RT max variation (m)

AOB

Template for report NEW


Check-list for DQA

a. Check the global hit maps and report the percentage of active multilayers for each muon station (Inner, Middle, Outer) in BA, BC, EA, EC.

b. Check the efficiency distributions: alert if the average efficiency is <99%.

c. Report the global flags on BA, BC, EA, EC.

d. Browse the dqmf web dispaly and report red alarms on:

i. Dead multilayers;

ii. TDC and ADC spectra;

iii. Low efficient or dead channels.

NEW


a. Percentage of active MultilayersPage 3 from report.pdf BARREL

NEW


a. Percentage of active MultilayersPage 4 from report.pdf ENDCAPS

NEW


b. Average efficienciesPage 10 from report.pdf BARREL

<99%

<99%

NEW


b. Average efficienciesPage 11 from report.pdf ENDCAPS

NEW


c. Global flags

Green Detector ON > 90%Red Detector ON < 90%Grey Low statistics


d. DQMF web display

1. Click on Barrel_A (or Barrel_C, Endcap_A/C);2. Click on MDT;3. Click on AllSectors

See this note for details on histograms and DQ assessment“The MDT data quality assessment at the Calibration Centers”ATL-MUON-INT-2011-001


Barrel_C/MDT/AllSectors

INNER MIDDLE OUTER

i. Check the hit maps with RED frame: HitsPerML_XXXX

Click on the histogram to see the details.


Barrel_A/MDT/AllSectors//A_HitsPerML_Inner

Detector alive (%):

Already knownSector 6 Eta 1

NEW DEADSector 14 Eta 6

Read the NEW_ML_OFF and check the chamber status inside the sector folder.


Barrel_A/MDT/Sector01/BML2A01/Chamber/A_TDC_ML1

ii. Report RED flags on TDC (or ADC) spectrum of the MDT multilayers.

ERROR on tdrift:

ERROR on t0:


Barrel_C/MDT/Sector03/BML2C03/DeadStatus

Multilayer 1

Multilayer 2

Layer 1 Layer 2

Layer 3 Layer 1 Layer 2

Layer 3


iii. Report RED flags on Dead or Low Efficient elements.


Barrel_C/MDT/Sector03/BML2C03/DeadStatus/DeadTubes_ML1_L2

DeadTube:

Calibration

Submit a Calibration job. Monitor the job processing. Check the calibration results. Compare them with the previous

calibration results.


Open the MDT Fits menu1. Click on fittables.2. “Click+Ctrl” on the

runs to be fitted (Default: last 20 runs in the list) and click on the button select.

3. Select the calibration regions for the analysis.

4. Start the calibration with the button on the bottom.

1

2 3

HOW TO CHOOSE RUNS FOR CALIBRATION#events >30M. The shifter can also decide which run is good according to the DQA.

4


1. Click on fits.2. Click on the set of runs to see the operation details.3. Check the status of each calibration region

FITTING = calibration is running,FAILED = calibration failed alert the expert if failures are >10%,DONE = calibration completed.

You can download the file on your pc. Click on status to see the job view and click on the job status to see the log file.


4. Click on calibdb observer


1. Choose the database **ROME_NEW** and click on the white arrow;2. Check that there is an entry for the last calibrated run;3. Click on t0 and rt.


Check the t0 results alert if large regions failed.


Check the rt results and count the failed regions.


Template for report NEW

Run Number ReferenceDQA

ReferenceFIT

#events on calib stream

BA BC EA EC

DQA - Macro Occupancy percentage (IMO)

Average Efficiencies (IMO)

Overall Flags

DQMF New Dead MLs

Red chambers 1

bad ADC spectra

Red chambers 2

bad TDC spectra

Red chambers 3

dead / low eff.

FIT check <t0>

RMS (t0)

<tdrift>

RMS (tdrift)

T0 jumps

(if any)

RT max variation (m)

AOB


Check-list for FIT

a. Check the t0 and tdrift distributions and report mean and RMS of the differences between last and reference calibrations.

b. Find t0-jump.

c. Report the max RT variation with respect to the reference calibration.

NEW


Check Last Calibration: a tool to see the results of the last calibration done, comparing it with previous results. You can:

modify the parameters and re-submit the script using “Change”; re-submit the script with the same parameters in case the DB is being updated

using “Refresh”

parameters

NEW

In the Calibration DB menu1. Click on “check last

calibration” button;2. Wait 3-4 minutes;3. A dialogue menu and a

list of histograms will appear.


Description of the histograms shownNEW

Sect-1: Presentation of the results of the last calibration.– Lines 1 to 4: distributions of t0s and tdrifts for last calibration and their dependence

on the sector. – Line 5: (left) distribution of the number of events for each fit; (right) distribution of

the rising edge slopes from the fits. – Line 6: (left) t0 uncertainty from the fit as a function of the number of events;

(right) tdrift uncertainty from the fit as a function of the number of events. Sect-2: Presentation of the difference btw last and reference calibration: t0s and tdrifts.

– Lines 7 to 10: distributions of the differences of t0s and tdrifts between last calibration and reference calibration.

– Lines 11 and 12: t0 and tdrift differences between last and reference calibrations as a function of the station index (0=BIL, 1=BIS, 2=BMS, 3=BML, 4=BOS, 5=BOL, …).

– Lines 13 to 24: t0 differences between last and reference calibrations for each station type as a function of the chamber index. (chamber index = 10 x phi + eta)

Sect-3: Presentation of the difference btw last and reference calibration: RT relations.

– Lines 25 and 26: RT relations from last calibration – Lines 27 to 29: difference between last and reference RT relations, mean (line

28) and width (line 29).


Sect-1: Presentation of the results of the last calibration

NEW

t0tdrift t0

tdrift

<t0> <tdrift > <t0> <tdrift >

BARREL ENDCAPS

#events Rising edge slope

vs sector

(t0) (tdrift)vs #events


Station Index code0 BIL1 BIS2 BML3 BMS4 BOL5 BOS7 BIR8 BMF9 BOF10 BOG12 BIM

14 EIL18 EML19 EMS21 EOL22 EOS23 EIS

NEWSect-2: Presentation of the difference btw last and reference calibration: t0s and tdrifts

Δt0

Δtdrift

BARREL ENDCAPSΔt0

Δtdrift

Δt0

Δtdrift

vs

Report mean and RMS of Δtdrift and Δt0


t0 variations chamber by chamber. This is the place to look for “t0 jumps”.

NEW

chamber index = 10xphi+eta


Some examples of 12.5 ns t0-jumps:

NEW


Sect-3: Presentation of the difference btw last and reference calibration: RT relations

NEWRT

BARREL ENDCAPS BARREL ENDCAPS

RT comparisons

Mean

Sigma


At the end of the shift Provide a DQA report for the

meeting (all the shifters). Attend the meeting and present the

results (afternoon shifter). Store the DQ flags in the database.


For those runs notified from theatlas-dataQuality-automatic-notifications: 1. The afternoon shifter presents the DQ analysis results at the

“Muon Daily Operations and Data Quality Meeting” at 15:10 (details of evo connection in agenda).

During the weekend, this meeting is replaced by a sharepoint signoff. Please, report any problems on sharepoint, links to the e-log and/or interesting histograms:

https://espace.cern.ch/atlas-dq/Lists/Team%20Discussion/AllItems.aspx

i. Joining the hn-atlas-DataQuality e-group is needed to post to this page.

ii.Sign up at https://e-groups.cern.ch or https://groups.cern.ch/Pages/default.aspx

NEW


For those runs notified from the atlas-dataQuality-automatic-notifications:

2. Fill flags of DQA analysis at Calibration Centers in the

ATLAS COOL DQA Database These flags should be considered as the overall result from the

Calibration Centers (they will be taken into account by the Muon offline DQ Shifter for the final decision).

►The flags MUST BE ISSUED BY the shifter Green Detector ON > 90%

Red Detector ON < 90%

Yellow Software errors, flag could become green after reprocessing

Grey Low statistics

the “four regions” flags provided by DQMF are a good suggestion.


a. Go to https://atlasdqm.web.cern.ch/atlasdqm/DQBrowser/DBQuery.php and insert your NICE username and password.

b. Set the Subsystem as “Muons”.c. Insert the run number.

d. Set the Database info as “COMP200 MUON CALIB”

e. Click on Display info, a new page will be opened with field for all the muon spectrometer regions of all the Muon detectors.

f. Click on each MDT region (MDTBA, MDTBC, MDTEA, MDTEC), set the DQA flag and click on Save change, password “Muons”.

For those runs notified from the atlas-dataQuality-automatic-notifications:

ATLAS COOL DQA Database

https://atlasdqm.web.cern.ch/atlasdqm/DQBrowser/DBQuery.php

https://atlasdqm.web.cern.ch/atlasdqm/DQBrowser/DBQuery.php


Experts for the Rome Calibration Center

[email protected]

• LCDS– Alessandro De Salvo [email protected]

• DQA– Mauro Iodice [email protected]

– Cesare Bini [email protected]

– Elena Solfaroli [email protected]

• CALIBRATION– Fabrizio Petrucci [email protected]

– Domizia Orestano [email protected]

Expert people in Rome

Backup

Some slides for simple trouble shooting


• Login on the Tier2 as calib (e.g. for Rome)> ssh -l calib atlas-ui.roma1.infn.it> ssh atlas-splitter> cd LCDS/> source setup.sh

• Check the proxy validity> voms_proxy_info

• Stop and restart the LCDS> splitter stop ( could be time demanding)> splitter start> logout

HOW TO RESTART THE LCDS


• Login on the Tier2 (e.g. for Rome) :> ssh atlas-ui.roma1.infn.it

• From your home with your grid certificate installed, provide your proxy and copy it in the calib home> voms_proxy_init -voms atlas -valid 96:0 -out $HOME/proxy

> scp $HOME/proxy calib@atlas-splitter:./LCDS/secret

HOW TO LOAD A NEW PROXY FOR THE LCDS


• Login on the Tier2 as calib (e.g. for Rome)> ssh -l calib atlas-ui.roma1.infn.it> ssh atlas-splitter> cd LCDS/> source setup.sh

• For problems on DQA job submission> unlock_dqa.py

• For problems on FIT job submission> unlock_fit.py

HOW TO FIX DQA or FIT PROBLEMS


A ntuple is not available for DQA analysis or calibration fit, if• it is FAILED it has to be unavailable (try to recreate it);• it has been already analyzed.

In the second case, the shifter can force the ntuple to be available for DQA/calibration again(i.e. to add this ntuple to others of different runs to increase the statistics).

a. Go to the ntuples page;b. Select the ntuples, that you want to force;c. Click the buttons on the bottom:

• DELETE = remove the ntuple• ENABLE = force the ntuple to be available for the calibration• DISABLE = force the ntuple to be unavailable for the calibration• RECREATE = create the ntuple again• ENABLE DQA= force the ntuple to be available for the DQA• DISABLE DQA = force the ntuple to be unavailable for the DQA

HOW TO MANAGE THE NTUPLES


a. Click on jobs.b. Check the Job type and its status.c. In case of anomaly, click on its status to investigate problems in the log file

and report to the expert.d. If many jobs are PENDING for a long time (>15 minutes), pause the job

submission with the button on the top and alert the expert.

HOW TO MONITOR THE JOBS


In case of merging failure:a. Go to the dqa merge page;b. Select the spectrometer region, that you want to re-merge;c. Click the buttons on the bottom:

HOW TO RE-SUBMIT THE HISTOGRAM MERGING


Click on the world link to browse the DQMF web display.

HOW TO BROWSE THE DQMF WEB DISPLAY


DQMF web display

1. Click on Barrel_A (or Barrel_C, Endcap_A/C);2. Click on MDT;3. Click on AllSectors

See this note for details on histograms and DQ assessment“The MDT data quality assessment at the Calibration Centers”ATL-MUON-INT-2011-001


Barrel_C/MDT/AllSectors

INNER MIDDLE OUTER

a. Check the hit map of the MDT chambers: HitsPerML_XXXX



Barrel_C/MDT/Sector03//AllChambers



Barrel_C/MDT/Sector03//AllChambers/HitsPerML_Inner

5. Report the OFF multilayer.

Multilayer OFF:

tutorial for shifter at the mdt calibration centers

Documents