online overview

1

Online OverviewOnline Overview

L. Coney – UCRL. Coney – UCR

MICE CM35 – Feb 2013MICE CM35 – Feb 2013

2Coney - CM35 - Feb 2013

Also in This Session:Also in This Session: Controls & Monitoring – PierrickControls & Monitoring – Pierrick

DAQ – Yordan DAQ – Yordan

Online MAUS – Alex RichardsOnline MAUS – Alex Richards

MICE Computing – Chris RogersMICE Computing – Chris Rogers

Will leave the specifics to them..Will leave the specifics to them..

Note: Online focus here – Operations Note: Online focus here – Operations focus tomorrowfocus tomorrow


Since CM34 in OctoberSince CM34 in October Significant events in the 4 months Significant events in the 4 months

since the October CM:since the October CM: December runDecember run Christmas breakChristmas break – Failure of AC in – Failure of AC in

PPD computing area – downtime of PPD computing area – downtime of PPD-hosted MICE PPD-hosted MICE services/computingservices/computing

Spectrometer Solenoid Controls Spectrometer Solenoid Controls ReviewReview

Restart of SS2 cooldown/testingRestart of SS2 cooldown/testing Activation run – Wednesday (13 Feb)Activation run – Wednesday (13 Feb)


December RunDecember Run Did not go smoothlyDid not go smoothly

Similar to October run issuesSimilar to October run issues Problems with DAQProblems with DAQ

Worked initially but problems developedWorked initially but problems developed Unable to solve remotelyUnable to solve remotely

Problems with C&MProblems with C&M HV control applicationsHV control applications Run ControlRun Control

Problems with Online Monitoring – goneProblems with Online Monitoring – gone

Prompted evaluation of reliability within Prompted evaluation of reliability within Online SystemsOnline Systems Need develop more robust pre-run procedures for Need develop more robust pre-run procedures for

DAQ, C&M, Online ReconstructionDAQ, C&M, Online Reconstruction Need DAQ pulser triggerNeed DAQ pulser trigger Need higher priority on documentation Need higher priority on documentation

More than one person to solve problems – must do More than one person to solve problems – must do better on handover of informationbetter on handover of information


December RunDecember Run Need fake data (more than cosmics) – test Need fake data (more than cosmics) – test

full chain of DAQ, unpacker, Online Recofull chain of DAQ, unpacker, Online Reco fake signals from TOFs? – no LED system ‘yet’fake signals from TOFs? – no LED system ‘yet’ Tracker will have LEDsTracker will have LEDs

Emphasized the need for additional Emphasized the need for additional expertiseexpertise

Personnel changes:Personnel changes: New network village manager – Chris Brew New network village manager – Chris Brew

(RAL)(RAL) New RAL network liaison – Antony Wilson (RAL)New RAL network liaison – Antony Wilson (RAL) New DAQ deputy – David Adey (FNAL)New DAQ deputy – David Adey (FNAL) New Online Monitoring owner – Rhys Gardner New Online Monitoring owner – Rhys Gardner

(Brunel grad student)(Brunel grad student) New C&M deputy – ???????????New C&M deputy – ???????????


PPD OutagePPD Outage PPD-hosted computing services loss over PPD-hosted computing services loss over

holidayholiday Loss of access to configuration database (CDB) Loss of access to configuration database (CDB)

prevented software development by US MICEprevented software development by US MICE Loss of access to micemineLoss of access to micemine

Highlighted confusion regarding these services Highlighted confusion regarding these services and how they relate to MICE activitiesand how they relate to MICE activities Including Online Group, Data-taking, FC testing, SS Including Online Group, Data-taking, FC testing, SS

testing testing

Prompted review of service loss – see Chris’ talkPrompted review of service loss – see Chris’ talk

Motivated improvement in computing Motivated improvement in computing documentation across the board – in PPD, on documentation across the board – in PPD, on micenet, hardware and servicesmicenet, hardware and services Micenet: Micenet:

http://micewww.pp.rl.ac.uk/projects/computing-software/wihttp://micewww.pp.rl.ac.uk/projects/computing-software/wiki/Micenet_Computerski/Micenet_Computers

PPD: PPD: http://micewww.pp.rl.ac.uk/projects/computing-software/wihttp://micewww.pp.rl.ac.uk/projects/computing-software/wiki/Computing_infrastructureki/Computing_infrastructure


Since CM34 cont’dSince CM34 cont’d Spectrometer Solenoid Controls ReviewSpectrometer Solenoid Controls Review

See Pierrick’s talkSee Pierrick’s talk Good feedback Good feedback improvements to the system improvements to the system Led to significant changes in priorities in Led to significant changes in priorities in

C&MC&M Knock-on effect on non-SS C&M workKnock-on effect on non-SS C&M work

Restart of SS2 magnet trainingRestart of SS2 magnet training See Pierrick’s talkSee Pierrick’s talk

Activation RunActivation Run Even with beam only to DSA – still an Even with beam only to DSA – still an

exercise of Online Systems (DAQ, C&M, exercise of Online Systems (DAQ, C&M, Online Reco)Online Reco)

Went much better than December RunWent much better than December Run


Overall OnlineOverall Online CompletedCompleted

Automated operating system updates including a Automated operating system updates including a MOM-accessible OFF switch for data-taking MOM-accessible OFF switch for data-taking (stability/performance)(stability/performance)

Spare hardware now organized in R9 (reliability)Spare hardware now organized in R9 (reliability) Computing documentation agreed go on micemine Computing documentation agreed go on micemine

(ease of operations/record keeping)(ease of operations/record keeping) Installation of additional UPSs (reliability)Installation of additional UPSs (reliability)

In progressIn progress Finalize monitoring of new UPS units Finalize monitoring of new UPS units

(reliability/stability)(reliability/stability) Installation of new Online Reconstruction machines Installation of new Online Reconstruction machines

(reliability/stability/performance)(reliability/stability/performance)

NOTE: Infrastructure largely in hand – NOTE: Infrastructure largely in hand – consistently good effort by Matt and Antonyconsistently good effort by Matt and Antony


Overall OnlineOverall Online22

DelayedDelayed Installation of new iocpc1 – 21 Dec 2012 (related to Installation of new iocpc1 – 21 Dec 2012 (related to

C&M reliability)C&M reliability) Delayed – requires Pierrick to be at RAL and to Delayed – requires Pierrick to be at RAL and to

coordinate with Matt – last visit December Run coordinate with Matt – last visit December Run Pierrick priority for trip was work on SS controlsPierrick priority for trip was work on SS controls Ran out of timeRan out of time

Computer monitoring info into Alarm Handler – 1 Computer monitoring info into Alarm Handler – 1 Feb 2013 (stability)Feb 2013 (stability)

Requires PierrickRequires Pierrick Pierrick priority is SS controlsPierrick priority is SS controls

Restriction of access to micenet – 1 Feb 2013Restriction of access to micenet – 1 Feb 2013

The link between Online Group, Software The link between Online Group, Software Group, and overall Computing has been weakGroup, and overall Computing has been weak Confusion regarding services, ownership, etc.Confusion regarding services, ownership, etc. Recent work to strengthen this areaRecent work to strengthen this area


Online Systems Online Systems Other Other ComputingComputing

MICE must be able to MICE must be able to take data without take data without connection to services or connection to services or computing external to computing external to micenetmicenet

StrengthsStrengths Use primary CDB – Use primary CDB –

located in MLCRlocated in MLCR Can store data in MLCR Can store data in MLCR

for days (so far none for days (so far none deleted from local deleted from local storage although is all storage although is all on GRID)on GRID)

Use local EPICS, Alarm Use local EPICS, Alarm Handler, and Archiver Handler, and Archiver for C&Mfor C&M

DAQ localDAQ local Online Reco uses local Online Reco uses local

MAUS installationMAUS installation

WeaknessesWeaknesses Elog – hosted on PPD – Elog – hosted on PPD –

useful but not criticaluseful but not critical Micemine – hosted on Micemine – hosted on

PPD – run plans, PPD – run plans, documentation, etc. – documentation, etc. – useful but not criticaluseful but not critical

Data Data GRID (see above) GRID (see above) External expert access – External expert access –

Archiver, EPICS gateway, Archiver, EPICS gateway, mousehole – convenience mousehole – convenience rather than necessityrather than necessity

Conclusions Conclusions Current arrangement of Current arrangement of

services works wellservices works well MICE data-taking not at MICE data-taking not at

riskrisk


Online ActivitiesOnline Activities Resurrected remote readout of neutron monitor Resurrected remote readout of neutron monitor

– Ian – Ian Updated/improved Online Reco plotsUpdated/improved Online Reco plots

TOFs – DurgaTOFs – Durga CKOVS – Gene CKOVS – Gene Online software – AlexOnline software – Alex

Major developments in C&MMajor developments in C&M Fully developed Focus Coil testingFully developed Focus Coil testing Micenet to R9 (thanks Antony!) Micenet to R9 (thanks Antony!) running as if in MLCR running as if in MLCR Transparent move (from Online Group perspective) to Transparent move (from Online Group perspective) to

MICE HallMICE Hall C&M focused on Spectrometer Solenoid C&M focused on Spectrometer Solenoid

changes in prioritieschanges in priorities C&M milestonesC&M milestones

Implement SS state machine – 1 Feb – see Pierrick’s Implement SS state machine – 1 Feb – see Pierrick’s talktalk

Full SS2 C&M – 1 Feb – see Pierrick’s talkFull SS2 C&M – 1 Feb – see Pierrick’s talk


C&M Milestones ShiftingC&M Milestones Shifting Complete Run Control for Step I elements – 21 Dec Complete Run Control for Step I elements – 21 Dec

1 March1 March Depends on data-taking – aimed at completion during December Depends on data-taking – aimed at completion during December

RunRun Depends on Pierrick available – busy on Spectrometer Solenoid Depends on Pierrick available – busy on Spectrometer Solenoid

workwork

Rack room environment monitoring plan – 1 Jan Rack room environment monitoring plan – 1 Jan 1 1 Feb Feb ? ? Require Pierrick – Pierrick busy on SS controlsRequire Pierrick – Pierrick busy on SS controls

Complete HV control user manual – 2 Feb Complete HV control user manual – 2 Feb 1 May 1 May Requires Pierrick to write documentation – Pierrick busy on SS Requires Pierrick to write documentation – Pierrick busy on SS

controls – documentation suffers – trickledown effect on controls – documentation suffers – trickledown effect on OperationsOperations

Complete Run Control manual and shifter guide – 1 Complete Run Control manual and shifter guide – 1 Jan Jan 15 April 15 April Requires Pierrick to write documentation – Pierrick busy on SS Requires Pierrick to write documentation – Pierrick busy on SS

controls – documentation suffers – again, affects Operationscontrols – documentation suffers – again, affects Operations


C&M Milestones cont’dC&M Milestones cont’d Bench test pneumatic proton absorber controls – 1 Bench test pneumatic proton absorber controls – 1

April April 1 May 1 May Requires Pierrick – busy on SS controlsRequires Pierrick – busy on SS controls

Install pneumatic proton absorber controls – 15 Install pneumatic proton absorber controls – 15 April April maybe 1 June maybe 1 June Requires Pierrick – busyRequires Pierrick – busy Requires ISIS shutdown: April 1-28 or June 17-30Requires ISIS shutdown: April 1-28 or June 17-30

Implement DS state-based Alarm Handler – 1 Feb Implement DS state-based Alarm Handler – 1 Feb 15 June15 June Priority for first state-based system shifted from DS to SSPriority for first state-based system shifted from DS to SS

New HV control – 15 Jan New HV control – 15 Jan 1 May 1 May Requires Pierrick & slow communication with CAEN – Pierrick Requires Pierrick & slow communication with CAEN – Pierrick

busybusy FC C&M review – 15 FebFC C&M review – 15 Feb

Depends on Pierrick availability and travel capability – not Depends on Pierrick availability and travel capability – not happening todayhappening today

WE NEED ANOTHER C&M PERSON!WE NEED ANOTHER C&M PERSON!


Online RecoOnline Reco Improved TOF & CKOV online plots – 21 Dec Improved TOF & CKOV online plots – 21 Dec

2012 – DONE2012 – DONE Requires data-taking with beam to completely vet new Requires data-taking with beam to completely vet new

plots – therefore tied into December Runplots – therefore tied into December Run Install new online reco machines – 1 Feb 2013Install new online reco machines – 1 Feb 2013

Likely to finish roughly on time – required new order by Likely to finish roughly on time – required new order by UniGeneve – installation now during Yordan’s MOM termUniGeneve – installation now during Yordan’s MOM term

KL online plots – 15 April 2013 KL online plots – 15 April 2013 1 June 2013 1 June 2013 No real online plot participation by KL groupNo real online plot participation by KL group Anticipate possible KL plots available after PID paper Anticipate possible KL plots available after PID paper

done?done? EMR online plots – 15 April 2013 EMR online plots – 15 April 2013 1 June 2013 1 June 2013

EMR schedule slipped – online plots follow hardware EMR schedule slipped – online plots follow hardware completioncompletion

NOTE: KL online plots is a guess – no NOTE: KL online plots is a guess – no participation = no plots = no infoparticipation = no plots = no info


By June CM36…By June CM36… Finish SS2 and SS1Finish SS2 and SS1

fully developed C&Mfully developed C&M SS state machine SS state machine

completedcompleted

Finish FC – move to HallFinish FC – move to Hall

Documentation – needs Documentation – needs higher priorityhigher priority

Improve stability & Improve stability & reliabilityreliability Complete installation of Complete installation of

iocpc1iocpc1 Complete computer Complete computer

monitoringmonitoring Stabilize C&M applicationsStabilize C&M applications Formalize development vs Formalize development vs

production C&Mproduction C&M Create comprehensive pre-Create comprehensive pre-

run test plan for DAQ and run test plan for DAQ and C&MC&M

SimplifySimplify Production version Run Production version Run

ControlControl Better limits w/in Alarm Better limits w/in Alarm

HandlerHandler Automate datamoverAutomate datamover

Improve functionalityImprove functionality Incorporate EMR into Incorporate EMR into

DAQDAQ Beam test EMRBeam test EMR Bring back Online Bring back Online

MonitoringMonitoring EMR online plotsEMR online plots Online analysis Online analysis

prototypeprototype

New C&M deputy.New C&M deputy.


Then….MICE Step IVThen….MICE Step IV Both Spectrometer Solenoids – both Both Spectrometer Solenoids – both

Trackers – one FC and an Absorber module.. Trackers – one FC and an Absorber module.. Requires robust Online systems – DAQ, Requires robust Online systems – DAQ,

C&M, Online Reco & Analysis, C&M, Online Reco & Analysis, InfrastructureInfrastructure

online overview

Documents

online focus

online systemsneed

online recowent

online overviewl

exercise of online systems

online recofake signals

antonyconey cm35

runconey cm35