enabling grids for e-science cod 19 meeting, bologna nordic rod experiences michaela lechner cod-19,...
TRANSCRIPT
Enabling Grids for E-sciencE
COD 19 meeting, Bologna
Nordic ROD experiences
Michaela Lechner
COD-19, Bologna
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 2/10
Overview
• NE ROC split into two regions (Federations)– BeNeLux (Belgium and the Netherlands) and Nordics (Denmark,
Finland, Norway, Sweden, and the Baltic states[MoU]).
• Nordic duty shifts– 6 teams, 3 from SNIC and 3 from NDGF.– http://www.egee-ne.org/internal/opdocs/NORDICroster2009 – 8 hours a day, 5 days a week coverage. Normal working time.– one team (usually one person) on duty for a week.– NDGF-T1 and EGEE operations overlap and are combined: ROD and Operator on Duty (OoD) at the same time!
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 3/10
ROD on-duty tasks
• On-duty tasks (for BeNeLux and Nordic ROD)– Perform all 1st line support and ROD tasks as defined in the 1st
line support and ROD operations manual:https://twiki.cern.ch/twiki/bin/view/EGEE/OperationalProceduresforRegionalCODAndInteractionsWithCCOD
– Watch other monitoring tools for T1 (Nagios, ganglia). (=OoD)– Use ROD dashboard tools– Assist sites where appropriate.
• Weekly duties (for Nordic OoD)– Handover– Fill out NDGF-T1 production report– Be present at WLCG weekly operation meeting next Mon 16:00– Prepare notes to weekly meeting on Friday 10:00
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 4/10
Nordic ROD communication
• Communication– All on-duty support via chat(s) and mailing list(s).– Mailing list for all NE ROC operators. The ROC mailing list:
[email protected] is archived under https://www.hpc2n.umu.se/mailman/private/roc/
– 1st line support over [email protected] – Shift handover via [email protected] mailing list.– XMPP (Jabber) chat: [email protected] – Contacts for sites and site admins are available on:
https://portal.nordu.net/display/ndgfwiki/Operation-Procedures – Communication through C-COD (Vera, Luuk) if necessary.– #cic-on-duty IRC chat @testbed011.cnaf.infn.it – Regular weekly status meetings, one each for EGEE (Thursday)
and NDGF (Friday) operations.
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 5/10
Nordic ROD knowledge sharing
• Short term– Mailing lists– Handover procedure– NDGF Chat– “None of the duties should be done acting alone”
• Longer term– Dashboard tools and notes therein. Notes: time and determination– GGUS: solutions are public and available. – NDGF wiki for OoD work– 1st line support over [email protected] (archived)
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 6/10
Daily Duties: Preliminaries
Global Grid User Support (GGUS) Overview:https://gus.fzk.de/ws/ticket_search.php?&supportunit=ROC_North&status=open&timeframe=no
CIC Portal:https://cic.gridops.org/index.php?section=roc&page=dashboard 1. You have to have a regional role in the GOCDB 2. Don't forget to set your preferences to one of these
regional roles, not site Administrator.
Log into the COD IRC chat
Change your nick in the NDGF chat eg. /nick caela(ROD, OoD)
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 7/10
Duties in detail: The Dashboard
• Home tab– overview fresh alarms, open tickets
• Ticket tab– React on colored tickets
• Dashboard tab– Actual work is done here– Set off alarms, when problems solved– read/write notes– Creating tickets or extract mail adresses used for 1st line support– Write informal mails
• Experience:– Nice with glide over and links, but slow in many ways– Not all tickets shown in ticket tab?– Creating tickets easier, administration fine
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 8/10
Bugs
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 9/10
When it burns (ASGC-DC)
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 10/10
Going for a cup of coffee
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 11/10
Nordic helpdesk solutions
• Regional helpdesk– BeNeLux have own ticketing system, but nothing similar
implemented for our Nordic sites, besides [email protected]
• Avoiding to talk with yourself– Being 1st line support, ROD and helpdesk all at the same time?– Too much responsibility on one person. – Furthermore one persons knowledge is not infinite: waiting one
week for new ideas?– Solution: Drawing more knowledge in (NDGF chat, mailinglists)
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 12/10
Nordic ROD workload
• Takes 2-3 days to get friendly with system– Hard to get an overview of new tools, new sites (Baltics)– Doing many things simultaneously– Too little time dedicated to real problem solving– Dashboard hideously slow (a feature, not a bug!)• Should definitely not be done at the same time as
TPM!
Enabling Grids for E-sciencE
COD 19 meeting, Bologna 13/10
Conclusion
• Found our own good practice procedures :-)• Dashboard
– On the way to become a great administrative tool, but beware not to forget real support
– Preferred sites option is of great help