GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, johannes ebke, tariq mahmoud automating of job submission in Athena/AthenaMC outline I. GangaTasks, motivation and overview II. news III. conclusion, to do tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC I. motivation started in summer 07: 1) need a tool which automates job (re)submission 2) need a tool which accounts for job chaining (interconnection) first report in october 07 tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC I. overview (motivation, mc) simul:5 recon:10 evgen:50 tariq mahmoud atlas sw week, cern, need automate job (re)submission need job chaining (interconnection) automating of job submission in Athena/AthenaMC I. overview (motivation, mc) tariq mahmoud atlas sw week, cern, complete dataset A,B,C: f1,f2,f3, ,fn A,B,C f1,f2,f3 A,B,C f4,f5,f6 A,B,C f7,f8,f9 A,B,C f4,f5,f6 A,B,C f7,f8,f9 A,B,C fn-2,fn-1,fn incomplete dataset A: f1,f2,f3 B: f1,f5,f6,f7,f8,f9 C: f1,f2,f7,f8,f9,f10 A,C f1,f2 B f5,f6 A f3 B,C f7,f8,f9 C f10 automating of job submission in Athena/AthenaMC I. overview (motivation, analysis) tariq mahmoud atlas sw week, cern, task few parameters: task parameters: float, name, number of events etc. process parameters: datasets, process name, geometry etc. input/output specifications Athena version AbstractJob generated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, AbstractJob job Ganga jobs automating of job submission in Athena/AthenaMC I. overview (concept) tariq mahmoud atlas sw week, cern, MCTask MCJob AnaTask AnaJob Task AnaJob repository/tools.data TaskList(GangaObject ) GPI.tasks automating of job submission in Athena/AthenaMC I. overview (concept) tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC V. conclusion GangaTask package: automates job management (small scale production system) works with Ganga Athena and Ganga AthenaMC. wiki in progress: https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTasks To do: 1. gather the largest number of files possible of an in-complete dataset. 2. test everything on Lxplus and elsewhere. 3. soon available in one of the next in Ganga releases automating of job submission in Athena/AthenaMC II. news https://twiki.cern.ch/twiki/bin/view/Atla/ - GangaTasks (explanation, gen. tutorial) - AnaTask (tutorial) 1.site-test-code implemented 2.more Ganga properties: EventView, Output dataset, Excluding sites/files, 3.act on user suggestions. 1.Ganga-site-index. 2.tested at cern and gridka (KFZ) 3.MCTasks: implemented since Ganga AnaTasks soon in Ganga (wget_ tgz) tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC II. news (control functions) 1.instructions, what to do in case of warnings and errors. 2.most attributes of GangaTasks are immutable. more control functions: extend/exclude sites/files 3.do not change used files: job option & user area pause task. 4.many other features tariq mahmoud atlas sw week, cern, safe analysis automating of job submission in Athena/AthenaMC V. conclusion GangaTask package: works with Ganga Athena and Ganga AthenaMC at several places. Two wikis: general and AnaTask more immutabilities to save jobs after submitting. more controlling functions. To do: 1. change site if job fails. 2. making use of GangaRobot 3. acting on users responds tariq mahmoud atlas sw week, cern, automating of job submission in Athena/AthenaMC III. illustration: chaining task steps mc.total_events = 200; mc.events_per_job={"evgen": 50, "simul": 5, "recon":10} evgen:i-0 simul:i-j recon:i-k mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10 AbstractJob.prerequisites() automating of job submission in Athena/AthenaMC III. illustration : interconnections, prerequisites automate job (re)submission job chaining (interconnection) the user sets few parameters and gets RESULTS