d0 farms 1 d0 run ii farms m. diesburg, b.alcorn, j.bakken, t.dawson, d.fagan, j.fromm, k.genser,...
TRANSCRIPT
![Page 1: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/1.jpg)
D0 Farms 1
D0 Farms
D0 Run II Farms
M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,
L.Loebel-Carpenter, I.Mandrichenko, C.Moore, S.Naymola, A.Moibenko, D.Petravick, M.Przybycien, H.Schellman, K.Shepelak, I.Terekhov,
S.Timm, J.Trumbo, S.Veseli, M.Vranicar, R.Wellner, S.White, V.White
![Page 2: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/2.jpg)
D0 Farms 2
D0 Farms
D0 Farm needs
• 250K event size• 50Hz trigger rate
– peak rate of 12.5 MB/sec– DC is less but reprocessing will bring back up
• Reconstruction 5- 20 seconds/event on 750 MHz PIII
– need 250->>500 CPU’s to handle peak rate– DC is 40% of peak – time constant for 1 GB file is 5- 10 hours.
RealityGoal
![Page 3: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/3.jpg)
D0 Farms 3
D0 Farms
![Page 4: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/4.jpg)
D0 Farms 4
D0 Farms
I/O machine• Purpose
– split/merge of farm output
– Serve home areas– Batch system control– File delivery master
• D0bbin– 4 CPU SGI 02000– 2 GB ethernet cards– 4 72 GB disk partitions
(2 way stripe)– peak I/O rates of 40-60
MB/sec
![Page 5: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/5.jpg)
D0 Farms 5
D0 Farms
Worker Nodes
• 40 Dual Pentium III 500MHz– 256MB/CPU
• 50 Dual Pentium III 750MHz– 512 MB/CPU
• 2 data disks (18 GB) + 6GB system
• 100Mb ethernet• CD/floppy for system
configuration
![Page 6: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/6.jpg)
D0 Farms 6
D0 Farms
Design Principles
• Use existing facilities– SAM/Enstore for data access and file tracking– Farm batch system (FBS) for most job control
• Keep D0 farm control scripts to a minimum– Batch system assigns machines– Data access system decides which file you get
• If worker process or machine dies, lose minimal number of files and don’t affect other processes
• No heroic recovery measures, track and resubmit those files
![Page 7: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/7.jpg)
D0 Farms 7
D0 Farms
Worker Configuration
• Workers act as generic FNAL farm machines– Only customization is pnfs for file delivery, home
area mount and startup of sam daemons on reboot.– D0 code environment downloads at job start– data access through SAM/encp/rcp, database server
• Batch system assigns workers to job, not D0FARM control process.
• D0FARM control never knows which workers are assigned to a job and does not need to.
![Page 8: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/8.jpg)
D0 Farms 8
D0 Farms
Data Access is SAM/enstore
• Integrated data handling system
• File and process data base
• Data base server• File servers• Enstore File delivery
systems• Pnfs file system
Farm Perspective
Can tell it you want a set of files
Can ask for the ‘next’ file
Can flag file as processed or error
Can get detailed accounting on what happened
Data transfers are from ~ 12 mover nodes to 90 farm nodes through 6509 switch – theoretically could move 100’s of MB/sec
Reality – online system has priority for drives.
![Page 9: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/9.jpg)
D0 Farms 9
D0 Farms
Farm Batch SystemTypical Farm Job
SECTION START
EXEC=startjob parameters
QUEUE=D0bbin
SECTION WORKER
EXEC=runjob parameters
NWORKERS=20
QUEUE=D0worker
SECTION END
EXEC=stopjob parameters
QUEUE=D0bbin
DEPEND WORKER(done)
• Queue tells the system what kind of machine to run on and how many.
• EXEC gives the script name and parameters
• DEPEND allows cleanup section to run when all worker sections are done.
• FBS assigns temporary disk on workers
• On end yanks disk and kills all processes.
![Page 10: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/10.jpg)
D0 Farms 10
D0 Farms
• START:– Tell SAM which files you will want– Go into wait state until get end signal
• WORKER: runs on N nodes– Download D0 environment– Inform SAM ready for data– Ask for SAM for next file– Process file and store output to output buffer– Inform SAM of success and ask for next file– On error or end of list, terminate.
• END: – Create job summary– Send message to Start process telling it to shut down the
SAM connection for input
Structure of a Farm Job
![Page 11: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/11.jpg)
D0 Farms 11
D0 Farms
50% of dual CPU
100% of dual
Farm Batch System Monitor
Efficient use of of CPU
![Page 12: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/12.jpg)
D0 Farms 12
D0 Farms
Query to see which input files were processed by a job
Query to see which input files were processed by a job
![Page 13: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/13.jpg)
D0 Farms 13
D0 Farms
Status• System has been in use for MC processing since before
CHEP 2000• System has been processing data as it comes off the D0
detector since March 2001• Hardware/control/monitoring can handle full data rates
well but…• Major problem is speed of executable and data
expansion during detector debugging– Output size is ~ input size by design– Currently factor of 2-3 larger due to debugging info.– Better thresholds and less noise will make life much easier
• Farms get more stress at beginning of run than later!!
![Page 14: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/14.jpg)
D0 Farms 14
D0 Farms
Results of typical farm startup
24.5 MB/sec
• Cold start of ½ of the D0 farm.
• 90 receiver nodes• 141 files of average size
376 MB• Read from 2-3 network
mounted Mammoth II tapes over 100 MB ethernet at ~10MB/sec/drive.
• Elapsed time of 44 minutes.
• This is twice peak rate from the detector.
![Page 15: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/15.jpg)
D0 Farms 15
D0 Farms
Current Production
Plot from FBSWWW product – out of the box
http://www-isd.fnal.gov/cgi-bin/fbsng/fbswww/fbswww?action=graphs&period=week&farm=D0
![Page 16: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/16.jpg)
D0 Farms 16
D0 Farms
W e candidateRun 125232 Event 183666Electron candidate recorded using EM triggerpT = 38 GeV isolation<0.2 EM fraction = 0.97
Pierre Petroff and Laurent Duflot
Layer 18 GeV
Layer 214 GeV
Layer 315 GeV
Layer 40.3 GeV
hot cell
![Page 17: D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649e965503460f94b9a112/html5/thumbnails/17.jpg)
D0 Farms 17
D0 Farms
Future
• It works now! but we will still:• Add ~100 more nodes over next 6 months• Make Improvements in automated running
Datasets currently defined and submitted by hand~ .25 FTE but still too much
• Local caching of filesGuarantee tape streams at full speedDon’t waste tape mounts if process file multiple times
1060204 %d0%bb%d0%b8%d1%81%d1%82%d0%be%d0%b2%d0%ba%d0%b0 %d0%bf%d0%be %d1%82%d0%b5%d0%bf%d0%bb%d0%be
%d0%9d%d0%b0%d0%b2%d1%87%d0%b0%d0%bb%d1%8c%d0%bd%d0%be %d0%bc%d0%b5%d1%82%d0%be%d0%b4%d0%b8%d1%87%d0