U OklahomaU Oklahomaand theand the

Open Science GridOpen Science GridHenry Neeman, Horst Severini,Chris Franklin, Josh Alexander

University of OklahomaCondor Week 2008, University of Wisconsin, Wednesday April 30 2008

OU and the OSGCondor Week 2008, Wed Apr 30 2008 2

Outline

Context: OU Hardware OU and the Open Science Grid Oklahoma Cyberinfrastructure Initiative OU NSF CI-TEAM Project

Context:OU Hardware

OU and the OSGCondor Week 2008, Wed Apr 30 2008 4

1,024 Pentium4 Xeon CPUs (“Irwindale” 3.2 GHz EM64T)

2,176 GB RAM (800 MHz FSB)23,000 GB disk (NFS+Lustre)Infiniband & Gigabit EthernetOS: Red Hat Linux Enterp 4Peak speed: 6,553.6 GFLOPs

Pentium4 Xeon Cluster

topdawg.oscer.ou.edutopdawg.oscer.ou.edu

OU and the OSGCondor Week 2008, Wed Apr 30 2008 5

Pentium4 Xeon Cluster

topdawg.oscer.ou.edutopdawg.oscer.ou.edu

DEBUTED 6/2005 #54 WORLDWIDE, #9 AMONG US UNIVERSITIES, #4 EXCLUDING BIG NSF CENTERS

CURRENTLY #289 WORLDWIDE, #29 AMONG US UNIVERSITIES, #20 EXCLUDING BIG 4 NSF CENTERS

www.top500.org

OU and the OSGCondor Week 2008, Wed Apr 30 2008 6

Condor Pool @ OU

OU IT has deployed a large Condor pool (775 desktop PCs in dozens of labs around campus).

OU’s Condor pool provides a huge amount of computing power – more than OSCER’s big cluster – in terms of Condorized PCs:

if OU were a state, we’d be the 12th largest state in the US;

if OU were a country, we’d be the 10th largest country in the world (other than “unknown”).

Also, we’ve been seeing empirically that lab PCs are available for Condor jobs about 80% of the time.

OU and the OSGCondor Week 2008, Wed Apr 30 2008 7

Proposed New Hardware

The following information is public. RFP: issued March 19, closed April 3 Architecture described:

quad core, dual socket, x86-64 ~5 times as fast as topdawg (similar compute node count) ~4 times as much RAM (probably 1333 MHz FSB) ~4 times as much disk (~100 TB) high performance interconnect (probably IB) plus GigE Red Hat Enterprise Linux 5 (assumed)

OU Board of Regents meets May 8-9.

OU andthe Open Science Grid

OU and the OSGCondor Week 2008, Wed Apr 30 2008 9

OU and the Open Science Grid Currently, OU’s relationship with the OSG primarily

benefits High Energy Physics: D0 project ATLAS project DOSAR project: cross disciplinary grid organization with members

in other OSG VOs (including D0 and ATLAS) We have 5 OSG resources:

OSCER’s large cluster (topdawg) – general purpose OSCER’s Condor pool (currently D0 only) – general purpose OUHEP’s Tier2 cluster – dedicated to HEP projects OUHEP’s desktop cluster – dedicated to HEP projects OUHEP’s OSG Integration TestBed (ITB): 8 nodes, used to test

new pre-production OSG releases. We recently installed OSG 0.9.0 and bestman-xrootd (a Storage Resource Manager) as part of the integration effort.

OU and the OSGCondor Week 2008, Wed Apr 30 2008 10

OU and D0

12/26/06 - 12/26/07 Events Data

#1 Michigan State U 33,677,505 2.81 TB

#2 U Oklahoma#2 U Oklahoma 16,516,50016,516,500 1.32 TB1.32 TB

#3 U Florida 13,002,028 1.07 TB

#4 UC San Diego 10,270,250 0.81 TB

#5 U Nebraska 8,956,899 0.71 TB

#6 Indiana U 4,111,740 0.35 TB

#7 U Wisconsin 3,796,497 0.30 TB

#8 Louisiana Tech U 3,224,405 0.25 TB

#9 Langston U (OK) 1,574,062 0.11 TB

OU and the OSGCondor Week 2008, Wed Apr 30 2008 11

OU D0 Breakdown OSCER’s big cluster (topdawg)

8,020,250 events (6th in the US), 0.66 TB OSCER Condor pool

6,024,000 events (6th in the US), 0.49 TB Dedicated OU HEP Tier3 cluster

2,472,250 events (9th in the US), 0.16 TBNotes: Without OSCER’s Condor pool, OU would be #4. Without OSCER’s cluster, OU would be #6. Without OU HEP’s dedicated Tier3 cluster, OU would still

be #2.

OU and the OSGCondor Week 2008, Wed Apr 30 2008 12

OU and ATLAS

4/4/2007 – 4/27/2008 Wallclock Hours

#1 Boston U 325,700

#2 U Chicago 297,600

#3 Indiana U 235,400

#4 Michigan State U 170,000

#5 UT Arlington 160,300

#6 U Oklahoma#6 U Oklahoma 145,700http://gratia-osg.fnal.gov:8880/gratia-reporting/

Note: A buggy version of gratia ran on OU’s resources until 4/3/2008.

OU and the OSGCondor Week 2008, Wed Apr 30 2008 13

OU: First in the World

OU was the first institution in the world to simultaneously run ATLAS and D0 grid production jobs on a general-purpose, multi-user cluster.

Most grid production jobs run on dedicated clusters that are reserved for one or the other of these projects, or on Condor pools.

OU and the OSGCondor Week 2008, Wed Apr 30 2008 14

OU’s Collaborations

OU plays key roles in: Oklahoma Center for High Energy Physics (OCHEP)

Collaboration between OU, Oklahoma State U and Langston U (HBU) Funded by a Dept of Energy EPSCoR grant

ATLAS Southwest Tier2: OU, Langston U, U Texas Arlington DOSAR (Distributed Organization for Scientific and

Academic Research)

OU, Langston U, U Arizona, Iowa State, Kansas State, U Kansas, Louisiana Tech, Louisiana State, Rice, U Mississippi, U Texas Arlington, Universidade Estadual Paulista (Brazil), Cinvestav (Mexico)

OU and the OSGCondor Week 2008, Wed Apr 30 2008 15

OU Helps with Condor

OU has helped set up Windows/coLinux/Fedora Condor pools at:

Oklahoma State U U Texas Arlington

Oklahoma Cyberinfrastructure

Initiative

OU and the OSGCondor Week 2008, Wed Apr 30 2008 17

OK Cyberinfrastructure Initiative Oklahoma is an EPSCoR state. Oklahoma recently submitted an NSF EPSCoR Research Infrastructure

Proposal (up to $15M). This year, for the first time, all NSF EPSCoR RII proposals MUST include

a statewide Cyberinfrastructure plan. Oklahoma’s plan – the Oklahoma Cyberinfrastructure Initiative (OCII) –

involves: all academic institutions in the state are eligible to sign up for free use of OU’s

and Oklahoma State U’s centrally-owned CI resources; other kinds of institutions (government, NGO, commercial) are eligible to use,

though not necessarily for free. OCII includes building a Condor flock between OU (775 PCs) and OSU

(~300 PCs). We’ve already helped OSU set up their Condor pool; they just need to roll out the deployment, and then we’ll be able to use it for HEP/OSG.

OU’s NSF CI-TEAM Project

OU and the OSGCondor Week 2008, Wed Apr 30 2008 19

OU’s NSF CI-TEAM Project

OU recently received a grant from the National Science Foundation’s Cyberinfrastructure Training, Education, Advancement, and Mentoring for Our 21st Century Workforce (CI-TEAM) program.

Objectives: Provide Condor resources to the national community Teach users to use Condor and sysadmins to deploy and

administer it Teach bioinformatics students to use BLAST over Condor

OU and the OSGCondor Week 2008, Wed Apr 30 2008 20

OU NSF CI-TEAM Project

teach students and faculty to use FREE Condor middleware, stealing computing time on idle PCs;

teach system administrators to deploy and maintain Condor on PCs;

teach bioinformatics students to use BLAST on Condor;

provide Condor Cyberinfrastructure to the national community (FREE).

Condor pool of 775 desktop PCs (already part of the Open Science Grid);

Supercomputing in Plain English workshops via videoconferencing;

Cyberinfrastructure rounds (consulting) via videoconferencing;

drop-in CDs for installing full-featured Condor on a Windows PC (Cyberinfrastructure for FREE);

sysadmin consulting for installing and maintaining Condor on desktop PCs.

OU’s team includes: High School, Minority Serving, 2-year, 4-year, masters-granting; 18 of the 32 institutions are in 8 EPSCoR states (AR, DE, KS, ND, NE, NM, OK, WV).

Cyberinfrastructure Education for Bioinformatics and Beyond

Objectives: OU will provide:

OU and the OSGCondor Week 2008, Wed Apr 30 2008 21

OU NSF CI-TEAM ProjectParticipants at OU(29 faculty/staff in 16 depts) Information Technology

OSCER: Neeman (PI) College of Arts & Sciences

Botany & Microbiology: Conway, Wren Chemistry & Biochemistry: Roe (Co-PI),

Wheeler Mathematics: White Physics & Astronomy: Kao, Severini (Co-PI),

Skubic, Strauss Zoology: Ray

College of Earth & Energy Sarkeys Energy Center: Chesnokov

College of Engineering Aerospace & Mechanical Engr: Striz Chemical, Biological & Materials Engr:

Papavassiliou Civil Engr & Environmental Science: Vieux Computer Science: Dhall, Fagg, Hougen,

Lakshmivarahan, McGovern, Radhakrishnan Electrical & Computer Engr: Cruz, Todd,

Yeary, Yu Industrial Engr: Trafalis

OU Health Sciences Center, Oklahoma City Biochemistry & Molecular Biology: Zlotnick Radiological Sciences: Wu (Co-PI) Surgery: Gusev

Participants at other institutions(62 faculty/staff at 31 institutions in 18 states)1. California State U Pomona (masters-granting, minority serving): Lee2. Colorado State U: Kalkhan3. Contra Costa College (CA, 2-year, minority serving): Murphy4. Delaware State U (masters, EPSCoR): Lin, Mulik, Multnovic, Pokrajac, Rasamny5. Earlham College (IN, bachelors): Peck6. East Central U (OK, masters, EPSCoR): Crittell,Ferdinand, Myers, Walker, Weirick,

Williams7. Emporia State U (KS, masters-granting, EPSCoR): Ballester, Pheatt8. Harvard U (MA): King9. Kansas State U (EPSCoR): Andresen, Monaco10. Langston U (OK, masters, minority serving, EPSCoR): Snow, Tadesse11. Longwood U (VA, masters): Talaiver12. Marshall U (WV, masters, EPSCoR): Richards13. Navajo Technical College (NM, 2-year, tribal, EPSCoR): Ribble14. Oklahoma Baptist U (bachelors, EPSCoR): Chen, Jett, Jordan15. Oklahoma Medical Research Foundation (EPSCoR): Wren16. Oklahoma School of Science & Mathematics (high school, EPSCoR): Samadzadeh17. Purdue U (IN): Chaubey18. Riverside Community College (CA, 2-year): Smith19. St. Cloud State University (MN, masters): J. Herath, S. Herath, Guster20. St. Gregory’s U (OK, 4-year, EPSCoR): Meyer21. Southwestern Oklahoma State U (masters, EPSCoR, tribal): Linder, Moseley, Pereira22. Syracuse U (NY): Stanton23. Texas A&M U-Corpus Christi (masters): Scherger24. U Arkansas Fayetteville (EPSCoR): Apon25. U Arkansas Little Rock (masters, EPSCoR): Hall, Jennings, Ramaswamy26. U Central Oklahoma (masters-granting, EPSCoR): Lemley, Wilson27. U Illinois Urbana-Champaign: Wang28. U Kansas (EPSCoR): Bishop, Cheung, Harris, Ryan29. U Nebraska-Lincoln (EPSCoR): Swanson30. U North Dakota (EPSCoR): Bergstrom, Hoffman, Majidi, Moreno, Peterson,

Simmons, Wiggen, Zhou31. U Northern Iowa (masters-granting): Gray

E

E E

E

OU and the OSGCondor Week 2008, Wed Apr 30 2008 22

Okla. Supercomputing Symposium

2006 Keynote:Dan Atkins

Head of NSF’sOffice ofCyber-

infrastructure

2004 Keynote:Sangtae KimNSF Shared

CyberinfrastructureDivision Director

2003 Keynote:Peter Freeman

NSFComputer & InformationScience &

EngineeringAssistant Director

2005 Keynote:Walt Brooks

NASA AdvancedSupercomputingDivision Director

http://symposium2008.oscer.ou.edu/

2007 Keynote:Jay Boisseau

DirectorTexas Advanced

Computing CenterU. Texas Austin

Tue Oct 7 2008 @ OUOver 225 registrations already!Over 150 in the first day, over 200 in the first week, over 225 in the first month.

FREE! Parallel Computing Workshop Mon Oct 6 @ OU

FREE! Symposium Tue Oct 7 @ OU

2008 Keynote: José Munoz

Deputy Office Director/ Senior

Scientific Advisor Office of Cyber-

infrastructure National Science

Foundation

OU and the OSGCondor Week 2008, Wed Apr 30 2008 23

To Learn More about OSCER

http://www.oscer.ou.edu/

Thanks for your attention!

Questions?