from compute intensive to data intensive grid computing

10
Miron Livny Computer Sciences Department University of Wisconsin-Madison [email protected] http://www.cs.wisc.edu/~miron From Compute Intensive to Data Intensive Grid Computing

Upload: keziah

Post on 12-Jan-2016

43 views

Category:

Documents


0 download

DESCRIPTION

From Compute Intensive to Data Intensive Grid Computing. Welcome to the 2 nd Annual Condor meeting!!! (Hope to see you all in the EuroGlobus+Condor meeting in southern Italy in mid June). The Condor Project (Established ‘85). - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: From Compute Intensive to Data Intensive  Grid Computing

Miron LivnyComputer Sciences DepartmentUniversity of Wisconsin-Madison

[email protected]://www.cs.wisc.edu/~miron

From Compute Intensive

to Data Intensive Grid Computing

Page 2: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

Welcome to the 2nd Annual Condor

meeting!!!

(Hope to see you all in the EuroGlobus+Condor meeting in southern

Italy in mid June)

Page 3: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

The Condor Project (Established ‘85)

Distributed Computing research performed by a team of 26 faculty, full time staff and students who

face software engineering challenges in a UNIX/Linux/NT environment,

are involved in national and international collaborations,

actively interact with academic and commercial users, maintain and support a large distributed production

environment, and educate and train students.

Funding - DoD, DoE, NASA, NSF, AT&T,IBM, INTEL, Microsoft and the UW Graduate School

Page 4: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

State of the project› Extended the scope and size of the project› Extremely busy› Defended and preserved our unique approach

to distributed/Grid computing› Enjoyed the impact we have on “real” users› Maintained the reputation of our research and

the quality of our technology› Worked hard to make a commercial Condor fly

alongside our project

Page 5: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

The Physicists & Us › Some of us have our roots in physics (Derek, Doug, Miron, Raj)› Long history of joint work (NIKHF 91,INFN 96, PPDG 99, SMC,

WA92 COROUS, PHINEX, CMS …) › Grid Physics Network (GriPhyN) project funded by the

Information Technology Research program of NSF ($10.9M for five years $2M to our group)

› Joint UW-HEP and Condor project funded by the UW Graduate School.

› 1M hours of Condor CPUs were allocated by the National Resource Allocation Board (NREC) to the CMS group at Caltech

› Our team selected as the main CS group for the next phase of PPDG (DOE-SciDAC proposal pending)

› CMS group of UW is one of the four sites of the Grid Laboratory of Wisconsin (GLOW) (NSF-MRI proposal pending)

Page 6: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

Physicists Have a

LOT

of data. (others have data problems too …)

Page 7: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

The Data Grids & Us

› Extend and enhance existing capabilities of Condor› Develop new storage and data transfer management

technologies (key element in PPDG proposal)› Interface “legacy” applications and the Grid “fabric”

with new capabilities/technologies› Work closely with the Globus team and follow their

leadership on data transfer and replica management tools (joint Data Grid SciDAC proposal is pending)

› Close collaboration with the EU Data Grid effort› Make ClassAds the “Esperanto” of Grid management

(joint Security SciDAC proposal with Globus is pending)

Page 8: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

And what about …

› Allocating Network Resources,› Scheduling of Storage Resources,› Co-allocation of Processing, Storage and

Network Resources,› Resource Reservation,› Data Shipping vs. Function Shipping,› Data Security,› Write Behind,› Data Affinity,› Or … ?

Page 9: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

Challenges Ahead

› Ride the “Grid Wave” without losing our balance

› Leverage the talent and expertise of our new faculty (Distributed I/O, Distributed Scheduling, Networking, Security)

› Expend our UW-Flock to a state-wide system (WiscNet?)

› Apply the Master Worker paradigm to domain decomposition problems (ongoing work with JPL)

› Scale our Master Worker framework to 10,000 workers.

› Open Source vs. Public Domain binaries vs. a Commercial version of Condor

Page 10: From Compute Intensive to Data Intensive  Grid Computing

www.cs.wisc.edu/condor

Don’t ask “what can the Grid do for

me?”ask “what can

I do with the

Grid?”