beyond big grid
DESCRIPTION
Beyond BiG Grid. Dr.ir. Anwar Osseyran SARA National High Performance Computing Services. BiG Grid was a Must. ATLAS Computing requirements over time 1995: 100 TB, 107 MIPS : 2001: 1900 TB, 7*107 MIPS : 2007: 70000 TB, 55*107 MIPS 2010 LHC START 2011: 83000 TB, 61*107 MIPS - PowerPoint PPT PresentationTRANSCRIPT
Big Grid and Beyond – Amsterdam 120926
Beyond BiG Grid
Dr.ir. Anwar OsseyranSARA National High Performance Computing Services
Big Grid and Beyond – Amsterdam 120926
BiG Grid was a Must
ATLAS Computing requirements over time
1995: 100 TB, 107 MIPS : 2001: 1900 TB, 7*107 MIPS : 2007: 70000 TB, 55*107 MIPS 2010 LHC START 2011: 83000 TB, 61*107 MIPS
From the start on it was clear that no center could provide ALL computing even for one LHC experiment
Similar needs in life science, Astronomy, …
15%40%45%
Big Grid and Beyond – Amsterdam 120926
BiG Grid: a National Effort
Partner agreement between BiG Grid and SARA already in 2008
National Grid infrastructure (Nikhef, RUG, SARA, Philips, …)
To support: LHC: Dutch Tier-1 LOFAR: Long Term Archive Life Science: Dutch LSG But also Alfa and Gamma
Sciences
Co-Development and Support
Big Grid and Beyond – Amsterdam 120926
BiG Grid: Big Effort
104 resource requests (35 cloud) projects (and counting !)More then 30 scientific communities served
From High Energy Physics: 1000s scientists
To Dutch Bio Banking Collaboration
SARA effort 06/2007 – 12/2012: Operations: 93k hours Support and Development: 31k hours BiG Grid Investment in HPC eco-system:
contribution to Supercomputer Huygens Grid Compute & Storage, Cluster
computing, Clouds, Big Data
Big Grid and Beyond – Amsterdam 120926
BiG Grid @SARAUser-Centric Innovation
Rapid prototyping and Proof-of-Concept in pilots with concrete use cases (small nr. of users)
Pre-production with limited number of users
Production for larger user community
3 examples: HPC Cloud Beehub storage for science Hadoop Big Data services
Started by SARA, early adopted and co-funded by BiG Grid
Big Grid and Beyond – Amsterdam 120926
HPC Cloud services in NL
Dynamic and flexible self-service
Define your own working environment
Define your own HPC cluster
Work loads: scale up workstation loads scale up complex work
flows
High User Productivity
Big Grid and Beyond – Amsterdam 120926
BeeHub: Storage for scientists
Integral scalable cloud storage service
Easy to use, store and share data, based on WebDAV
For all desktops OS’s & distributed computing infra
Integrated with SURFconext
Big Grid and Beyond – Amsterdam 120926
Hadoop Big Data Services
Open source Apache Hadoop along with a framework for map reduce jobs over data
Move processing to the data
Seamless scalability
2009: Pilot Hadoop on Cloud
2010: Test cluster available for scientists
2011: Funding for production servicesPoC: 20 users
2012: Production started with 72 machines * 8 cores / 8 TB storage / 64GB RAM, 3 devops, team of consultants
Lot of interest from science and business
Big Grid and Beyond – Amsterdam 120926
Life after BiG Grid…
BiG Grid the project ends this year
(SURF)SARA is committed to continue its effort:
with the current infrastructure partners
With suppliers and support partners with YOU…
We will strive for a seamless transition to the new situation
Beyond BiG Grid: Sector Clouds and Big Data…
Big Grid and Beyond – Amsterdam 120926
Beyond the Grid @CERNHelix Nebula: The Science Cloud
Establish a sustainable European cloud computing infrastructure to provide stable computing capacities and services that elastically meet demand
Pilot phase : Proof of Concept deployments on 3 commercial cloud providers (ATOS, CloudSigma, T-Systems)
Big Grid and Beyond – Amsterdam 120926
Dutch Federative CloudsA Prosumer Approach
Sustainability: Federation of existing data centers.
Scalability & Economy of Scale Maximize utilization factor
Security and performance: Sector specific security in place
Community support: Customer intimacy & sector expertise Community collaboration and sharing
Also for better partnership with Public Clouds
Big Grid and Beyond – Amsterdam 120926
To end with: Big Data is Changing Science
Big Data is changing science, medicine, business, and technology.
A whole new way of science: correlation supersedes causation, coherent models or unified theories…
Biggest challenge for science & business is not storing data but how to make sense of it.