at louisiana state university university of washington – e-science introduction to the teragrid...

AT LOUISIANA STATE UNIVERSITYUniversity of Washington – e-Science

Introduction to the TeraGridJeffrey P. Gardner

Sr. Research Scientist, High Performance Computing

University of Washington

Dept. of Physics

Thanks to TeraGrid community

(the source of many of these slides)

esp. Daniel S. Katz, LSU

University of Washington – e-Science

Overview

• TeraGrid is US national resource• Funded by the NSF Office of Cyberinfrastructure• Gives any researcher in the U.S. access to

leading-edge computational resources• Detailed info about the TeraGrid• How to start using the TeraGrid

2


What is Cyberinfrastructure?

• “Cyberinfrastructure is a technological solution to the problem of efficiently connecting data, computers, and people with the goal of enabling derivation of novel scientific theories and knowledge.”1

• Term was used by the NSF Blue Ribbon committee in 2003 in response to the question: “How can NSF… remove existing barriers to the rapid evolution of high performance computing, making it truly usable by all the nation's scientists, engineers, scholars, and citizens?”

• The TeraGrid2 is the NSF’s response to this question.• Cyberinfrastructure is also called e-Science3

1Source: Wikipedia2More properly, the TeraGrid in it’s current form: the “Extensible Terascale Facility”3Source: NSF


What is the TeraGrid?• World’s largest infrastructure for open scientific

discovery• Leadership class resources at eleven partner sites

combined to create an integrated, persistent computational resource– High-performance networks– High-performance computers (>750 TFlops)– Visualization systems– Data resources and tools (>30 PB, >100 discipline-specific

databases)– Science Gateways– User portal– User services - Help desk, training, advanced app support

• Allocated through national peer-review process• (It’s free!)

4

~2 PFlops in another year!


TeraGrid Resources

5


TeraGrid Systems 2007-8

Computational Resources (size approximate - not to scale)Slide Courtesy Tommy Minyard, TACC

SDSC

TACC

UC/ANL

NCSA

ORNL

PU

IU

PSC

NCAR

TennesseeLONI/LSU

6


Who Uses TeraGrid

Molecular Biosciences

31%

Chemistry17%

Physics17%

Astronomical Sciences

12%

Materials Research6%

Earth Sciences3%

All 19 Others4%

Advanced Scientific Computing

2%

Atmospheric Sciences

3%

Chemical, Thermal Systems

5%

7


How TeraGrid Is Used

8

Use ModalityUse ModalityCommunity SizeCommunity Size

(rough est. - number of (rough est. - number of users)users)

Batch Computing on Individual Resources 850Exploratory and Application Porting 650Workflow, Ensemble, and Parameter Sweep 250Science Gateway Access 500Remote Interactive Steering and Visualization 35Tightly-Coupled Distributed Computation 10

2006 data from


25

50

75

100

125

150

175

200

225

250

275

J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J

2004 2005 2006 2007

NU

s (m

illi

on

s)Specific

Roaming

TeraGrid Usage

33% Annual Growth

Specific Allocations Roaming Allocations

200

100

Normalized Units (millions)

TeraGrid currently delivers an average of 420,000 cpu-hours per day Dave Hart ([email protected])


TG usage: Predicting storms

10

• Hurricanes and tornadoes cause massive loss of life and damage to property

• TeraGrid supported spring 2007 NOAA and University of Oklahoma Hazardous Weather Testbed– Major Goal: assess how well ensemble

forecasting predicts thunderstorms, including the supercells that spawn tornadoes

– Nightly reservation at PSC– Delivers “better than real time” prediction– Used 675,000 CPU hours for the season– Used 312 TB on HPSS storage at PSC

Slide courtesy of Dennis Gannon, IU, and LEAD Collaboration


TG Usage: Gravitational Waves

Observations

ModelsAnalysis & Insight

Visualization Credits: Werner Benger, Ralf Kaehler, LSU/AEIData Simulation Credits: LSU/AEI relativity groups

CactusGravitational waves predicted from colliding black holes, neutron stars, supernovae


TG Usage: Cosmology

The Cosmic WebCosmological evolution and galaxy formation using a 3D cosmological n-body gravity+hydrodynamics code, Gasoline.

100 million light yearsCredits: Tom Quinn, Jeff Gardner, Univ. of Washington


TG usage: Biology

13

High resolution 3-D reconstruction of infectious viruses Wen Jiang, Weimin Wu, Purdue UniversityMatthew L. Baker, Joanita Jakana and Wah Chiu, Baylor College of MedicinePeter R. Weigele and Jonathan King, MIT

High resolution 3-D structure of virus particles provide important insights to the development of effective prevention and treatment strategies. This work used the electron cryo-microscopy to demonstrate the 3-D reconstruction of the infectious bacterial virus ε15 at 4.5 Å resolution, which allowed tracing of the polypeptide backbone of its major capsid protein gp7. The structure reveals similar protein architecture to that of other tailed double-stranded DNA viruses, even in the absence of detectable sequence similarity. However, the connectivity of the secondary structure elements (topology) in gp7 is unique.

Large numbers (104-105) of 2-D images (8002 pixels/image), representing the projections of identical 3-D structure viewed at different angles, were collected. These images require intensive computation to accurately determine their relative orientations before the 2-D images can be coherently merged into a single high resolution 3-D structure.

These results have been just published on Nature (Feb 28, 2008).Slide courtesy of Purdue and TeraGrid


Solve any Rubik’s Cube in 26 moves?

• Rubik's Cube is perhaps the most famous combinatorial puzzle of its time

• > 43 quintillion states (4.3x10^19)

• Gene Cooperman and Dan Kunkle of Northeastern Univ. proved any state can be solved in 26 moves

• 7TB of distributed storage on TeraGrid allowed them to develop the proof

Source: http://www.physorg.com/news99843195.html


Community Engagement through Science Gateways

• Increasing investment by communities in their own cyberinfrastructure, but heterogeneous:– Resources– Users – from expert to K-12– Software stacks, policies

• Three common forms:– Web portal with users in front and services in back– Client server model where application programs run on users'

machines (i.e. desktops) and access services– Bridges across multiple grids, allowing communities to utilize

both community developed grids and shared grids

• Science Gateways– Provide “TeraGrid Inside” capabilities– Leverage community investment

Slide courtesy of Nancy Wilkins-Diehr

15


Current Science Gateways• Biology and Biomedicine Science Gateway• Open Life Sciences Gateway• The Telescience Project• Grid Analysis Environment (GAE)• Neutron Science Instrument Gateway• TeraGrid Visualization Gateway, ANL• BIRN

• Open Science Grid (OSG)• Special PRiority and Urgent Computing

Environment (SPRUCE)• National Virtual Observatory (NVO)• Linked Environments for Atmospheric

Discovery (LEAD)• Computational Chemistry Grid (GridChem)• Computational Science and Engineering

Online (CSE-Online)• GEON(GEOsciences Network)• Network for Earthquake Engineering

Simulation (NEES)• SCEC Earthworks Project• Network for Computational Nanotechnology

and nanoHUB• GIScience Gateway (GISolve)• Gridblast Bioinformatics Gateway• Earth Systems Grid• Astrophysical Data Repository (Cornell)


16


SGW Highlight: National Virtual Observatory - Facilitating Scientific Discovery

17

• Access to telescope images from around the world

• NVO provides access to combined sky surveys– Different views of the same

cosmological phenomenon can reveal new insights

• New science enabled by enhancing access to data and computing resources– Data correlation– Understanding of physical

processes– Identification of new

phenomenon

• NVO is a set of tools used to exploit the data avalanche


SGW Highlight: Linked Environments for Atmospheric Discovery (LEAD)

•Providing tools that are needed to make accurate predictions of tornados and hurricanes

•Meteorological data•Forecast models•Analysis and visualization tools

•Data exploration and Grid workflow


18


SGW Highlight: GridChem’s Client-Server Approach Provides Power and a Rich Feature Set

Slide courtesy of Sudhakar Pamidighantam, NCSA

19


Gateways

20

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

Jan-

07

Feb-

07

Mar

-07

Apr

-07

May

-07

Jun-

07

Jul-0

7

Aug

-07

Sep

-07

Oct

-07

Nov

-07

Dec

-07

# o

f G

atew

ay J

ob

s

Nearly 500k gateways jobs in CY2007GridChem: 192k jobs, >210k TG SUsCIGportal: 94k jobs, >154k TG SUsLEAD: 40k jobs, >54k TG SUs



TG New Large Resources

• Ranger@TACC– First NSF ‘Track2’

HPC system– 504 TFlops– 15,744 Quad-Core

AMD Opteron processors– 123 TB memory, 1.7 PB disk

• Kraken@NICS (UT/ORNL)– Second NSF ‘Track2’ HPC system– 170 TFlops Cray XT4 system– Will be upgraded to Cray XT5 at nearly 1 PFlops

• 10,000+ compute sockets• 100 TB memory, 2.3 PB disk

21


So how do I get on the TeraGrid?

• The best thing to do: Talk to your local TeraGrid “Campus Champion” (for UW, that’s me)

• Campus Champion can:– Direct you to the most appropriate TeraGrid platforms– Give you an experimental TeraGrid account– Help you write proposals to acquire TeraGrid time


TeraGrid Resource Allocations

• Every TeraGrid award of time is either:– System-specific (“Type S”): Time is awarded for a

specific system, e.g. PSC Cray XT3.– TeraGrid Roaming (“Type R”): Time is awarded that

can be used on any* TeraGrid system.

• TeraGrid time is awarded in “Service Units” or “SUs”. SUs correspond roughly to CPU-hours:– For system-specific awards, 1 SU = 1 CPU-hour on

that machine.– For TeraGrid Roaming awards, 1 SU = 1 CPU-hour

on a 1.5GHz Itanium2 system and will be converted for whichever machine you actually run on.

*With a few exceptions


TeraGrid Resource Allocations

• The easiest type of allocation to get is a “Development Allocation” or “DAC”*:– Currently DACs are 30,000 SUs– Submit a single page (i.e. no more than 3 paragraphs)

description of your research and your goals for trying TeraGrid.

– Development applications are reviewed and awarded continuously

– You will be up and running within a few weeks.

*DAC = “Development Allocation Committee”


POPS - Allocations

• POPS is the on-line system used for the allocations process (pops-submit.teragrid.org)– Allocation Requests– Peer reviews– Usage information

• pops.teragrid.org for now – also accessible from the TeraGrid user portal (portal.teragrid.org)

25


Allocation Process

• Types of Requests– DAC (up to 30k SUs, continually reviewed)

• DACS on Ranger can be larger, same for other new large resources (Kraken, Track2c, etc.)

– MRAC (up to 500k SUs, reviewed quarterly)– LRAC (over 500k SUs, reviewed semi-annually)– Can apply for compute, data, support resources– Also, there are community accounts…

• Awards– Most awards are granted in full!– For one or more 12 month periods – can be renewed– Can rebut reviewers who reject or cut award

26


How You Can Use TeraGrid

ComputeService

VizService

DataService

Network, Accounting, …

Site 1

Site 3

Site 2

TeraGrid Infrastructure (Accounting, Network, Authorization,…)

POPS (for now)

Science Gateways

UserPortal

Command Line

Slide courtesy of Dane Skow and Craig Stewart

27


User Portal: portal.teragrid.org

28


Access to resources

• Terminal: ssh, gsissh• Portal: TeraGrid user

portal, Gateways– Once logged in to

portal, click on “Login”

29


User Portal – Compute/Viz Resources

30


User Portal – Other Resources

31


User Portal – Other Information

32

• Knowledge Base for quick answers to technical questions

• Documentation

• Science Highlights

• News and press releases

• Education, outreach and training events and resources


Data Storage Resources

• GPFS-WAN– 700 TB disk storage at SDSC, accessible from machines at

NCAR, NCSA, SDSC, ANL

• Data Capacitor– 535 TB storage at IU, including databases

• Data Collections– Storage at SDSC (files, databases) for collections used by

communities

• Tape Storage– Available at IU, NCAR, NCSA, SDSC, PSC

• Access is generally through GridFTP • Typical data transfer speeds are 100MB/s!

33


Conclusions

• TeraGrid is not a secret government agency• Just a collection of universities working together,

funded by NSF• Currently, an abundance of cycles available• Talk to current users/participants or your

Campus Champion for help with proposals and other information

34

at louisiana state university university of washington – e-science introduction to the teragrid...

Documents