uk hpc review 05 sept 2005 paris, 31 march 2003 give me a suitable title.... peter coveney centre...

UK HPC Review 05 Sept 2005Paris, 31 March 2003

Give me a suitable title.... Peter Coveney

Centre for Computational Science

University College London

UK HPC Review 05 Sept 2005

HPC and Grid activities (1)

Group general interests in HPC include scalable codes for Group general interests in HPC include scalable codes for capability computing in order to address grand challenge capability computing in order to address grand challenge problems in several areas of comptuational science through use problems in several areas of comptuational science through use of novel computational methods such as computational steering of novel computational methods such as computational steering and grid computingand grid computing

RealityGrid: RealityGrid: A £6M ($11) project funded by UK EPSRC. Initially A £6M ($11) project funded by UK EPSRC. Initially 2002-2005, now 2005-2009 (selected for Platform Grant)2002-2005, now 2005-2009 (selected for Platform Grant)..

118 PhD students over 8 years as special DTA award to the project8 PhD students over 8 years as special DTA award to the project

Through RealityGrid access to Through RealityGrid access to NSF PACI NSF PACI && NRAC NRAC computing awards computing awards

Spawned several sub-projects: Ongoing Joint US NSF-UK Spawned several sub-projects: Ongoing Joint US NSF-UK SPICE (2005-SPICE (2005-2006) 2006) andand TeraGyroid (2003) TeraGyroid (2003)

ESLEA - Exploitation of Switched Lightpaths for eScience ESLEA - Exploitation of Switched Lightpaths for eScience Applications. Applications. (£1.33M; 2005-2007). HPC component(£1.33M; 2005-2007). HPC component

Primary Primary UK project to join international effort in exploiting Lambda UK project to join international effort in exploiting Lambda networksnetworks


HPC and Grid activities (2)

Partners in EPSRC e-Science Pilot Project funded Partners in EPSRC e-Science Pilot Project funded Integrative Integrative BiologyBiology (£2.4 M; 2004-2007): Deploying RealityGrid capabilities (£2.4 M; 2004-2007): Deploying RealityGrid capabilities for steering, high performance computing and grid deployment for steering, high performance computing and grid deployment of both monolithic and coupled applications in cardiac dynamics of both monolithic and coupled applications in cardiac dynamics and tumour growth.and tumour growth.

IntBioSim:IntBioSim: An integrated approach to multi-level biomolecular An integrated approach to multi-level biomolecular simulations. E-Science project in computational systems biology simulations. E-Science project in computational systems biology (£976K;2004-2007)(£976K;2004-2007)

EPSRC e-Science Best Practice project entitled EPSRC e-Science Best Practice project entitled Rapid Rapid Prototyping of Usable Grid MiddlewarePrototyping of Usable Grid Middleware (£331K; 2005-2006). (£331K; 2005-2006).

Robust application hosting in WSRF::LiteRobust application hosting in WSRF::Lite (£170K; 2004-2006). (£170K; 2004-2006). OMII project provides users with a general hosting environment OMII project provides users with a general hosting environment for grid-enabled applications that exploits WSRF capabilities. for grid-enabled applications that exploits WSRF capabilities.


LB3D: Three dimensional Lattice-Boltzmann simulations• LB3D code is written in Fortran90

and parallelized using MPI

• Scales linearly on all available resources (CSAR, HPCx, Lemieux,

Linux/Itanium clusters)

• Fully steerable

• Uses parallel data format PHDF5

• Data produced during a single large scale simulation can exceed

hundreds of gigabytes to terabytes

• Simulations require supercomputers

• High end visualization hardware and parallel rendering software

(e.g. VTK) needed for data analysis

3D datasets showing snapshots from a simulation of spinodal decomposition: A binary mixture of water and oil phase separates. ‘Blue’ areas denote high water densities and ‘red’ visualizes the interface between both fluids.


Computational steering

Initial condition: Random water/ surfactant mixture.

Self-assembly starts.

Rewind and restart from checkpoint.

Lamellar phase: surfactant bilayers between water layers.

Cubic micellar phase, low surfactant density gradient.

Cubic micellar phase, high surfactant density gradient.


monitoring

checkpointing

steering and control


Known experimentally for several years

Many interesting properties, theoretical and practical

Discovered using LB3D in 2003;announced at DSFD2003

Liquid crystalline structure leads to many questions—role of defects in materials and fluid properties, etc

Liquid crystalline gyroid cubic mesophase

N. González-Segredo and P. V. Coveney, "Self-assembly of the gyroid cubic mesophase: lattice-Boltzmann simulations." Europhys. Lett., 65, 6, 795-801 (2004);


•Funded by EPSRC (UK) & NSF (USA)-to federate the UK e-Science Grid and US TeraGrid

•Main objective was to deliver high impact science which it would not be possible to perform without the combined resources of the US and UK grids

•Study of defect dynamics in liquid crystalline surfactant systems using lattice-Boltzmann methods

•Four month project including work exhibited at Supercomputing 2003 and SC Global

TeraGyroid: A high performance computing grid project


TeraGyroid Grid

VisualizationComputation

Starlight (Chicago)

Netherlight (Amsterdam)

BT provision

PSC

ANL

NCSA

Phoenix

Caltech

SDSC

UCL

Daresbury

Manchester

SJ4MB-NG

Network PoP

Access Grid nodeService Registry

production network

Dual-homed system

10 Gbps

2 x 1 Gbps


Defects and their dynamics in gyroid phases

Developed analysis tools for identifying &tracking defects; classified defects

Studied time evolution of defects

Mechanical and rheological propertiesLattice sizes up to 10243

Huge data analysis challenge


Spherical Pickering emulsion droplets = “colloidosomes”

http://www.deas.harvard.edu/projects/weitzlab also Hull group webpages

COLLOIDS IN BINARY SOLVENTS

Colloids in binary solvents

Spherical colloids superposed on lattice:

Surfaces are cut by boundary links

Particle centres move smoothly

Bounceback on links algorithm

N.-Q. Nguyen and A. Ladd, Phys Rev E 66, 046708 (2002)

Net force/torque on particle:

Sum of contributions from all boundary links

Used to update the colloid velocity

BBL causes no overall change in mass if flow incompressible

COLLOID HYDRODYNAMICS IN LB

LUDWIG

• MPI, Open MP, etc; 40,000 lines• Supercomputer + workstation compatible• Gold Star rating on HPCx

‘BIG SCIENCE’Algorithm design, coding, problem definition, parameter steering, data production, analysis, interpretation....

SYSTEM SIZES• 30/40 fluid/fluid interfaces across system; about 10,000 colloids•1283 lattice size maximum with colloids• larger with MPI on BlueGene/L expected Q3 2005

LIMITATIONS

•Careful parameter steering / testing needed to keep systematic errors in check.

• Large but finite range of parameters achievable: maintain hierarchy


SPICE: Simulated Pore Interactive Computing

Environment

●Transport of biomolecules through protein pores not well understood.

● Fully atomistic simulations crucial to capture pore-protein interaction.

●Systems of this size and complexity are computationally intensive.

● Time scale of transport is typically tens of microseconds. Traditional computational approaches (MD) manage barely few nanoseconds.

DNA beginning its translocation through an alpha-hemolysin protein pore embedded in a lipid layer. There are 275,000 atoms in the system. Water is not shown


• Grid approach facilitates more

effective and greater resource

utilization; permits novel analysis

approaches; uniform interface makes

utilization “easier”.

● Appropriate choice of algorithms is

required to make the problem amenable

to a grid-based solution.

SMD+ JE: Steered Molecular Dynamics

(SMD) to “pull DNA through the pore”

and Jarzynksi's Equation (JE) to compute

the equilibrium free energy profile from

the non-equilibrium pulling.

A view into the pore from the top end. Note the seven fold symmetry of the system.

SPICE: Grid Computing Using Novel Algorithms


Step I: Understand structural features using static visualization

Step II: Interactive simulations for dynamic and energetic features Steered simulations: Visualizer used to apply forces; forces sent to simulation; simulation computes effect of forces; simulation sends updated information back to visualizer Haptic interaction: Use haptic device to feel feedback forces

Step III: Simulations to compute “optimal” parameters values. 75 simulations on 128/256 processors each. (100pN/A spring constant; 12.5 A/ns pulling velocity)

Step IV: Use computed “optimal” values to calculate full FEP along the cylindrical axis of the pore. Will requires 250,000 CPU hours (100 simulations of 2500 CPU hours) at least!!

Computing the Free Energy Profile (FEP)


The infrastructure used similar to the TeraGyroid project -- Software & Hardware. Some resources (e.g UK-NGS) more mature. Grid middleware not much better!

High-end systems required to provide realtime interactivity. Simulation might stall due to “unreliable” sim-vis communication.

Advanced networks provide schedulable capacity and high QoS in terms of packet loss, jitter and latency. Better performance using UKLight

SPICE: Grid Infrastructure


Calculating binding free energies

• Usually possible to only compute ∆∆Gbind

• Consider the thermodynamic cycle

+

∆GA

+

∆GB

∆G1

∆G2

•Free energy is a thermodynamic state function ∆G1 + ∆GB - ∆G2 - ∆GA = 0

∆∆Gbind = ∆GB - ∆GA

∆∆Gbind = ∆G2 - ∆G1


Thermodynamic integration•The free energy change is calculated using the thermodynamic integral1

GAB =

€

∂H

∂λ0

1

∫ dλ

1. Run 10+ simulations, each at a distinct value of (e.g. 0.1,0.2…)

2. For each, compute

3. Plot as a function of and integrate the function using the trapezium rule

€

∂H

∂λ

€

∂H

∂λ

-100

-80

-60

-40

-20

0

20

0 0.5 1

Example TI plot

1. Leach, A. R. 2001 Molecular Modelling. Principles and Applications 2nd edition Edinburgh: Pearson Education Ltd.


=0.1

=0.2

=0.3

…

=0.9

Starting conformation

t

Seed successive simulations

(10 sims, each 2ns)

Check for convergence

lambda

Combine and calculate integral

time

Use steering to launch, spawn and terminate - jobs

Run each independent job on the Grid

This calculation is ideally suited for a computational grid

STIMD


A User’s PerspectiveNAMD2.5-

ReG

SGS

US TeraGrid

Steering client

Registry

find

publish

bindManchester

laptop/PDA/portal

UK NGS

1. Using the RealityGrid (ReG) launching wizard, the user launches a parallel NAMD2.5-ReG job on the US TeraGrid. The exchange of locations and data is facilitated by the Registry.

2. A second simulation is launched with a different value of . This is spawned from the first simulation and, for example, is displayed in a different tab of the ReG steering client.

3. The ensemble averages are plotted by the ReG steering client. Once a simulation has converged, it is terminated.

4…the second simulation is migrated

spawned

NAMD2.5-ReG

SGS

NAMD2.5-ReG

SGSSTIMD


The SH2 calculation at the AHM2004 was a success

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

STIMD


Immunoinformatics: Agonist or antagonistFree energy differences of Tax to P6A

UnboundPPG

21 Bound

PPG21

BindingPG

1

+

+

P1

P2

P1

P2BindingPG

2

TI TI

Experiment

Experiment

102K atoms

11 windows

1.2ns/window

256 procs

LeMieux (PSC)

30K CPU hours

120h Wallclock

53K atoms

11 windows

2ns/window

256 procs

LeMieux (PSC)

33K CPU hours

130h Wallclock

Wan, S., et al J. Im munol. 157, 1715-1723, 2005.

∆∆Gbind = -2.9 +/- 0.2 kcal/mol Experiment

∆∆Gbind = -1.9 +/- 0.4 kcal/mol TI

Possibly worlds largest TI calculation!!


Immunological Synapse

330K atoms

Newton (CSAR):1ns: 16hours on

192 procs3100 CPU hours

~1 million atoms

Newton (CSAR):1ns: 48hours on 192 procs (estimated)

9300 CPU hours (estimated)

Turn immune responses:

Immunological synapse


WEDS• WSRF::Lite Environment for Distributed Simulation

• Prototype application hosting environment built using WSRF::Lite (UoM)

• Provides a WSRF-compliant wrapper around an application.

• Services for discovering and instantiating applications and for file staging.

• Application binary remains unaltered.

• WEDS version 1.0 released (2005)


Application launching using WEDS

machine_01(launcher)

machine_02 machine_03

...

machine_NN

multiprocessor

( OR )


AHE (‘Application Hosting Environment’)

• Perl-based successor to WEDS; design incorporates experience gained with WEDS

• Provide a web services presentation of existing (‘legacy’) applications; no requirement to modify app. source or binary

• Delegates access to compute resources (Globus, OMII, Sun Grid Engine) to GridSAM services (OMII)

• Uses WS-Security; MyProxy for user credentials

• Registry and discovery services for locating hosted apps.


Application launching via OMII_1 job launcher and GridSAM

uk hpc review 05 sept 2005 paris, 31 march 2003 give me a suitable title.... peter coveney centre...

Documents

uk hpc review

uk epsrc

escience project

grid computing realitygrid

omii project

epsrc escience pilot

grid activities

control slide