06 session3 g8exascaleworkshop ingenious
TRANSCRIPT
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 1/38
INGENIOUS:Using next generation computers and algorithms for
modeling the dynamics of large biomolecular
systems
Makoto Taiji
Computational Biology Research Core
RIKEN Quantitative Biology Center
Processor Research Team
RIKEN Advanced Institute for Computational Sciences
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 2/38
Our future targets
Cilia of mouse embryo
Bacterial FlagellumFluid dynamic mechanism responsible for breaking left-
right symmetry of the Human Body: The Nodal Flow ,
N. Hirokawa, Y. Okada, Y. Tanaka,
Annual Review of Fluid Mechanics 41, 53-72 (2009).
https://www.youtube.com/watch?v=3y_P67KwuvU
https://www.youtube.com/watch?v=vxiwhfgzL0Q
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 3/38
Challenges in Molecular Dynamics
simulations of biomolecules
S. O. Nielsen, et al, J. Phys. (Condens. Matter.), 15 (2004) R481
30,000 year・ ExaFLOPS
Strong
Scaling
Weak Scaling
K computer
Anton/MDG4 Target Region
1021
J energy(~3x1018J
is spent in Japan
in each year)
Multiscale approachis essential
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 4/38
Organization
n Molecular Fluctuations (Aston Group)
n Molecular Fluctuations ↔ Fluid Dynamics
(Moscow Group)
n Multiscale Fluid Dynamics (Univ. London/
Cambridge Group)
n HPC (RIKEN)
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 5/38
Mercedes-Benzwaterasabridgetothemacroscale
Implementa8oninmolecular
dynamics
ArtursScukinsandDmitryNerukh
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 6/38
WhyMBwater?
• Arela8velysimple,2Dmodelthathasallfeaturesandpeculiari8esofrealwater
• Being2DscalesmuchbeFerwithsize:allowstoreachthespa8alsizesofhydrodynamics
• Welldevelopedtheore8cally(startedbyBen-Naiminearlyseven8es)
• Computa8onallywellstudiedbyMonteCarlo,butnoinves8ga8onsbyMolecularDynamics• WellsuitedfroourpurposeofdevelopinghybridMolecularDynamics–luidDynamicsapproach
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 7/38
MercedesBenzpoten8al:,
whereisLennard-Jonespoten8al,
,
isorienta8ondependantpoten8al,isaGaussianfunc8on.
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 8/38
Wehavederivedtheformulasforcalcula8ng
thermodynamicsfromMDtrajectories
• Temperature:
• Pressure:
• Heatcapacity:
• Heatexpansioncoefficient:
• Compressibility:
whereisa8meaverage,Visanarea,N-numberofmolecules,T-temperature,K–kine8c
energy,–density.
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 9/38
Results
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 10/38
Structure
TheRDqualita8velydiffers
fromLennard–JonesRD
butcoincideswiththeresults
obtainedusingMonteCarlo
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 11/38
Conclusions
• The2DMercedes-Benzmodelmimicsrealwaterbehaviour.
• Capturesminimumofpressure(volume),nega8veexpansioncoefficient,minimumof
compressibilityandhighheatcapacity.
• RDqualita8velydiffersfromLennard-JonesRD.
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 12/38
Towardsaccuratemodelingacrossdifferentscales:high-
resolu6onmethodsforFluctua6ngHydrodynamics
equa6onsV.Y.Glotov,V.M.Goloviznin,A.V.Danilin
( )
( ) ( )
22
2
0
40
3
40
3
u
t x
u P u u s
t x x x
E P u q u s E u T ut x x x x x
ρ ρ
ρ ρ η
ρ ρ η κ
∂ ∂+ =
∂ ∂
∂ +∂ ∂ ∂+ − ⋅ ⋅ − =
∂ ∂ ∂ ∂
∂+
∂+
⋅∂ ∂ ∂ ∂⎛ ⎞+ − ⋅ ⋅ + ⋅ − =⎜ ⎟∂ ∂ ∂ ∂ ∂ ∂⎝ ⎠
2
;2
v
u
E c T ρ ρ ρ = +
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )2
8, , ;
3
2, , ;
k T s x t s x t x x t t
k T q x t q x t x x t t
η δ δ
σ
κ δ δ
σ
⋅ ⋅ ⋅ʹ ʹ ʹ ʹ= ⋅ − ⋅ −
⋅
⋅ ⋅ ⋅ʹ ʹ ʹ ʹ= ⋅ − ⋅ −
( )
( )
( )
( )
( )
( )
2
12 2 2 2
2
22 2 2 2
11 1;
1 1
11 1;
1 1
1ln
v
su P u P u c G
t t x xc s c s
su P u P u c G
t t x xc s c s
P s
t c T t γ
γ
ρ ρ ρ γ ρ ρ γ
γ
ρ ρ ρ γ ρ ρ γ
ρ ρ
⎛ ⎞ ⎛ ⎞⎧ ⎫−∂ ∂ ∂ ∂⎪ ⎪⎜ ⎟ ⎜ ⎟+ ⋅ + + − ⋅ + ⋅ =⎨ ⎬⎜ ⎟ ⎜ ⎟∂ ∂ ∂ ∂⋅ − − ⋅ − −⎪ ⎪⎩ ⎭⎝ ⎠ ⎝ ⎠
⎛ ⎞ ⎛ ⎞⎧ ⎫−∂ ∂ ∂ ∂⎪ ⎪⎜ ⎟ ⎜ ⎟− ⋅ + − − ⋅ − ⋅ =⎨ ⎬⎜ ⎟ ⎜ ⎟∂ ∂ ∂ ∂⋅ − − ⋅ − −⎪ ⎪⎩ ⎭⎝ ⎠ ⎝ ⎠
⎛ ⎞ ⎛∂ ∂− ⋅⎜ ⎟ ⎜∂ ∂⎝ ⎠ ⎝
3
1ln ;
v
P su G
x c T xγ
ρ ρ
⎡ ⎤ ⎡ ⎤⎞ ⎛ ⎞ ⎛ ⎞∂ ∂+ ⋅ − ⋅ =⎢ ⎥ ⎢ ⎥⎟ ⎜ ⎟ ⎜ ⎟∂ ∂⎠ ⎝ ⎠ ⎝ ⎠⎣ ⎦ ⎣ ⎦
( )
2
1
c
s
ρ
γ
⋅
<
−
One dimensional case
Characteristic form of LL-NS equations
Condition for hyperbolicity
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 13/38
Stochas6cfluxes
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )2
8, , ;
32
, , ;
k T s x t s x t x x t t
k T q x t q x t x x t t
η δ δ
σ κ
δ δ σ
⋅ ⋅ ⋅ ⎫ʹ ʹ ʹ ʹ= ⋅ − ⋅ − ⎪⋅ ⎪ ⇒⎬
⋅ ⋅ ⋅ ⎪ʹ ʹ ʹ ʹ= ⋅ − ⋅ −⎪⎭
( ) ( )
( ) ( )2
8, 0,1 ;
3
2, 0,1 ;
h
h
k T s x t Gauss
x t
k T q x t Gauss
x t
η
σ
κ
σ
⋅ ⋅ ⋅= ⋅
⋅ ⋅ Δ ⋅ Δ
⋅ ⋅ ⋅= ⋅
⋅ Δ ⋅ Δ
Stochastic fluxes approximation
• For high value of stochastic forcing (large s and q fluxes) the
solution of the LL Navier-Stokes equations is very challenging
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 14/38
•
Iserlis 1986• Roe 1998
• Samarskii and Goloviznin 1998
• Goloviznin and Karabasov 1998
• Karabasov, Hynes and Goloviznin 2001
• Tran and Scheurer 2002
• Kim 2004
• Goloviznin 2005
• Karabasov and Goloviznin 2007, 2009
c 0t x
∂ϕ ∂ϕ+ ⋅ =
∂ ∂
• Explicit, second-order in space and time
• Non-dissipative and low-dispersive
• Very compact stencil• Conservation form
• Staggered variables: one-cell stencil in space and time
• Nonlinear flux correction based on maximum principle
• Nonlinear flux reconstruction based on the minimum
solution variation
• Highly scalable method and has already been
successfully used in unsteady convection-dominatedflow modelling
x
t
Our choice: Compact Accurately Boundary Adjustinghigh-REsolution Technique (CABARET)
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 15/38
Comparisonofseveralcomputa8onalschemesfortheBellproblem
MacCormackscheme 2.61 -.4%
Piecewiseparabolicmethod 2.5 -9.4%
Third-orderRunge-KuFa 2.7 0.9%
CABARET 2.75 -3.2%
MolecularSimula8on 2.7 -2.1%
Variance in conserved quantities at equilibrium
MacCormackscheme 2.01 -14.3%
Piecewiseparabolicmethod 1.97 -16.0%
Third-orderRunge-KuFa 2.34 -1.3%
CABARET 2.31 -1.7%
MolecularSimula8on 2.35 0%
MacCormackscheme 13.31 -0.3%
Piecewiseparabolicmethod 13.27 -0.5%
Third-orderRunge-KuFa 13.65 2.3%
CABARET 13.1 -1.2%
MolecularSimula8on 13.21 -1%
2δρ Exact value:
2
J δ Exact value:
82.35 10
−
×
2 E δ Exact value:
13.34
102.84 10×
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 16/38
§ Towards micro- and nano-scales• Temperature uctuations important, large density and velocity uctuations
• Acoustics: ultra-sound / Biological applications: Coupling with MD
§ Interesting phenomena concurrently occur at small andlarger scale, both in time and space• Numerically difficult to deal efficiently with large time/space diff erences
• A multi-space-time algorithm is demonstrated
AMULTI-SPACE-TIMEALGORITHMFORCONCURRENTLARGE/SMALL
SCALEFLUIDDYNAMICSSIMULATIONS
AntonMarkesteijnandSergeyKarabasov
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 17/38
Mul6Space-Timealgorithm-Overview
§ Fluctuating Hydrodynamics (Landau&Lifshitz)• Mimic microscopic behaviour at macroscopic scales
• Dissipative uxes treated as stochastic variables• Thermodynamics: “Fluctuation-Dissipation theorem”
§ “Scale Function”: A (pre)dened “meshless” zoom value• The value of this function is increased where small time and space
phenomena are dominant• The scale function also determines the actual comp. grid
§ Equation transformations both in space and time• Transformations are dependent on scale function
• Transformed (Computational Domain) / Untransformed (Physical Domain) § Special time marching (local and global time)
• Local time step controlled by scale function
• Cells only updated when necessary (local<global time)
• (CFL curse): increased efficiency, decreased error
l f l
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 18/38
SomeExamplesofScaleFunc6ons
§ 1D Scale diff erence of 5• Both mesh size and local time scaled
• Computational domain simple Cartesian mesh
§ 2D Mesh (radial 25 to 1 (200x200 mesh)
l l 6 d d i
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 19/38
2DExample:Fluctua6ngHydrodynamics
§ Scale diff erence of 100, on a 200x200 mesh
§ Probe in centre of domain• Measure density transient
• Acoustic signal recovered from noise‒ Time ensemble
• Variables are Maxwellian
Fl 6 H d d i MD
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 20/38
Fluctua6ngHydrodynamicsvsMD
§ Density uctuations and the speed of sound• Domain 250x40, Scale Function 1 to 25 to 1 in
plateaus• Smallest volume 0.6x0.6x0.6 nm3 (liquid water)
• Speed of sound obtained by t (~1510 m/s)
• Continuum results compared to MD results
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 21/38
Scaling challenges in MD
n 〜50,000 FLOP/particle/step
n Typical system size : N=105
n 5 GFLOP/step
n 5TFLOPS eff ective performance
1msec/step = 170nsec/day
Rather Easy
n 5PFLOPS eff ective performance
1μsec/step = 200μsec/day???
Di fficult, but important
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 22/38
Scaling of MD on K Computer
Strong scaling〜50 atoms/core
~3M atoms/Pflops
22
1,674,828 atoms
Since K Computer is still under development,
the result shown here istentative.
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 23/38
GRAPE: special-purpose computer
for classical particle simulations
n GRAvity PipEn Originaly proposed by Prof. Chikada, NAOJ
n Special-purpose accelerator
▷ Astrophysical N -body simulations
▷ Molecular Dynamics Simulations
J. Makino & M. Taiji, Scientific Simulations with Special-Purpose Computers,John Wiley & Sons, 1997.
Host
Computer GRAPE
Most of Calculation → GRAPE Others → Host computer
Particle Data
Results
Problem in Heterogeneous System GRAPE/
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 24/38
Problem in Heterogeneous System - GRAPE/
GPUsn In small system
▷ Good acceleration, High performance/costn In massively-parallel system
▷ Scaling is often limited by host-host network,
host-accelerator interface
HostComputer
Accelerator
HostComputer
Accelerator
Host Network
High-Latency
Low-Bandwidth
HostComputer
Accelerator
General-purpose core
Embeddedmemories
Accelerator
General-purpose core
Embeddedmemories
Low-Latency
High-BandwidthLow-Latency
System-on-Chip
Low-BandwidthHigh-LatencyNetwork
Typical Accelerator System SoC-based System
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 25/38
Anton
n D. E. Shaw Research
n Special-purpose pipeline
+ General-purpose CPU core
+ Specialized network
n Anton showed the importance of
the optimization in communicationsystem
R. O. Dror et al., Proc. Supercomputing 2009, in USB memory.
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 26/38
MDGRAPE-4
n Special-purpose computer for MD simulation
n Test platform for special-purpose machines
n Target performance
▷ 20μsec/step for 100K atom system
▷ 8.6μsec/day (2fsec/step)
n Target application : GROMACS
n Completion: ~2013
n Enhancement from MDGRAPE-3
▷ 130nm→ 40nm process
▷ Integration of Network / CPU
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 27/38
MDGRAPE-4 System
48 Optical Fibers
12 lane6Gbps
Optical
12 lane6Gbps
Electric = 7.2GB/s
(after 8B10Bencoding)
Node(2U Box)
Total 64 Nodes(4x4x4)=4 pedestals
MDGRAPE-4SoC
Total 512 chips
(8x8x8)
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 28/38
MDGRAPE-4 System-on-Chip
n 40 nm (Hitachi HDL4S), ~ 230mm2
n 64 force calculation pipelines
@ 0.8GHz
~ 2.5 TFLOPS equivalentn 64 general-purpose processors
Tensilica Extensa LX4
@0.6GHz
n 72 lane SERDES @6GHz
n 65W
S C Bl k Di
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 29/38
SoCBlockDiagram
Network Unit FPGA IF
Global Memory
x y z 6Gbps x 12 x 6100MHz x 128
Bus Arbiter /DMAC
IMem
DMemCore
IMem
DMemCore
8 Pipelines
8 Pipelines
8 Pipelines
Bus Arbiter /DMAC
IMem
DMemCore
IMem
DMemCore
IMem
DMemCore
InstructionMemory (1)
InstructionMemory (2)
InstructionMemory (CGP)
Pipeline Blocks GP BlocksControl GP
MessageQueue
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 30/38
Embedded Global Memories in SoC
n ~1.8MBn 4 Block
n For Each Block
▷ 128bit X 2 for General-
purpose core▷ 192bit X 2 for Pipeline
▷ 64 bit X 6 for Network
▷ 256bit X 2 for Inter-block
GM4 Block 460KB
GM4 Block
460KB
GM4 Block 460KB
GM4 Block
460KB
Network
2 PipelineBlocks
2 GP Blocks
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 31/38
General-Purpose Core
n Tensilica LX @ 0.6 GHz
n 32bit integer / 32bit Floating
n 4KB I-cache / 4KB D-cache
n 8KB Local Memory
▷ DMA or PIF access
n 8KB Local Instruction Memory
▷ DMA read from 512KB Instruction memory
Core
4KB
Dcache
8KB
D-ram
4KBIcache
8KB
I-ram
Core
Core Core
Core Core
Core Core
Core Core
DMAC
QueueIF
Integer
Floa:ngQueue
PIF
Inst-
ruc:on
DMAC
GP Block
Barrier
Global Memory
Control Processor
InstructionMemory
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 32/38
Software evaluation platform for
MDGRAPE-4: RTL model
n RTL-based simulator on Candence Ncsimn Cycle accurate
n Slow (>10ms/cycle)
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 33/38
Software evaluation platform for
MDGRAPE-4:
n Under construction (4Q 2012)n Tensilica XTMP based multicore processor
simulator (non-free)
n Includes behavior models of
▷ Network
▷ Special-purpose pipeline
▷ Memories (latency can be considered)
n Two-levels▷ Precise memory models for instruction
▷ Innite memory for instruction
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 34/38
Software evaluation platform for
MDGRAPE-4 (2)
n Programming language▷ C▷ C++ (without malloc)
n Direct control of network unitsn No operating system, with simple monitor
l l f
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 35/38
Evaluation platform
based on MDGRAPE-4 simulator
n Extend MDGRAPE-4 simulatorn Change Balance▷ More resources for general-purpose cores
n General-purpose cores▷ Shared on-chip memory for 8-16 cores
▷ off -chip memory
▷ synchronization mechanism
n Special-purpose pipelines
n Network interface
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 36/38
Special-
purposeblock
General-
purposecores
Local/Cache
memories
On-chip
Network
Special-
purposeblock
General-
purposecores
Local/Cache
memories
On-chip
Network
Off-chip
Network
Off-chip
Memory
Special-
purposeblock
General-
purposecores
Local/Cache
memories
On-chip
Network
Special-
purposeblock
General-
purposecores
Local/Cache
memories
On-chip
Network
Off-chip
Network
Off-chip
Memory
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 37/38
Toward Exascale
For Molecular Dynamicsn Single-chip system
▷ >1/30 of the MDGRAPE-4 system can be embedded with11nm process
▷ Local MD + Multiscale▷ Still network is necessary inside SoC
n For further strong scaling for MD
▷ # of operations / step / 20Katom ~ 109
▷ # of arithmetic units in system ~ 106 /PopsExascale means “Flash” (one-path) calculation
▷ More specialization is required
7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS
http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 38/38
Meetings & Visits
n Past▷ Nov 2011 @ Cambridge
▷ UK PI(Dr. Nerukh)’s visit to RIKEN for a month in Dec 2011
▷ Sep 2012 @ Kobe
▷ UK Researcher (Mr. Skukins)→ RIKEN (Sep-Nov 2012)n Future related events
▷ Dec 2012: UK-Japan bilateral workshop at British embassyin Tokyo (supported by British embassy Japan)
▷ Jul 2013: Royal Society Kavli Seminar in UK “Multiscale systems: linking quantum chemistry, moleculardynamics, and microuidic hydrodynamics”
▷ Project workshops in UK or/and Japan (2013)