molecular dynamics (md) on gpus · supercharger library, vasp & more green = application...

Feb. 2, 2017

Molecular Dynamics (MD) on GPUs

2

Accelerating Discoveries

Using a supercomputer powered by the Tesla

Platform with over 3,000 Tesla accelerators,

University of Illinois scientists performed the first

all-atom simulation of the HIV virus and discovered

the chemical structure of its capsid — “the perfect

target for fighting the infection.”

Without gpu, the supercomputer would need to be

5x larger for similar performance.

3

Overview of Life & Material Accelerated Apps

MD: All key codes are GPU-accelerated

Great multi-GPU performance

Focus on dense (up to 16) GPU nodes &/or large # of

GPU nodes

ACEMD*, AMBER (PMEMD)*, BAND, CHARMM, DESMOND, ESPResso,

Folding@Home, GPUgrid.net, GROMACS, HALMD, HOOMD-Blue*,

LAMMPS, Lattice Microbes*, mdcore, MELD, miniMD, NAMD,

OpenMM, PolyFTS, SOP-GPU* & more

QC: All key codes are ported or optimizing

Focus on using GPU-accelerated math libraries,

OpenACC directives

GPU-accelerated and available today:

ABINIT, ACES III, ADF, BigDFT, CP2K, GAMESS, GAMESS-

UK, GPAW, LATTE, LSDalton, LSMS, MOLCAS, MOPAC2012,

NWChem, OCTOPUS*, PEtot, QUICK, Q-Chem, QMCPack,

Quantum Espresso/PWscf, QUICK, TeraChem*

Active GPU acceleration projects:

CASTEP, GAMESS, Gaussian, ONETEP, Quantum

Supercharger Library*, VASP & more

green* = application where >90% of the workload is on GPU

4

MD vs. QC on GPUs

“Classical” Molecular Dynamics Quantum Chemistry (MO, PW, DFT, Semi-Emp)Simulates positions of atoms over time;

chemical-biological or chemical-material behaviors

Calculates electronic properties; ground state, excited states, spectral properties,

making/breaking bonds, physical properties

Forces calculated from simple empirical formulas (bond rearrangement generally forbidden)

Forces derived from electron wave function (bond rearrangement OK, e.g., bond energies)

Up to millions of atoms Up to a few thousand atoms

Solvent included without difficulty Generally in a vacuum but if needed, solvent treated classically (QM/MM) or using implicit methods

Single precision dominated Double precision is important

Uses cuBLAS, cuFFT, CUDA Uses cuBLAS, cuFFT, OpenACC

Geforce (Workstations), Tesla (Servers) Tesla recommended

ECC off ECC on

5

GPU-Accelerated Molecular Dynamics Apps

ACEMD

AMBER

CHARMM

DESMOND

ESPResSO

Folding@Home

GPUGrid.net

GROMACS

HALMD

HOOMD-Blue

LAMMPS

mdcore

Green Lettering Indicates Performance Slides Included

GPU Perf compared against dual multi-core x86 CPU socket.

MELD

NAMD

OpenMM

PolyFTS

6

Benefits of MD GPU-Accelerated Computing

• 3x-8x Faster than CPU only systems in all tests (on average)

• Most major compute intensive aspects of classical MD ported

• Large performance boost with marginal price increase

• Energy usage cut by more than half

• GPUs scale well within a node and/or over multiple nodes

• K80 GPU is our fastest and lowest power high performance GPU yet

Try GPU accelerated MD apps for free – www.nvidia.com/GPUTestDrive

Why wouldn’t you want to turbocharge your research?

http://www.nvidia.com/GPUTestDrive

www.acellera.com

470 ns/day on 1 GPU for L-Iduronic acid (1362 atoms)

116 ns/day on 1 GPU for DHFR (23K atoms)

M. Harvey, G. Giupponi and G. De Fabritiis, ACEMD: Accelerated molecular dynamics simulations in the microseconds timescale, J. Chem. Theory and Comput. 5, 1632 (2009)

www.acellera.com

NVT, NPT, PME, TCL, PLUMED, CAMSHIFT1

1 M. J. Harvey and G. De Fabritiis, An implementation of the smooth particle-mesh Ewald (PME) method on GPU hardware, J. Chem. Theory Comput., 5, 2371–2377 (2009)2 For a list of selected references see http://www.acellera.com/acemd/publications

June 2017

AMBER 16

11

JAC_NVE on GP100s

Running AMBER version 16

The green nodes contain Dual Intel(R) Core(TM) i7-4820K @ 3.70GHz CPUs +

Quadro GP100s GPUs (PCIe and NVLink)

320.19 320.14

370.32

404.09

0

50

100

150

200

250

300

350

400

450

1 node + 1x GP100per node (PCIe)

1 node + 1x GP100per node (NVLink)



ns/

day

23,558 atoms PME 2fs

12

JAC_NVE on GP100s

614.42 613.16

714.23

782.11

0

100

200

300

400

500

600

700

800

900





ns/

day





13

JAC_NPT on GP100s

295.75 295.42

333.03

360.64

0

50

100

150

200

250

300

350

400





ns/

day





14

JAC_NPT on GP100s

580.47 578.48

654.66

706.53

0

100

200

300

400

500

600

700

800





ns/

day





15

FactorIX_NVE on GP100s

106.23 105.98

142.45

166.61

0

20

40

60

80

100

120

140

160

180





ns/

day

90,906 atoms PME




16

FactorIX_NPT on GP100s

102.27 102.26

126.75

146.34

0

20

40

60

80

100

120

140

160





ns/

day

90,906 atoms PME




17

Cellulose_NVE on GP100s

24.01 24.02

31.35

36.91

0

5

10

15

20

25

30

35

40





ns/

day

408,609 atoms PME




18

Cellulose_NPT on GP100s

22.76 22.8

28.76

32.37

0

5

10

15

20

25

30

35





ns/

day

408,609 atoms PME




19

STMV_NPT on GP100s

15.64 15.43

20.22

23.13

0

5

10

15

20

25





ns/

day

1,067,095 atoms PME




20

TRPCAGE on GP100s

1216.561187.3

0

250

500

750

1000

1250

1500



ns/

day

304 atoms GB




21

Myoglobin on GP100s

470.41458.28

443.49 447.23

0

150

300

450

600





ns/

day

2,492 atoms GB




22

Nucleosome on GP100s

11.47 11.29

21.2920.51

0

5

10

15

20

25





ns/

day

25,095 atoms GB




February 2017

AMBER 16

24

PME-Cellulose_NPT on K80s

Running AMBER version 16.3

The blue node contains Dual Intel Xeon E5-2699 [email protected] [3.6GHz Turbo]

(Broadwell) CPUs

The green nodes contain Dual Intel Xeon E5-2699 [email protected] [3.6GHz

Turbo] (Broadwell) CPUs + Tesla K80 (autoboost) GPUs

➢ 1x K80 is paired with Single Intel Xeon E5-2699 [email protected] [3.6GHz

Turbo] (Broadwell)2.35

11.36

15.43

0

4

8

12

16

20

1 Broadwell node 1 node +1x K80 per node

1 node +2x K80 per node

ns/

day

PME-Cellulose_NPT

4.8X

6.6X

25

PME-Cellulose_NPT on P100s PCIe



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla P100 PCIe (16GB) GPUs

➢ 1x P100 PCIe is paired with Single Intel Xeon E5-2699 [email protected]

[3.6GHz Turbo] (Broadwell)2.35

21.85

30.00

0

5

10

15

20

25

30

35

40

1 Broadwell node 1 node +1x P100 PCIe (16GB)

per node

1 node +2x P100 PCIe (16GB)

per node

ns/

day

PME-Cellulose_NPT

9.3X

12.8X

26

PME-Cellulose_NPT on P100s SXM2



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla P100 SXM2 GPUs

➢ 1x P100 SXM2 is paired with Single Intel Xeon E5-2698 [email protected]


23.37

32.22

36.65

0

5

10

15

20

25

30

35

40

1 Broadwell node 1 node +1x P100 SXM2

per node

1 node +2x P100 SXM2

per node


per node

ns/

day

PME-Cellulose_NPT

9.9X

13.7X15.6X

27

PME-Cellulose_NVE on K80s



(Broadwell) CPUs





11.85

16.53

0

4

8

12

16

20



ns/

day

PME-Cellulose_NVE

4.8X

6.7X

28

PME-Cellulose_NVE on P100s PCIe



(Broadwell) CPUs





23.34

32.55

0

5

10

15

20

25

30

35

40


per node


per node

ns/

day

PME-Cellulose_NVE

9.4X

13.2X

29

PME-Cellulose_NVE on P100s SXM2



(Broadwell) CPUs





24.94

35.16

40.88

0

5

10

15

20

25

30

35

40

45


per node


per node


per node

ns/

day

PME-Cellulose_NVE

10.1X

14.2X16.6X

30

PME-FactorIX_NPT on K80s



(Broadwell) CPUs





48.54

66.68

0

10

20

30

40

50

60

70

80



ns/

day

PME-FactorIX_NPT

4.2X

5.8X

31

PME-FactorIX_NPT on P100s PCIe



(Broadwell) CPUs





98.77

132.86

0

20

40

60

80

100

120

140


per node


per node

ns/

day

PME-FactorIX_NPT

8.6X

11.6X

32

PME-FactorIX_NPT on P100s SXM2



(Broadwell) CPUs





106.25

144.11

159.80

0

20

40

60

80

100

120

140

160

180


per node


per node


per node

ns/

day

PME-FactorIX_NPT

9.3X

12.6X14.0X

33

PME-FactorIX_NVE on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

11.98

51.14

71.49

0

10

20

30

40

50

60

70

80



ns/

day

PME-FactorIX_NVE

5.4X

6.0X

34

PME-FactorIX_NVE on P100s PCIe



(Broadwell) CPUs





105.86

145.83

0

20

40

60

80

100

120

140

160


per node


per node

ns/

day

PME-FactorIX_NVE

8.8X

12.2X

35

PME-FactorIX_NVE on P100s SXM2



(Broadwell) CPUs





114.88

159.24

178.02

0

20

40

60

80

100

120

140

160

180

200


per node


per node


per node

ns/

day

PME-FactorIX_NVE

9.6X

13.3X14.9X

36

PME-JAC_NPT on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

45.89

162.09

216.78

0

50

100

150

200

250



ns/

day

PME-JAC_NPT

3.5X

4.7X

37

PME-JAC_NPT on P100s PCIe



(Broadwell) CPUs




[3.6GHz Turbo] (Broadwell)

45.89

283.60

327.69

0

50

100

150

200

250

300

350


per node


per node

ns/

day

PME-JAC_NPT

6.2X

7.1X

38

PME-JAC_NPT on P100s SXM2



(Broadwell) CPUs





310.52

360.64

423.09

0

50

100

150

200

250

300

350

400

450

1 Broadwell node 1 node +1x P100 PCIe

per node

1 node +2x P100 PCIe

per node


per node

ns/

day

PME-JAC_NPT

6.8X7.9X

9.2X

39

PME-JAC_NVE on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

47.90

173.20

234.99

0

50

100

150

200

250



ns/

day

PME-JAC_NVE

3.6X

4.9X

40

PME-JAC_NVE on P100s PCIe



(Broadwell) CPUs





47.90

308.46

363.79

0

50

100

150

200

250

300

350

400


per node


per node

ns/

day

PME-JAC_NVE

6.4X

7.6X

41

PME-JAC_NVE on P100s SXM2



(Broadwell) CPUs





339.81

402.18

473.10

0

50

100

150

200

250

300

350

400

450

500


per node


per node


per node

ns/

day

PME-JAC_NVE

7.1X

8.4X9.9X

42

GB-Myoglobin on K80s



(Broadwell) CPUs





288.47

339.45

0

50

100

150

200

250

300

350

400



ns/

day

GB-Myoglobin

10.0X

11.8X

43

GB-Myoglobin on P100s PCIe



(Broadwell) CPUs





483.37

561.94

0

100

200

300

400

500

600


per node


per node

ns/

day

GB-Myoglobin

16.7X

19.5X

44

GB-Myoglobin on P100s SXM2



(Broadwell) CPUs





534.28

639.37

0

100

200

300

400

500

600

700


per node


per node

ns/

day

GB-Myoglobin

18.5X

22.2X

45

GB-Nucleosome on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

0.40

5.84

11.31

20.55

0

5

10

15

20

25




ns/

day

GB-Nucleosome

14.6X

28.3X

51.4X

46

GB-Nucleosome on P100s PCIe



(Broadwell) CPUs





11.91

22.77

39.91

45.92

0

5

10

15

20

25

30

35

40

45

50


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

ns/

day

GB-Nucleosome

29.8X

56.9X

99.8X

114.8X

47

GB-Nucleosome on P100s SXM2



(Broadwell) CPUs





13.36

25.53

46.2948.29

0

10

20

30

40

50

60


per node


per node


per node


per node

ns/

day

GB-Nucleosome

33.4X

63.8X

115.7X

120.7X

48

Rubisco-75K on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

0.01

0.35

0.69

1.34

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6




ns/

day

Rubisco-75K

35.0X

69.0X

134.0X

49

Rubisco-75K on P100s PCIe



(Broadwell) CPUs





0.71

1.40

2.69

4.20

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

ns/

day

Rubisco-75K

71.0X140.0X

269.0X

420.0X

50

Rubisco-75K on P100s SXM2



(Broadwell) CPUs





0.80

1.57

3.06

4.46

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0


per node


per node


per node


per node

ns/

day

Rubisco-75K

80.0X

157.0X

306.0X

446.0X

AMBER 14

52

AMBER 14 vs. AMBER 12

Courtesy of

Scott Le Grand

From GTC 2014

presentation

53

AMBER 14; large P2P and small Boost Clocks impacts

2 x Xeon E5-2690 [email protected] + 4 xTesla K40@745Mhz (no P2P)

2 x Xeon E5-2690 [email protected] + 4 xTesla K40@875Mhz (no P2P)

2 x Xeon E5-2690 [email protected] + 4 xTesla K40@745Mhz (P2P)

2 x Xeon E5-2690 [email protected] + 4 xTesla K40@875Mhz (P2P)

Series1 125.77 132.97 196.68 215.18

125.77132.97

196.68

215.18

0

50

100

150

200

250

ns/d

ay

AMBER 14 (ns/day) on 4x K40; P2P and Boost Clocks ImpactDHFR NVE PME, 2fs Benchmark (CUDA 6.0, ECC off)

Boost

P2P

Boost

No P2P

No Boost

P2PNo Boost

No P2P

5454

AMBER Performance Over Time

Courtesy of

Scott Le Grand

From GTC 2014

presentation

55

Cellulose on K40s, K80s and M6000s


The blue node contains Dual Intel E5-2698 [email protected], 3.6GHz Turbo CPUs

The green nodes contain Dual Intel E5-2698 [email protected], 3.6GHz Turbo CPUs + either NVIDIA Tesla K40@875Mhz, Tesla K80@562Mhz (autoboost), or Quadro

M6000@987Mhz GPUs

1.93

8.96

7.87

11.76

10.49

13.67

15.3814.90

0

4

8

12

16

20

1 HaswellNode

1 CPU Node+ 1x K40

1 CPU Node+ 0.5x K80

1 CPU Node+ 1x K80

1 CPU Node+ 1x M6000

1 CPU Node+ 2x K40

1 CPU Node+ 2x K80


Sim

ula

ted T

ime (

ns/

day)

PME-Cellulose_NVE

4.1X

6.1X

5.4X

8.0X7.7X

4.6X

7.1X

56

Factor IX on K40s, K80s and M6000s




M6000@987Mhz GPUs

9.68

40.48

33.59

50.7047.80

61.18 60.93

66.89

0

10

20

30

40

50

60

70

80

1 HaswellNode

1 CPU Node+ 1x K40


1 CPU Node+ 1x K80


1 CPU Node+ 2x K40

1 CPU Node+ 2x K80


Sim

ula

ted T

ime (

ns/

day)

PME-FactorIX_NVE

3.5X

5.2X5.0X

6.4X6.3X

7.0X

4.2X

57

JAC on K40s, K80s and M6000s




M6000@987Mhz GPUs

37.38

134.82

121.30

174.34

161.53

200.34

225.34219.83

0

50

100

150

200

250

1 HaswellNode

1 CPU Node+ 1x K40


1 CPU Node+ 1x K80


1 CPU Node+ 2x K40

1 CPU Node+ 2x K80


Sim

ula

ted T

ime (

ns/

day)

PME-JAC_NVE

3.2X

4.7X

4.3X

5.4X

6.0X 5.9X

3.6X

58

Cellulose on M40s


The blue node contain Single Intel Xeon E5-2698 [email protected] (Haswell) CPUs

The green nodes contain Single Intel Xeon E5-2697 [email protected] (IvyBridge)

CPUs + Tesla M40 (autoboost) GPUs

1.07

10.12

14.40

15.90

0

2

4

6

8

10

12

14

16

18

1 Node 1 Node +1x M40 per node

1 Node +2x M40 per node


Sim

ula

ted T

ime (

ns/

Day)

PME - Cellulose_NPT

9.5X

13.5X14.9X

59

Cellulose on M40s

1.07

10.50

15.41

17.13

0

2

4

6

8

10

12

14

16

18




Sim

ula

ted T

ime (

ns/

Day)

PME - Cellulose_NVE

9.8X

14.4X

16.0XRunning AMBER version 14




60

FactorIX on M40s

5.38

46.90

67.37

72.96

0

10

20

30

40

50

60

70

80




Sim

ula

ted T

ime (

ns/

Day)

PME - FactorIX_NPT

8.7X

12.5X

13.6XRunning AMBER version 14




61

FactorIX on M40s

5.47

49.33

73.00

80.04

0

10

20

30

40

50

60

70

80

90




Sim

ula

ted T

ime (

ns/

Day)

PME - FactorIX_NVE

9.0X

13.3X14.6X





62

JAC on M40s

20.88

149.40

211.97

226.63

0

50

100

150

200

250




Sim

ula

ted T

ime (

ns/

Day)

PME - JAC_NPT

7.2X

10.2X10.9X





63

JAC on M40s

21.11

157.68

230.18

246.15

0

50

100

150

200

250

300




Sim

ula

ted T

ime (

ns/

Day)

PME - JAC_NVE

7.5X

10.9X

11.7X





64

Myoglobin on M40s

9.83

232.20

300.86

322.09

0

50

100

150

200

250

300

350




Sim

ula

ted T

ime (

ns/

Day)

GB - Myoglobin

23.6X

30.6X32.8X





65

Nucleosome on M40s

0.13

4.67

9.05

16.11

0

2

4

6

8

10

12

14

16

18




Sim

ula

ted T

ime (

ns/

Day)

GB - Nucleosome

35.9X

69.6X

123.9X





66

TrpCage on M40s

408.88

831.91

551.36

464.63

0

100

200

300

400

500

600

700

800

900




Sim

ula

ted T

ime (

ns/

Day)

GB - TrpCage2.03X

1.3X

1.1X





67

Recommended GPU Node Configuration for AMBER Computational Chemistry

Workstation or Single Node Configuration

# of CPU sockets 2

Cores per CPU socket 6+ (1 CPU core drives 1 GPU)

CPU speed (Ghz) 2.66+

System memory per node (GB) 16

GPUs Kepler K20, K40, K80, P100

# of GPUs per CPU socket1-4

GPU memory preference (GB) 6

GPU to CPU connection PCIe 3.0 16x or higher

Server storage 2 TB

Network configuration Infiniband QDR or better

Scale to multiple nodes with same single node configuration67

July 2016

CHARMM DOMDEC-GUI

69

CHARMM DOMDEC-GUI 465 K System Benchmark

Running CHARMM version c40a1

The blue node contains Dual Intel Xeon E5-2698 [email protected] GHz (Haswell) CPUs

The green nodes contain Dual Intel Xeon E5-2698 [email protected] GHz (Haswell) CPUs + Tesla K80 (autoboost) GPUs

Benchmarks were done based on the STANDARD CHARMM c40a1 version by the Yang group (FSU), who

is responsible for possible benchmarking error.

0.36

2.15

0

1

2

3

4

1 Haswell node 1 node + 1x K80 per node

ns/

day

465 K System (Her1_HER1_membrane)

6.0X

*Higher is better

70




The green nodes contain Dual Intel Xeon E5-2698 [email protected] GHz (Haswell) CPUs + Tesla K80 (autoboost) GPUs



0.18

1.43

0.0

0.5

1.0

1.5

2.0

1 Haswell node 1 node + 1x K80 per node

ns/

day

534 K System (POPC_PSPC_CHL1mixture)

*Higher is better

8.0X

71




The green nodes contain Dual Intel Xeon E5-2698 [email protected] GHz (Haswell)

CPUs + Tesla M40 GPUs



16.00

59.68

0

20

40

60

80

1 Haswell node 1 node + 1x M40 per node

ns/

day

20 K System (Crambin)

*Higher is better

3.7X

72







is responsible for possible benchmarking error.3.90

25.08

0

5

10

15

20

25

30

35


ns/

day

61 K System (GlnBP)

6.4X

*Higher is better

73








0.36

2.27

0

1

2

3

4


ns/

day

465 K System (Her1_HER1_membrane)

*Higher is better

6.3X

October 2016

GROMACS 2016

75

Erik Lindahl (GROMACS developer) video

76

Water 1.5M on K80s

Running GROMACS version 2016


(Broadwell) CPUs



NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

2.79

5.22

6.14

0

1

2

3

4

5

6

7

1 Broadwell node 1 node + 2x K80 per node 1 node + 4x K80 per node

ns/

day

Water 1.5M

1.9X

2.2X

77

Water 3M on K80s



(Broadwell) CPUs




1.32

2.66

3.05

0

1

1

2

2

3

3

4

1 Broadwell node 1 node + 2x K80 per node 1 node + 4x K80 per node

ns/

day

Water 3M

2.0X

2.3X

78

Water 1.5M on M40s



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla M40 (autoboost) GPUs


2.79

6.15

7.60

0

1

2

3

4

5

6

7

8

1 Broadwell node 1 node + 2x M40 per node 1 node + 4x M40 per node

ns/

day

Water 1.5M

2.2X

2.7X

79

Water 3M on M40s



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla M40 (autoboost) GPUs


1.32

2.97

3.94

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

1 Broadwell node 1 node + 2x M40 per node 1 node + 4x M40 per node

ns/

day

Water 3M

2.3X

3.0X

80

Water 1.5M on P40s



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla P40 GPUs


2.79

6.60

8.07

0

1

2

3

4

5

6

7

8

9

1 Broadwell node 1 node + 2x P40 per node 1 node + 4x P40 per node

ns/

day

Water 1.5M

2.4X

2.9X

81

Water 3M on P40s



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla P40 GPUs


1.32

3.36

4.19

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

1 Broadwell node 1 node + 2x P40 per node 1 node + 4x P40 per node

ns/

day

Water 3M

2.5X

3.2X

82

Water 1.5M on P100 PCIes



(Broadwell) CPUs




2.79

6.34

7.11

0

1

2

3

4

5

6

7

8

1 Broadwell node 1 node + 2x P100 PCIe (16GB)per node

1 node + 4x P100 PCIe (16GB)per node

ns/

day

Water 1.5M

2.3X

2.5X

83

Water 3M on P100 PCIes



(Broadwell) CPUs




1.32

3.16

3.43

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1 Broadwell node 1 node + 2x P100 PCIe (16GB)per node

1 node + 4x P100 PCIe (16GB)per node

ns/

day

Water 3M

2.4X

2.6X

February 2017

GROMACS 5.1.2

85

Water 1.5M on K80s

Running GROMACS version 5.1.2


(Broadwell) CPUs




Turbo] (Broadwell)

3.04

3.49

5.75

0

1

2

3

4

5

6

7



ns/

day

Water 1.5M

1.1X

1.9X

86

Water 1.5M on P100s PCIe

3.04

4.39

6.967.21

0

2

4

6

8

10


per node


per node


per node

ns/

day

Water 1.5M

1.4X

2.3X2.4X



(Broadwell) CPUs





87

Water 1.5M on P100s SXM2



(Broadwell) CPUs





3.04

4.11

6.70

7.18

7.88

0

1

2

3

4

5

6

7

8

9


per node

1 node +2x 100 SXM2

per node


per node


per node

ns/

day

Water 1.5M

1.4X

2.2X2.4X

2.6X

88

Water 3M on K80s

1.38

1.59

2.98

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5



ns/

day

Water 3M

1.2X

2.2X



(Broadwell) CPUs




Turbo] (Broadwell)

89

Water 3M on P100s PCIe

1.38

1.96

3.43

3.80

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0


per node


per node


per node

ns/

day

Water 3M

1.4X

2.5X

2.8X Running GROMACS version 5.1.2


(Broadwell) CPUs





90

Water 3M on P100s SXM2



(Broadwell) CPUs





1.38

1.84

3.50

3.82

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5


per node


per node


per node

ns/

day

Water 3M

1.3X

2.5X2.8X

91

Recommended GPU Node Configuration for GROMACS Computational Chemistry


# of CPU sockets 2

Cores per CPU socket 6+


System memory per socket (GB) 32

GPUs Kepler K20, K40, K80

# of GPUs per CPU socket

1x

Kepler GPUs: need fast Sandy Bridge or Ivy Bridge, or

high-end AMD Opterons

GPU memory preference (GB) 6

GPU to CPU connection PCIe 3.0 or higher

Server storage 500 GB or higher

Network configuration Gemini, InfiniBand91

February 2017

HOOMD-Blue 1.3.3

93

lj-liquid on K80s

Running HOOMD-Blue version 1.3.3


(Broadwell) CPUs





1324.84

1594.37

1942.12

0

500

1000

1500

2000

2500




avg

tim

e s

teps/

sec

lj-liquid

4.1X

4.9X

5.9X

94

lj-liquid on P100s PCIe



(Broadwell) CPUs





2912.66

3217.68

0

500

1000

1500

2000

2500

3000

3500


per node


per node

avg

tim

est

eps/

sec

lj-liquid

8.9X

9.9X

95

lj-liquid on P100s SXM2

326.52

3129.11

3397.74

0

500

1000

1500

2000

2500

3000

3500

4000


per node


per node

avg

tim

est

eps/

sec

lj-liquid

9.6X

10.4XRunning HOOMD-Blue version 1.3.3


(Broadwell) CPUs





96

lj_liquid_512k on K80s



(Broadwell) CPUs





220.10

334.59

526.47

0

100

200

300

400

500

600




avg

tim

est

eps/

sec

lj_liquid_512k

5.1X

7.7X

12.1X

97

lj_liquid_512k on P100s PCIe



(Broadwell) CPUs





398.12

534.54

770.18

1045.50

0

200

400

600

800

1000

1200


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

avg

tim

est

eps/

sec

lj_liquid_512k

9.2X

12.3X

17.7X

24.1X

98

lj_liquid_512k on P100s SXM2



(Broadwell) CPUs





443.74

568.51

793.36

1119.76

0

200

400

600

800

1000

1200


per node


per node


per node


per node

avg

tim

est

eps/

sec

lj_liquid_512k

10.2X

13.1X

18.3X

25.8X

99

lj_liquid_1m on K80s



(Broadwell) CPUs





109.54

181.42

303.00

0

50

100

150

200

250

300

350




avg

tim

est

eps/

sec

lj_liquid_1m

5.0X

8.2X

13.7X

100

lj_liquid_1m on P100s PCIe



(Broadwell) CPUs





204.67

294.88

465.58

672.46

0

100

200

300

400

500

600

700

800


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

avg

tim

est

eps/

sec

lj_liquid_1m

9.3X

13.4X

21.1X

30.5X

101

lj_liquid_1m on P100s SXM2



(Broadwell) CPUs





221.02

315.07

488.04

707.73

0

100

200

300

400

500

600

700

800


per node


per node


per node


per node

avg

tim

est

eps/

sec

lj_liquid_1m

10.0X

14.3X

22.1X

32.1X

102

Microsphere on K80s



(Broadwell) CPUs





64.87

98.43

166.74

0

20

40

60

80

100

120

140

160

180




avg

tim

est

eps/

sec

microsphere

3.7X

5.6X

9.5X

103

Microsphere on P100s PCIe



(Broadwell) CPUs





145.71

179.54

257.58

371.24

0

50

100

150

200

250

300

350

400


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

avg

tim

est

eps/

sec

microsphere

8.3X

10.2X

14.7X

21.2X

104

Microsphere on P100s SXM2



(Broadwell) CPUs





151.51

186.01

271.21

384.72

0

50

100

150

200

250

300

350

400

450


per node


per node


per node


per node

avg

tim

est

eps/

sec

microsphere

8.6X10.6X

15.5X

21.9X

105

Polymer on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

362.19

975.14

1209.45

1518.99

0

200

400

600

800

1000

1200

1400

1600




avg

tim

est

eps/

sec

polymer

2.7X

3.3X

4.2X

106

Polymer on P100s PCIe



(Broadwell) CPUs





362.19

1999.64

2143.15

2480.70

0

500

1000

1500

2000

2500

3000


per node


per node


per node

avg

tim

est

eps/

sec

polymer

5.5X

5.9X

6.8X

107

Polymer on P100s SXM2



(Broadwell) CPUs





362.19

2111.99

2272.27

2651.56

0

500

1000

1500

2000

2500

3000


per node


per node


per node

avg

tim

est

eps/

sec

polymer

5.8X

6.3X

7.3X

108

Quasicrystal on K80s



(Broadwell) CPUs





502.53

767.90

1280.44

0

200

400

600

800

1000

1200

1400




avg

tim

est

eps/

sec

quasicrystal

6.4X

9.8X

16.3X

109

Quasicrystal on P100s PCIe



(Broadwell) CPUs





851.29

1199.64

1791.41

2261.72

0

500

1000

1500

2000

2500


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

avg

tim

steps/

sec

quasicrystal

10.9X15.3X

22.9X

28.9X

110

Quasicrystal on P100s SXM2



(Broadwell) CPUs





939.53

1249.90

1940.29

2429.68

0

500

1000

1500

2000

2500

3000


per node


per node


per node


per node

avg

tim

steps/

sec

quasicrystal

24.8X

31.0X

12.0X

16.0X

111

Triblock-copolymer on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

361.42

953.01

1170.47

1492.01

0

200

400

600

800

1000

1200

1400

1600




avg

tim

est

eps/

sec

triblock-copolymer

2.6X

3.2X

4.1X

112

Triblock-copolymer on P100s PCIe



(Broadwell) CPUs





361.42

1999.14

2155.27

2456.09

0

500

1000

1500

2000

2500

3000


per node


per node


per node

avg

tim

est

eps/

sec

triblock-copolymer

5.5X6.0X

6.8X

113

Triblock-copolymer on P100s SXM2



(Broadwell) CPUs





361.42

2132.922253.83

2587.91

0.00

500.00

1000.00

1500.00

2000.00

2500.00

3000.00


per node


per node


per node

avg

tim

est

eps/

sec

triblock-copolymer

5.9X

6.2X

7.2X

February 2017

LAMMPS 2016

115

Atomic-Fluid Lennard-Jones 2.5 Cutoff on K80s

Running LAMMPS version 2016


(Broadwell) CPUs



0.37

0.57

0.00

0.20

0.40

0.60

0.80

1.00


1/se

conds

Atomic-Fluid Lennard-Jones 2.5 Cutoff

1.5X

116

Atomic-Fluid Lennard-Jones 2.5 Cutoff on P100s PCIe



(Broadwell) CPUs



0.37

0.62

0.00

0.20

0.40

0.60

0.80

1.00


per node

1/se

conds


1.7X

117

Atomic-Fluid Lennard-Jones 2.5 Cutoff on P100s SXM2



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla P100 SXM2 (autoboost) GPUs

0.37

0.64

0.00

0.25

0.50

0.75

1.00

1 Broadwell node 1 node + 2x P100 SXM2 per node

1/se

conds


1.7X

118

Atomic-Fluid Lennard-Jones 5.0 Cutoff on K80s



(Broadwell) CPUs





0.14

0.26

0.36

0.00

0.20

0.40

0.60

0.80

1.00




1/se

conds


1.4X2.6X

3.6X

119

Atomic-Fluid Lennard-Jones 5.0 Cutoff on P100s PCIe



(Broadwell) CPUs





0.22

0.350.37 0.38

0.00

0.20

0.40

0.60

0.80

1.00


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

1/se

conds


2.2X

3.5X 3.7X 3.8X

120

Atomic-Fluid Lennard-Jones 5.0 Cutoff on P100s SXM2



(Broadwell) CPUs





0.22

0.36

0.41

0.00

0.25

0.50

0.75

1.00


per node


per node


per node

1/se

conds


2.2X3.6X

4.1X

121

Course-grain Water on K80s



(Broadwell) CPUs



0.00437 0.00444

0.0000

0.0020

0.0040

0.0060

0.0080

0.0100


1/se

conds

Course-grain Water

1.0X

122

Course-grain Water on P100s PCIe



(Broadwell) CPUs



0.0044

0.0061

0.0093

0.0000

0.0010

0.0020

0.0030

0.0040

0.0050

0.0060

0.0070

0.0080

0.0090

0.0100


per node


per node

1/se

conds

Course-grain Water

1.4X

2.1X

123

Course-grain Water on P100s SXM2



(Broadwell) CPUs



0.0044

0.0069

0.0110

0.0000

0.0020

0.0040

0.0060

0.0080

0.0100

0.0120


per node

1 node +8x 100 SXM2

per node

1/se

conds

Course-grain Water

1.6X

2.5X

124

EAM on K80s



(Broadwell) CPUs





0.02

0.04

0.07

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07




1/se

conds

EAM

2.0X

4.0X

7.0X

125

EAM on P100s PCIe



(Broadwell) CPUs





0.03

0.05

0.08

0.13

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

1/se

conds

EAM

3.0X

5.0X

8.0X

13.0X

126

EAM on P100s SXM2



(Broadwell) CPUs





0.03

0.05

0.08

0.13

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14


per node


per node


per node


per node

1/se

conds

EAM

3.0X

5.0X

8.0X

13.0X

127

Gay-Berne on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

0.010.02

0.03

0.04

0.00

0.01

0.02

0.03

0.04

0.05




1/se

conds

Gay-Berne

2.0X

3.0X

4.0X

128

Gay-Berne on P100s PCIe



(Broadwell) CPUs





0.01

0.02

0.04

0.05

0.00

0.01

0.02

0.03

0.04

0.05


per node


per node


per node

1/se

conds

Gay-Berne

2.0X

4.0X

5.0X

129

Gay-Berne on P100s SXM2



(Broadwell) CPUs





0.01

0.02

0.04

0.05

0.00

0.01

0.02

0.03

0.04

0.05

1 Broadwell node 1 node +1x SXM2per node

1 node +2x SXM2per node

1 node +4x SXM2per node

1/se

conds

Gay-Berne

2.0X

4.0X

5.0X

130

Rhodopsin on K80s



(Broadwell) CPUs




Turbo] (Broadwell)

0.22 0.22

0.31

0.38

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40




1/se

conds

Rhodopsin

1.4X

1.7X

131

Rhodopsin on P100s PCIe



(Broadwell) CPUs





0.22

0.29

0.33

0.48

0.52

0.00

0.10

0.20

0.30

0.40

0.50

0.60


(16GB) per node


(16GB) per node


(16GB) per node


(16GB) per node

1/se

conds

Rhodopsin

1.3X1.5X

2.2X

2.4X

132

Rhodopsin on P100s SXM2



(Broadwell) CPUs





0.22

0.30

0.38

0.490.50

0.00

0.10

0.20

0.30

0.40

0.50

0.60


per node


per node


per node


per node

1/se

conds

Rhodopsin

1.4X

1.7X

2.2X 2.3X

133

Recommended GPU Node Configuration for LAMMPS Computational Chemistry


# of CPU sockets 2

Cores per CPU socket 6+


System memory per socket (GB) 32

GPUsGTX Titan X,

Kepler K20, K40, K80, M40

# of GPUs per CPU socket 1-2

GPU memory preference (GB) 6+

GPU to CPU connection PCIe 3.0 or higher

Server storage 500 GB or higher

Network configuration Gemini, InfiniBand

Scale to thousands of nodes with same single node configuration13

3

July 2017

NAMD 2.12

135

APOA1 on K80s

Running NAMD version 2.12


(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla K80 (autoboost) GPUs3.45

14.92

17.73

0

4

8

12

16

20



ns/

day

APOA1

136

APOA1 on P100s PCIe



(Broadwell) CPUs


Turbo] (Broadwell) CPUs + Tesla P100 PCIe (16GB) GPUs3.45

22.58 22.85

0

4

8

12

16

20

24

28


(16GB) per node


(16GB) per node

ns/

day

APOA1

137

APOA1 on P100s SXM2



(Broadwell) CPUs



3.45

22.98 23.44 23.87

0

5

10

15

20

25

30


per node


per node


per node

ns/

day

APOA1

138

F1ATPASE on K80s



(Broadwell) CPUs



1.15

4.81

6.27

0

2

4

6

8



ns/

day

F1ATPASE

139

F1ATPASE on P100s PCIe



(Broadwell) CPUs



1.15

7.346.99

7.40

0

2

4

6

8

10


(16GB) per node


(16GB) per node


(16GB) per node

ns/

day

F1ATPASE

140

F1ATPASE on P100s SXM2



(Broadwell) CPUs



1.15

7.116.85

7.11

0

2

4

6

8

10


per node


per node


per node

ns/

day

F1ATPASE

141

STMV on K80s



(Broadwell) CPUs



0.292

1.274

2.085

0.0

0.5

1.0

1.5

2.0

2.5

3.0



ns/

day

STMV

142

STMV on P100s PCIe



(Broadwell) CPUs



0.29

2.15

2.32

0.0

0.5

1.0

1.5

2.0

2.5

3.0


(16GB) per node


(16GB) per node

ns/

day

STMV

143

STMV on P100s SXM2



(Broadwell) CPUs



0.292

2.077

0.0

0.5

1.0

1.5

2.0

2.5

3.0


per node

ns/

day

STMV

NAMD 2.11 – Up to 2X Faster

145

New GPU features in NAMD 2.11

• GPU-accelerated simulations up to twice as fast as NAMD 2.10

• Pressure calculation with fixed atoms on GPU works as on CPU

• Improved scaling for GPU-accelerated particle-mesh Ewald calculation

• CPU-side operations overlap better and are parallelized across cores.

• Improved scaling for GPU-accelerated simulations

• Nonbonded force calculation results are streamed from the GPU for better overlap.

• NVIDIA CUDA GPU-acceleration binaries for Mac OS X

Selected Text from the NAMD website

146

NAMD 2.11 is up to 2x faster

0

5

10

15

20

25

1 Node 2 Nodes 4 Nodes

Sim

ula

ted T

ime (

ns/

day)

APoA1 (92,224 atoms)

1.2X

1.6X2.0X

NAMD 2.10 & NAMD 2.11 contain Dual Intel E5-2697 [email protected] (IvyBridge) CPUs + 2 Tesla K80 (autoboost) GPUs

147

NAMD 2.11 APoA1 on 1 and 2 nodes


The blue nodes contain Dual Intel E5-2698 [email protected] (Haswell) CPUs

The green nodes contain Dual Intel E5-2698 [email protected] (Haswell) CPUs + Tesla

K80 (autoboost) GPUs

2.77

11.67

16.99

5.22

19.73

24.31

0

5

10

15

20

25

1 Node 1 Node +1x K80

1 Node +2x K80

2 Nodes 2 Nodes +1x K80

2 Nodes +2x K80

Sim

ula

ted T

ime (

ns/

day)

APoA1(92,224 atoms)

4.2X

6.1X

3.8X

4.7X

148

NAMD 2.11 APoA1 on 4 and 8 nodes





10.27

20.64

23.52

16.85

27.83 27.74

0

5

10

15

20

25

30


4 Nodes +2x K80


8 Nodes +2x K80

Sim

ula

ted T

ime (

ns/

day)

APoA1 (92,224 atoms)

2.0X

2.3X1.7X 1.6X

149

NAMD 2.11 is up to 1.8x faster

0

2

4

6

8

10


Sim

ula

ted T

ime (

ns/

day)

F1-ATPase (327,506 atoms)

1.1X

1.8X 1.4X


150

NAMD 2.11 F1-ATPase on 1 and 2 nodes





0.94

3.87

6.11

1.86

7.23

10.58

0

5

10

15


1 Node +2x K80


2 Nodes +2x K80

Sim

ula

ted T

ime (

ns/

day)

F1-ATPase(327,506 atoms)

4.1X

6.5X

3.9X

5.7X

151

NAMD 2.11 F1-ATPase on 4 and 8 nodes





3.63

11.66

12.62

6.88

14.22

15.74

0

5

10

15

20


4 Nodes +2x K80


8 Nodes +2x K80

Sim

ula

ted T

ime (

ns/

day)

F1-ATPase(327,506 atoms)

3.2X

3.5X2.1X

2.3X

152

NAMD 2.11 is up to 1.5x faster

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0


Sim

ula

ted T

ime (

ns/

day)

STMV (1,066,628 atoms)

1.5X

1.1X

1.5X


153

NAMD 2.11 STMV on 1 and 2 nodes



The green nodes contain Dual Intel E5-2698 [email protected] CPUs (Haswell) + Tesla


0.23

1.03

1.75

0.46

1.98

3.27

0

1

2

3

4


1 Node +2x K80


2 Nodes +2x K80

Sim

ula

ted T

ime (

ns/

day)

STMV(1,066,628 atoms)

4.5X

7.6X4.3X

7.1X

154

NAMD 2.11 STMV on 4 and 8 nodes



The green nodes contain Dual Intel E5-2698 [email protected] CPUs (Haswell) + Tesla


0.90

3.61

4.54

1.74

5.86

6.24

0

2

4

6

8


4 Nodes +2x K80


8 Nodes +2x K80

Sim

ula

ted T

ime (

ns/

day)

STMV (1,066,628 atoms)

4.0X

5.0X

3.4X

3.6X

155

Benefits of MD GPU-Accelerated Computing

• 3x-8x Faster than CPU only systems in all tests (on average)

• Most major compute intensive aspects of classical MD ported

• Large performance boost with marginal price increase

• Energy usage cut by more than half

• GPUs scale well within a node and/or over multiple nodes

• K80 GPU is our fastest and lowest power high performance GPU yet

Try GPU accelerated MD apps for free – www.nvidia.com/GPUTestDrive

Why wouldn’t you want to turbocharge your research?

http://www.nvidia.com/GPUTestDrive

Dec. 19, 2016

Molecular Dynamics (MD) on GPUs

157

GPU-Accelerated Quantum Chemistry Apps

Abinit

ACES III

ADF

BigDFT

CP2K

GAMESS-US

Gaussian

GPAW

LATTE

LSDalton

MOLCAS

Mopac2012

NWChem

Green Lettering Indicates Performance Slides Included

GPU Perf compared against dual multi-core x86 CPU socket.

Quantum SuperChargerLibrary

RMG

TeraChem

UNM

VASP

WL-LSMS

Octopus

ONETEP

Petot

Q-Chem

QMCPACK

Quantum Espresso

molecular dynamics (md) on gpus · supercharger library*, vasp & more green* = application...

Documents

molecular dynamics (md) on gpus · supercharger library, vasp & more green = application...