system packaging solution for future high … eps shunichi kikuchi.pdffuture high performance...

2018 IEEE 68th Electronic Components and Technology Conference │ San Diego, California │ May 29 – June 1, 2018 1

May 31, 2018Shunichi KikuchiFujitsu Limited

System Packaging Solution for Future High Performance Computing


• Future HPC spec. from the transition of TOP500• Possible System Packaging for the future

Outline

For the advent of Exascale Computing‐ A prediction from TOP500 history ‐


1.0E‐01

1.0E+00

1.0E+01

1.0E+02

1.0E+03

1.0E+04

1.0E+05

1.0E+06

1.0E+07

1.0E+08

1.0E+09

1.0E+10SUM

TOP

#500

1PF

1EF

1TF

“TOP”s have been recorded with 3.5‐4 year delay from “SUM”s. 1EF will be recorded in 2021‐2022.

ʼ97

ʻ08

Max

imal

LIN

PACK

Per

form

ance

(Fl

ops)

6 times4 times

845PF

The number of CPUs and Accelerators (TOP10)


1

10

100

1,000

10,000

100,000

1,000,000

Minimum

Maximum

On averageBG/L

ASCI Red

BG/LBG/Q

K

The number of them seems to have saturated since 2007.

Trend of the number of cores in system (TOP10)


Performance improvement needs to increase the number of cores.

1

10

100

1,000

10,000

100,000

1,000,000

10,000,000

100,000,000

Minimum

Maximum

On average

Multicore CPU6 out of TOP10

BG/L

Tianhe-2TaihuLight

# of CPUs and Accelerators : 10,000 – 100,000+ pcsPerformance per each : 100,000 – 10,000 GFlops[Exascale era]

(2core CPU)

(12core CPU)

(260core CPU)

(57core ACC)

Performance per power


25‐50 GF/W will be a key factor to implement 1EF.

Linpack Top1

TOP10 feature @ Nov. 2017


CPU Accelerator(bootable) CPU+Accelerator

JCAHPC

NSCC

JAMSTEC

ORL

LANL

NERSCLLNL

NRCPC

6.1GF/W by 260core CPU

14.2GF/W by 1984coreAcc.

8.6GF/W by GPU & CPU8.6GF/W by GPU & CPU

TaihuLight

Sequoia

K computer

Shoubu

TitanOak‐forest PACS

Cori

TrinityTianhe‐2

Piz Daint

No.1

No.6

No.10

No.2

No.3 CSCS

No.4

No.5

No.7

No.8

No.9

There are three types of configurations.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Other

Intel

Cray

IBM

InfiniBand

Ethernet

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Other

Intel

Cray

IBM

InfiniBand

Ethernet

I/O interconnects in system


Top500

Top20

Still major

Custom

Major

Both InfiniBand and Ethernet are popular as a whole.

Communication links for short‐distance


ACSI Purple ‐3K links @2.5G in 2005Roadrunner ‐40K links @5G in 2008Power 775 ‐60K fibers/rack @10G in 2011Sequoia ‐620K links @10G in 2011

Transition of interconnects in system


Optical interconnects have been applied between racks and between shelfs.They will be near to CPUs and Accelerators, depending on system requirements, technology evolution (including costs).

Speedconnected -10Gbps 25Gbps 50Gbps-

Distance ComponentExampleSystems

K computerFX10 FX100 Future

systemBetween

racks Elec Opto Opto - 100ｍ Cable

Between shelfs Elec Opto Opto - 3ｍ Cable

Between boards in

shelf

-- Opto Opto- 1m

Cable-- -- Opto Cable

Elec -- Opto Board, Back plane

On board

Elec -- Elec

- 0.3ｍ

Board-- -- Opto Board(Fiber sheet, Waveguide)

-- Elec/Opto Elec/Opto Board&cable

-- -- Elec Cable-- -- Opto Cable

Interconnects on board


Altera, Avago, 2012

PETRA/NEDO, 2016

Cu cableOptical cables,Fiber sheet orpolymer waveguides

Cu patternCu pattern

board

CPU/ACC package Fiber sheet orpolymer waveguides

Related technologies in the past ECTCs


Compass EOS@ECTC2014

336ch, 64Gb/s/mm2

(168*8Gb/s=1.344Tb/s)348ch, 15Gb/s/mm2

IBM@ECTC2016 AIST/PETRA@ECTC2016

96ch, 2.4Tb/s

CPU/Accelerator chip trend (in general)


0.1

1.0

10.0

1995 2000 2005 2010 2015 2020Year

Clock Frequency (GHz) Die size (mm2)

200

300

400

500

600

700

800

900

1995 2000 2005 2010 2015 2020Year Year

0

50

100

150

200

250

300

350

400

450

500

1995 2000 2005 2010 2015 2020

Heat dissipation (Watts)

There seems no significant growth in clock frequency since 2007.Die size has been reached to the manufacturing limit.Heat dissipation may be controlled toward green computing.

Power delivery trend


Source: Daimler

Exascaler Inc. @SC17 nVIDIA

Mounted DDCs(ex. 48V1V)

Creating electrical power from

At Package/Module Level

At Datacenter LevelHydrogenNatural gas etc.

Possible Processor Package and Module


High density package Standard I/F module

Package substrate

Silicon‐Photonics

LD

Processor stacking(optional)

Transition of package substrate technology


Technology progress and design optimization may be able to mitigate the current patterning and cost issues.

2010 20202015

1μm

100μm

10μm

0.1μm

year

Patternpitch

Chip(Semiconductor)

Interposer

Substrate

Future prospect of system packaging


High density rack mountingHigh density system packaging

CPU/MEM package

Acceleration package

socket

High performance card with standard interface Highly dense rack mounting


Double sided cooling unit(CPU/MEM/DDC)

CPU

GPU

DLU

FPGA

DAU

FPGA

Rackmountable

CDU

Cooling platejackets

Future prospect of system packaging

Quick connecting/disconnecting plug

Cooling technology


A wide range of choices should be prepared in order to match with customers’ requirements :

Coolant availabilityLimitation of ambient temperatureLimitation of power consumptionLimitation of installation space

Liquid immersion cooling・Submersion of all parts・Natural convection・Forced circulation・Boiling

Liquid circulation cooling・Hot water・Chilled water・Micro-channel・Impinging

IBM

Air-forced cooling・Heat sink・Heat sink w/ heat pipe・Vapor and liquid loop・Vapor chamber・Thermo-siphon

airradiator

pump

Coolingunit

liquid circulation, and liquid immersionAir‐forced,

Cooling from the viewpoint of system packaging


A lot of PCB units

Liquid immersion

High density main board

Liquid circulation

17GF/W@2017

1.E-04

1.E-02

1.E+00

1.E+02

1.E+04

1.E+06

1990 2000 2010 2020 2030 2040 2050 2060

1.85x106 GF/W

2039 2055

BG/L

Earth SimlRoadrunner

Shoubu

NWT0.012GF/W@2002

Toward Big‐data era and the beyond


Source: FORTUNE: July 10, 2017

Human Brain37 PF (*1)/20W=1.85x106 GF/W

200PF@2018, 10times/4yrs, 200*10^(21/4)=36ZF@203920MW /20W = 1Million Processors, 37PF*1M=37ZF@2039

If technology is continuously evolved, we may reach…

1 billion times as fast as Earth Simulator(36TF) in 2002‐2004

Performan

ce/Pow

er

(*1) Source: 36.8PF, 3.2PBytesBest of H+ Magazine 2008‐2010 Volume 1

Tera Peta Exa Zeta

17GF/W@2017

1.E-04

1.E-02

1.E+00

1.E+02

1.E+04

1.E+06

1990 2000 2010 2020 2030 2040 2050 2060

1.85x106 GF/W

2039 2055

BG/L

Earth SimlRoadrunner

Shoubu

NWT0.012GF/W@2002

Toward Big‐data era and the beyond


Source: FORTUNE: July 10, 2017

Human Brain

Or else?Quantum Computing?Digital Annealing?

37 PF (*1)/20W=1.85x106 GF/W

200PF@2018, 10times/4yrs, 200*10^(21/4)=36ZF@203920MW /20W = 1Million Processors, 37PF*1M=37ZF@2039

If technology is continuously evolved, we may reach…

Which type of computing can be near to human brains?

Quantum Annealing?

1 billion times as fast as Earth Simulator(36TF) in 2002‐2004

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

Performan

ce/Pow

er

(*1) Source: 36.8PF, 3.2PBytesBest of H+ Magazine 2008‐2010 Volume 1

Conventional HPC?

Tera Peta Exa Zeta

Various device configurations for different issues


Health care Traffic Relaxation Sustainable Food and Agriculture

Reproducible Energy Education for everyone Sustainable Environment

It is difficult to solve and find a better goal by using one type of HPC. A suitable system packaging solution can be chosen in the coming several years for each.

Fujitsu’s High‐end HPC Development


Fujitsu has provided HPC systems with original technologies, developed for over 40 years, to accelerate advanced research

© RIKEN

The K computer continues to be competitive in various fields; from advanced research to manufacturing

K computer

RIKEN and Fujitsu are developing the Post‐K to achieve superior application performance

Post‐K computer

HPCGNo.1(2017)

Graph500No.1(2017)

Gordon Bell Prize Finalist

(2016)

PRIMEHPC FX10

PRIMEHPC FX100

24

system packaging solution for future high … eps shunichi kikuchi.pdffuture high performance...

Documents