technology and architectures for future large‐scale ......interconnect technology roadmap long...
TRANSCRIPT
![Page 1: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/1.jpg)
TechnologyandArchitecturesforFutureLarge‐ScaleCompu;ngSystems
RickStevens
ArgonneNa;onalLaboratory
TheUniversityofChicago
![Page 2: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/2.jpg)
![Page 3: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/3.jpg)
Argonne’sIBMBlueGene/P–556TFs
![Page 4: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/4.jpg)
CrayXT5atORNL–1.6Pflop/s
Jaguar Total XT5 XT4Peak Performance 1,645 1,382 263AMD Opteron Cores 181,504 150,176 31,328System Memory (TB) 362 300 62Disk Bandwidth (GB/s) 284 240 44Disk Space (TB) 10,750 10,000 750Interconnect Bandwidth(TB/s)
532 374 157
ThesystemswillbecombinedaRer
acceptanceofthenewXT5upgrade.Each
systemwillbelinkedtothefilesystemthrough
4x‐DDRInfiniband
![Page 5: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/5.jpg)
Tradi;onalSourcesofPerformanceImprovementareFlat‐Lining(2004)
• New Constraints– 15 years of exponential
clock rate growth has ended
• Moore’s Law reinterpreted:– How do we use all of those
transistors to keepperformance increasing athistorical rates?
– Industry Response: #coresper chip doubles every 18months instead of clockfrequency!
FigurecourtesyofKunleOlukotun,LanceHammond,HerbSu_er,andBurtonSmith
Transistorscon;nuetoscaleClockhasleveledoff(2‐4Ghz)Powerleveledoff(~100W‐200W)Performanceperclock(2‐4ops/clock)
![Page 6: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/6.jpg)
Mul;corecomesinawidevariety– Mul;pleparallelgeneral‐purposeprocessors(GPPs)– Mul;pleapplica;on‐specificprocessors(ASPs)
“The Processor is thenew Transistor”
[Rowen]
Intel 4004 (1971):4-bit processor,2312 transistors,
~100 KIPS,10 micron PMOS,
11 mm2 chip
Sun Niagara8 GPP cores (32 threads)
Intel®XScale
™ Core32K IC32K DC
MEv210
MEv211
MEv212
MEv215
MEv214
MEv213
Rbuf64 @128B
Tbuf64 @128BHash
48/64/128Scratch
16KBQDR
SRAM2
QDRSRAM
1
RDRAM1
RDRAM3
RDRAM2
GASKET
PCI
(64b)66
MHz
IXP280IXP28000 16b16b
16b16b
1188
1188
1188
1188
1188
1188
1188
64b64b
SPI4orCSIX
Stripe
E/D Q E/D Q
QDRSRAM
3E/D Q1188
1188
MEv29
MEv216
MEv22
MEv23
MEv24
MEv27
MEv26
MEv25
MEv21
MEv28
CSRs-Fast_wr
-UART-Timers
-GPIO-BootROM/SlowPort
QDRSRAM
4E/D Q1188
1188
Intel Network Processor1 GPP Core
16 ASPs (128 threads)
IBM Cell1 GPP (2 threads)
8 ASPs
Picochip DSP1 GPP core248 ASPs
Cisco CRS-1188 Tensilica GPPs
![Page 7: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/7.jpg)
![Page 8: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/8.jpg)
What’sNext?
Source: Jack Dongarra, ISC 2008
![Page 9: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/9.jpg)
E3AdvancedArchitectures–Findings(2007)
• Exascalesystemsarelikelyfeasibleby2017±2• 10‐100Millionprocessingelements(mini‐cores)withchipsas
denseas1,000corespersocket,clockrateswillgrowslowly• 3Dchippackaginglikely• Large‐scaleop;csbasedinterconnects• 10‐100PBofaggregatememory• >10,000’sofI/Ochannelsto10‐100Exabytesofsecondary
storage,diskbandwidthtostoragera;osnotop;malforHPCuse
• HardwareandsoRware(OS)basedfaultmanagement• Simula;onandmul;plepointdesignswillberequiredto
advanceourunderstandingofthedesignspace• Achievableperformanceperwa_willlikelybetheprimary
metricofprogress
![Page 10: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/10.jpg)
TopTechnicalChallenges
• PowerConsump;on– Proc/mem,I/O,op;cal,memory,delivery
• Chip‐to‐ChipInterfaceScaling– pin/wirecount⇒3Dpackaging
• Package‐to‐PackageInterfaces– Scalableinterconnects
• CostPressureinOp;csandMemory
• FaultTolerance– FITratesandfaultmanagementconcepts
• E.g.Inplacespares,chipkill,etc.– Reliabilityofirregularlogic,designprac;ce
Exascalesystemsbuiltfromtoday’stechnologywouldconsumeoveraGigawa_
Exascalesystemsbuiltfromtoday’stechnologywouldcostoveraGigabuck
Exascalesystemsbuiltfromtoday’stechnologywouldhaveaMTBFof10minutes
![Page 11: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/11.jpg)
LookingouttoExascaleConcurrencywillbeDoublingevery18months
![Page 12: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/12.jpg)
SystemsScalingProjec;ons
![Page 13: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/13.jpg)
ITRSRoadmapforLogicDevices
PowerDensity20x‐30xthan2005Voltageassump;onis.5v‐.7v
![Page 14: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/14.jpg)
DarpaExascaleStudy
ConcludedthatitwillbeaMajorchallengetogettoSustainedExaopsperformanceLevelsby2020
![Page 15: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/15.jpg)
TotalSystemConcurrencyLargestmachinesnowhavecombinedsystem’slevelconcurrencyapproaching1M
![Page 16: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/16.jpg)
ThreadLevelConcurrencyEachthreadmustdomul;pleopera;onsperclockcycle
BG“doublehummer”
VectorMachines
Superscalar
![Page 17: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/17.jpg)
ParallelismandLocalityTrendsAlgorithmsarepushingintheoppositedirec;onasthehardwaree.g.advancedalgorithmsarebecomingless“local”whilehardwareisbecomingmoreparallelandmorelocal
![Page 18: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/18.jpg)
PowerConstrainedClockRateClock=Power_Density/(Capacitance_per_device*Transistor_Density*V2
dd)
Powerdensity(i.e.cooling)willlimittheclockrateasfeaturesizesshrink
![Page 19: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/19.jpg)
GflopsperWa_mustgrow1000x(0.1⇒100)toachieveExascale
![Page 20: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/20.jpg)
PowerandFPUscalingneededtoReachExa‐opera;ons/s(onlyflops)
Picojoule/FLOP FPUsperExaflop
![Page 21: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/21.jpg)
InterconnectTechnologyRoadmapLongdistanceInterconnectsWillbeop;cal
Op;calmayreplaceCopperforoncarrierInterconnects
InterconnectpowerwillGrowto10%‐30%oftotalpower
![Page 22: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/22.jpg)
HeatRemovalApproachesHighlylikelyallfuturemachinesatscalewillbeliquidcooled
![Page 23: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/23.jpg)
High‐EndPackagingOp;ons
![Page 24: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/24.jpg)
3DPackagingExamples
![Page 25: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/25.jpg)
SecondaryStorageProjec;ons(Scratchat25x,Archiveat200x)
20,000to>1,000,000diskdrivesforsecondarystoragePhasechangememorymayreplacediskin2018;meframe
![Page 26: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/26.jpg)
Evolu;onofStorage
![Page 27: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/27.jpg)
![Page 28: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/28.jpg)
AggressiveStrawman
~200,000nodeseachwith~750cores=~150McoresMemoryis16GBfora~5Tflopnodewith~1000threadsTotalsystemmemoryis~4PB.
![Page 29: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/29.jpg)
SystemsScalingProjec;ons
![Page 30: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/30.jpg)
TheBo_omLine
• Levelsofconcurrency(106⇒109) 1000x• ClockrateofCore(1‐4GHz⇒1‐4GHz) 1x• RAMperCore(1‐2GBnowto<1GB) <1x• TotalNumberofcores(200K⇒100M) 500x• Numberofcorespernode(8⇒>64‐512) >8x‐64x• AggressiveFaultManagementinHWandSW• I/Ochannels(>103⇒105) >100x• PowerConsump;on(10MW⇒40MW‐150MW)4x‐16x• ProgrammingModel(MPI⇒MPI+X)
![Page 31: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/31.jpg)
ParallelProgrammingModels:TwentyYearsandCoun;ng
• Inlarge‐scalescien;ficcompu;ngtodayessen;allyallcodesaremessagepassingbased.Addi;onallymanywillusesomeformofmul;threadingonSMPormul;corenodes.
• Mul;coreischallengingprogrammingmodelsbuttherehasnotyetemergedadominatemodeltoaugmentmessagepassing
• Thereisaneedtoiden;fynewhierarchicalprogrammingmodelsthatwillbestableoverlongtermandcansupporttheconcurrencydoublingpressure
![Page 32: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/32.jpg)
QuasiMainstreamProgrammingModels
• C,Fortran,C++andMPI
• OpenMP,pthreads
• (CUDA,RapidMind,Cn)OpenCL
• PGAS(UPC,CAF,Titanium)
• HPCSLanguages(Chapel,Fortress,X10)
• HPCResearchLanguagesandRun;me
• HLL(ParallelMatlab,GridMathema;ca,etc.)
![Page 33: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/33.jpg)
April2009 Programmingmodels,Salishanconference 33
Clusters
w/Accelerators
Poten;alMigra;onPathsProgrammingModels
BaseandMPI
Base/OpenMPandMPI
CUDA
Base/OpenCL
libspe
Base/OpenMP+andMPI
Base/OpenMP
C/C++/Fortran/Java(Base)
Charm++
RapidMind
GEDAE/Streamingmodels
PGAS/APGAS
ReduceMPItasks
Makeportable,open
Makeportable,open
HarnessacceleratorsPr
oduc;vity,maintainability
Scale,produc;vity,maintainability
Scale
Scaleb
eyond
oneno
de
Scale
Green:open,widelyavailable*Blue:somewhereinbetweenRed:proprietary*OpenCLavailabilitypredicted
Base/OpenCLandMPI
Harne
ssaccelerators
Harnessaccelerators
ALF
Makeportable,open
Produc
;vity
![Page 34: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/34.jpg)
FromVivekSarkarSC08ExascaleWorkshop
![Page 35: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/35.jpg)
FromVivekSarkarSC08ExascaleWorkshop
![Page 36: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/36.jpg)
ar
FromVivekSarkarSC08ExascaleWorkshop
![Page 37: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/37.jpg)
![Page 38: Technology and Architectures for Future Large‐Scale ......Interconnect Technology Roadmap Long distance Interconnects Will be opcal Opcal may replace Copper for on carrier Interconnects](https://reader036.vdocuments.us/reader036/viewer/2022071423/611d54b0d10c6a590376eac6/html5/thumbnails/38.jpg)