DISCOBOLUSDISCOBOLUS

DESIGNS

Memory for Memory for

De o y oe o y o

Telecommunications,Telecommunications,the Backbone of the Cloudthe Backbone of the Cloud

Bill GervasiBill GervasiNovember 2011November 2011

Care &Care &Care &Care &Care &Care &

F Y CF Y CF Y C2

Feeding for Your CloudFeeding for Your CloudFeeding for Your Cloud

Memory comes in many flavors

Defined to meet the widest market

Sometimes this means

Compromise …Compromise …

3

For some reason

Suppliers pursue customers withSuppliers pursue customers with

MMoney…

4

Market segments such as PC, server, etc get the features they need

This drives a mass market and low price

However, other applications sometimes need to work around the approved

5

features…

Let’s look at memory architectures for…y

Line Card

B tServer

6

Brouter

Line Card• Fairly small memory capacity• Cache data packets some Line Card • Cache data packets, some

low level MAC

Solder-down: just enough to fit the buffer

72b-SO-DIMM: Fast,72b SO DIMM: Fast,rotate & latch socket

7

Mini-RDIMM: Fast,horizontal or vertical direct insert socket 7

BroutersBrouters

Controller

Address Header Address Header

Controller

Payload Payload

PacketBuffer

Memory

AddressLookupTable

Memory

8

Memory

Brouter RequirementsBrouter Requirements

• High speed, low latency header g p , ylookup: typically SRAM like QDR

• Medium (2-8GB) DRAM bufferMedium (2 8GB) DRAM buffer–A few ms of data buffering–Exactly 50% reads 50% writesExactly 50% reads, 50% writes–Mostly 64-byte min packets

9

DDR3/DDR4 Breaks > 1600DDR3/DDR4 Breaks 1600

• DDR3/DDR4 designed for large / g gsystems with many ranks of memoryy

• Interleaving ranks hides– tFAW = four activations windowtFAW = four activations window– tRRD = row to row access time–Short and long bank group limits

10

–Short and long bank group limits

Bank 0 Bank 1

Bank 2 Bank 3

Bank 4 Bank 5

Bank 6 Bank 7

Bank 4

DRAMRank 0

ACT0/0 ACT0/1 ACT0/2 ACT0/3 RD0/0 RD0/1 RD0/2 RD0/3ACT0/2 RD0/2

What the controller wants to do

Commands

Data

ACT0/4ACT0/4 RD0/4

Data

ACT0/0 ACT0/1 ACT0/2 ACT0/3 RD0/0 RD0/1 RD0/2 RD0/3ACT0/2 RD0/2

What the controller is forced to do

Commands ACT0/4 RD0/4

Performancelost

11

DataGap caused by core

limitations

Bank 0 Bank 0Bank 1 Bank 0 Bank 1 Bank 0 Bank 1Bank 1

Bank 3

Bank 4 Bank 5

Bank 2 Bank 3

Bank 4 Bank 5

Bank 2 Bank 3

Bank 4 Bank 5

Bank 2 Bank 3

Bank 5Bank 4

Bank 2

DRAMBank 6 Bank 7

DRAMBank 6 Bank 7

DRAMBank 6 Bank 7

DRAMBank 6 Bank 7

Rank 0 Rank 1 Rank 2 Rank 3

Servers avoid core penalties by adding ranksInterleave rank accesses

Increases memory capacity at the same time

12

How Brouters Avoid Core Penalties

ACT0/0 ACT0/1 RD0/1Commands RD0/0

• Can’t add ranks: too much wasted memory

Data

• Can t add ranks: too much wasted memory• Solution: Lots of 32 bit buses with 16-word

bursts (16x4 = 64 byte packet)• Pushes core limitations aside by increasing

time spent on data transactions• However, increases pin count

13

However, increases pin count• Can’t use standard memory modules

What do Servers Eat for Lunch?What do Servers Eat for Lunch?

RDIMMs

1, 2, 4 ranks

Every rank loads the bus

LRDIMMs

1, 2, 4, 8 ranks

One load on bus

14

Problem with multi-drop bus:Controller

p

Decrease in frequency with loading

RDIMM: Currently limited toDDR3-1066 with two DIMMsDDR3 1066 with two DIMMs4 Ranks per DIMM

LRDIMM: Currently limited toDDR3-1066 with three DIMMs

15

8 Ranks per DIMM

DDP Limits 4Rank RDIMMsDDP Limits 4Rank RDIMMs

Face down, face up dual die k (DDP) DRAMDRAM

Bond

package (DDP) DRAMs– High inductance– Asymmetrical

DRAMDRAM

y

Face up DRAM die with redistribution layers

Hi h itDRAMDRAM

Bond RDL

– High capacitance

JEDEC ballout optimizes

DRAM

16

monolithic DRAMs, not DDP

3DS: Game Changer?3DS: Game Changer?

DRAMDRAMDRAM Through Silicon

Via technologyDRAM

Logic interface

Via technology

Presents one load on busI f f RDIMMIncreases frequency of RDIMM8 Ranks/channel @ 1600 enabled

17

3DS: Dead End for Now3DS: Dead End for Now

Known good diewith speed bin yieldp y

is not feasibleuntil 2016

at the earliest…

18

Invensas DFD: Game Changer!Invensas DFD: Game Changer!DRAM

DRAM

Dual Face Down

Reduced impedance, reduced capacitanceSymmetric signaling ballout tuned for DDP2 DIMMs x 4 ranks at DDR3 1600 enabled (1 5V)

19

2 DIMMs x 4 ranks at DDR3-1600 enabled (1.5V)2 DIMMs x 4 ranks at DDR3L-1333 enabled (1.35V)

SummarySummary

JEDEC provides a foundation for mass market expansion

Niche applications get inventive to leverage the resulting technologies

Inventive companies find ways to

20

p yimprove on the standards

DISCOBOLUSDISCOBOLUS

DESIGNSDThank youThank you

Questions?Questions?