TAKE IT TO THE NTH

Frederic VecovenSun Microsystems

SunFire range of servers

System Components

Ultra−SPARCCPU &

compilers

Fireplane Shared

MemoryInterconnect

Applications &Middleware

Clustering & Networking

Operating Environment

Storage

Cache Coherency Basics

CPU 1 CPU 2 CPU 3

1. Read to Share

Memory

CPU 4

Coherency blocks(Aligned 32, 64, or 128 bytes)

2. Read to Share

4. Read to Share3. Read

to Own

5. Writeback

3. Invalidate 5. Invalidate

Cache Coherency Types

1. Broadcast (snoopy) coherency\\

P P P P\\

M M M MIO IO

\\

Snooping coherence domai n

PPPP\\

AgentMMMM

IOIO

P

M

IO

Processor

Memory

I/O controller

2. Point−to−point (directory) coherency

Glo

bal

Inte

rcon

nect

Snooping coherence domai n

Agent

Sun Generation Timeline

‘90 ‘95 ‘00 Now

UltraSPARC−I / UPA

SuperSPARC / XDBus

Cypress SPARC / MBus

UltraSPARC−III / Fireplane

Developmen t

Productio n

CPU core / Interconnect1

2

3

4

UltraSPARC−V5

Increasing CPU Integration

Memory controller

Cypress

IUCypress

FPU Cache tagsCache controller and Coherency controller

Ultra−IIIProcessor

Coherency controller

Memory controller

External cache

controller

External cache tags

Ultra−IProcessor

Coherency controller

Memory controller

External cache

controller

External cache tags

Coherency controller

Memory controller

SuperSparcProcessor

External cache controller and cache tags

Cypress SPARC / MBus — 1990

SuperSPARC / XDBus — 1993

UltraSPARC−I / UPA — 1996

UltraSPARC−III / Fireplane — 2000

1

2

3

4

Generation 1: MBus1

0.1 GBps4 CPUs

MBusCypress SPARC

1990

SnoopyCoherency

CPU

MEM

IO

1 bus

Generation 2: XDBus1 2

0.1 GBps4 CPUs

1.28 GBps64 CPUs

MBusCypress SPARC

1990

XDBusSuperSPARC

1993

SnoopyCoherency

SnoopyCoherency

CPU

MEM

IO

1, 2, or 4buses

1 bus

CPU

MEM

IO

Generation 3: UPA31 2

0.1 GBps4 CPUs

1.28 GBps64 CPUs

12.8 GBps64 CPUs

SnoopyCoherency

MBusCypress SPARC

1990

XDBusSuperSPARC

1993

UPAUltraSPARC−I/II

1996

SnoopyCoherency

SnoopyCoherency

CPU

MEM

IO

CPU

MEM

IOD

ata

Cro

ssba

ror

Data bus or xbar

1 or 4address buses

1, 2, or 4buses

1 bus

CPU

MEM

IO

31 42

Generation 4: Fireplane

0.1 GBps4 CPUs

1.28 GBps64 CPUs

43 /172 GBps72–106 CPUs

12.8 GBps64 CPUs

DirectoryCoherency

SnoopyCoherency

MBusCypressSPARC

1993

XDBusSuperSPARC

1993

UPAUltraSPARC−I/II

1996

SnoopyCoherency

SnoopyCoherency

CPU

MEM

IO

Dat

a C

ross

bar

FireplaneUltraSPARC−III

2000

CPU

MEM

IO

Add

ress

Cro

ssba

r

Dat

a C

ross

bar

CPUMem

IO

CPUMem

IO

CPUMem

IO

Res

pons

e C

ross

bar

Snoopy

9.6 GBps

or

Data bus or xbar

1 or 4address buses

1, 2, or 4buses

1 bus

CPU

MEM

IO

Worldwide Unix Server Market

1997 1998 1999 2000 2001$0

$5

$10

$15

$20

$25

$30

Fac

tory

Rev

enue

(B

illio

ns)

YearYears are 4 quarters endin g June 30Source: IDC Sept 01

System CPU capacity:

17−32

5−8

1−2

3−4

9–16

33−6465–128>128

32%

30%

38%

Solaris on SPARC on Fireplane Interconnect

Sun Fire Servers

280R

• 2 CPUs

• 8 GB RAM

• 4 PCI slots

V880

• 8 CPUs

• 32 GB RAM

• 9 PCI slots

3800−6800

• 8– 24 CPUs

• 64–192 GB RAM

• 16–32 PCI or12–16 cPCI slots

• 1 to 4 domains

15K

• 72–106 CPUs

• 576 GB RAM

• Up to 72 hot−swap PCI slots

• 1 to 18 domains

1−level Fireplane: Small Server

Workstation or Small server

• CPU/Memory pair

• PCI bridge

CPU Data Switch

CPU

CPU

PCIbridge

33 MHz slot(s)

66 MHz slot

Memory

Memory2.4 GBps2.4 GBps

2.4 GBps2.4 GBps

1.2 GBps

Level 0

2−level Fireplane: Workgroup

CPU Data Switch

CPU Memory

CPU

CPU Data Switch

CPU

CPU

Address Repeater

6x6Data

Switch

PCIbridge 66 MHz slot

PCIbridge 66 MHz slot

Memory

Memory

Memory2.4 GBps2.4 GBps4.8 GBps

2.4 GBps2.4 GBps

1.2 GBps

Workgroup server

• 4 Dual−CPU/ Memory boards

• 2 PCI bridges

Level 1 Level 0

33 MHz slot(s)

33 MHz slot(s)

3−level Fireplane: Mid−size

CPU Data Switch

CPU Memory

CPU

CPU Data Switch

CPU

CPU

Address Repeater

Data Switch

PCIbridge 66 MHz slot

PCIbridge

Data Switch

10x10Data

Switch66 MHz slot

Memory

Memory

Memory

Address Repeater

10−wayAddress Repeater

2.4 GBps2.4 GBps4.8 GBps4.

8 G

Bps

2.4 GBps2.4 GBps

2.4 GBps 1.2 GBps

4 Fireplane Switch Boards

I/O Assembly

6 Uniboards

4 I/O boards

CPU/Mem Uniboard

Level 2 Level 1 Level 0

33 MHz slot(s)

33 MHz slot(s)

Level 2 Level 1 Level 04−level Fireplane: Large Server

CPU Data Switch

CPU Memory

CPU

CPU Data Switch

CPU

CPU

Address Repeater

Data Switch

PCIbridge 66 MHz slot

PCIbridge

Data Switch

Data Switch

18x18 Data Xbar

18x18 Response

Xbar

18x18 Address

Xbar

66 MHz slot

Memory

Memory

Memory

Address Repeater

Address Repeater

2.4 GBps2.4 GBps4.8 GBps4.

8 G

Bps

2.4 GBps2.4 GBps

2.4 GBps4.8 GBps 1.2 GBps

Level 3

Expander Board

18 B

oard

sets

I/O Assembly

Centerplane

33 MHz slot(s)

33 MHz slot(s)

CPU/Mem Uniboard

CPU/Memory Uniboar d

Four banks of 8 SDRAM DIMMs

4 Data Switch ASICs

DataControl ASIC

AddressASIC

Boot bus ASIC

Boot bus

ASIC

Two sets of 8 CPU Data Switch ASICs

19.35"

16.5"C

PU

E$ DIMMs

CP

U

CP

U

CP

U

Pow

erP

ower

Pow

er

I/O Assemblies4 slot cPCI 6 slot cPCI

8 slot PCI 4 slot hot−swap PCI

Mid Range CabinetsSun Fire 68003 x Sun Fire 3800

3 x (8 CPUs + 12cPCI slots ) 24 CPUs + 32 PCI slot s

Sun Fire 4800/4810

12 CPUs + 16 PCI slot s

High−end Cabinet4 Fan Trays

18 Boardsets with18 CPU−Memory Boards

and 18 I/O orDual CPU Boards

4 Fan Trays

2 System Controllers

Six Dual Input 4 KWAC to 48 volt DCPower Supplies

75"

33"65"

SF 15K Components

CPU−Memory board (18)

I/O or MaxCPU board (18)

System Controller board (2)

System Controller peripheral board (2)

Control expander frame (2)

System expander frame (18)

Expander board (18)

Fan trays (8)

Fan Center−planes (8)

Power centerplane

Logic centerplane

Control expander sockets (2)

System expander sockets (18)

Centerplane ASICs (20)

(One side shown )

Mid−range Micro BenchmarkParallel pointer−chasin gUPA to Fireplane Generation

250

350

450

550

650

0 4 8 12 16 20 24

Mem

ory

late

ncy

(ns)

Processors

Sun Fire 6800(Switch)

SunEnterprise6500 (Bus)

(Loweris better)

0

1

2

3

4

5

6

0 4 8 12 16 20 24

Mem

ory

band

wid

th (

GB

ps)

Processors

(Higheris better)

SunEnterprise

Linea

r

6500 (Bus)

Sun Fire 6800(Switch)

Sun Interconnect Generations

MBus XDBus UPA Fireplane

Year (in mid−size servers) 1991 1993 1996 2001

System clock (MHz) 40 50–55 83–100 150

Coherency type

Packet switching

Address & Data

Coherency block (bytes)

Clocks/snoop

Address bus BW (GBps)

# Address buses

Datapath width (bytes)

Wiring

Broadcast Broadcast & point−to−point

Circuit Packet switched

Together Separate

32 64

16 11 2 1

0.08 0.3 3.0 9.6

1 4 4 18

8 16 32

Bused Switched

Max data B/W (GBps) 0.08 1.3 12.8 172

Mid: BusedHigh: Switched

31 42

Sun Snooping Bandwidth Progress

0.04

0.1

1

10

40

1990 1992 1994 1996 1998 2000 2002 2004

Bro

adca

st−b

us

band

wid

th (

GB

ps)

Year of first shipment in medium−sized servers

1. MBus

2. XDBus

3. UPA

Doubling every 18 months trend line

4. Fireplane

Now 5. US−V bus

Snoopy Bandwidth Progress

0.1

1

10

US−V

XDBus• 50 MHz (11 clock snoop)

• Packet switched

• 64−byte cache line

• Multiple buses

UPA

Fireplane

MBus• 40 MHz

Five Sun SMPGenerations

.05

1990 1993 1996 2000

Sno

opy

band

wid

th (

GB

ps)

1

2

3

4

• 100 MHz

• 2−clock snoop

• Separate address & data

• Routers & domains

• 150 MHz

• 1−clock snoop

• 2−level coherency

5

2003