WildFireWildFire: A Scalable Path : A Scalable Path for SMPsfor SMPs

Erick Hagersten and Michael Erick Hagersten and Michael KosterKoster

Sun Microsystems Inc.Sun Microsystems Inc.Presented by Terry Arnold IIPresented by Terry Arnold II

IntroductionIntroductionWhat was the goal?What was the goal?How did they achieve it?How did they achieve it?CMRCMRHASHASCompetitive ComparisonsCompetitive ComparisonsResultsResultsQuestionsQuestions

The GoalThe Goal In the past people have been In the past people have been

skeptical about the ability of SMPs to skeptical about the ability of SMPs to continue to scale due to their continue to scale due to their bandwidth limitationsbandwidth limitations

The trend has been to switch to cc-The trend has been to switch to cc-NUMANUMA

To improve the scalability of SMP To improve the scalability of SMP technologiestechnologies

Cc-NUMA issuesCc-NUMA issuesGreat scalability but have less than Great scalability but have less than

optimal “access patterns”optimal “access patterns”Require high software optimization Require high software optimization

for capacity and conflict missesfor capacity and conflict missesNon trivial scheduling, etc. (resource Non trivial scheduling, etc. (resource

and memory management)and memory management)

How?How?The answer is the same as the answer The answer is the same as the answer

to all engineering problems, that is, to all engineering problems, that is, throwing new acronyms at the problemthrowing new acronyms at the problem

Coherent Memory Replication (CMR)Coherent Memory Replication (CMR)Hierarchical Affinity Scheduling (HAS)Hierarchical Affinity Scheduling (HAS)Both of these exploit locality as a Both of these exploit locality as a

means of increasing performance (that means of increasing performance (that is for OLTP workloads)is for OLTP workloads)

The OverviewThe Overview

The Acronyms: CMRThe Acronyms: CMR S-COMA with fixed home locations for each S-COMA with fixed home locations for each

addressaddress Shadow physical pagesShadow physical pages Coherence at hardware level (64 byte)Coherence at hardware level (64 byte) Things start out cc-NUMA and changed into Things start out cc-NUMA and changed into

CMR based on hardware counters that CMR based on hardware counters that monitor memory access patternsmonitor memory access patterns

Limitations – memory-resident pages and Limitations – memory-resident pages and large physical pages can only be replicated large physical pages can only be replicated explicitlyexplicitly

The Acronyms: HASThe Acronyms: HASSchedules in the following way:Schedules in the following way:Last processor it ran onLast processor it ran onSame node processorSame node processorRemote node processor (when load Remote node processor (when load

balances exceeds “threshold”)balances exceeds “threshold”)

ImplementationImplementation2 ASICs – NIAC (coherence), NIDC (bit 2 ASICs – NIAC (coherence), NIDC (bit

sliced interconnect)sliced interconnect)These improve upon latency of a These improve upon latency of a

switchswitchNIAC – Interface and Global-NIAC – Interface and Global-

Coherence LayerCoherence LayerTranslators and CountersTranslators and Counters

CompetitionCompetition The SGI Origin and The SGI Origin and

Sequent’s NUMA-QSequent’s NUMA-Q

Results 1Results 1

Results 2Results 2

Questions?Questions?

Is this “solution” too dependent on the software (kernel modifications)?

How compatible are CMR and HAS with the other DSM solutions?