sun fire x4600 m2 server architecture

SUN FIRE™ X4600 M2

SERVER ARCHITECTURE

Highly-Scalable x64 Computing Based on Powerful Dual-Core and Quad-Core AMD Opteron™ Processors

White PaperJune 2008

Sun Microsystems, Inc.

Table of Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Delivering Massive x64 Single-System Scalability . . . . . . . . . . . . . . . . . . . . . . . . . 2

The Sun Fire X4600 M2 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

System Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Enterprise-Class Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

AMD Opteron Processor Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Dual-Core AMD Opteron Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Quad-Core AMD Opteron Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Server Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

System-Level Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

CPU/Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Sun Fire X4600 M2 Server Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Enterprise-Class Operating System and Management Software . . . . . . . . . . . . . 35

Solaris 10 OS Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Integrated Lights Out Management (ILOM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Sun xVM Ops Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

For More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Executive Summary


Executive Summary

Even as the x64 computing architecture has grown in ubiquity, organizations continue

to demand new levels of scalability from their computing platforms. Databases need to

process ever-larger number of transactions. Growing data warehouses require larger

analysis efforts to extract useful information. High performance computing (HPC)

applications are increasingly pressed into service to answer our most critical business

needs and scientific inquiries. While the latest x64 processors offer very impressive

levels of performance, organizations need systems that offer balanced scalability and

performance, to preserve and capitalize on investments.

At the same time, many organizations are responding to new levels of server sprawl

and complexity that have resulted from deploying large numbers of legacy systems.

Administrative, energy, and real estate costs continue to grow unabated — severely

straining IT budgets. Increasingly, many are responding with consolidation strategies

that combine ever more powerful standard x64 systems with innovative new

approaches to virtualization. Not only does consolidation offer to improve vital resource

utilization, but it can also greatly enhance business agility by providing computational

resources where they are needed most.

The Sun Fire™ X4600 M2 server employs the latest high-performance AMD Opteron™

processors in a massively-scalable system that demonstrates Sun's proven datacenter-

centric design focus. Ideal for databases, HPC applications, and mission-critical

applications alike, this server delivers scalable and manageable x64 computing in

dense and servicable rackmount enclosures. Using the latest Dual-Core and Quad-Core

AMD Opteron processors, the Sun Fire X4600 M2 server is particularly well suited for

consolidation and virtualization initiatives. With compelling compute power, memory

capacity, and I/O density, as well as considerable energy efficiency, organizations can

live within their means even as they grow their IT infrastructure.

With a choice of operating systems — including the Solaris™ Operating System (Solaris

OS), Linux, Microsoft Windows, and VMware — the Sun Fire X4600 M2 server effectively

combines Dual-Core or Quad-Core AMD Opteron processors with balanced system

design. Large memory support, considerable I/O bandwidth, and up to eight processor

sockets give this server the capacity to handle the most demanding applications. With

datacenter operation as a fundamental design assumption, the server also offers

redundant and hot-swappable components, efficient front-to-back air flow, highly-

efficient power supplies, and built-in system management tools. Beyond supporting a

large number of processors, the Sun Fire X4600 M2 server is engineered to provide

mission-critical application availability, even as it conserves valuable energy resources

and lowers operational expenses. This document details the systems architecture of

Sun Fire X4600 M2 server, along with key software components.

2

Delivering Massive x64 Single-System Scalability


Chapter 1


To stay competitive, organizations must deploy applications quickly, delivering compute

power where it is needed most, all with increasingly little margin for error. Databases,

high performance computing applications, and the growth of Web-centric business

models are all placing significant demands on IT infrastructure. At the same time,

systems deployed in the datacenter must now offer considerable investment protection

and agility, scaling gracefully under pressure even as they deliver business-critical and

mission-critical applications. Regrettably, many datacenters have become sprawling

and complex, and most are up against very real constraints in terms of power, cooling,

and real estate.

Consolidation through virtualization has emerged as an effective strategy for

addressing the very real need for computing scalability while increasing the work that

can be done in a given power, thermal, and physical footprint. Consolidation can

improve resource utilization, reduce administrative complexity, and drive down IT costs.

Consolidating many smaller legacy servers into fewer more powerful systems can also

help to minimize administrative workloads while increasing capacity and conserving

valuable datacenter floor space. Energy costs can be drastically reduced, vastly

improving available performance relative to the amount of energy consumed.

Nonetheless, effective consolidation strategies require truly robust, scalable, and

available computing platforms.

The Sun Fire™ X4600 M2 Server

To help IT managers address the challenge of increasing capacity while managing

datacenter growth, the Sun Fire X4600 M2 server provides industry-leading scalability

and performance based on the latest AMD Opteron processors. This server features high

performance and unprecedented density in an energy-efficient and compact four-rack-

unit (4U) form-factor. With capabilities that complement the rest of the Sun™ server

product line, the Sun Fire X4600 M2 server raises the bar for 32- and 64-bit enterprise-

class computing. These systems offer:

• Best-in-Class Performance

The Sun Fire X4600 M2 server features a choice of either Dual-Core or Quad-Core

AMD Opteron processors. With a sophisticated cache hierarchy, and on-chip

memory management, these processors offer high system performance and

throughput compared with systems based on earlier-generation x86 chipsets.

Making the most of the AMD Direct Connect Architecture, the Sun Fire X4600 M2

server can house up to eight Second-Generation Dual-Core AMD Opteron

processors or Third-Generation Quad-Core AMD Opteron processors — yielding up

to 32 AMD Opteron cores in a single 4U datacenter footprint.

3



• Remarkable Compute and Memory Density

Density is the cornerstone of the Sun’s server designs, and the Sun Fire X4600 M2

server is no exception. When populated in a 40-rack unit (RU) enclosure, this

server facilitates a single rack with up to 320 cores, 640 DIMM slots, and 60 PCI

Express slots. In addition, each server can house up to four internal 2.5-inch SAS

disk drives. The Sun Fire X4600 M2 server provides the densities needed to achieve

consolidation and virtualization efficiencies. Such density facilitates the

consolidation of many smaller servers, helping to conserve real estate, lower

energy expense, and make the most of costly administrative talent. In addition,

these servers support multiple operating systems, which helps to simplify

consolidation efforts and diminish server sprawl.

• Extensive System Expandability

The ability to expand and upgrade a server over time reduces the need for

additional capital acquisitions and lowers application lifecycle costs. The Sun Fire

X4600 M2 server supports two, four, six, or eight processor sockets per system and

provides maximum memory configurations up to 256 GB (using 4 GB DIMMs). Four

Gigabit Ethernet ports are standard, as are four USB ports (two front, two back),

and one video port (HD-15). With breakthrough system densities, this server is

designed to scale to support new users, more transactions, or new 32-bit or 64-bit

applications — enhancing system longevity and increasing overall return on

investment (ROI).

• Improved Energy Efficiency

Sun offers a portfolio of eco-responsible products and computing solutions to

address a broad range of infrastructure requirements. In Sun Fire X4600 M2

servers, AMD Opteron processors incorporate new technologies that minimize

power use and enhance energy efficiency.

AMD PowerNow!

technology optimizes

processor performance relative to the power consumed, allowing CPU clock

frequency to be adjusted to the needs of applications.

Enhanced AMD PowerNow!

features in Quad-Core AMD Opteron processors offer features such as

Independent

Dynamic Core technology

and

Dual Dynamic Power Management

. High-efficiency

power supplies in the server chassis lessen overall power consumption. Variable-

speed fans, effective disk carrier design, and front-to-back air flow in the chassis

help to effectively cool the system and maintain appropriate ambient

temperatures for both the processor and the system.

• Enterprise-Class High Availability

The Sun Fire X4600 server is designed with enterprise-class reliability, availability,

and serviceability (RAS) features. To maximize uptime, systems include redundant

hot-swappable fans and can be configured with redundant hot-swappable power

supplies. Internal SAS disk drives can be configured for RAID 0 or 1, and disk drives

4



are also hot-swappable. Four integrated Gigabit Ethernet ports enhance network

availability and can be installed in failover configurations. On-board system

management tools encourage proactive remote monitoring and intervention.

• Tightly-Integrated Management

To support out-of-band management, the Sun Fire X4600 M2 server incorporates

an Integrated Lights Out Management (ILOM) service processor. This built-in

hardware-based management functionality allows administrators to monitor and

manage systems remotely, letting them take corrective action as necessary to

minimize unplanned downtime.

Sun Fire X4600 M2 servers combine best-in-class performance with noteworthy

compute, memory, and I/O capacities. As a result, these systems are designed to scale

up, scale out, and scale within, enabling deployment in a wide range of application

architectures:

•

Scale-up architectures

— With up to 32 cores available per system, this server is well

suited to scale for growing workloads that deliver Web, database, and other key

infrastructure services.

•

Scale-out architectures

— With large memory capacities, significant internal storage,

four Gigabit Ethernet ports, and high-bandwidth PCI Express expansion, these servers

can scale to solve complex computing problems that demand intensive computing

power and data bandwidth.

•

Scale-within capabilities

— With the ability to support Solaris 10 virtualization

technology and VMware, Sun Fire X4600 M2 servers are ideal systems to consolidate

multiple applications within a single extensible platform.

Figure 1 illustrates the 4U Sun Fire X4600 M2 server enclosure.

Figure 1. The Sun Fire X4600 M2 server offers up to 32 AMD Opteron cores in only four rack units (4U).

Sun Fire X4600 M2 Server

5



System Comparisons

The Sun Fire X4600 M2 server complements Sun’s extended x64 server product line.

Sun provides a wide range of AMD Opteron based servers with diverse capabilities.

Table 1 compares the features of Sun Fire X4140, X4440, and X4600 M2 servers.

Table 1. Sun Fire X4140, X4440, and X4600 M2 server feature comparison

Features of the Sun Fire X4600 M2 server include:

• Support for multiple Dual-Core or Quad-Core AMD Opteron processors

• Large memory capacities (up to 256 GB)

• Internal storage, with support for four internal SAS disk drives

• Internal hardware RAID support provided

• PCI-X and PCI Express expandability

• Built-in quad Gigabit Ethernet support

• Built-in VGA video port

• Integrated Lights Out Management (ILOM) service processor and firmware

• Support for multiple operating systems

Multiple off-the-shelf configurations are available, along with a wide spectrum of

options to tailor the system for specific workload requirements.

Feature Sun Fire X4140 Server Sun Fire X4440 Server Sun Fire X4600 M2 Server

Processors One or two Dual-Core or Quad-Core AMD Opteron series 2000 processors

Two or four Dual-Core or Quad-core AMD Opteron Series 8000 processors

Two, four, six, or eight Dual-Core or Quad-core AMD Opteron Series 8000 processors

Memory capacity Up to 64 GB (1, 2, or 4 GB DDR2 DIMMs)

Up to 128 GB(1, 2, of 4 GB DDR2 DIMMs)

Up to 256 GB(1, 2, of 4 GB DDR2 DIMMs)

Maximum internal disk drives

Up to eight SFF 2.5-inch SAS 73 or 146 GB disk drives, choice of RAID HBAs

a

Up to eight SFF 2.5-inch SAS 73 or 146 GB disk drives, choice of RAID HBAs

a

Up to four SFF 2.5-inch SAS 73 or 146 GB disk drives, internal RAID

RAID Hardware RAID

0, 1, 1E, 10, 5, 5EE, 50, 6, and 60

with SAS RAID HBA

Hardware

0, 1, 1E, 10, 5, 5EE, 50, 6, and 60

with SAS RAID HBA

Hardware

0 or 1

Removable and pluggable I/O

Slimline DVD+/-RWFive USB 2.0 ports (Two front, two rear, one internal)

Slimline DVD-RFive USB 2.0 ports (Two front, two rear, one internal)

Slimline DVD/CD-RWFour USB 2.0 ports (Two front, two rear)

Expansion slots Two x8 PCI Express slots, One x16 PCI Express slot

One x4 PCI Express slot, Four x8 PCI Express slots, andOne x16 PCI Express slot

Two PCI-X slots, Four x8 PCI Express slots, andTwo x4 PCI Express slots

Ethernet Four on-board Gigabit Ethernet ports (10/100/1000Base-T)

Four on-board Gigabit Ethernet ports (10/100/1000Base-T)

Four on-board Gigabit Ethernet ports (10/100/1000Base-T)

Power supplies Dual redundant hot-swappable AC 650 W power supply units (N+N redundancy)

Dual redundant hot-swappable AC 1050 W power supply units(N+N redundancy)

Four redundant hot-swappable AC 950 W power supply units(N+N redundancy)

Fans Redundant, hot-swappable fan modules (N+N redundancy)

Redundant, hot-swappable fan modules (N+N redundancy)

Redundant, hot-swappable fan modules (N+N redundancy)

Form factor 1 rack unit (1U) 2 rack units (2U) 4 rack units (4U)

a.The RAID HBA consumes a single PCI Express slot

6



Enterprise-Class Capabilities

Ultimately, datacenter requirements go beyond mere performance and scalability.

Effective servers must provide considerable flexibility to serve a wide variety of needs in

an agile fashion. The Sun Fire X4600 M2 server is designed for datacenter operation —

running multiple operating systems and providing significant reliability, availability,

and availability (RAS) features. Ultimately, Sun’s innovative chassis design across its x64

product line contributes directly to the datacenter utility of these powerful systems.

A Choice of Operating Systems

In order to provide maximum flexibility and investment protection, Sun Fire X4600 M2

server supports a choice of operating systems, including:

• The Solaris Operating System

• The Linux operating system (64-bit Red Hat or SuSE Linux)

• Microsoft Windows

• VMware ESX Server

Table 2 lists supported OS releases supported by the Sun Fire X4600 M2 server as of this

writing. Please see

sun.com/x64

for the latest supported operating systems and

environments.

Table 2. Supported OS releases for Sun Fire X4600 M2 server

• The Solaris™ Operating System

Distributed under a commercial and open source licensing model, the Solaris 10

OS offers many innovative technologies that change the equation for

organizations needing to reduce costs, minimize complexity, and help eliminate

risk. The Solaris 10 OS is optimized for Sun systems and is supported on over one

thousand third-party x86/x64 systems. In addition, the Solaris 10 OS is free for

download without requirement to purchase a support contract, providing an

economic advantage over other community-based operating system offerings.

OS Release 32-bit or 64-bit Dual-Core Quad-Core

Solaris 10 OS 64-bit Yes Update 4

Microsoft Windows Server 2003 Standard or Enterprise Edition, SP1 or later

32/64-bit Yes Yes

Microsoft Windows Server 2008 64-bit TBD TBD

VMware ESX 3.0 64-bit Yes TBD

a

VMware ESX 3.5 64-bit Yes TBD

a

Red Hat Enterprise Linux 4, Update 4 minimum 32/64-bit Yes Yes (64-bit only)

Red Hat Enterprise Linux 5 (RHEL) 64-bit Yes Yes

SuSE Linux Enterprise Server 10, SP1 64-bit Yes Yes

a.Please see

sun.com/vmware

for the latest information on supported VMware releases

7



Taking advantage of the Solaris 10 OS can bring added flexibility and power to the

enterprise. Supporting systems from laptops and single-board computers to

datacenter and cluster installations, the Solaris 10 OS serves applications ranging

from military command and control systems, to telecommunication switch gear,

to stock trading. The Solaris 10 OS also includes more than 180 applications from

the free and open source software (F/OSS) community, and thousands of others

are freely available for download over the Internet.

Provided on all Sun systems at no charge, the Solaris OS delivers performance,

security, scalability, and reliability advantages for scale-out computing

environments. Underlying technologies, such as a high-performance networking

stack, advanced file system, and modern memory model combine to optimize the

performance of hosted applications. A suite of security features previously only

found in Sun’s military-grade Trusted Solaris™ operating system are now included

to fortify the commercial enterprise.

The Solaris OS supports near linear scalability from 1 to 72 CPUs and

addressability of up to 2

64

bytes of memory, well beyond the physical memory

limits of even Sun’s largest server. In addition, by providing the ability to

automatically recover from hardware faults, the Solaris OS provides maximum

data and application availability.systems. The Solaris 10 OS is free for download

without requirement to purchase a support contract, offering an economic

advantage over other community-based operating system offerings.

• Linux Environments

Sun offers and supports the leading Linux variants on Sun Fire x64 servers,

including Red Hat Enterprise Linux and Novell SUSE Linux Enterprise Server. As the

leader in enterprise services for UNIX®, Sun brings decades of expertise to Linux

environments. Sun support contracts for Linux provide all front-line support and

transparent access to back-line support from Red Hat and Novell.

Sun is one of the largest contributors to the open-source community. Areas of

contribution include OpenOffice.org, Mozilla, GNOME, and X.org. In addition, Sun

provides key software offerings for Linux including

– Lustre™ parallel file system

– Sun Ray™ Server Software

– Sun xVM software

– StarOffice™ productivity suite

– Sun Java™ Desktop Powered Program

– Sun Studio, Sun Java Studio Creator, and NetBeans™ IDE software

– MySQL™ database

8



• Microsoft Windows Environments

Organizations are constantly seeking to reduce the variety of platforms in the

datacenter, even when a wide range of workloads are present. To address this

need, the Sun Fire X4600 server is ideal for Microsoft SQL and Microsoft Windows

consolidation and virtualization efforts. This server has passed stringent Microsoft

compatibility test suites, achieving the Designed for Windows certification and a

listing in Microsoft Windows catalogs. Support contracts for Microsoft Windows

are also available from Sun. This certification and support demonstrates Sun’s

commitment to providing the best platforms to run not only the Solaris OS and

Linux, but Microsoft Windows as well.

• VMware

Ground-breaking virtualization solutions from VMware help improve asset

utilization, operational efficiency, and business agility. Sun offers the VMware

Infrastructure product suite on Sun hardware systems with full support from Sun.

VMware virtualization technology also combines with key Solaris 10 OS features

such as DTrace, Solaris Containers, and Solaris Predictive Self Healing software. As

a result, organizations can create breakthrough approaches to virtualization. In

fact, utilizing VMware virtual infrastructure software with the Solaris 10 OS for

consolidation projects can increase system utilization by up to ten times.

Effective and Consistent System Design for the Datacenter

Beyond the capabilities of individual systems, Sun understands that datacenters have

unique and pressing needs that require attention on the part of system designers.

Density, performance, and scalability are all essential considerations, but systems must

also fit in with modern datacenter strategies that consider power, cooling, and

serviceability. The Sun Fire X4600 M2 server shares an innovative design philosophy that

extends across Sun’s volume x64 and SPARC® server platforms.

Principals of this design philosophy include:

•

Common chassis design

— Shared chassis design leverages key system innovations

across multiple architectures, provides for common components and subassemblies,

and greatly simplifies administration for those deploying multiple processor

architectures.

•

Maximum Compute Density

— Sun’s volume servers provide leading density in terms

of CPU cores, memory, storage and I/O. This focus on density often lets Sun’s smaller

rackmount servers replace larger rackmount servers, for a considerable space

savings.

•

Common, Shared Management

— The Sun Fire X4600 M2 server is designed for ease

of management and serviceability with service processors in common with other Sun

volume server platforms. Systems and components are designed for easy

identification and hot-swap components facilitate on-line replacement.

9



•

Continued Investment Protection

— Sun designs for maximum investment

protection. Even with breakthrough x64 technology, Sun’s Solaris Binary

Compatibility Guarantee means that applications simply run without modification.

Chassis Design Innovations

Sun volume server platforms feature basic chassis designs that are shared with other

Sun x64 and SPARC server platforms. This approach not only provides a consistent look

and feel across the product line, but it simplifies administration through consistent

component placement and shared components. Beyond mere consistency, this

approach provides a datacenter design focus that places key technology where it can

make a difference for the operations.

• Enhanced System and Component Serviceability

Finding and identifying servers and components in a modern datacenter can be a

challenge. The Sun Fire X4600 M2 server is optimized for lights-out datacenter

configurations with easy-to-identify servers and modules. Color-coded operator

panels provide straightforward diagnostics. Systems are designed for deployment

in hot-isle / cold-isle multiracked deployments with both front and rear diagnostic

LEDs and Fault Remind features to help identify faulty or failed components.

Consistent connector layouts for power, networking, and management make

moving between Sun’s systems straightforward. All hot-plug components are tool-

less and easily available for serviceability. For instance, Sun Fire X4600 M2 servers

provides direct access to fan modules so that fans can be serviced without

exposing sensitive components, or causing unnecessary downtime.

• Robust Chassis, Component, and Subassembly Design

Sun’s volume servers share chassis that are carefully designed to provide

reliability and cool operation. In spite of their computational, I/O, and storage

density, Sun’s servers are able to maintain adequate cooling using conventional

technologies. Efficient modular fan assemblies keep the chassis within an effective

operating temperature range.

• Minimized Cabling for Maximized Airflow

To minimize cabling and increase reliability, Sun Fire X4600 M2 servers are

designed with a minimum of cabling. This approach not only provides more

reliable connectors, but it maximizes airflow through the chassis for more

effective cooling.

• Leading Reliability, Availability, and Serviceability (RAS)

The Sun Fire X4600 M2 server provides excellent reliability, availability, and

serviceability (RAS) characteristics. Highly-reliable parts and a relatively low total

component count minimize the opportunity for system errors.

10



The following key design elements help increase the dependability of IT services:

– Reduced parts count

– Built-in RAID capabilities

– Redundancy and hot-swap components

– Parity protection and error correction capabilities

– Integrated Lights Out Management (ILOM) service processor

– Superior energy efficiency

– Robust virtualization technology

– Comprehensive fault management

11

AMD Opteron™ Processor Technology


Chapter 2


The Sun Fire X4600 M2 server is powered by AMD Opteron processors, utilizing AMD’s

Direct Connect Architecture and NVIDIA chipsets for scalability and fast I/O throughput.

The server supports both Second-Generation Dual-Core AMD Opteron processors as well

as Third-Generation Quad-Core AMD Opteron processors. The sections that follow

describe the architecture and feature set of AMD Opteron processors.

Dual-Core AMD Opteron Processors

Second-Generation AMD Opteron processors are native Dual-Core AMD Opteron

processors that feature AMD’s Direct Connect Architecture. These processors offer a

common core architecture that is consistent across multisocket systems, and is also

consistent with previous AMD Opteron processors. This strategy helps organizations

minimize the cost of transitions while they maximize past investments in software and

hardware optimization. AMD Opteron processors are offered in three series:

• 1000 Series — Single socket

• 2000 Series — Up to two sockets

• 8000 Series — Four to Eight sockets

Sun Fire X4600 M2 servers employ 8000 Series AMD Opteron processors.

Innovative Processor Technology

The AMD Opteron processor extends the ubiquitous x86 architecture to accommodate

64-bit processing. Formerly known as x86-64, AMD’s enhancements to the x86

architecture allow seamless migration to the superior performance of 64-bit

technology. Dual-Core AMD Opteron processors offer considerable advantages,

including:

•

AMD64 technology

— AMD64 technology lets 64-bit operating systems provide full,

transparent, and simultaneous 32-bit and 64-bit platform application multitasking.

This approach lets systems run the existing installed-base of 32-bit applications and

operating systems at peak performance, while providing a 64-bit migration path.

•

Direct Connect Architecture

— AMD’s Direct Connect Architecture helps to reduce the

very real challenges and bottlenecks of system architecture.

– Memory is directly connected to the processor, optimizing memory performance

– I/O is directly connected to the processor, for more balanced throughput and I/O

– Processors are directly connected to other processors, allowing for more linear

symmetrical multiprocessing

12



•

Integrated DDR2 memory controller

— A 144-bit wide, on-chip DDR2 memory

controller provides 128 bits for data and 16 bits for ECC and Enhanced ECC

technologies, while providing low-latency memory bandwidth that scales as

processors are added.

•

AMD HyperTransport technology

— AMD HyperTransport Technology provides a

scalable bandwidth interconnect between processors, I/O subsystems, and other

chipsets.

•

Quad-core upgradeability

— AMD Opteron processors with DDR2 memory are

designed to offer a seamless upgrade path from dual-core to quad-core processors.

Similar power and thermal envelops help protect investments, letting organizations

upgrade to Quad-Core AMD Opteron processors while realizing similar power

efficiencies.

•

AMD Virtualization (AMD-V)

— AMD Virtualization reduces overhead by selectively

intercepting instructions destined for guest environments while the Direct Connect

Architecture helps guest operating systems run at near native speed. A virtualization-

aware integrated memory controller provides efficient isolation of virtual machine

memory.

•

Enhanced performance per watt

— Energy-efficient DDR2 memory uses up to 30

percent less power than DDR1 memory, and up to 58 percent less power than FB-

DIMM memory. In addition,

AMD PowerNow!

technology with Optimized Power

Management can deliver performance on demand, while minimizing power

consumption.

Dual-Core Processor Architecture

The AMD Opteron processor (Figure 2) was designed from the start for multicore

functionality, with a crossbar switch and system request interface. This approach

defines a new class of computing by combining full x86 compatibility, a high-

performance 64-bit architecture, and the economics of an industry-standard processor.

Figure 2. High-level architectural perspective of a Dual-Core AMD Opteron processor

DDR2Memory

Controller

HyperTransport 0

HyperTransport 1

HyperTransport 2

System Request Interface

Crossbar Switch

Second-Generation Dual-Core AMD Opteron

Core 1 Core 2

128 KB L1 Cache

1MB L2 Cache

128 KB L1 Cache

1MB L2 Cache

13



Enhancements of the AMD Opteron processor over the legacy x86 architecture include:

• 16 64-bit general-purpose integer registers that quadruple the general-purpose

register space available to applications and device drivers as compared to x86

systems

• 16 128-bit XMM registers provide enhanced multimedia performance to double the

register space of any current SSE/SSE2 implementation

• A full 64-bit virtual address space offers 40 bits of physical memory addressing and 48

bits of virtual addressing — supporting systems with up to 256 terabytes of memory

Each processor core has a dedicated 1 MB Level-2 cache, and both cores use the System

Request Interface and Crossbar Switch to share the Memory Controller and access the

three HyperTransport links. This sharing represents an effective approach since

performance characterizations of single-core based systems have revealed that the

memory and HyperTransport bandwidths are typically under-utilized, even while

running high-end server workloads. The Second-Generation AMD Opteron processor

integrates three HyperTransport technology links, providing a scalable bandwidth

interconnect among processors, I/O subsystems, and other chip-sets. At 16 x 16 bits and

1 GHz operation, HyperTransport technology provides support for up to 8 GB/s

bandwidth per link.

HyperTransport Technology

HyperTransport technology is a high-speed, low-latency, point-to-point link designed to

increase the communication speed between integrated circuits in computers, servers,

embedded systems, and networking and telecommunications equipment. Second-

Generation and Third-Generation AMD Opteron processors continue to use

HyperTransport technology links to provide a scalable bandwidth interconnect among

processors, I/O subsystems, and other chip sets. HyperTransport technology:

• Helps increase overall system performance by removing I/O bottlenecks typically

found in Front Side Bus (FSB) architectures, efficiently integrating with legacy buses,

increasing bandwidth and speed, and reducing latency of processors.

• Provides up to 8 GB/second bandwidth per link at 16 x 16 bits, 1 GHz operation,

offering significantly more bandwidth than most current technologies, and sufficient

bandwidth for supporting new interconnects such as PCI Express

• Uses low-latency responses and low pin counts for enhanced performance and

reliability

• Maintains compatibility with legacy PC buses while being extensible to new Systems

Network Architecture (SNA) buses

• Appears transparent to operating systems, so that peripheral drivers continue to

operate

14



Quad-Core AMD Opteron Processors

Native Quad-Core AMD Opteron processors incorporate four processor cores on a single

silicon die. Despite this innovation, Quad-Core AMD Opteron processors are electrically,

thermally, and socket-compatible with Second-Generation AMD Opteron Socket F (1207)

processors. Figure 3 provides a block-level diagram of the third-generation Quad-Core

AMD Opteron processor.

Figure 3. Third-Generation Quad-Core AMD Opteron processor block-level diagram

Quad-Core AMD Opteron processors go beyond simply adding two additional cores.

With a native multicore design, all four cores share the same silicon, and are directly

connected via AMD’s Direct Connect Architecture. Processors, I/O, and memory

controller logic are all connected to each other, aiding performance and reducing

bottlenecks. Quad-Core AMD Opteron processors provide a broad set of significant

enhancements, described in the sections that follow.

• Enhanced AMD PowerNow! Technology

Enhanced AMD PowerNow! technology provides significant power-management

advancements.

AMD CoolCore technology

can reduce energy consumption and

heat generation by turning off unused parts of the processor.

Independent

Dynamic Core technology

allows each core to vary its clock frequency depending

on the specific performance requirements of the applications it is supporting,

helping to reduce power consumption.


(formerly called “splitplane”) provides an independent power supply to the cores

and to the memory controller, allowing them to operate on different voltages,

depending on usage.

DDR2Memory

Controller

HyperTransport 0

HyperTransport 1

HyperTransport 2


Crossbar Switch

2 MB L3 Cache

512 KB L2 Cache

Core 1 Core 2 Core 4Core 3

128 KBL1 Cache

512 KBL2 Cache

128 KBL1 Cache

128 KBL1 Cache

128 KBL1 Cache

512 KBL2 Cache

512 KBL2 Cache

512 KBL2 Cache

Third-Generation Quad-Core AMD Opteron

15



• Investment Protection

Not only are Quad-Core AMD Opteron processors the first native x86 quad-core

processors, but they are the first quad-core processors designed to operate within

similar thermal and power envelops as AMD’s current Second-Generation

processors. This consistency allows simplified upgradeability and protects

organizations investments in AMD Opteron based systems with a seamless

upgrade path.

• Virtualization Enhancements

Virtualization is memory intensive, and Quad-Core AMD Opteron processors

provide exceptional memory throughput with an integrated memory controller.

AMD Virtualization introduces Rapid Virtualization Indexing (formerly called

“nested page tables”) and a tagged translation look-a-side buffer (TLB). While TLBs

exist in almost every processor architecture, AMD implemented tagged TLBs to

improve virtual to physical memory lookups from one virtual machine to another.

AMD’s Rapid Virtualization Indexing feature is designed to reduce the overhead

penalty associated with virtualization technologies by moving the process of

managing virtual memory from software to hardware. This approach reduces the

complexity of existing x86 virtualization solutions and facilitates increased

performance and efficiency for many virtualized workloads.

• Support for High Performance Computing (HPC)

A variety of features coalesce in Quad-Core AMD Opteron processors to make them

ideal for HPC workloads.

AMD Memory Optimizer Technology

increases memory

throughput by up to 50 percent compared to previous generations of the AMD

Opteron processor.

AMD Wide Floating Point Accelerator

provides 128-bit

Streaming SIMD Extensions (SSE) floating point capabilities, letting each core

simultaneously execute up to four floating point operations (FLOPS) per clock —

four times the floating-point computations of previous-generation AMD opteron

processors.

AMD Balanced Smart Cache

provides significant cache enhancements

with 128 KB of Level-1 cache, and 512 KB of Level-2 cache per core, combined with

2 MB of shared Level-3 cache shared across all cores.

Processor Design for Energy Efficiency

Power consumption continues to be one of the top concerns for managing today's

datacenters. Quad-Core AMD Opteron processors address this concern by providing

industry-leading overall power-efficiency that can deliver significant performance gains

over Dual-Core AMD Opteron processors while operating in the same thermal envelope.

All processor series — current Single-Core, Dual-Core, and Quad-Core AMD Opteron

processors — have been designed to a consistent power and thermal specification.

16



Average CPU Power (ACP)

Average CPU Power (ACP) is a metric that offers a relevant estimation of the power

consumption for Quad-Core AMD Opteron processors. ACP is determined by breaking

down multiple components of the power consumed within the processor, including the

power dedicated to the cores, the integrated memory controller, and to HyperTransport

technology links. In contrast, thermal design power (TDP) refers to the power that

processors are capable of consuming. ACP and TDP are both valid indicators of

processor power.

AMD has referenced processor power consumption based on TDP values to date.

However, ACP represents a relevant measure that reflects power consumption while

running server-class enterprise workloads. In particular, ACP is a useful metric for

datacenter operators to use when estimating power budgets to size their datacenters,

while TDP is more useful and relevant to system designers. Table 3 illustrates both TDP

and ACP for Quad-Core AMD Opteron processors.

Table 3. TDP and ADP for Quad-Core AMD Opteron processors

Enhanced AMD PowerNow! and Independent Dynamic Core Technology

Enhanced AMD PowerNow! Technology is designed to reduce power consumption of

the entire quad-core processor. The native quad-core design of Third-Generation AMD

Opteron processors lets enhanced power management address each of the four cores

independently. In particular, Independent Dynamic Core technology allows each core to

vary its frequency, based on the specific needs of the application. This ability allows for

more precise power management to reduce datacenter energy consumption and

thereby reduces total cost of ownership (TCO).

Power consumption is related to the voltage level of the voltage supply to the processor

as well as the frequency of operation. General purpose systems are designed to operate

at a voltage level and frequency level that meets their peak computational

performance. Unfortunately, this level of operation can consume significant amounts of

power, especially when peak processor performance is not required. Power can typically

be saved by reducing the supply voltage of the processor when peak performance is not

needed. With this approach, the sections of the processor which are unused have the

clock frequency reduced which reduces power consumption.

AMD Opteron Processor Low Power CPU Modules (HE)

Standard Power Modules

Performance Optimized Power (SE)

Quad-core processors TDP 79 W 115 W 137 W

ACP 55 W 75 W 105 W

17



As shown in Figure 5, the core frequency with the Dual-Core AMD Opteron processor is

locked based on the load characteristics of Core 0. Core 1 will operate at the same core

frequency even though it's load characteristics are low. With Independent Dynamic

Core Technology, the native quad-core processor can operate each of the cores at

different frequencies based on the load characteristics of that particular core.

Figure 4. Independent Dynamic Core technology adjusts frequency on a per-core basis


Dual Dynamic Power Management (formally “splitplane”) allows each processor to

maximize the power-saving benefits of AMD PowerNow! technology without

compromising performance. Dual Dynamic Power Management can reduce idle power

consumption and allow for per-processor power management in multisocket systems to

decrease power consumption. Figure 5 illustrates a Quad-Core AMD Opteron processor

powered by a conventional unified power supply, as well as one powered from

independent voltage supplies.

Figure 5. With Dual Dynamic Power Management system motherboards can deliver separate power for the processor cores and memory controller

Core 0 Core 1

Idle MHzMHzIdle

75% 1%Idle

Idle

IdleCore 0

MHzMHz

Core 1

IdleMHzMHzCore 2 Core 3

Dual-Core Native Quad-Core

75% 35%

1%10%

MHz and Voltage is lockedto the highest utilized

core’s p-state.

MHz is independently adjustedseparately per core.

DDR2Memory

Controller

HyperTransport 0

HyperTransport 1

HyperTransport 2


Crossbar Switch

2 MB L3 Cache

Core 1 Core 2 Core 4Core 3

Unified PowerSupply

DDR2Memory

Controller

HyperTransport 0

HyperTransport 1

HyperTransport 2


Crossbar Switch

2 MB L3 Cache

Core 1 Core 2 Core 4Core 3CPU Power

Memory ControllerPowerr

18



The illustration shows that power for the CPU and Memory Controller on Third-

Generation AMD Opteron processors can be provided from independent voltage

supplies — if supported by the system motherboard — offering greater performance

and better power management. These same processors can also run in legacy systems

with a unified power supply. Second-Generation AMD Opteron processors use a unified

voltage plane for the memory and processor cores. These processors are still

compatible with motherboards designed to support Dual Dynamic Power Management

but they will deliver the same voltage to the CPU and Memory Controller power.

High-Bandwidth Chip-Level I/O for High Performance Computing

Implementing a multicore processor is only one part of providing fast and reliable

performance. In addition to computational performance, high performance computing

and other demanding applications require the ability to move data between processors,

memory, and I/O with a minimum of bottlenecks. Third-Generation Quad-Core AMD

Opteron processors are designed to provide high-bandwidth chip-level interconnects to

other processors, memory, and system I/O. HyperTransport links connect between

multiple processors and to system I/O bridges. Integrated memory controllers provide

fast low-latency access to memory.

Integrated Memory Technology

AMD Opteron processors integrate a DDR memory controller directly into the processor.

The memory controller runs close to the processor’s core frequency and greatly

increases bandwidth to the processor at significantly reduced latencies. The

performance-enhancing effect is even more dramatic within multisocket AMD Opteron

systems, because each additional processor has its own memory controller, allowing

memory bandwidth to scale within the server.

AMD Opteron processors are designed to work with Double Data Rate (DDR) SDRAM.

Similar to first-generation DDR memory, DDR2 memory cells transfer data both on the

rising and falling edge of the clock (a technique called “dual pumping”). The key

difference between DDR and DDR2 is that in DDR2 the bus is clocked at twice the speed

of the memory cells, so four words of data can be transferred per memory cell cycle. As

a result, DDR2 can effectively operate at twice the bus speed of DDR, without speeding

up the memory cells themselves.

Figure 3 illustrates the Quad-Core AMD Opteron processor architecture featuring the

cache controller and three stages of caches. The dedicated per-core 128 KB L1 cache

provides a 64 KB instruction cache and a 64KB for data, and is capable of delivering two

data loads per cycle instead of one load per cycle of competing x86 processors. The

latency for the L1 cache is three clock cycles with very fast access time. The quad-core

architecture also features a dedicated per-core 512 KB L2 cache to eliminate conflicts

common in shared caches. These caches are 16-way set associative, and the latency for

each core to retrieve data from its L2 cache is 12 clock cycles. A large, 2 MB L3 cache is

19

AMD Opteron™ Processor Technology Sun Microsystems, Inc.

shared between all processor cores in Quad-Core AMD Opteron processors. The L3 cache

is 32-way set associative and is based on a non-inclusive victim cache architecture. The

latency for any core to retrieve data from the L3 cache is less than 38 clock cycles.

The L2 cache was designed for those applications that are running on a single core and

consume most or all of the 2 MB L3 cache. This situation can cause a problem on other

processor architectures that do not have three levels of cache, since the shared cache

can be busy serving one core while the others are starved. In AMD Opteron processors,

even if one thread is consuming the L3 cache, other threads run effectively from the

core’s L2 cache, which is sized to accommodate the majority of modern working sets.

AMD Virtualization Technology

Virtualization technology lets organizations achieve higher levels of efficiency,

utilization, and flexibility by dividing a given system into several virtual machines —

allowing the consolidation of many legacy systems onto one physical system. AMD's

Virtualization (AMD-V) technology provides an enhanced AMD Opteron instruction set

that subsumes some tasks that virtual machine managers (VMMs) typically perform

through software emulation.

Quad-Core AMD Opteron processors with Direct Connect Architecture help enable

industry leading virtualization platform efficiency. Featuring AMD-V technology with

Rapid Virtualization Indexing, Quad-Core AMD Opteron processors can accelerate the

performance of virtualized applications and improve the efficiency of switching among

virtual machines. This feature allows organizations to host more virtual machines and

users per system to maximize the consolidation and power-saving benefits of

virtualization.

Third-Generation Quad-Core AMD Opteron processors offer enhancements to AMD-V

that provide a balanced approach to improve virtualization performance and help

enable near-native performance for virtualized applications. AMD Opteron processors

also provide silicon feature-set enhancements that are designed to improve

performance, reliability, and security of existing and future virtualization environments

to support more users. Some of the AMD-V enhancements that are built into the Third-

Generation AMD Opteron architecture include:

• Direct Connect Architecture to Host More Virtual Machines (VMs) Per Server

AMD’s Direct Connect Architecture helps improve application performance within

a virtual machine. This architecture provides direct CPU-to-memory, CPU-to-I/O,

and CPU-to-CPU connections to streamline server virtualization. The Integrated

Memory Controller is designed to improve performance on memory-intensive

virtualization environments through high bandwidth, low latency, and scalable

access to memory. HyperTransport technology optimizes the movement of data

and the sharing of resources among VMs and I/O subsystems for greater system

scalability.

20 AMD Opteron™ Processor Technology Sun Microsystems, Inc.

• Tagged Translation Look-aside Buffer for Increased Responsiveness

Unique to AMD Opteron processors, the Tagged Translation Look-aside Buffer (TLB)

allows for faster switching times between virtual machines by maintaining a

mapping to the VM’s individual memory spaces. Competing solutions cannot

distinguish one VM's memory space from another’s, resulting in additional

memory management overhead and reduced responsiveness when switching

between virtual machines.

• Device Exclusion Vector (DEV) for More Efficient Security

AMD’s Device Exclusion Vector (DEV) performs security checks in hardware,

protecting memory access to un-authorized requests from external devices. The

DEV controls access to virtual machine memory based on permission, isolating

virtual machines for secure operation. The DEV performs these security checks in

hardware, rather than software — resulting in efficiency. The DEV creates

Protection Domains that deny memory access for unauthorized requests from

external devices, such as hard disks, network controllers, and other devices.

• Rapid Virtualization Indexing for Better Performance in a Virtualization Environment

Rapid Virtualization Indexing is an enhancement to AMD-V technology in Quad-

Core AMD Opteron processors. This feature is designed to dramatically increase

the performance of virtualized applications while providing faster switching

between virtual machines. Rapid Virtualization Indexing allows users to host more

VMs per server and maximize the benefits of virtualization. This feature must be

supported in the virtualization software.

21 Server Architecture Sun Microsystems, Inc.

Chapter 3

Server Architecture

The Sun Fire X4600 server is designed to provide multisocket scalability and high

performance while offering high reliability and low power consumption. The sections

that follow detail physical and architectural aspects of the system.

System-Level ArchitectureFigure 6 illustrates a high-level block diagram of the Sun Blade X4600 server

motherboard, illustrating a relatively simple and elegant design. The server supports up

to eight Dual-Core or Quad-Core AMD Opteron processors on CPU/Memory modules

that insert directly into the motherboard. Eight on-board DIMM slots are provided on

each CPU/Memory module, and are directly connected to the processor. CPU/Memory

Modules are added in pairs, and are interconnected by dedicated 8.0 GB/second

HyperTransport links. System configurations with two, four, six, and eight processors

are supported. All processors in a system must be the same type (dual-core or quad-

core), same stepping, and the same frequency.

Figure 6. Sun Fire X4600 M2 motherboard block-level diagram

CPU

Mod

ule

A

CPU 0

CPU

Mod

ule

B

CPU 2

CPU

Mod

ule

C

CPU 1

CPU

Mod

ule

D

CPU 3

CPU

Mod

ule

E

CPU 4

CPU

Mod

ule

F

CPU 5

CPU

Mod

ule

G

CPU 6

CPU

Mod

ule

H

CPU 7

SAS HDDs

4x 1 GBEthernet

RearUSB

FrontUSB

IDECD/DVD

VGAVideo

Serial ManagementEthernet

PCI-X

100

MHz

PCI-X

100

MHz

PCI-E

8-L

ane

PCI-E

8-L

ane

PCI-E

4-L

ane

PCI-E

8-L

ane

PCI-E

8-L

ane

PCI-E

4-L

ane

Shared100 MHz Bus

ILOM

32 MBFlash

Videoover LANRedirect

64 MBDDR

SDRAM

SuperI/O

Rage XLRage XL

ManagedPower Supplies

IO-04CK8-04+ +

+ +

8132PCI-X

USB 0

USB 1

USB 5

USB 2 .. 4

IDE

All HT links - 1 GHz 8 GB/sec

+ +

+ +

SAS 1064LSI

USBHub

DVI

Video

BCM5221

MPC8248SP


The CPU/Memory modules are connected to a set of components that provide I/O for

the system, including:

• A tunneling HyperTransport bridge (AMD-8132 HyperTransport PCI-X Tunnel)

• A HyperTransport-attached Southbridge component (NVIDIA nForce4 Professional

2200/CK8-04)

• A HyperTransport-attached Southbridge component (NVIDIA nForce4 Professional

2050/IO-04)

AMD Direct Connect Architecture Implementation with HyperTransport Links

AMD’s Direct Connect Architecture provides system designers with a powerful tool for

building tightly-coupled systems with large numbers of processors. Because processors

are directly connected to memory, I/O, and each other via HyperTransport links,

multisocket systems can be designed with low latency and without extra bus and chip

overhead. The Sun Fire X4600 M2 server leverages HyperTransport links and the AMD

Direct Connect Architecture to create a design that scales easily from two to eight

sockets.

The design of a HyperTransport topology for larger numbers of processors must take

hop distance into account. In particular, average hop distance increases when building

eight-socket systems when compared to that of four-socket systems. Minimizing the

number of hops is ideal, and Sun Fire X4600 servers accomplish minimum hop distance

by enhancing the ladder and twisted ladder topologies as shown in Figure 7.

Figure 7. The Enhanced Twisted Ladder topology yields reduced hop counts for larger numbers of processors

Sun’s approach using HyperTransport technology minimizes hop count and latency

between processors. Since memory is controlled by each processor, latency for remote

memory accesses is also reduced. Figure 8 illustrates the hop counts for two-socket,

four-socket, six-socket, and eight-socket Sun Fire X4600 M2 server configurations.

Ladder Enhanced Twisted LadderTwisted Ladder


Though many choose to divide the resources of Sun Fire X4600 servers using

virtualization technology, workloads with higher CPU components and lower memory

latency/bandwidth needs scale best beyond four sockets.

Figure 8. Hops between processors in Sun Fire X4600 servers

Connectivity between CPU/Memory modules changes depending on the actual number

of modules installed. Figure 9 illustrates the various connectivity scenarios depending

on the number of installed CPU/Memory modules.

Figure 9. Hypertransport topology in Sun Fire X4600 M2 servers for 2-processor, 4-processor, 6-processor, and 8-processor configurations

p gy

8 CPU Modules

4 CPU Modules 2 CPU Modules

CPUModule B

CPUModule D

CPUModule C

CPUModule A

CPUModule F

CPUModule E

CPUModule G

CPUModule H

CK8-04P

+ +

+ +

8132PCI-X

IO-04

CPUModule F

CPUModule D

CPUModule A

CPUModule H

CPUModule E

CPUModule B

CK8-04P

+ +

+ +

8132PCI-X

IO-04

6 CPU Modules

CPUModule E

CPUModule A

CPUModule H

CPUModule D

CK8-04P

+ +

+ +

8132PCI-X

IO-04

CPUModule B

CPUModule A

CK8-04P

+ +

+ +

8132PCI-X


I/O Subsystem

The Sun Fire X4600 M2 I/O subsystem is connected to the CPU/Memory complex

through a HyperTransport link from processors A and H. The I/O bridges are

implemented with the NVIDIA NForce4 Professional 2200 (CK8-04) and the NVIDIA

nForce4 Professional 2050 (IO-04) media and communications processors (MCPs). Other

I/O components connect to these three main chipsets, including:

• A quad-channel SAS RAID controller (LSI SAS 1064) attached to one of the AMD-8132

PCI-X tunnels on a shared bus with the PCI-X slot 0.

• Two 10/100/1000 dual Gigabit Ethernet MAC/PHY devices (Intel FW82546GB NIC)

• An ATI Rage XL video controller device attached to the PCI bus of the NVIDIA nForce

Professional 2200

• A Super I/O device connecting the Integrated Lights Out Management (ILOM) service

processor to the NVIDIA nForce Professional 2200 chip.

AMD-8132 HyperTransport PCI-X Tunnel

The AMD-8132 HyperTransport PCI-X 2.0 tunnel is a high-speed device that provides two

independent, high-performance PCI-X bus bridges, integrated with a high-speed

HyperTransport technology tunnel. The AMD-8132 component brings high-performance

PCI-X 2.0 connectivity to AMD Opteron processor-based systems with increased

throughput, improved RAS capabilities, robust data management, and enhanced

HyperTransport technology connectivity.

As shown in Figure 10, the front-end HyperTransport interface (Side A) provides a

configurable 16-bit wide communication path to the host, offering up to 8 GB/second of

aggregate bandwidth. Similarly, the back-end HyperTransport interface (Side B)

provides a configurable 16-bit wide communication path to a downstream device,

offering up to 8 GB/second of aggregate bandwidth.

The AMD-8132 HyperTransport PCI-X Tunnel provides high-speed PCI-X capability and

offers the following features:

• 16-bit HyperTransport interfaces (Side A and B) offering a maximum aggregate

bandwidth up to 8 GB/sec. on each side

• Two PCI-X bridges (A and B), each of which supports a 64-bit data bus

• Each bridge also supports Mode 1 PCI-X, conventional PCI protocol, and Mode 2

operation

• Each bridge supports independent transfer rates and operational modes:

– In PCI-X mode, the bridge supports transfer rates of 133, 100, 66, and 50 MHz

– In PCI mode, the bridge supports transfer rates of 66, 50, 33, and 25 MHz


Figure 10. AMD-8132 HyperTransport PCI-X tunnel architecture block diagram

NVIDIA nForce Professional Media and Communications Processors (MCPs)

Both MCPs provide I/O connectivity to a CPU via a 1 GHz 8 GB/second HyperTransport

connection. Each MCP provides 20 lanes of PCI Express I/O divided into 3 links (two x8

links and one x4 link) for a total of 40 lanes of PCI Express I/O bandwidth. These links

are all directly wired to PCI Express expansion slots exposed through the rear of the

chassis. The CK8-04 bridge also implements the southbridge functionality including

multiple USB2.0 ports, a serial port, and an IDE bus.

The NVIDIA nForce Professional 2200 (CK8-04) supports:

• Three PCI Express ports (two x8 and one x4)

• Independent IDE connection to the DVD/CD-RW player

• LPC connection to the Super I/O chip

• Six USB 2.0 ports (three for external use and three for the ILOM service processor)

• A PCI connection to the ATI Rage XL video controller

The NVIDIA nForce Professional 2050 (IO-04) supports three PCI Express ports (two x8

and one x4).

LSI SAS1064 SAS Controller

The Sun Fire X4600 M2 server includes an integrated LSI SAS1064 4-port 3-GB/sec. SAS

HW RAID disk controller. The LSI SAS1064 is a versatile controller that provides 4 SAS

ports capable of 3 GB/second data transfers for each PHYS, for a total maximum

bandwidth of 12 GB/second. SAS features of the LSI SAS1064 include:

• Four fully-independent PHYS

• Support for 3 GB/second SAS data transfers for each PHYS

• High-performance, serial, point-to-point, enterprise-level storage interface

• Simplified cabling between devices

• Data transfers using SCSI information units

• 133 MHz 64-bit PCI-X interface (shared bus with PCI-X 100 MHz slots)

• Integrated RAID0 and RAID1 solutions provide Integrated Mirroring technology and

Integrated Striping technology

• Up to four hot-swappable 2.5-inch SAS disk drives are supported.

AMD-8132 Device

PCI-X

Bridge A

PCI-X

Bridge B

Side A Side B

HyperTransport Link

16 bits upstream

16 bits downstream

HyperTransport Link

16 bits upstream

16 bits downstream

Tunnel

Slots and Devices

HostDownstream

Device


Intel 82546GB Dual-Port Gigabit Ethernet Controllers

The Sun Fire X4600 M2 server is equipped with two Intel 82546GB Dual Port Gigabit

Ethernet controllers that provide quad Gigabit Ethernet interfaces to the server

platform. The Intel 82546GB Dual Port Gigabit Ethernet controller provides two 64-bit

fully-integrated Gigabit Ethernet Media Access Control (MAC) and physical layer (PHY)

functions. The controller is capable of transmitting and receiving data at 10/100/

1000MB/second data rates with half or full duplex capabilities. Additional features of

the Intel 82546GB Dual Port Gigabit Ethernet controller include:

• 802.3ab PHY compliance and compatibility (CAT-5 use)

• 802.3ab auto-negotiation

• 802.3x full-duplex flow control

• 802.9q VLAN tag insertion, stripping, and packet filtering

• Preboot eXecution Environment (PXE) Flash Interface support (32- and 64-bit)

ATI Rage XL Graphics Controller

The Sun Fire X4600 M2 server employs an ATI Rage XL graphics controller to deliver 2D

and 3D graphics acceleration. The DVI port is connected to the ILOM service processor

for remote graphics redirection. The ATI Rage XL has the following features:

• 64-bit 125 MHz memory clock

• 8 MB external memory

• 64-bit SDR (SDRAM/SGRAM) memory path

• 64-bit AGP/PCI bus

• Integrated TMDS: DVI, DFP, and VESA P&D interface

• Support for 24-bit TTL

• 1600x1200 maximum resolution

• 16.7M maximum color depth

• 1024 x 768 resolution

ILOM Daughter Card

Each Sun Fire X4600 M2 server provides an Integrated Lights Out Management (ILOM)

service processor located on a daughter card. This service processor is similar to those

used in other Sun x64 and SPARC servers. ILOM provides multiple in-band and out-of-

band management solutions. IPMI and SNMP OS-resident agents provide in-band

management through the host operating system or platform. Out-of-band

management is provided through the serial port or dedicated Ethernet port. A

command line interface (CLI) is provided for out-of-band serial port connectivity. The

dedicated Ethernet connectivity to out-of-band management features include:

• Web interface

• CLI via SSH

• IPMI 2.0

• SNMP v1, v2c, and v3


The ILOM daughter card is a standalone system with no dependencies on the operating

system. Physical interfaces are provided through the connections listed above. The

actual connected devices, however, are located on the main system board. More

information on ILOM capabilities is provided in Chapter 4. ILOM daughter card

connections include:

• DVI output port connected to the ATI Rage XL via FPGA to support remote graphics

redirection from the ILOM service with a maximum VGA resolution of 1024 x 768

• Three USB connections to the NVIDIA nForce Professional 2200 to provide remote

keyboard, mouse, and storage functionality

• LPC connection to the NVIDIA nForce Professional 2200 via an FPGA

• 10/100 MB Ethernet connected to the on-board Broadcom BCM5221 10/100 Ethernet

controller, which automatically detects straight or cross-over Ethernet cables

• External serial port multiplexed with the main system serial port for access to the

ILOM CLI and system console using serial port redirection (S-o-L, Serial-over-LAN)

CPU/Memory ModulesThe Sun Fire X4600 M2 server supports up to eight Dual-Core or Quad-Core AMD

Opteron Series 8000 processors — each provided on a CPU/Memory module with

associated slots for memory DIMMs. The CPU/Memory module contains:

• 1 AMD Opteron CPU and heatsink

• 8 DDR2 memory DIMM slots

• 1 CPU/Memory Voltage Regulator Module

• Airflow redirection baffles for greatest cooling efficiency

• A Fault Remind button and per-DIMM LEDs to help identify a failed component.

The CPU/Memory module (Figure 11) is connected to the system motherboard using a

pair of ganged connectors. Each processor receives core voltage input from an

individual voltage regulator module (VRM). The VRM regulates the system current and

voltage according to the VID code output by the CPU package. The VRM is a DC-to-DC


point-of-load convertor specified for +12 Volt input and programmable 100 Amp output.

The latest Sun Fire X4600 CPU/Memory module also supports split-plane operation for

support of AMD Dual Dynamic Power Management.

Figure 11. The Sun Fire X4600 server CPU/Memory Module provides support for a Dual-Core or Quad-Core AMD Opteron processor and up to 8 DDR2 DIMMs.

Memory Subsystem

Each AMD Opteron processor includes a low-latency, high-bandwidth, integrated

memory controller that reduces latencies during memory access over traditional front-

side bus-based memory controllers, and each supports up to eight ECC Registered

DDR2/667 memory modules. Although the AMD Opteron processor's memory controller

works in 64-bit or 128-bit mode ECC operation, for best performance results, it is

recommended to run 128-bit ECC operation mode. To run in 128-bit mode, DIMMs are

populated in pairs such that they occupy one-half of the AMD Opteron processor's 128-

bit controller interface. The controller supports 1 bit per byte ECC, and the Sun Fire

X4600 M2 server uses DDR2/667 (PC2-5300) registered DDR SDRAM modules.

Eight DDR DIMM slots are provided per processor, and they are color-coded white and

black to indicate population order (white = 0,1 and black = 2,3; white pair first). LED

fault indicators controlled by the ILOM Service Processor provide the ability to easily

identify failed DIMM modules. Memory capacity scales with the number of processors,

so memory attached to an unpopulated processor slot is unaddressable. As a result, a

dual-socket system can support a maximum of sixteen DIMMs (up to 64 GB with 4 GB

DIMMs). A fully populated, eight-processor server supports a maximum of 64 DIMMS or

up to 256 GB of memory with 4 GB DIMMs.

Other features of the Sun Fire X4600 M2 server memory architecture include:

• Dedicated on-die 128-bit wide DDR memory controller

• Memory bandwidth up to 10.7 GB/sec. @ DDR2/667

• Under 80 ns direct memory latency


• Registered ECC DDR2/667 (PC2-5300) DIMMs supported

• Up to 32 GB per CPU with 4 GB DIMMs in 8 DIMM slots per CPU

A Broad Range of Configuration Options

The Sun Fire X4600 M2 server can support two, four, six, or eight processors through

HyperTransport links. Table 4 illustrates the CPU population rules and numbering.

Systems with two processors simply deploy two CPU/Memory modules, leaving the rest

of the slots empty. Systems deploying four and six CPU/Memory modules require filler

modules that bridge the HyperTransport connections and help direct air-flow through

the chassis. In eight-processor systems, all of the slots are populated.

Table 4. CPU module population and numbering

Sun Fire X4600 M2 Server OverviewThe Sun Fire X4600 M2 server provides considerable computational power and

scalability in a space-efficient 4U rackmount package. By closely coupling up to eight

Dual-Core or Quad-Core AMD Opteron processors, these systems extend the ubiquitous

x64 architecture for mission-critical applications and OLTP database workloads as well

as high performance computing and consolidation. The server is designed to address

the challenges of modern datacenters with reduced power consumption and a small

physical footprint. Depending on the model selected, the Sun Fire x4600 M2 server

features two, four, six, or eight Dual-Core or Quad-Core AMD Opteron processors and up

to 256 GB of memory.

Front of the Server

Slot A Slot B Slot C Slot D Slot E Slot F Slot G Slot H

8 Processors Module 0 Module 2 Module 1 Module 3 Module 4 Module 5 Module 6 Module 7

6 Processors Module 0 Module 1 Filler Module 2 Module 3 Module 4 Filler Module 5

4 Processors Module 0 Filler Filler Module 1 Module 2 Filler Filler Module 3

2 Processors Module 0 Module 1 Empty Empty Empty Empty Empty Empty

Rear of the Server


System Motherboard and Chassis Perspective

Figure 12 provides a top-down perspective of the Sun Fire X4600 M2 server with the top

cover removed and a full complement of CPU/Memory modules installed. For some

four-socket and six-socket configurations, filler modules are installed in empty slots to

help ensure proper airflow and front-to-back cooling. All eight PCI Express slots are low

profile, and are wired to either x8 or x4 PCI Express interfaces.

Figure 12. Sun Fire X4600 M2 server top view

Four system fans insert from the top of the chassis, and four power supplies insert from

the rear of the chassis. The four power supplies provide N+N redundancy. The system

can continue to operate at full capacity with any combination of two of the four power

supplies.1

Enclosure

The 4U Sun Fire X4600 M2 server enclosure is designed for use in a standard 19-inch

rack (Table 5).

Table 5. Dimensions and weight of the Sun Fire X4600 M2 server

1.Some restrictions apply when apply when all 64 DIMM slots are populated with 137 Watt CPUs.

Dimension U.S. International

Height 6.9 inches (4 RU) 176 millimeters

Width 17.5 inches 445 millimeters

Depth 24.75 inches 629 millimeters

Maximum standalone weight (without rackmount kit and cable management arm)

88 pounds 40 kilograms

CPU and Memory Modules

Hot-Swap Fan Modules

ILOM Daughter Card


The Sun Fire X4600 M2 server includes the following major components:

• Support for two, four, six, or eight CPU/Memory modules, with each module

containing one Dual-Core or Quad-Core AMD Opteron 8000 series processor

• Eight DIMM slots per CPU/Memory module, supporting Up to 256 GB of memory with

Registered ECC DDR2/667 (PC2-5300) DIMMs

• Four drive bays for SAS disk drives and additional DVD-ROM

• Four on-board 10/100/1000Base-T Ethernet ports

• Four USB 2.0 ports (2 forward, 2 rear facing)

• Six PCI Express expansion slots (four x8, and two x4)

• Two 64-bit PCI-X slots @ 100 MHz

• ILOM service processor daughter card with 10/100Base-T Ethernet port

• 19-inch rackmount kit

• Cable management arm (optional)

• Four 950 Watt AC power supplies (hot-swappable in N+N redundant configuration)

with integral fans

• Four (N+1 redundant) hot-swappable cooling fans under environmental monitoring

and control, accessed directly from the top of the chassis


Front and Rear Perspectives

Figure 13 illustrates the front and rear panels of the Sun Fire X4600 M2 server.

Figure 13. Sun Fire X4600 M2, front panel perspective

External features of the Sun Fire X4600 M2 server include:

• Front and rear system and component status indicator lights provide locator (white),

service required (amber), and activity status (green) for the system.

• Four hot-plug SAS disk drives insert through the front panel of the system.

• One slimline, slot-accessible DVD/CD-RW drive is accessed through the front panel.

• Four USB 2.0 ports are provided, two on the front panel, and two on the rear.

• Four hot-plug/hot-swap (N+N) power supplies with integral fans insert from the rear.

• Rear power-supply indicator lights convey the status of each power supply.

• A single AC plug is provided on each hot-plug/hot-swap power supply.

• Four 10/100/1000Base-T autosensing Ethernet ports are provided.

• A 15-pin dinn connector is provided for VGA port.

• A total of six PCI Express card slots are provided, along with two PCI-X slots.

• Two management ports are provided for use with the ILOM system controller. The

RJ-45 serial management port provides the default connection to the ILOM controller.

USB 2.0 Ports

DVD driveLocate Button/LED

Hard Disk Drives

System status indicators

Redundant (N+N)

10/100/1000Base-T

VGA port

Serial and network

PCI-X slots

Management ports

USB ports

power supply units

Ethernet portsPCI Express slotsPCI Express slots


Cooling and Power

The Sun Fire X4600 M2 server is designed with stringent datacenter power and cooling

requirements in mind.

Cooling

Because heat is so destructive to electronic components, lowering operating

temperature can substantially increase reliability. Sun's cooling design enables the

CPUs to operate approximately 15 percent more efficiently than in some competitive

systems.

Figure 14. Sun Fire X4600 M2 servers provide front-to-back airflow with airflow segregated between power supplies and the main chassis

As shown, the Sun Fire X4600 M2 server is partitioned by a chassis divider that

segregates airflow into two distinct chambers:

• The motherboard/PCI and fan chamber, and

• The Power Supply Unit (PSU) and storage chamber

Both areas are air cooled front-to-back in concert with typical modern datacenter

practice. The PSU and storage chamber is cooled by individual fans on the back of each

power supply. Air is drawn through the front of the server by these rear-located fans.

The motherboard/PCI and fan chamber is cooled by redundant rows of fans mounted in

Chassis Divider

PSU Airflow

MotherboardAirflow

MotherboardAirflow


front of the server behind the bezel (Figure 15). Fans are hot swappable, and each fan

has an LED fault indicator. Fans are accessible from the top of the server so that they

can be accessed without interrupting system operation.

Figure 15. Four N+1 redundant fans are accessed through the top of the Sun Fire X4600 M2 server

Power

The Sun Fire X4600 M2 server is powered by dual redundant hot-swappable power

supply units (PSUs) with integral fans. The server requires two PSUs, but up to four

PSUs can be used per system for redundancy. For optimal redundancy, separate circuits

can be used to eliminate the dependence on a single electrical circuit, and to provide

for consistent N+N redundancy with the PSUs themselves. Power supplies are

monitored and managed by the ILOM service processor. PSU fans are not individually

managed by ILOM, but are instead considered part of the PSU unit. As a result, the

failure of a PSU fan is reported by ILOM as a failure of the entire PSU.

35 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc.

Chapter 4

Enterprise-Class Operating System and Management Software

Unlike many x64 systems, Sun Fire X4600 M2 servers are ideally suited for enterprise

environments. Not only are the systems designed with the datacenter in mind, but the

operating system and management software provided with these systems allow them

to serve the most important and mission-critical applications. The sections that follow

describe the Solaris 10 OS and key enterprise-grade management technology.

Solaris 10 OS SupportAmong the available operating systems, the Solaris OS is ideal for large-scale enterprise

deployments. Supported on all of Sun’s x64 and SPARC platforms, the Solaris OS has

specific features that can enhance flexibility and performance — as described in the

sections that follow.

• Solaris Containers for Consolidation, Secure Partitioning, and Virtualization

Solaris Containers comprise a group of technologies that work together to

efficiently manage system resources, virtualize the system, and provide a

complete, isolated, and secure runtime environment for applications. Solaris

Containers can be used to partition and allocate the considerable computational

resources of Sun Fire X4600 M2 servers. Solaris containers are comprised of Solaris

Zones and Solaris Resource Management which work together with the Solaris

fair-share scheduler.

– Solaris Zones — Solaris Zones can be used to create an isolated and secure envi-

ronment for running applications. A zone is a virtualized operating system envi-

ronment created within a single instance of the Solaris OS. Zones can be used to

isolate applications and processes from the rest of the system. This isolation

helps enhance security and reliability since processes in one zone are prevented

from interfering with processes running in another zone.

– Resource Management — Resource management tools provided with the Solaris OS lets administrators dedicate resources such as CPU cycles to specific applications. CPUs in multicore multiprocessor systems — such as Sun Fire X4600 M2 servers — can be logically partitioned into processor sets and bound to a resource pool, and can ultimately be assigned to a Solaris zone. Resource pools provide the capability to separate workloads so that consumption of CPU resources does not overlap. Resource pools also provide a persistent configura-tion mechanism for processor sets and scheduling class assignment. In addition, the dynamic features of resource pools let administrators adjust system resources in response to changing workload demands.


• Solaris Dynamic Tracing (DTrace) to Instrument and Tune Live Software Environments

When production systems exhibit nonfatal errors or sub-par performance, the

sheer complexity of modern distributed software environments can make accurate

root-cause diagnosis extremely difficult. Unfortunately, most traditional

approaches to solving this problem have proved time-consuming and inadequate,

leaving many applications languishing far from their potential performance levels.

The Solaris DTrace facility provides dynamic instrumentation and tracing for both

application and kernel activities — even allowing tracing of application

components running in a Java Virtual Machine (JVM™)1. DTrace lets developers and

administrators explore the entire system to understand how it works, track down

performance problems across many layers of software, or locate the cause of

aberrant behavior. Tracing is accomplished by dynamically modifying the

operating system kernel to record additional data at locations of interest. Best of

all, although DTrace is always available and ready to use, it has no impact on

system performance when not in use, making it particularly effective for

monitoring and analyzing production systems.

• NUMA Optimization in the Solaris OS

With memory managed by each processor on Sun Fire X4600 M2 servers, the

implementation represents a non-uniform memory access (NUMA) architecture. In

NUMA architectures, the speed needed for a processor to access its own memory

is slightly different than that required to access memory managed by another

processor. The Solaris OS provides technology that can specifically help

applications improve performance on NUMA architectures.

– Memory Placement Optimization (MPO) — The Solaris 10 OS uses MPO to

improve the allocation and placement of memory across the physical memory of

a server, resulting in increased performance. Through MPO, the Solaris 10 OS

works to help ensure that memory is as close as possible to the processors that

access it, while still maintaining enough balance within the system. As a result,

many database and HPC applications are able to run considerably faster with

MPO.

– Hierarchical lgroup support (HLS) — HLS improves the MPO feature in the

Solaris OS. HLS helps the Solaris OS optimize performance for systems with

more complex memory latency hierarchies. HLS lets the Solaris OS distinguish

between the degrees of memory remoteness, allocating resources with the low-

est possible latency for applications. If local resources are not available by

default for a given application, HLS helps the Solaris OS allocate the nearest

remote resources.

1.The terms "Java Virtual Machine" and "JVM" mean a Virtual Machine for the Java platform.


• Solaris ZFS

Solaris ZFS offers a dramatic advance in data management, automating and

consolidating complicated storage administration concepts and providing

unlimited scalability with the world’s first 128-bit file system. ZFS is based on a

transactional object model that removes most of the traditional constraints on I/O

issue order, resulting in dramatic performance gains. ZFS also provides data

integrity, protecting all data with 64-bit checksums that detect and correct silent

data corruption.

• A Secure and Robust Enterprise-Class Environment

Best of all, the Solaris OS doesn’t require arbitrary sacrifices. The Solaris Binary

Compatibility Guarantee helps ensure that existing Solaris applications continue

to run unchanged, protecting investments. Certified multilevel security protects

Solaris environments from intrusion. Moreover, Sun’s comprehensive Fault

Management Architecture means that elements such as Solaris Predictive Self

Healing can communicate directly with the hardware to help reduce both planned

and unplanned downtime.

Integrated Lights Out Management (ILOM)The Sun Fire X4600 M2 server offers an Integrated Lights Out Manager (ILOM) service

processor on a removable daughter card, allowing remote management for all

activities that do not require physically touching the system. Industry standards are

embraced throughout, letting these systems easily integrate into existing

environments. In addition, since the ILOM service processor is a core component of Sun

Fire X4600 M2 servers, there is no additional charge for this functionality.

On-Board ILOM Firmware and Connections

The ILOM service processor connects to all major components via on-board interfaces

such as I2C with a separate management network provided for remote access. Equipped

with field-upgradeable firmware, the ILOM service processor supplies management

functions for fan speed control and diagnostic LEDs, and provides a wealth of

connections to individual server components. Sensors can use the ILOM service

processor to generate entries in the system event log when the sensor crosses a certain

threshold value. Examples in Sun Fire X4600 M2 servers include:

• Chassis sensors for intrusion, power supply failure, temperature failure, or fan failure

• Front and back panel sensors sensing the state of various LEDs and system locate

button

• Motherboard temperature sensors to monitor the ambient temperature chip on the

motherboard

• Power supply sensors to determine whether power supplies are present, connected

to AC power, and/or powering the system


• Hard disk drive backplane sensors to determine the presence and health of the disk

backplane

• Fan sensors to determine the presence of the fan tray, the speed of individual fans,

and to detect the failure of individual fans.

ILOM Communication Channels, User Management, and Security

Access to ILOM functionality on Sun Fire X4600 M2 servers can be made through a

variety of both out-of-band and in-band communication channels. Out-of-band

communication helps ensure that effective management can take place even in the

event of hardware or networking failures, and includes:

• A management serial port that provides direct console access via a command line

interface (CLI)

• A dedicated Ethernet port that provides a web-based GUI (over HTTPS), a CLI via SSH,

IPMI 2.0, and SNMP v1, v2c, and v3

In-band communication to the ILOM service processor is provided via the host OS

running on the server. A variety of management tools can be used to access

management information on individual servers:

• Intelligent Platform Management Interface (IPMI) with IPMItool

IPMItool is a simple command-line interface to systems that support the

Intelligent Platform Management Interface (IPMI) v2.0 specification. IPMItool

provides the ability to read the sensor data repository and print sensor values,

display the contents of the system event log, print field-replaceable unit

information, read and set LAN configuration parameters, and perform remote

chassis power control. IPMItool was originally written to take advantage of IPMI-

over-LAN interfaces but it is also capable of using the system interface as provided

by a Linux kernel device driver such as OpenIPMI or a Solaris OS driver called BMC

that is provided with the Solaris 10 OS. IPMItool is available under a

BSDcompatible license.

IPMItool is not designed to replace the OpenIPMI library but instead provides a

completely command-line oriented tool that can be used by administrators in

conjunction with other tools. Where possible, IPMItool supports comma-

separated values for output to facilitate parsing by other scripts or programs.

IPMItool is designed to run quick command-response functions that can be as

simple as turning the system on or off, or as complex as reading in the sensor data

records while extracting and printing detailed sensor information for each record.

• Simple Network Management Protocol (SNMP) Management

SNMP management provides remote access by SNMP-compliant entities to

monitor and control network devices and manage configurations including

statistics collection, performance, and security on a network. SNMP is a network

management protocol used almost exclusively in TCP/IP networks. Sun Fire X4600


M2 servers provide SNMP MIBs to manage and monitor the servers using any

SNMP-capable network management system, such as HP OpenView Network

Node Manager (NNM), Tivoli, CA Unicenter, or IBM Director. The MIB data

describes the information being managed, reflects current and recent server

status, and provides server statistics.

SNMP v1, v2c, and v3 are supported with v3 selected by default (v1 and v2c are

disabled by default). SNMP “sets” can be selected or disabled (default). An IPMI-

specific trap called a Platform Event Trap, or PET, may also be generated. The

following SNMP MIBs are supported:

– SNMP-FRAMEWORK-MIB

– SNMP-USER-BASED-SM-MIB

– SNMP-MPD-MIB

– ENTITY-MIB

– SUN-PLATFORM-MIB

• Role-Based Administration

Different management users can be defined with corresponding roles and

responsibilities. Up to 10 user IDs can be created locally on the service processor

with each user ID consisting of a user name and the roles that are allowed. By

default, Administrator and Operator roles are defined. Authentication is carried

out against a local service processor database. Alternately, an LDAP client is

implemented in the ILOM service processor as well to allow authentication

against an LDAP server (LDAP groups must be mapped to service processor roles).

Up to 10 concurrent active sessions are supported on the service processor,

including serial, secure shell and web clients. User accounts can be authenticated

through LDAP, Radius, and Active Directory.

Remote Keyboard, Video, Mouse, and Storage (RKVMS)

To facilitate effective and full-featured remote management, the ILOM service processor

provides remote keyboard, video, mouse, and storage (RKVMS) support that is tightly

integrated with Sun Fire X4600 M2 servers. Together these capabilities allow the servers

to be administered remotely, while accessing keyboard, mouse, video and storage

devices local to the administrator (Figure 16). ILOM Remote Console support is provided


on the ILOM service processor and can be downloaded and executed on the

management console. Input/output of virtual devices is handled between ILOM on the

server and ILOM Remote Console on the web-based client management console.

Figure 16. Remote keyboard, video, mouse, and storage (RKVMS) support in the ILOM service processor allows full-featured remote management for Sun Fire X4600 M2 servers

• Remote Keyboard and Mouse Support

The Sun Fire X4600 M2 server detects a USB keyboard and mouse through the

ILOM service processor. ILOM Remote Console captures mouse and keyboard input

on the management console and sends it to the ILOM service processor. The

service processor then transmits mouse and keyboard inputs on the respective

USB buses on the server. The server receives keyboard entries and mouse

movements as if they were generated by local USB devices.

• Remote Video Support

The server incorporates a VGA graphics controller. Graphics from the ATI graphics

controller is sent to the ILOM service processor. The service processor then

redirects the video signal to ILOM Remote Console running on the management

system over a network connection, where the video is displayed on the

management console. ILOM Remote Console supports 16-bit video to

accommodate higher quality.

ManagementConsole

CDROM, DVDROMor .iso Image

Keyboard, Mouse, CDROM,and Floppy are Seen as

USB Devices by BIOS and OS

ILOM Remote ConsoleDisplays Remote Video in

Application Window

Video(Up to 1024x768@60Hz)

ILOM Remote ConsoleConnected to ILOM OverManagement Ethernet

Local Mouse andKeyboard

Sun Fire X4600 M2Server

Graphics Redirect Over Ethernet

Floppy Disk orFloppy Image

Remote Keyboard, Mouse and StorageEmulated as USB Devices by ILOM


• Remote virtual storage

The Sun Fire X4600 M2 servers detects two USB storage devices through the ILOM

service processor that can be set up in the BIOS as floppy, CD/DVDROM, or disk

image (floppy and CD/DVDROM by default). When the server tries to access either

the virtual floppy disk or the virtual CDROM, the ILOM service processor redirects

the request over the Ethernet connection to ILOM Remote Console on the

management console. ILOM Remote Console accesses the content from the

physical floppy disk drive, CD/DVDROM, or disk image, returning it across the

network to the server. The ILOM service processor presents the data to the server

as if it were coming from a local USB storage device. This capability allows the

server to boot remotely from a virtual USB storage device. The content of the

storage device can be an actual CD, floppy disk, or disk image (ISO or IMG).

Sun xVM Ops CenterBeyond local and remote management capabilities, datacenter infrastructure needs to

be agile and flexible, allowing not only fast deployment but streamlined redeployment

of resources as required. Sun xVM Ops Center technology (formerly Sun N1™ System

Manager and Sun Connection) provides an IT infrastructure management platform for

integrating and automating management of thousands of heterogeneous systems. To

improve life-cycle and change management, Sun xVM Ops Center supports the

management of applications and the servers on which they run, including the Sun Fire

X4600 M2 server.

Sun xVM Ops Center simplifies infrastructure life-cycle management by letting

administrators perform standardized actions across logical groups of systems. Sun xVM

Ops Center can automatically discover and group bare-metal systems, performing

actions on the entire group as easily as operating on a single system. Sun xVM Ops

Center remotely installs and updates firmware and operating systems, including

support for:

• Solaris 8, 9, and 10 on SPARC systems

• Solaris 10 on x86/x64 platforms

• Red Hat and SuSE Linux distributions

In addition, the software provides considerable lights-out monitoring of both hardware

and software, including fans, temperature, disk and voltage levels — as well as swap

space, CPU utilization, memory capacity, and file systems. Role-based access control

lets IT staff grant specific management permissions to specific users. A convenient

hybrid user interface integrates both a command-line interface (CLI) and an easy-to-use

graphical user interface (GUI), providing remote access to manage systems from

virtually anywhere.


Sun xVM Ops Center provides advanced management and monitoring features to the

Sun Fire X4600 M2 server. The remote management interface discovers and presents

the the servers, making operations, detailed inventory, and status pages available to

administrators. Servers can be discovered and organized into logical groups. Organizing

servers into groups also allows features such as OS deployment across multiple

systems.

Some of the functions available through Sun xVM Ops Center software include

operating system provisioning, firmware updates (for both the BIOS and ILOM service

processor firmware), and health monitoring. In addition, Sun xVM Ops Center includes

a framework allowing administrators to easily access inventory information, simplifying

the task of running jobs on multiple servers with server grouping functionality.

43 Conclusion Sun Microsystems, Inc.

Chapter 5

Conclusion

To curb the impact of sprawling datacenters, many organizations need x64 servers with

truly massive levels of scalability. Whether serving large database or high performance

computing applications, or consolidating multiple lower-powered servers, systems

must scale gracefully. The balanced design of Sun Fire X4600 M2 servers lets them scale

in processing power, memory, and I/O bandwidth, without arbitrary limitations.

The Sun Fire X4600 M2 server features an innovative enterprise-class design that makes

the most of AMD’s Direct Connect Architecture. Based on innovative Dual-Core and

Quad-Core AMD Opteron processors, this server can supply a powerful building block to

help scale, virtualize, and consolidate datacenter infrastructure. Offering best-in-class

performance, remarkable density, and extensive system expandability, the Sun Fire

X4600 M2 server makes the most of constrained datacenter resources. In addition, the

server offers extensive reliability, availability, and serviceability features along with

improved energy efficiency. Integrated Lights Out Management technology provides

advanced monitoring and management — at no extra cost.

Ultimately, even the most capable servers require enterprise-class operating systems

and tools. With a choice of the Solaris OS, Linux, Microsoft Windows, and VMware

virtualization software, these servers provide a wealth of popular options. The Solaris

OS in particular offers advanced and innovative features such as Solaris Containers and

DTrace that can help consolidate and tune enterprise deployments. With powerful

management tools such as Sun xVM Ops Center, organizations can deploy Sun Fire

X4600 M2 servers quickly with confidence, knowing that their choices can improve their

results as well as the bottom line.


For More Information

To learn more about Sun products and the benefits of Sun Fire X4600 M2 server, contact

a Sun sales representative, or consult the related documents and Web sites listed in

Table 6.

Table 6. Related Websites

Web Site URL Description

sun.com/x64 Sun Fire X4600 M2 server

sun.com/solaris The Solaris Operating System

sun.com/xvm Sun xVM Ops Center

sun.com/vmware VMware Virtual Infrastructure 3 (VI3) on Sun x64 servers

Sun Fire X4600 M2 Server Architecture On the Web sun.com/x64

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com

© 2008 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, Java, JVM, Lustre, NetBeans, MySQL, N1, Solaris, StarOffice, Sun Fire, Sun Ray, and Trusted Solaris are trademarks or registered trademarks of Sun

Microsystems, Inc. or its subsidiaries in the United States and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the US and other

countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. AMD and Opteron are trademarks or registered trademarks of Advanced Micro Devices, Inc. Information

subject to change without notice. Printed in USA SunWIN #: 470898 6/08

sun fire x4600 m2 server architecture

Documents