sunffire6800-4810-4800-3800 system overview manual805-7362-13

8/3/2019 SunfFire6800-4810-4800-3800 System Overview Manual805-7362-13

1/48

Sun Microsystems, Inc.www.sun.com

Submit comments about this document at: http://www.sun.com/hwdocs/feedback

Sun Fire 6800/4810/4800/3800Systems Overview

Part No. 805-7362-13February 2004, Revision 01


2/48

Copyr ight 2004 Sun Microsystems, Inc., 4150 Netw ork Circle,Santa Clara, California 95054, U.S.A. All rights reserved .

Sun Microsystems, Inc.has intellectual property rights relating to technology that is described in this d ocument. In particular, and withoutlimitation, these intellectual property rights may include one or more of the U.S. patents listed at http:/ / ww w.sun.com/ p atents and one ormore add itional patents or pend ing patent app lications in the U.S. and in other countries.

This docum ent and the p roduct to which it pertains are distributed und er licenses restricting their use, copying,d istribution, anddecompilation.N o part ofth e product or ofthis document may be reproduced in any form by any means without prior written authorization ofSun and its licensors, if any.

Third-pa rty softwa re, includin g font technology, is copyrigh ted and licensed from Sun supp liers.

Parts of the prod uct may be der ived from Berkeley BSD systems, licensed from the University ofCa lifornia. UNIX is a registered tradem ark inthe U.S.a nd in other countries, exclusively licensed through X/ Open Comp any, Ltd.

Sun, Sun Microsystems, the Sun logo, AnswerBook2, docs.sun.com, Sun Fire, and Solaris are trademarks or registered trad emarks of SunMicrosystems, Inc.in the U.S.and in other countries.

All SPARC trad ema rks are used u nd er license and are trad ema rks or registered trad ema rks of SPARC Intern ational, Inc. in the U.S. and in othercountr ies. Prod ucts bearing SPARC trad ema rks are based up on an architecture developed by Sun Microsystems, Inc.

The OPEN LOOK and Sun Grap hical User Interface was dev eloped by Sun Microsystems, Inc. for its users and licensees. Sun acknowled gesthe pioneering efforts of Xerox in researching and d eveloping the concept of visual or graph ical user interfaces for the comp uter indu stry.Sunholds a non-exclusive license from Xerox to the Xerox Graph ical User Interface, wh ich license also covers Suns licensees wh o imp lement OPENLOOK GUIs and oth erw ise comp ly with Suns wr itten license agreem ents.

U.S. Government RightsCommercial use. Government u sers are subject to the Sun Microsystems, Inc.stand ard license agreement andapp licable provisions of the FAR and its supp lements.

DOCU MENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CON DITIONS, REPRESENTATION S AND WARRANTIES,INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON -INFRINGEMENT,ARE DISCLAIMED, EXCEPT TO TH E EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Copyr ight 2004 Sun Microsystems, Inc., 4150 Netw ork Circle,Santa Clara, Californie 95054, Etats-Unis. Tous droits rservs.Sun Microsystems, Inc. a les droits de propr it intellectuels relatants la technologie qui est dcrit da ns ce docume nt. En par ticulier, et sans lalimitation, ces droits de propr it intellectuels peu vent inclure un ou plus des brevets amricains num rs http :/ / ww w.sun.com/ p atents etun ou les brevets plus sup plmentaires ou les app lications de brevet en attente da ns les Etats-Unis et dan s les autres pays.

Ce produ it ou documen t est protg par un copyright et distribu avec des licences qui en restreignent lutilisation, la copie, la distribution, et ladcompilation.Au cune partie de ce produ it ou document ne peut tre reproduite sous aucune forme, par quelque moyen qu e ce soit,sanslautor isation pra lable et crite de Sun et d e ses bailleurs d e licence, sil y ena.

Le logiciel dten u par des tiers, et qui comp rend la technologie relative au x polices de caractres, est protg par un copyright et licenci par d esfournisseurs d e Sun.

Des parties de ce prod uit pou rront tre dr ives des systmes Berkeley BSD licencis par lUniversit de Californie. UNIX est une ma rquedp ose aux Etats-Unis et da ns dautres pays et licencie exclusivem ent par X/ Open Comp any, Ltd.

Sun, Sun M icrosystems, le logo Sun, AnswerBook2,d ocs.sun.com, Sun Fire, et Solaris sont des marqu es de fabrique ou des m arques dp osesde Sun Microsystems, Inc.au x Etats-Unis et dan s dautres pays.

Toutes les marqu es SPARC sont utilises sous licence et sont des ma rques d e fabrique ou d es marqu es dposes d e SPARC Internation al, Inc.aux Etats-Unis et dans dautres pays. Les prod uits porta nt les mar ques SPARC sont bass sur une architecture dvelop pe par SunMicrosystems, Inc.

Linterface dutilisation graph ique OPEN LOOK et Sun a t dvelop pe par Sun Microsystems, Inc. pou r ses utilisateurs et licencis. Sunreconnat les efforts de pionn iers de Xerox pour la recherche et le dvelopp emen t du concep t des interfaces dutilisation visuelle ou grap hiquepou r lindu strie de linforma tique. Sun dtient u ne license non exclusive de Xerox sur linterface dutilisation graphiqu e Xerox, cette licencecouvra nt galemen t les licencies de Sun qu i metten t en place linterface d utilisation graph ique OPEN LOOK et qui en outre se conformen taux licences crites de Sun .

LA DO CUMENTATION EST FOURNIE "EN LTAT" ET TOUTES AUTRES CON DITIONS, DECLARATION S ET GARAN TIES EXPRESSESOU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PARLA LOI APPLICABLE, Y COMPRIS NOTAMMENTTOUTE GARAN TIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A LAPTITUDE A UNE UTILISATION PARTICULIERE OU ALABSENCE DE CON TREFAON .


3/48

iii

Contents

Preface xi

1. Sun Fire Product Overv iew 11

1.1 Standard Features 11

1.2 Machine Configurations 14

1.2.1 Sun Fire 6800 System 14




2. System Features and Capabil ities 21

2.1 Partit ions and Domains 22

2.1.1 Partit ions 22

2.1.2 Domains 23

2.2 Reliability 24

2.2.1 Redu cing the Probability of Error 24

2.2.2 Correcting Errors Using Error Correcting Code 24

2.2.2.1 Error-Correcting Code Protection of the DataInterconnect 25

2.2.3 Detecting Uncorrectable Errors 252.2.3.1 Multiple-Bit Data Errors 25


4/48


5/48

Contents v

3.3 Console Bus Interconnect 33

4. Sun Fire System Components 41

4.1 CPU/ Memory Board 41

4.2 I/ O Assemblies 42

4.2.1 PCI I/ O 42

4.2.2 CompactPCI I/ O 42

4.3 Repeater Board 42

4.4 System Controller Board 43

4.4.1 Redundant System Controllers 44

4.4.2 Virtual Domain Key Switches 44

4.4.3 Solaris Console 45

4.4.4 Virtual Time Of Day 45

4.4.5 Environmental Monitoring 45


6/48

vi Sun Fire 6800/4810/4800/3800 Systems Overview February 2004


7/48

vii

Figures

FIGURE 1-1 Sun Fire Systems and Sun Fire Cabinet 2

FIGURE 1-2 Sun Fire 6800 System CabinetFront and Rear Views 5

FIGURE 1-3 Sun Fire 4810 System Mounted in Optional Sun Fire CabinetFront and Rear Views 7

FIGURE 1-4 Sun Fire 4800 SystemFront and Rear Views 9FIGURE 1-5 Sun Fire 3800 System Mounted in Optional Sun Fire CabinetFront and Rear Views 11

FIGURE 2-1 Partitions and Domains on a Sun Fire 6800 System 2

FIGURE 3-1 Standard Operation of the Sun Fire 6800/4810/4800/3800 Systems 2


8/48

viii Sun Fire 6800/4810/4800/3800 Systems Overview February 2004


9/48

ix

Tables

TABLE 1-1 Sun Fire Shared Components 3

TABLE 1-2 Sun Fire 6800 System Features 4

TABLE 1-3 Sun Fire 4810 System Features 6

TABLE 1-4 Sun Fire 4800 System Features 8TABLE 1-5 Sun Fire 3800 System Features 10


10/48

x Sun Fire 6800/4810/4800/3800 Systems Overview February 2004


11/48

xi

Preface

This document provides th e following information about the Sun Fire systemfamily of servers:

s Machine configurations of the four serverss Hardw are overviews System comp onentss Reliability, availability, and serviceability features


12/48

xii Sun Fire 6800/4810/4800/3800 Systems Overview February 2004

Typographic Conventions

Related Docum entation

Typeface or

Symbol Meaning Examples

AaBbCc123 The nam es of comm and s, files,and directories; on-screencomputer output.

Edit your .login file.

Use ls -a to list all files.

AaBbCc123 What you type, whencontrasted w ith on-screencomputer output.

% su

Password:

A aBbCc123 Book titles, new w ords orterms, words to be emphasized.

Command-line variable;replace with a real name or

value.

Read Ch apter 6 in the Users Guide.

These are called class options.

You must be root to do this.

To d elete a file, type rm filename.

App lication T itle

Installation Sun Fire 6800 System Installation Guide

Sun Fire 4810/4800/3800 Systems Installation Guide

Sun Fire 4810/4800/3800 S ystems Cabinet M ount ing Guide

Operation Sun Fire Cabinet Installation and Reference Guide

Sun Fire 6800/4810/4800/3800 Systems Getting StartedSun Fire 6800/4810/4800/3800 Systems Service Manual

Software Sun Fire Midrange Sy stems Platform A dministration M anual

Sun Fire Midrange Systems Controller Command Reference Manual


13/48

Preface xiii

Accessing Sun Documentation OnlineYou can view, print, or purchase a broad selection of Sun documentation, includinglocalized versions, at:

http://www.sun.com/documentation

Contacting Sun Technical SupportIf you ha ve technical questions about this produ ct that are not answ ered in thisdocument, go to:

http://www.sun.com/service/contacting

Sun Welcomes Your CommentsSun is interested in imp roving its documen tation and w elcomes your comm ents andsuggestions. You can subm it your comments by going to:

http://www.sun.com/hwdocs/feedback

Please include the title and pa rt nu mber of your d ocument w ith your feedback:

Sun Fire 6800/4810/4800/3800 Systems Overview , part number 805-7362-13


14/48


15/48

1-1

CHAPTER 1

Sun Fire Prod uct Overview

This chapter discusses the features and capacity of four Sun Fire servers:

s Sun Fire 6800 systems Sun Fire 4810 systems Sun Fire 4800 systems Sun Fire 3800 system

This fam ily of servers p rovid es entry-level to high-end server fun ctionality. The SunFire 6800 system has space for internal peripherals mounted within the 19-inchcabinet. You have the flexibility with the remaining three systems to install them inindu stry-stand ard 19-inch cabinets or have th em preinstalled in a Sun Firesystemcabinet. The Sun Fire system cabinet can hold one Sun Fire 4810 system, oneSun Fire 4800 system, or a m aximu m of three Sun Fire 3800 systems.

1.1 Standard FeaturesThe standard features of these systems include:

s Rackmou ntable in indu stry stand ard 19-inch rack1

s Sup port for up to 24 CPUs and 192 Gbytes of memorys Sup port for up to 32 I/ O slots, PCI, and Com pactPCI I/ O m odu les

s Extensive redu nd ancys System controllerss Support for multiple domainss Concurrent hardware maintenances Common componentss Redu ndant power and coolings 9.6-Gbyte bus band wid th

1. The Sun Fire 6800 system is not ra ckmou ntable, it comes already m oun ted in a cabinet.


16/48

1-2 Sun Fire 6800/4810/4800/3800 Systems Overview February 2004

FIGURE 1-1 Sun Fire Systems a nd Sun Fire Cabinet

Sun Fire 6800 system

Sun Fire cabinetfor Sun Fire 4810/4800/3800 systems

Sun Fire 4800 deskside system


17/48

Chapter 1 Sun Fire Product Overview 1-3

The Sun Fire family of servers, of which the Sun Fire 6800/ 4810/ 4800/ 3800 systemsare mem bers, share man y common comp onents, as shown in the following table.

TABLE 1-1 Sun Fire Shared Components

Components Sun Fire 6800 Sun Fire 4810 Sun Fire 4800 Sun Fire 3800

CPU/Memory board x x x x

CPU processors x x x x

Memory DIMMs x x x x

PCI I/O assembly x x x NA

PCI I/O cards x x x NA

CompactPCI I/O assembly x x x 1

CompactPCI I/O cards x x x x

System Controller board x x x 2

Repeater board x x x 3

1

The CompactPCI I/ O assembly for the Sun Fire 3800 system is unique.2 The System Controller board for the Sun Fire 3800 system is unique.3 The Sun Fire 3800 system d oes not h ave Repeater board s.


18/48


1.2 Machine ConfigurationsFour machine configurations are available:

s Sun Fire 6800 system

s Sun Fire 4810 rackm oun table system

s Sun Fire 3800 rackm oun table system

s Sun Fire 4800 deskside or rackmountable system

1.2.1 Sun Fire 6800 System

The Sun Fire 6800 system has support for six CPU/ Memory boards, four I/ Oassemblies, four Repeater boards an d two System Controller board s. Although thereare four Repeater boards, they are logically two red un dan t repeaters (two boardstogether make up one logical repeater). FIGURE 1-2 shows front and rear views of the

Sun Fire 6800 system cabinet. TABLE 1-2 lists the featu res of the Sun Fire 6800 system.

TABLE 1-2 Sun Fire 6800 System Features

Features Quantity or Description

CPU / M em or y board s 6

CPUs 24

M axim u m m em ory 192 Gbytes

I/ O assem blies 4 (PCI or Com p actPCI)

System Controller boards 2

Repeater boards 4

Domains 4 maximum

Power supplies 6

Power requirements 200240 VAC

Red un dan t coolin g Yes

Red u nd an t A C in pu t Yes

Internal peripherals None (However, space is available in thecabinet for p eripherals options.)

Packaging Su n Fire 6800 cabinet


19/48


FIGURE 1-2 Sun Fire 6800 System CabinetFront and Rear Views

Front Rear


20/48



The Sun Fire 4810 system has su pp ort for three CPU/ Memory boards, two I/ O

assemblies, two Repeater boards, and two System Controller board s. FIGURE 1-3shows front and rear views of the Sun Fire 4810 system mounted in an optional SunFire cabinet. TABLE 1-3 lists the featu res of the Sun Fire 4810 system.




CPUs 12

Maxim um m em ory 96 Gbytes



Repeater boards 2

Domains 2 maximumPower supplies 3



Red un dan t AC in pu t N o

In tern al p er ip her als N on e

P ack ag in g Ra ck mou n ta ble or mou n ted in Su n Fire sy st em ca bin et


21/48


FIGURE 1-3 Sun Fire 4810 System Mounted in Optional Sun Fire CabinetFront and Rear

Views

Front Rear


22/48



The Sun Fire 4800 system has su pp ort for three CPU/ Memory boards, two I/ O

assemblies, two Repeater boards, and two System Controller board s. FIGURE 1-4show s front and rear view s of the Sun Fire 4800 system. TABLE 1-4 lists the featu res ofthe Sun Fire 4800 system.




CPUs 12




Repeater boards 2

Domains 2 maximumPower supplies 3




In tern al p er ip her als N on e

Packag ing Rackmountable , deskside , o r mounted in a Sun Fire systemcabinet


23/48


FIGURE 1-4 Sun Fire 4800 SystemFront and Rear Views

Front Rear


24/48



The Sun Fire 3800 system supports two CPU/ Memory board slots, two I/ O

assemb lies, and two System Controller board s. The Sun Fire 3800 system is d ifferentfrom th e Sun Fire 6800/ 4810/ 4800 systems in that it does n ot hav e Repeater b oard s,and uses a different System Controller board and I/ O assemblies.

FIGURE 1-5 shows front an d rear views of the Sun Fire 3800 system m ounted in anoptional Sun Fire cabinet. TABLE 1-5 lists the featu res of the Sun Fire 3800 basesystem.

TABLE 1-5 Sun Fire 3800 System FeaturesFeatures Quantity or Description


CPUs 8


I/ O assem blies 2 (Com p actPCI on ly)


Domains 2 maximum

Power supplies 3

Power requirements 100120 VAC or 200240 VAC



P ack ag in g Ra ck mou n ta ble or mou n ted in a Su n Fir e sy stem ca bin et


25/48


FIGURE 1-5 Sun Fire 3800 System Mounted in Optional Sun Fire CabinetFront and RearViews

Front Rear


26/48



27/48

2-1

CHAPTER 2

System Features and Cap abilities

The Sun Fire family of servers has features and capabilities that were not availableon p revious midran ge systems. Primary features includ e the ability to par tition you rsystem and create domains. These features provide greater reliability, availability,and serviceability, which means uptime. These features and capabilities are asfollows:

s Partition The ability for the system to logically behave as two separate systems.

s DomainThe ability to create logically independent multiple sections within apartition, with each domain running its own operating system.

s Reliability A fun ction of the care with wh ich the ha rdw are and software d esignwas executed, the qu ality of the components selected, and the qu ality of themanufacturing process (for example, ESD protection, clean rooms, and so forth).

s Availability The percentage of time the customer s system is able to d oproductive w ork.

s

Serviceability The system ensu res that repair time (d own time) is kept to aminimum.


28/48


2.1 Partitions and DomainsThe Sun Fire system can be divided into partitions and dom ains. A single physicalsystem can have m ultiple ind epend ent logical systems, each runn ing its ownoperating system, by using partitions and d omains. Partitions and d omains d ifferonly in terms of their flexibility and isolation.

FIGURE 2-1 Partitions and Domains on a Sun Fire 6800 System

2.1.1 Partitions

A single physical Sun Fire 6800 system can be divided into two partitions. All

connections between boards of one partition and boards of the other par tition aredisabled. The system logically behaves as tw o separa te systems.

Partition A Partition B

Board 0 Board 1 Board 2 Board 3 Board 4 Board 5

Partition A Partition B

Board 0 Board 1 Board 2 Board 3 Board 4 Board 5

Partitions A and B have separate Repeater boards.

A partition can have up to two domains.

Domain 1

Domain 0

Partition A

Domain 0

Partition A

Domain 0

Partition B

Domain 0

Partition B

Active domain connection

Active domain connectionInactive logical connection

Active domain connection


29/48

Chapter 2 System Features and Capabilities 2-3

If the partitions are assigned to the physical half of the Sun Fire 6800 system, thenthe power planes associated with each partition are also isolated. A Sun Fire 6800system can be divided into two partitions by logically isolating one set of Repeaterboard s for each p artition. The Sun Fire 4810/ 4800/ 3800 systems a lso sup por t tw o

partitions.

Each partition on th e Sun Fire 6800 system can have up to two d oma ins, allowing forup to four dom ains tota l. For th e Sun Fire 4810/ 4800/ 3800 systems, if a singlepartition is established, it can support two domains; if two partitions are established,how ever, each p artition w ill supp ort only one dom ain.

2.1.2 Dom ainsThe Sun Fire system can be logically divided into multiple domains. Since eachdom ain is comprised of one or more system boards, a dom ain can have between oneand 24 processors. Each dom ain ru ns its own instance of the operating system andhas its own periphera ls and n etwork connections. You can configure d omainswithout interrupting the operation of other domains on the same system.

Domains can be used for:s Testing new applicationss Operating system updatess Configuring several domains to support separate departments

While prod uction work continues on the remaining (and usu ally larger) dom ain,there w ill not be an y ad verse interaction between any of the dom ains. You can gainconfidence in the correctness of app lications w ithout d isturbing prod uction wor k.

When the testing work is complete, the system can be rejoined logically withoutrebooting (there are no p hysical changes wh en you use d omains). Thu s, if problemsoccur, the rest of your system is not affected.

The Sun Fire 6800 system can hav e u p to four dom ains. The Sun Fire 4810/ 4800/3800 systems can each have up to two domains. Each instance of the SolarisOperating Environment runs in its own domain. Domains do not depend on eachother and do not interact with each other.

A single partition on a Sun Fire 6800 system can be divided into two domains.Unlike partitions, domains share th e Repeater boards. Each dom ain gets half theaddress bandwidth of a full system bus.


30/48


2.2 ReliabilityThe reliability capa bilities of the Sun Fire system fall into fou r categories:

s Reducing the probability of errorss Correcting errors using error correcting code (ECC)s Detecting uncorrectable errorss Environmental sensing

2.2.1 Redu cing the Probability of ErrorAll the ASICs are designed for worst case temperature, voltage, frequency, andairflow combinations. The high level of logic integration in the ASICs reducescomponent an d interconnect count.

A distributed p ower system imp roves power sup ply performan ce and reliability.

Extensive self-test up on p ower-on reboot after a hard wa re failure screens all of thekey logic blocks in th e Sun Fire system:

s Built-in self-test logic in all the ASICs.s The power-on self-test (POST)controlled from the System Controller board

tests each logic block first in isolation, and then with progressively more of thesystem. Failing components are electrically isolated from the centerplane. Theresult is that the system is booted only w ith logic blocks that have p assed thisself-test and th at mu st operate w ithout error.

All I/ O cables have a positive lock mechanism and a strain-relief support to preventaccidental disconnections.

2.2.2 Correcting Errors Using Error Correcting Code

The Sun Fire system system contains a nu mber of subsystems that are capable ofrecovering from errors w ithout failing. Subsystems that have a large num ber ofconnections have greater od ds of failure. The subsystems that have th e highestprobability of errors are protected from transient errors through the use of single-biterror correction that uses an error correcting code.


31/48


2.2.2.1 Error-Correcting Code Protection of the Data Interconnect

The entire data path from the local data crossbars and the m emory su bsystem iserror correcting code protected. Single-bit-data errors detected in these subsystems

are corrected by the receiving UltraSPARC III module, and the system is notifiedfor logging pur poses that an error has occurred .

The memory subsystem d oes not check or correct errors, but p rovides the extrastorage bits. The Sun Fire data bu ffer chips u se the error correcting codes to assist infault isolation.

If a correctable error is detected by the int erconnect, the system controller is notifiedand enough informa tion is saved to isolate the failure to a single net w ithin the

interconnect system. The d ata containing the error is sent through the interconnectunchanged, and the error is reported.

Memory errors are logged by software so that defective DIMMs can be identifiedand replaced during scheduled maintenance.

2.2.3 Detecting Uncorrectable ErrorsAlmost all internal system p aths are protected by some form of redu nd ant checkmechanism. Transmission of bad d ata is thus d etected, preventing p ropagation ofbad data withou t notification. All uncorrectable errors will result in a n errorcondition. The recovery requires an operating system autom atic reboot.

2.2.3.1 Multiple-Bit Data ErrorsMultiple-bit ECC errors are detected by the receiving port, which notifies theoperating system, so that depend ing up on w hat p rocess is affected, the system as awh ole can a void failure.

Parity errors on external cache reads to th e interconnect become mu ltibit ECC dataerrors and are han dled as other m ultibit errors.

2.2.3.2 Address Errors

Any single-bit or m ultiple-bit errors detected in th e ad dress interconnect areunrecoverable and a re fatal to the operating system.


32/48


2.2.3.3 System Time-Out Errors

Time-out errors d etected by the p ort controller or mem ory controller are anindication of lost transactions. Time-outs are therefore always unrecoverable.

2.2.3.4 Power Corrected Failures

The Sun Fire system system uses a highly reliable distributed power system. Each I/O su bsystem, CPU/ Memory board, System Controller board , or Repeater boardwithin th e system ha s DC-to-DC converters for that board only, with m ultipleconverters for each voltage. When a DC-to-DC converter fails, the system controller

is notified. The system board reporting the failure will then be deconfigured fromthe system. No gu arantee is made regarding continued system operation at the timeof the failure.

2.2.4 Environmental Sensing

The system chassis environment is monitored for key measures of system stability,such as temp erature, airflow, and p ower su pp ly performance. The system controlleris constantly monitoring the system environm ental sensors in ord er to have enou ghadva nce warn ing of a potential condition so that the m achine can be brough tgracefully to a haltavoiding p hysical dam age to the system an d possiblecorruption of data.

2.2.4.1 TemperatureThe internal temp erature of the system is monitored at key locations as a fail-safemechanism. Based on temp erature read ings, the system can notify the adm inistratorof a potential problem, begin an orderly shutd own , or power off the systemimmediately.

2.2.4.2 Power SubsystemThe Sun Fire system system p erform s add itional sensing to enhance the reliability byenabling constant health checks. DC voltages are monitored at key points within thesystem. DC current from each power supply is monitored and reported to thesystem controller. The CPU p ower control will shut d own any overheating CPUwithout shutting down the system.

The reset signals in the Sun Fire system system is sequenced w ith the DC p ower

levels to gua rantee stability of voltage throughou t the cabinet, prior to removing th ereset signals and enabling norm al operation of the Sun Fire system system logic.


33/48


2.3 AvailabilityFor organizations w hose goal is to make information instantly available to usersacross the ent erp rise, high levels of availability are essential. This is especially tru efor a large shared resource system such as the Sun Fire system.

2.3.1 Availability Goals for the Sun Fire System

The Reliability, Availability, and Serviceability (RAS) goals for the Sun Fire systemsystem is to protect the integrity of the customers data and to maximize availability.The focus is on three areas:

s Problem detection and isolationknowing what went wrong and ensuring thatthe problem is not propagated

s Tolerance and recoveryabsorbing abnormal system behavior and fixing it, ordyn amically circum venting it

s Redu nd ancyreplicating critical components

To ensure dat a integrity at th e hard wa re level, all data is error correction code (ECC)protected, and control buses are protected by p arity checks out to the data on thedisks. These checks ensure that errors are contained.

For tolerance to errors, resilience capabilities are designed into the Sun Fire systemto ensure that the system continues to operate, even in a d egraded mod e. Because itis a symmetrical multiprocessing system, the Sun Fire system system can function

with on e or m ore processors disabled. In recovering from a problem, the system ischecked qu ickly to determine the fault and to ensure minimu m d owntim e. Thesystem can be configured with redundant hardware to reduce downtime.

2.3.2 High Availability Capabilities of the Sun FireSystem

The Sun Fire system system capabilities raise its availability from the normalcommercial category to th e high availability category. These capabilities are groupedas follows:

s Fault tolerant capabilities Any single point of failure is entirely transparent tousers. Users see no loss of performance or capability in the specific areas of thesystem that are fault tolerant.


34/48


s Resiliency capabilitiesThese capabilities enable processing and data access tocontinue in spite of a failure, possibly with reduced resources. These capabilitiesusu ally require that you reboot your system.

s Serviceability capabilitiesThese capabilities lower or eliminate the repair time

wh en a failure occurs.

2.3.2.1 Cooling

The Sun Fire system system has redundant cooling. If one fan fails, the remainingfans autom atically increase their speed, thereby enabling the system to continue tooperate, even at the m aximu m specified am bient. Therefore, operation need n ot be

suspend ed w hen a fan fails. Also, you can rep lace a fan w hile the system isoperating, again withou t any adverse imp act on the availability metric. Of course,the Sun Fire system system has comprehensive and fail-safe temperature monitoringto ensure that th ere is no over-tempera ture stressing of components in the event of acooling failure.

2.3.2.2 AC Power Switching

AC power is supplied to the Sun Fire system through up to four independent,30-amp ere, single-ph ase Redu nd ant Tran sfer Switches (RTS). Each RTS mod ulecarries pow er to tw o or three 1,280-watt bu lk pow er sup plies.

The AC connections m ust be controlled by separate custom er circuit breakers, andcan be on isolated power grids if a high level of availability is required. Optionally,third-party battery backed-up p ower can be used to provide AC pow er in the eventof utility failure.

2.3.2.3 ECC

On th e Sun Fire system system, data errors are d etected, corrected, and / or reportedby the data buffer on behalf of its associated processor. Additionally, data errorspassing through the interconnection w ill be detected and will cause a record stopcondition for the ASICs. The ASICs detect and initiate this condition. These history

buffers and record stop -condition bits can th en be read and used by offlinediagnostics.

2.3.3 Resiliency Capabilities

Resiliency capabilities enable processing and data access to continue in spite of a

failure, possibly with reduced resources. These capabilities usually require that youreboot the system, and th is is counted as repair time in the availability equation.

2 3 3 1 C


35/48


2.3.3.1 DC Pow er

The Sun Fire logic DC power system is modular at the system board level. Bulk56-VDC is supp lied throu gh a circuit protector to each system board . This 56 volts is

converted through several small DC-to-DC converters to the specific low voltagesneeded on the board. Failure of a DC-to-DC converter affects only that particularsystem board. You n eed to configure only as m any bu lk DC pow er sup plies as areneeded for the particular system configuration. The standard redundantconfigurations are three DC power supp lies for up to three system board s and sixDC pow er sup plies for a maximum configuration on the Sun Fire 6800 system.

2.3.3.2 Logic BoardsThe System Controller board contains the system controller interface as well as theclock source and the emergency shutdown logic. Optionally, you can configure twoSystem Controller boards in th e system for red und ancy.

The Repeater, CPU/ Memory boards, and the I/ O su bsystems hold th e DC-to-DCconverters that p ower the ad dress repeater, the system data controller, the systemdata crossbar, and all other ASICs. If one Repeater board fails, the system willcontinu e to operate in a degrad ed m ode, which includes tw o of the four add ressbuses and data buses.

2.3.3.3 Processor

If you have a failure of an UltraSPARC-III processor, the dual data switch, theexternal cache SRAMs, or the associated support ASICs, the failed processor can beisolated from the remainder of the system by a power-on self-test (POST)configu ration step . As long as th ere is at least one fun ctioning processor available inthe configuration, the system can operate.

2.3.3.4 Mem ory

When POST comp letes testing the mem ory subsystem, any faulty banks of mem ory

will be identified. POST can then reconfigure the memory configuration using onlyreliable memory banks, taking adva ntage of the h ighly configurable natu re of theaddress-match logic in the memory controller.

2 3 4 R d d t C t


36/48


2.3.4 Redundant Components

Both the custom er mean time between failure and the customer av ailabilitymeasures of the system are enh anced by the Sun Fire system systems capability toconfigure redun dan t components. There are no comp onents in the system tha tcannot be configured red und antly if the customer desires. Each system board iscapable of independent operation. The Sun Fire system system is built with multiplesystem boards and is inherently capable of operating w hen only a subset of theconfigured boards are fun ctional.

In add ition to the basic system boards, redu nd antly configurable comp onentsinclude:

s System Controller boardss Repeater boardss Bulk pow er subsystemss Bulk pow er supp liess Peripheral controllers and channels

You can configure systems with multiple connections to the peripheral devicesenabling redun dan t controllers and chann els. Softwa re maintains the m ultiple paths

and can switch to an alternate path on the failure of the prim ary path .The system controller is controlled though a console interface workstation.Redund ant system controllers and interfaces can be configured if the customerdesires.

2.4 Serviceability CapabilitiesTo redu ce repair time, the Sun Fire system system has been d esigned with a nu mberof maintenance capabilities and aids. These are u sed by the Sun Fire systemadm inistrator and by the service provider.

There are several capabilities that enable service to be p erformed withou t forcingscheduled dow ntime. Failing comp onents are identified in the failure logs in such a

wa y that the field-replaceable unit is clearly identified. All boards and pow ersup plies in a properly configured system can be removed and replaced d uringsystem operation without scheduled downtime.

2 4 1 M h i l S i bilit


37/48


2.4.1 Mechanical Serviceability

Connectors are keyed so that board s cannot be installed u pside d own . Special toolsare not requ ired to access the inside of the system. This is becau se all voltages withinthe cabinet ar e considered extra-low voltag es (ELVs) as d efined by ap plicable safetyagencies.

No jum per s are required for configur ation of the Sun Fire system system. This makesfor a mu ch easier installation of new and / or up graded system components. Thereare no slot d epend encies other tha n th e special slots required for the SystemController and Repeater boards.

The Sun Fire system system cooling-system design includ es cap abilities that prov idestrength in the area of RAS. Stand ard proven p arts and components are u sedwherever possible. Field-replaceable units (FRUs) and subassemblies are designedfor quick and easy replacement w ith minimal u se of tools required.

2.4.1.1 Bulk Power Supp ly Removal and Replacement

Bulk 56-volt pow er sup plies can be hot-swap ped with n o interrup tion to the system.This assum es that the system is configured from the factory for pow er sup plyredundancy.

2.4.1.2 Fan Tray Removal and Replacement

If a fan fails, the rem aining working fans are set to high-speed operation by thesystem controller in order to compensate for the reduced airflow. The system isdesigned to op erate norm ally un der these conditions un til the failed fan a ssembly isserviced. The fan trays can be hot-swap ped with n o interrup tion to the system.

2.4.1.3 Domain Isolation

The Sun Fire system system has an interconnect domain facility that enables thesystem boards to be assigned to separate software systems. For examp le, one

domain can do production while a second domain experimentally runs the nextrevision of the operating system or exercises a suspected bad board with production-type work.

2.4.2 Nonconcurrent Service

Nonconcurrent service requires the entire system to be pow ered off.

2 4 3 Remote Service


38/48


2.4.3 Remote Service

Every System Controller board has remote access capability that enables remotelogin to the system controller. Through this remote connection, all system controllerdiag nostics are accessible. You can ru n d iagnostics remotely or locally ondeconfigured system boards wh ile the operating system is run ning on the othersystem boards.

CHAPTER 3


39/48

3-1

CHAPTER 3

Hardware Overview

The Sun Fire systems are a family of symm etrical shared-mem ory m ultiprocessors(SMPs).

You can view the Sun Fire systems at several functions of detail:

s Standard operation (simple SMP, OS functions)

s Interconnect (details of interconnect for OS boot and RAS features)

s

Console bus interconnect (details of how the System Controller board controls thesystem)

3.1 Standard Operation

The standard operation is simply that of an SMP run ning an operating system w ithstandard functions. It consists of CPU/ Memory devices and I/ O devices connectedthrough an interconnect bus. Although the d ata interconnect is actually a crossbarswitch, it is logically a bu s. This is illustra ted in FIGURE 3-1.


40/48


FIGURE 3-1 Stand ard Operation of the Sun Fire 6800/ 4810/ 4800/ 3800 Systems

3.2 Data InterconnectAlthou gh the stand ard op eration of the Sun Fire system is that of a simp le bu s-likeinterconnect, it is actually a point-to-point switched interconnect, with two levels ofrepeaters or sw itches.

The switch is capable of complex functions such as:

s Dividing the system into comp letely isolated partitionss Dividing the partition into logically isolated domains

To boot the operating system and to exercise the functions listed above, the systemcontroller needs to be aw are of the logical structure of the sw itch interconnect.

EPCIP C I

I/O controller

Data interconnect

CPUCPU dataswitch

Memory

Data address bus

EPCI P C I

I/O controller

EPCIP C I

I/O controller

CPUCPU dataswitch

Memory

CPUCPU dataswitch

Memory

The Sun Fire 6800 system h as six slots for CPU/ Mem ory b oard s. The Sun Fire 4810/


41/48

Chapter 3 Hardware Overview 3-3

4800 systems have three slots for CPU/ Mem ory boa rd s. The Sun Fire 3800 systemhas tw o slots for CPU/ Memory board s. Each CPU/ Memory board h as up to fourUltraSPARC III 750-MHz or UltraSPARC III 900-MHz CPUs. The CPU also includes

a mem ory controller, and each CPU can sup port one m emory bank w ith up to eightDIMMs.

The Sun Fire 6800 system has fou r bay s for the I/ O a ssemblies. Two bays ar eincluded in the Sun Fire 4810/ 4800/ 3800 systems for I/ O assemblies. The Sun Fire6800/ 4810/ 4800 system s sup por t both PCI and Comp actPCI I/ O chassis. The SunFire 3800 system h as tw o bays for Com pactPCI only I/ O chassis. Each PCI I/ Oassembly has tw o I/ O controllers, each w ith one 66-MHz PCI bus and one 33-MHzPCI bus.

The Sun Fire 6800 system is design ed to grea tly imp rove r eliability, serviceability,and availability (RAS) over previous generations of systems. The Sun Fire system isdesigned to be able to recover from any hardware failure. Some failure recovery willnot imp act users (for examp le, a pow er sup ply failure) if the system is configured forredu nd ant p ower sup plies. Some failure recovery (for examp le, a CPU failure) willrequire a reboot, and w ill impact users, but a p roperly configured system w illalways be able to recover from an y hard ware failure.

The add ress path and data p ath are treated in slightly different w ays. The add resspath has two completely redundant repeaters. A complete address repeater pathrequires two Repeater boards, as the Address Repeater (AR) function is bit-slicedacross two ARs. On the Sun Fire 6800 system, the da ta path is bit-sliced across allfour Repeater boards for standard operation. Optionally, a single pair of Repeaterboards can be u sed in d ouble-pu mp ed mod e, so that full functionality, althoughwith lowered data bandwidth, is retained.

The Repea ter board s have active devices. Because centerp lanes are relatively hard toservice, the Sun Fire 6800/ 4810/ 4800 systems w ere d esigned so tha t n o activedevices are present on the centerplane. The Sun Fire 3800 system, however,incorporates all the active comp onents onto its centerplane.

3.3 Console Bus InterconnectThe console bus enables the system controllers to read and write registers on the restof the system. Only one of the tw o SCs can be master on the console bus at a time.Each system controller is connected to a console bus hub (CBH), and the two CBHsarbitrate for the u se of the console bus.


42/48


CHAPTER 4


43/48

4-1

Sun Fire System Com ponents

The Sun Fire 6800/ 4810/ 4800 systems sh are m any of the sam e comp onent s. Thesecomponents are th e CPU/ Memory board s, the I/ O assemblies, the Repeater board s,and the System Controller boards. The Sun Fire 3800 system also sha res theCPU/ Memory boards, but uses different System Controller boards and I/ O assemblies.

4.1 CPU/ Memory BoardThe CPU/ Memory board is the same across the Sun Fire 6800/ 4810/ 4800/ 3800systems. This board supports up to four UltraSPARC III CPU modules, eight EcacheSIMMs, and eight banks of mem ory, tw o bank p er CPU with four DIMM sockets perbank , for a total of 32 DIMMs.

All DIMMs mu st be the same capacity and sizes within a bank and mu st not beintermixed on a board.

Two CPU/ Mem ory boa rd typ es are ava ilable, one w ith th e UltraSPARC III 750-Mhzprocessor. The second board typ e has th e UltraSPARC III 900-Mhz p rocessor whichwill now be referred to as the CPU/ Memory+ board.

Mixing CPU/ Memory board s with CPU/ Mem ory+ boards in the same chassis is notsupp orted. You m ust install CPU/ Memory board s of the same typ e (either

CPU/ Memory board s or CPU/ Mem ory+ boards) in th e same chassis. You cannotmix CPU/ Memory boards in th e same chassis. The CPU/ Memory+ board s are notbackward comp atible with the CPU/ Memory boards.


44/48


4.2 I/ O AssembliesThe Sun Fire 6800/ 4810/ 4800 systems su pp ort b oth PCI I/ O and Comp actPCI I/ Odevices. The Sun Fire 3800 system supports only CompactPCI I/ O devices.

4.2.1 PCI I/ O

The I/ O assemblies are logically and physically the same for the Sun Fire

6800/ 4810/ 4800 systems. The basic PCI I/ O assembly will have six slots forstandard PCI (33-MHz) device boards, plus two slots for PCI-66 (66-MHz) deviceboards.

4.2.2 CompactPCI I/ O

The Compa ctPCI I/ O assembly is designed for the Compa ctPCI form factor d eviceboards. The Sun Fire 3800 system I/ O assembly for CompactPCI supports a total ofsix slots. The Sun Fire 6800/ 4810/ 4800 systems I/ O assemblies for CompactPCIsup port a total of four slots each.

4.3 Repeater BoardThe Sun Fire 6800/ 4810/ 4800 systems are designed to be repaired and upgradedeasier and faster than prev ious systems. This is du e to the placemen t of active ASICsmou nted on the Repeater boards. With two Repeater boards installed in the system,an a lternate path is available through the second board , if one board fails. The SunFire 3800 system is the only system that comes with all of the active components ofthe Repeater board bu ilt onto the centerplane.

The Repeater board s provide two functions: redu nd ancy for reliability, and a higherband wid th. The system can operate w ith only one Repeater board. The Repeaterboard acts as a switch and connects mu ltiple CPU/ Memory and I/ O boardstogether. The three components are the Address Repeater (AR), the Sun Fire systemdata controller (SDC), and the data crossbar (DX).

In standard operation, the Sun Fire 6800 system has four Repeater boards which areused to route ten buses (six CPU and four I/ O) If one of the Repeater boards fail


45/48

Chapter 4 Sun Fire System Components 4-3

used to route ten buses (six CPU and four I/ O). If one of the Repeater boards fail,the system can continue to operate in a degrad ed m ode w ith one pair of adjacentRepeater board s. The data w idth is cut in half and the tw o Repeater board s will

route the ten bu ses.Because the Sun Fire 4810 and 4800 systems support only two Repeater boards, thetwo Repeater board s operate together to rou te five buses (three CPU and two I/ O).If one of the Repeater boards fail, the data wid th is cut in half and one Repeaterboard can rou te the five buses.

4.4 System Controller BoardThe System Controller board contains the system clock and a service processor. OneSystem Controller board is required per system. You can install one additionalSystem Controller board for redundancy in all the Sun Fire systemsystems. The SunFire 6800 system comes standard with two System Controller boards installed at the

factory. The processor on the board is a microSPARC IIep with its own POST/ OBPflash PROM an d 8 Mbytes of DRAM. The processor also has a 33-MH z PCI bus w ithtwo d evices on it.

With two System Controller boards in each system, if one System Controller boardfails, the other System Controller board can take control of the system withoutcausing a disruption in the main system operation. The System Controller board hasa 10/ 100-BASE-T Ethern et conn ection and an Ebus inter face for a variety of dev ices.These include a TOD/ N VRAM d evice, flash PROM for extra N VRAM sp ace, a large

(4 Mbyte) flash PROM to hold the OS code, and one 16552 dual serial port device.

The System Controller board performs the following main functions:

s Sets up the system and coordinates the boot p rocesss Generates system clockss Monitors the environmental sensors throughout the systems Analyzes errors and takes corrective actions Sets up the system partitions and domains

s Provides the system console functionalitys Provides virtual doma in key switchess Provides network Solaris software console for each domains Provides virtual time-of-day for each d omain


46/48

4.4.3 Solaris Console


47/48

Chapter 4 Sun Fire System Components 4-5

The system controller provides a Solaris software console for each domain. TheEthernet or serial port of the System Controller board is the physical connection for

the Solaris softwa re console. The serial port can su pp ort only one console at a tim e.How ever, the Ethernet p ort can su pp ort m any consoles simu ltaneously. The systemcontroller multiplexes these physical connections to provide console services foreach domain and for the system controller.

4.4.4 Virtual Time Of Day

The Sun Fire system TOD/ NVRAM chip is located on the System Controller board.The system controller multiplexes the physical TOD chip to provide TOD servicesfor each domain and for the System Controller board. The system controller alsoprovides for synchronizing the TOD between the m ain System Controller board andthe redu nd ant System Controller board.

4.4.5 Environmental MonitoringThe Sun Fire systemsystem h as a large num ber of sensors that m onitor temperatu re,voltage, and current. The system controller polls these devices periodically. Ifthresholds are exceeded, the system controller shuts d own various components toprevent damage.


48/48


sunffire6800-4810-4800-3800 system overview manual805-7362-13

Documents