1

Taxonomy and Trends

Dan SiewiorekCarnegie Mellon University

June 2012

2

Outline Taxonomy and Trends General Purpose Examples High Availability Examples A Methodology Conclusion

3

Application Taxonomy General purpose

• Wide range of applications; frequently high performance High availability

• Occasional loss of single user but not system; rapid restart Long life

• No human maintenance; automatically detect and reconfigure; high coverage Critical computations

• Usually real-time control systems; low recovery time; high coverage

4

5

General Purpose Examples

6

Error Detection Techniques in Typical General-Purpose System Memory

• Double-error-detection code on memory data• Parity on address and control information

Cache• Parity on data, address, control information

I/O Unit• Parity on data and control

CPU• Parity on data paths• Parity on control store• Duplication and comparison of control logic

7

Error Recovery Techniques in Typical General-Purpose System

Memory• Single-error-detection code on data• Retry on address or control information parity error

Cache• Retry on data, address, control information parity

error I/O Unit

• Retry on data or control parity errors CPU

• Retry on control store parity error• Invert sense of control store• Macroinstruction retry

8

IBM 3090 Series Fault-Tolerance Features Reliability

• Low intrinsic failure rate technology• Extensive component burn-in during manufacture• Dual processor controller that incorporates switchover• Dual 3370 Direct Access Storage units support

switchover• Multiple consoles for monitoring processor activity and

for backup• LSI packaging vastly reduces number of circuit

connections• Internal machine power and temperature monitoring• Chip sparing in memory replaces defective chips

automatically

9

IBM 3090 Series Fault-Tolerance Features Availability

• Two or four central processors• Automatic error detection and correction in central and

expanded storage– Single bit error correction and double bit error detection in

central storage– Double bit error correction and triple bit error detection in

expanded storage• Storage deallocation in 4K-byte increments under system

program control• Ability to vary channels off line in one channel increments• Instruction retry• Channel command retry• Error detection and fault isolation circuits provide improved

recovery and serviceability• Multipath I/O controllers and units

10

IBM 3090 Series Fault-Tolerance Features

Data integrity• Key controlled storage protection (store and fetch)• Critical address storage protection• Storage error checking and correction• Processor cache error handling• Parity and other internal error checking• Segment protection (S/370 mode)• Page protection (S/370 mode)• Clear reset of registers and main storage• Automatic Remote Support authorization• Block multiplexer channel command retry• Extensive I/O recovery by hardware and control

programs

11

IBM 3090 Series Fault-Tolerance Features Serviceability

• Automatic fault isolation (analysis routines) concurrent with operation

• Automatic remote support capability – auto call to IBM if authorized by the customer

• Automatic customer engineer and parts dispatching

• Trade facilities• Error logout recording• Microcode update distribution via remote support

facilities• Remote service console capability• Automatic validation tests after repair• Customer problem analysis facilities

12

ED/FI in IBM 308X / 3090 Hundreds of thousands of isolation domains Parity checks account for 70-80% of checkers – data,

address, and shift/increment parity predictors Decoder/encoder checkers 25% of IBM 3090 circuits for RAS Can instantaneously detect 90% of all errors 25% of faults assumed solid for the technology If less that two weeks between events, the cause is

assumed to be the same intermittent Call service if 24 errors in 2 hours

13

High Availability Examples

14

Tandem Design Objectives “Nonstop” operation where failures detected,

components configured out of service, repaired components configured back in without stopping other system components

No single hardware failure can compromise data integrity of the system

Modular system expansion through adding more processing power, memory, and peripherals without impacting application software

15

16

17

Fault Containment Software processes do not share state – only message

passing Hardware – no shared memory, dual porting I/O,

multiple power supply

18

Fast-Fail Modules (detection) Software – consistency checks, defensive programming Hardware – software generated status probes,

hardware self-tests

19

Software Bugs

Backup process does not encounter same state and environment, code takes a different path

20

Software Process pairs Transaction processing – two phase commit protocol Log write-ahead protocol – record before and after-

image of database in an audit trail Network systems management – programmed operators

help reduce administrative errors Tandem maintenance and diagnostic system – analyze

event loss to successfully call out FRU 90% of time

21

22

Error Handling Error detection logic records error Operating system runs diagnostics

• Incident of failure algorithm• If transient return board to service• If permanent call Customer Assistant Center – CAC

CAC determines problem• Selects board of same revision level• Print installation instructions• Ship via overnight courier

22 field engineers support 400 systems Service 6% / year of LCC vs. 9% for others

23

24

A Methodology

25

A Methodology Define objectives Limit the scope Define confinement regions Design error handling mechanisms Design error reporting mechanisms Testing of error handling/reporting mechanisms Evaluate design

26

27

28

29

30

31

32

33

34

35

Exercising Latent Faults

Dormant Area Exercise

Memory locations MCU periodically reads every array location (scrubbing)

Detection mechanisms Software* periodically forces error conditions into the detection mechanisms

Reporting mechanisms Software* periodically initiates and observes error reports

Recovery mechanisms Software* periodically invokes recovery operations

*Special commands to support exercising dormant areas are provided in BIUs and MCUs

36

Recovery Mechanisms and Coverage

Mechanism Coverage

Retry Transient errors

ECC Storage array address and data

Spare bit DRAM replacement

Memory bus pairs Memory bus failure

Module shadowing Module failure, GDP, IP, or memory

37

Conclusion

38

Conclusion Designing from first principles to produce an

architecture to tolerate failures achieves better reliability, availability, and cost-effectiveness than an ad-hoc, add-on approach

It is possible to build systems in which the activities of fault detection, diagnosis, and recovery are completely automated and transparent to the user