Exploring Failure Transparency and the Limits of Generic Recovery
Dave LowellCompaq Western Research Labxxx
Subhachandra Chandra andPeter M. Chen, University of Michigan
2
Introduction
Failure transparency: abstraction of failure-free operation
OS recovers app after hardware, OS, and application failures
– No programmer help– No slow down
Will explore theory, performance, and limitations
3
Consistent recovery
Visible output equivalent to failure-free run
– equivalence: allows duplicates– avoids “exactly once” problem
Failure transparency consistent recovery with generic techniques
4
Guaranteeing consistent recovery
Key players: non-deterministic events, visible events, commit events
Save-work invariant (simplified):– There’s a commit after each non-
deterministic event that happens-before a visible event.
– Full theorem handles liveness, distinguishes causality and ordering
5
Commit All CAND CAND-LOG
Effort to identify/convert ND events
6
CAND CAND-LOG
Effort to identify/convert ND events
CPVS
CPV-2PCE
ffort
to c
om
mit
onl
y vi
sib
le e
vent
s
CBNDVS
CBNDV-2PC
CBNDVS-LOG
7
CAND CAND-LOG
Effort to identify/convert ND events
CPVS
CPV-2PCE
ffort
to c
om
mit
onl
y vi
sib
le e
vent
s
CBNDVS
CBNDV-2PC
CBNDVS-LOG
Coord. CheckpointingManethoOptimistic Logging
Targon/32SBL Hypervisor
8
Effort to identify/convert ND events
Effo
rt to
co
mm
it o
nly
visi
ble
eve
nts increasing recovery time
app
lica
tion
failu
re r
eco
very
incre
asing
sim
plicit
y
incre
asing
per
form
ance
9
Performance study
Discount Checking: fast checkpoints to reliable memory (Rio)
– Logging and two-phase commit– Disk version
Mostly interactive applications– Localized and distributed
10
CAND1%
43%
CAND-LOG0%
13%
Effort to identify/convert ND events
CPVS1%44%
Effo
rt to
co
mm
it o
nly
visi
ble
eve
nts
CBNDVS1%42%
CBNDVS-LOG0%12%
Nvi Text Editor
11
CAND199%
11499%
CAND-LOG126%
7700%
Effort to identify/convert ND events
CPVS129%7346%
CPV-2PC12%319%
Effo
rt to
co
mm
it o
nly
visi
ble
eve
nts
CBNDVS101%5743%
CBNDV-2PC12% 252%
CBNDVS-LOG73%4973%
TreadMarks Barnes-Hut
12
Have only considered “stop” failures
Committing everything is okay– Save-work: when we must commit
Some failures affect application state– Can we commit too much?
13
Dangerous Paths
14
Dangerous Paths
15
Lose-work invariant
To recover from propagation failure, never commit on a “dangerous path”.
Save-work and Lose-work conflict!– Visible event on dangerous path– Can’t guarantee consistent recovery
from propagation failures
Do we see this conflict in practice?
16
Measuring Lose-work violations
Fault-injection study : OS crashes– injected faults into running kernel– induced 350 OS crashes– recovered nvi and postgres using
Discount Checking
Results– nvi: 15% crashes violate Lose-work– postgres: 3% crashes violate Lose-work
17
Application crashes
Fault-injection study: ND bugs– nvi: 37% violate Lose-work– postgres: 33% violate Lose-work
Published bug distributions: 85-95% of application bugs are deterministic
– intrinsically violate Lose-work
Perhaps > 90% app crashes violate Lose-work!
18
Conclusions
Save-work and Lose-work invariants Save-work protocol space Invariants fundamentally conflict Failure transparency performance:
– 0-12% overhead on reliable memory– 13-40% overhead on disk (interactive apps)
> 90% application failures violate Lose-work
20
21
Chart example
0
10
20
30
40
50
60
70
80
90
100
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
East
West
North
South
