COP 5611 Operating Systems Spring 2010

Dan C. MarinescuOffice: HEC 439 BOffice hours: M-Wd 1:00-2:00 PM

222222

Lecture April 5, 2010 Homework 3 due on Wednesday – April 14 Last time:

All or nothing atomicity Today:

Before or after atomicity Next time

Conclude atomicity

_________

begin all-or-nothing action_________ arbitrary sequence of___ lower-layer actions___

end all-or-nothing action_________

}

___ first step of all-or-nothing action_________ Pre-commit discipline: can back out, ___ leaving no trace_________ Commit point_________ Post-commit discipline: completion is inevitable______ last step of all-or-nothing action

}}

The golden rule of atomicityNever modify the only copy!

History of earlier versionsTentativenext version

Current version

1614112295207Variable A:

Version history of one variable V from the journal storage

All-or-nothing Journal Storage System

Cell StorageSystem

NEW_ACTI ON

– catalogs

– versions

– outcomerecords

Journal

READ

W RI TE

ALLOCATE

DEALLOCATE

ABORT

COMMI T

READ_CURRENT_VALUE

WRI TE_NEW _VALUE StorageManager

Outcome record

Created when the application calls the NEW_ACTION primitive of the journal storage manager call

It is created instate PENDING When the application calls WRITE_NEW_VALUE of a data object it

should supply The identifier of the data object The tentative new value The identification of the all_or_nothing_action

8

Example of an outcome record

9

7

03

outcomerecords

Object A

pending1794:aborted1423:1101: committed

1101 1423 1794all-or-nothingaction id:

751524value:

new all-or-nothingaction iscreated

aborted

all-or-nothingactionaborts

outcome recordstate no longerof any interest

all-or-nothingactioncommits committed

pendingnon-existent

discarded

The state transitions of an outcome record

procedure NEW _ACTION ()id NEW_OUTCOME _RECORD ()id.outcome_record.state PENDINGreturn id

procedure COMMIT (reference id)id.outcome_record.state COMMITTED

procedure ABORT (reference id)id.outcome_record.state ABORTED

Algorithms to read the current value and write a new value

If the current action fails before the COMMIT the new version of the variable is not visible to anyone wishing to read the current value.

If an action changes multiple values but it fails before commit none of the modified values will be visible.

12

procedure READ _CURRENT_VALUE (data_id, caller_id)starting at end of data_id repeat until beginning

v previous version of data_id // Get next older versiona v.action_id // Identify the action a that created its a.outcome_record.state // Check action a’s outcome recordif s = COMMITTED then

return v.valueelse skip v // Continue backward search

signal (“Tried to read an uninitialized variable!”)

procedure WRITE _NEW_VALUE (reference data_id, new_value, caller_id)if caller_id.outcome_record.state = PENDING

append new version v to data_idv.value new_valuev.action_id caller_id

else signal (“Tried to write outside of an all-or-nothing action!”)

procedure TRANSFER (reference debit_account, reference credit_account, amount)my_id NEW_ ACTION ()xvalue READ _CURRENT_VALUE (debit_account, my_id)xvalue xvalue - amountWRITE _ NEW_VALUE (debit_account, xvalue, my_id)yvalue READ _CURRENT_VALUE (credit_account, my_id)yvalue yvalue + amountWRITE _ NEW_VALUE (credit_account, yvalue, my_id)if xvalue > 0 then

COMMIT (my_id)else

ABORT (my_id)signal(“Negative transfers are not allowed.”)

procedure TRANSFER (debit_account, credit_account, amount)GET dbdata, debit_accountdbdata dbdata - amountPUT ( dbdata, debit_account)GET crdata, credit_accountcrdata crdata + amountPUT ( crdata, credit_account)

Atomicity logs

The recording to the journal storage optimized for speed of writing by creating a single interleaved version history of all variables.

The writing to cell storage optimized for fast retrieval. The log is the authoritative record of the outcome of an action. The log resides in non-volatile storage

15

WRI TE_NEW_VALUE

Log

Journal Storage

Cell

loginstall

Storage

READ_CURRENT_VALUE

currentend of log

Write first to the journal and then to the cell storage

Volatile storage Non-volatile storage

log

log

Applicationprogram

Applicationprogram

Applicationprogram

cellstorage

cellstorage

logcell

storagecache

In-memory database:

High-performance

Ordinary database:

database:

Write-ahead-log protocolLog the update before installing it.

Logging protocols

Logging record

A log record contains the following information: The identity of the action A component action that if performed installs the intended value in cell

storage A component action which if performed reverses the effect of the action

upon the cell storage An application appends a log record by invoking the lower level

primitive action LOG. LOG is an atomic action itself

19

procedure TRANSFER (debit_account, credit_account, amount)my_id LOG ( BEGIN _TRANSACTION )dbvalue.old GET (debit_account)dbvalue.new dbvalue.old - amountcrvalue.old GET (credit_account, my_id)crvalue.new crvalue.old + amountLOG ( CHANGE , my_id,

“PUT (debit_account, dbvalue.new)”, //redo action“PUT (debit_account, dbvalue.old)” ) //undo action

LOG ( CHANGE , my_id, “PUT (credit_account, crvalue.new)” //redo action“PUT (credit_account, crvalue.old)”) //undo action

PUT (debit_account, dbvalue.new) // installPUT (credit_account, crvalue.new) // installif dbvalue.new > 0 then

LOG ( OUTCOME , COMMIT , my_id)else

LOG (OUTCOME , ABORT , my_id)signal(“Action not allowed. Would make debit account negative.”)

LOG ( END_TRANSACTION , my_id)

All-or-nothing TRANSFER with logging

9979

PUT(debit_account, $120)newer log records

action_id:redo_action:

undo_action:

CHANGEtype:9974

COMMITTEDaction_id:

status:

OUTCOMEtype:9979

PUT(credit_account, $40)

PUT(credit_account, $10)

action_id:redo_action:

undo_action:

CHANGEtype:

older log records

…PUT(debit_account, $90)

Examples of two CHANGE records for a TRANSFER action with action_id=9979 and an OUTCOME record for action_id=9974

procedure ABORT (action_id) starting at end of log repeat until beginning

log_record previous record of logif log_record.id = action_id then

if (log_record.type = OUTCOME )then signal (“Can’t abort an already completed action.”)

if (log_record.type = CHANGE )then perform undo_action of log_record

if (log_record.type = BEGIN ) then break repeat

LOG (action_id, OUTCOME , ABORTED ) // Block future undos.LOG (action_id, END)

Recovery procedure

The recovery is function of the database layout If an in-core database:

Two passes First pass scan the log backwards from the last record (LIGO scan).

Collect the identity and the completion status of all all_or_noting actions.The action which were either COMMITT-ed or ABORT-ed are called winners.

Forward scan perform all the REDO actions

23

procedure RECOVER () // Recovery procedure for a volatile, in-memory database.winners NULLstarting at end of log repeat until beginning

log_record previous record of logif ( log_record.type = OUTCOME )

then winners winners + log_record // Set addition.

starting at beginning of log repeat until endlog_record next record of logif ( log_record.type= CHANGE )

and ( outcome_record find (log_record.action_id) in winners) and (outcome_record.status = COMMITTED ) thenperform log_record.redo_action

procedure RECOVER () // Recovery procedure for non-volatile cell memorycompleteds NULLlosers NULLstarting at end of log repeat until beginning

log_record previous record of logif ( log_record.type = END )

then completeds completeds + log_record // Set addition.if ( log_record.action_id is not in completeds) then

losers losers + log_record // Add if not already in set.if ( log_record.type = CHANGE ) then

perform log_record.undo_action

starting at beginning of log repeat until endlog_record next record of logif ( log_record.type = CHANGE )

and ( log_record.action_id.status = COMMITTED ) thenperform log_record.redo_action

for each log_record in losers dolog (log_record.action_id, END ) // Show action completed.

procedure RECOVER () // Recovery procedure for rollback recovery.completeds NULLlosers NULLstarting at end of log repeat until beginning // Perform undo scan.

log_record previous record of logif ( log_record.type = OUTCOME )

then completeds completeds + log_record // Set addition.if ( log_record.action_id is not in completeds) then

losers losers + log_record // New loser.if (log_record.type = CHANGE ) then

perform log_record.undo_action

for each log_record in losers dolog (log_record.action_id, OUTCOME , ABORT ) // Block future undos.

Before or after atomicity

How to provide atomicity for concurrent actions? The version history mechanism does not prevent concurrent actions

from accessing pending changes. The trivial solution – simple serialization

Recall that the version history assigns a unique identifier to each atomic action to link tentative versions of a variable with the outcome record of the action.

Now we require that the atomic actions are assigned consecutive integer values.

Each newly created transaction, transaction n, must wait before reading or writing any shared variable until the previous (n-1) transactions have either committed or aborted.

27

procedure BEGIN _TRANSACTION ()id NEW_OUTCOME _RECORD (PENDING ) // Create, initialize, assign id.previous_id id – 1wait until previous_id.outcome_record.state ° PENDINGreturn id

Simple serialization requires a modified BEGIN_TRANSACTION

Object value of object at end of transaction

1 2 3 4 5 6

A

B

C

D

Committed Committed Committed Aborted Committed Pending

0

0

0

0

+10

-10 -6

-4

+12

-2

-12

+2

0

-2

outcomerecordstate

transaction 1: initialize all accounts to 02: transfer 10 from B to A3: transfer 4 from C to B4: transfer 2 from D to A (aborts)5: transfer 6 from B to C6: transfer 10 from A to B

Serialization

Threads can still run in concurrently as serializations affects them only during the execution of transactions which handle shared variables.

Let see if we can relax the stringent requirement of serialization. Place dotted boxes for variables not affected by each transaction.

30

1 2 7Object Value of object at end of transaction

3 4 5 6

A

B

C

D

Committed Committed Committed Aborted Committed Pending

0

0

0

0

+10

-10 -6

-4

+12

-2

-12

+2

0

-2

OUTCOMErecordstate

Pending

Unchanged value

Changed value

+10 +12 0

-6 -2

0 -4 +2 +2

0 0 -2 -2 -2

Mark-point discipline

Concurrent thread which invoke READ_CURRENT_VALUE should not see the PENDING version of any variable.

A transaction can reveal all its results by changing the value of the OUTCOME record to COMMITTED.

A transaction should also be able to wait for a PENDING value. Solution create pending values for all variables an application

intends to modify and announce when it has finished this process Marking point the time when all pending values have been created Marking point discipline no transaction can begin reading its

inputs until the preceding transactions has reached its marking point or is no longer pending.

32

procedure READ _CURRENT _VALUE (data_id, this_transaction_id)starting at end of data_id repeat until beginning

v previous version of data_idlast_modifier v.action_idif last_modifier this_transaction_id then skip v // Keep searchingwait until ( last_modifier.outcome_record.state ° PENDING )if (last_modifier.outcome_record.state = COMMITTED )

then return v.stateelse skip v // Resume search

signal (“Tried to read an uninitialized variable”)

READ_CURRENT_VALUE for the mark point discipline: 1. Skip all versions created by transaction later than the current transaction2. Wait for a pending version created by an earlier transaction until that transaction either COMMITs or ABBORTs

procedure NEW _VERSION (reference data_id, this_transaction_id)if this_transaction_id.outcome_record.mark_state = MARKED then

signal (“Tried to create new version after announcing mark point!”)append new version v to data_idv.value NULLv.action_id transaction_id

procedure WRITE _VALUE (reference data_id, new_value, this_transaction_id)starting at end of data_id repeat until beginning

v previous version of data_idif v.action_id = this_transaction_id

v.value new_valuereturn

signal (“Tried to write without creating new version!”))

Mark-point discipline versions of:1. NEW_VERSION 2. WRITE_VALUE 3. BEGIN_TRANSACTION 4. NEW_OUTCOME_RECORD

procedure BEGIN _TRANSACTION ()id NEW_OUTCOME _RECORD (PENDING )previous_id id - 1wait until (previous_id.outcome_record.mark_state = MARKED )

or (previous_id.outcome_record.state ° PENDING )return id

procedure NEW_OUTCOME _RECORD ( starting_state)ACQUIRE (outcome_record_lock) // Make this a before-or-after action.id TICKET ( outcome_record_sequencer)allocate id.outcome_recordid.outcome_record.state starting_stateid.outcome_record.mark_state NULLRELEASE (outcome_record_lock)return id

procedure MARK _POINT _ANNOUNCE (reference this_transaction_id) this_transaction_id.outcome_record.mark_state MARKED

procedure TRANSFER (reference debit_account, reference credit_account,amount)

my_id BEGIN _TRANSACTION ()NEW_VERSION (debit_account, my_id)NEW_VERSION (credit_account, my_id)MARK_POINT _ANNOUNCE (my_id);xvalue READ _CURRENT_VALUE (debit_account, my_id)xvalue xvalue - amountWRITE _VALUE (debit_account, xvalue, my_id)yvalue READ _CURRENT_VALUE (credit_account, my_id)yvalue yvalue + amountWRITE _VALUE (credit_account, yvalue, my_id)if xvalue > 0 then

COMMIT (my_id)else

ABORT (my_id)signal(“Negative transfers are not allowed.”)

Mark-point version of the TRANSFER procedure

Optimistic atomicity

Presume that interference between concurrent transactions is not likely. Allow the transactions to proceed without waiting. Watch for actual interference and take some recovery acction if it indeed

happens. Read-capture discipline:

A transaction does not need to predict the identity of every object it will update – it will discover the identity of these objects as it proceeds.

When calling READ_CURRENT_VASLUE make a mark of the thread position in the version history.

If a thread earlier in the serial ordering wants to insert but arrives late it must abort and try agaian using a later version in the version history.

37

1 2 7

Value of object at end of transaction

3 4 5 6

A

B

C

D

Committed Committed Committed Aborted Committed Pending

0

0

0

0

+10

-10 -6

-4

+12

-2

-12

+2

0

-2

Outcome recordstate

Pending

Conflict

Changed value

HWM=3

HWM=2 HWM=6

HWM=3 HWM=5HWM=2

HWM=5

Conflict: Must abort!

HWM=6

High-water markHWM=

HWM=7

HWM=7HWM=4-4

+2

procedure READ _CURRENT _VALUE (reference data_id, value, caller_id) starting at end of data_id repeat until beginning

v previous version of data_idif v.action_id caller_id then skip vexamine v.action_id.outcome_record

if PENDING thenWAIT for v.action_id to COMMIT or ABORTif COMMITTED then

v.high_water_mark max (v.high_water_mark, caller_id)return v.value

else skip v // Continue backward searchsignal (“Tried to read an uninitialized variable!”)

procedure NEW _VERSION (reference data_id, caller_id)if ( caller_id < data_id.high_water_mark) // Conflict with later reader.or (caller_id < ( LATEST _VERSION [data_id].action_id)) // Blind write conflict.then ABORT this transaction and terminate this threadadd new version v at end of data_idv.value 0v.action_id caller_id

procedure WRITE _VALUE (reference data_id, new_value, caller_id)locate version v of data_id.history such that v.action_id = caller_id

(if not found, signal (“Tried to write without creating new version!”))v.value new_value

n

n + 1

n + 2

R5

R4

R5

42

61

29

three entries in the reorder buffer

architecturalregister

physicalregisterinstruction

physical register file

0

127

with 128 registers

n R5 R4 R2 // Write a result in register five.n + 1 R4 R5 + R1 // Use result in register five.n + 2 R5 READ (117492) // Write content of a memory cell in register five.

procedure PAY _ INTEREST (reference account)if account.balance > 0 then

interest = account.balance * 0.05TRANSFER (bank, account, interest)

elseinterest = account.balance * 0.15TRANSFER (account, bank, interest)

procedure MONTH _END _ INTEREST : ()for A each customer_account do

PAY _ INTEREST (A)

outcome:

superior:

MONTH_END_ I NTERES T

PENDING

outcome:

superior:

PAY_ I NTERES T 1 (1st invocation )

COMMI TTED

MO NTH_END_ I NTEREST

outcome:

superior:

TRANSFER 1

COMMI TTED

PAY_ INTEREST 1

creator: TRANS FE R 1

newest version of

OK for TRANSFER 2

none

outcome:

superior:

PAY_ I NTERES T 2 (2nd invocation)

PENDI NG

MONTH_END_ I NTERES T

outcome:

superior:

TRANSFER 2

PENDING

PAY_ INTEREST 2

account bank

to read?

From: AliceTo: BobRe: my transaction 91if (Charles does Y and Dawn does Z) then do X, please.

From:AliceTo: BobRe: my transaction 271

Please do X as part of my transaction.

From:BobTo: AliceRe: your transaction 271

My part X is ready to commit.

Two-phase-commit message #1:From:AliceTo: BobRe: my transaction 271

PREPARE to commit X.

Two-phase-commit message #2:From:BobTo: AliceRe: your transaction 271

I am PREPARED to commit my part. Have you decided to commit yet?

Two-phase-commit message #3From:Alice To: BobRe: my transaction 271My transaction committed. Thanks for your help.

AliceWorker

Bob Charles Dawn

PREPARE X

PREPARE Y

PREPARE Z

Bob is PREPARE D

Charles is PREPAREDDawn is PREPARED

COMMIT

COMMI T

COMMI T

Time

Coordinator Worker Worker

log BEGIN

log

log BEGIN

log PREPARED

log COMMI TTED

COMMI TTED

From:Julius CaesarTo: Titus LabienusDate:11 January

I propose to cross the Rubicon and attack at dawn tomorrow. OK?

From:Titus Labienus or From:Titus LabienusTo: Julius Caesar; To: Julius CaesarDate:11 January Date:11 January

Agreed, dawn on the 12th. No. I am awaitingreinforcements from Gaul.

From:Julius CaesarTo: Titus LabienusDate:11 January

The die is cast.

procedure ALL _OR _NOTHING _DURABLE _GET (reference data, atomic_sector)ds CAREFUL_ GET (data, atomic_sector.D0)if ds = BAD then

ds CAREFUL_GET (data, atomic_sector.D1)return ds

procedure ALL _OR _NOTHING _DURABLE _PUT (new_data, atomic_sector)SALVAGE (atomic_sector)ds CAREFUL_ PUT (new_data, atomic_sector.D0)ds CAREFUL_ PUT (new_data, atomic_sector.D1)return ds

procedure SALVAGE (atomic_sector) //Run this program every Td seconds.ds0 CAREFUL _GET (data0, atomic_sector.D0)ds1 CAREFUL _GET (data1, atomic_sector.D1)if ds0 = BAD then

CAREFUL_PUT (data1, atomic_sector.D0)else if ds1 = BAD then

CAREFUL_PUT (data0, atomic_sector.D1)if data0 ° data1 then

CAREFUL_PUT (data0, atomic_sector.D1)

data0 data1D0: D1 :