distributed shared memory (part 1). distributed shared memory (dsm) mem0 proc0 mem1 proc1 mem2 proc2...
TRANSCRIPT
Distributed Shared Memory (part 1)
Distributed Shared Memory (DSM)
mem0
proc0
mem1
proc1
mem2
proc2
memN
procN
network
...
shared memory
Shared memory programming
• Standard – pthread• synchronizations
– Barriers – Locks– Semaphores
Sequential SOR
for some number of timesteps/iterations {for (i=0; i<n; i++ )
for( j=1, j<n, j++ )temp[i][j] = 0.25 *
( grid[i-1][j] + grid[i+1][j]
grid[i][j-1] + grid[i][j+1] );for( i=0; i<n; i++ )
for( j=1; j<n; j++ )grid[i][j] = temp[i][j];
}
Parallel SOR with Barriers (1 of 2)
void* sor (void* arg){
int slice = (int)arg;int from = (slice * (n-1))/p + 1;int to = ((slice+1) * (n-1))/p + 1;
for some number of iterations { … }}
Parallel SOR with Barriers (2 of 2)
for (i=from; i<to; i++) for (j=1; j<n; j++)
temp[i][j] = 0.25 * (grid[i-1][j] + grid[i+1][j] + grid[i][j-1] + grid[i][j+1]);
barrier();for (i=from; i<to; i++)
for (j=1; j<n; j++) grid[i][j]=temp[i][j];
barrier();
Differences between SMP and Software DSM
• Delay: tradeoffs, such as block size• Software => traps: cost of
read/write misses• Goals of caches: multiprocessor =
performance, dist. system = transparency
• bus vs. long networks: reliance on serialization and broadcast.
Consequent differences in protocols and applications
• Bigger block size– Cost amortization, higher hit ratio for larger
blocks?– Reduced overhead
• But therefore...– Migration vs. Replication– False sharing increases
• DSM protocol more complex: Must handle lost, corrupted, and out-of-order packets
• Above, coupled with cost of traps, => SDSM consistency cost much higher!
Results of high consistency costs
• Manage sharing more carefully• Align data to page boundaries
Consistency Models
• Sequential Consistency– All processors observe the same order– Must correspond to some serial order– Only ordering constraint is that
reads/writes of P1 appear in the same order, but no restrictions on relative ordering between processors.
Common consistency protocols
• Write update– Multicast update to all replicas
• Write invalidate– Invalidate cached copies in p2, p3– Cache miss if p2/p3 access X
• Valid data from other cache
Conventional Implementation
• As proposed by Li & Hudak, TOCS ‘86.• Use virtual memory to implement
sharing.• Shared memory divided up by virtual
memory pages.• Use single-writer, multiple-reader write-
invalidate coherence protocol.• Keep pages in one of three states:
– invalid, read-only, read-write
Example
proc0 proc1 proc2 procN
shared memory
Example: Read Access Hit
proc0 proc1 proc2 procN
read
Example: Write Access Hit
proc0 proc1 proc2 procN
write
Example: Read Access Miss
proc0 proc1 proc2 procN
read
Example: Read Fault
proc0 proc1 proc2 procN
readfault
Example: Replication on Read
proc0 proc1 proc2 procN
read
Example: Write Access Miss
proc0 proc1 proc2 procN
write
Example: Write Fault
proc0 proc1 proc2 procN
writefault
Example: Write Invalidation
proc0 proc1 proc2 procN
write
Example: Write Access to Read-Only
proc0 proc1 proc2 procN
write
Example: Write Fault
proc0 proc1 proc2 procN
writefault
Example: Write Invalidation
proc0 proc1 proc2 procN
write
How to Remember Locations?
• Broadcast on miss (as in SMP).• Static home.• Dynamic home or owner.
Ownership and Owner Location
• Owner is the last writer.• Owner maintains copyset.• Every processor maintains
probable owner (not always the real owner).
Ownership Location
• Every read or write miss is sent to (local) probable owner.
• If owner, handle appropriately, else forward to probable owner.
Ownership Modification
• If write miss, new writer becomes owner, and all forwarders set probable owner to requester.
• If read miss, set probable owner to responding processor.
Example
• Initially, owner(page0) = p0, and probable owner(page0) = p0 everywhere.
• Write miss by p1, sends message to its probable owner (p0), handled there, new owner = p1, probable owner(0) on p0 = 1.
• Read miss by p2, sends message to probable owner (p0), forwarded to probable owner (p1), handled there, probable owner(0) on p2 becomes p1.
Implement synchronizations
• Use messages to implement synchronizations
Barriers
• Designate one processor as barrier manager.
• When a process waits at a barrier, it sends an arrival message to the barrier manager and waits.
• When barrier manager has received all messages, it sends a departure message to all processes.
Locks
• Designate one process as the lock manager for a particular lock.
• When a process acquires a lock, it sends an acquire message to the manager and waits.
• Manager forwards message to last acquirer.
• If lock free, send lock grant message.• If lock held, hold on to request until
free, and then send lock grant message.
Problem: False Sharing
• Concurrent access to different data within the same consistency unit.
• With page as consistency unit, lots of opportunity for false sharing.
• Two flavors:– read-write – write-write
Read-Write False Sharing
x
y
Read-Write False Sharing (Cont.)
w(x)
r(y) r(y) r(x)
synch
w(x) w(x)
Read-Write False Sharing (Cont.)
w(x)
r(y) r(y) r(x)
synch
w(x) w(x)
Write-Write False Sharing
w(x)
w(y) w(y) r(x)
synch
w(x) w(x)
Summary
• Software shared memory on distributed memory hardware.– Uses virtual memory.
• Home migration to improve locality– important because of high latencies.
• Sequential consistency suffers from false sharing