dryad and dryalinq
DESCRIPTION
Dryad and DryaLINQ. Dryad and DryadLINQ. Dryad provides automatic distributed execution DryadLINQ provides automatic query plan generation. Dryad. General-purpose execution environment for distributed, data-parallel applications - PowerPoint PPT PresentationTRANSCRIPT
Dryad and DryaLINQ
Dryad and DryadLINQ
Dryad provides automatic distributed executionDryadLINQ provides automatic query plan generation
Dryad
• General-purpose execution environment for distributed, data-parallel applications
• Focus on simplicity, reliability, scalability, efficiency and not latency, unreliable networks
• Automatic management of scheduling, distribution, fault tolerance
• Exploits Data Parallelism
Dryad
• Computations expressed as a Directed Acyclic Graph– Jobs executed on vertices– Edges are communication channels– Each vertex has several input and output edges– Data transport mechanisms: Files, TCP pipes,
shared memory FIFOs
Job = Directed Acyclic Graph
Processingvertices Channels
(file, pipe, shared memory)
Inputs
Outputs
Dryad vs. MapReduce, Parallel DB
• More control to developer than MapReduce• MapReduce aims at simplicity at the expense
of generality and performance• Computation Graph is implicit in Parallel DB
Dryad System Architecture
• Job manager – coordinates jobs, constructs graph• Name server – exposes computers with network
topology• Daemons run on each computer in the cluster
Communication
Job (Graph) Construction
• Using graph operators implemented in C++ to describe the graph (from simpler sub graphs).
Job Execution• Job manager not currently fault tolerant• Vertices may be scheduled multiple times due to
failures – Each execution versioned– Execution record kept- including versions of incoming vertices– Outputs are uniquely named (versioned)– Final outputs selected if job completes– Non-file communication (TCP pipe, Shared Memory FIFO)
may cascade failures• Vertices specify hard constraints or preferences for set
of computers required• Scheduling is greedy assuming only one job
11
Policy Managers
R R
X X X X
Stage RR R
Stage X
Job Manager
R managerX ManagerR-X
Manager
Connection R-X
Cluster network topology
rack
top-of-rack switch
top-level switch
Run-time Graph Refinement
14
S S S S
A A A
S S
T
S S S S S S
T
# 1 # 2 # 1 # 3 # 3 # 2
# 3# 2# 1
static
dynamic
rack #
Dynamic Aggregation
Fault Tolerance
SkyServer DB Query
• 3-way join to find gravitational lens effect• Table U: (objId, color) 11.8GB• Table N: (objId, neighborId) 41.8GB• Find neighboring stars with similar colors:
– Join U+N to findT = U.color,N.neighborId where U.objId = N.objId
– Join U+T to findU.objId where U.objId = T.neighborID
and U.color ≈ T.color
D D
MM 4n
SS 4n
YY
H
n
n
X Xn
U UN N
U U
• Took SQL plan• Manually coded in Dryad• Manually partitioned data
SkyServer DB query
u: objid, colorn: objid, neighborobjid[partition by objid]
select u.color,n.neighborobjidfrom u join nwhere u.objid = n.objid
(u.color,n.neighborobjid)[re-partition by n.neighborobjid][order by n.neighborobjid]
[distinct][merge outputs]
select u.objidfrom u join <temp>where u.objid = <temp>.neighborobjid and |u.color - <temp>.color| < d
Optimization
D
M
S
Y
X
M
S
M
S
M
S
U N
U
D D
MM 4n
SS 4n
YY
H
n
n
X Xn
U UN N
U U
Optimization
D
M
S
Y
X
M
S
M
S
M
S
U N
U
D D
MM 4n
SS 4n
YY
H
n
n
X Xn
U UN N
U U
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
0 2 4 6 8 10
Number of Computers
Spe
ed-u
pDryad In-Memory
Dryad Two-pass
SQLServer 2005
High level Programming Languages
• Nebula – limited to existing binaries• SSIS – SQLServer workflow engine, distributed• DryadLINQ – Supports both imperative and
declarative operations on datasets
Dryad/DryadLINQ
• Decoupling of Dryad and DryadLINQ– Dryad: execution engine (given DAG, do scheduling and
fault tolerance)– DryadLINQ: programming model (given query, generate
DAG)
DryadLINQ• Exploits LINQ (Relational queries integrated in C#) to provide
a hybrid of imperative and declarative programming• LINQ has a design choice that is easy to express computations
also giving runtime leeway implementing them.• Sequential program composed of LINQ expressions• Performs side-effect free transformations on datasets• Written and Debugged using .NET development tools• More general than distributed SQL• Programs can be automatically optimized and efficiently
executed on large cluster
DryadLINQ
• Serialization for dryad are provided by High level software layers like DrayLINQ
• DrayLINQ preserves the LINQ programming model and defines new operators and datatypes for data parallel programming
DryadLINQ Architecture
DryadLINQ Data Model
Partition
Partitioned Table
.Net objects
Data Model is distributed implementation of LINQ CollectionsEach Dataset is distributed (disjoint) across the clusterPartitioned table exposes metadata information
– type, partition, compression scheme, serialization, etc.
DrayLINQ Constructs
• Expressions must be side-effect free• Allows programmer to specify annotations
(hints) to guide optimization• Operators
– Hash Partition– Range Partition– Apply: Allows arbitrary streaming computations– Fork: Takes single input and generates multiple
output datasets
System Implementation
• Execution Plan Graph: Starts by converting raw LINQ expressions into EPG
• DryadLINQ Optimizations– Static Optimizations– Dynamic Optimizations
• Code Generation: Uses dynamic code generation to automatically synthesize LINQ code to be run at the Drayad vertex
Conclusions• Goal: Use a compute cluster as if it is a single
computer– Dryad/DryadLINQ represent a significant step
• Requires close collaborations across many fields of computing, including– Distributed systems– Distributed and parallel databases– Programming language design and analysis
References
• Dryad: Distributed Data-parallel Programs from Sequential Building Blocks (Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly March 2007)
• DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language (Yuan Yu, Michael Isard, Dennis Fetterly, Mihai Budiu, Úlfar Erlingsson, Pradeep Kumar Gunda, and Jon CurreyDecember 2008)
Thank you