Efficient Large Scale Efficient Large Scale Content DistributionContent Distribution

WDAS 2004WDAS 2004

By Danny Bickson, By Danny Bickson,

Dahlia Malkhi,Dahlia Malkhi,

David RabinowitzDavid Rabinowitz

Lecture outlineLecture outline

Problem definitionProblem definition Previous workPrevious work Julia algorithmJulia algorithm Comparison to other algorithmsComparison to other algorithms Future directionsFuture directions DemonstrationDemonstration

Problem StatementProblem Statement

Several names:Several names:• Large scale content distributionLarge scale content distribution• High speed data disseminationHigh speed data dissemination• Application multicastApplication multicast

Problem modelProblem model

Network of Network of NN nodes, network fully nodes, network fully connected.connected.

A single source node A single source node SS has complete has complete file Foo of size |Foo| bits.file Foo of size |Foo| bits.

All nodes want to download the file All nodes want to download the file Foo.Foo.

All links have the same bandwidth All links have the same bandwidth (However not the same cost).(However not the same cost).

Performance AnalysisPerformance Analysis

Common criteria for comparison Common criteria for comparison between algorithms:between algorithms:• Download timeDownload time• ConnectivityConnectivity

Locality motivation – Tree exampleLocality motivation – Tree example

Locality Motivation – Tree exampleLocality Motivation – Tree example

Performance AnalysisPerformance Analysis

Fair Sharing- upload/ download ratioFair Sharing- upload/ download ratio Work – number of bits X link distanceWork – number of bits X link distance

All criteria are taken in the worst All criteria are taken in the worst case.case.

Naïve solutionsNaïve solutions

Application Multicast treeApplication Multicast tree

Download time: klogk(n)

Connectivity: n-1

Fair sharing: k

Work: |Foo|(D/2)(n-1)

Locality awareness improves this to be : |Foo|Dn1-1/log(k)

FastReplica FastReplica Cherkasova,Lee -HP Labs - 2003Cherkasova,Lee -HP Labs - 2003

Phase 1Phase 1Source

Clients

FastReplica FastReplica Cherkasova,Lee -HP Labs - 2003Cherkasova,Lee -HP Labs - 2003

Phase 2Phase 2Source

Clients

FastReplica FastReplica Cherkasova,Lee -HP Labs - 2003Cherkasova,Lee -HP Labs - 2003

Scalability?Scalability?

FastReplica FastReplica Cherkasova,Lee -HP Labs - 2003Cherkasova,Lee -HP Labs - 2003

Scalability?Scalability?

Time: O(|Foo|logk(n))Connectivity: (n-1)(k-1)/2Fair sharing: 1Work: |Foo|Dn1- , =1/log(k)

SplitStream SplitStream Castro, Kermarrec, Castro, Kermarrec, Ms Research, Rice - 2003Ms Research, Rice - 2003

Time: O(|Foo|logk(n))Connectivity: (n-1)kFair sharing: 1Work: |Foo|Dn1- , =1/log(k)

Our approachOur approach

Previous approaches do not take Previous approaches do not take locality into account.locality into account.

Enforce “fair sharing” / load Enforce “fair sharing” / load balancingbalancing

Reduce download time to optimalReduce download time to optimal

Julia algorithm motivation: Julia algorithm motivation: “Divide and conquer”“Divide and conquer”

First phase

2nd phase 2nd phase

Julia algorithm motivation: Julia algorithm motivation: “Divide and conquer”“Divide and conquer”

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

Round 0

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8

Round 1 –

Exchange 1 part along longest links

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8

Round 2 –

Exchange 2 parts along half length links

1,3,5,7 2,4,6,8 1,3,5,7 1,3,5,7 1,3,5,72,4,6,8 2,4,6,8 2,4,6,8

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8

Round logN – exchange half of the file along shortest links

1,3,5,7 2,4,6,8 1,3,5,7 1,3,5,7 1,3,5,72,4,6,8 2,4,6,8 2,4,6,8

1-8 1-8 1-8 1-8 1-8 1-8 1-8 1-8

Algorithm propertiesAlgorithm properties

Load balance / Fair sharingLoad balance / Fair sharing Optimal finishing timeOptimal finishing time Local transfer of most parts of the Local transfer of most parts of the

filefile

Comparison SummaryComparison SummaryDownload timeDownload time Number of Number of

edgesedgesTotal workTotal work Fair Fair

sharingsharing

Application Application multicast treemulticast tree

K|F|logK|F|logkk(n)(n) n-1n-1 |F|(D/2)(n-1)|F|(D/2)(n-1) kk

SplitStream SplitStream protocolprotocol

|F|(log|F|(logkk(n)+1)(n)+1) (n-1)k(n-1)k (|F|Dn(|F|Dn1-1-

FastReplica FastReplica protocolprotocol

(2-1/k)|F|log(2-1/k)|F|logkk(n)(n) (n-1)*(n-1)*

(k+1)/2(k+1)/2(|F|Dn(|F|Dn1-1- 11

Our protocolOur protocol 2|F|2|F| nlog(n)nlog(n) (|F|D log(n))(|F|D log(n)) 11

=1/log2(k)

Implementation issues and future Implementation issues and future workwork

Fuzzy vs. structured networkFuzzy vs. structured network How to measure distances in the How to measure distances in the

networknetwork Fault tolerance, dynamic Fault tolerance, dynamic

membershipmembership ScalabilityScalability

Demonstration of software toolsDemonstration of software tools

THE ENDTHE END

Thank you!Thank you!