minimal broker overlay design for content-based publish/subscribe systems naweed tajuddin...
TRANSCRIPT
![Page 1: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/1.jpg)
Minimal Broker Overlay Design for Content-Based Publish/Subscribe
Systems
Naweed Tajuddin
Balasubramaneyam ManiymaranHans-Arno Jacobsen
University of TorontoNovember 18, 2013, CASCON 2013
MSRG.ORG
![Page 2: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/2.jpg)
2MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013 2
Introduction to Publish/Subscribe
• Messaging platform that decouples information sources and sinks
• GooPS – Google P/S– AdSense, Docs, YouTube
• Yahoo Message Broker (within PNUTs)– Data replication for Web Apps– Eventual consistency
![Page 3: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/3.jpg)
3MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
1. Advertise
2. Subscribe
3. Publish
Content-Based Publish/Subscribe
S
P
![Page 4: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/4.jpg)
4MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Challenges for Content-Based Publish/Subscribe
B5 B6
B2B3
B7
S1
B8 B9
P1
P2
B1
B4
Let me know when HP book <
$15
Let me know when HP book <
$15
Sale! HP: $14.99
Sale! MW: $15.99
Sale! OED: $24.99
Amazon.ca
Chapters.ca
![Page 5: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/5.jpg)
5MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Problem Statement
Given a set of publishers and subscribers, how can we design a pub/sub overlay that maximizes performance (delivery latency) and minimizes cost (number of brokers).
INPUT
• Brokers available for deployment (processing capacities)
• Publishers (advertisements)
• Subscribers (subscriptions)
• Publication rates per advertisement
OUTPUT
• Set of deployed brokers
• Client-broker allocation
• Overlay topology
CONSTRAINT
• Broker processing capacities not exceeded
![Page 6: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/6.jpg)
6MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Content Space
priceprice
volu
me
volu
me
2 4 6 8 10 12
2
4
6
8
sub: [price in (2,10)][volume in (2,7)]
pub: [price = 3][volume = 5]
![Page 7: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/7.jpg)
7MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Similarity Model: Interest
• Publisher-subscriber similarity
• Likelihood that a publication will match a subscription
• Geometric intersection between advertisement and subscription over advertisement size
• I = α12 / |a1|
α12
a1
s2
12
7
4
2
2 5 10 12att1att1
att2
att2
![Page 8: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/8.jpg)
8MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Similarity Model: Commonality
• Subscriber-subscriber similarity
• Likelihood that publications matching one subscription will match another subscription
• Geometric intersection over subscription size
• C = α122 / |s1||s2|
α12
s1
s2
12
7
4
2
2 5 10 12att1att1
att2
att2
![Page 9: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/9.jpg)
9MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Estimating Load Impact
• Publishers– Publication rate
• Subscriber– Σpub(Interest * pub rate)
• Brokers– Sum of load impact of local publishers and
subscribers– Load compensation factor: reserve broker capacity to
account for pure forwarding traffic
S1
P1
P2
10 msgs/s8 msgs/s
![Page 10: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/10.jpg)
10MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Solution Overview
• Two-phase Algorithm1. Allocate clients across minimal set of
brokers
2. Cluster brokers with high similarity to form good overlay topology
• Both problems are NP-complete, see paper for proof
![Page 11: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/11.jpg)
11MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
B2
B1
B3
B4
B5
Client 1
Client 2
Client 3
Client 4
Client 5
Brokers Ranked by Capacity
Clients Ranked by Client Ranking Function
B1
Client 1
Client 2
Most similar broker
Client 3
B2
Client 3
Client 4
Client 4
B1
Client 4
Client Allocation
Client-Broker Allocation Algorithm
![Page 12: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/12.jpg)
12MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Client Ranking Function
• Determines the order in which clients are deployed– Impacts how broker capacities are consumed
• #1 Greatest load impact (GLI)– Clients ranked by load imposed on broker
• #2 Greatest interest (GI)– Groups consisting of single publisher-subscriber-pair ranked by
interest • #3 Greatest interest per group (GIg)
– Groups consisting of single publisher and all subscribers with non-zero interest, ranked by greatest interest
• #4 Baseline– Clients ranked and allocated in random order
![Page 13: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/13.jpg)
13MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Overlay Topology Construction
1. Assign weight to every link equal to commonality of broker pair
2. Compute max spanning tree to get overlay topology
a1 a2
a3 a4
s4 s5
s6 s7
s3
s1 s2
A B
C D
![Page 14: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/14.jpg)
14MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Evaluation Overview / Experiment Steps
• Algorithms implemented in Java and simulated using JiST discrete event simulator
1. Execute algorithms to compute overlay design2. Configure pub/sub system simulator overlay
according to overlay design3. Run experiment and record statistics
![Page 15: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/15.jpg)
15MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Overlay Performance
Message count: number of messages generated per publication (or number of broker hops publication must travel to reach all interested subscribers)
![Page 16: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/16.jpg)
16MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Conclusions
• Optimization framework for pub/sub overlay construction
• Similarity framework: Interest and commonality – Tools for overlay construction– Leverage existing semantics of content-based pub/sub
• Load modeling framework– Broker congestion significantly reduced
• Client allocation and overlay topology construction algorithms– Low latency overlay topologies at reduced cost
• Future work– Support additional constraints and incorporate network congestion– Account for physical network and broker capacity model
![Page 17: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/17.jpg)
Thank You!Questions?
![Page 18: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/18.jpg)
** Extra Slides **
![Page 19: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/19.jpg)
19MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013 19
Similarity Model: Commonality
• Subscriber-subscriber similarity• Likelihood that publications matching one subscription
will match another subscription• Geometric intersection over subscription size
• C = α122 / |s1||s2|
7α34
s3
s4
α12
s1
s2
12
7
4
2
2 5 10 12 2 3 8 9
8
54
att1att1
att2
att2
att1att1
att2
att2
class = “sale”class = “sale” class = “sale”class = “sale”
![Page 20: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/20.jpg)
20MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Evaluation Overview
• Algorithms implemented in Java
• Pub/Sub system built using JiST discrete event simulator
• Workload details– 200-1000 publishers– 600-3000 subscribers– 12000-48000 pubs– Pub rates: 1-10 msgs/s– Broker capacity: 1000
msg/s
# of Advs
# of Subs
# of Pubs
![Page 21: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/21.jpg)
21MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Overlay Cost
![Page 22: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/22.jpg)
22MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013 22
Maximum Peak Load
![Page 23: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/23.jpg)
23MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Load and Congestion Effects
![Page 24: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/24.jpg)
24MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
LCF Cost–Performance Tradeoff
![Page 25: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/25.jpg)
25MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013 25
Overlay Design – Related Work
B5 B6
B2B3
B7
S1
B8 B9
P1
P2
B1
B4B5 B6
B2B3
B7
S1
B8
P1
P2
B1
B4
B9
• Rewire overlay[Baldoni et al., 2007][Yoon et al., 2013]
• Move publishers[Cheung et al., 2010]
![Page 26: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/26.jpg)
26MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Estimating Load Impact of Clients
S1
P1
P2• Publisher
– Publication Rate
• Subscriber– Σ(Interest * pub rate)
sS1
aP1 aP2
i1 i2
iS1-P1 = 0.15 iS1-P2 = 0.20
10 msgs/s8 msgs/s
Load impact (S1) = 8(0.15) + 4(0.2)
= 2 msgs/s
![Page 27: Minimal Broker Overlay Design for Content-Based Publish/Subscribe Systems Naweed Tajuddin Balasubramaneyam Maniymaran Hans-Arno Jacobsen University of](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649f525503460f94c76749/html5/thumbnails/27.jpg)
27MIDDLEWARE SYSTEMS
RESEARCH GROUPMSRG.ORG
Nov. 18, 2013
Estimating Broker Load
S1
P1
P2
10 msgs/s8 msgs/s
• Sum of load impact of local publishers and subscriber
• Pure forwarding traffic
• Load compensation factor: reserve broker capacity by factor for pure forwarding traffic