2008_Kalman.Graffi_QuaP2P_Kolloquium_Efficiency.Management.ppt
KOM - Multimedia Communications LabProf. Dr.-Ing. Ralf Steinmetz (director)
Dept. of Electrical Engineering and Information TechnologyDept. of Computer Science (adjunct professor)
TUD – Technische Universität Darmstadt Merckstr. 25, D-64283 Darmstadt, Germany
Tel.+49 6151 164959, Fax. +49 6151 166152 www.KOM.tu-darmstadt.de
© author(s) of these slides 2008 including research results of the research network KOM and TU Darmstadt otherwise as specified at the respective slide8. April 2023
Dipl.-Math. Dipl.-Inform. Kalman Graffi
Efficiency and Information Management in Peer-to-Peer Systems
QuaP2P Kolloquium, 9. April 2008
7.31.10.25
peer-to-peer.info
12.5.7.31
95.7.6.10
86.8.10.18
planet-lab.orgberkeley.edu 89.11.20.15
Efficiency and Information Management
KOM – Multimedia Communications Lab 2
Overview
1 The Peer-to-Peer Paradigm
1.1 Trends in Peer-to-Peer Research
1.2 Efficiency in Peer-to-Peer Systems
2 Towards QoS & Emergency Call Handling
2.1 Serious Application: Emergency Call Handling
2.2 Towards QoS for Overlay Flows
2.3 Overlay Bandwidth Management
3 Lessons Learned for QoS in P2P Systems
4 Towards a Kind of „Efficiency Management”
4.1 Current State of Efficiency Management
4.2 Our Vision of an Efficiency Management Lifecycle
4.3 Efficiency Management Module
4.4 Current Research Focus: Improved Architecture for Efficiency Management
5 Lessons Learned for Efficiency Management in P2P Systems
KOM – Multimedia Communications Lab 3
The Peer-to-Peer Paradigm
Peer-to-Peer Systems: Users of a system provide the infrastructure of the system Service is provided from users/peers to users/peers Peer-to-Peer overlays:
virtual networks, providing new functionality E.g. Distributed Hash Tables, Keyword-based Search
Evolution of applications File sharing:
No Quality of Service (QoS) requirements Voice over IP
Real-time requirements Video-on-demand
Real-time and bandwidth requirements
H(„my data“)
= 31072207
7.31.10.25
peer-to-peer.info12.5.7.31
95.7.6.10
86.8.10.18
planet-lab.orgberkeley.edu
29063485
201116221008709
611
61.51.166.150
?
1
KOM – Multimedia Communications Lab 4
Trends in Peer-to-Peer Research
Quality aspects gain importance Reliability: expected professionalism Ease of Use:
Multimedia and interactivity
Critical success factor for complex P2P applications modular P2P applications
Quality aspects: Adaptability – to scenario, system scale Validity – of stored data Trust – of users and mechanisms Efficiency – ratio between performance and costs
Costs Security
Quality of P2P Systems
Retrievability
Coherence
Consistency
Correctness
PerformanceScalability
Flexibility
Stability
Dependability
Service Provisioning
Overlay Operations
Individual Node
Complete System
IP Infrastructure
Availability
Reliability
Robustness/ Fault tolerance
Integrity
Confidentiality
Authentication
Non- repudiation
TrustValidityEfficiencyAdaptability
Costs Security
Quality of P2P Systems
Retrievability
Coherence
Consistency
Correctness
PerformanceScalability
Flexibility
Stability
Dependability
Service Provisioning
Overlay Operations
Individual Node
IP Infrastructure
Availability
Reliability
Robustness/ Fault tolerance
Integrity
Non-
TrustValidityEfficiencyAdaptability
1.1
KOM – Multimedia Communications Lab 5
Efficiency in Peer-to-Peer Systems
Future Peer-to-Peer based applications Modular, component based composition A module has to
be highly efficient provide Quality of Service
Efficiency can be viewedin three aspects: Resource usage
in the peer in the overlay in the network
Question in peer-centric view: local knowledge vs. system wide solution Devices
IP Infrastructure
3
$
$
P2P Overlay
2
^Peer
IN OUT
10100110110001000110
1
1.2
KOM – Multimedia Communications Lab 6
Efficiency Management in Peers
Taxonomy on Sched. and AQMMechanisms in P2P ScenariosK. Graffi et al.Technical Report KOM-2007-1/2, 2007
Taxonomy of Scheduling and Active Queue Mechanisms on Overlay Flows
Welche Strategien aus der Netzwerk-schicht sind auch im Overlay anwendbar?
01/2007
1. Message SchedulingBefore:
After:
2. Queue ManagementBefore:After:
Queue Limit
Overlay Bandwidth Management – Scheduling and Active Queue Management of Overlay Flows
Overlay Bandwidth Mana-gement: Sched. and AQM of Overlay Flows, K.Graffi et al., IEEE LCN ‘07
Wie kann man QoS für Overlay Flows erbring-en? Welche Verfahren sind geeignet? Sind lokale Lösungen aus-reichend?
10/2007
Wie kann man QoS (geringe Latenz, Loss) für P2P-basierte Notrufdienste erbringen? Anwendung der Erkenntnisse.
ECHoP2P: Emergency Call Handling over P2P Overlays
ECHoP2P: Emergency CallHandling over P2P OverlaysK.Graffi, A.Kovacevic, et al. P2P-NVE ‘07
11/2007
Connect me to an emergency station!
Emergency Call Handling
QoS Provisioning
05/2007
P2P Forschung - Übersicht und Herausforderungen, K.Graffi, Quap2P, it-Informa-tion Technology, 2007
Peer-to-Peer Research – Overview and Challenges
Was ist Peer-to-Peer? Gemeinschaftsarbeit von QuaP2P mit einem Überblick zu Peer-to-Peer und seinen Herausforder-ungen.
Network Wrapper
Overlay Layer
User
UDP TCP
Online-timeModel
Behavior
PackageLoss
DelayModel
Bandwidth
Kademlia
Sim
ulatio
n E
ng
ine
Application LayerDistribution
Strategy
Chord
ReplicationStrategy
KOM – Multimedia Communications Lab 7
Efficiency Management in P2P Networks
05/2008
Load Balancing for Multimedia Streaming in Heterogeneous Peer-to-Peer SystemsK.Graffi,…,A.Kovacevic, et al. ACM NOSSDAV ‘08
Load Balancing for Multimedia Streaming in Heterogeneous Peer-to-Peer Systems
Wie kann die Ressourc- en und Aufgabenver-teilung in P2P Netzen fair gestaltet werden?
Wie kann man Lasten-verteilung und Hetero-genität unterstützen?
Eine allgemeines Over-Overlay für Attribute-based Peer Search. Wie gestaltet man skalierende Informationsarchitekturen?
Information and Efficiency Management: Attribute-based Peer Search for Structure P2P Overlays
Towards and Information and Efficiency Management Arch… K. Graffi et al. Technical Report KOM-2008-2, 2008
4/2008
7.31.10.25
peer-to-peer.info12.5.7.31
95.7.6.10
86.8.10.18
graffi.org-lifesocial.org 130.83.139.139
KBR Layer
UnifiedID Space
with Overlay specific IDs
KBR using Unified ID Space
CoordinatorSupport Peer
Peer
From Cells to Organisms: Long-Term Guarantees on Service Provisioning in P2P NetworksK. Graffi, A.Kovacevic, et al. ACM SIGAPP NOTERE ’08
From Cells to Orga-nisms: Long-term Guarantees on Service Provisioning in Peer-to-Peer Networks
Wie kann man die Optimierung der Res- sourcenverteilung im Overlay als Aufgabe des Overlays gestalten?
06/2008
Information and Efficiency Management in the case of FreePastry
Information and Efficiency Management in the Case of FreePastry, K.Graffi et al., to be published ‘08
Ein Statistik-Modul (für Overlay Service Provider) und Attribute-based Peer Search.
3/2008
7.31.10.25
peer-to-peer.info12.5.7.31
95.7.6.10
86.8.10.18
graffi.org-lifesocial.org 130.83.139.139
KOM – Multimedia Communications Lab 8
Towards QoS & Emergency Call Handling
Connect me to an emergency
station!
Emergency Call Handling
QoS Provisioning
2
KOM – Multimedia Communications Lab 9
Serious Future Peer-to-Peer Applications
Application Areas To exploit self-organization abilities of P2P
Catastrophe scenarios require robust mechanisms E.g. coping with churn
Example: Emergency Call Handling Hard QoS requirements Peer-to-peer mechanisms provide
failure-tolerance and Quality of Service
See how the efficiency can be improved in this example Learn from it
KOM – Multimedia Communications Lab 10
Serious Application: Emergency Call Handling
Emergency Call Handling is not supported in VoIP (Skype) 2009: mandatory for VoIP providers P2P fits: all-IP, scalable,
but Quality of Service?
Requirements1. Location critical service:
Find closest/responsible Emergency Station
2. Quality of Service for P2P flows needed QoS policy: low delay, low loss
contact Emergency Station as soon as possible without message loss
Goal: How to solve problem locally ? OR
do we need system wide management? Alabama Emergency Zones
Snapshot of the simulated scenarios
Source: US census
Source: NENA
Paper at: K. Graffi et al., “ECHoP2P…”, Int. Workshop on P2P-NVE, Nov. 2007
Population density in Alabama
2.1
KOM – Multimedia Communications Lab 11
Our Approach for P2P-based Emergency Call Handling
Challenge 1: Location-based search requirements
Approach: Globase.KOM - Geographical LOcation BAsed SEarch Engineered for requirements of location based services A logical neighbor is a geographical neighbor (like in CAN) Tree structure enables search/lookup in O(log N)
Extended with following search mechanisms: Closest peer (Emergency Station) Peer fulfilling a specific criteria (responsibility)
A
B
C
D
E
G
H
I
JK
F
A
B C D E
F G H I JK Search Query
Paper at: A. Kovacevic et al., “Location Awareness…”, Special Issue of the Proc. of the IEEE on Adv. In Distr. Multim. Comm., Jan. 2008
KOM – Multimedia Communications Lab 12
Towards QoS for Overlay Flows
Types of flows in P2P systems
Layer 4 traffic flows: Underlay perspective Out of scope
Direct P2P communication: File transfer, application traffic, … Few, but large data streams (elephants) (Often) with low priority for the system
Overlay flows Multi-hop: maintenance, user queries… Many, small messages (mice) Varying relevance for system
Provide QoS to overlay flow according to its relevance
2.2
P2P Overlay
2
^Peer
IN OUT
10100110110001000110
1
KOM – Multimedia Communications Lab 13
Scheduling and Active Queue Management
Learn from network layer Scheduling
Reorder tasks/messages in queue Reference: First-in-First-Out
Active Queue Management Drop msg. upon congestion of queue Reference: Drop Tail
Research challenge: how to apply on overlay layer?
1. Message SchedulingBefore:
After:
2. Queue ManagementBefore:
After:Queue Limit
MessageSched.+AQM
See: K. Graffi et al., “Taxonomy on Scheduling/AQM Strategies…” Tech. Report, KOM-TR-2007-1,2, TU Darmstadt
KOM – Multimedia Communications Lab 14
Analyzing Overlay Flows in Kademlia
Simulation of Kademlia
Setup: PeerfactSim.KOM 10,000 peers, all join,
random store/lookup
Metrics: # of contacts per peer # of msg per contact
Observation Huge number of contacts with few messages each Overlay “flows” and traditional flows differ significantly
Conclusion Stateless scheduler needed Quality of service information are stored in messages
Paper at: K. Graffi et al., “Overlay Bandwidth Management …” in Proc. of IEEE Loc. Comp. Networks, Oct. 2007
KOM – Multimedia Communications Lab 15
Overlay Bandwidth Management
Novel substrate “Network Wrapper” Between overlay and transport layer:
Queues messages Applies Scheduling and AQM solution: HiPNOS.KOM
HiPNOS.KOM: Highest Priority First, No Starvation Introduce message priorities for Loss and Delay AQM: at congestion, drop message with lowest loss-prio. Scheduling: at free bandwidth, send message with highest delay-prio. Avoid starvation: Periodically increase delay-prio. of queued messages
Properties of HiPNOS.KOM Focus on QoS for overlay flows Easy to apply on existing overlays
Paper at: K. Graffi et al., “Overlay Bandwidth Management …” in Proc. of IEEE Loc. Comp. Networks, Oct. 2007
2.3
KOM – Multimedia Communications Lab 16
Overlay Bandwidth Management Results
Observation: Proportional relations:
Delay to delay-priority Loss to loss-priority
Results: HiPNOS.KOM provides QoS
Regarding delay and loss According to chosen priorities
Paper at: K. Graffi et al., “Overlay Bandwidth Management …” in Proc. of IEEE Loc. Comp. Networks, Oct. 2007
KOM – Multimedia Communications Lab 17
Lessons Learned for QoS in P2P Systems
Results for Scheduling and AQM Delay and delay-priority, loss and loss-priority are proportional Emergency Calls have always highest priority All other messages have lower priority Quality of service can be provided
Lessons learned:IF … known:
Optimization criteria Set of all alternatives
THEN mechanisms for Quality of Service are easy to adopt
Question from beginning:How to solve problem locally ? OR do we need system wide management?
Solve it system wide requires information Necessary for efficient decisions in distributed systems Often missing in Peer-to-Peer systems
3
KOM – Multimedia Communications Lab 18
Towards a Kind of „Efficiency Management”
Peers
αβλ
μ
Parameters
f(α, β)=…=xg(λ, μ)=…=yh(α, λ)=…=z
ModelsInterpreted state
Architecture
Choose priorities
Efficiency Management Architecture
Analysis, Modeling and Interpretation
Using Info. to Gain
Efficiency
4
KOM – Multimedia Communications Lab 19
Current State of Efficiency Management
Each functional layerhas its own information/analysis architecture
To gather, analyze layer specific information Examples
BitTorrent: for tit-for-tat peer selection Replication: which data, on which peers
Give me 10 peers with specific availability
Skype: for Superpeer selection Find one peer with specific bandwidth and online time
Network wrapper: underlay awareness Identify most sent message type
Security: Find 5 trustworthy peers
PASTPAST
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
4.1
KOM – Multimedia Communications Lab 20
Desired State of Efficiency Management
Each functional layerhas its own information/analysis architecture
To gather, analyze layer specific information
Common basic functionality can be “abstracted”, i.e. “extracted” To gather layer specific information
Attribute-based peer search To analyze information, (derive optimization goals) To apply results for better decisions
Separate Information/Efficiency Management Layer for this task
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
KOM – Multimedia Communications Lab 21
Our Vision of an Efficiency Management Lifecycle
Efficiency Management System:
To engineer & to build architecture To gather information from peers To retrieve system parameters
To analyze component To use system model To prepare statistics To interpret system state
With application component To Provide QoS
Based on above issues
Peersα
βλμ
Parameters
f(α, β)=…=xg(λ, μ)=…=yh(α, λ)=…=z
ModelsInterpreted state
Architecture
Choose priorities
Efficiency Management Architecture
Analysis, Modeling and Interpretation
Using Info. to Gain
Efficiency
Paper at: K. Graffi et al., “From Cells to Organisms …” in Proc. of ACM SIGMM NOTERE, June 2008
4.2
KOM – Multimedia Communications Lab 22
Efficiency Management Module
Offers:
Statistics / Monitoring Number of peers, avg. online times Aggregable information
Attribute-based peer search Give me:
3 peers with Storage space > 20Mb Bandwidth > 100kb/s
Uses: Can be used on all structured P2P overlay Requires Key-based Routing (KBR)
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
4.3
EfficiencyManagement
Module
KBR
KOM – Multimedia Communications Lab 23
First Version of the Efficiency Mgmt. Module
First step: specific solution for FreePastry For Monitoring / Statistics Attribute-based peer search
Built a module for FreePastry Using Scribe to spread data Pub/sub principle
Module can be used!
Valuable component forPIPE Architecture
7.31.10.25
peer-to-peer.info
12.5.7.31
95.7.6.10
86.8.10.18
graffi.org-lifesocial.org 130.83.139.139
Underlay:The Internet
StructuredOverlay: DHT
EfficiencyManagement
System
Monitoring, Statistics
KOM – Multimedia Communications Lab 24
Current Research Focus: Improved Architecture for Efficiency Management
For all structured P2P overlays Covered by common API KBR Usable by all functional layers in a P2P system
Function: Overlay-independence Monitoring capabilities Attribute-based peer search
Features: Scalability (# of info and peers) Robustness (double-churn) Load-balancing Considering peer heterogeneity Adapting to usage patterns Low overhead
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
7.31.10.25
peer-to-peer.info
12.5.7.31
95.7.6.10
86.8.10.18
graffi.org-lifesocial.org 130.83.139.139
StructuredOverlay: DHT
Underlay:The Internet
KBR Layer
UnifiedID Space
with Overlay specific IDs
KBR using Unified ID Space
EfficiencyManagement
System
CoordinatorSupport Peer
Peer
4.4
KOM – Multimedia Communications Lab 25
Efficiency Management Architecture
Efficiency Management Architecture Built on underlying structured overlay Communicates via common API
Route to PeerID Just an add-on, easy to deploy
Principle Each node publishes information
updates in the architecture (bottom up) Update-tree is established Each node knows where to send updates to Queries are processed bottom up
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
UnifiedID Space
KBR using Unified ID Space
CoordinatorSupport Peer
Peer
KOM – Multimedia Communications Lab 26
Efficiency Management Architecture Details
Over-overlay: ID space separated in intervals (domains) Peer responsible for a specific ID (e.g. middle) is responsible for ID domain Peers in the domain send updates to this Coordinator Updates propagated upwards the tree
Supporting Peers for Load Balancing Coordinator may chose Supporting Peers Good peers chosen by 50/50 ratio
Pick e.g. 20 best peers in the domain Best 10 peers in domain advertised one level up Second best 10 peers can be used as support
Workload can be delegated to supporting peers Tree depth / peer load adjustable
IDSpace
Peers
Coordinator Supporting Peer
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
KOM – Multimedia Communications Lab 27
Queries in the Efficiency Management System
Query Type: Give me M peers Fulfilling specific requirements on
Bandwidth, storage space, computational capabilities, Online time, peer load, reputation … (wide set of requirements definable)
Query processing First sent to coordinator of lowest domain Query traverses bottom-up, until M matching peers found Result is sent then to requesting peer Tradeoff:
Upper peers in tree know more Load should be kept on lower levels of the tree
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
KOM – Multimedia Communications Lab 28
Structure of the Efficiency Management Arch.
Query Performance: O(log N) hopsScalability:
Tree-structure of coordinators form information architecture Supporting peers: Strong peers can take the load
Robustness: No additional maintenance needed (done by structured overlay) Any peer can fail, no unwanted effects
See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt
KOM – Multimedia Communications Lab 29
Example Application: Replication Layer
Content storage in P2P systems Churn is a problem
Data may get lost
Replication is a solution
Challenges Which files to replicate?
Most requested, rarest? At which peers?
Most reliable? Highest bandwidth? How many replicas?
Depends on requirements on availability. By which peers?
Efficiency Management System allows for answers
KOM – Multimedia Communications Lab 30
Lessons Learned for Efficiency Management in P2P Systems
Information Management is just ONE part of the Efficiency Management Lifecycle
Next steps: Publish Technical Report on distributed Efficiency Management Architecture on
IEEE P2P 2008 Fine tune on some aspects / limitations
Long-term vision: P2P network regulates itself
According to QoS constraints towards efficiency From self-organization of the peers to self-consciousness of the system* Requires P2P modeling, data mining, distributed decision making, …
Upcoming Applications: P2P-based Grid: Share resources, negotiate service in return with the system* Modularized, layer-interactive, complex applications
* Paper at: K. Graffi et al., “From Cells to Organisms …” in Proc. of ACM SIGMM NOTERE, June 2008
5
KOM – Multimedia Communications Lab 31
Questions?
Kalman Graffi
Peer-to-Peer Research Group
Dept. of Electrical Engineering and Information TechnologyMultimedia Communications Lab · KOM
Merckstr. 25 · 64283 Darmstadt · GermanyPhone(+49) 6151 – 16 49 59Fax (+49) 6151 – 16 61 [email protected]
Further information: http://www.KOM.tu-darmstadt.de/
Publications: http://www.KOM.tu-darmstadt.de/Research/Publications/publications.html