Adaptable bandwidth planning for enhanced QoS support in user-centric

broadband architectures

Dr. Ilka Milouchewa (FHG) Dirk Hetzer (T-Systems, M&B)

2

Bandwidth on Demand Planning.Topics

Background Bandwidth planning based on reinforcement learning Operations research for bandwidth scheduling Combining reinforcement learning with scheduling

Q-learning Informed learning Relational learning

QORE system for adaptable bandwidth planning Integration of bandwidth planning

in research projects and Telecom networks

3

Bandwidth on Demand Planning.Background (1)

need for planning of resources in new on-demand services (rich-media, IPTV, VoD, gaming) on all IP core, replacing ISDN, ATM etc.

high bandwidth demands, efficiency achievable by bandwidth planning and resource reservation

Scenario: Planning based resource reservation in advance at aggregation points (DSLAM, BB-PoP)

4

Bandwidth on Demand Planning.Background (2)

Need for capacity planning based on ADVANCE resource reservations for different kinds of QoS based applications and best effort traffic

Total resources (bandwidth) are always restricted, therefore advance reservation could be used for planning and enhanced utilization

Learning of performance to predict resource requirements Optimized resource allocations

adapting advance resource requests to predicted requests Find compromise between different resource requests

and enhance QoS for all kinds of applications (traffic)

Network accesslink

BroadbandInfrastructureInternet traffic

TrafficSource / Sink

Network accesslink

Network accesslink

Flexible connectivityScare resources

Advance resource reservation requests

Planning of resource allocation to applications

Immediate resource reservation requests

Best effort traffic

Environment state

Feedback

Bandwidth broker

Scenario: Planning is based on resource reservation “in advance” at access routersconsidering learning of performance data

5

Radio Access Others

Fixed Line Access

• Microwave • Satellite• Laser/

Optical• WiMax

• GSM• GPRS• EDGE• 3G W-CDMA• HSDPA• Broadband

wireless access

• PSTN/ISDN• FTTP• Cable• Broadband

wireline access

Communications Services evolution path:

•h

Telephony (F/M)

Television

Broadband Internet

TriplePlay

+ DSL-Internet

+ VoIP

+ IPTV

+ Smart Home

+ DSL-/ 3G-Internet+ VoIP+ TV+ VoD

+ Cable Internet+ Communication

UNIFIED USER EXPERIENCE

Voice Voice + Text Voice + Text + Picture + Sound

Voice + Text + Picture + Sound + Video

SMS MMS Rich Media

Convergence & Triple Play.Challanging the way of Service Creation. Content delivery on demand

Access evolution path: Triple Play evolution path:

6

Integrates telephony service with a user’s television

Supports the delivery of telephony services in conjunction with cable, DSL, and IP-based video services

PSTN, mobile,or VoIP phones

PSTN /VoIP

VideoDistribution

NetworkSIP Servlet Bill Smith

732-699-3232

• TV Calling Name• User directed routing• Click-to-dial • Participation TV - Voting - Gaming - Shopping• Messaging

- Picture Sharing- Multimedia Message Display- Content Services- Voice Mail Screening

User-centric approach for bandwidth planning.Scenarios and applications Planning for Triple Play.Example 1: TV-based Telecom Services.

7

User-centric approach for bandwidth planning.Scenarios and applications Planning for Triple Play.Example 2: TV-Mobile Convergent Participation Service (Blogging).

Participation TV as a Mobile Communication/ Broadcast Communication convergence case.

Participation TV denotes the integration of user feedback/interaction into TV-formats (such as game shows).

Traditionally very limited e.g. by calling into the game, sending SMS, …

Viewer Mobile Net MMS-Server Broadcast Station

httpSOAPMM7

MM7UCP

UMTSGSM

Release

Embeddingin MHP

MMS-Server

Net ProviderMMS Mobile

MHP

DVB-T Net

DVB-T Net

1 2 3

4

56

MultimediaContent

PublishingService

Viewer Mobile Net MMS-Server Broadcast Station

httpSOAPMM7

MM7UCP

UMTSGSM

Release

Embeddingin MHP

MMS-Server

Net ProviderMMS Mobile

MHP

DVB-T Net

DVB-T Net

1 2 3

4

56

MultimediaContent

PublishingService

MMS diaries (blogging) – uploads from personal pictures, texts via MMS

Virtual pubs/discos tour (spying on events)

Virtual classroom

Contests (e.g. Best amateur news report of the day/month)

MMS/iTV chat

Other services (personalized weather reports, group contests, alerts,etc.)

8

Bandwidth on Demand Planning.Bandwidth planning based on reinforcement learning (1)

Different “learning” approaches to optimize and plan bandwidth

Reinforcement learning benefits-> Rewards from interactions (action) with environment at each state (dynamical learning)-> Adaptive control considering states and actions (change of bandwidth scheduling dependent on the performance “feedback”)

.

Supervised learning

- correct action supplied to each state

- current actual actions have no effect on next state

- each interaction is independent, self contained

Reinforcement Learning

- agent never told which action is corr ect at given state

- agent told nothing about actions not selected

- current actions may affect next state

object is to maximize all future rewards

Stochastic automata

- a stochastic policy by associating a probability with each action, so that actions are chosen at random according to their probabilities - the policy does not take into consideration the current state of the system, when choosing an action.

9

Bandwidth on Demand Planning.Bandwidth planning based on reinforcement learning (2)

Allocate resource in advance for different traffic classes -> Multimedia Conference -> GRID Transfers

Adaptation of advance reservations to resource needs of traffic classes -> On-demand service

Learning performance (delay, throughput) and predict resource needs -> In case of HDTV

-Resource usage is derived in interaction with environment- Using reinforcements, prediction is done for the period T

- Optimal sharing of resources in advance for traffic classesconsidering reinforcementsfor each traffic

-Different strategies for advance allocation of resources for on-demand traffic considering predicted and reinforcements of actual resource usage of traffic classes

Best effort traffic

Resource reservation for advance and immediate resource reservation requests

Dynamic sharing of resources

Scenarios for usage of reinforcement learning for bandwidth planning

10

Bandwidth on Demand Planning.Usage of QORE for bandwidth planning (1)

Reservation in advance parameters depend on applications and users

GridOrder of allocations to applicationsDependency

Distributed multimedia, Grid, multimedia conferencing

Flexibility of usage:

- Fixed start,

- Interval based (flexible start, flexible end)

- Deadline

Time constraints

Tele-Radiology, VoIP, videoDependent on application users (cost wanted to pay)

Cost

GRID, multimedia, mission criticalDependent on application usageDuration

VoIP aggregate, content deliveryAdditional resource demand for traffic aggregation

Increment

Multimedia streaming, content delivery, file download

Dependent on application and possibilities for varying QoS levels

Alternative Bandwidth

Example ApplicationOptionsReservation in advance Parameter

11

Bandwidth on Demand Planning.Usage of QORE for bandwidth planning (2) Problem: Reinforcement learning

based on rewards from environment (delay of best effort traffic) -->finding optimal bandwidth allocation (optimal schedule)

which satisfies resource requirements in advance of QoS based applications and enhancing QoS (delay) of best effort traffic

Reinforcement learning problems for bandwidth planning-> User interface for bandwidth allocation in advance considering traffic classes-> Reinforcements: periodical performance measurements for best effort traffic-> Value function cumulative rewards evaluating the schedule for the planning period-> Interaction with Bandwidth Broker

Adaptable - bandwith planning agent interacting with environment - > optimal resource allocation policies for advance resource requests using value functions

Bandwidth broker

Monitoring data base

(QoS, best effort traffic )

Periodical Performance measurements

Network

Reward ( end-to-edn delay)

Reward ( end-to-edn delay)

12

Combining reinforcement learning methods and scheduling (1) Obtaining optimal bandwidth schedules for proactive and reactive planningusing combination of operation research & reinforcement learning methods

A(R ) Set of conflict free schedules based on resource restrictions R

- Apr (R ) Set of conflict free schedules for predicted patterns based on resource restrictions R

Simple Q-Learning Informed Q-Learning

Arel (R ) Set of conflict free schedules found in online operational networks based on resource restrictions R

Relational Q-Learning

Proactive planning

Reactive planning

Conflict-free schedule with minimum duration

Pattern based scheduling

Partial displacement scheduling

13

Combining reinforcement learning methods and scheduling (2).Simple Q-Learning - Approach

A model-free RL approach combining conflict-free scheduling with minimum duration

− Bandwidth schedules are characterized with Q-values based on rewards and value function (cumulative sum of rewards evaluating exceeded end-to-end delay threshold for planning periods)

Selection of bandwidth schedule for evaluation

− Random selection, e-greedy (the best Q-value) Q-value of bandwidth schedule updated every time

at the end of planning period, for instance a day

− Update based on Learning rate (Recency Weighted Average..)

Pure “trial” and “error”

− for practical usage not efficient, because predictions of resource usage of applications are not considered

14

Combining reinforcement learning methods and scheduling (3) .Simple Q-Learning - Example

Optimization Plan, BW Limit = 3500, Algorithm = Mixed BW

0

500

1000

1500

2000

2500

3000

3500

Time Slots [s]

Data Tele-Radiology (primary Appl.) M ilitary Application (primary Appl.) M ilitary Netmeeting (dependent Appl.)

Courier e-mail Service Shopping Mall Voice Tele-Radiology

T-DSL Connection

15

Combining reinforcement learning methods and scheduling (9).Relational Q-Learning - Modified schedule based on patterns

When pattern is detected, the planned allocation is rescheduledUsing rewards from end-to-end delay measurements for daily planning period

16

Bandwidth on Demand Planning.Usage of QORE for bandwidth planning (1)

Integration of patterns in scheduling algorithms based on reinforcement learning

Different kinds of patterns considered for bandwidth planning:

- Outliers

- Threshold overload patterns

- Patterns describing traffic, QoS parameter behavior, routing events and anomalies of network connections

Bandwidth planning focus

Patterns describing “normal”and “abnormal” behavior of multivariate time series data

End systems

Routers

Network connections

QoS parameter

Traffic volum

e Immediate resource

request

Routing event

s

Failure events

Patterns

- abstractions for structures describing behaviour of performance parameters for network connections

- extracted from monitoring data bases

17

Bandwidth on Demand Planning.Usage of QORE for bandwidth planning (2)

QORE components and interaction with monitoring and planning database

Bandwidth scheduling

Advance resource specifications

Application QoS Monitoring

Effective bandwidth estimation

Scenario manager for bandwidth allocation

Knowledge database for bandwidth scheduling - Application traffic definitions - QoS measurement scenario and results - Bandwidth estimation - Connection resource simulations - Scenario specification - Bandwidth schedules and results - Patterns

Simulator of resource constraints for connections

Pattern analyser and outlier detection

Constrained based scheduling algorithms

Visual data mining for bandwidth scheduling

User interface for bandwidth scheduling

QORE: automated tool for adaptable QoS-oriented proactive and reactive bandwidth planning

Components using common knowledge database for monitoring & planning

- User interface for „advance“ reservation      

- QoS parameter monitoring

- Scheduling algorithms

- Resource constraints simulator

- Effective bandwidth estimation

- Pattern analyser

- Visual data mining for bandwidth planning

18

Scenario for bandwidth planning in converged fixed / mobile environment Selection of optimal access network for content delivery

Access Networks Core network

QORE Advance reservation QORE Advance

reservation

Access router

Content Server

QoS Broker

QoS Broker

Measurement DB

19

Summary.Automated adaptable bandwidth planning in Internet based on reinforcement learning, QoS parameter patterns and scheduling heuristics

Novel management strategies for an automated proactive and reactive bandwidth planning in Internet using QoS monitoring data (QORE system)

Integration of data mining technologies using patterns for bandwidth planning

Bridging a gap in operations research techniques for bandwidth planning integrating dynamic learning of QoS parameters (pattern detection)

Integrated architecture based on reinforcement learning, operations research and data mining for bandwidth planning

20

Bandwidth on Demand Planning.Outlook & ongoing work in EU projects

NETQOS

Application specific bandwidth planning concepts

Policy based bandwidth management and planning

DAIDALOS

Enhanced QoS brokerage architectures based on resource reservation in advance

Advance resource reservation for mobile QoS based services

Practical integration of the adaptable bandwidth planning of QORE in network management systems