1

Support for Video Streaming over CERNET

Congxiao BaoTsinghua University

2008-7-12

Outline

• Background• Problem & challenges• Design goals• Solutions• Cases study• Conclusion

2

Background-Network deployment &Video streaming

activities

Network deployment

http://lake.net.edu.cn:8033/map-history.html

3

CERNET/CERNET2•IPv4 national backbone•IPv4 unicast and multicast•38 GigaPops in 36 cities •2,200 universities connected•Self-funded operation•High utilization (70%+)

•Pure IPv6 National Backbone•IPv6unicast and multicast•25 GigaPops in 20 cities•120 universities connected•Free usage•Low utilization (10%)

Video Streaming Trials over our networks

• Distance-learning in – Language– Technology– Music– Art performance– Medicine

• Conference attending• Project seminar• Regular NOC working meeting• etc

4

SIGCOMM2007

Tsinghua-KISTI uncompressed HD transmission

APAN telemedicine

5

Video Streaming System• LD video

– compressed 1M H.323/Accessgrid/Isabel• SD video

– Compressed 30 Mbps DVTS– Uncompressed 220 Mbps MCGILL

• HD Video– Compressed 22 Mbps VLC– Uncompressed 800 Mbps –1.2 Gbps KISTI-code

• 3D video– Compressed 60Mbps 2*DVTS

• Huge data transmission– EUChinagrid-1– MaZeP2P 500Mbps

Key technologies

• Network support– Unicast/Multicast– IPv4/IPv6

• protocol feature– HP UDP real-time video applications– HP TCP applications

• Interactive way– Point to point transmission– Multi-points to multi-points transmission

6

Remarks• HP video applications are required over

distributed backbones from users• but……from our experiences

– Multicast based applications is very important among academic communities, but enabling multicast is a hard work

– End to end performance guarantee for HP UDP or TCP applications is impossible without advanced measurement and bandwidth resources adjustment

• Analysis of the fundamental problems and try to find solutions

Problems & challenges-multicast network and distributed UDP/TCP

applications

7

From our previous presentations

Multicast network monitoring

• Enabling multicast on the backbone is a tough work

• The router supported commands are not enough for the multicast debugging

• End to end (user-controlled) monitoring tools are mandatory

8

HP-UDP applications

• Take DVTS as an example – Hardware is based on consumer products– Software is free and open source– Useful for mission-critical applications– Special demand for network support

• 30Mbps/single video stream

DVTS is a good start point!!!

Setup a DVTS demonot easy

1. Simplest case----point to point with intradomain routing

User B

UserA

DVTS unit

DVTS unitsending

receivingsending

receiving

9

Normal procedures for a DVTS setup

Step 1: Solicit an application Step2: Exchange contact info Step3: Mailing list setupStep4: Email bomb (equipment config/test schedule/…)Step5: 1st test - A to B is not ok, B to A is not ok

2nd test - maybe network /application system problemping/traceroute/ network adjusting

3nd test - network is ok. But still can not see or hear …software reinstallation /even change anther pc/camera checking/speak checking….

……10nd test (rehearsal) - Everything is ready, engineers are

exhausted!!! Step6: Formal event - let’s keep fingers cross!!!

Very difficult ……

2. More complex cases• Multi-points with intradomain routing• Point to point with interdomain routing• Multi-points with interdomain routing

Especially with asymmetrical routing….. for international collaboration

10

What is the problem and where are the bottlenecks

AS2AS2 AS3AS3

AS1AS1

access access

Bad cablingWrong speed

Wrong duplex mode

Bad cablingWrong speed

Wrong duplex mode

firewall firewallBandwidth bottleneck

Host parameters

Host parameters

Problem statement• Technique issues

– DVTS application behavior and network test results may have different meaning

• DVTS application: 30Mbps, UDP port 8000, etc• Network test: 1Kbps, ICMP, show AS-path

– The packets are traveling cross difference AS, the path is likely asymmetric, and the end2end path may not be the same as what the NOC sees.

• The end users do not know the network traffic and the NOCs do not know the end2end

– The firewalls and the QoS policies may deployed somewhere in the path

• Ratelimite match acl 140 in g0/0

11

Problem statement(cont.)

– The network behavior may change between the rehearsals

• Murphy’s law– It is not a easy job to setup DVTS system

• DV camera, 1394 cable, computer, clicking the menu

– The end users and the NOCs are usually speaking different languages

• ^_^

Problem statement (cont.)

• Non-technique issues--- heavy burden of negotiation– Language

• Natural language– Time differences (sleepless testing)– POC (right person to find right person)– Etc

• Multi-sites work together– Need enough patience for more than 5

sites to work together at the same time if each site could not fixed everything ASAP

12

Why HP TCP applications

• Reliable mass data transfer

33

TCP performance

use the Principle of Network Efficiency:– only inject more data into a loaded network

when you believe that the receiver has removed the same amount of data from the network

– TCP uses ACKs as the sender’s timer

ACK packets

Data packets

RS

13

36

TCP session behaviour

TCP Rate Control

0

1 0

2 0

3 0

4 0

5 0

6 0

7 0

0 5 1 0 15 20 25 30

TC

P W

indo

w S

ize

Network congestion level

Slow Start Behaviour Congestion Avoidance Behaviour

HP TCP application test

max__ _

_ _throughput

receive window sizeround trip time

=

max_,.

. / secthroughput bytes= =65 535

0 56117 027

max_,.

. / secthroughput bytes= =24 567

0 5643 886

Formula

SUN §WINDOWS

Theory

TCP transfer speed

14

Problem statement (cont.)

• Packet loss– Packet loss is the key factor for auto-

adjustment of TCP rate-control, it should be very very small

• Window size – If the packet loss is zero, but RTT is large for

a specific physical link, increase the window size of the operation system of the end system can improve the transmission throughput

Summary• More and more demonstration and trials on high-

performance applications– Application-specific stream or data– Consume large network bandwidth– Heavily rely on network support

• User controlled tools, work with NOC closely– Measurement and monitoring

• Network performance measurement tool– Unicast/multicast

• Application-specific measurement tools• Distributed measurement servers

– End to end performance guarantee• Inter-AS routing adjustment• Agreement on enabling application-related ACLs

15

Design Goals-Requirements and answers

Goals

• Use measurement tools to– Built a common language between the application

users/engineers and the network administrators• Separate the network problems from the end system

problems

– Help users to do self-service• Test without the help of the counterpart• Simulate the real DVTS behavior without using the real

DVTS system

– Find the network segments which cause the problem• Find the bottleneck of the network

16

Requirements and answers• Open and share the network and application test

statistics both for network administrators ,application system engineers and end-users– Similar 5+ elements for both simulated random data and video

• To do the test automatically with the measurement server at any time without partners– Measurement servers setup and running

• To do the trouble-shooting hop by hop for inter-AS collaboration– Distributed measurement servers in each AS management

How about setup a measurement servers at every AS and user’s subnet?

Solutions-Scalable Application-specific measurement

framework

17

Application-support Architecture

Network Infrastructure(Unicast/Multicast)

Open Network monitoring and management

Middleware(Directory services/dial plan/Security)

Application systemEngineering environment

Application protocols

Otherprotocols

Network protocols

ApplicationSpecialist

Software developer

Networkengineer

Humanprotocols

Application self-testing Tools

SASM building blocks• General client-server architecture

– Unidirectional and bidirectional• User-controlled measurement tools integrated

– Basic network performance monitoring– Application-specific measurement – Real application trial

• Distributed measurement servers– Sending or reflecting mode– Distributed server management

• User access interface– Easy to use & visualization

• web service– Prefix authentication & DDOS

18

User-controlled measurement tools• Basic network performance monitoring

– unicast network connectivity • TCP-ICMP-ping

– multicast network connectivity• Ssmping/asmping

• Application-specific measurement tools– Simulated DVTS streaming

• UDP-throghput-dvping– Simulated UHD streaming

• UDP-throghput-uhdvping– HP TCP throughput measurement

• iperf• Real-application reflecting

– DVTS– UHD– FTP

Focus…….

Operation modes of basic tools

19

The technical specification

tein2-hk2-dvts33 % dvping -hUsage:dvping [-h] [-p snd_port] [-rp rcv_port] [-t period] [-tt tt] [-bw bandwidth] host

tein2-hk2-dvts33 % dvmcast -hdvmcast [-h] [-t time_limit]

[-l log_file] [-lp log_period] [-ot open_th] [-ft fluc_th][-noloop] [-dvf dv_file] [-4|-6] [-rp rcv_port] [-asm grp] [-ssm src grp] [-sf src_flt] [-sff sf_file][-ech ech_port] [-d [dst_addr] [dst_port]] [-df dst_file]

Introduction & usage

20

Sending mode

Reflecting mode

21

Distributed measurement servers

• Server discovery alone the packet traveling path in each AS– AS path and DNS server

• New server registration – By manual or automatically

Sender

Receiver

display

Browser

DMServ

httpd

Sender

Receiver

display

Browser

DMServ

httpd

DMServ

httpd

DMServ

httpd

DMServ

httpd

User AUser B

AS1 AS2 AS3

22

Find the measurement servers

SASM Framework

23

SASM Framework

Web interface

http://202.112.35.200:8056/ddvts.htmlUser: tein2, passwd: tein2

24

TEIN2 DM servers distribution

The TCP measurement Tool

25

The testing steps• Default test

– The default TCP transfer speed for the e2e peer• Window size test

– Check the client window• Parallel session test

– Check the server window– Physical speed limit

• Comparison with ping and UDP (dvping) tests– Check packet loss (ping and dvping)– Check RTT (ping and dvping)– Check throughput (dvping)

A Document

http://noc.tein2.net:8036/support-for-hp/hp-mc-20070507.html

Login: tein2 passwd: tein2

26

Summary of the TEIN2 NOC measurement servers

server clinet

ICMPICMP

U/UDPU/UDPM/UDP

U/UDP

U/UDP

U/TCP

U/TCP

U/UDP

ping

ssmping

dvping

iperf –c

smjoin

iperf –s

smclock

dvmcast

ssmpingd

Socket stack

U – UnicastM - Multicast

The specification is important

• Application Area– Medicine

• Application system– DVTS

• Network specification– UDP/p=8000/bw=30Mbps/RTT<150ms/ Erlang– IP/prefix/ASN

27

Cases study-dvping/dvmcast tools usage

Tsinghua-kISTI UHD transmission trialCERNET2 DVTS deployment

TEIN2 multicast support for canalAVIST

From Arlington to TEIN2 Beijing server

using WALN Not good

using RJ45 Good

28

Tsinghua-kISTI UHD transmission trial

• Basic config– Uhd packet format

• FRC4751

– Changeable Packet size• From 1472Bytes

– Same port • 5004

– Linux configurations• memory:2G• CPU:3.2G(2.4G)*4

Test topology (IPv4/IPv6)

s

s

BJ(6IX-CERNET2)

HK

KOREANet2

s

1G

1G

1G1G

1G

1G1G

10G

210.25.189.1262001:252:0:2::1001

210.25.189.1252001:252:0:2::1002

MSSender/receiver

134.75.21.902001:320:10:4000:/64

MSSender/receiver

Video senderVideo receiver

134.75.21.802001:320:10:4000::1001

134.75.21.842001:320:10:4000::1002

29

Test results-IPv4UHD-specific measurement (1)

• Sending mode( from BJ-KISTI)[cernet@localhost bin]$ ./hdvmcast -4 -l<source_log ip="210.25.189.126" port="32771" start="2007/10/30 16:20:44" stop="2007/10/30 16:20:54" packets="764166" lost="0%" bytes="1124852352" avg_bw="900Mbps"><bw_change time="2007/10/30 16:20:45" bandwidth="897Mbps"/>

<bw_change time="2007/10/30 16:20:46" bandwidth="905Mbps"/><bw_change time="2007/10/30 16:20:47" bandwidth="900Mbps"/><bw_change time="2007/10/30 16:20:48" bandwidth="899Mbps"/><bw_change time="2007/10/30 16:20:49" bandwidth="901Mbps"/><bw_change time="2007/10/30 16:20:50" bandwidth="896Mbps"/>

<bw_change time="2007/10/30 16:20:51" bandwidth="905Mbps"/><bw_change time="2007/10/30 16:20:52" bandwidth="899Mbps"/>

<bw_change time="2007/10/30 16:20:53" bandwidth="898Mbps"/><bw_change time="2007/10/30 16:20:54" bandwidth="905Mbps"/>

</source_log>• Sending mode( from KISTI-BJ)

bao@debian:~/hdvts/bin$ ./hdvmcast -l<source_log ip="134.75.21.90" port="32773" start="2007/10/30 15:33:39" stop="2007/10/30 15:33:49"

packets="764008" lost="1.08%" bytes="1112475776" avg_bw="889Mbps"><bw_change time="2007/10/30 15:33:40" bandwidth="890Mbps"/><bw_change time="2007/10/30 15:33:41" bandwidth="893Mbps"/><bw_change time="2007/10/30 15:33:42" bandwidth="889Mbps"/><bw_change time="2007/10/30 15:33:43" bandwidth="890Mbps"/><bw_change time="2007/10/30 15:33:44" bandwidth="892Mbps"/><bw_change time="2007/10/30 15:33:45" bandwidth="890Mbps"/><bw_change time="2007/10/30 15:33:46" bandwidth="894Mbps"/><bw_change time="2007/10/30 15:33:47" bandwidth="894Mbps"/><bw_change time="2007/10/30 15:33:48" bandwidth="891Mbps"/><bw_change time="2007/10/30 15:33:49" bandwidth="894Mbps"/>

</source_log>

Test results-IPv4UHD-specific measurement (2)

• Reflecting mode

bao@debian:~/hdvts/bin$ ./hdvping 134.75.21.90 -bw 900M -t 300ping 134.75.21.90 with uncompressed HDV data:send port:5004recv port:5004packet size:1472period :300 ssnd_pkt rcv_pkt snd_rate rcv_rate loss rtt22906891 21542645 899Mbps 846Mbps 5.96% 4856ms

Due to CPU usage problem………………

30

Test results-IPv4UHD-specific measurement (3)

Test results-real video transmission (IPv4)

BJ

HK

KOREANer2

10G1G

receiver

sender 1G1G sender

receiver

Tsinghua University

6IX/CERNET2

900Mbps

900Mbps

KISTI

31

Test results-IPv6(1)• Sending mode (from BJ-KISTI)<source_log ip="2001:252:0:2::1001" port="32770"

start="2007/10/29 18:37:23" stop="2007/10/29 18:37:33" packets="812201" lost="47%" bytes="624801408" avg_bw="499Mbps">

• Sending mode (from KISTI-BJ) <source_log ip="2001:320:10:4000:230:48ff:fe90:564f"

port="32771" start="2007/10/28 17:42:45" stop="2007/10/28 17:42:55" packets="808379" lost="47.2%" bytes="619944468" avg_bw="496Mbps">

Test results-IPv6(2)video-reflecting-measurement

Dear Zhang Xuan,

KREONET NOC found a number of bottleneck points in domestic IPv6 network.I guess it is coupled with using dual-stack ipv4 & ipv6 in KR side.

Currently, ipv6 performance is confined with 450Mbps so i'm looking for the other ways tocontinue our tests for ipv6.

Best Regards,

Jinyong Jo from KISTI.

210.25.189.125（BJ receiver）210.25.189.123（BJ sender）134.75.21.90（korea echo）

32

CERNET2 DVTS deployment

• Connection topology• Node configuration• Data flow• Reflector development• Traffic monitoring• Website support• DVTS-specific measurement tools

• 25 DVTS nodes (diameter is about 1000+ kms)

• Bidirectional transmission

• 750MbpsDVTS

DVTS

DVTS

DVTS

DVTS

DVTS DVTS

DVTS DVTS

THU

XA

WH

HF

SJTU

GZ

CD

SY

TJ

CHQ

XM

NJ

LZ

DL

HABCHC

JN

HZ

CHS

ZZ

PKUBUPT

BUAA

FDU

TJU

DVTS DVTS DVTS

DVTS

DVTS

DVTS

DVTSDVTS

DVTS

DVTS

DVTSDVTS

DVTS

DVTSDVTSDVTS

BJ

SH

CERNET2 DVTS Connection Topology

33

CERNET2 DVTS Node configuration

CERNET2CERNET2

DVTSsender

DVTS Testing server

R

SW

DVTSreceiver

Node configuration—Central Node (1)

R R RReflector

array

sender

switch GE

Multicast/unicast

T640

12 receivers 13 receiversbackup

CE

RN

ET

2

THU sending

Reflector array

Each reflector receives

multicast/unicast stream from sender and converts to

12(13)unicast duplicated copies

to 25 different IPv6 hosts

distributed in 25 Pops

30Mx25=750Mbps

34

Equipment Configuration— Central Node (2)

bj wh gz nj sh

xa cd sy tjn lzh

chq chs zhz hef jnn

xmn hzh dln chc hrb

pku bupt buaa fdu tju

PAD displayarray

PCcluster

switchGE

30Mx25=750Mbps

PC clusterEach PC receives a

DVTS stream from one of the

25 Pops, respectively T640

CERNET2

THU Receiving

SJTU XJTU …… JLU

CERNET2

switch switch

………

…

北大

吉大 网管

DV

CERNET2 national Center

THU

R R RGE GE

CERNET2 25 DVTS nodes

DVTSsending PKU …… JLU

CERNET2

switch switch

………

…

PKU

JLU NMS

DV

CERNET2 national center

THU

R R RGE GE

DVTSsending

DVTSreceiving

DVTSreceiving

SJTU

Data Flow

35

Video Wall

Reflector Development

• It runs on Linux OS and developed using g++• It supports IPv4 and IPv6• It supports Unicast and multicast (SSM and

ASM)• It can reflects 20+ streams with gigabit

network card• It will be released as Open source soon

source

reflector

reflector

reflector

SSSM multicast

unicast

36

Network Traffic Monitoring

CERNET2 DM servers distribution

37

Website Support

TEIN2 multicast support for CanalAVIST

• Support for distributed DVTS application• User-controlled monitoring tools &

distributed servers • All protocols work together• documentation

38

Conclusion

39

Conclusion

• High performance UDP (real-time video) enabling is not only a pure application test

• Involves network performance monitoring and application-specific measurement

• Distributed servers are important for user-controlled tools

• With the measurement results, users can enable a HP application sucessfully

Thank You!