tnc 2007 bandwidth-on-demand to reach the optimal throughput of media brecht vermeulen stijn...

TNC 2007

Bandwidth-on-demand to reach the optimal throughput of media

Brecht Vermeulen

Stijn Eeckhaut, Stijn De Smet, Bruno Volckaert, Joachim Vermeir, Filip De Turck, Piet Demeester (Ghent University – IBBT)Ibrahim Habib, Zhaoming Li (City University of New York )

With acknowledgment to IBBT FIPA & GEISHA project, VRT, IBM & University of Antwerp

Brecht VermeulenTNC 2007

p. 2

Broadcaster workflow

Archiving

Media production

Raw videomaterial

ingest

ingest

Playout


p. 3

Tape based workflow

Digital tapes

Linear editing

Rough cut

Final editing

Voice over

Playout

Non-Linear editing

Local conversion to file


p. 4

File based workflow

Central storage

Playout

NLE clients

Archive

Digital files

Digital tape from camera

or memory device

Windows or Apple


p. 5

Files: so ?

SD

DV25 DNxHD 2.2 HDCAM HDCAM-SR Proxy

Mb/s 28,95 220 144 440 1,51h video (in GB) 13,0 99,0 64,8 198,0 0,7Transfer 1h video in seconds over Gb/s 111 843 551 1685 6

HD

Research issues: Optimal large file transfer: network & server

performance Offsite transcoding/rendering farms (& editing &

voice-over & subtitles, ...) File-based archiving Disaster recovery


p. 6

Contents

Introduction Optimising server networking

TCP/IP offloading vs. CPU based FTP vs. NFS vs. CIFS

Network based vision Ongoing research Conclusion


p. 7

TCP tuning options

Adapt kernel TCP parameters (free) Bigger receive window: more data in-transit Important if bandwidth*delay is high Linux: rmem,wmem,tcp_rmem, tcp_wmem,

mem,netdev_max_backlog Windows registry: Tcp1323Opts=3,

GlobalMaxTcpWindowSize,TcpWindowSize,AFD DefaultReceive(Send)window

e.g. Buffers and window on 4MB Jumbo frames ($)

MTU 9000 bytes, ..., 16000 bytes Not really a standard-> NICs, switches to be tested


p. 8

TCP offloading

TCP checksum & segmentation offload ($) Most modern good nics Works with standard kernel Warning: some cards say that they do offloading,

but it is done in the driver software Full TCP offload ($$)

Complete TCP/IP stack on the NIC (incl. retransmits, slow start...)

TCP setup/teardown still by host Webserver short connections vs. long transfers

Problems with e.g. Bonding Kernel patch needed (linux)


p. 9

TCP offloading

Normal NIC Offloading


p. 10

TCP offloading tests

Back-to-back tests between AMD dual Opteron systems (Opteron 246 @ 2GHz)

Intel PRO/1000 NIC (4 x 1 Gbps) TCP checksum & segm offload

Chelsio T204 TOE (4 x 1 Gbps) full TCP offload (= TCP Offload Engine)

TCP throughput measured with Iperf Generates TCP streams on different interfaces Transfers are memory-to-memory

Limitations PCI-X bus: 64 bit @ 133 MHz ~ 1GB/s

PCI-X is a half-duplex bus, PCI Express is a full-duplex point-to-point connection

Maximal (unidir) TCP efficiency: 94.1% 941 Mbps per link 99% for 9000 byte MTU


p. 11

Chelsio TOE vs. Intel Pro 1000 (MTU 1500) 4 links unidir: 3.7 Gb/s vs. Intel NIC 2.7 Gb/s 4 links bidir: 7 Gb/s vs. Intel NIC 3.2 Gb/s

Jumbo frames on Intel: throughput +, CPU -

TCP offloading results

Inte

l

Inte

lC

hels

io

Chels

io

MTU 1500

MTU 9000

4 Gb/s

8 Gb/s

50%

100%


p. 12

Protocol comparison setup

Transfers between storage and memory GPFS fibre channel storage used

360MB/s write, 690MB/s read from one server 2.88Gb/s write, 5.52 Gb/s read


p. 13

Protocol comparison

FTP > NFS > CIFS for reads FTP > CIFS > NFS for writes FTP with chelsio close to GPFS performance


p. 14

CIFS (synchr.) vs. Latency: model


p. 15

Contents

Introduction Optimising server networking Network based vision

Broadcasters’ problems Media grid farms Archiving Disaster recovery

Ongoing research Conclusion


p. 16

Broadcasters’ problems

Typically broadcasters work together with production houses, remote studios, ...

Storage and computing is not core business of broadcasters -> outsource to datacenters ?

Networking seems THE solution, BUT... FTP > NFS > CIFS+delay issue: but remember

windows editing clients -> CIFS HDCAM-SR: 440Mb/s video codec Storage bandwidth: both for archiving (and retrieve

something from archive), disaster recovery Time-critical (journals)


p. 17

Mediagrid farms

Editing on standard definition, rendering on rendering farms on HD (editing effects, cuts, ...)

Problems: Standard grid infrastructure is more directed

towards computing intensive vs. storage/dataset intensive tasks

For broadcasters: guarantees are needed on bandwidth and computing availability

Bandwidth to the rendering farms should be high, but can be by reservation (e.g. for non-live productions).


p. 18

Archive: (S)ATA disk price evolution

19992000 2001

20022003

20042005

20062007

Price per GB

0,0

1,0

2,0

3,0

4,0

5,0

6,0

7,0

8,0

9,0

10,0

Source: own purchase prices 1999-2007

Max Capacity €/GB

SATA 2000 75 7,4SCSI 2000 73 17,4SATA 2007 750 0,3SCSI 2007 300 2,6


p. 19

Archive

Cheaper disks and tape library systems: Online/nearline file-based archiving

Storage management


p. 20

Archive providers


p. 21

Archive needs

Only high bandwidth when retrieving content

Uploading of content may be slower Some content may be duplicated to two

sites, other to only one site Reservations for guaranteed bandwidth ?


p. 22

Disaster recovery

Central storage

Playout

NLE clients

Archive

Central storage is large Production is done on this Total restore = > 24 hours Solution:

Working on remote copy ? Networking/server performance ? Client CIFS ? Bandwidth guarantees on-demand for this ?


p. 23

Contents

Introduction Optimising server networking Network based vision Ongoing research

VPN between Gent and New York

Conclusion


p. 24

VPN: Gent – New York

For now: only 100Mb/s

Cuny (NY)

HOPINew YorkNew York

BELNETAmsterdamIBBT

Ingress LSP in NY Egress LSP in NL

HOPI-Ghent_Nyc_Ams

Transit LSP in BELNET’s GhentEgress LSP in GEANT2 NL

HOPI-Ghent_Ghent_AmsAmsterdam

New York

Ghent

Ingress LSP in GEANT2 NL

HOPI-Ghent_Ams_Ghent

LDP session to exchange vlan 4003

Ingress LSP in NLEgress LSP in NY

HOPI-Ghent_Ams_Nyc

Figure provided by Dante


p. 25

CVLSR

CHEETAH Virtual Label Switching Router Linux control PC with GMPLS engine Ethernet switch with

bandwidth reservations

Due to delay in setup and performance issues, research is still ongoing

One possible way


p. 26

Conclusions

Demand from broadcasters: Bandwidth and remote storage/computing Large files

Research: Optimal configuration and tuning of protocol

parameters and servers to use the bandwidth Is bandwidth reservation a solution for network

distribution of this functionality ?

tnc 2007 bandwidth-on-demand to reach the optimal throughput of media brecht vermeulen stijn...

Documents

tcp setupteardown

gbpsfull tcp offload

intel nic

iperfgenerates tcp streams

readsftp cifs nfs

memorylimitationspcix

ftp nfs cifs delay issue

ghzintel pro1000 nic