AdvantagesofTCPpacingusingFQ

BrianTierney,ESnetbl>erney@es.nethBp://fasterdata.es.net

TCPop'on:FairQueuingScheduler(FQ)AvailableinLinuxkernel3.11(releasedlate2013)orhigher–  availableinFedora20,Debian8,andUbuntu13.10–  Backportedto3.10.0-327kernelinv7.2CentOS/RHEL(Dec2015)

ToenableFairQueuing(whichisoffbydefault),do:–  tcqdiscadddev$ETHrootfqOraddthisto/etc/sysctl.conf:net.core.default_qdisc=fq

Tobothpaceandshapethetraffic:–  tcqdiscadddev$ETHrootfqmaxrateNgbit•  Canreliablypaceuptoamaxrateof32Gbpsonafastprocessors

Canalsodoapplica>onpacingusinga‘setsockopt(SO_MAX_PACING_RATE)’systemcall–  iperf3supportsthisviathe“—bandwidth’flag

AdvantagesofPacing

•  ThefollowingplotsshowaclearbenefitfrompacingTCP–  ProperpacingofflowscancompletelyeliminateTCP‘sawtooth’

•  Theadvantageisevengreaterwithparallelflows•  WerecommendallDataTransferNodes(DTNs)useFQ-basedpacing

•  Pacingalsohelpswitheliminatepacketlossduetounder-bufferednetworkhardware,andunder-poweredsecuritydevices.

9/30/163

9/30/164

NewYorktoTexas:WithPacing

Moreexamplesofpacinghelping

ParallelStreamTest1LeHside:sumof4streamsRightside:tputofeachstreamStreamsappeartobemuchbeLerbalancedwithFQ,pacingto2.4performedbest

ParallelStreamTest2LeHside:sumof4streamsRightside:tputofeachstreamStreamsappeartobemuchbeLerbalancedwithFQ

100GHost,ParallelStreams:nopacingvs20Gpacing

9/30/168

WealsoseeconsistentlossontheLANwith4streams,nopacingPacketlossduetosmallbuffersinDellZ9100switch?

100GHostto10GHost

9/30/169

FastHosttoSlowhost

9/30/1610

ThroBledthereceivehostusing‘cpupower’command:/bin/cpupower -c all frequency-set -f 1.2GHz

FQPacketsaremuchmoreevenlyspacedtcptrace/xplotoutput:FQonleH,StandardTCPonright

11

Runyourowntests

•  FindaremoteperfSONARhostonapathofinterest–  Mostofthe2000+worldwideperfSONARhostswillaccepttests•  See:hBp://stats.es.net/ServicesDirectory/

•  Runsometests–  bwctl-chostname-t60--parsable>results.json

•  ConvertJSONtognuplotformat:–  hBps://github.com/esnet/iperf/tree/master/contrib