kek network qi fazhi 2004.8. kek sw l2/l3 switch for outside connections central l2/l3 switch a...
TRANSCRIPT
KEK Network
Qi Fazhi
2004.8
KEK SWL2/L3 Switch for
outside connections
Central L2/L3 Switch A
Netscreen
Firewall
Super Sinet Router
10GbE
2 x GbE
2 x GbE
IDSGbE
KEK GW2 x GbE
KEK SWL2/L3 Switch for
outside connections
Central L2/L3 Switch B
Netscreen
Firewall
2 x GbE
2 x GbE
2 x GbE
4 x GbE
NOODSwitch to other places
NOODSwitches to Buildings
4 x GbE
10GbE
Tskuba Center
10GbE
Other Univ. HEP Lab.
GbE
4 x GbE4 x GbE
up to: 14 Vlan, Based MAC Static & DHCP(for wireless)
IDSGbE
KEK’s firewall architecture
L2/L3 Switch for outside connections
Central L2/L3 Switch
Firewall
10GbE
10GbE/GbE
10GbE
2 x GbE
2 x GbE
Securitymonitor
High Throughput Access RouteHigh Throughput Access Route
Regular flowsRegular flowsIDSGbE
KEK
universities/institutesin Asia-Pacific
universities in Japan
SLAC,FNAL,BNL,
CERN,DESY,SDSC,
.....
SuperSINET
SuperSINETSuperSINET
APAN/Transpac
APAN
Since 2004.1
SuperSINET’s links to US/EU
SuperSINETNII’s router at NewYork
OC48 x 4
Gèant
Abilene
ESnet
OC48
OC48
GbE
GbE
GbE
GbE
GbE ISP(10Gbps)
MAN LAN10GbE Switch
POS Channel
10GbE
10GbE
10GbE
10GbE
GbE x 3
IEEAF Atlantic
SLAC is on ESnet
CERN is on GEANT
WAN Performance Measurement and Monitor
-- Study and Work at KEK
Qi Fazhi
2004-07
Study
• WAN Performance Terminology
• WAN Performance Measurement Tools
• How to get larger throughput
Terminology / Elements be relationship to WAN Performance
• Bandwidth– Physical bandwidth, or capacity (C)– Available bandwidth (A)
• Throughput– Maximum throughput– Achievable throughput
• Latency / RTT
TERMINOLOGY•Latency
•Packet Loss
•Bandwidth:the speed that a network element can forward traffic.
(is independent of end hosts and protocol type)
–Physical/capacity:maximum number of bits per second a network element can transfer(is determined by the slowest network element along the path)
–Available:the capacity minus utilization over a given time interval.
•Throughput:amount of data that is successfully sent from one host to another via a network(may be limited by every component along the path from source host to destination host)
–Maximum throughput
–Achievable throughput
Network Performance Measurement Tools
• Iperf/Netperf: tools for measuring end-to-end TCP/UDP performance
• pipechar: hop-by-hop bottleneck analysis tool
• pchar: hop-by-hop performance measurement tool
• traceroute: lists all routers from current host to remote host
How to get larger throughput
Decrease Latency / RTTPacket Loss
Choose the best path Routing coordination
Larger Throughput
TCP Tuning?
Latency / RTT Packet Loss
Parallel stream TCP
TCP Tuning
• Why?– Bandwidth increases– TCP shortage
• How?– Modify the TCP Windows Size– Modify the System Buffer Size– ……– Some New Protocol: FAST
TCP
• TCP is adaptive• It is constantly trying to go faster• It slows down when it detects a loss
• How much it sends is controlled by windows• When it sends is controlled by received
ACK’s(or timeouts)
Summary of techniques to maximize TCP WAN
throughput • congestion window (CWND):TCP uses it to determine how
many packets can be sent at one time, The larger the congestion window size, the higher the throughput.
• TCP “slow start” and “congestion avoidance” algorithms determine the size of the congestion window
• The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket (System)
• For each socket, there is a default value for the buffer size, which can be changed by the program using a system library call just before opening the socket.
• Kernel enforced maximum buffer size.
Why Tuning?
Bandwidth*Delay Product
• Bandwidth * Delay = number of bytes in flight to fill path
• The optimal buffer size is twice the bandwidth*delay product of the link(recover from errors)
• buffer size = 2 * bandwidth * delay = bandwidth * RTT • bandwidth :get it from pipechar or pchar • RTT:get it from Ping• Example:ping time is 50 ms ; the end-to-end network consists of
all 100 BT Ethernet and OC3 (155 Mbps).then TCP buffers should be .05 sec * (100 Mbits / 8 bits) = 625 KBytes
• most of today UNIX OS's by default have a maximum TCP buffer size of only 256 KB! So we should do some tuning……
buffer Windows throughputBW*Delay
Important Points About TCP
• Throughput• congestion window :determine how many packets can be sent at one
time ,the larger the congestion window size, the higher the throughput. • the size of the congestion window :slow start (constantly trying
to go faster)and congestion avoidance (slows down when it detects a loss)• maximum congestion window :is related to the amount of buffer
space that the kernel allocates for each socket
• buffer size :• 2*Bandwidth*Delay Products ( BDP ) = bandwidth * RTT
Summary
Routing coordinationKEK<->IHEP
• Before Change Status– Go through Russia and USA
• KEK->IHEP: KEK->Sinet->Abilene->RBNet->CSTnet->IHEP
• IHEP->KEK: IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK
– RTT is very large
• After Change Status– KEK->IHEP: KEK->SINet->APAN-JP->CERnet->NSFC->CSTnet->IHEP
– IHEP->KEK(not changed): IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK
– For the reason of not changing IHEP->KEK route,the RTT is still large,but there are some changes in the throughput
KEK-IHEP Route Tuning
KEK2505
Sinet2907 Abilene11537 APAN_JP7660
RBnet5568
Dargon_tap9407 Cernet4538 NSFC9406 CSTNet7497
AS1239
AS9405
AS3356
Level 3 Sprint
APAN_JP&APAN_CN Link
IHEP3460
KEK->I HEP(Vi a USA/ RUSSI A)
0
0. 2
0. 4
0. 6
0. 8
1
1. 210 15 20 30 40 50 60 70 80 90 100
110
120
130
140
150
160
Test Durati on (Seconds)
Thr
ough
put (
Mbps
)
KEK->I HEP(Vi a APAN)
0
0. 5
1
1. 5
2
2. 5
10 15 20 30 40 50 60 70 80 90 100
110
120
130
140
150
160
Test Durati on (Seconds)
Thr
ough
put (
Mbps
)
Parallel TCP Stream KEK->IHEP
KEK->I HEP(Vi a APAN)
0
0. 5
1
1. 5
2
2. 5
10 15 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160
Test Durati on (Seconds)
Throu
ghpu
t(Mb
ps)
One TCP Stream Parallel TCP Streams
WAN Performance Measurement/Monitor
• Target– Packet loss & latency– Throughput
• Tools
• Mechanism
WORK
Test PC Data
Remote RemoteRemote
WWW
Traceroute/Ping iperf netperf
HTTP
Ping/iperf/netperf
Mechanism (Base on www)
CMDResult
System Structure
WAN Performance Measurement and Monitor
WAN Performance Measurement WAN Performance Monitor
Performance data collect
Data File
Performance data analysisPerformance measurement resultPerformance measurement command
The System Interface
Measurement: (RTT & Packet Loss/traceroute & ping)
Traceroute and Ping Result
Traceroute and Ping Result with graph
ms
Measurement: Throughput with Iperf
Measurement Result: Throughput with Iperf
Measurement: Throughput with Netperf
Measurement Result: Throughput with Netperf
Monitor: RTT and Packet loss with ping
Monitor Result: RTT and Packet loss with ping
Monitor: Throughput with iperf
Monitor Result: Throughput with iperf
README
• INSTALL– creat a user account: monitor
– cd ~monitor
– tar xvf monitor.tar
– edit /usr/local/apache/conf/httpd.conf
– edit the ~monitor/ping_list.txt and ~monitor/iperf_list.txt
– edit the /etc/crontab
– http://serveraddress/monitor/
Useful links
• http://www.apan.net
• http://ws.arin.net/cgi-bin/whois.pl