-1- wise* trafview wise * trafview etri’s content-aware internet application traffic measurement...
TRANSCRIPT
-1-
Wise*TrafView
Wise*TrafView
ETRI’s Content-aware Internet Application Traffic Measurement and Analysis System
APAN Network Technology WSJanuary 29, 2004
IP Networking Technology Team, ETRI{jungsp, chunghs, choits, tsjeong}@etri.re.kr
-2-
Wise*TrafView
Contents
▣ Current Internet Application Traffic Characteristics
▣ Wise*TrafView : Our Approach
▣ Wise*TrafView : Implementation and Deployment Experiences
▣ Summary
-3-
Wise*TrafView
Measurement Application Areas
▣ Network Problem Determination and Analysis
▣ Traffic Report Generation
▣ Intrusion & Hacking Attack (e.g., DoS, DDoS) Detection
▣ Service Level Monitoring (SLM)
▣ Network Planning
▣ Usage-based Billing/Accounting (both between SPs and SP-and-Customer)
▣ Customer Relationship Management (CRM)
▣ Marketing
-4-
Wise*TrafView
2. CURRENT INTERNET APPLICATION TRAFFIC
CHARACTERISTICS
-5-
Wise*TrafView
Current Internet Traffic Characteristics
▣ High-speed networks (Mbps Gbps Tbps)
▣ High-volume traffic
▣ Variety of Applications◈ Streaming media (Windows Media, Real Media, Quicktime)
◈ P2P traffic
◈ Network Games
◈ Network Security Attacks
◈ Etc.
-6-
Wise*TrafView
Application Recognition(1)
▣ Limitations of port-based recognition◈ The port database maintained by IANA doesn’t reflect the
real-world situation– Most newer applications simply do not register their ports
– Sometimes they even invade well-known port area to pass thorough firewalls
◈ Most bandwidth hogs, nowadays, dynamically allocate ports
– They are not linked up with any fixed ports!
-7-
Wise*TrafView
Port/ApplicationPort-basedAccounting
Contents-awareAccounting
80/HTTP 67 GB59.1 GB
(11.8% reduced)
21/FTP_CTRL 0.29 GB 0.28 GB
20/FTP_DATA 43 GB 42 GB
?/FTP_DATA_PASSIVE n/a6 GB
(14.3% of FTP_DATA,2% of the total volume)
5003/? 692 MB
HTTP: 13.2 MB
BUGS_MUSIC: 420.8 MB
EDONKEY: 172.3 MB
etc.: 85.7 MB
PosTech Traffic Breakdown
- PosTech Campus Network(24h sum in May, 304GB total volume)
Application Recognition(2) :Trend in Internet Application Traffic Characteristics
-8-
Wise*TrafView
Application Recognition(3)
▣ Many applications require to be identified by payload inspection
▣ Why is payload inspection necessary?
◈ Several applications can use the same port number
◈ Identification error can be occurred by ephemeral port number
◈ Some applications can use a dynamic port number
◈ Etc.
-9-
Wise*TrafView
▣ Application example : Passive FTP
Application Recognition(4)
client.1302 server.21 (FTP_CTRL_REQ)
server.21 (FTP_CTRL_REP)client.1302
client.1303
client.1303
server.20 (FTP_DATA_DOWN)
server.20 (FTP_DATA_UP)
server.49152 (FTP_DATA_PSV_UP)
client.1306
client.1306
server.49152 (FTP_DATA_PSV_DOWN)
% ls % passive% get wmggw.mp3
% quit
% ftp server
49152
-10-
Wise*TrafView
Why Port-based Approach is not enough?
▣ Non-flow based measurement◈ Not enough for the above requirements
▣ Typical Flow-based Measurement (like NetFlowTM, cflowd, LFAP)
◈ Typically a flow is defined as a set of packets passing an observation point in the network during a certain time interval and having a set of common properties
◈ 5-tuple packet header fields are used for this
◈ New applications such as P2P, streaming and network games have characteristics of dynamic port allocation
▣ More Detailed Analysis is needed!!◈ Typical Flow-based Measurement is not enough
◈ Need more detailed analysis depending on applications– It may require content filtering
-11-
Wise*TrafView
3. Wise*TrafView : OUR APPROACH
-12-
Wise*TrafView
Motivation
▣ Develop precise Internet application traffic measurement and analysis system
◈ Precise application analysis
◈ Passive flow-based measurement
◈ Sub-transaction(flow) level detailed application analysis
◈ Pseudo-realtime analysis
◈ No loss capture and analysis
◈ No sampling but capturing all
◈ For various Internet measurement purposes
-13-
Wise*TrafView
Flow Concept
▣ A “flow” is◈ a sequence of packets whose <src and dst IP addresses, src and dst port numb
ers, and protocol id> are all identical
▣ Why flow?◈ The size of entire raw packet streams for a given unit time are prohibitively en
ormous to be analyzed in time
◈ Each individual packets in a flow contain duplicate information
◈ Packets in the same flow are correlated; we can identify more packets which were previously categorized as unknown application
a packet
a distinctive signatureof application “X”
a flow generated by application “X”
Now, these pkts can alsobe identified as “X”
-14-
Wise*TrafView
Internet Application Classification
▣ Type S: Simple Application Type ◈ for an application which uses a well-known port number or which uses a regi
stered port number but is popularly used
◈ Applications : WWW, FTP, SMTP, BGP, etc.
▣ Type P: Payload Application Type◈ for an application which uses a registered port number but requires payload i
nspections for precise classification
◈ Applications : HTTP_ALT(8080,8081,9000), MSNMessenger(6891-6900), KAZZA(1214), …
▣ Type R: Reverse Application Type◈ for an application which uses a registered but requires comparison with a cor
related reverse flow for the precise classification
◈ Applications : eDonkey down, WINMX down, GuruGuru BBS(9999)…
▣ Type C: Co-related Application Type◈ for an application which uses a dynamic port number assignment
◈ Applications : Passive FTP, RTSP, Windows Streaming, …
-15-
Wise*TrafView
Capture AgentCapture Agent
Analysis Server
Database
GUI
...
NIC IPCAPCard
...
NIC IPCAPCard
splitter
flow and packetrecords (NFS)
recognition and analysisresults (ODBC)
ARCLConfig-File
... ......
System Architecture Overview
-16-
Wise*TrafView
Agent : Generating Flow & Packet Records
▣ Carries on simple filtering and signature matching functions
▣ Generates flow records & packet records◈ Flow record
– For flow information
– Fields : IP addr, port, protocol, flow duration, packets, bytes, …
◈ Packet record – for individual packet
– Fields : timestamp, TOS, TTL, TCP flags, payload, …
– Important for analysis server’s precise application identification
◈ This procedure aggregates and organizes the traffic information and reduces the amount of traffic volume transferred to the server
-17-
Wise*TrafView
Analysis Server :Enhanced Application Recognition
▣ Wise*TrafView utilizes some enhanced proprietary recognition mechanisms in a comprehensive way◈ Application specific signature matching,
◈ temporal and spatial flow correlation,
◈ dynamic port recognition and utilization, and
◈ some heuristics
▣ Not only capable of discriminating applications, but also their sub-flows◈ e.g., HTTP HTTP_REQ, HTTP_REP, HTTP_REQACK, etc.
-18-
Wise*TrafView
Analysis Server : AS and Country Mapping
▣ Identifying flow sources and destinations◈ Both source and destination IP address of a flow are mapp
ed to ASes and finally to country codes
◈ This helps to locate the source and the sink of a flow– enables discrimination among transit, inbound, and outbound traffi
c flows
-19-
Wise*TrafView
Application Recognition Configuration Language (ARCL)
▣ Configurability and Adaptability▣ Why adaptability so important?
◈ The highly frequent nature of Internet applications’ appearance and disappearance
◈ Swift mutation of applications◈ Localization of the use patterns of applications
▣ Wise*TrafView copes with the problem by introducing ARCL (Application Recognition Configuration Language)
▣ By taking advantage of ARCL, Wise*TrafView
◈ doesn’t need to be re-built or re-installed by any module for extending or modifying recognition coverage; editing the configuration in ARCL and re-enforcing suffices
-20-
Wise*TrafView
Config-file by ARCLapplication WWW {
port_rep_name HTTP port 80 protocol TCP{ // S type
decision_group HTTP_REQ_REP_ACK {
src_port >= 1024
dst_port == 80
}
decision_group HTTP_REP_REQ_ACK {
src_port == 80
dst_port >= 1024
}
}
port_rep_name HTTP_ALT port 8080 protocol TCP{ // P type
src_disc_pattern=="HTTP" in pkt 0-2 at byte 0 - 4
( dst_disc_pattern=="GET" in pkt 0-3 at byte 0 - 10 ||
dst_disc_pattern=="POST" in pkt 0-3 at byte 0 - 10 )
decision_group HTTP_ALT_REQ_REP_ACK {
src_port >= 1024
dst_port == 8080
}
decision_group HTTP_ALT_REP_REQ_ACK {
src_port == 8080
dst_port >= 1024
}
}
}
application EDONKEY { // R typeport_rep_name EDONKEY_DOWN port 4662 protocol TCP{
dst_disc_pattern=="0xe33d000000" in pkt 2-3 at byte 0 - 4decision_group EDONKEY_DOWN_REQ_REP_ACK {
src_port >= 1024dst_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555
}decision_group EDONKEY_DOWN_REP_REQ_ACK {
src_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555dst_port >= 1024
} } ……}
application FTP { // C type port_rep_name FTP port 21 protocol TCP{ src_ref_pattern=="r/227 Entering Passive Mode \(\d{1,3},\d{1,3},\d{1,3},\d{1,3},(\d{1,4}),(\d{1,4})\)/$src_port = atoi($1)*1024 + atoi($2)" in pkt any at byte 0-35 induce FTP_DOWN_P decision_group FTP_REQ_REP_ACK {
src_port >= 1024dst_port == 21
} decision_group FTP_REP_REQ_ACK {
src_port == 21dst_port >= 1024
} }}
-21-
Wise*TrafView
4. Wise*TrafView : IMPLEMENTATION & DEPLOYMENT EXPERIENCES
-22-
Wise*TrafView
Deployment Experiences
▣ ETRINet◈ Link speed : 100Mbps FastEthernet, using libpcap◈ Traffic volume : 70Mbps◈ Period : May 2003 – Current◈ Analysis result : S(52.83%), P(9.99%), R(2.38%), C(4.92%), Unknown(28.88%)
▣ Postech◈ Link speed : 1Gbps Ethernet, using libpcap◈ Traffic Volume : 60 – 70Mbps◈ Period : May 2003(1week)
▣ Univ. of Andong◈ Link speed : FastEthernet, using capturing card developed by ETRI◈ Traffic volume : 60-70Mbps◈ Period : Oct. 2003 - Current
▣ Other experiences◈ Deployment on the International link of one of Korean Internet Exchange point using O
C-3 POS card developed by ETRI
-23-
Wise*TrafView
Port/ApplicationPort-based
Accounting (A)Contents-awareAccounting (B)
Accuracy(A/B)
80/HTTP 67 GB59.1 GB
(11.8% reduced)0.882/1.0
21/FTP_CTRL 0.29 GB 0.28 GB 0.965/1.0
20/FTP_DATA 43 GB 42 GB 0.977/1.0
?/FTP_DATA_PASSIVE
n/a6 GB
(14.3% of FTP_DATA,2% of the total volume)
0.0/1.0
5003/? 692 MB
HTTP: 13.2 MB 0.0/1.0
BUGS_MUSIC: 420.8 MB 0.0/1.0
EDONKEY: 172.3 MB 0.0/1.0
etc.: 85.7 MB 0.0/1.0
- PosTech Campus Network(24h sum in May, 304GB total volume)
PosTech Traffic Analysis Result
-24-
Wise*TrafView
▣ Hardware◈ For lower speed links (<= 622Mbps)
– Capture agent– high performance PC: Zeon 2.4GHz * 2 + CPU, 2GB+ RAM
– Analysis server– high performance PC: Zeon 2.8GHz * 2 + CPU, 1GB+ RAM, 100GB+ H
DD
◈ For Higher speed links ( > 1 Gbps, under developing)– Clustered capture system– Hardwired logic for supporting wire-speed processing
▣ Software◈ Capture agent
– Linux
◈ Analysis server– Linux, MySQL
System Spec.(1)
-25-
Wise*TrafView
System Spec.(2)
▣ Link Signal Splitters◈ Electrical
– Ethernet tap, DS-3 tap, etc.
◈ Optical– ordinary optical splitter
– independent of physical and data-link layer protocols
▣ High Performance Packet Capture Cards◈ Model A: for lower speed links
– Ethernet, FastEthernet, DS-3/(E3)
◈ Model B: for middle speed links– ATM at OC-3, and POS at OC-3, OC-12 (622Mbps)
-26-
Wise*TrafView
User Interface
▣ Web-based Interface◈ simple
◈ easy to use
◈ intuitive
◈ portable
▣ A web site for each measurement site can be easily established◈ Autonomous authentication and authorization can be supported
-27-
Wise*TrafView
GUI (Traffic Report)
-28-
Wise*TrafView
-29-
Wise*TrafView
GUI (Traffic Matrix)
-30-
Wise*TrafView
5. SUMMARY
-31-
Wise*TrafView
The Merits of Wise*TrafView
▣ Transparent Packet Capture◈ Complete independence of the existing networking equipment
▣ Flow-based Measurement and Analysis◈ Reduced load
◈ Higher degree of recognition
▣ Understanding Application Specific Contexts◈ By means of enhanced application recognition algorithms, sub-flows can be d
etected
▣ Scalable◈ Can scale up from tens of Mbps to Gbps
◈ Supports various physical and data-link layer technologies
▣ Highly Extensible and Adaptable◈ Easy configuration with ARCL
