distributed network traffic monitoring and analysis...
TRANSCRIPT
WebTrafMon II POSTECHDP&NM Lab.
Distributed Network Traffic Monitoring and Analysis using Load Balancing Technology
Distributed Network Traffic Monitoring and Analysis using Load Balancing Technology
Soon-Hwa Hong, Jae-Young Kim, Bum-Rae Choand James W. Hong
DP&NM Lab. Dept. of Computer Science and Engineering
POSTECH, Pohang KoreaEmail: {padosori, jay, brcho, jwkhong}@postech.ac.kr
http://dpnm.postech.ac.kr/
WebTrafMon II POSTECHDP&NM Lab.
(2/16)
Introduction?Network traffic (text, image, software, audio, video) is
increasing continuously both on the Internet and Intranet.? A simple, accurate and efficient network traffic monitoring
and analysis is required to understand the current usage as well as to plan for future.
?Many shortcomings exist in currently available monitoring systems.? cannot analyze long-term traffic.? do not have monitoring capability from multiple network points.? capture, analysis and presentation all in one machine.? cannot prevent packet drops from the system overload.
?WebTrafMon II attempts to overcome these shortcomings using distributed architecture and load balancing technique.
WebTrafMon II POSTECHDP&NM Lab.
(3/16)
Related WorkUserInterface
Distributed Architecture
Support for Multiple network points
Analysis Scope Analysis Interval
Analysis Method
CaptureMethod
Textnonolayer 7currentreal timelibpcaptcpdump
Webyesyeslayer 7current, hourly, daily, monthly, yearly
real time, batch
libpcapWebTrafMon II
Webnonolayer 7current, hourlyreal time, batch
libpcapWebTrafMon
Webyesyeslayer 25 minute, hourly, daily, weekly, monthly
batchsnmpagent
MRTG
X-Windowsnonolayer 7
current, user-specified time
real time, batch
libpcapethereal
Webnonolayer 75 second, hourlybatchlibpcapntop
WebTrafMon II POSTECHDP&NM Lab.
(4/16)
WebTrafMon II Requirements
?analyze various types of information: host information, network, transport, and application layer protocols. ?analyze present real-time, hourly, daily, monthly
and yearly network traffic data automatically.?analyze multiple network points traffic.?no packet drops.?web-based graphical user interface.
WebTrafMon II POSTECHDP&NM Lab.
(5/16)
Centralized vs. DistributedCentralized Traffic Analysis Architecture
presentation
analysis
capture
network point user
■ system overload occurs frequently and manypackets are dropped
■ cannot support for multiple network points■ presentation time is slow
Distributed Traffic Analysis Architecture
analysis
capture presentation
usernetwork point
■ capture, analysis and presentation modules execute on separate machines to minimize system overload
■ can support for multiple network points■ presentation time is fast
network traffic datapacket header information
packet header information network traffic data
analyzed information
distributedenvironment
analyzed information
WebTrafMon II POSTECHDP&NM Lab.
(6/16)
WebTrafMon II: Design
database
analyzer web viewer
… … … … … … …
1. multiple network point packet capture and analyze packet header
(real-time, hourly, daily, monthly, yearly)
statistics network traffic data
2. analyze packet header and save into DB and make short term and long term traffic data
3. query to database from user request and give information to user
probe
network point
promiscuous mode packet capture
probe
network point
promiscuous mode packet capture
analyze packet header and save into DB make real-time, short term,
long term traffic data
user
distributedenvironment
request
response
packet header information
WebTrafMon II POSTECHDP&NM Lab.
(7/16)
Packet Capture Module (Probe)
probe(analysis packet header)
network point
Log File
log format
… … … … … … … … … … …
log format
log format
log format
?Probe captures packet with promiscuous mode, analyzespacket header and saves into log file?No packet drops from system overload using an independent
packet capture module.
Log format : time, length, frame_type, source ip, destination ip,protocol, source port and destination port information
save packet header information into log file
promiscuous mode packet capture
WebTrafMon II POSTECHDP&NM Lab.
(8/16)
Log Format
6 6 2 4data CRC
46-1500 bytes
TCP data
Ethernet
IP
TCP
Log format2 2
time length frame_type src_ip dst_ip protocol src_port dst_port4 4 42 2 2 21 (bytes)
frame capture time
total Ethernet frame size(Ethernet header + data + CRC)
Dst addr
Srcaddr Type
4 1 IP data4Dst addrSrc addr Protocol
dst portsrc port
WebTrafMon II POSTECHDP&NM Lab.
(9/16)
Packet Analysis Module (Analyzer)
… … … … …
log file Log Transformerlog data
log file
log file
database
IP-based, non IP-based data
DB Analyzer
IP-based, non IP-based data
statistical real-time, hourly, daily, monthly and yearly data.
?An Analyzer is divided into a Log Transformer module and DB Analyzer module. ?Log Transformer assorts log files into IP-based data, and non IP-based data (e.g., ARP, RARP, IPX). ?Log Transformer saves these assorted data to database. ?DB Analyzer analyzes assorted data in database and makes statistical real-time,hourly, daily, monthly and yearly data.
WebTrafMon II POSTECHDP&NM Lab.
(10/16)
Data Translation by Analyzer for Long-Term Traffic
Log files
hourly_application
hourly_data_sent
hourly_data_received
hourly_data_exchanged
hourly_ network
hourly_transport
daily_*_table monthly_*_table yearly_*_table
raw_ip_table raw_non_ ip_table
hourly
monthly yearly
daily
WebTrafMon II POSTECHDP&NM Lab.
(11/16)
DB Schema
SMALLINT UNSIGNEDdst_port
SMALLINT UNSIGNEDsrc_port
TINYINT UNSIGNED
protocol
INTdst_ip
INTsrc_ip
SMALLINT UNSIGNED
frame_type
SMALLINT UNSIGNEDlength
INTtime
Type Name
SMALLINT UNSIGNED
ether_type
SMALLINT UNSIGNED
length
INTtime
TypeName
BIGINTcount
BIGINTbytes
VARCHAR(100)frame_name
VARCHAR(20)frame_type
TINYINT UNSIGNEDday
TypeName
BIGINTcount
BIGINTbytes
VARCHAR(100)frame_name
VARCHAR(20)frame_type
TINYINT UNSIGNEDyear
TypeName
BIGINTcount
BIGINTbytes
VARCHAR(100)frame_name
VARCHAR(20)frame_type
TINYINT UNSIGNED
month
TypeName
BIGINTcount
BIGINTbytes
VARCHAR(100)frame_name
VARCHAR(20)frame_type
TINYINT UNSIGNED
hour
TypeName
raw_ip_table
raw_non_ ip_table
hourly_network_table
daily_network_tablemonthly_network_table
yearly_network_table
hourly
dailyyearly
monthly
This is a *_network_table example. network_table has IP information from raw_ip_table and ARP and so on information from raw_non_ip_table.
WebTrafMon II POSTECHDP&NM Lab.
(12/16)
Web Viewer Module Design
raw_ip, non_ip table
data_sent table
data_received table
data_exchanged table
network table
transport table
application table
Database
Web Browser(user)
Web Viewer
Database
Client
Web
Serverquery
reply
CGI
requestresponse
WebTrafMon II POSTECHDP&NM Lab.
(13/16)
http
Database
Implementation
Binary Log File
packet capture using libpcap
probe
analyzer
save packet header information into log file
save packet header information into database raw ip, non-ip, table
raw_ip, non_ip_table
data sent table
data received table
data exchanged table
network table
transport table
application table
MySQLClient
NFS distributed environmentNFS distributed environment
MySQLServer
web viewerC-CGI,MySQL Client
query
reply
C-CGI, Apache web-server
Internet
make real-time, hourly, daily, monthly and yearly host, protocol and application information table
user
WebTrafMon II POSTECHDP&NM Lab.
(14/16)
Web-based User Interface: Main View
menuanalysis time
hourly total traffic
detailed views
time interval menu
WebTrafMon II POSTECHDP&NM Lab.
(15/16)
Detailed Views1
3
2
4
5
6
WebTrafMon II POSTECHDP&NM Lab.
(16/16)
Summary & Future Work
?WebTrafMon II overcomes many shortcomings of existing monitoring and analysis systems.
? can analyze real-time and hourly, daily, monthly and yearly network traffic data.
?With load balancing, independent packet capture prevents packet drops from monitoring system overload.
?WebTrafMon II can analyze multiple network points traffic.
? Future work on WebTrafMon II?More analysis on host and application relationships.? Adapt to monitor and analyze other types of IP networks (IPoA,
IPoWDM, etc.)? Traffic analysis based on contents (video, audio, etc.)