network intrusion detection applications and research like zhang
TRANSCRIPT
![Page 1: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/1.jpg)
Network Intrusion DetectionNetwork Intrusion DetectionApplications and ResearchApplications and Research
Like ZhangLike Zhang
![Page 2: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/2.jpg)
OutlineOutline
• Recent debate over NIDS
• Introduction to NIDS
• A survey of current NIDS products
• Research on anomaly NIDS
• Conclusion
![Page 3: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/3.jpg)
Is NIDS dead?Is NIDS dead?
“Hype Cycle for Information Security”
Gartner Report, 2003
• False positives and negatives
• Requiring full-time monitoring (24 hours a day, seven days a week, 365 days a year)
• Market failure
• Will be obsolete by 2005
![Page 4: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/4.jpg)
Current SituationCurrent Situation
• Intrusion Detection evolves into Intrusion Prevention
• New types of IDS come into play (distributed IDS, application-based IDS,etc.)
• NIDS is applied to firewall, anti-virus system, optional plug-in for server-side program, or deployed as a standalone product
![Page 5: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/5.jpg)
NIDS TechniquesNIDS Techniques
• Signature-based
• Anomaly-based
• Stateful detection
• Application-level detection
![Page 6: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/6.jpg)
Signature-base NIDSSignature-base NIDS
Similar to the traditional anti-virus applicationsExample:Martin Overton, “Anti-Malware Tools: Intrusion Detection Systems”,
European Institute for Computer Anti-Virus Research (EICAR), 2005
Signature found at W32.Netsky.p binary sample
Rules for Snort:
![Page 7: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/7.jpg)
Anomaly DetectionAnomaly Detection
• Already used by industry--Protocol Anomaly
--Statistical/Threshold based
• In Research--Data mining
![Page 8: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/8.jpg)
Protocol Anomaly DetectionProtocol Anomaly Detection
Based on the well established RFCs
Focus on the packet headerExample:--All SMTP commands have a fixed maximum size. If the size exceeds
the limit, it could be a buffer overflow or malicious code inserting
attack
--SYN flood attack: attacker sends SYN with fake source address
--Teardrop attack: fragmented IP packets with overlapped offset
![Page 9: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/9.jpg)
Threshold basedThreshold based
Using training data to generate a statistical
model, then select proper thresholds for
network environment (traffic volume, TCP
packet count, IP fragments count, etc.)
-- usually used as an complementary tool
![Page 10: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/10.jpg)
Stateful IDSStateful IDS
• No practical Solutions
• Very simple implementing
Example:
Snort uses patter matching in continuous Packets.
Traditional signature rules: “pattern1” “pattern1 || pattern2”
The rule now can be defined as: “pattern1.*pattern2”
![Page 11: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/11.jpg)
Application-level IDSApplication-level IDS
Focus on specific services or programs
(Web Server, Database, etc.)
Example
--Monitoring all invocation for Microsoft RPCs
--Analyze HTTP request for malicious query strings
Products:
--mod_security: an optional IDS component for Apache
Web Server
![Page 12: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/12.jpg)
IDS TodayIDS TodayProducts and ApplicationsProducts and Applications
• Snort
• McAfee Intrushield
• ISS RealSecure
• Cisco IPS
• Symantec IDS
![Page 13: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/13.jpg)
SnortSnort
• Open Source, since 1998
• Used by many major network security products
• Signature-based (more than 3000)
• Simple IP header protocol anomaly detection
• Simple stateful pattern matching
![Page 14: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/14.jpg)
McAfeeMcAfee
• Profile-based anomaly detection--Manually create profile
--Create profile by self-learning through a training period
• Using profile plus threshold for defending against DOS and DDOS
• Inspect encrypted traffic by collecting the server side private keys
![Page 15: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/15.jpg)
ISS RealSecureISS RealSecure
• About 2000 signatures• Application-based approach
--identifying any possible exploit to the published vulnerabilities of MS RPC, IIS, Apache, Lotus, etc.
• Additional support for P2P,Instant Messengers• Virtual Prevention System
--a virtual environment to examine the execution of a file in order to find any possible malicious behaviors
• Support for IPv6--Detect possible backdoors which enable the IPv6 of a system (usually off)
![Page 16: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/16.jpg)
Cisco IPS produtcsCisco IPS produtcs
Protocol decoding
Threshold based property checking
Signature matching
Protocol Anomaly Detection
Checking file behaviors by intercepting all calls to the system resources
![Page 17: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/17.jpg)
SymantecSymantec
• Multi-steps (protocol, vulnerability, signature, DOS, traffic, evasion check)
• Unique feature: evasion check
e.g. request “/index.html” can be replace with “/%69nd%65x.html” to evade the signature matching
![Page 18: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/18.jpg)
Summary of Current ProductsSummary of Current Products
Snort McAfee Intrushield
ISS RealSecure
Cisco IDS Symantec IMUNE
Signature General x x x x x
Application based x
Anomaly Detection
Profile-based x
Vulnerability-based x x
Statistical-based x x x
Protocol-based x x x
Self-learning x
Application specific x x
Stateful x x
Behavior x x
Encrypted Traffic Detection x
IPv6 Support x
![Page 19: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/19.jpg)
Challenges for NIDSChallenges for NIDS
• High false positives-- FP of 0.1% means a normal packet will be misclassified as an alert for every 1000 normal packets, which is about one error alert per minute on a 100M network
• Zero day attack (unknown attack)--Most current products rely on signature-based detection, difficult to detect new attacks.
• Poor at automatically preventing ability--Human interaction is required when attack is detected
![Page 20: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/20.jpg)
Research on Intrusion DetectionResearch on Intrusion Detection
• Columbia University--Data mining based (since 1997)
• University of California at Santa Barbara--Service Specific (HTTP)--Stateful IDS
• Florida Institute of Technology--Protocol Anomaly (Statistical based)
• University of Minnesota--MIND (Minnesota Intrusion Detection System)
![Page 21: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/21.jpg)
Columbia Univ. IDSColumbia Univ. IDS
• 1997, Applied RIPPER rule learning algorithm on UNIX system calls monitoring for malicious events detection
• 1998, Applied the algorithm on off-line network traffic data (clean training data)
• 2000, Applied EM and clustering algorithm for dealing with noisy dataset
• 2001, Developed an complete experiment NIDS based on those algorithms.
• 2004, New approach towards payload anomaly detection
![Page 22: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/22.jpg)
Implementing ProcedureImplementing Procedure
Pre-Processing
Feature construction
Apply RIPPER algorithm
Process raw packet data
Create statistic features
Rule learning
Wenke Lee, Sal Stolfo, and Kui Mok., “A Data Mining Framework for Building Intrusion Detection Models”, Proceedings of the 1999 IEEE Symposium on Security and Privacy, Oakland, CA, May 1999
![Page 23: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/23.jpg)
Pre ProcessingPre Processing
SYN flood attack
![Page 24: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/24.jpg)
Feature ConstructionFeature Construction
(service=http, flag=S0, dst_host=victim),
(service=http, flag=S0, dst_host=victim)
-> (service=http, flag=S0, dst_host=victim)
[0.93, 0.03, 2]
93% of the time, after two http connections with S0
flag are made to host victim, within 2 seconds from
the first of these two, the third similar connection is
made, and this pattern occurs in 3% of the data
![Page 25: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/25.jpg)
RIPPLE RulesRIPPLE Rules
smurf :- service=ecr_i, host_count >= 5,
host_srv_count>=5( if the service is icmp echo request, and connections with the same
destination host are at least 5, and connections with the same service
are at least 5,then it is a smurf/DOS attack)
satan :- host_REJ_%>=83%, host_diff_srv_% >=
87%( for connections with the same destination host, if the rejection rate is at least
83%, and the percentage of different services is at least 87%, then it is a
santa/PROBING attack)
![Page 26: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/26.jpg)
Experiment ResultsExperiment Results
Applied on DARPA’98 Intrusion Detection Evaluation Data Set
![Page 27: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/27.jpg)
Payload based ApproachPayload based Approach
K. Wang, S. J. Stolfo, “Anomalous Payload-based Network
Intrusion Detection”, RAID 2004• Construct the statistical model for all bytes in the header• Use Mahananobis distance to measure the difference
Problems:• Clean training data is required• False positive (unacceptable)
![Page 28: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/28.jpg)
Service Specific IDS by UCSBService Specific IDS by UCSB
V.Giovanni et al at University of California at Santa Barbara
Since 2002
• Application level
• Focuses on HTTP request
• HTTP request analyzing
• Constructing models for important fields in the request instead of all bytes of the payload (Columbia payload approach)
![Page 29: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/29.jpg)
Sample RequestSample Request
RequestGET /scripts/access.pl?user=johndoe&cred=admin
Properties for DetectionRequest Type: e.g. GET
Request Length: e.g. Length(“GET /scripts/access.pl?user=johndoe&cred=admin”)
Payload Distribution
![Page 30: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/30.jpg)
Request TypeRequest Type
Assumption: If a rare used request type was found, it is very possible it
will initiate malicious activity
Anomaly Score:AStype=-log2(p[type])
P[type] stands for the probability of a certain type
![Page 31: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/31.jpg)
Request LengthRequest Length
Assumption:The request length should not vary much of a certain type.
Otherwise, it is probably caused by some attacks
(e.g. overflow)
Anomaly Score:ASlen=1.5(1-)/(2.5*)
P[type] stands for the probability of a certain type
![Page 32: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/32.jpg)
Characters DistributionCharacters Distribution
256 ASCII Characters
e.g. “passwd” -> “112 97 115 115 119 100”
Distributions: {0.33, 0.17, 0.17, 0.17, 0.17}
2=f(Oi, Ei) (i corresponds from segment 0 to 5)
Aspd= 2*(15/L) (L stands for the payload length)
Segment 0 1 2 3 4 5
ASCII Value 0 1-3 4-6 7-11 12-15 16-255
![Page 33: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/33.jpg)
Final Anomaly ScoreFinal Anomaly Score
AS=0.3*AStype + 0.3*ASlen+0.4*ASpd
![Page 34: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/34.jpg)
Later Research at UCSBLater Research at UCSB
Structure Inference with Markov Model
![Page 35: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/35.jpg)
Other Properties UsedOther Properties Used
• Token Finderif the query parameter is drawn from known candidates
• Attribute Presence or absencemalicious crafted request usually ignore the order of parameters
• Access Frequency
• Invocation order
• Request time interval
![Page 36: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/36.jpg)
Experiment ResultsExperiment Results
• Tested at UCSB campus network and Google
• False positive 0.06%
Major cons:
Limited to HTTP service
![Page 37: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/37.jpg)
Packet Header Anomaly DetectionPacket Header Anomaly Detection
Packet Header Anomaly Detection (PHAD)developed by Florida Institute of Technology since 2001
Basic Assumption:If an event x happened n times with r different results in the
training period, the probability of a novel data is r/n
![Page 38: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/38.jpg)
ImplementingImplementing
Step 1:Assign the novel data probability to important fields
of the packet header (protocol type, flags, etc.)
Step 2:Adding all the novel data probability together as a
threshold
![Page 39: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/39.jpg)
MINDSMINDS
MINDS (Minnesota Intrusion Detection System)
Statistic outlier-based anomaly detection
Compared 5 outlier-based scheme:• K-th nearest neighbor• Nearest neighbor• Mahalanobis-distance based• Local Outlier Factor (LOF)• Unsupervised SVMs
![Page 40: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/40.jpg)
Comparison ResultComparison ResultA. Lazarevic, et al, “A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection”, Proceedings of the 3rd SIAM Conference on Data Mining, San Francisco, 2003
![Page 41: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/41.jpg)
Some Emerging ApproachesSome Emerging Approaches
• SVMs(unsupervised and supervised)
• PCA
• PCA + SVMs
• Neural Network
![Page 42: Network Intrusion Detection Applications and Research Like Zhang](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649dac5503460f94a9b3c0/html5/thumbnails/42.jpg)
ConclusionConclusion
• Signature based approaches still play the major part in practical IDS
• Anomaly detection has only very limited success• New approaches are proposed everyday, but
false positive and detection rate are still the major problem
• Various mechanisms should work together for maximum success