intrusion detection/prevention systems · 2020. 2. 22. · – intrusion detection systems –...
TRANSCRIPT
-
Intrusion Detection/Prevention SystemsIntrusion Detection/Prevention SystemsGaps, Research Challenges and SolutionsGaps, Research Challenges and Solutions
Ali A. GhorbaniAli A. Ghorbani
-
IDS/IPS: Gaps, Research Challenges and Solutions2
AgendaAgenda
• Introduction• Network Security
– Intrusion Detection Systems– Lessen learned so far
• QRadar• Research Challenges and Solutions
– Automatically detecting anomalies in flow or event streams– Automated Rules Tuning, Learning and Adaptation– Data visualization – Hardware event and flow correlation– Automatic discovery of network applications
• Conclusions & Future Work
-
IDS/IPS: Gaps, Research Challenges and Solutions3
OverviewOverview
• University of New Brunswick• Established in 1790• Oldest English-speaking University in Canada
-
IDS/IPS: Gaps, Research Challenges and Solutions4
OverviewOverview
• Faculty of Computer Science• First Faculty of Computer Science in Canada
– Artificial Intelligence– Bioinformatics– Computational Geometry– Hardware– Hardware/Software Co-design– Parallel/Distributed Processing– Network and Information Security– Software Engineering – Wireless/Mobile computing
-
IDS/IPS: Gaps, Research Challenges and Solutions5
Research GroupsResearch Groupsias.cs.unb.caias.cs.unb.ca ; ; nsl.cs.unb.cansl.cs.unb.ca
• Intelligent and Adaptive Systems (IAS) – Machine Learning– Intelligent Agents and Multiagent Systems– Web Intelligence & Adaptive Web Systems– Privacy, Security and Trust
• Network Security– All aspects of Network Security– Application of Artificial Intelligence to Network Security
• Critical Infrastructures Protection– Agent-based Interdependencies Modelling and
Simulation (AIMS)
-
IDS/IPS: Gaps, Research Challenges and Solutions6
Computer Computer SecuritySecurity
SecuritySecuritySecure SystemsSecure SystemsBreach:Interception
Confidentiality
Assets are accessible only by authorized parties
Availability
Assets are accessible to authorized parties at appropriate times
Breach:Interruption
Breach:Modificationfabrication
Integrity
Assets can be modified only by authorized parties
Secure
-
IDS/IPS: Gaps, Research Challenges and Solutions7
Secure SystemSecure System
Authentication
Signature
Availability
ConfidentialityData Integrity
Non-Repudiation
Access Control
• Robust Protocols• Access controls• All the normal anti-virus,
anti-intrusion, anti-spam, anti-scam techniques
• Robustness - security implies availability– deny DOS – minimize false positive
-
IDS/IPS: Gaps, Research Challenges and Solutions8
Threat and IntrusionThreat and Intrusion
• Threat: A potential violation of security
– Flaws in design, implementation and operation
• Intrusion/Attack: Unauthorized access to a network or computer system that could compromise CIA !
-
IDS/IPS: Gaps, Research Challenges and Solutions9
Prevent Attacks!Prevent Attacks!
• Wishful hope!• Completely secure systems still a dream• New features and software new vulnerabilities• It is hard to get ride of existing problems
– TCP/IP, OS, etc.• New systems contain old vulnerabilities• Attack sophistication vs. Intruder Knowledge
-
IDS/IPS: Gaps, Research Challenges and Solutions10
Attack vs. KnowledgeAttack vs. Knowledge
-
IDS/IPS: Gaps, Research Challenges and Solutions11
Threat Evolution: Malicious CodeThreat Evolution: Malicious Code
“Sasser”
Human response impossibleAutomated response required,i.e., automated remediation and automated attribution
Seconds
Minutes
Hours
Days
Weeks or months
Time
Human response impossibleAutomated response unlikelyProactive blocking possible
Human response difficult/impossibleAutomated response possible
Human response possible
File Viruses
MacroViruses
e-mailWorms
Blendedthreats
“Warhol”threats
“Flash”Threats,
i.e., Sasser
Boot, ComInfectors
Early 1990s Mid 1990s Late 1990s 2000 2003 2004
Adopted from McAfee
combine the characteristics of viruses, worms, Trojan Horses, and malicious code with serverand Internet vulnerabilities to initiate, transmit,
and spread an attack.
Through advanced scanning, Warhol worms would first start an infection using a list of about 50,000 sites, and then use coordinated scanning techniques to infect the rest of the Internet.
http://www.webopedia.com/TERM/B/virus.htmlhttp://www.webopedia.com/TERM/B/virus.htmlhttp://www.webopedia.com/TERM/B/worm.htmlhttp://www.webopedia.com/TERM/B/Trojan_horse.htmlhttp://www.webopedia.com/TERM/B/Trojan_horse.htmlhttp://www.webopedia.com/TERM/B/server.htmlhttp://www.webopedia.com/TERM/B/server.htmlhttp://www.webopedia.com/TERM/B/Internet.html
-
IDS/IPS: Gaps, Research Challenges and Solutions12
The State of Network Security
• Not good at all! • Why?
– Vulnerabilities exist at level of OSI layers– Very few sysadmins check in real-time– High false alarm rates/Low detection rates– Does not differ by network or team size– Sysadmins feel poorly about current tools– Network incidents are increasing– The speed of attacks accelerates
-
IDS/IPS: Gaps, Research Challenges and Solutions13
The Speed of Attacks AcceleratesThe Speed of Attacks Accelerates
• SQL Slammer:• Blended threat exploits known vulnerability• Payload was single 404 byte UDP packet• Doubled every 8.5 seconds• Achieved full scanning rate (over 55 million scans per
second) after approximately 3 minutes• Infected 90% vulnerable hosts worldwide within 10 minutes
Adopted from McAfee
-
IDS/IPS: Gaps, Research Challenges and Solutions14
IntrusionsIntrusions
• Misuse of protocols• Misuse of service ports• DoS based on crafted
payloads• Bandwidth or Flood
attacks– ICMP echo request Flood– TCP data segment Flood– TCP SYN/RST Flood– TCP SYN Floods– TCP, UDP, ICMP floods
• Buffer Overflows
• Protocol Attacks– SYN Flood– ICMP echo reply flood– UDP Flood
• Protocol Tunneling• Backdoor Intrusions• Low-bandwidth
DoS/DDOS attacks• Logic Attacks
– Land attack– Ping of Death– Teardrop
HTTP traffic on a non-standard port, say port 53
Backdoor service on well-known standard port, e.g., peer-to-peer file sharing using Gnutella on port 80
process table exhaustion, IP stack crashing, or a Web application .soft spot
-
IDS/IPS: Gaps, Research Challenges and Solutions15
How about Detecting IntrusionsHow about Detecting Intrusions
• Intrusions detected through Audit of LogsBut audit logs can get very large!
• Reactive, not pro-active• Develop models of normal usage• Detect anomalies
-
IDS/IPS: Gaps, Research Challenges and Solutions16
Detection ApproachesDetection Approaches
• Misuse Detection: examine network/system activity for known misuses.– rule based– signature based
• Anomaly Detection: compare network/system activity against profile of normal behavior.
Strength:Fewer false positives
Strength:Can detect unknown attacks
Problem:Detects only known attacks
Problem:Hard to define normal behaviorLarge number of false positives
-
IDS/IPS: Gaps, Research Challenges and Solutions17
Intrusion Detection SystemsIntrusion Detection Systems
• Host-based– Parses audit logs for evidence of suspicious or
malicious activity– Monitors key system files for tampering
• Network-based– Deployed in front of firewalls, key points– Watches live packets; looks for attacks, misuse,
anomalies
-
IDS/IPS: Gaps, Research Challenges and Solutions18
Lessen learned so far?Lessen learned so far?
• Network security is a continuous process:Step 1: SecureStep 2: MonitorStep 3: TestStep 4: Improve
• Problems with malicious activity are increasing• Research must be focused to keep up with the
problems• What are we doing?
– Work to improve IDS through better technology– Create partnership with government, industry, etc.
Secure
Monitor
Test
Improve Security Policy
-
IDS/IPS: Gaps, Research Challenges and Solutions19
Lessen learned so far?Lessen learned so far?
• Improve IDS through better technology …• Because:
– Problems with malicious activity are increasing– Products are available to solve some of the problems– Research must be focused to keep up with and
eventually get ahead of problems– Partnership among government, industry, and
academia is the solution
-
Quick Overview of QRadarQuick Overview of QRadar
-
IDS/IPS: Gaps, Research Challenges and Solutions21
QRadarQRadar
• A Network Security Management platform• Key Features
– Auto-discovery of log and flow generators– Application layer network knowledge– Native creation of asset profiles, integration of VA data– Auto-tuning– Security event correlation capabilities
-
IDS/IPS: Gaps, Research Challenges and Solutions22
QRadar : Basic ComponentsQRadar : Basic Components
Surveillance
Analysis
Control
• Network data (NetFlow, JFlow, ...) • Security Events and Logs• Vulnerability Info
• Anomaly Detection• Misuse Detection• Event Correlation• Risk Analysis
• Automatic Offence Resolution• Flexible Reports
-
IDS/IPS: Gaps, Research Challenges and Solutions23
Stealthy activity
Worm activity
Diverse Reports addressing Internal Diverse Reports addressing Internal And External RisksAnd External Risks
-
IDS/IPS: Gaps, Research Challenges and Solutions24
Unified Security and Network ViewUnified Security and Network View
-
IDS/IPS: Gaps, Research Challenges and Solutions25
CustomersCustomers
• Harvard University uses QRadar for:– Real-time analysis of threats on the network
• Hartford uses QRadar to:– Monitor application use policies within the network– Support network and security compliance reporting
efforts for internal and external auditors
• NASA uses QRadar to:– Investigate events among large sets of network data– Implement detection and remediation capabilities in
existing network infrastructure
-
SpecificationSpecification--basedbasedDetectionDetection
-
IDS/IPS: Gaps, Research Challenges and Solutions27
IntroductionIntroduction
• Three avenues of research for intrusion
detection
– Misuse detection
– Specification-based detection
– Anomaly Detection
Misuse Detection
Rules
Input
Match
Input
AnomalyDetection
Normal Model
Machine Learning Techniques
DeviationDeviationSpecification-basedDetection
Specifications(Protocols, Policies, …)
Expert Knowledge
Input
-
IDS/IPS: Gaps, Research Challenges and Solutions28
SpecificationSpecification--based Detectionbased Detection
-
IDS/IPS: Gaps, Research Challenges and Solutions29
SpecificationSpecification--based Detectionbased Detection
• Using system behavioral specifications to detect attacks– Middle ground between misuse detectors and anomaly
detectors
• Design Principles:– Expert knowledge of a system or a network allowed
behavior is used; no learning; no threshold-based rules
– Least privilege principle (default deny principle)
-
IDS/IPS: Gaps, Research Challenges and Solutions30
ChallengesChallenges
Inability to accurately specify the legitimate behavior:
• Limiting the operations to a subset of what is assumed to be legitimate
• Generating false positives
-
IDS/IPS: Gaps, Research Challenges and Solutions31
Specification of Network ProtocolsSpecification of Network Protocols
• Specification of each layer of protocol stack can be used to model the expected behavior of the machines that communicate over a network.
• RFCs 791 and 793 are used as the reference specifications for different implementations of IP and TCP protocols, respectively.
• Some previous research works have used state machines to model the allowed behavior of protocol implementations.
-
IDS/IPS: Gaps, Research Challenges and Solutions32
IPIP--level Detectorlevel Detector
• In general, IP can be seen as a stateless specification.
• The only part implemented as a state machine is the segmentation-reassembly verification.
– Sanity checking for all current segmentation-reassembly operations from the viewpoint of a third-party observer.
• The RFC imposes (explicitly or implicitly) restrictions on the values of some network features.
-
IDS/IPS: Gaps, Research Challenges and Solutions33
Categories and Examples ofCategories and Examples ofFeature Value RestrictionsFeature Value Restrictions
• Reserved fields– IP header padding that must be zero
• Valid range, or set of values– Protocol field can get only values defined in RFC 790– The minimum (normal) value for IHL is 5
• Consistency of values of different features– Source and destination addresses cannot be the same
-
IDS/IPS: Gaps, Research Challenges and Solutions34
Categories and Examples ofCategories and Examples ofFeature Value RestrictionsFeature Value Restrictions
• Consistency of feature values and the actual data– The value of IHL (Internet Header Length) and Total
Length must match the actual length of the received datagram
• Consistency of feature values and the current state– The combination of Fragment Offset and length features
should not indicate that the current datagram has overlap with the existing ones
-
IDS/IPS: Gaps, Research Challenges and Solutions35
Implementation and ResultsImplementation and Results
• The implemented specification-based detector performs a wide range of sanity checking for each packet, looking for any violation of IP specification.– IP header fields– IP options
• The implemented detector is tested to detect some IP-level known attacks.– Ping of death– Teardrop
• The detector also checks for the rate of failedreassembly processes.
-
IDS/IPS: Gaps, Research Challenges and Solutions36
TCPTCP--level Detectorlevel Detector
• TCP is a connection-oriented protocol (RFC 793)• TCP protocol is the target of a large number of
attacks– Finger-printing– SYN-flood– ICMP Ping Flood
• Some weaknesses of TCP protocol– Lack of handling abnormal packets or state transitions– Absence of timers on some states (e.g., CLOSE_WAIT)
the state of the connections is traced by TCP state machine (initially proposed in RFC 793) that is implemented by TCP stacks in different OS’s.
-
IDS/IPS: Gaps, Research Challenges and Solutions37
TCPTCP--level Detectorlevel Detector
Normal transitionsSpecial Transitions
-
IDS/IPS: Gaps, Research Challenges and Solutions38
TCPTCP--level Detectorlevel Detector
• The detection is based on three types of checks
1. Suspicious state transitions
HostA HostB
No Connection
Connection Initiated
Half OpenNo
Connection
SYN
SYN,
ACK
SYN
SYN, ACK
TimeOut2
High frequency of this arc might be indicator of SYN-Flood attack
-
IDS/IPS: Gaps, Research Challenges and Solutions39
TCPTCP--level Detectorlevel Detector
• The detection is based on three types of checks1. Suspicious state transitions2. Bad state transitions3. TCP protocol legitimacy (imposed by RFC 793)
• !=
• Header Length >= 20 bytes.• If packet has the URG flag set, then it should have
data.
In every state, all the input packets which don’t have a corresponding output arc from the current state will result in bad state transition. (e.g., In the initial state, we do not expect to see any packet whose FIN or ACK flags are set)
-
Automatic Rule TuningAutomatic Rule Tuning
-
IDS/IPS: Gaps, Research Challenges and Solutions41
IntroductionIntroduction
• “IF-THEN” rules are widely used in SIM (Security Information Management) systems for different purposes– IDS : Detecting malicious behaviors– Firewall : Accepting/denying connections
• Condition part• Any logical combination of tests• Each test is based on one feature of the system (e.g.,
# of authentication failures, # of probes) • Action part
– Application dependent (accept, deny, log, …)
-
IDS/IPS: Gaps, Research Challenges and Solutions42
RulesRules
• Correlation rules
Custom Rule Engine
Rule1Rule2…Rulek
Raw Events Alerts
Condition part : # of authentication failures with the same IP across more than 3 destination IP(s) within 10 minutes exceed 4
Sample event correlation rule
-
IDS/IPS: Gaps, Research Challenges and Solutions43
Rules and TestsRules and Tests
• Other examples of tests– Number of Reconnaissance and Suspicious
activities from a source to a destination within 5 minutes exceeds 10
– Number of Scan events followed by Exploitfrom a single IP address against severaldestination addresses exceeds 5
-
IDS/IPS: Gaps, Research Challenges and Solutions44
What is the problem?What is the problem?
• Rules are both time-dependent and deployment-dependent
• Two aspects of adaptation– Test-level– Threshold-level
• Difficulties with manual tuning
-
IDS/IPS: Gaps, Research Challenges and Solutions45
Our ApproachOur Approach
• Three key components of our method– Abstracting the rule
– Moving Average-based Tuning • Automatic
– Deviation-based Tuning• Semi-Automatic
•If the # Authentication failures is more than 3 in 5 minutes
•If the # Authentication failures is more than k* SD in 5 minutes
K = Deviation Factor
SD = Standard Deviation
Current Threshold
-
IDS/IPS: Gaps, Research Challenges and Solutions46
Automatic Rule Tuner (ART)Automatic Rule Tuner (ART)
Moving Average
Input Events
Administrator Feedback
AlertsAttribute1(μ, σ) Attribute2(μ,σ) …
Rule Engine
Attributen(μ,σ)
RT2RT1…
RTn
-
IDS/IPS: Gaps, Research Challenges and Solutions47
Moving AverageMoving Average--based Tuningbased Tuning
• Update rules in the Moving Average
– Mean
– Standard Deviation :
11
1 ++×
= ++ NXNXX NNN
222 ))(()())(()( XEXEXEXEXVar −=−=
-
IDS/IPS: Gaps, Research Challenges and Solutions48
Moving Average
Input Events
Administrator Feedback
AlertsAttribute1(μ, σ) Attribute2(μ,σ) …
Rule Engine
Attributen(μ,σ)
RT2RT1…
RTn
Internal design of ARTInternal design of ART
Rule Tuner
Rule Logic
Classifier+ / -
+ / -
Input Events
Administrator Feedback
Decision Function
Update + / -
An ART is allocated for each rule of the system
-
IDS/IPS: Gaps, Research Challenges and Solutions49
ClassifierClassifier
• What is the role of classifier?• When does it trigger?• What is the reaction of the system?
– Statenew = classification_mode– Increase the weights of learning for input feedbacks
from administrator– Choose the output of classifier as the output of the
rule(+/-)– Update the rule logic when the false feedbacks ratio are
decreased enough
when the deviation factor of a threshold should be changed (e.g. if it should be changed to 3.5*SD from the previous value of 3:SD)
when the number of false feedbacks given by the administrator increases.
-
IDS/IPS: Gaps, Research Challenges and Solutions50
ClassifierClassifier
• What is the point of a hybrid solution?
– Classifier update is dependent on the administrator feedback
– Moving average-based tuning are more frequent than deviation-based tuning
-
IDS/IPS: Gaps, Research Challenges and Solutions51
ClassifierClassifier
• Desired Characteristics of the classification method– Efficient incremental learning– Ability of extracting current thresholds (decision
boundaries)• Candidate : Support Vector Machines
-
IDS/IPS: Gaps, Research Challenges and Solutions52
Administrator FeedbackAdministrator Feedback
• How can the administrator give a false negative feedback?
N L H
Low Confidence Alerts
High confidence Alerts
Current Threshold
-
IDS/IPS: Gaps, Research Challenges and Solutions53
Challenges and Future WorksChallenges and Future Works
• Decreasing the cost of classifiers• Test-level adaptation
– Test pruning– Adding a new test
• Implementation and test in a real-world network
-
Event CorrelationEvent Correlation
-
IDS/IPS: Gaps, Research Challenges and Solutions55
Problem StatementProblem Statement
• Managing a large volume of events is time consuming and labor intensive
• QRadar : 3000 events/sec for an average network
• Multi-stage attacks
• Large number of false positive or normal events
• Similar root causes
Probing Compromising Hacking Covering Escalating
-
IDS/IPS: Gaps, Research Challenges and Solutions56
Alert CorrelationAlert Correlation
• Definition– “Alert correlation a process that analyzes the alerts produced by one or more
intrusion detection systems and provides a more succinct and high-level view of occurring or attempted intrusions. Even though the correlation process is often presented as a single step, the analysis is actually carried out by a number of components”
- Valeur et al., 2005
• Goals– Reduces Alert volume– Eliminates Redundancies– Presents higher proportion of relevant information– Learns Attack Strategies
-
IDS/IPS: Gaps, Research Challenges and Solutions57
Some Correlation TechniquesSome Correlation Techniques
• Based on feature similarity• Alfonso Valdes and Keith Skinner
• Based on known scenarios• STATL, LAMDBA
• Based on precondition/post-condition relationship
• Ning et al., JIGSAW and MIRADOR
• Based on graph theory & neural nets• Bin Zhu and Ali Ghorbani
-
IDS/IPS: Gaps, Research Challenges and Solutions58
Different aspects of correlationDifferent aspects of correlation
• Aggregation
• Correlation
• False Positive Analysis
• Prioritization
A B
A
AA
A
-
IDS/IPS: Gaps, Research Challenges and Solutions59
Application in QRadarApplication in QRadar
Raw Events
Rule1
Rule2
…
Rulen
Alerts
Custom Rule Engine
Sample RuleIf Scan event is followed by Exploit from a single source IP against several hosts
-
IDS/IPS: Gaps, Research Challenges and Solutions60
Rule Wizard in QRadarRule Wizard in QRadar
-
IDS/IPS: Gaps, Research Challenges and Solutions61
What is missing?What is missing?
• Limitations of current design
– Hard-coding of correlation rules
– Unable to discover new types of correlations
• Automatic Rule Discovery
-
IDS/IPS: Gaps, Research Challenges and Solutions62
Our MethodologyOur Methodology
• Discovering interesting event patterns using data mining techniques
• Patterns of Interest– Burst Patterns– Partially Periodic Patterns– Similar Patterns– Causality Patterns
-
IDS/IPS: Gaps, Research Challenges and Solutions63
Apriori AlgorithmApriori Algorithm
ABCD
ABC ABD ACD BCD
AB AC BC AD BD CD
A B C D
{}
Apriori
1-itemset
2-itemset
3-itemset
4-itemset
Downward closure propertyA pattern is not a candidate unless all of its subsets are frequent
-
IDS/IPS: Gaps, Research Challenges and Solutions64
Burst EventsBurst Events
• Burst events : – Events or group of events that are issued with
abnormal rates
EventAEventB
-
IDS/IPS: Gaps, Research Challenges and Solutions65
Burst EventsBurst Events
• Mining Burst Events:
Basket Events Description1 A, B, C
A, BA,B, DA
…
Events with odd rates in this time window2 …3 …4 …
…
-
IDS/IPS: Gaps, Research Challenges and Solutions66
Repetitive PatternsRepetitive Patterns
– Partially periodic patterns for one event (P, δ, minsup)• P : period• δ : tolerance• Minsup : minimum support
-
IDS/IPS: Gaps, Research Challenges and Solutions67
Partial Periodic Temporal AssociationPartial Periodic Temporal Association
• P-Patterns (P, δ, minsup, W)– P : period– δ : tolerance– Minsup : minimum support– Temporal Association
Off OnOn
Event groups with temporal association W
∆t < P + δ∆t < W
-
IDS/IPS: Gaps, Research Challenges and Solutions68
Partial Periodic Temporal AssociationPartial Periodic Temporal Association
P-Pattern Mining Algorithm1. Determine possible periods for each event type
Chi-Square test is used as a measure of randomness of an event type given event time series
2. Mine temporal patterns with each period found in step 1 using Apriori algorithm
Application– Useful in detecting both normal and malicious behaviors
-
IDS/IPS: Gaps, Research Challenges and Solutions69
Causality PatternsCausality Patterns
• Basic Format of Episode Rules
• Sample episode rule:
A [W1] -> B [W2] (Support, Confidence)A : (A1,A2,…Am) -> B : (B1,B2,…Bn),
A is a sub-episode of BAi and Bi are predicates on event types (event type most often)
If an event of type A is followed by an event of type B within t1 seconds, an event of type C is seen within t2 seconds with probability 0.95
-
IDS/IPS: Gaps, Research Challenges and Solutions70
Multiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern
• MAF-Patterns– (n, G, S) : pattern signature or mining camp
• n : length of the pattern• G : grouping attributes• S : Itemizing attributes
Example: The event types probe and exploit tend to be generated from the same host and within the same minute.
n = 2, G = {Event Source, Interval}S = {Event Type}Mining camp : (2, {Event Source, Interval}, {Event Type})
-
IDS/IPS: Gaps, Research Challenges and Solutions71
Multiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern
• MAF-Pattern mining– Finding all patterns with signature (n, G, S) with more
than sup_thres support (given by the user)• Scalability is a real challenge
– All combination of attributes are possible as G and S• Solution:
– downward closure property
c ofsuccessor 3- type theis }){AS G, (n, -c ofsuccessor 2- type theis S) },{AG (n, -
c ofsuccessor 1- type theis S) G, 1,(n-
i
i
∪∪
+
c = (n, G, S)
Ai is a new attribute
-
IDS/IPS: Gaps, Research Challenges and Solutions72
Search Space for MAFSearch Space for MAF
(1, {T}, {A}) (1, {T}, {B})
(1, {T}, {})
(2, {T}, {A}) (1, {TB}, {A}) (1, {T}, {AB}) (1, {TA}, {B}) (2, {T}, {B})
(3, {T}, {A}) (2, {TB}, {A}) (2, {T}, {AB}) (2, {TA}, {B}) (3, {T}, {B})
(4, {T}, {A}) (3, {TB}, {A}) (3, {T}, {AB}) (3, {TA}, {B}) (4, {T}, {B})
-
IDS/IPS: Gaps, Research Challenges and Solutions73
Multiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern
• Sample MAF patterns– (n, {Interval}, {Host})
– (n, {Interval, Event Type}, {Host})
• Application of MAF pattern mining– Finding similar patterns– Might be useful for causality and false positive
analysis
-
IDS/IPS: Gaps, Research Challenges and Solutions74
Alerts
Events
Integration with QRadarIntegration with QRadar
Filtering RulesMalicious
Correlation Rules
Offline Pattern MinerPattern Analysis
& Rule Discovery
Event DB
Custom Rule Engine Similar Patterns
Causality Patterns
Event Bursts
Periodic Patterns
Raw Events
-
Hardware based Network Hardware based Network Intrusion DetectionIntrusion Detection
-
IDS/IPS: Gaps, Research Challenges and Solutions76
IntroductionIntroduction
• A major problem in NIDS is the speed of operation .
• Software based NIDS are not able to work at line speed, – especially when deployed on high speed links .
• In signature based NIDS, the problem is more significant– occasionally whole packet payload is to be inspected
to detect a known signature pattern.
-
IDS/IPS: Gaps, Research Challenges and Solutions77
Signature based NIDSSignature based NIDS
• Signature based intrusion detection is one of the commonly used and yet accurate methods of intrusion detection
– The attack pattern is studied
– The signature is extracted
– A most common NIDS that uses signature based approach is SNORT
-
IDS/IPS: Gaps, Research Challenges and Solutions78
NIDS Classification Based on Hardware NIDS Classification Based on Hardware PlatformsPlatforms
Reconfigurable Discrete Logic(RDL)
Reconfigurable Discrete Logic(RDL)
Static Random Access memories(SRAM)
Static Random Access memories(SRAM)
Ternary Content Addressable Memories(TCAM)
Ternary Content Addressable Memories(TCAM)
Network Processors(NP)
Network Processors(NP)
-
IDS/IPS: Gaps, Research Challenges and Solutions79
RDL: Reconfigurable Discrete LogicRDL: Reconfigurable Discrete Logic
• Implements Snort Rules in hardwired logic
• Usually implemented in FPGA
• Need hardware reconfiguration for updating rules.
-
IDS/IPS: Gaps, Research Challenges and Solutions80
RDL: Reconfigurable Discrete Logic RDL: Reconfigurable Discrete Logic exampleexample
-
IDS/IPS: Gaps, Research Challenges and Solutions81
RDL: Reconfigurable Discrete LogicRDL: Reconfigurable Discrete Logic
• Pros– Very high throughput– FPGAs are the fastest and probably cheapest way to
market– Rules can be reconfigured
• Cons– Rule update needs reconfiguration (new HDL, compilation,
reprogramming)– Compilation sometimes needs days– Not suitable for automatic rule tuning – Large rule-sets may not fit in FPGA
-
IDS/IPS: Gaps, Research Challenges and Solutions82
SRAM based NIDS exampleSRAM based NIDS example
• SRAM based architectures typically use Hashing.
-
IDS/IPS: Gaps, Research Challenges and Solutions83
SRAM based NIDS exampleSRAM based NIDS example
-
IDS/IPS: Gaps, Research Challenges and Solutions84
SRAMSRAM
• Pros– Updating rule does not require hardware reconfiguration– Use of hashing may result in single memory access to detect
a match– Less hardwired logic needed
• Cons– Need hash logic without hash collision, hash collisions may
result in multiple memory access– Hash logic may need hardware update if exact hashing is
used to update rules– Hard to detect multiple patterns simultaneously– Need to perform hashing for different input string sizes
-
IDS/IPS: Gaps, Research Challenges and Solutions85
Ternary CAMTernary CAM
• TCAM stores three logic values ‘1’, ‘0’ and ‘x’
• In case of multiple matches, only the lowest indexed output is selected.
• Need extra logic for content matching.
-
IDS/IPS: Gaps, Research Challenges and Solutions86
Ternary CAM exampleTernary CAM example
k bytes> 1K
entries
192.128.101.100
168.100..xxx.xxx
192.128.xxx.xxx
Match192.128.101.xxx
Input
TCAM
-
IDS/IPS: Gaps, Research Challenges and Solutions87
Ternary CAMTernary CAM
• Pros– Rule update does not need hardware reconfiguration– Single memory access is needed to find a match
• Cons– Large TCAMs need a lot of extra comparison logic– High access time as compared to SRAM– Expensive– Hard to detect multiple patterns simultaneously– Pattern length is constrained by TCAM width.
-
IDS/IPS: Gaps, Research Challenges and Solutions88
Network Processor based NIDS Network Processor based NIDS exampleexample
-
IDS/IPS: Gaps, Research Challenges and Solutions89
Network ProcessorsNetwork Processors
• Pros– Easier to implement, and fast time to market.– Parallel computation with micro engines and StrongARM.– Complete NIDS can be implemented and not just pattern
matching.
• Cons– Normally needs extra logic for pattern matching to catch up
with line speed.
-
IDS/IPS: Gaps, Research Challenges and Solutions90
ConclusionConclusion
• Improvement in hardware based NIDS is still needed
• Approaches without the need for hardware reconfiguration are most considered
• We are considering approaches based on PFGA and bloom filters
-
QUESTIONS?QUESTIONS?
Intrusion Detection/Prevention Systems Gaps, Research Challenges and Solutions AgendaOverviewOverviewResearch Groups�ias.cs.unb.ca ; nsl.cs.unb.caComputer SecuritySecure SystemThreat and IntrusionPrevent Attacks!Attack vs. KnowledgeThreat Evolution: Malicious CodeThe State of Network SecurityThe Speed of Attacks AcceleratesIntrusionsHow about Detecting IntrusionsDetection ApproachesIntrusion Detection SystemsLessen learned so far?Lessen learned so far?Quick Overview of QRadarQRadarQRadar : Basic ComponentsCustomersSpecification-based�DetectionIntroductionSpecification-based DetectionSpecification-based DetectionChallengesSpecification of Network ProtocolsIP-level DetectorCategories and Examples of�Feature Value RestrictionsCategories and Examples of�Feature Value RestrictionsImplementation and ResultsTCP-level DetectorTCP-level DetectorTCP-level DetectorTCP-level DetectorAutomatic Rule TuningIntroductionRulesRules and TestsWhat is the problem?Our ApproachAutomatic Rule Tuner (ART)Moving Average-based TuningInternal design of ARTClassifierClassifierClassifierAdministrator FeedbackChallenges and Future WorksEvent CorrelationProblem StatementAlert CorrelationSome Correlation TechniquesDifferent aspects of correlationApplication in QRadarRule Wizard in QRadarWhat is missing?Our MethodologyApriori AlgorithmBurst EventsBurst EventsRepetitive PatternsPartial Periodic Temporal AssociationPartial Periodic Temporal AssociationCausality PatternsMultiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern Search Space for MAFMultiple Attribute Frequent Pattern Integration with QRadarHardware based Network �Intrusion DetectionIntroductionSignature based NIDSNIDS Classification Based on Hardware PlatformsRDL: Reconfigurable Discrete LogicRDL: Reconfigurable Discrete Logic exampleRDL: Reconfigurable Discrete LogicSRAM based NIDS exampleSRAM based NIDS exampleSRAMTernary CAMTernary CAM exampleTernary CAMNetwork Processor based NIDS exampleNetwork ProcessorsConclusionQUESTIONS?