intrusion detection/prevention systems · 2020. 2. 22. · – intrusion detection systems –...

91
Intrusion Detection/Prevention Systems Intrusion Detection/Prevention Systems Gaps, Research Challenges and Solutions Gaps, Research Challenges and Solutions Ali A. Ghorbani Ali A. Ghorbani

Upload: others

Post on 10-Feb-2021

8 views

Category:

Documents


0 download

TRANSCRIPT

  • Intrusion Detection/Prevention SystemsIntrusion Detection/Prevention SystemsGaps, Research Challenges and SolutionsGaps, Research Challenges and Solutions

    Ali A. GhorbaniAli A. Ghorbani

  • IDS/IPS: Gaps, Research Challenges and Solutions2

    AgendaAgenda

    • Introduction• Network Security

    – Intrusion Detection Systems– Lessen learned so far

    • QRadar• Research Challenges and Solutions

    – Automatically detecting anomalies in flow or event streams– Automated Rules Tuning, Learning and Adaptation– Data visualization – Hardware event and flow correlation– Automatic discovery of network applications

    • Conclusions & Future Work

  • IDS/IPS: Gaps, Research Challenges and Solutions3

    OverviewOverview

    • University of New Brunswick• Established in 1790• Oldest English-speaking University in Canada

  • IDS/IPS: Gaps, Research Challenges and Solutions4

    OverviewOverview

    • Faculty of Computer Science• First Faculty of Computer Science in Canada

    – Artificial Intelligence– Bioinformatics– Computational Geometry– Hardware– Hardware/Software Co-design– Parallel/Distributed Processing– Network and Information Security– Software Engineering – Wireless/Mobile computing

  • IDS/IPS: Gaps, Research Challenges and Solutions5

    Research GroupsResearch Groupsias.cs.unb.caias.cs.unb.ca ; ; nsl.cs.unb.cansl.cs.unb.ca

    • Intelligent and Adaptive Systems (IAS) – Machine Learning– Intelligent Agents and Multiagent Systems– Web Intelligence & Adaptive Web Systems– Privacy, Security and Trust

    • Network Security– All aspects of Network Security– Application of Artificial Intelligence to Network Security

    • Critical Infrastructures Protection– Agent-based Interdependencies Modelling and

    Simulation (AIMS)

  • IDS/IPS: Gaps, Research Challenges and Solutions6

    Computer Computer SecuritySecurity

    SecuritySecuritySecure SystemsSecure SystemsBreach:Interception

    Confidentiality

    Assets are accessible only by authorized parties

    Availability

    Assets are accessible to authorized parties at appropriate times

    Breach:Interruption

    Breach:Modificationfabrication

    Integrity

    Assets can be modified only by authorized parties

    Secure

  • IDS/IPS: Gaps, Research Challenges and Solutions7

    Secure SystemSecure System

    Authentication

    Signature

    Availability

    ConfidentialityData Integrity

    Non-Repudiation

    Access Control

    • Robust Protocols• Access controls• All the normal anti-virus,

    anti-intrusion, anti-spam, anti-scam techniques

    • Robustness - security implies availability– deny DOS – minimize false positive

  • IDS/IPS: Gaps, Research Challenges and Solutions8

    Threat and IntrusionThreat and Intrusion

    • Threat: A potential violation of security

    – Flaws in design, implementation and operation

    • Intrusion/Attack: Unauthorized access to a network or computer system that could compromise CIA !

  • IDS/IPS: Gaps, Research Challenges and Solutions9

    Prevent Attacks!Prevent Attacks!

    • Wishful hope!• Completely secure systems still a dream• New features and software new vulnerabilities• It is hard to get ride of existing problems

    – TCP/IP, OS, etc.• New systems contain old vulnerabilities• Attack sophistication vs. Intruder Knowledge

  • IDS/IPS: Gaps, Research Challenges and Solutions10

    Attack vs. KnowledgeAttack vs. Knowledge

  • IDS/IPS: Gaps, Research Challenges and Solutions11

    Threat Evolution: Malicious CodeThreat Evolution: Malicious Code

    “Sasser”

    Human response impossibleAutomated response required,i.e., automated remediation and automated attribution

    Seconds

    Minutes

    Hours

    Days

    Weeks or months

    Time

    Human response impossibleAutomated response unlikelyProactive blocking possible

    Human response difficult/impossibleAutomated response possible

    Human response possible

    File Viruses

    MacroViruses

    e-mailWorms

    Blendedthreats

    “Warhol”threats

    “Flash”Threats,

    i.e., Sasser

    Boot, ComInfectors

    Early 1990s Mid 1990s Late 1990s 2000 2003 2004

    Adopted from McAfee

    combine the characteristics of viruses, worms, Trojan Horses, and malicious code with serverand Internet vulnerabilities to initiate, transmit,

    and spread an attack.

    Through advanced scanning, Warhol worms would first start an infection using a list of about 50,000 sites, and then use coordinated scanning techniques to infect the rest of the Internet.

    http://www.webopedia.com/TERM/B/virus.htmlhttp://www.webopedia.com/TERM/B/virus.htmlhttp://www.webopedia.com/TERM/B/worm.htmlhttp://www.webopedia.com/TERM/B/Trojan_horse.htmlhttp://www.webopedia.com/TERM/B/Trojan_horse.htmlhttp://www.webopedia.com/TERM/B/server.htmlhttp://www.webopedia.com/TERM/B/server.htmlhttp://www.webopedia.com/TERM/B/Internet.html

  • IDS/IPS: Gaps, Research Challenges and Solutions12

    The State of Network Security

    • Not good at all! • Why?

    – Vulnerabilities exist at level of OSI layers– Very few sysadmins check in real-time– High false alarm rates/Low detection rates– Does not differ by network or team size– Sysadmins feel poorly about current tools– Network incidents are increasing– The speed of attacks accelerates

  • IDS/IPS: Gaps, Research Challenges and Solutions13

    The Speed of Attacks AcceleratesThe Speed of Attacks Accelerates

    • SQL Slammer:• Blended threat exploits known vulnerability• Payload was single 404 byte UDP packet• Doubled every 8.5 seconds• Achieved full scanning rate (over 55 million scans per

    second) after approximately 3 minutes• Infected 90% vulnerable hosts worldwide within 10 minutes

    Adopted from McAfee

  • IDS/IPS: Gaps, Research Challenges and Solutions14

    IntrusionsIntrusions

    • Misuse of protocols• Misuse of service ports• DoS based on crafted

    payloads• Bandwidth or Flood

    attacks– ICMP echo request Flood– TCP data segment Flood– TCP SYN/RST Flood– TCP SYN Floods– TCP, UDP, ICMP floods

    • Buffer Overflows

    • Protocol Attacks– SYN Flood– ICMP echo reply flood– UDP Flood

    • Protocol Tunneling• Backdoor Intrusions• Low-bandwidth

    DoS/DDOS attacks• Logic Attacks

    – Land attack– Ping of Death– Teardrop

    HTTP traffic on a non-standard port, say port 53

    Backdoor service on well-known standard port, e.g., peer-to-peer file sharing using Gnutella on port 80

    process table exhaustion, IP stack crashing, or a Web application .soft spot

  • IDS/IPS: Gaps, Research Challenges and Solutions15

    How about Detecting IntrusionsHow about Detecting Intrusions

    • Intrusions detected through Audit of LogsBut audit logs can get very large!

    • Reactive, not pro-active• Develop models of normal usage• Detect anomalies

  • IDS/IPS: Gaps, Research Challenges and Solutions16

    Detection ApproachesDetection Approaches

    • Misuse Detection: examine network/system activity for known misuses.– rule based– signature based

    • Anomaly Detection: compare network/system activity against profile of normal behavior.

    Strength:Fewer false positives

    Strength:Can detect unknown attacks

    Problem:Detects only known attacks

    Problem:Hard to define normal behaviorLarge number of false positives

  • IDS/IPS: Gaps, Research Challenges and Solutions17

    Intrusion Detection SystemsIntrusion Detection Systems

    • Host-based– Parses audit logs for evidence of suspicious or

    malicious activity– Monitors key system files for tampering

    • Network-based– Deployed in front of firewalls, key points– Watches live packets; looks for attacks, misuse,

    anomalies

  • IDS/IPS: Gaps, Research Challenges and Solutions18

    Lessen learned so far?Lessen learned so far?

    • Network security is a continuous process:Step 1: SecureStep 2: MonitorStep 3: TestStep 4: Improve

    • Problems with malicious activity are increasing• Research must be focused to keep up with the

    problems• What are we doing?

    – Work to improve IDS through better technology– Create partnership with government, industry, etc.

    Secure

    Monitor

    Test

    Improve Security Policy

  • IDS/IPS: Gaps, Research Challenges and Solutions19

    Lessen learned so far?Lessen learned so far?

    • Improve IDS through better technology …• Because:

    – Problems with malicious activity are increasing– Products are available to solve some of the problems– Research must be focused to keep up with and

    eventually get ahead of problems– Partnership among government, industry, and

    academia is the solution

  • Quick Overview of QRadarQuick Overview of QRadar

  • IDS/IPS: Gaps, Research Challenges and Solutions21

    QRadarQRadar

    • A Network Security Management platform• Key Features

    – Auto-discovery of log and flow generators– Application layer network knowledge– Native creation of asset profiles, integration of VA data– Auto-tuning– Security event correlation capabilities

  • IDS/IPS: Gaps, Research Challenges and Solutions22

    QRadar : Basic ComponentsQRadar : Basic Components

    Surveillance

    Analysis

    Control

    • Network data (NetFlow, JFlow, ...) • Security Events and Logs• Vulnerability Info

    • Anomaly Detection• Misuse Detection• Event Correlation• Risk Analysis

    • Automatic Offence Resolution• Flexible Reports

  • IDS/IPS: Gaps, Research Challenges and Solutions23

    Stealthy activity

    Worm activity

    Diverse Reports addressing Internal Diverse Reports addressing Internal And External RisksAnd External Risks

  • IDS/IPS: Gaps, Research Challenges and Solutions24

    Unified Security and Network ViewUnified Security and Network View

  • IDS/IPS: Gaps, Research Challenges and Solutions25

    CustomersCustomers

    • Harvard University uses QRadar for:– Real-time analysis of threats on the network

    • Hartford uses QRadar to:– Monitor application use policies within the network– Support network and security compliance reporting

    efforts for internal and external auditors

    • NASA uses QRadar to:– Investigate events among large sets of network data– Implement detection and remediation capabilities in

    existing network infrastructure

  • SpecificationSpecification--basedbasedDetectionDetection

  • IDS/IPS: Gaps, Research Challenges and Solutions27

    IntroductionIntroduction

    • Three avenues of research for intrusion

    detection

    – Misuse detection

    – Specification-based detection

    – Anomaly Detection

    Misuse Detection

    Rules

    Input

    Match

    Input

    AnomalyDetection

    Normal Model

    Machine Learning Techniques

    DeviationDeviationSpecification-basedDetection

    Specifications(Protocols, Policies, …)

    Expert Knowledge

    Input

  • IDS/IPS: Gaps, Research Challenges and Solutions28

    SpecificationSpecification--based Detectionbased Detection

  • IDS/IPS: Gaps, Research Challenges and Solutions29

    SpecificationSpecification--based Detectionbased Detection

    • Using system behavioral specifications to detect attacks– Middle ground between misuse detectors and anomaly

    detectors

    • Design Principles:– Expert knowledge of a system or a network allowed

    behavior is used; no learning; no threshold-based rules

    – Least privilege principle (default deny principle)

  • IDS/IPS: Gaps, Research Challenges and Solutions30

    ChallengesChallenges

    Inability to accurately specify the legitimate behavior:

    • Limiting the operations to a subset of what is assumed to be legitimate

    • Generating false positives

  • IDS/IPS: Gaps, Research Challenges and Solutions31

    Specification of Network ProtocolsSpecification of Network Protocols

    • Specification of each layer of protocol stack can be used to model the expected behavior of the machines that communicate over a network.

    • RFCs 791 and 793 are used as the reference specifications for different implementations of IP and TCP protocols, respectively.

    • Some previous research works have used state machines to model the allowed behavior of protocol implementations.

  • IDS/IPS: Gaps, Research Challenges and Solutions32

    IPIP--level Detectorlevel Detector

    • In general, IP can be seen as a stateless specification.

    • The only part implemented as a state machine is the segmentation-reassembly verification.

    – Sanity checking for all current segmentation-reassembly operations from the viewpoint of a third-party observer.

    • The RFC imposes (explicitly or implicitly) restrictions on the values of some network features.

  • IDS/IPS: Gaps, Research Challenges and Solutions33

    Categories and Examples ofCategories and Examples ofFeature Value RestrictionsFeature Value Restrictions

    • Reserved fields– IP header padding that must be zero

    • Valid range, or set of values– Protocol field can get only values defined in RFC 790– The minimum (normal) value for IHL is 5

    • Consistency of values of different features– Source and destination addresses cannot be the same

  • IDS/IPS: Gaps, Research Challenges and Solutions34

    Categories and Examples ofCategories and Examples ofFeature Value RestrictionsFeature Value Restrictions

    • Consistency of feature values and the actual data– The value of IHL (Internet Header Length) and Total

    Length must match the actual length of the received datagram

    • Consistency of feature values and the current state– The combination of Fragment Offset and length features

    should not indicate that the current datagram has overlap with the existing ones

  • IDS/IPS: Gaps, Research Challenges and Solutions35

    Implementation and ResultsImplementation and Results

    • The implemented specification-based detector performs a wide range of sanity checking for each packet, looking for any violation of IP specification.– IP header fields– IP options

    • The implemented detector is tested to detect some IP-level known attacks.– Ping of death– Teardrop

    • The detector also checks for the rate of failedreassembly processes.

  • IDS/IPS: Gaps, Research Challenges and Solutions36

    TCPTCP--level Detectorlevel Detector

    • TCP is a connection-oriented protocol (RFC 793)• TCP protocol is the target of a large number of

    attacks– Finger-printing– SYN-flood– ICMP Ping Flood

    • Some weaknesses of TCP protocol– Lack of handling abnormal packets or state transitions– Absence of timers on some states (e.g., CLOSE_WAIT)

    the state of the connections is traced by TCP state machine (initially proposed in RFC 793) that is implemented by TCP stacks in different OS’s.

  • IDS/IPS: Gaps, Research Challenges and Solutions37

    TCPTCP--level Detectorlevel Detector

    Normal transitionsSpecial Transitions

  • IDS/IPS: Gaps, Research Challenges and Solutions38

    TCPTCP--level Detectorlevel Detector

    • The detection is based on three types of checks

    1. Suspicious state transitions

    HostA HostB

    No Connection

    Connection Initiated

    Half OpenNo

    Connection

    SYN

    SYN,

    ACK

    SYN

    SYN, ACK

    TimeOut2

    High frequency of this arc might be indicator of SYN-Flood attack

  • IDS/IPS: Gaps, Research Challenges and Solutions39

    TCPTCP--level Detectorlevel Detector

    • The detection is based on three types of checks1. Suspicious state transitions2. Bad state transitions3. TCP protocol legitimacy (imposed by RFC 793)

    • !=

    • Header Length >= 20 bytes.• If packet has the URG flag set, then it should have

    data.

    In every state, all the input packets which don’t have a corresponding output arc from the current state will result in bad state transition. (e.g., In the initial state, we do not expect to see any packet whose FIN or ACK flags are set)

  • Automatic Rule TuningAutomatic Rule Tuning

  • IDS/IPS: Gaps, Research Challenges and Solutions41

    IntroductionIntroduction

    • “IF-THEN” rules are widely used in SIM (Security Information Management) systems for different purposes– IDS : Detecting malicious behaviors– Firewall : Accepting/denying connections

    • Condition part• Any logical combination of tests• Each test is based on one feature of the system (e.g.,

    # of authentication failures, # of probes) • Action part

    – Application dependent (accept, deny, log, …)

  • IDS/IPS: Gaps, Research Challenges and Solutions42

    RulesRules

    • Correlation rules

    Custom Rule Engine

    Rule1Rule2…Rulek

    Raw Events Alerts

    Condition part : # of authentication failures with the same IP across more than 3 destination IP(s) within 10 minutes exceed 4

    Sample event correlation rule

  • IDS/IPS: Gaps, Research Challenges and Solutions43

    Rules and TestsRules and Tests

    • Other examples of tests– Number of Reconnaissance and Suspicious

    activities from a source to a destination within 5 minutes exceeds 10

    – Number of Scan events followed by Exploitfrom a single IP address against severaldestination addresses exceeds 5

  • IDS/IPS: Gaps, Research Challenges and Solutions44

    What is the problem?What is the problem?

    • Rules are both time-dependent and deployment-dependent

    • Two aspects of adaptation– Test-level– Threshold-level

    • Difficulties with manual tuning

  • IDS/IPS: Gaps, Research Challenges and Solutions45

    Our ApproachOur Approach

    • Three key components of our method– Abstracting the rule

    – Moving Average-based Tuning • Automatic

    – Deviation-based Tuning• Semi-Automatic

    •If the # Authentication failures is more than 3 in 5 minutes

    •If the # Authentication failures is more than k* SD in 5 minutes

    K = Deviation Factor

    SD = Standard Deviation

    Current Threshold

  • IDS/IPS: Gaps, Research Challenges and Solutions46

    Automatic Rule Tuner (ART)Automatic Rule Tuner (ART)

    Moving Average

    Input Events

    Administrator Feedback

    AlertsAttribute1(μ, σ) Attribute2(μ,σ) …

    Rule Engine

    Attributen(μ,σ)

    RT2RT1…

    RTn

  • IDS/IPS: Gaps, Research Challenges and Solutions47

    Moving AverageMoving Average--based Tuningbased Tuning

    • Update rules in the Moving Average

    – Mean

    – Standard Deviation :

    11

    1 ++×

    = ++ NXNXX NNN

    222 ))(()())(()( XEXEXEXEXVar −=−=

  • IDS/IPS: Gaps, Research Challenges and Solutions48

    Moving Average

    Input Events

    Administrator Feedback

    AlertsAttribute1(μ, σ) Attribute2(μ,σ) …

    Rule Engine

    Attributen(μ,σ)

    RT2RT1…

    RTn

    Internal design of ARTInternal design of ART

    Rule Tuner

    Rule Logic

    Classifier+ / -

    + / -

    Input Events

    Administrator Feedback

    Decision Function

    Update + / -

    An ART is allocated for each rule of the system

  • IDS/IPS: Gaps, Research Challenges and Solutions49

    ClassifierClassifier

    • What is the role of classifier?• When does it trigger?• What is the reaction of the system?

    – Statenew = classification_mode– Increase the weights of learning for input feedbacks

    from administrator– Choose the output of classifier as the output of the

    rule(+/-)– Update the rule logic when the false feedbacks ratio are

    decreased enough

    when the deviation factor of a threshold should be changed (e.g. if it should be changed to 3.5*SD from the previous value of 3:SD)

    when the number of false feedbacks given by the administrator increases.

  • IDS/IPS: Gaps, Research Challenges and Solutions50

    ClassifierClassifier

    • What is the point of a hybrid solution?

    – Classifier update is dependent on the administrator feedback

    – Moving average-based tuning are more frequent than deviation-based tuning

  • IDS/IPS: Gaps, Research Challenges and Solutions51

    ClassifierClassifier

    • Desired Characteristics of the classification method– Efficient incremental learning– Ability of extracting current thresholds (decision

    boundaries)• Candidate : Support Vector Machines

  • IDS/IPS: Gaps, Research Challenges and Solutions52

    Administrator FeedbackAdministrator Feedback

    • How can the administrator give a false negative feedback?

    N L H

    Low Confidence Alerts

    High confidence Alerts

    Current Threshold

  • IDS/IPS: Gaps, Research Challenges and Solutions53

    Challenges and Future WorksChallenges and Future Works

    • Decreasing the cost of classifiers• Test-level adaptation

    – Test pruning– Adding a new test

    • Implementation and test in a real-world network

  • Event CorrelationEvent Correlation

  • IDS/IPS: Gaps, Research Challenges and Solutions55

    Problem StatementProblem Statement

    • Managing a large volume of events is time consuming and labor intensive

    • QRadar : 3000 events/sec for an average network

    • Multi-stage attacks

    • Large number of false positive or normal events

    • Similar root causes

    Probing Compromising Hacking Covering Escalating

  • IDS/IPS: Gaps, Research Challenges and Solutions56

    Alert CorrelationAlert Correlation

    • Definition– “Alert correlation a process that analyzes the alerts produced by one or more

    intrusion detection systems and provides a more succinct and high-level view of occurring or attempted intrusions. Even though the correlation process is often presented as a single step, the analysis is actually carried out by a number of components”

    - Valeur et al., 2005

    • Goals– Reduces Alert volume– Eliminates Redundancies– Presents higher proportion of relevant information– Learns Attack Strategies

  • IDS/IPS: Gaps, Research Challenges and Solutions57

    Some Correlation TechniquesSome Correlation Techniques

    • Based on feature similarity• Alfonso Valdes and Keith Skinner

    • Based on known scenarios• STATL, LAMDBA

    • Based on precondition/post-condition relationship

    • Ning et al., JIGSAW and MIRADOR

    • Based on graph theory & neural nets• Bin Zhu and Ali Ghorbani

  • IDS/IPS: Gaps, Research Challenges and Solutions58

    Different aspects of correlationDifferent aspects of correlation

    • Aggregation

    • Correlation

    • False Positive Analysis

    • Prioritization

    A B

    A

    AA

    A

  • IDS/IPS: Gaps, Research Challenges and Solutions59

    Application in QRadarApplication in QRadar

    Raw Events

    Rule1

    Rule2

    Rulen

    Alerts

    Custom Rule Engine

    Sample RuleIf Scan event is followed by Exploit from a single source IP against several hosts

  • IDS/IPS: Gaps, Research Challenges and Solutions60

    Rule Wizard in QRadarRule Wizard in QRadar

  • IDS/IPS: Gaps, Research Challenges and Solutions61

    What is missing?What is missing?

    • Limitations of current design

    – Hard-coding of correlation rules

    – Unable to discover new types of correlations

    • Automatic Rule Discovery

  • IDS/IPS: Gaps, Research Challenges and Solutions62

    Our MethodologyOur Methodology

    • Discovering interesting event patterns using data mining techniques

    • Patterns of Interest– Burst Patterns– Partially Periodic Patterns– Similar Patterns– Causality Patterns

  • IDS/IPS: Gaps, Research Challenges and Solutions63

    Apriori AlgorithmApriori Algorithm

    ABCD

    ABC ABD ACD BCD

    AB AC BC AD BD CD

    A B C D

    {}

    Apriori

    1-itemset

    2-itemset

    3-itemset

    4-itemset

    Downward closure propertyA pattern is not a candidate unless all of its subsets are frequent

  • IDS/IPS: Gaps, Research Challenges and Solutions64

    Burst EventsBurst Events

    • Burst events : – Events or group of events that are issued with

    abnormal rates

    EventAEventB

  • IDS/IPS: Gaps, Research Challenges and Solutions65

    Burst EventsBurst Events

    • Mining Burst Events:

    Basket Events Description1 A, B, C

    A, BA,B, DA

    Events with odd rates in this time window2 …3 …4 …

  • IDS/IPS: Gaps, Research Challenges and Solutions66

    Repetitive PatternsRepetitive Patterns

    – Partially periodic patterns for one event (P, δ, minsup)• P : period• δ : tolerance• Minsup : minimum support

  • IDS/IPS: Gaps, Research Challenges and Solutions67

    Partial Periodic Temporal AssociationPartial Periodic Temporal Association

    • P-Patterns (P, δ, minsup, W)– P : period– δ : tolerance– Minsup : minimum support– Temporal Association

    Off OnOn

    Event groups with temporal association W

    ∆t < P + δ∆t < W

  • IDS/IPS: Gaps, Research Challenges and Solutions68

    Partial Periodic Temporal AssociationPartial Periodic Temporal Association

    P-Pattern Mining Algorithm1. Determine possible periods for each event type

    Chi-Square test is used as a measure of randomness of an event type given event time series

    2. Mine temporal patterns with each period found in step 1 using Apriori algorithm

    Application– Useful in detecting both normal and malicious behaviors

  • IDS/IPS: Gaps, Research Challenges and Solutions69

    Causality PatternsCausality Patterns

    • Basic Format of Episode Rules

    • Sample episode rule:

    A [W1] -> B [W2] (Support, Confidence)A : (A1,A2,…Am) -> B : (B1,B2,…Bn),

    A is a sub-episode of BAi and Bi are predicates on event types (event type most often)

    If an event of type A is followed by an event of type B within t1 seconds, an event of type C is seen within t2 seconds with probability 0.95

  • IDS/IPS: Gaps, Research Challenges and Solutions70

    Multiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern

    • MAF-Patterns– (n, G, S) : pattern signature or mining camp

    • n : length of the pattern• G : grouping attributes• S : Itemizing attributes

    Example: The event types probe and exploit tend to be generated from the same host and within the same minute.

    n = 2, G = {Event Source, Interval}S = {Event Type}Mining camp : (2, {Event Source, Interval}, {Event Type})

  • IDS/IPS: Gaps, Research Challenges and Solutions71

    Multiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern

    • MAF-Pattern mining– Finding all patterns with signature (n, G, S) with more

    than sup_thres support (given by the user)• Scalability is a real challenge

    – All combination of attributes are possible as G and S• Solution:

    – downward closure property

    c ofsuccessor 3- type theis }){AS G, (n, -c ofsuccessor 2- type theis S) },{AG (n, -

    c ofsuccessor 1- type theis S) G, 1,(n-

    i

    i

    ∪∪

    +

    c = (n, G, S)

    Ai is a new attribute

  • IDS/IPS: Gaps, Research Challenges and Solutions72

    Search Space for MAFSearch Space for MAF

    (1, {T}, {A}) (1, {T}, {B})

    (1, {T}, {})

    (2, {T}, {A}) (1, {TB}, {A}) (1, {T}, {AB}) (1, {TA}, {B}) (2, {T}, {B})

    (3, {T}, {A}) (2, {TB}, {A}) (2, {T}, {AB}) (2, {TA}, {B}) (3, {T}, {B})

    (4, {T}, {A}) (3, {TB}, {A}) (3, {T}, {AB}) (3, {TA}, {B}) (4, {T}, {B})

  • IDS/IPS: Gaps, Research Challenges and Solutions73

    Multiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern

    • Sample MAF patterns– (n, {Interval}, {Host})

    – (n, {Interval, Event Type}, {Host})

    • Application of MAF pattern mining– Finding similar patterns– Might be useful for causality and false positive

    analysis

  • IDS/IPS: Gaps, Research Challenges and Solutions74

    Alerts

    Events

    Integration with QRadarIntegration with QRadar

    Filtering RulesMalicious

    Correlation Rules

    Offline Pattern MinerPattern Analysis

    & Rule Discovery

    Event DB

    Custom Rule Engine Similar Patterns

    Causality Patterns

    Event Bursts

    Periodic Patterns

    Raw Events

  • Hardware based Network Hardware based Network Intrusion DetectionIntrusion Detection

  • IDS/IPS: Gaps, Research Challenges and Solutions76

    IntroductionIntroduction

    • A major problem in NIDS is the speed of operation .

    • Software based NIDS are not able to work at line speed, – especially when deployed on high speed links .

    • In signature based NIDS, the problem is more significant– occasionally whole packet payload is to be inspected

    to detect a known signature pattern.

  • IDS/IPS: Gaps, Research Challenges and Solutions77

    Signature based NIDSSignature based NIDS

    • Signature based intrusion detection is one of the commonly used and yet accurate methods of intrusion detection

    – The attack pattern is studied

    – The signature is extracted

    – A most common NIDS that uses signature based approach is SNORT

  • IDS/IPS: Gaps, Research Challenges and Solutions78

    NIDS Classification Based on Hardware NIDS Classification Based on Hardware PlatformsPlatforms

    Reconfigurable Discrete Logic(RDL)

    Reconfigurable Discrete Logic(RDL)

    Static Random Access memories(SRAM)

    Static Random Access memories(SRAM)

    Ternary Content Addressable Memories(TCAM)

    Ternary Content Addressable Memories(TCAM)

    Network Processors(NP)

    Network Processors(NP)

  • IDS/IPS: Gaps, Research Challenges and Solutions79

    RDL: Reconfigurable Discrete LogicRDL: Reconfigurable Discrete Logic

    • Implements Snort Rules in hardwired logic

    • Usually implemented in FPGA

    • Need hardware reconfiguration for updating rules.

  • IDS/IPS: Gaps, Research Challenges and Solutions80

    RDL: Reconfigurable Discrete Logic RDL: Reconfigurable Discrete Logic exampleexample

  • IDS/IPS: Gaps, Research Challenges and Solutions81

    RDL: Reconfigurable Discrete LogicRDL: Reconfigurable Discrete Logic

    • Pros– Very high throughput– FPGAs are the fastest and probably cheapest way to

    market– Rules can be reconfigured

    • Cons– Rule update needs reconfiguration (new HDL, compilation,

    reprogramming)– Compilation sometimes needs days– Not suitable for automatic rule tuning – Large rule-sets may not fit in FPGA

  • IDS/IPS: Gaps, Research Challenges and Solutions82

    SRAM based NIDS exampleSRAM based NIDS example

    • SRAM based architectures typically use Hashing.

  • IDS/IPS: Gaps, Research Challenges and Solutions83

    SRAM based NIDS exampleSRAM based NIDS example

  • IDS/IPS: Gaps, Research Challenges and Solutions84

    SRAMSRAM

    • Pros– Updating rule does not require hardware reconfiguration– Use of hashing may result in single memory access to detect

    a match– Less hardwired logic needed

    • Cons– Need hash logic without hash collision, hash collisions may

    result in multiple memory access– Hash logic may need hardware update if exact hashing is

    used to update rules– Hard to detect multiple patterns simultaneously– Need to perform hashing for different input string sizes

  • IDS/IPS: Gaps, Research Challenges and Solutions85

    Ternary CAMTernary CAM

    • TCAM stores three logic values ‘1’, ‘0’ and ‘x’

    • In case of multiple matches, only the lowest indexed output is selected.

    • Need extra logic for content matching.

  • IDS/IPS: Gaps, Research Challenges and Solutions86

    Ternary CAM exampleTernary CAM example

    k bytes> 1K

    entries

    192.128.101.100

    168.100..xxx.xxx

    192.128.xxx.xxx

    Match192.128.101.xxx

    Input

    TCAM

  • IDS/IPS: Gaps, Research Challenges and Solutions87

    Ternary CAMTernary CAM

    • Pros– Rule update does not need hardware reconfiguration– Single memory access is needed to find a match

    • Cons– Large TCAMs need a lot of extra comparison logic– High access time as compared to SRAM– Expensive– Hard to detect multiple patterns simultaneously– Pattern length is constrained by TCAM width.

  • IDS/IPS: Gaps, Research Challenges and Solutions88

    Network Processor based NIDS Network Processor based NIDS exampleexample

  • IDS/IPS: Gaps, Research Challenges and Solutions89

    Network ProcessorsNetwork Processors

    • Pros– Easier to implement, and fast time to market.– Parallel computation with micro engines and StrongARM.– Complete NIDS can be implemented and not just pattern

    matching.

    • Cons– Normally needs extra logic for pattern matching to catch up

    with line speed.

  • IDS/IPS: Gaps, Research Challenges and Solutions90

    ConclusionConclusion

    • Improvement in hardware based NIDS is still needed

    • Approaches without the need for hardware reconfiguration are most considered

    • We are considering approaches based on PFGA and bloom filters

  • QUESTIONS?QUESTIONS?

    Intrusion Detection/Prevention Systems Gaps, Research Challenges and Solutions AgendaOverviewOverviewResearch Groups�ias.cs.unb.ca ; nsl.cs.unb.caComputer SecuritySecure SystemThreat and IntrusionPrevent Attacks!Attack vs. KnowledgeThreat Evolution: Malicious CodeThe State of Network SecurityThe Speed of Attacks AcceleratesIntrusionsHow about Detecting IntrusionsDetection ApproachesIntrusion Detection SystemsLessen learned so far?Lessen learned so far?Quick Overview of QRadarQRadarQRadar : Basic ComponentsCustomersSpecification-based�DetectionIntroductionSpecification-based DetectionSpecification-based DetectionChallengesSpecification of Network ProtocolsIP-level DetectorCategories and Examples of�Feature Value RestrictionsCategories and Examples of�Feature Value RestrictionsImplementation and ResultsTCP-level DetectorTCP-level DetectorTCP-level DetectorTCP-level DetectorAutomatic Rule TuningIntroductionRulesRules and TestsWhat is the problem?Our ApproachAutomatic Rule Tuner (ART)Moving Average-based TuningInternal design of ARTClassifierClassifierClassifierAdministrator FeedbackChallenges and Future WorksEvent CorrelationProblem StatementAlert CorrelationSome Correlation TechniquesDifferent aspects of correlationApplication in QRadarRule Wizard in QRadarWhat is missing?Our MethodologyApriori AlgorithmBurst EventsBurst EventsRepetitive PatternsPartial Periodic Temporal AssociationPartial Periodic Temporal AssociationCausality PatternsMultiple Attribute Frequent Pattern Multiple Attribute Frequent Pattern Search Space for MAFMultiple Attribute Frequent Pattern Integration with QRadarHardware based Network �Intrusion DetectionIntroductionSignature based NIDSNIDS Classification Based on Hardware PlatformsRDL: Reconfigurable Discrete LogicRDL: Reconfigurable Discrete Logic exampleRDL: Reconfigurable Discrete LogicSRAM based NIDS exampleSRAM based NIDS exampleSRAMTernary CAMTernary CAM exampleTernary CAMNetwork Processor based NIDS exampleNetwork ProcessorsConclusionQUESTIONS?