technical university of crete packet pre-filtering for network intrusion detection ioannis sourdis,...

Technical University of Crete

Packet Pre-filtering for Network Intrusion Detection

Ioannis Sourdis, Vasilis Dimopoulos,

Dionisios Pnevmatikatos and Stamatis Vassiliadis

CE, TU Delft, the NetherlandsECE, TU Crete, Greece

ICS-FORTH, Greece

5.12.2006 © Ioannis Sourdis 2

TU Crete

Introduction

Intrusion Detection System (IDS):

Packet Classification (Header Matching)

Payload Scan (Pattern Matching)

Related work on Pattern matching & Packet Classification

Internet

Where From?Where to?

What Content?

Packets

Is this Packet “suspicious”?

IDS

Is that enough for the core of a high-speed IDS???


TU Crete

Motivation

IDS rulesets contain thousands of rules (SNORT IDS: 3,000-4,000)

IDS rule syntax becomes more complicated:

Significant Cost

Performance limitation

The high-speed Pattern matching techniques are required but are not adequate to accommodate the new IDS syntax features

HEADER

P1

payload

depth

HEADER

P1 P2min Distance

offset

within


TU Crete

What do we need?

Need for different levels of processing packets

1st Level: use a simplified version of the rules It’s a Fact: Not ALL rules will match every single packet Exclude the majority of the rules

2nd Level: use advanced attack descriptions for the remaining rules

Reduce the cost and maintain high performance

Incoming Packets

Candidate rules IDs 2nd Level

Complete Match Enginefor the reported rules

(HW or SW)

Matching Rules IDs1st Level

Packet Pre-filtering


TU Crete

How to accomplish this?

We are looking for a fast-inexpensive way to exclude/ “filter-out” for each incoming packet most of the IDS rules + point out a small subset of possibly matching rules?

Partially match each rule of the set

Considering only header info per rule groups of hundreds of rules. Many rules may have the same header description!

Suggestion: header + a small portion of payload pattern in the filter

Rule #1

Rule #2

Rule #3

.

.

.

.

.

.

.

Rule #N

Rule #N+1

.

.

Rule #M

IDS ruleset

Possibly matching

rules


TU Crete


How: Put a small FIRST part of each rule in the pre-filtering Match (if any) the header description of the rule (Source and

Destination Address, protocol) Match a prefix of the first payload pattern of the rule (constant

number of bytes)

IF for a packet, this part of the rule matches the rule needs to be fully matched

ELSE the rule is excluded/filtered-out


Incoming Packets

Candidate rules IDs

Complete Match Engine

for the reported rules(HW or SW)

Matching Rules IDs


TU Crete

Example

IDS Ruleset:1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix]2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix]…N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix]

P(1)…P(N): prefixes of payload patterns (i.e. 4-10 characters long)

The pre-filtering won’t be efficient if incoming packets look like this:

H(1) P(1) … P(2) … P(3) … P(N) …

Candidate: Rule(1) Rule(3) ……..Rule(N)Rule(2)


TU Crete

Example

IDS Ruleset:1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix]2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix]…N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix]

P(1)…P(N): prefixes of payload patterns (i.e. 4-10 characters long)

The pre-filtering will be efficient if incoming packets look like this:

The less rules activated per single packet the most efficient the pre-filtering

H(1) P(1) … P(2) …

Candidate: Rule(1) Rule(2)


TU Crete

HW Implementation

Field extractor: Extract header and payload

Payload Matching: Prefix of the 1st payload

pattern (i.e. 2-10 chars) Implementation DCAM Regular Expressions may also

be used (matching again Reg. Expr. prefixes)

Field Extractor

Header

Payload

Incoming Packets

Partial PayloadMatching Rule 0

Rule 1

Rule 2

Rule N

Priority Encoder

Header Matching

Bitmask

Candidate rules IDs

Header matching: Src/Dest IP+Port, Protocol Implementation simple

comparators

Bitmask, each bit corresponds to a rulePriority Encoder: Pipelined, encodes/outputs every SET bit of the bitmask.


TU Crete

2nd Level: Complete Match Engine


Incoming Packets

Candidate rules IDs

Complete Match Engine

for the reported rules(HW or SW)

Matching Rules IDs

HW or SW? SW assign in multiple threads to match the candidate rulesNetwork processor multiple processing engines...Fully customized HW system

We propose (not excluding other solutions):Reconfigurable Hardware: HW performance, flexibility, reconfiguration to update the ruleset


TU Crete

Reconfigurable IDS core Organization

Pre-filtering points out the rules to be fully matchedSpecialized Engines: For each candidate rule:

A PE is reserved A firmware is transferred to the PE PE released EoP or rule mismatch

Coprocessors (Static patterns & Regular expression matching) perform payload scanPEs select the coprocessor info and decide whether a rule matches or not

Pre-Filtering Specialized Engines

PE PE PE PE

PE PE PE PE

PE PE PE PE

FirmwareMemory

Coprocessors

OUTPUT: MATCHING rule ID

Header Matching

Partial PayloadMatch

Priority Encoder Possible Match

Rules

Static Patterns

Regular Expressions

INCOMINGPackets

MATCHING rule ID

Output I/F

MATCHING rule ID


TU Crete

What if (Candidate rules > PEs) ?

What does (Candidate rules>PEs) mean?

A single packet partially matches more than x rules (e.g. x=32 or 64)

Can such a packet be a normal packet?

What happens when (Candidate rules>PEs)?

In order to guarantee performance, the packet is reported, Admin policies determine the next step (i.e. drop)

Pre-Filtering Specialized Engines

PE PE PE PE

PE PE PE PE

PE PE PE PE

FirmwareMemory

Coprocessors

OUTPUT: MATCHING rule ID

Header Matching

Partial PayloadMatch

Priority Encoder

Possible Match Rules

Static Patterns

Regular Expressions

INCOMINGPackets

MATCHING rule ID

Output I/F

MATCHING rule ID


TU Crete

Experimental Results

Defcon11 traces9 trace files~10 millions packets4.6 million packets have payloadpayload length:

Mean 698 bytes Max 1460 bytes

Milions of packets per trace

0

0.5

1

1.5

2

eth0

znb0

znb1

znb2

znb3

znb4

znb5

znb6

znb7

Millions

Header Only With payload

SNORT v2.43,191 rules

2,271 rules with payload description

920 only header


TU Crete

Simulation Results

Pre-Filtering setup: Header matching Scr/dest IP+Port, Protocol Payload Pattern match 2-10 chars prefix match

For prefix>2 chars: Average Candidate rules per packet=[1,3] per traceOverall average: 1.8 rules per packet Only header match ~45 rules per packet

Average Candidate Rules After Pre-filtering

0

2

4

6

8

10

12

eth0

znb0

znb1

znb2

znb3

znb4

znb5

znb6

znb7

4 bytes6 bytes

8 bytes10 bytes

2 bytes


TU Crete

Simulation Results

Payload prefix match= 2 chars: max 63 candidate rules per packetsPayload prefix match>=4 chars: max 32 candidate rules per packets

What does this mean:Max number of rules for further processing1% or 32 out of 3,200 rulesThe Max degree of parallelism needed (processing engines, threads etc.)

Maximum Candidate Rules After Pre-filtering

0

8

16

24

32

eth0

znb0

znb1

znb2

znb3

znb4

znb5

znb6

znb7

4 bytes

6 bytes

8 bytes

10 bytes


TU Crete

Implementation Results

Field Extractor

Header

Payload

Incoming Packets

Partial PayloadMatching Rule 0

Rule 1

Rule 2

Rule N

Priority Encoder

Header Matching

Bitmask

Candidate rules IDs

0 2K 4K 6K 8K 10K 12K 14K 16K

8-bit

32-bit

Dat

apat

h w

idth

Slices

field extractor Pattern matching header matching Priority Encoder Control

Packet Prefiltering Area Cost

Datapath 8 bits/cycle:Virtex2: 2.7 GbpsVirtex4: 4 GbpsArea 11K slices (medium-small FPGA)

Datapath 32 bits/cycle:Virtex2: 9.7 GbpsVirtex4: 14 GbpsArea 15K slices (medium-small FPGA)

Priority encoder takes most of the area


TU Crete

Conclusions

Packet Pre-filtering Points out a small rule subset per incoming packet

for further processing Offloads an IDS core Allows to utilize more sophisticated attack

descriptions on the 2nd phase of the system (for a few rules)

We include in the filter (per rule) Header matching (Source/Destination Address &

Protocol) 4-8 characters payload pattern match


TU Crete

Conclusions

Performance:

99% of the IDS rules per incoming packet do not need further processing (in Defcon11 traces), without loosing detection precision.

Throughput: 2.7-10 Gbps (Virtex2) or 4-14 Gbps (Virtex4), 8 or 32 bits datapaths

Requirements:

Lightweight system, requires 10-15K slides, can fit in a medium-sized FPGA

Can be integrated in both HW or SW based systems


TU Crete

Questions?

technical university of crete packet pre-filtering for network intrusion detection ioannis sourdis,...

Documents

tu crete

h1 payload

ids rule syntax

matching rules

filter rule

n rule

rule source

p1 pattern