traffic anomaly detection and attack

Traffic Anomaly Detection and Attack Recognition

QRATOR Labs

Anomaly Recognition Qrator Labs2

The threat

Network attack is becoming a major threat on nations, governmental institutions, critical infrastructures and business organizations. Some attacks are focused on exploiting software vulnerabilities to implement denial of service attacks, damage or steal important data. Other use a large number of infected machines to implement denial-of-service attacks. In this presentation we are focusing on detecting network attacks by detecting the anomalies in network traffic flow data and anomalous behavior of the network applications. The goal is to detect the beginning of the attack in a real-time and to detect when the system is returned back to the normal state.

Anomaly Recognition Qrator Labs3

The threat

The network traffic flow data can be represented by a set of network-level metrics (amount of packets for different protocols, inbound and outbound traffic, etc.) and application-level metrics (like the response duration histogram for web server). These metrics are collected by the traffic analyser at fixed rate. The goal for the state analyzer is to detect anomalous network and application behavior basing on these metrics.The input data for the analyzer is statistics matrix that contains a single row for every traffic time slice. Each row contains the network-level and application-level features that come from different scales. This matrix is the input for the intrusion detection processes (both training and detection steps).

Anomaly Recognition Qrator Labs4

DARPA: simulated attacks on air base[1]

The example of IP-domen traffic’s features due one day and its relations (features)The stochastic process X={x1,…xn} where x_i- all features at the moment of the time

Anomaly Recognition Qrator Labs5

The threat

Challenge: How to process an “ocean” of data in order to find abnormal patterns in the data? How to fuse data from different sources (sensors) to find correlations and anomalies? How to find distances in high-dimensional data? How can we determine whether a point belongs to a cluster/segment or not? The goal is to identify points that deviate from normal behaviour which reside in the cluster. How we treat huge high dimensional data that is dynamically and constantly changes? How can we model the high dimensional data to find deviations from normal behavior?

Anomaly Recognition Qrator Labs6

Network Intrusion Detection Systems

Anomaly Recognition Qrator Labs7

Electronic intelligence and Cyber threat management:Generic approach

Theory, efficient algorithms, software and prototypes (integrated system) which process data in real time to detect anomalies that deviate from normal behavior

Anomaly Recognition Qrator Labs8

DARPA: simulated attacks on air base[1]

Anomaly Recognition Qrator Labs9

DARPA: simulated attacks on air base[1]

Anomaly Recognition Qrator Labs10

Problem setup

Anomaly Recognition Qrator Labs11

Standard approach: Diffusion Maps (DM)

Anomaly Recognition Qrator Labs12

Standard approach: Diffusion Maps (DM)

[2] R.R. Coifman, S. Lafon, Diffusion maps, Applied and ComputationalHarmonic Analysis, 21, 5-30, 2006.

Anomaly Recognition Qrator Labs13

Standard approach: Diffusion Maps (DM)

It is easy to see that the map has the following properties:• The map represents the data in a space of dimension m.• The map is not linear.• The distance between the images of points is equal to the diffuse distance, that is, the probability to get from point x to point y via random walk on the graph for the time t.

Anomaly Recognition Qrator Labs14

Standard approach: Diffusion Maps (DM)

The figure illustrates the effectiveness of the separation of mixed known clusters via “diffusion maps”. If the generated data is represented as two interlocking rings (marked different shades of blue), no any linear methods is able to divide it. Nevertheless, a random walk on the graph represented by these rings, have ability to divide the classes. The probability remain inside the same ring by random walk is greater than the probability of jumping from one ring to another.

Anomaly Recognition Qrator Labs15

Diffusion Maps (DM): The problem

Classification background and anomaly?

Anomaly Recognition Qrator Labs16

Diffusion Maps (DM): The problem

BAD RESULT

Anomaly Recognition Qrator Labs17

Diffusion Maps (DM): The problem

Anomalies are not grouped in clusters

Anomaly Recognition Qrator Labs18

Advanced approach: Homotopy in Temporal Diffusion Maps (DM)

2

2

21

2

2

mod)(

ji xxDji

ij eeG

Diffusion operator

The diffusion geometry is oriented around a smooth parametric curve. The curve represents the day and night

Anomaly Recognition Qrator Labs19

Advanced approach: Homotopy in Temporal Diffusion Maps (DM)

Once X is mapped - extension of to , usingrepresentatives from X (sampling)

f Xx

Xx

Anomaly Recognition Qrator Labs20

Advanced approach: Homotopy in Temporal Diffusion Maps (DM)

iELet

be approximating curve and Xx

iE

Define homotopy G(x)

i

i

iEx

iExiExxG

))(,(

))(,()()(

)(xG

Anomaly Recognition Qrator Labs21

Advanced approach: Homotopy in Temporal Diffusion Maps (DM)

iELet

be approximating curve and Xx

iE

Define homotopy G(x)

i

i

iEx

iExiExxG

))(,(

))(,()()(

)(xG

Anomaly Recognition Qrator Labs22

Advanced approach: Alpha-stream process for anomaly detection

Anomaly Recognition Qrator Labs23

Advanced approach: Alpha-stream process for anomaly detection

Anomaly Recognition Qrator Labs24

Advanced approach: Alpha-stream process for anomaly detection

Image Processing application of “alpha-stream”: Object segmentation

Anomaly Recognition Qrator Labs25

The results:

Anomaly Recognition Qrator Labs26

The results:

The features(left) and its representation in DM (right)

Anomaly Recognition Qrator Labs27

The results:

Anomaly Recognition Qrator Labs28

The results:

The features(left) and its representation in DM (right)

Anomaly Recognition Qrator Labs29

The results:

Anomaly Recognition Qrator Labs30

Comparison to projection on the PCA (existing method)

anomalies background

anomalies 0,95 0,05

background 0,03 0,97

Table 1: distribution of the “false-positive” and “true-negative” for the result of presented algorithm.

anomalies background

anomalies 0,63 0,37

background 0,29 0,71

Table 2: distribution of the “false-positive” and “true-negative” for the result of projection on PCA.