traffic anomaly detection and attack

Traffic Anomaly Detection and Attack Recognition

QRATOR Labs

Anomaly Recognition Qrator Labs2

The threat

Network attack is becoming a major threat on nations, governmental institutions, critical infrastructures and business organizations. Some attacks are focused on exploiting software vulnerabilities to implement denial of service attacks, damage or steal important data. Other use a large number of infected machines to implement denial-of-service attacks. In this presentation we are focusing on detecting network attacks by detecting the anomalies in network traffic flow data and anomalous behavior of the network applications. The goal is to detect the beginning of the attack in a real-time and to detect when the system is returned back to the normal state.


The threat

The network traffic flow data can be represented by a set of network-level metrics (amount of packets for different protocols, inbound and outbound traffic, etc.) and application-level metrics (like the response duration histogram for web server). These metrics are collected by the traffic analyser at fixed rate. The goal for the state analyzer is to detect anomalous network and application behavior basing on these metrics.The input data for the analyzer is statistics matrix that contains a single row for every traffic time slice. Each row contains the network-level and application-level features that come from different scales. This matrix is the input for the intrusion detection processes (both training and detection steps).


DARPA: simulated attacks on air base[1]

The example of IP-domen traffic’s features due one day and its relations (features)The stochastic process X={x1,…xn} where x_i- all features at the moment of the time


The threat

Challenge: How to process an “ocean” of data in order to find abnormal patterns in the data? How to fuse data from different sources (sensors) to find correlations and anomalies? How to find distances in high-dimensional data? How can we determine whether a point belongs to a cluster/segment or not? The goal is to identify points that deviate from normal behaviour which reside in the cluster. How we treat huge high dimensional data that is dynamically and constantly changes? How can we model the high dimensional data to find deviations from normal behavior?


Network Intrusion Detection Systems


Electronic intelligence and Cyber threat management:Generic approach

Theory, efficient algorithms, software and prototypes (integrated system) which process data in real time to detect anomalies that deviate from normal behavior


Problem setup


Standard approach: Diffusion Maps (DM)



[2] R.R. Coifman, S. Lafon, Diffusion maps, Applied and ComputationalHarmonic Analysis, 21, 5-30, 2006.



It is easy to see that the map has the following properties:• The map represents the data in a space of dimension m.• The map is not linear.• The distance between the images of points is equal to the diffuse distance, that is, the probability to get from point x to point y via random walk on the graph for the time t.



The figure illustrates the effectiveness of the separation of mixed known clusters via “diffusion maps”. If the generated data is represented as two interlocking rings (marked different shades of blue), no any linear methods is able to divide it. Nevertheless, a random walk on the graph represented by these rings, have ability to divide the classes. The probability remain inside the same ring by random walk is greater than the probability of jumping from one ring to another.


Diffusion Maps (DM): The problem

Classification background and anomaly?



BAD RESULT



Anomalies are not grouped in clusters


Advanced approach: Homotopy in Temporal Diffusion Maps (DM)

2

2

21

2

2

mod)(

ji xxDji

ij eeG

Diffusion operator

The diffusion geometry is oriented around a smooth parametric curve. The curve represents the day and night



Once X is mapped - extension of to , usingrepresentatives from X (sampling)

f Xx

Xx



iELet

be approximating curve and Xx

iE

Define homotopy G(x)

i

i

iEx

iExiExxG

))(,(

))(,()()(

)(xG


Advanced approach: Alpha-stream process for anomaly detection



Image Processing application of “alpha-stream”: Object segmentation


The results:


The results:

The features(left) and its representation in DM (right)


The results:


The results:

The features(left) and its representation in DM (right)


The results:


Comparison to projection on the PCA (existing method)

anomalies background

anomalies 0,95 0,05

background 0,03 0,97

Table 1: distribution of the “false-positive” and “true-negative” for the result of presented algorithm.

anomalies background

anomalies 0,63 0,37

background 0,29 0,71

Table 2: distribution of the “false-positive” and “true-negative” for the result of projection on PCA.

traffic anomaly detection and attack

Internet