multi-layer intrusion detection and prevention … intrusion detection and prevention system: a new...
TRANSCRIPT
Multi-Layer Intrusion Detection and Prevention System:A New Approach
A Thesis Submitted
in Partial Fulfillment of the Requirements
for the Degree of
Master of Sciencein
Cyber Security
by
Vikash Kumar Saini14/MS/029
Under the Supervision of
Dr. B.M. MehtreAssociate Professor
Center for Cyber Security
Institute for Development and Research in Banking Technology, Hyderabad
(Established by Reserve Bank of India)
COMPUTER SCIENCE AND ENGINEERING DEPARTMENTSARDAR PATEL UNIVERSITY OF POLICE, SECURITY AND CRIMINAL
JUSTICE
JODHPUR – 342304, INDIAMay, 2016
UNDERTAKING
I declare that the work presented in this thesis titled “Multi-
Layer Intrusion Detection and Prevention System: A New Ap-
proach”, submitted to the Computer Science and Engineering De-
partment, Sardar Patel University of Police, Security and Criminal
Justice, Jodhpur, for the award of the Master of Science degree in
Cyber Security, is my original work. I have not plagiarized or sub-
mitted the same work for the award of any other degree. In case
this undertaking is found incorrect, I accept that my degree may be
unconditionally withdrawn.
May, 2016
Jodhpur
(Vikash Kumar Saini)
ii
CERTIFICATE
Certified that the work contained in the thesis titled “Multi-Layer
Intrusion Detection and Prevention System: A New Approach”, by
Vikash Kumar Saini, Registration Number 14/MS/029 has been car-
ried out under my supervision and that this work has not been sub-
mitted elsewhere for a degree.
May, 2016
Dr. B.M. MehtreAssociate Professor
Center for Cyber Security,
Institute for Development and Research in
Banking Technology, Hyderabad
(Established by Reserve Bank of India)
iii
Acknowledgment
I would like to take this opportunity to express my deep sense of gratitude to all who
helped me directly or indirectly during this thesis work.
First, I would like to thank my supervisor, Associate Professor Dr. B.M. Mehtre , for
being a great mentor and the best adviser I could ever have. His advice, encouragement
and critics are source of innovative ideas, inspiration and causes behind the successful
completion of this dissertation. The confidence shown on me by him was the biggest
source of inspiration for me. It has been a privilege working with him from last five
months.
I wish to express my sincere gratitude to Dr. Bhupendra Singh , Vice Chancellorand Sh. M.L. Kumawat, (Former) Vice Chancellor, for providing me all the facilities
required for the completion of this thesis work.
I would like to express my sincere appreciation and gratitude towards faculty members
at S.P.U.P., Jodhpur, especially Mr. Arjun Choudhary & Mr. Vikas Sihag for their en-
couragement, consistent support and invaluable suggestions. I thanks to Mr. Ghanshyam
Bopche PhD. Scholar, who helped me, guided me at the time I needed the most. When-
ever I get nervous, I used to talk with my colleagues. They always tried to encourage me,
without all mentioned above, this work could not have achieved its goal.
iv
Finally, I am grateful to my father Mr. Pratap Saini , my mother Mrs. Vidya Devifor their support. It was impossible for me to complete this thesis work without their love,
blessing and encouragement.
-Vikash Kumar Saini
v
Biographical Sketch
Vikash Kumar Saini
Gurjar Wala Kuwan, Narnaul Road, Singhana.(jhunjhunu)-Raj. PIN-333516
E-Mail: [email protected], Mobile. No. +91- 998216 6843
Father’s Name : Mr. Pratap Saini
Mother’s Name : Mrs. Vidya Devi
Education
• Pursuing Master of Science in Cyber Security, Computer Science & Engineering
from S.P.U.P., Jodhpur, 2016.
• B.Tech. in Information & Technology from Gyan Vihar University, Jaipur, with
69% in 2013.
• Intermediate from Jhunjhunu Academy, Jhunjhunu, with 78% in 2006.
• High School from B.S.S.S, Pilani, with 59% in 2009.
vi
Devoted to My Loving Family for their kind affection and backing,To my companions for indicating trust in me.
vii
}Only Two Things are Infinite, The Universe and Human Stupidity,And I’m not Sure about the Former~
-Albert Einstein
}Security is Not a Product,It’s a Process~
-Bruce Schneier
viii
Synopsis
A vulnerability in a single system constitutes a hole in the entire network. Exploiting
the new vulnerability by an intruder comes under zero-day threat. Just deploying net-
work intrusion detection and prevention system does not detect zero-day threats because
ids/ips work on the signature based detection mechanism. Signatures are based on the
known attacks. There is a possibility of bypassing single-level (ids/ips) security by using
intrusions. So, an attack which enters in the system is a big setback to any network.
In this thesis, A multi-layer architecture is proposed for intrusions detection and pre-
vention. Two devices (Snort, Suricata) are used in this scheme. Multilayer architecture is
based on the signatures as well as anomaly based detection and prevention mechanisms
and also a decision-making process is implemented in the architecture. Intrusions and
data flood attacks are detected and blocked by the proposed design. On the basis of test
results, it is clear that proposed architecture gives better performance compared to indi-
vidual (single) unit of ids/ips. It also collects dropped packets for analysis. This can be
used for prediction and prevention of new attacks. Thus, the proposed architecture gives
enhanced security to the network.
ix
Contents
Acknowledgment iv
Biographical Sketch vi
Synopsis ix
1 Introduction 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Network Protocols . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 Different Forms of Network . . . . . . . . . . . . . . . . . . . . 2
1.2.3 The ISO/OSI Reference Model . . . . . . . . . . . . . . . . . . . 3
1.3 Network Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.1 Types of Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Network Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4.1 Various security threats . . . . . . . . . . . . . . . . . . . . . . . 5
1.4.2 Network Security Components . . . . . . . . . . . . . . . . . . . 6
1.4.3 Network Security Level’s . . . . . . . . . . . . . . . . . . . . . . 6
1.5 Organization of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
x
2 Existing Tools and Techniques for Securing Network 102.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Literature Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 Intrusion detection system(IDS) . . . . . . . . . . . . . . . . . . . . . . 11
2.3.1 Types of IDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Intrusion Prevention System(IPS) . . . . . . . . . . . . . . . . . . . . . 13
2.4.1 Types of IPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.5 Working Mechanism of Network Intrusion Detection and Prevention System 14
2.5.1 Signature Based IDS/IPS . . . . . . . . . . . . . . . . . . . . . . 15
2.5.2 Anomaly Base IDS/IPS . . . . . . . . . . . . . . . . . . . . . . 15
2.6 Technology used in IDS/IPS . . . . . . . . . . . . . . . . . . . . . . . . 15
2.6.1 Working Mechanism of IDS/IPS . . . . . . . . . . . . . . . . . . 15
2.7 Tools in IDS/IPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7.1 SNORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7.2 SURICATA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.8 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3 Multi-Layer Intrusion Detection and Prevention System : A New Approach 273.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 Proposed Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.1 What is Multilevel IDS/IPS? . . . . . . . . . . . . . . . . . . . . 29
3.3.2 Meaning of Proactive Approach in field of Network Security . . . 30
3.3.3 Working of MultiLevel IDS/IPS System . . . . . . . . . . . . . . 30
3.3.4 Analysis of Dropped Packet . . . . . . . . . . . . . . . . . . . . 34
3.3.5 Collecting Valuable Information from Dropped Packet . . . . . . 35
3.3.6 Comparison Between Snort and Suricata Results . . . . . . . . . 35
3.3.7 Various Cases of Decision-Making Machine . . . . . . . . . . . . 37
4 Experimental Setup and Results 394.1 Configurations for Experimental Performance . . . . . . . . . . . . . . . 39
4.1.1 Tools Configurations . . . . . . . . . . . . . . . . . . . . . . . . 39
xi
4.1.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.2 Results of Single-Layer Intrusion Detection System . . . . . . . . . . . . 44
4.2.1 Processing in Snort Machine . . . . . . . . . . . . . . . . . . . . 45
5 Conclusion and Future Work 51
6 Author’s Publications 52
References 53
xii
List of Figures
1 Architecture of Router . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Firewall Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3 Deployment of NIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4 Deployment of HIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5 Process of Detection Mechanism . . . . . . . . . . . . . . . . . . . . . 16
6 Alert Generated Console . . . . . . . . . . . . . . . . . . . . . . . . . 17
7 Detailed Structure of Signature . . . . . . . . . . . . . . . . . . . . . . 21
8 MultiLevel IDS/IPS Architecture. . . . . . . . . . . . . . . . . . . . . 29
9 Replication Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
10 Normal Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
11 Data Flood Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
12 Installed Snort on Ubuntu Machine . . . . . . . . . . . . . . . . . . . 42
13 IP address of protecting machine . . . . . . . . . . . . . . . . . . . . . 45
14 Different types of Rules Saved in Machine . . . . . . . . . . . . . . . . 46
15 User Define Rule For Demo Purpose . . . . . . . . . . . . . . . . . . . 47
16 SYN Flood Record in Attacker Machine . . . . . . . . . . . . . . . . . 47
17 Alert Generating in Live Mode . . . . . . . . . . . . . . . . . . . . . . 49
xiv
18 File Generated when Malicious Activity Detected . . . . . . . . . . . . 49
19 Data in Dropped File . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
xv
List of Tables
1 ISO/OSI Reference Model . . . . . . . . . . . . . . . . . . . . . . . . 3
2 NIDS Output Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3 Structure of Snort Rule . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4 Distinctive Category of Signatures . . . . . . . . . . . . . . . . . . . . 22
5 Decision Making Table . . . . . . . . . . . . . . . . . . . . . . . . . . 34
6 Attacks on Various types of Packet . . . . . . . . . . . . . . . . . . . . 35
7 Valuable Information Relate to Packet . . . . . . . . . . . . . . . . . . 36
8 Snort vs Suricata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
9 Configuration of Host Machine . . . . . . . . . . . . . . . . . . . . . . 40
10 Snort Pre-Requisites File . . . . . . . . . . . . . . . . . . . . . . . . . 41
xvi
Chapter 1
Introduction
1.1 Overview
“Multi-Level Intrusion Detection and Prevention System”. Scheme have analyzed and all
the desired result collected from the work. The new architecture is proposed for giving
enhanced security to the network. The work is explained in this thesis work in coming
parts.
Securing a network is a big and important task. Because the network is the best source
for an intruder to make the cyber crime. Most of the time we fail to prevent the zero-day
threat because of no prior information for that particular attack. The proposed architecture
improve the graph level zero-day threat prevention and also detect and prevent malicious
activity on the network. In the proposed design a proactive approach is dependent on the
analysis of dropped packet.
The single-level intrusion detection and prevention is experimentally performed for
verification the performance of the single tool. Malicious activity is detected and pre-
vented by single level ids/ips but not for all intrusions. A single level device not capable
of preventing all types of attacks. So for giving strength to the network, the proposed
1
design works on two tools (Snort, Suricata). Multilevel security inspects the data and
prevents intrusions.
The intelligence of a machine is used for making intrusion on another intelligence/non-
intelligence machine. The purpose of doing this is either intentionally, or it may be the
business purpose, the scenario comes under cyber crime. The mechanism of Cyber Secu-
rity is designed in such a way that protection of Network, Program, and data from attacks
and an intruder, with the help of various technologies and processes. Various elements of
cyber security are given below
• Securing Network
• Securing Information
• Securing Application
1.2 Networking
Communication between two or more devices for sharing data or using various services
is either with the internet or without internet is called networking. For making communi-
cation between devices there are so many factors play’s role, explaining below
1.2.1 Network Protocols
when communication takes place this is the very basic step, for making the connection
there must a language required and in the cyber world the communication language is
known as Network Protocol.
e.g. TCP/IP mostly found on the internet because this protocol is widely used for com-
munication.
1.2.2 Different Forms of Network
The network deployment depends on the requirement; sometimes there will be a broad
requirement of the system or sometimes small, so according to that network can be deploy.
2
The networking shown in these forms which are given below 1.
• LAN: Services uses for small number of people located.
• WAN: Across a large geographical area.
• MAN: Over radio transreceivers2.
1.2.3 The ISO/OSI Reference Model
Table 1: ISO/OSI Reference Model
Top Most Layer: ApplicationPresentation
SessionTransport
Internet Layer: NetworkData Link
Bottom Layer: Physical
In the Table [1] the seven layer of OSI3 model start their role from the originating of
data from source to destintion till the data received by end user. Open Systems Intercon-
nect(OSI) model has seven layers, control is gone starting with one layer then onto the
next, beginning at the application layer in one station, and continuing to the base layer i.e.
physical layer. The purpose of the OSI reference model is the digital communication.
1.3 Network Attacks
An intrusion on the network infrstructure is called network attack[3]. A system assault
can be characterized as any strategy, process, or means used to malignantly endeavor to1Within the network, there are so many options available for using any topology. According to the
requirement or security point of view, the topologies are uses. e.g. Star, Bus, Ring, Hybrid, etc.2These network services are deployed or used according to the requirement, sometime bigger or some-
time less requirement.3OSI was officially adopted as an international standard by the International Organization of Standards
(ISO).
3
trade off system security. Intrusion collects the information by analyzing the network and
exploit the vulnerability or existing open ports[11].
1.3.1 Types of Attacks
• Passive Attack: Scanned for open ports, vulnerabilities, looks for clear text pass-
word, monitors unencrypted traffic and sensitive information. Passive attacks in-
clude monitoring of unprotected communication.
• Active Attack: Active attack is network exploit intruder attempts to committ changes
to data.
– DDoS4
– DoS
– Message modification attack
– Session reply attack
• Distributed Attack: Trojan horse or back-door program are the components of
distributed attack.
• Insider Attack: Attack is commenting by someone from the inside the organiza-
tion, such as employee.
• Close-in Attack: Intruder trying to get physically access to network components
or data.
• Phishing Attack: Phishing attack intruder creates a fake web site looks like as
original. In Phishing attack intruder try to read information about login credentials,
account information, email identity or other communication channels.
• Hijack Attack: Intruder get access between me and another’s session and discon-
nects the another from the communication.4There are a few sorts of DDoS assaults; These system assaults are developing all the more capable
consistently, and some send more than 100 Gbps at the crest.
4
• Spoof Attack: Source address is spoofed by intruder.5
• Buffer overflow: Intruder sends large data to an system than is expected.
• Exploit Attack: Intruder know the hole in system so by developing code for hole
exploiting data can be thieft.
• Password Attack: Crack the passwords stored in a network account database.
1.4 Network Security
For securing the network there are many techniques available which may be in the form
of hardware or the form of software. But most of the time all the existing techniques are
unable to detect the new attack and network get hacked. The are so many technologies
are uses for detecting or preventing the unwanted or malicious activity on the network
or block the sophisticated threats.6 But most of the time new attacks are not detected or
prevented by available security mechanism.
1.4.1 Various security threats
• Viruses, Worms and Trojan Horses
• Identity Theft
• DDoS
• DoS
• Zero-Day Threats
• Data Interception & Theft.
5The spoofing attack can be possible in these forms, IP Address Spoofing Attacks, ARP Spoofing At-tacks, DNS Server Spoofing Attacks
6The threat is a piece of program which is developed for making the cyber crime. The malicious activitycaused by an intruder and this activity are detected by the software/hardware which is deployed in thenetwork. The security mechanism can be different for every network system.
5
1.4.2 Network Security Components
The network security is provided in the different way and by the following components
which are given below.
• Anti Viruses
• Firewall
• IDS/IPS
• Virtual Private Network(VPN’s)7
1.4.3 Network Security Level’s
Securities at any network is the very first task. Now a days intruders are always trying to
make intrusions on the network. So intrusion detection can be possible in different levels
from basic to higher levels. The levels are explained below.
Basic Level Securities at Network
If any data wants to enter in the network there is identity verification on every data is
processed by security mechanism. The security structure is defined according to the se-
curity policies of the network. The security policies are the various rules which are set
in the machine according to the required data. And also the combination of equipment,
means which equipment is put at the initial level of network or which one afterward. e.g.
Gateway Router has putted at the initial level then after firewall. Some basic securities
are given, and type of securities they do is also explain.
Security by Router
• What is Router?
Device which connects multiple devices together by a network, either wireless or
wired andor it is layer 3 network gateway. Router contains CPU (processor), mem-
ory and I/O interfaces.7VPN creates encrypted connection between private networks. VPN relies on IPsec or SSL connection.
6
• Architecture of Router
In the Figure [1] the Architecture of router 8 and various components of router and
Figure 1: Architecture of Router
their processing is given.
Putting router at initial stage of the network entry have advantages and the func-
tionality are given below.
• Functionality and Security by Router
– Authentication
– SSID
– Mac Address filtering
– Disable SSID Broadcast9
– Assigns static IP Address to devices
All the above securities are provided by router and these are the basic thing for
security point of view. If any malicious data comes under any of above category
then it is blocked by router and unwanted traffic will remove.
8Router can be of wireless or wired and the functionality of both the router is same in both the cases. Indifferent types of security some more feature can be vary, its depend on the company of router.
9At the point when a remote gadget looks the region for remote systems it will recognize the SSID.Disabling the SSID Broadcast is one method for securing the remote system.
7
Security by Firewall
• What is Firewall?
A designed software or hardware which work on the basis of defined policy ac-
cording to the requirement of data. The policies are defined for the protection from
unwanted or untrusted data 10.
• Deployment of Firewall
Figure 2: Firewall Deployment
• Functionality and Security by firewall
– Excellent auditor
– Use to restrict the specific services
– Excellent at alerting.
Security by Antiviruses A designed software which detect/prevent and remove soft-
ware viruses, and other malicious software like Trojan’s, Worms, AdWare etc.
• Remove or prevent spyware and adware
• Uses as safeguard a computer from malware, trojan horses and computer worms.10Firewall can be of two types hardware and software firewall. Firewall cannot stop network attacks if
any relate policy not define
8
Higher Level Securities at Network
Higher level security at any network can provided by deep packet inspection of all incom-
ing and outgoing data[5]. IDS/IPS works as deep packet inspection for every data. Means
IDS/IPS monitor every packet which wants to enter in the network. IDS/IPS support
in-depth security policy and can be used to detect a wide range of threats.
• Software Vulnerability Exploits
• DoS or DDoS
• Malicious Activity
• Buffer Overflow
• Password Cracking
• Protocols Attacks
Every IDS/IPS system works on the principle of signature, which are defined in the ma-
chine. The signatures are based on the previously happened attacks or we can also add
more signature according the requirement11. For increasing the intelligence of machine,
we have added anomaly based detection mechanism. The anomaly-based is explained by
graphical view in the proposed architecture figures [10],[11].
1.5 Organization of Thesis
The remaining part of the thesis is unified as follows:
Chapter 1 Highlighted the introduction, summary of cyber security, Network security and
work carried out by various researchers in the field of Network Security.
Chapter 2 Describes about problem statement, literature survey and existing tools and
technique.
Chapter 3 Proposed a new Architecture in the network security field.
Chapter 4 Configuration and results discussing in this section.
Chapter 5 Conclusion and the Future Work is discussing in this section.11Some time the user defined signature are not able to give good result. So the machine gives false
positive and false negative generating at high rate by the ids/ips machine.
9
Chapter 2
Existing Tools and Techniques for
Securing Network
2.1 Overview
For securing the network, there are many tools and techniques available; they are in the
form of hardware or the kind of software. But most of the time all the existing methods are
unable to detect the new attacks and system get hacked. The network security architecture
deployment is vary from organization to organization. The security policies are different
for every company. There are various techniques which are followed by a company for
detect and prevent the intruders activity.
There are different types of ids/ips tools are available in the security field, and the
deployment of these tools help to prevent malicious activity.
10
2.2 Literature Survey
The idea of deep packet analysis had given by “D. E. Denning” in “1986”[5]. The idea for
inspecting data deeply is used now for analysing all the incoming and outgoing data on
the network [2]. The purpose of doing this for securing the network from new attacks and
preventing malicious activity on the network [11]. The intrusion detection and prevention
system is defined in two ways; Signature based detection and prevention mechanism and
Anomaly based detection and prevention mechanism. Both the mechanisms have used
for network level ids(NIDS) and host level ids(HIDS). The signature defined by previous
attacks. So the ids/ips work by past attacks, new attacks are not detectable by current
ids/ips system. Because of the signature-based detection mechanism, ids/ips generate a
large number of alert for false positive cases, which is overhead for the analyst. Some-
times malicious data successed to enter into the network. Therefore, the case comes under
false negative. The false negative is the big setback for any network. So for providing
more securities and reducing the false negative rate the new design has proposed in this
paper.
Network attacks are the best medium for intruder to committing a large crime [7]. Be-
cause if the entire network is compromised by the intruder then it can be used as zombie
network and another crime can be succeed. In the paper different security mechanism
had discussed for preventing network attacks [19]. The network attacks and their secu-
rity mechanism have discussed, how a network can be secured and how intruders can be
prevented to enter in the network [10], [11].
2.3 Intrusion detection system(IDS)
Intrusion is an unauthorized attempt it may or may not successful [9]. Intrusion detection
system has used for detecting malicious activity. A system is used to detect unauthorized
intrusions onto the network. IDS system work on the principle of Signature and Anomaly
based intrusion detection mechanisms by these two mechanisms, the malicious activity
are blocked.
11
2.3.1 Types of IDS
IDS can be deployed at two places for securing the system. Which are given below.
Netwok level IDS(NIDS)
The entire network as the monitoring scope. The network traffic is monitored to detect
intrusions [6]. NIDS detect data that may be harmful to the system. NIDS capture and
inspect every packet that is destined to the network whether it is permitted or not. NIDS
can run in many ways. It can be run on a separate machine and inspect the whole network,
or it can be run on a computer and investigate itself. e.g. It can watch itself or entire
network, if somebody is attempting an SYN Flood or a TCP Port scanning.
• Architecture of NIDS
The architecture of NIDS is given in Figure [3].
Figure 3: Deployment of NIDS
12
• Network Load Balancer
The network data is capture by NIDS and distribute to the whole network. It is a
software that runs from the sensor.
• Alert Notifier1
It contacts to the network security team responsible for handling if any incidents
occur on the network on the behalf of organization security policy.
• Database
The database maintained for capturing dropped packets, and these packets are uses
for prevention of future attacks on the network.
Host Level IDS(HIDS)
Inspection of the system within the organization network is called host-based intrusion
detection system. HIDS2 is detecting malicious activity within the single machine.
• Architecture of HIDS
The architecture of HIDS is given in Figure [4].
2.4 Intrusion Prevention System(IPS)
IDS are passive components they only detect and reports blindly. IPS has prevented the
attacks; IPS is also working like IDS in two forms.
2.4.1 Types of IPS
Network level Intrusion Prevention System(NIPS)
The bundle work on the network which detects and prevents is preferred for security
perspective, this prevention of attacks on the network level comes under NIPS. The entire
1The network analyst defines alert, and action specified in the machine so according to that machine willperform action.
2Host level ids work for a single system only, monitoring the data which arrive on the host machine.Within a single network, more than one HIDS can not be deployed
13
Figure 4: Deployment of HIDS
networks data are monitoring and intrusions prevented by NIPS.
Host level Intrusion Prevention System(HIPS)
Most HIPS prevents occurs by agent resides on the host machine. If any malicious code
is running on host machine, then it is detected and prevent takes place by HIPS.
2.5 Working Mechanism of Network Intrusion Detection
and Prevention System
Network Intrusion/Prevention System work by two mechanisms, either the malicious ac-
tivity made by the intruder is unknown by IDS/IPS machine, or device already knows it.
The principle of IDS/IPS is explained below.
14
2.5.1 Signature Based IDS/IPS
The various signature category is given on Table [4]. The signature type is defined by dif-
ferent types of attacks which have already exploited by the intruder and for that particular
attacks a signature is defined for preventing. The signature list updated mostly if any new
malicious activity is happening on the network. It is because an intruder does not repeat
the same activity.
2.5.2 Anomaly Base IDS/IPS
The anomaly-based detection cover the sensing capability of the machine. The machine
is not like human thinking, but we can define code in the machine. The graph has shown
in this thesis for detecting and preventing anomaly-based attacks [10][11]. e.g. Any DoS
or DDoS is happening on the network then bandwidth is field more than the threshold
value then IDS sense this activity because it is not the part of daily network activity and a
machine will block the attacks.
2.6 Technology used in IDS/IPS
IDS/IPS work’s only on two major principle’s one is signature based detection/prevention
and another is anomaly based detection /prevention system. There are so many IDS/IPS
tools available; some are open source, and some of them are licensed version. The func-
tionality of devices is varied it depends on the signature defined and sensing capability of
the machine.
e.g. Tools which are open sourced.
• SNORT
• SURICATA
2.6.1 Working Mechanism of IDS/IPS
Working of ids/ips is on the basis of two mechanisms are explained below.
15
Signature based IDS/IPS Detection Mechanism
This is the simple tasks for IDS to detect malicious activity if this is the defined in the
signatures. For detecting the malicious activity, there is a program stored in the machine.
The program has developed in such a way that if any intrusion made on the network by
the intruder, then it will detect and generate an alert. Various signature is defined by past
attacks. For a single type of attack, a single rule is developed. If the intruder changes the
malicious code, then it comes under a new kind of attacks. Update of the signature has
done in two phases either it is update by the developer team or it can be manually update.
We can update the signatures according to the requirement, any rules can be add or can
be removed.
An example of rules are shown below.
• Alert TCP $EXTERNAL NET any Ô $HOME NET 139 (msg:“NETBIOS NT
NULL session”: flags:A++:
content: “x 00 00 00 00 57 00 69 00 6E 00 64 00 6F 00 77 00 73 00 20 00 4E 00 54
00 20 31 00 33 00 38 00 31 x” ; reference: bugtraq, 1163 ; refernce: cve, cve-2000-
0347; reference : arachnids, 204 ; classtype: attempted-recon; sid:530 ; rev : 5;)
How a signature is devleoped and how port number and ip address are stored this
detailed explaination is on “Detailed structure of signature”
Process of Detection Mechanism The process of intrusion detection mechanism is
given in Figure [5]. And the alert is generated if intrusions has arrived. The alert is then
Figure 5: Process of Detection Mechanism
16
Figure 6: Alert Generated Console
process to the network security engineer, by trigger or by alert. Some form of secondary
processing on the information.
Pattern Matching In this process the matching of individual signature against of indi-
vidual packets. Let suppose a web server attack is commenting on the network then the
pattern is matched only under the category of web server area, which will reduce the time
complexity of the machine.
StatefulPattern Matching In this process the matching of a pattern on the entire ses-
sion not on a single packet. Stateful pattern matching to reassemble communication ses-
sion like TCP (Transmission Control Protocol) session and IP (Internet Protocol) frag-
mentation. State of each communication transaction is tracking and reassembling of
packets stream is on the receiving in the same manner.
Listening Alert When a malicious activity is happening on the network, then ids/ips
will inspect the activity and according to rule, machine will take action.
Challenges to Signature Based IDS Signature based ids is a very useful tool, as also
face some problems. Primary activity detects very well while sometimes not serving well.
Factors such as speed, network architecture, and signature are added regularly. Some ex-
amples have given below which are the problem in signature based IDS.
False Positive Data is okay, generate the alert. Some companies deployed IDS/IPS for
17
detecting malicious activity. But after few weeks, they removed because it generates hun-
dreds of alert per day, which is the unnecessary trigger. The reason of false positive case.
TCP/IP has designed very open and very flexible suit of protocol. Modifications are made
to the protocols daily basis, and Companies regularly find new users for existing protocol.
So for detecting harmful activity is a tough task.
Solution: False positive is overhead for the network analyst, so the architecture and their
functionality has proposed in such a way that it has decreased false positive case. The
functionality has discussed in this thesis by using the different location for saving dropped
packets. The complete discussion is given in topic [3.3.4]
False Negative Data is not okay(bad data) entered in the network. IDS/IPS is not capable
of detecting all kind of attacks. Therefore, false negative(attack successful) data entered
into the network which is the big setback for any network. When ids fail to generate alert
when malicious activity occurs. Because attacker knows the IDS/IPS has deployed and
various rules are defined, so they develop code apart to specified standards.
Solution: Analysis is necessary: The database has maintained by machine for collecting
dropped packets. These dropped packets were received from the single and multi-layer
ids/ips machine. The analysis has made for getting all the valuable information against
intruder and new attacks and data flood attacks prevented.
Anomaly Based IDS/IPS
Anomaly based ids work on the statistics of data on the daily basis record. Let suppose in
any company there is uploading, and downloading working is going on, and daily records
have counted by network monitoring tool (Wire-Shark). According to daily records, the
value is assigned in the graph. If any data flood is happening on the network, the machine
will treat the attack as anomaly detection. Therefore, an alert will generate. The graph
has discussed in figure [10], [11].
18
2.7 Tools in IDS/IPS
security on the network by ids/ips is comes under higher level security. Using term higher
level security has an important meaning because ids/ips inspect deep packet analysis of
coming data in the network. Other than ids/ips security component provides security
according the defined policy. Different types of security tools are available in ids/ips
some gives better result and some are not. Different types of tools are explained here
which are used in the proposed architecture Figure [8].
2.7.1 SNORT
• Snort is Network Intrusion Detection and Prevention System (NIDS)3
• Packet sniffer
• Inspect the traffic in real time
• Work on basis of signatures, can also work on the basis of anomaly detection.
Configuration modes of Snort
Configuration of Snort can be possibile in the three modes. Which are given below
• Sniffer mode
Reads the packet off network and displays on the screen (console). In sniffer mode
various types of process can be done which are discuss below
– We can print TCP/IP packet to the screen (./snort -v)
– Can show IP and TCP/UDP/ICMP header (./snort -vd)
– Displays packets and the headers (./snort -vde) or (./snort -d -v -e)
• Packets Logger Mode
Logs are generate and stored in disk. the path of disk is automatically saved in snort
or it can be change according to the requirement.3Network level intrusion detection inspect the entire network’s data. Deep inspection detection mecha-
nism provides better security to the network.
19
– ./snort -dev -1 ./log
• Network level ids(NIDS)
Detect, prevent and analysis on network traffis.
– ./snort -d -h “ipaddress” -1 ./log -c snort.conf
– NIDS mode outputs
Table 2: NIDS Output Mode
Choice Explaination-A Fast Fast Alert Mode-A Full Full Alert Mode
-A None Alerts are turn off-A Console Send alerts to screen(console) mode-A Unsock Alerts are sends to UNIX socket
– High performance configuration
We need to use barnyard2 it allows snort to log alerts in a binary form. If we
want tcpdump then logging with the fast Output mechanism. e.g. ./snort -b -A
fast -c snort.conf
– Alert order:
We can change default Pass rules; Drop rules, Alert, Log rules.
– Working principle of snort
Snort detect and prevent intrusions on the basis of signatures. Signatures are
used to verify all the parts of a data packets. Snort 1.x version can analyze
transport and network layer but in Snort 2.x version it is also added that it can
analyze application layer protocols.
Structure and Development of a Signature
Signatures are the collection of instruction which written for the detection and prevention
of malicious activity.
Snort signature defined in two logical parts, which are given in table [3].
20
Table 3: Structure of Snort Rule
Rule Header Rule Option
What action rule takes this information stored in rule header. Additionally contains crite-
ria for coordinating a standard against packets. Alternative part include an instant message
and tell which part of the packets ought to be utilized to create the instant message.
E.g. Alert ICMP any any Ô any any ( msg: “ping with TTL=100”;)
Figure 7: Detailed Structure of Signature
• Protocols:
IP, ICMP, TCP, UDP
• Directions:
Ô ,←, <>
21
Categories of Snort Rules
Table 4: Distinctive Category of Signatures
S.NO Category Various forms Action
1 Appdetect.
rules
Network activity Control parts of an applica-
tion carry on
2 Blacklist.
rules
Public list of malicious
URLs, Talos virus
sandboxes
Contains DNS, URI, USER-
AGENT and IP address
chooses that have been
set out to be pointers of
intruder’s development
3 Browser.
rules
Chrome, Firefox, ie,
webkit, plug-in
Detection for vulnerabilities
in browsers
4 Content Replace, Deleted Check for content replace and
detection
5 Exploit kit Searches for vulnerability
against programming in a
software
6 File Executable, flash, im-
age, identify, multime-
dia, office, pdf
Vulnerabilities are checked
that are found or conveyed
through executable records
7 Indicator Compromise, obfusca-
tion, shell code
Contains rules for positively,
obfuscated and identification
markers of shell code in traf-
fic
8 Malware Backdoor, cnc, tools Contains rules for the iden-
tification of activity bound
to known listening command
channels
Continued on next page
22
Table 4 – Continued from previous page
S.NO Category Various forms Action9 OS Linux, Solaris, win-
dows
Looking for vulnerabilities in
different OSes
10 Policy Multimedia, social,
spam
Detect potential violations of
different policies
11 Protocol ftp, icmp, imap, pop,
services, VoIP
Show the nearness of given
conventions activity or vul-
nerabilities in the system
12 Pua adwarep2p toolbars Potentially Unwanted Appli-
cations (pua) that deal with
unused application which in-
stalled as background process
13 Server Apache, iis, mssql,
mysql, oracle, webapp,
Decides that identify vulner-
abilities in or assaults against
servers
2.7.2 SURICATA
• Network Intrusion Detection and Prevention security Monitoring Engine [12].
• Open Source
Engine
• NIDS & NIPS engine
• Network security monitoring(NSM) engine
• Traffic recording using PCAP logger
• Offline analysis of PCAP files
23
• Processing of PCAP files in UNIX socket mode.
O.S. Support
• Windows, FreeBSD, OpenBSD, MacOSX, Linux
Protocol passer
Support for packet decoding given below
• UDP, ICMP, IPV4, IPV6, TCP
If any of the above given protocols have arrived in encoding formate, then protocols
are decode by Suricata machine.
Detection Engine
• Protocol keywords
• Rule Profiling
• File Matching
– File Magic & Name and Extension & Size
Outputs
• All JSON alert and event output
• Request logging for Http
• Alert fast log
• SysLogÝ alert to sys log
• Flow logging
Output from Suricata machine can be captured in various forms, which have given
above. The capturing of the outputs from Suricata machine can be visualized at live
mode or can be kept in the database.
24
Packet WorkShop
• High Performance capture
– AF PACKET : Read Raw Socket & Read Load Balancing
– PF RING : Fast packet capturing, analysis and investigation
– NETMAP : Netmap is an extremely productive structure for line-rate for raw
packets input/output from client space,
• Standard capture
– PCAP : Is instrument in libpcap library.
– NFLOG (Net Filter Integration)
• IPS mode
– Netfilter based on linux
– NETMAP
• Multi Threading:-
– Fully Configurable threading from single thread to dozens of threads
– Optional CPU Affinity setting
– Optional lock profilling.
Working mechanisms, all the configurations and requirements of Snort and Suricata
has discussed in this chapter. The working process and outcomes is discussing in the next
chapter.
2.8 Problem Statement
Traditional security mechanisms use single IDS/IPS device. As a result, many intruders
succeed to bypass the ids/ips security and also new attacks are not detectable by used
methodology because ids/ips work on the signature based detection mechanism. So in
25
some cases, False Negative (attack successful) data has entered into the network. Also, a
drawback of ids/ips is generating more number of unnecessary alert for some data these
cases comes under the false positive case. The available tools for securing the network
are Snort & Suricata. Individual deployment of any tool has some problem but the combi-
nation of both the tools gives strength to the network. The explanation is given incoming
parts.
26
Chapter 3
Multi-Layer Intrusion Detection and
Prevention System : A New
Approach
3.1 Abstract
Traditional security mechanisms use the single unit of IDS/IPS device. In some cases,
intrusions are succeeded to enter into the network. These successful attacks are a big set-
back for any network. In this paper, We propose a Multi-Layer Architecture for intrusions
detection and prevention. The design is based on the two ids/ips tools Snort and Suricata.
It is found that the parallel combination of Snort and Suricata tools gives better results
compared to individual performance. Our test results show that the false negative (attack
Successful) cases are blocked and failed to enter into the network, and this scheme also
reduces the false positive cases. This is achieved by using dropped packets analysis in
the proposed design. Our results show that deployment of multilayer ids/ips system gives
enhanced security.
27
• Keywords: Snort, Suricata, Wire-shark, MySQL Router, Firewall.
3.2 Introduction
Here in this paper, the intelligence of the machine is increased by using more number
of tools in the ids/ips security mechanism. Because the combination of two or more
tools gives power to the detection mechanism. So intruder unable to bypass the security
system. The proactive approach of proposed architecture is based on the dropped packet;
these dropped packets are the collected from multi-level architecture. So by making the
analysis of these dropped packets will help to prevent the new attack. And also remove
the false positive rate which are dropped unnecessarily . When the data is flooded on the
network, the security mechanism can’t handle that flood and threshold value goes beyond
the defined value.
The motive of proposed architecture Figure [8] is preventing false negative cases 1 and
new attack [16]. The multilevel security will inspect the downloading and uploading data
on two different machines at the same point in time. The multilevel inspection of a single
data increases the threat detection and prevention probability.
There are so many types of attacks are possible on the network. The most useable
protocol is TCP/IP protocol for every service over the internet, so attackers try to find
vulnerability over TCP/IP protocol [14], [7].
3.3 Proposed Architecture
The given architecture Figure [8] is for increasing the security strength of any network.
That will help to detect the various types of attacks and prevent them. The proposed
architecture has defined in two forms to give the strength to a network. The first part is
Multilevel Intrusion Detection and Prevention system, and the another part is the Analysis
of dropped packet. Multilevel IDS/IPS system has a unique functionality that the data
wants to enter the network will go through multi-tools inspection, the tools are monitoring
1At the time of DoS/DDoS attack is happening on the network, the data are flooded on the networkwith a high rate, and the rate may be of 100-300GbPS or more. The whole system goes down, and all theservices of network failed. So by this, the intention of an attacker will be completed.
28
Figure 8: MultiLevel IDS/IPS Architecture.
same data at the different machine at a single point in time. Therefore, the system is
known as multilevel IDS/IPS system.
The another part is an analysis of dropped packet; these dropped packets are generated
from multilevel IDS/IPS because any intrusion made by an intruder comes under dropped
packets. The analysis of these packets falls within the proactive approach in the field of
network security. Because these dropped packet will tell the next step of an attacker. The
proposed architecture is deployed on the network level intrusion detection and prevention
system (NIDS). 2
3.3.1 What is Multilevel IDS/IPS?
Using two or more IDS/IPS tools (Snort, Suricata) for network security and parallel in-
spect the same data at a same point of time will increase the chances of detecting malicious
data, because if using a single tool it may bypass the security but using two or more tools
its difficult to bypass both the tool’s security. The default mechanism of detection/pre-
vention of both the tools are signature based but in the proposed architecture one more
functionality has added anomaly based detection and prevention mechanism. So both the
mechanism are used in the multilevel IDS/IPS system. The signature is predefined by the
developer team Table [4] and the regular update is generated, according to the requirement
2The multi-layer intrusion detection and prevention system has deployed on network level intrusiondetection/prevention(nids).
29
at the user level.
3.3.2 Meaning of Proactive Approach in field of Network Security
By the multilevel ids/ips system, the malicious activity and flood attack is on the network
is prevented. If any malicious activity is blocked by ids/ips, then it generates a file which
comes under dropped packet. These dropped packet used for analyzing them. Because
these packets are attempted intrusion by intruder, and used tools block them. All the
valuable information from these files are collected for securing the network. The valuable
information is used for preventing the zero-day threat/vulnerability. And by this, we can
protect the network from the zero-day threat. What information collected from dropped
packet is shown in Table [7].
3.3.3 Working of MultiLevel IDS/IPS System
The multilevel IDS/IPS is explained in the three phases.
Phase 1 : Replication Phase
Replication machine Figure [9] works only for duplication of packets there is no packet
inspection at this phase. Blindly replication processes and sends them to the next phase
on two different machines. The replication of packet can be possible by any available
tools, or this replication process can be defined manually by any script.
Figure 9: Replication Machine
30
Phase 2 :Inspection phase
The inspection of data is processed on phase 2 of given architecture. The examination of
data happens and also on the anomaly based detection mechanism. The anomaly detection
graphs have shown in figures [10],[11] in this thesis work.
Signature Based Detection and Prevention The signature [13] based detection is de-
fined by previously happened attacks. If any attack is repeating by the attackers, then it
will detect by the machine. Thats why a particular signature is used for particular types
of attacks. And the machine will block that attack and generate an alert.
Anomaly based detection and prevention This technique is based on the intelligence
of machine, but the machine is not like human thinking. Therefore here in this proposal,
some intelligence has given to the machine in the form of the graphs. It was found that
according to given technique machine work like human mind in the form of sensing ca-
pability of data only and prevent the DoS and DDoS attack if exploiting on the network.
The graph has developed by daily downloading and uploading data; the average has taken,
and here it is generalize based on an example. The machine is work on the behalf of the
average graph and senses the data which have defined in the algorithm. If other than
the algorithm is given data are downloading in the network, the machine will take action
according to the given instruction.
• Normal Graph
The value in the graph has assumed of a company for explaining the concept; The
expected data is a software company and according to the work of company the up-
loading data and downloading data are defined here. In the Figure [10] the average
is found that per second 0.01Gb data uploading and 0.005Gb data is downloading.
The average has found by network monitoring tool (Wire-Shark3). In wire-shark
every packet has captured and also count the number of packets arrived in the sys-
tem. Some more functionality wire-shark having but here we use only the packet
counting features.3Wire-Shark is network monitoring tool in which every incoming and outgoing data has captured and
relative information is shown in the system.
31
Figure 10: Normal Graph
• Data Flood Graph
In Figure [11] DoS or DDoS attack is happening and the data is flooding 4 on the
network and defined graph and equation show there is something anomaly action
on the network. Then IDS/IPS will generate alert and takes action. At the time of
attack, the average data is tiny as compared to attacked data means defined aver-
age graph has crossed the limit and graph go beyond aspect by IDS/IPS machine.
Therefore, immediate action will take place by machine and more than threshold
value data will not allow. This is the feature of sensing capability of the machine
and comes under Proactive approach in the field of network security. The data flood
graph is shown in Figure [11]. The above graphs are given on the basis of average
data of any company and the maximum data of a day is given. The maximum
data is calculated on the basis of working hours of the company. Here the working
hours are assumed 12 hours per day. And in working hours the downloading and
uploading data are capture and used in graph.
The average data have used by 12 hours per day, which are the working hours in the
company.
Threshold values are:
• Uploading = 432 Gb
4The flooding of data is at very high rate in DoS or DDoS attack, the data is flooded by attacker becauseintention is jamming the bandwidth and achieving goal.
32
Figure 11: Data Flood Graph
• Downloading = 216 Gb
In flood graph when DoS or DDoS attack is happening then, the downloading data
cross the threshold value, and IDS/IPS will take action. So more than threshold value no
data will allow entering in the network, according to defined action.
The above two algorithms are given for anomaly detection and prevention mecha-
nisms. Traditional ids/ips tools work on the signature-based detection system so for in-
creasing the intelligence of machine anomaly detection and prevention is included. Which
will help to protect the network from data flood attacks. If any data flood attack (DoS or
DDoS) is happening on the network then defined graph will cross the threshold point of
normal data. Therefore, according to defined action machine will work. And prevention
of DoS or DDoS attack is possible. So the used algorithm is a proactive approach in
network security field.
Phase 3 : Decision Making Phase
The action taking part is processed on the phase 3 machine which is shown in Table [5],
and all the input of this machine are dependent on the phase 2 outputs and according to
them, relevant action will take place. The decision-making condition will calculate by
AND GATE properties. In the table [5] showing that the data is allowed or not allowed to
enter in the network.
The cases give high-end security to the network because the data is allowed to enter
the system only in a single case, the given table[5]. The allow case of data is dependent on
33
both the IDS/IPS tools which mean if both cases allow condition then only data will enter
the network. If any of one give not allow state, then data will not come into the system,
and the alert will generate, and data keep in the database. The database has maintained
for collecting dropped packets because these dropped packets are used for making the
analysis.
Table 5: Decision Making Table
Case SNORT SURICATA SNORT ANDSURICATA
Action Taken by Ma-chine
Case1
Allow Allow Allow Data will enter in the net-work
Case2
Allow Not Allow Not Allow Blocked, Alert Generate,Keep in database 1
Case3
Not Allow Allow Not Allow Blocked, Alert Generate,Keep in database 2
Case4
Not Allow Not Allow Not Allow Blocked, Alert Generate,Keep in database 3
3.3.4 Analysis of Dropped Packet
Taking the Advantage from hackers Activity for securing the network.
The dropped packets are the advantages for us. The meaning of saying Advantage from
hackers Activity is that hacker is giving the hint to us something is vulnerable in your
network that’s why they are attracting. These dropped packets are the malicious activity
which was made by the attacker, but blocked by used scheme. The probability of success
of zero-day threat is not 1 at the first attempt. So many intrusions had made by the attacker,
and IDS/IPS blocks these attacks. So by these dropped attacks we are able to know the
target of an attacker.
Activity made by an attacker on Network The intruder tries to gain all the valuable in-
formation about the network. Without knowing the relevant information about the system,
the intruder is unable to exploit the network [10]. All kind of information like which type
34
network security is uses or various rules using for the network security, all the security
levels, etc.
Table 6: Attacks on Various types of Packet
Name Of Packet Name of Protocols Attacks based on ProtocolsTCP/IP ARP, IP, RARP, HTTP,
HTTPSPassword sniffing, Denial ofservice, TCP sequence num-ber, TCP session hijacking,Ping O Death, etc.
UDP FTP, TELNET,TFTP, SMTP,TIME,POP3, FINGER, NTP,IMAP2, SNMAP
DNS, SSDP, CharGEN,QOTD, BitTorrent, Kad,Steam Protocol, NTP, Net-BIOS, Quake NetworkProtocol, RIPv1, MulticastDNS(mDNS), Portmap
Various types of given packets and protocols on which attacker made attacks [11]. Some
of the attack may repeat and according to attack; the signature has defined for detecting
the attack.
3.3.5 Collecting Valuable Information from Dropped Packet
Every packet has some unique feature, unique information, and structure. The dropped
packet are of different types and valuable information from each dropped packet. Because
what the information will be access from dropped packet will only tell what action should
take for securing the network from new threat, Various types of packets and their valuable
information is given in Table [7]
3.3.6 Comparison Between Snort and Suricata Results
Using two types of machine give strength to the network because at least one of the two
machines gives the better result for all malicious activities. Because both the machine
35
Table 7: Valuable Information Relate to Packet
Name of Packet Valuable InformationIP Version, Type Of services(ToS), Flags, Destination Address, protocol
TCP Destination port, Control Bit Flags(URG, PSH), Urgent PointerUDP Destination port, Destination, Length
have different detection mechanism in some cases[1]. Combined both the device will
play the good role instead of single machine. The difference given in table [8] .
Table 8: Snort vs Suricata
Parameter Snort SuricataThread Single-Thread Multi-ThreadRules Talos(VRT) : Snort
rules, Shared Objectrules and EmergingThreat rules
Talos(VRT) : Snort rulesand Emerging Threat rules
Shellcodes Less Detectable More DetectableEvasion Technique More Detectable Less DetectableMalware and Viruses Less Detectable More DetectableMultiple failed logins More Detectable More DetectableFragmented packets More Detectable Less DetectableDenial of Service (DoS) More Detectable More Detectable
Both the tools are different in some cases of detection mechanism like in some cases both
are equally detectable or in some cases anyone is highly detectable and other may be less
detectable. We are using two tools and at least one has highly detectable in all the cases.
Cases are shown in table [8].
Deploying multilevel ids/ips on the network gives multifunctionality processes. The
initial stage is under detection and prevention of malicious activity on the network. The
intruder does not bypass the multilayer security system because using two or more tools
gives strength to the network. The another part of proposed architecture is preventing
new attacks on the network. Another advantage of multilayer ids/ips is decreasing the
false negative rate in the network. Because the intruders unable to bypass the security
36
area, it is possible by using two or more tools in the proposed architecture. The reason is
at least one of all the different tools give a true positive result for a particular malicious
activity.
3.3.7 Various Cases of Decision-Making Machine
In the decision-making machine, there are four cases arises by the AND Gate property.
These four cases play different role for every output from the machines. Because the
output is dependent on phase 2 of architecture. If decision-making machine drops any
data, then there may be the possibility of the false positive case. So for removing false
positive cases, there are three types locations have used for storing dropped packets. The
data location and various cases have discussed below which have discussed in table [5].
• Case 1:
Both the devices of Snort, Suricata on phase 2 will say the data is not malicious
therefore the incoming data have entered into the network.
• Case 2:
Snort machine said the data is not malicious, and Suricata machine said the data
is malicious. So there are chances of this case of false positive or it can be possi-
bility of malicious data. Therefore these categorie’s data are stored in the database
location 1.
• Case 3:
Snort machine said the data is malicious and suricata machine said the data is not
malicious. So there is chances of this case of false positive or it can be possibility of
malicious data. Therefore these categorie’s data are stored in the database location
2.
• Case 4:
Both the machines of phase 2 said the data is malicious, so it is the possibility of
data is malicious and dropped by ids/ips. These categories data are uses for making
analysis for preventing new attacks because the intruders attempts for intrusions
37
goes failed5.
5 Case 4 gives the high probability of intrusions because both the used tools Snort and Suricata sayingthe data is malicious. This type of dropped packets are analyzed regarding new attacks and false negativecases can be prevented.
38
Chapter 4Experimental Setup and Results
4.1 Configurations for Experimental Performance
In this area I tried to improve the graph level ”Intrusion Detection System” as well as re-
ducing “False Negative” cases and also tried to prevent the new threat by making analysis
of dropped packet. The work performed on the following terms which are given below.
• Detection of Malicious Activity
• Prevention of Malicious Activity
• Alert Generating
• Dropped Packet File Generating
• Analysis of Dropped Packet.
4.1.1 Tools Configurations
• Host Machine Configuration
The experimental performance has done on a single machine. Some more virtual
39
machines have installed on the single host machine, and the network has created
between all of these. The configuration has discussed below.
Table 9: Configuration of Host Machine
PROCESSOR Intel Core i7 @ 3.40GHzRAM 32.0 GB
OS Windows 8.1HDD 1TB
• Tools Used
– Vmware Workstation 12 player
– WireShark
– Nmap
– Zenmap
– Snort
– Suricata
– MySQL
– Software Firewall
All the above tools are used in the host machine and Snort configured machine
for network monitoring.
• Machines Installed in Vmware
– Snort Installed Machine(Ubuntu 14.04)
– Protection Machine(Windows 7)
– Attacker Machine(Kali 2.0)
– Traffic Generator Machine(Ubuntu 14.04)
40
4.1.2 Performance
The work is performed on the single machine and the configuration of machine have given
in Table [9]. Four VMware machine are installed on the host machine, and every machine
have different functionality and uses.
• Snort Installed Machine(Ubuntu 14.04)
The Snort tool gives better result in the Linux OSes because the functionality of
snort is more compatible with Linux. Here the reason for using Ubuntu is only
this. Snort has installed as Network Intrusion Detection System. Network Intru-
sion Detection and Prevention System has monitored the whole network which has
discussed in Figure [3]. NIDS inspect with the functionality of “Deep Packet In-
spection” [8]. Each and every packet is monitor by nids machine and check for
malicious activity.
– Snort Pre-Requisites Snort have four main Pre-Requisites 1
Table 10: Snort Pre-Requisites File
Library/FileName
Name to Search Available Source
libdnet libdumbnet-dev Accessible by Ubuntuvault(repository)
pcre libpcre3-dev Accessible by Ubuntuvault(repository)
pcap libpcap-dev Accessible by Ubuntuvault(repository)
daq www.snort.org compiled from source
– For installing snort some basic commands are as follows:
1Before installing Snort pre-requisites configuration is must, without configuring these libraries Snortwill not give accurate result.
41
Listing 4.1: Installing Snort in Ubuntu by Following commands
cd / s n o r t s r c <<<Line 1>>>
wget ‘ ‘ l i n k o f s n o r t f i l e ” <<<Line 2>>>
t a r −xvz f s n o r t 2 . 9 . 8 . 0 . t a r . gz % Unta r t h e
downloaded f o l d e r . <<<Line 3>>>
cd s n o r t 2 . 9 . 8 . 0 % E n t e r i n t h e s n o r t f o l d e r
where a l l t h e f i l e c o n s i s t i n g . <<<Line 4>>>
c o n f i g u r e − s o u r c e f i l e <<<Line 5>>>
make <<<Line 6>>>
sudo make i n s t a l l <<<Line 7>>>
Figure 12: Installed Snort on Ubuntu Machine
In Figure [12] the version of snort is shown. And also showing the pre-requisites
libraries.
– Two Libraries give strength to snort which has configured in snort machine.
∗ Barnyard
Barnyard2 is an open source mediator for Snort unified2 parallel yield
records. Snort write output to the hard drive and by using Barnyard han-
dle database inserts into binary form. For installing barnyard in snort
machine the following commands are uses which are given below.
Listing 4.2: Configuring Barnyard with Snort by Following commands
42
sudo wget ‘ ‘ l i n k o f b a r n y a r d 2 f i l e ” <<<Line
1>>>
sudo t a r −z x f m a s t e r % Unta r t h e downloaded
f i l e <<<Line 2>>>
cd f i r n s y−b a r n y a r d 2 ∗ % Wil l go i n t h e
b a r n y a r d f o l d e r <<<Line 3>>>
sudo a u t o r e c o n f − f v i −I . / m4 <<<Line 4>>>
sudo . / c o n f i g u r e w i t h −mysql w i t h −mysql−l i b r a r i e s = / u s r / l i b / i386−l i n u x−gnu %
C o n f i g u r i n g b a r n y a r d wi th s n o r t c o n f i g u r i n g
f i l e <<<Line 5>>>
sudo make <<<Line 6>>>
sudo make i n s t a l l <<<Line 7>>>
sudo cp / ‘ ‘ l o c a t i o n o f b a r n y a r d 2 . con f f i l e ” /
b a r n y a r d 2 . con f / ‘ ‘ l o c a t i o n o f s n o r t f o l d e r ” /
% Coping t h e b a r n y a r d 2 . con f f i l e a t t h e
l o c a t i o n o f s n o r t f i l e s <<<Line 8>>>
sudo cp schemas / c r e a t e mysql / u s r / s r c <<<Line
9>>>
sudo mkdir / v a r / l o g / b a r n y a r d 2 <<<Line 10>>>
∗ Oinkmaster
Oinkmaster is a tool to update snort rules file. Written in perl, installing
perl is must. Some rules are own created and if not to modified then
defined in oinkmaster not to update. For updating the snort signature the
oinkmaster is uses. The configuration process of oinkmaster is shown
below.
Listing 4.3: Configuring Oinkmaster with Snort by Following commands
cd / ‘ ‘ a c c e s s o i n k m a s t e r d i r ” <<<Line 1>>>
43
sudo bash −c ‘ ‘ sudo . / c r e a t e−s idmap . p l / e t c /
s n o r t / r u l e s >/ e t c / s n o r t / s i d−msg . map” <<<
Line 2>>>
sudo / ‘ ‘ l o c a t i o n o f b a r n y a r d 2 d i r ” −c / ‘ ‘
l o c a t i o n o f s n o r t f o l d e r / b a r n y a r d 2 . con f −d
/ ‘ ‘ l o c a t i o n o f s n o r t l o g s ” −f s n o r t . l o g −w /
v a r / l o g / b a r n y a r d 2 / by log . waldo −C / ‘ ‘ l o c a t i o n
o f s n o r t d i r ” / c l a s s i f i c a t i o n . c o n f i g <<<
Line 3>>>
CRTL+C t o e x i t b a r n y a r d 2 <<<Line 4>>>
• Protection Machine(Windows 7)
In my project the windows 7 machine configured as protected machine. Because
working only on a single host machine, the protected machine is defined only one
machine in snort machine. This protected machine assumed as a complete network,
and a range of IP addresses is given in snort machine.
• Attacker Machine(Kali 2.0)
Kali Linux is used as attacker machine because in the Kali Linux so many tools are
available. The various tools are uses for committing various network attacks. The
attacks have done on the Windows 7 machine because snort machine protects this
machine and that is my task for protecting the network by using Snort tool.
• Traffic Generator Machine(Ubuntu 14.04)
Traffic generator machine has used for generating unwanted traffic on the network
because of testing snort working. The data is massive which has sent on the win-
dows machine.
4.2 Results of Single-Layer Intrusion Detection System
The Snort has configured for detecting and preventing malicious activity on the network.
The Snort has installed on the Ubuntu machine, and three other machines are also in-
stalled. The detailed explanation is given below.
44
4.2.1 Processing in Snort Machine
The Network Intrusion Detection and Prevention system have configured on the Ubuntu
machine. The malicious activity made by the intruder on the whole network is detected
and prevented by snort machine. The detection/prevention mechanism is based on the
signature which are given on Table [4]. The snort mechanism has explained below.
1. Snort Running Commands
• snort -c /“snort installed location”/snort.confCommand will give information regarding correctness of snort configuration.
• sudo snort -i eth0 -vNormally we get a live packet sniffing.
• sudo snort -A console -u snort -g snort -c /“snort installed location”/snort.conf-i eth0 -TDoing a config loading test.
2. IP address of Protecting system
Figure 13: IP address of protecting machine
In Figure [13] the IP address is shown, this IP address is of Windows 7 machine
because I want to protect Windows 7 machine from intrusions. The network which
is to be protected, the IP addresses of the network is defined in the snort machine.
We can also define any particular IP address within the same network.
45
Figure 14: Different types of Rules Saved in Machine
3. Pre-defined Signatures List
In the Figure [14] the list of defined rules have shown. These signature are de-
termined by previous attacks which had already detected. And some rules can be
user specified by the requirement. The update of snort signature is possible by the
Oinkmaster tool. Oinkmaster updates the snort signature regularly, if any new sig-
nature is added to the developer team, then it will be maintained by oinkmaster. The
oinkmaster has configured in the snort machine after configuring the Snort tool.
4. User defined Signature
In Figure [15] the rule is given for demo purpose, and it is the user defined signa-
ture. The location of user defined rule is /etc/snort/rules/zzalert.rules and shown
in Figure [14] in the circle. The signature is developed for ICMP packet testing;
46
Figure 15: User Define Rule For Demo Purpose
any ICMP attack is made on protected machine by intruder then the alert will be
shown in console mode. Which is shown in Figure [6]
5. MySQL Server
MySQL server is configured in snort machine for keeping the file in the database.
Keeping the dropped files in the database for analysis purpose. And this analysis
of dropped file will help to prevent the zero-day threat as well as reduce the false
negative rate. And in the network security, it is the proactive approach. Because the
intention of the intruder has known.
6. SYN Flood on Protected Machine
The attacker machine does SYN flood on the protected computer. The protection
has given by snort machine which has discussed in Figure [13]. The packets have
sent at the slight time of interval at targeted device is shown in Figure [16]. And
also, IP address is shown, this is the target machine’s IP address.
Figure 16: SYN Flood Record in Attacker Machine
47
Features of Snort Machine
Snort have the capability of detection and prevention of malicious activity. And also,
the packet dropping facility is there. When a malicious activity made by intruder then
according to defined signature the action will be taken by the machine. All the features of
Snort have explained below.
Detection of Malicious Activity Snort detection mechanism is based on signature and
anomaly based. The list of signatures in Figure [14] defined in the snort machine accord-
ing to the past malicious activity. The anomaly-based detection mechanism is based on
the sensing capability of the machine. The machine will sense the network and according
to the network behavior, action will take place. For testing the machine, User define rule
is generated which is shown in Figure [15]. And from attacker machine, the SYN flood
attack is happening on protected machine, and detection of that attack is successful which
is shown in Figure [17].
Prevetion of Malicious Activity Prevention of malicious activity by snort is next phase
of detection mechanism. The basic step of snort against malicious activity is detection
and after that prevention activity is happening. The prevention of malicious activity is
also based on two mechanisms, signature and anomaly based detection and prevention
mechanisms. The prevention of malicious activity generated dropped file which is shown
in Figure [18]. At this location only the malicious activity is located. So we can say this
is the malicious activity and prevention of malicious activity is the successful.
Alert Generating Snort machine generates the alert according to the defined signature
in the computer, in Figure [17] Generating alert means if malicious activity is happening
on the network then detection/prevention is completed by the snort machine. Then after
the process is informed by the snort machine to the network analyst by alarm or by a
trigger. Or it is also possible of seeing this process in the console mode of snort machine.
For running snort in live console mode, the command is used “Root access” snort -Aconsole -q -u snort -g snort -c /“snort installed location”/snort.conf -i eth0 and the
output of snort machine is shown in Figure [17].
48
Figure 17: Alert Generating in Live Mode
Dropped Packet File Generating Snort machine blocks the malicious activity, and
this action comes under dropped packets. Snort generate the dump packages, and these
packets are uses for analysis purpose. Snort machine gives the file name according to the
date and time of attack this is the inbuilt features of Snort machine. The dropped file is
shown in Figure [18]. And the default location of file generating is /var/log/snort.
Figure 18: File Generated when Malicious Activity Detected
Analysis of Dropped Packet Analysis of dropped packet is the source for preventing
zero-day threat because these are the malicious activity which is made by the intruder.
And intruder trying to exploit the vulnerability on the network by malicious activity means
something wrong with the network. The detailed study of dropped packet has explained
in the Topic [3.3.4]. The file has generated once the malicious activity has detected, the
file has shown in Figure [18]. What this file contains? Is the dropped data can give any
information against new attacks? All the valuable information which will help for pre-
venting zero-day threat are collected. The valuable information collection has discussed
49
in Topic [3.3.5]. In the Figure [19]. Seeing all the packets of SYN flood is possible but
here in the Figure, only a few of them is possible to see. With the help of “x less” only one
page has seen here.
Figure 19: Data in Dropped File
The destination IP addresses and other related information which has discussed in table 7
are collected and kept in the database for analyzing. Because of using different locations
for dropped packets storage, it is helpful for making analysis[3.3.7].
50
Chapter 5Conclusion and Future Work
We have implemented and tested a scheme of multi-layer scheme using the combination
of Snort and Suricata. It is observed that Snort detects and prevents more intrusions
compared to Suricata, and some more results are discussed below.
• Suricata detected more intrusions on Shell Codes.
• Snort detected more intrusions on Multifailed logins.
• Both snort and suricata equally detected intrusions on Denial of Service attack.
A drawback of our proposed design is generated more alert unnecessary in some cases.
Future works include reducing unnecessary alert by machine learning concepts.
51
Author’s Publications
• Vikash Kumar Saini, B.M. Mehtre. “Multi-Layer Intrusion Detection and Pre-
vention System: A New Approach”, Submitted to Journal of Computers & Secu-rity, ELSEVIER, in May, 2016.
52
References
[1] Aldeid, . Digital forensics, computer-forensics, malware-network-detection,
snort-suricata. https://www.aldeid.com/wiki/Suricata-vs-snort;
2013.
[2] Aldwairi, M., Conte, T., Franzon, P.. Configurable string matching hardware for
speeding up intrusion detection. SIGARCH Comput Archit News 2005;33:99–107.
[3] Ballmann, B.. Understanding Network Hacks Attack and Defense with Python.
springer, 2015.
[4] CTDP, . The computer technology documentation project. http:
//www.comptechdoc.org/independent/networking/protocol/
protnet.html; 2010.
[5] Denning, D.E.. An intrusion-detection model. In: Security and Privacy, 1986 IEEE
Symposium on. 1986. p. 118–118.
[6] Guimaraes, M., Murray, M.. Overview of intrusion detection and intrusion pre-
vention. In: Proceedings of the 5th Annual Conference on Information Security
Curriculum Development. ACM; 2008. p. 44–46.
[7] Harris, B., Hunt, R.. Review: Tcp/ip security threats and attack methods. Comput
Commun 1999;22:885–897.
53
[8] He, X.D., Ling, T.C.. Enhanced automated intrusion prevention in network secu-
rity. In: Computer Engineering and Technology (ICCET), 2010 2nd International
Conference on. volume 2; 2010. p. 286–291.
[9] Kizza, J.M.. GUIDE TO COMPUTER NETWORK SECURITY. volume 2nd
Edition. Springer, 2013.
[10] Lan, F., Chunlei, W., Guoqing, M.. A framework for network security situation
awareness based on knowledge discovery. In: Computer Engineering and Technol-
ogy (ICCET), 2010 2nd International Conference on. volume 1; 2010. p. 226–231.
[11] NEEDHAM, R., LAMPSON, B.. Network attack and defense. 2010.
[12] OISF, . Open source ids and ips and nsm engine. 2015.
[13] Pearson, . Working with snort rules. https://www.pearsonhighered.
com/samplechapter/0131407333.pdf; 2015.
[14] Prowell, S., Kraus, R., Borkin, M.. Seven Deadliest Network Attacks. volume 1st.
Syngress Publishing, 2010.
[15] Roesch, M.. Description of snort. https://www.snort.org.
[16] Sahay, R., Blanc, G., Zhang, Z., Debar, H.. Towards autonomic ddos mitigation
using software defined networking. In: SENT 2015 : NDSS Workshop on Security
of Emerging Networking Technologies. .
[17] Shin, S., Wang, H., Gu, G.. A first step toward network security virtualization:
From concept to prototype. IEEE Transactions on Information Forensics and Secu-
rity 2015;10(10).
[18] Umesh Hodeghatta Rao, U.N.. THE INFOSEC HANDBOOK, An Introduction to
Information Security, 2014.
[19] Yan, F., Jian-Wen, Y., Lin, C.. Computer network security and technology re-
search. In: 2015 Seventh International Conference on Measuring Technology and
Mechatronics Automation. 2015. p. 293–296.
54