Internet Traffic Analysis for Threat Detection

Internet Traffic Analysis for Threat Detection

Joshua Thomas, CISSP

Thomas Conley, CISSP

Ohio University

Communication Network Services

Joshua Thomas, CISSP

Thomas Conley, CISSP

Ohio University

Communication Network Services

AbstractAbstract

Useful logs may already exist at your institution.

Network transaction logging is a very useful, flexible, and inexpensive tool for network security.

Comprehensive network security relies on log collection and analysis.

Analysis of log files can be automated, and can provide information that can be the basis for prevention and response procedures.

Useful logs may already exist at your institution.

Network transaction logging is a very useful, flexible, and inexpensive tool for network security.

Comprehensive network security relies on log collection and analysis.

Analysis of log files can be automated, and can provide information that can be the basis for prevention and response procedures.

Start with what you haveStart with what you have

The collection and analysis of network transaction data is useful for a wide range of tasks Security management Network billing and accounting Network operations management Performance analysis

As a result, some form of network transaction logs may already exist within your institution, even if not specifically implemented for network security reasons.

The collection and analysis of network transaction data is useful for a wide range of tasks Security management Network billing and accounting Network operations management Performance analysis

As a result, some form of network transaction logs may already exist within your institution, even if not specifically implemented for network security reasons.

“Pointed stick”“Pointed stick”

Low cost, high returns

Simple to implement

Nonspecific, flexible

Non-restrictive

Low cost, high returns

Simple to implement

Nonspecific, flexible

Non-restrictive

Fundamental needFundamental need

Network transaction logs are arguably the most basic, necessary countermeasure in network security.

Logs should form the basis for decisions regarding other security initiatives.

Traffic analysis will be necessary to validate the performance of other security countermeasures.

Network transaction logs are arguably the most basic, necessary countermeasure in network security.

Logs should form the basis for decisions regarding other security initiatives.

Traffic analysis will be necessary to validate the performance of other security countermeasures.

Needs pyramid: Maslow’s HierarchyNeeds pyramid: Maslow’s Hierarchy

Biological and Physiological needs

Safety needs

Esteem needs

Belongingness and Love needs

Self-actualization

Needs pyramid: Network SecurityNeeds pyramid: Network Security

Network Transaction Logs

Security Staff

Firewalls

Host Security

IDS/IPS

Transparent monitorTransparent monitor

Acts as a passive device, gathering traffic and performance statistics at appropriate places in networks (server or client locations)

Is not necessarily a point of failure in your network

Cannot alter network traffic, as active devices such as firewalls or IDS/IPS systems.

However, monitoring can co-exist with other network security devices, such as IPS/IDS

Acts as a passive device, gathering traffic and performance statistics at appropriate places in networks (server or client locations)

Is not necessarily a point of failure in your network

Cannot alter network traffic, as active devices such as firewalls or IDS/IPS systems.

However, monitoring can co-exist with other network security devices, such as IPS/IDS

Transparent monitor: Simple setupTransparent monitor: Simple setup

Upstream ProviderUpstream Provider

HubHub

Network MonitorNetwork Monitor

NetworkNetwork

ScalableScalable

Mirroring traffic is relatively inexpensive.

Institutions may choose to capture as much data as possible and only perform limited analysis as needed.

There are appropriate solutions for implementing network transaction monitoring at just about every level of a network. Small lab environment Single department University border

Mirroring traffic is relatively inexpensive.

Institutions may choose to capture as much data as possible and only perform limited analysis as needed.

There are appropriate solutions for implementing network transaction monitoring at just about every level of a network. Small lab environment Single department University border

Transparent monitor: Large-scaleTransparent monitor: Large-scale

ISP 1ISP 1 ISP 2ISP 2

Network MonitorNetwork Monitor

Selective memorySelective memory

In order to be able to store and analyze high volumes of traffic, the memory demands must be reduced in some way.

In order to be able to store and analyze high volumes of traffic, the memory demands must be reduced in some way.

Selective memory: DepthSelective memory: Depth

IPS/IDS systems generally select certain transactions (via signature matching, etc.) for storage and analysis. In other words, only communications that match a selection criteria are recorded, and all other data is ignored.

IPS/IDS systems generally select certain transactions (via signature matching, etc.) for storage and analysis. In other words, only communications that match a selection criteria are recorded, and all other data is ignored.

!!!!

Selective memory: BreadthSelective memory: Breadth

Flow monitoring accounts for every transaction, but does not retain the content of the transactions.

Transactions contain both routing information and content. Only routing information is retained.

Applications that can capture this sort of transaction data include Argus, tcpdump, Ethereal, cflowd, etc.

Flow monitoring accounts for every transaction, but does not retain the content of the transactions.

Transactions contain both routing information and content. Only routing information is retained.

Applications that can capture this sort of transaction data include Argus, tcpdump, Ethereal, cflowd, etc.

Flow metricsFlow metrics

Metrics generally captured in network transaction logs include: Source, destination IP addresses (for IP traffic) Beginning, end times Packet count Byte count TTL (for IP traffic) TCP flags (for TCP/IP traffic) TCP state progression (for TCP/IP traffic) Base sequence numbers (for TCP/IP traffic)

Metrics generally captured in network transaction logs include: Source, destination IP addresses (for IP traffic) Beginning, end times Packet count Byte count TTL (for IP traffic) TCP flags (for TCP/IP traffic) TCP state progression (for TCP/IP traffic) Base sequence numbers (for TCP/IP traffic)

InferenceInference

Certain traffic characteristics are very useful in making inferences about the nature of the traffic.

Examples: Amount of bandwidth consumed Number of connection attempts Connections to unused address ranges

Certain traffic characteristics are very useful in making inferences about the nature of the traffic.

Examples: Amount of bandwidth consumed Number of connection attempts Connections to unused address ranges

AutomationAutomation

Identifying problems through inference can be automated.

Once the criteria has been clearly defined, then the tasks that were once done by humans can be performed by simple programs.

Once the identification of problems is automated, then those results can be fed into response procedures.

Identifying problems through inference can be automated.

Once the criteria has been clearly defined, then the tasks that were once done by humans can be performed by simple programs.

Once the identification of problems is automated, then those results can be fed into response procedures.

ExamplesExamples

Compare logs with blacklists, such as known- spyware or spam source IP lists

Examine traffic destined for non-populated subnets

Noise-floor analysis

TCP port usage

Compare logs with blacklists, such as known- spyware or spam source IP lists

Examine traffic destined for non-populated subnets

Noise-floor analysis

TCP port usage

Endless possibilitiesEndless possibilities

We are constantly discovering new uses for network transaction logs

We are constantly discovering new uses for network transaction logs

About our institutionAbout our institution

4,820 employees (1,069 full-time faculty) 20,143 students (18,497 full-time students) 90+ Mbps Internet bandwidth (2 ISP’s) 6,000,000,000+ packets per day

3,000,000,000+ source packets 3,000,000,000+ destination packets

2,400+ GB per day (500+ DVD-ROMs) 727 source GB per day 1,675 destination GB per day

~12 GB Argus log files generated per day, on average (0.6% of the total bytes represented)

4,820 employees (1,069 full-time faculty) 20,143 students (18,497 full-time students) 90+ Mbps Internet bandwidth (2 ISP’s) 6,000,000,000+ packets per day

3,000,000,000+ source packets 3,000,000,000+ destination packets

2,400+ GB per day (500+ DVD-ROMs) 727 source GB per day 1,675 destination GB per day

~12 GB Argus log files generated per day, on average (0.6% of the total bytes represented)

References/ResourcesReferences/Resources

RFC 2724, “RTFM: New Attributes for Traffic Flow Measurement.” (http://www.rfc-editor.org/rfc/rfc2724.txt)

Argus: http://www.qosient.com/argus

RFC 2724, “RTFM: New Attributes for Traffic Flow Measurement.” (http://www.rfc-editor.org/rfc/rfc2724.txt)

Argus: http://www.qosient.com/argus