fun with flow - carnegie mellon university...summarize silk flow records by a user-specified key...
TRANSCRIPT
![Page 1: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/1.jpg)
© 2010 Carnegie Mellon University
Fun with Flow
Richard Friedberg rf [at] cert.org
![Page 2: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/2.jpg)
2 © 2010 Carnegie Mellon University
Objectives
• Flow Primer • Why do I care? • Tools • Capabilities and examples • Almost live demo • Build it! • Where to go for more
![Page 3: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/3.jpg)
3 © 2010 Carnegie Mellon University
What is flow? • The simple version: a very brief summarization of a
network connection • The key values
— IP address source & destination — Protocol — Transport source & destination port
• Other stuff that can be useful — Time/date — Flags — Number of packets sent / received — Number of bytes sent / received
![Page 4: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/4.jpg)
4 © 2010 Carnegie Mellon University
Why flow?
Pros • Cheap • Quick (relative) • Plenty of tools • Minimal Privacy concerns • Scale • Provides a large data set
Cons • No payload • Provides a large data set
![Page 5: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/5.jpg)
5 © 2010 Carnegie Mellon University
Why do I care? • Network Billing • Utilization • Network modeling • Behavior-based analytics • QoS monitoring • Data Loss / Exfil Detection • Malware Detection
• Watchlist / Intrusion set monitoring
• Botnet tracking • Acceptable use monitoring • Application troubleshooting • Forensics / Incident Response
![Page 6: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/6.jpg)
6 © 2010 Carnegie Mellon University
Why flow? A perspective on storage Fully saturated 100mb link 1 month of storage = 50GB Fully saturated GigE link 1 month of storage = 500GB Fully saturated 10GigE link 1 month of storage = 5TB A little more reasonable than full packet capture http://tools.netsa.cert.org/releases/SiLK-Provisioning-v204.xls
![Page 7: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/7.jpg)
© 2010 Carnegie Mellon University
Tools
![Page 8: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/8.jpg)
8 © 2010 Carnegie Mellon University
YAF Yet Another Flowmeter
To be used with SiLK suite or other IPFIX compliant
tools
YAF IPFIX
Network libpcap capture
DAG capture
Napatech capture
Dumpfile input
![Page 9: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/9.jpg)
9 © 2010 Carnegie Mellon University
YAF – Flow and then some…
YAF
![Page 10: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/10.jpg)
10 © 2010 Carnegie Mellon University
YAF
• HTTP • SSH • SMTP • Gnutella • Yahoo Messenger • DNS • FTP • SSL/TLS • SLP • IMAP
• IRC • RTSP • SIP • RSYNC • PPTP • NNTP • TFTP, • Teredo • MySQL • POP3
Application labeling can recognize
![Page 11: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/11.jpg)
11 © 2010 Carnegie Mellon University
YAF Capture DNS – All or just Authoritative and NXDomain
responses HTTP
• Server, User-Agent, GET, Connection • HTTP, Referer, Location, Host • Content-Length, Age, Content-Type • Accept, Accept-Language,(Result Code)
FTP, IMAP, RTSP, SIP, SMTP, SSH Soon to be added X.509 Certificates Primarily from recognized SSL/TLS protocol
negotiations
![Page 12: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/12.jpg)
12 © 2010 Carnegie Mellon University
SiLK What is SiLK? Set of collection and analysis tools (30+) Accepts IPFIX, Netflow v5, Netflow v9 YAF
(Sensor) Network
SiLK Traffic Summaries
(Flow Data)
Command Line Tools
iSiLK GUI
Collection
![Page 13: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/13.jpg)
13 © 2010 Carnegie Mellon University
out
RootDir
Sensor2 silk.conf Sensor1
in inweb innull outweb outnull
year
day
month
hour
SiLK Repository
![Page 14: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/14.jpg)
14 © 2010 Carnegie Mellon University
Basic commands rwfilter Partition SiLK Flow records into one or more 'pass' and/or 'fail' output streams. rwfilter is the primary
tool for pulling flows from the data store. rwsort Sort SiLK Flow records using a user-specified key comprised of record attributes, and write the
records to the named output path or to the standard output. Users can define new key fields using plug-ins written in C or PySiLK.
rwcut Print the attributes of SiLK Flow records in a delimited, columnar, human-readable format. Users
can define new printable attributes using plug-ins written in C or PySiLK. rwuniq Bin (group) SiLK Flow records by a user-specified key comprised of record attributes and print the
total byte, packet, and/or flow counts for each bin. rwuniq can also print distinct source IP and destination IP counts. Users can define new key fields and value fields using plug-ins written in C or PySiLK.
rwcount Summarize SiLK Flow records across time, producing textual output with counts of bytes, packets,
and flow records for each time bin. rwstats Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute
values from the flow records that match each key, sort the results by the value to generate a Top-N or Bottom-N list, and print the results. Users can define new key fields and value fields using plug-ins written in C or PySiLK.
![Page 15: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/15.jpg)
15 © 2010 Carnegie Mellon University
iSiLK – Why? It helps me to choose SiLK tools
• Toolbar buttons allow quick perusal of tools It lets me avoid SiLK tool syntax
• Menus & other GUI elements show my choices It lets me avoid Linux command syntax and file
names • iSiLK organizes my data sets and results
It has an integrated graphing capability
![Page 16: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/16.jpg)
16 © 2010 Carnegie Mellon University
GUI - iSiLK
SiLK on Linux system
Flow
Repository
iSiLK on Windows system (or Mac or Linux)
SSH
![Page 17: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/17.jpg)
17 © 2010 Carnegie Mellon University
iSiLK Query Builder
![Page 18: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/18.jpg)
18 © 2010 Carnegie Mellon University
iSiLK Query Builder
![Page 19: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/19.jpg)
19 © 2010 Carnegie Mellon University
iSiLK Query Builder
![Page 20: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/20.jpg)
20 © 2010 Carnegie Mellon University
SiLK Analysis Pipeline SiLK was built to effectively query a repository
• Everything is retroactive Closest to real time is batched jobs Pipeline is a streaming analysis engine for SiLK flow
files • Versus retroactive SiLK tool analysis
“Near-time” alerting
![Page 21: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/21.jpg)
21 © 2010 Carnegie Mellon University
SiLK Analysis Pipeline
Filters • All flows go through each filter • Filter based on any field in flow record
Evals • Filtered flows passed to associated eval • Time sensitive state kept here
Alerts • Alerts created when eval thresholds met • Can be rate-limited
![Page 22: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/22.jpg)
22 © 2010 Carnegie Mellon University
SiLK Pipeline Filters Each evaluation gets its flows from one filter A filter can provide for multiple evaluations A single filter is specified in the configuration file for
each evaluation. Operators for any field in flow record
• <, <=, >, >=, ==, !=, IN_LIST, NOT_IN_LIST • Each filter can have multiple “anded” comparisons
![Page 23: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/23.jpg)
23 © 2010 Carnegie Mellon University
SiLK Pipeline Evaluations Can have time restrictions:
• Alert if “this” happens in any 5 minute period Made up of a number of independent checks
• E.g. Bytes > 1000 and packets > 500 in 5 minutes Each check can be limited by its own time window
• Examples — Sum of Packets > 1000 in 10 minutes — Number of Unique Source IP Addresses > 10 in an hour — Total Flow Count > 10000 in 1 minute
• If all checks meet threshold, the evaluation alerts
![Page 24: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/24.jpg)
24 © 2010 Carnegie Mellon University
SiLK Analysis Pipeline Capabilities Finite State Beacon Detection Sensor Outage Detection IPv6 Tunnel Detection Passive FTP Detection Watchlists Flow counts Flow field based capabilities (Can be combined)
• Sum or Average of the field value (bytes, packets, durations, etc)
• Proportion of flows with a given field value (TCP, Web, etc)
![Page 25: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/25.jpg)
25 © 2010 Carnegie Mellon University
SiLK Pipeline Alerts When deemed able to alert, they contain:
• The flow record • Evaluation name as identifier • Metrics that triggered alert and its threshold • Timestamp
Currently output to arcSight files Can output to files and logs
![Page 26: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/26.jpg)
26 © 2010 Carnegie Mellon University
Tools – Bringing it all together
YAF (Sensor) Network
IPFIX Mediator Protocol
Analysis
fixbuf
SiLK Traffic Summaries
(Flow Data)
Command Line Tools
iSiLK GUI
Pipeline SiLK
Collection Analysis Scripts
Alerts
![Page 27: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/27.jpg)
27 © 2010 Carnegie Mellon University
Prism Data examined on a schedule (e.g. once daily) Data diverted by first filter hit (filters designed to match known traffic) Partitions data by filter expressions, order matters! Visualization of known traffic examined for anomalies Remainder of data analyzed for anomalies, malicious activity
Data Manifold
Filter Filter … Filter
Visualization engine
Remainder
Known traffic Known traffic Known traffic
Flow data
![Page 28: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/28.jpg)
28 © 2010 Carnegie Mellon University
Prism Output example
![Page 29: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/29.jpg)
29 © 2010 Carnegie Mellon University
Beacons SiLK scripts or watchlists to identify possible beaconing Time series plots of possible beacons
![Page 30: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/30.jpg)
30 © 2010 Carnegie Mellon University
After the fact forensics/modeling Who talked to who? What else did this box / emplyoee do? Who else had this malware? Large outbound flows? Who are the top talkers on the network?
![Page 31: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/31.jpg)
31 © 2010 Carnegie Mellon University
Motivation • Improve transition and
uptake of analytics • Provide basic
visualization SOC analysts can use easily
• Integrate well into existing workflow (i.e. command-line)
• Inclusion in iSiLK
Rayon – Viz Library and Tools
![Page 32: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/32.jpg)
32 © 2010 Carnegie Mellon University
Data fusion DNS Data - Fast Flux domains, Phishing URLs Malware analysis - Network Touchpoints Simple analytics from flow feeding SIEM for
correlation
![Page 33: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/33.jpg)
33 © 2010 Carnegie Mellon University
Leveraging DPI YAF Inspector – Initial proof of concept for YAF’s
extended capabilities
![Page 34: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/34.jpg)
34 © 2010 Carnegie Mellon University
Leveraging DPI
![Page 35: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/35.jpg)
35 © 2010 Carnegie Mellon University
Yinspector – Almost Live!
![Page 36: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/36.jpg)
36 © 2010 Carnegie Mellon University
Yinspector – Almost Live!
![Page 37: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/37.jpg)
37 © 2010 Carnegie Mellon University
Yinspector – Almost Live!
![Page 38: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/38.jpg)
38 © 2010 Carnegie Mellon University
Yinspector – Almost Live!
![Page 39: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/39.jpg)
39 © 2010 Carnegie Mellon University
Yinspector – Almost Live! Top Talkers
![Page 40: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/40.jpg)
40 © 2010 Carnegie Mellon University
Build it! Tools available: http://tools.netsa.cert.org
• Source • RPM • LiveCD • Reference Data
![Page 41: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/41.jpg)
41 © 2010 Carnegie Mellon University
Where to go for more http://tools.netsa.cert.org
• Software • Documentation • Installation Guide, Analyst Handbook • Online (CBT) training • Wiki / Tooltips • Scripts
FloCon 2012 (mid-January) www.flocon.org Specifics coming soon We are hiring! Come see me. http://cert.org/jobs
![Page 42: Fun with Flow - Carnegie Mellon University...Summarize SiLK Flow records by a user-specified key comprised of record attributes, compute values from the flow records that match each](https://reader035.vdocuments.us/reader035/viewer/2022062609/60fc20fbe33e651a2f7cd235/html5/thumbnails/42.jpg)
42 © 2010 Carnegie Mellon University
Q&A http://tools.netsa.cert.org netsa-help [at] cert.org rf [at] cert.org