finding a needle in a pcap - sei digital library. yaf pcap features. rolling pcap dump • rotates...
TRANSCRIPT
![Page 1: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/1.jpg)
© 2013 Carnegie Mellon University
Finding a Needle in a PCAP
Emily Sarneso
Flocon 2015
![Page 2: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/2.jpg)
2
Copyright 2014 Carnegie Mellon University.
This material is based upon work supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.
References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University of its Software Engineering Institute.
NO WARRANTYTHIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material has been approved for public release and unlimited distribution except as restricted below.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].
Carnegie Mellon®, CERT ® , CERT Coordination Center® and Flocon® are registered marks of Carnegie Mellon University.
DM-0001893
![Page 3: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/3.jpg)
3
GoalDescribe a full packet capture solution that can quickly and efficiently produce requested information.
Show analysis capabilities of YAF, super_mediator, and SiLK.
Demonstrate PCAP features in YAF.
![Page 4: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/4.jpg)
4
PCAP ChallengesVolume (4Gbps):• 1 Hour: 1.7TB• 1 Day: 40.8TB• 1 Week: 285.6TB• 1 Month: 1.1PBData Stored on Sensors• Separate from analysisIndexing:• Timestamp Files• BPF Filters• GUI tools• Splunk
![Page 5: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/5.jpg)
5
YAF PCAP FeaturesRolling PCAP dump
• Rotates files using time or size.• Creates meta file with flows contained in each PCAP file.
Index a PCAP File• Uses flow key hash and start time.
PCAP per flow• Creates a PCAP file for each flow.• Use with BPF filters.
![Page 6: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/6.jpg)
6
Gh0st Rat Investigation
![Page 7: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/7.jpg)
7
Gh0stRemote Access TrojanFree source codeEasy to modifyDistinctive Network Signature
SignatureUsually 5 BYTES
Compressed Length
4 BYTES
Uncompressed Length
4 BYTES
ZLIBHDR0x789C
2 BYTES
Data
![Page 8: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/8.jpg)
8
Method29,000 (15G) PCAP samplesUse YAF to index and produce flow, DPIYAF Signatures
Flow
Enhanced Flow (DPI)
PCAP
![Page 9: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/9.jpg)
9
Tool setup
![Page 10: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/10.jpg)
10
Initial Results
![Page 11: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/11.jpg)
11
YAF SignaturesNorman ASA 2012 Report identifies 85 Gh0st variants
download01.norman.no/documents/ThemanyfacesofGh0stRat.pdf
![Page 12: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/12.jpg)
12
Results with YAF Signatures
![Page 13: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/13.jpg)
13
Super_mediatorA very configurable IPFIX mediatorCollects every IPFIX information element YAF can exportMultiple exportersMultiple collectors (v.1.0)
YAF SUPERMEDIATOR
flowcap
FileStorage
SiLK
![Page 14: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/14.jpg)
14
Super_mediator configurationListing application label first allowed for quick binning by variant.
Super_mediator Results:• 227,833 Total Bi-flows• 60,816 Bi-flows Gh0st• 86,053 Unidentified
ApplicationHashStimemsDomainSipDipSportDportProtocolvlanintIflagsUflagsRiflagsRuflagsPkts,Rpkts
BytesRbytesDatabytesRdatabytesSmallpktsRsmallpktsLargepktsRlargepktsNonemptypktsRnonemptypktsMaxsizeRmaxsizeFirsteight
![Page 15: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/15.jpg)
15
Finding a Pattern
![Page 16: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/16.jpg)
16
Analysis Part 1Remove unwanted flows from unidentified flows:
• Remove flows with source/destination port 138,139.• Remove flows with initialTCPFlags = ‘R’• Remove flows with dataByteCount = 0
Find flows with pattern:• No more than 1 small packet (forward), 0 reverse• Non-empty packets = 1 or 2 (forward), 1 reverse• maxPacketSize = reverseMaxPacketSize• firstEightPacketDirection = 0x02
Results:• 44,468 bi-flows removed• 37,500 bi-flows with pattern• 4,085 bi-flows did not follow pattern
![Page 17: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/17.jpg)
17
Finding Gh0st Variants and Signatures
![Page 18: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/18.jpg)
18
Analysis Part 2Run unidentified PCAP files through YAF again and export first 100 bytes of payload
![Page 19: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/19.jpg)
19
ResultsIdentified several signature variants of Gh0stFound 55 new Gh0st variantsCreated YAF Application Label for Gh0st
• Correctly identifies 97% of Gh0st traffic.
Collected over 3,000 unique domain names• Correlated with Gh0st variants.
![Page 20: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/20.jpg)
20
Searching for Gh0st in DEFCON CTF PCAP
![Page 21: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/21.jpg)
21
DEFCON CTF PCAP DataGoal: Test new Gh0st application labelDefcon CTF PCAP Data
• 409 GB• Separated by team and
day
![Page 22: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/22.jpg)
22
Investigating “Gh0st” in DEFCON
![Page 23: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/23.jpg)
23
YafMeta2PcapInput:
• Large PCAP file or list of PCAP files• PCAP meta file created by YAF• Flow key hash and start time
Output• PCAP file with desired flow
![Page 24: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/24.jpg)
24
DEFCON AnalysisUsed YAF signatures to determine other flows with “DmdT” and “eliza”
“eliza” was a text-based space economy simulator challenge at CTF
80% of DmdT traffic went to last place team.
![Page 25: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/25.jpg)
25
Method Comparison
PCAP ->FLOW
yafMeta2Pcap
DeterminePCAP(S)
that contain
flow
MergePCAP files w/
mergecap
PCAP
Write a BPF filter that will return
session
SeparateFlows
TCPDUMP YAF
![Page 26: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP](https://reader031.vdocuments.us/reader031/viewer/2022021901/5b83370b7f8b9a940b8c9fc5/html5/thumbnails/26.jpg)
26
Questions?CERT NetSA tools website:tools.netsa.cert.org
Contact:[email protected]
[email protected]@cert.org