linux memory analysis workshop session 1 andrew...
TRANSCRIPT
![Page 1: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/1.jpg)
Linux Memory Analysis Workshop – Session 1
Andrew Case
![Page 2: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/2.jpg)
Who Am I?
• Security Analyst at Digital Forensics Solutions
Also perform wide ranging forensics investigations
• Volatility Developer
• Former Blackhat, SOURCE, and DFRWS speaker
• Computer Science degree from UNO
• GIAC Certified Forensics Analyst (GCFA)
2
![Page 3: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/3.jpg)
Format of this Workshop
• I will be presenting the Linux kernel memory analysis capabilities of Volatility
• Along the way we will be seeing numerous examples of Linux kernel source code as well as Volatility’s plugins source code
• Following along with me while I use Volatility to recover data will get you the most out of this workshop
3
![Page 4: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/4.jpg)
Setting up Your Environment
4
![Page 5: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/5.jpg)
Agenda for Today’s Workshop1. Recovering Vital Runtime Information
2. Investigating Live CDs (Memory Analysis)
3. Detecting Kernel Rootkits
5
![Page 6: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/6.jpg)
Agenda for This Hour
• Memory Forensics Introduction
• Recovering Runtime Information
– Will discuss kernel internals necessary to recover processes, memory maps, loaded modules, etc
– Will discuss how these are useful/relevant to forensics & IR
– We will be recovering data with Volatility as we go
• Q&A / Comments
6
![Page 7: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/7.jpg)
7
Memory Forensics Introduction
![Page 8: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/8.jpg)
Introduction
• Memory analysis is the process of taking a memory capture (a copy of RAM) and producing higher-level objects that are useful for an investigation
• A memory capture has the entire state of the operating system as well as running applications
– Including all the related data structures, variables, etc
8
![Page 9: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/9.jpg)
The Goal of Memory Analysis
• The higher level objects we are interested in are in-memory representations of C structures, custom data structures, and other variables used by the operating system
• With these we can recover processes listings, filesystem information, networking data, etc
• This is what we will be talking about throughout the workshop
9
![Page 10: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/10.jpg)
Information Needed for Analysis
• The ability to:
1. Locate needed data structures in memory
2. Model those data structures offline
3. Report their contents
10
![Page 11: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/11.jpg)
Locating Data Structures• To locate static data structures, we use the
System.map file
– Contains the name and address of every static data structure used in the kernel
– Created in the kernel build process by using nm on the compiled vmlinux file
11
![Page 12: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/12.jpg)
Modeling Data Structures
• The parts of the Linux kernel we care about are written in C
• All data structures boil down to C structures
• These have a very simple in-memory representation (next slide)
12
![Page 13: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/13.jpg)
C Structures in Memory
• Source Code:
struct blah {
int i;
char c;
short s; };
struct blah *b = malloc(…);
13
• In Memory:
Lets say we have an instance of ‘b’ at 0x0
Then:
b->i goes from 0x0 to 0x4
b->c goes from 0x4 to 0x5
b->s goes from 0x5 to 0x7
![Page 14: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/14.jpg)
Modeling Structures• During analysis we want to automatically model
each C structure of interest
• To do this, we use Volatility’s dwarfparse.py: Builds a profile of C structures along with members,
types, and byte offsets
Records offsets of global variables
• Example structure definition'ClassObject': [ 0xa0, { Class name and size
'obj': [0x0, ['Object']], Member name, offset, and type
14
![Page 15: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/15.jpg)
15
Introducing Volatility
![Page 16: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/16.jpg)
Volatility• Most popular memory analysis framework
– Written in Python
– Open Source
– Supports Windows {XP, Vista, 7, 2003, 2008}
– Support Linux 2.6.9 to 2.6.3x on Intel and ARM
• Allows for analysis plugins to be easily written
• Used daily in real forensics investigations
• Will be the framework used in this workshop
16
![Page 17: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/17.jpg)
Volatility Object Manager
• Once we have a model of a kernel’s data structures (profiles) we can then just rely on Volatility
• Its object manager takes care of parsing the struct definitions, including types, and then providing them as requested
– Example on next slide
17
![Page 18: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/18.jpg)
Example Plugin Code
• Accessing a structure is as simple as knowing the type and offset
intval = obj.Object(“int”, offset=intOffset, ..)
• Volatility code to access ‘descriptor’ of an ‘Object’:
o = obj.Object("Object", offset=objectAddress, ..)
c = obj.Object("ClassObject", offset=o.clazz, …)
desc = linux_common.get_string(c.descriptor)
18
![Page 19: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/19.jpg)
Volatility Address Spaces
• Address spaces are used to translate virtual addresses to offsets within a memory capture
– Same process used to translate to physical addresses on a running OS
• Plugin developers simply need to pass the given address space to functions that need it
– Manual change only required to access userland(will see an example in a bit)
19
![Page 20: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/20.jpg)
Current Address Spaces
• x86 / x64
• Arm (Android)
• Firewire
• Windows Hibernation Files
• Crash Dumps
• EWF Files
20
![Page 21: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/21.jpg)
21
Recovering Runtime Information
![Page 22: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/22.jpg)
Runtime Information
• This rest of this session is focused on orderly recovery of data that was active at the time of the memory capture
• We will be discussing how to find key pieces of information and then use Volatility to recover them
22
![Page 23: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/23.jpg)
Information to be Recovered
• Processes
• Memory Maps
• Open Files
• Network Connections
• Network Data
• Loaded Kernel Modules
23
![Page 24: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/24.jpg)
Recovering Process Information
• Each process is represented by a task_struct
• Once a task_struct is located, all information about a process can be quickly retrieved
– Possible to do it through other methods, but much more convoluted
24
![Page 25: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/25.jpg)
Locating Processes – Method 1• init_task is the symbol for the task_struct of
“swapper”, the PID 0 process
– Statically initialized
• task_struct->tasks holds a linked list of all active processes
– NOT threads! (more on this later)
– Simply walking the list gives us a process listing
25
![Page 26: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/26.jpg)
Locating Processes – Method 2
• Newer kernels have a Process ID (pid) namespace
• The hash table can be located through init_pid_ns and then enumerated
• These hash tables are used to populate /proc/<pid>
26
![Page 27: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/27.jpg)
Wanted Per-Process Information
• Name and Command Line Arguments
• UID/GID/PID
• Starting/Running Time
• Parent & Child Processes
• Memory Maps & Executable File
• Open Files
• Networking Information
27
![Page 28: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/28.jpg)
Needed task_struct Members• Name
– char comm[TASK_COMM_LEN]; // 16
– Command line arguments in later slides
• User ID / Group ID
– Before 2.6.29
• uid and gid
– Since 2.6.29
• struct cred *cred;
• cred->uid and cred->gid
28
![Page 29: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/29.jpg)
task_struct Members Cont.
• Parent Process
– struct task_struct *real_parent;
• Child processes
– struct list_head children; /* list of my children */
• Process times
– utime, stime, start_time, real_starttm
29
![Page 30: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/30.jpg)
Recovery with Volatility
• Option 1:
– In: volatility/plugins/linux_task_list_ps.py
– Walks the task_struct->tasks list
• Option 2:
– In: volatility/plugins/linux_task_list_psaux.py
– Reads command line invocation from userland
• Will cover algorithm after discussing memory management structures
30
![Page 31: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/31.jpg)
Process Gathering Demo/Hands On• Will be using:
– linux_task_list_ps
– linux_task_list_psaux
31
![Page 32: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/32.jpg)
Process Memory Maps
• Viewed on a running system within /proc/<pid>/maps
• Lists all mappings within a process including:
– Mapped file, if any
– Address range
– Permissions
32
![Page 33: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/33.jpg)
Accessing the Mappings
• Each mapping is stored as a vm_area_struct
• Stored in two places:
– task_struct->mm->mm_rb
Red black tree of mappings
– task_struct->mm->mmap
List of mappings ordered by starting address
33
![Page 34: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/34.jpg)
Needed Members of vm_area_struct
• unsigned long vm_start, vm_end
– The starting and ending addresses of the mapping
• vm_area_struct vm_next
– The next vma for the process (linked list from mm->mmap)
• struct file vm_file
– If not NULL, points to the mapped file (shared library, open file, main executable, etc)
34
![Page 35: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/35.jpg)
Recovery with Volatility• Listing mappings implemented in
volatility/plugins/linux_proc_maps.py
• Analyzing specific mappings implemented in volatility/plugins/linux_dump_maps.py
– Can specify by PID or address
35
![Page 36: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/36.jpg)
Getting the Command Line Args
• Each processes’ mm_struct contains a number of variables that hold the address of the .text, .data, heap, stack, and command line args
• Command line args:
– Beginning address stored in arg_start
– End address stored in arg_end
– Arguments are stored in a list of null terminated “strings”
36
![Page 37: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/37.jpg)
Using ->mm to get **argv# switch pgd (address space)
tmp_dtb = self.addr_space.vtop(task.mm.pgd)
# create new address space
proc_as = self.addr_space.__class__(self.addr_space.base, self.addr_space.get_config(), dtb = tmp_dtb)
# read in command line argument buffer
argv = proc_as.read(task.mm.arg_start, task.mm.arg_end - task.mm.arg_start)
37
![Page 38: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/38.jpg)
Gathering Open Files• Want to emulate /proc/<pid>/fd
• task_struct->files->fdt->fd is array of file structures
• Each array index is the file descriptor number
• If an index is non-NULL then it holds an open file
• Use max_fds of the fdt table to determine array size
38
![Page 39: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/39.jpg)
Information Per-File
• Path information stored in the f_dentry and f_vfsmnt members– To get full path, need to emulate __d_path
function
• Inode information stored in f_dentry structure– Contains size, owner, MAC times, and other
metadata
• Recovering file contents in-memory requires use of the f_mapping member– Come back for session 2!
39
![Page 40: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/40.jpg)
Memory Maps and Open Files Demo
• Memory Maps
– Listing process mappings
– Acquiring the stack and heap from interesting processes
• Open Files
– Lists open files with their file descriptor number
40
![Page 41: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/41.jpg)
Networking Information
• The kernel contains a wealth of useful information related to network activity
• This info is immensely helpful in a number of forensics and incident response scenarios
41
![Page 42: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/42.jpg)
Netstat Plugin
• Used to emulate the netstat command
• This information is found on a running machine found in these /proc/net/ files:– tcp/tcp6
– udp/udp6
– unix
• We find open sockets by enumerating open files of processes and then finding those which are socket file descriptors
42
![Page 43: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/43.jpg)
Volatility’s linux_netstat.pyopenfiles = lof.linux_list_open_files.calculate(self)
# for every open filefor (task, filp, _i, _addr_space) in openfiles:
d = filp.get_dentry() # the files dentry
if filp.f_op == self.smap["socket_file_ops"] or filp.d.d_op == self.smap["sockfs_dentry_operations"]:
# it is a socket, can get the protocol informationiaddr = d.d_inodeskt = self.SOCKET_I(iaddr)inet_sock = obj.Object("inet_sock", offset = skt.sk, …)
43
![Page 44: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/44.jpg)
ARP Cache
• Emulates arp -a
• The ARP cache stores recently discovered IP and MAC address pairs
– It is what facilities ARP poisoning
• Recovery of this cache provides information on other machines the target machine was communicating with
44
![Page 45: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/45.jpg)
Recovering the ARP Cache
• Implemented in linux_arp.py
• This code walks the neigh_tables and their respective hash_buckets to recover neighbor structures
• These contain the device name, mac address, and corresponding IP address for each entry
45
![Page 46: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/46.jpg)
Routing Table
• Emulates route -n
• The routing table stores routing information for every known gateway device and its corresponding subnet
• The linux_route plugin recovers this information
46
![Page 47: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/47.jpg)
Routing Cache• Emulates route –C
• This cache stores recently determined source IP and gateway stores
• A great resource to determine recent network activity on a computer
47
![Page 48: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/48.jpg)
Network Recovery Demo/Hands On
• Many plugins!
48
![Page 49: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/49.jpg)
Dmesg• The simplest plugin in all of Volatility
• Simply locates and prints the kernel debug buffer
49
![Page 50: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/50.jpg)
Dmesg Plugin Codeptr_addr = self.smap["log_buf"]
# the bufferlog_buf_addr = obj.Object("long", offset = ptr_addr, vm =
self.addr_space)
# its lengthlog_buf_len = obj.Object("int", self.smap["log_buf_len"], vm =
self.addr_space)
# read in the bufferyield linux_common.get_string(log_buf_addr, self.addr_space,
log_buf_len)
50
![Page 51: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/51.jpg)
Loaded Kernel Modules
• Want to emulate the lsmod command
• Each module is represented by a structmodule
• Each active module is kept in the modules list
• We can simply walk the list to recover all needed information
51
![Page 52: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/52.jpg)
Information Per Module• char name[MODULE_NAME_LEN]
– The name of the module
• void module_init– .text + .data of init functions
• void module_core– .text + .data of core functions
• symtab/strtab– Symbol and string tables
• struct list_head list– Entry within the list of loaded modules
52
![Page 53: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/53.jpg)
Recovery with Volatility• In: volatility/plugins/linux_lsmod.py
• Volatility code:
mods_addr = self.smap["modules"]
modules = obj.Object("list_head",offset=mods_addr,)
for module in
linux_common.walk_list_head("module", "list", modules, …):
yield module
53
![Page 54: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/54.jpg)
Questions/Comments?
• Please fill out the feedback forms!
• Contact:
– @attrc
54
![Page 55: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/55.jpg)
Linux Memory Analysis Workshop – Session 2
Andrew Case
![Page 56: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/56.jpg)
Who Am I?
• Security Analyst at Digital Forensics Solutions
Also perform wide ranging forensics investigations
• Volatility Developer
• Former Blackhat, SOURCE, and DFRWS speaker
• Computer Science degree from UNO
• GIAC Certified Forensics Analyst (GCFA)
56
![Page 57: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/57.jpg)
Format of this Workshop
• I will be presenting the Linux kernel memory analysis capabilities of Volatility
• Along the way we will be seeing numerous examples of Linux kernel source code as well as Volatility’s plugins source code
• Following along with me while I use Volatility to recover data will get you the most out of this workshop
57
![Page 58: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/58.jpg)
Setting up Your Environment
58
![Page 59: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/59.jpg)
Agenda for Today’s Workshop1. Recovering Vital Runtime Information
2. Investigating Live CDs Through Memory Analysis
3. Detecting Kernel Rootkits
59
![Page 60: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/60.jpg)
Agenda for This Hour
• Discuss Live CDs and how they disrupt the normal forensics process
• Present research that enables traditional investigative techniques against live CDs
• We will be recovering files and data as we go along
• Q&A / Comments
60
![Page 61: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/61.jpg)
61
Introducing Volatility
![Page 62: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/62.jpg)
Volatility• Most popular memory analysis framework
– Written in Python
– Open Source
– Supports Windows {XP, Vista, 7, 2003, 2008}
– Support Linux 2.6.9 to 2.6.3x on Intel and ARM
• Allows for analysis plugins to be easily written
• Used daily in real forensics investigations
• Will be the framework used in this workshop
62
![Page 63: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/63.jpg)
Volatility Object Manager
• Once we have a model of a kernel’s data structures (profiles) we can then just rely on Volatility
• Its object manager takes care of parsing the struct definitions, including types, and then providing them as requested
– Example on next slide
63
![Page 64: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/64.jpg)
Example Plugin Code
• Accessing a structure is as simple as knowing the type and offset
intval = obj.Object(“int”, offset=intOffset, ..)
• Volatility code to access ‘descriptor’ of an ‘Object’:
o = obj.Object("Object", offset=objectAddress, ..)
c = obj.Object("ClassObject", offset=o.clazz, …)
desc = linux_common.get_string(c.descriptor)
64
![Page 65: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/65.jpg)
Volatility Address Spaces
• Address spaces are used to translate virtual addresses to offsets within a memory capture
– Same process used to translate to physical addresses on a running OS
• Plugin developers simply need to pass the given address space to functions that need it
– Manual change only required to access userland(will see an example in a bit)
65
![Page 66: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/66.jpg)
Current Address Spaces
• x86 / x64
• Arm (Android)
• Firewire
• Windows Hibernation Files
• Crash Dumps
• EWF Files
66
![Page 67: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/67.jpg)
67
Live CD Introduction
![Page 68: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/68.jpg)
Normal Forensics Process
Acquire Disk Image
Verify Image
Process Image
Perform Investigation
Obtain Hard Drive
68
![Page 69: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/69.jpg)
Traditional Analysis Techniques
• Timelining of activity based on MAC times
• Hashing of files
• Indexing and searching of files and unallocated space
• Recovery of deleted files
• Application specific analysis
– Web activity from cache, history, and cookies
– E-mail activity from local stores (PST, Mbox, …)
69
![Page 70: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/70.jpg)
Problem of Live CDs
• Live CDs allow users to run an operating system and all applications entirely in RAM
• This makes traditional digital forensics (examination of disk images) impossible
• All the previously listed analysis techniques cannot be performed
70
![Page 71: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/71.jpg)
The Problem Illustrated
Acquire Disk Image
Verify Image
Process Image
Perform Investigation
Obtain Hard Drive
71
![Page 72: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/72.jpg)
No Disks or Files, Now What?• All we can obtain is a memory capture
• With this, an investigator is left with very limited and crude analysis techniques
• Can still search, but can’t map to files or dates
– No context, hard to present coherently
• File carving becomes useless
– All fragmented in memory
• Good luck in court
72
![Page 73: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/73.jpg)
People Have Caught On…• The Amnesic Incognito Live System (TAILS) [1]
– “No trace is left on local storage devices unless explicitly asked.”
– “All outgoing connections to the Internet are forced to go through the Tor network”
• Backtrack [2]
– “ability to perform assessments in a purely native environment dedicated to hacking.”
73
![Page 74: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/74.jpg)
What It Really Means…
• Investigators without deep kernel internals knowledge and programming skill are basically hopeless
• It is well known that the use of live CDs is going to defeat most investigations
– Main motivation for this work
– Plenty anecdotal evidence of this can be found through Google searches
74
![Page 75: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/75.jpg)
What is the Solution?
• Memory Analysis!
• It is the only method we have available…
• This Analysis gives us:
– The complete file system structure including file contents and metadata
–Deleted Files (Maybe)
–Userland process memory and file system information
75
![Page 76: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/76.jpg)
76
Recovering the Filesystem
![Page 77: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/77.jpg)
• Steps needed to achieve this goal:
1. Understand the in-memory filesystem
2. Develop an algorithm that can enumerate directory and files
3. Recover metadata to enable timelining and other investigative techniques
77
Goal 1: Recovering the File System
![Page 78: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/78.jpg)
The In-Memory Filesystem• AUFS (AnotherUnionFS)
– http://aufs.sourceforge.net/
– Used by TAILS, Backtrack, Ubuntu 10.04 installer, and a number of other Live CDs
– Not included in the vanilla kernel, loaded as an external module
78
![Page 79: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/79.jpg)
AUFS Internals• Stackable filesystem
• Presents a multilayer filesystem as a single one to users
• This allows for files created after system boot to be transparently merged on top of read only CD
• Each layer is termed a branch
• In the live CD case, one branch for the CD, and one for all other files made or changed since boot
79
![Page 80: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/80.jpg)
Walkthrough
• Explore AUFS on a running system
80
![Page 81: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/81.jpg)
AUFS Userland View of TAILS
# cat /proc/mountsaufs / aufs rw,relatime,si=4ef94245,noxino
/dev/loop0 /filesystem.squashfs squashfs
tmpfs /live/cow tmpfs
tmpfs /live tmpfs rw,relatime
# cat /sys/fs/aufs/si_4ef94245/br0
/live/cow=rw
# cat /sys/fs/aufs/si_4ef94245/br1
/filesystem.squashfs=rr
81
Mount points relevant to AUFS
The mount point of each AUFS branch
![Page 82: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/82.jpg)
Forensics Approach
• No real need to copy files from the read-only branch
– Just image the CD
• On the other hand, the writable branch contains every file that was created or modified since boot
– Including metadata
– No deleted ones though, more on that later
82
![Page 83: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/83.jpg)
83
Linux Internals
![Page 84: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/84.jpg)
Struct Dentry• Represents a directory entry (directory, file, …)
• Contains the name of the directory entry and a pointer to its inode structure
• Members used:
– d_inode – pointer to inode structure
– d_name – name of file
84
![Page 85: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/85.jpg)
Struct Inode• FS generic, in-memory representation of a disk
inode
• Contains address_space structure that links an inode to its file’s pages
• Also provides metadata:
– MAC Times
– Size
– Owner
– more…
85
![Page 86: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/86.jpg)
Struct Address_Space
• Links physical pages together into something useful
• Holds the search tree of the struct pages for a file
This pages then lead to physical pages
86
![Page 87: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/87.jpg)
Linux Internals Overview II
• Page Cache
– Used to store struct page structures that correspond to physical pages
– address_space structures contain linkage into the page cache that allows for ordered enumeration of all physical pages pertaining to an inode
• Tmpfs
– In-memory filesystem
– Used by TAILS to hold the writable branch
87
![Page 88: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/88.jpg)
Enumerating Directories
• Once we can enumerate directories, we can recover the whole filesystem
• Not as simple as recursively walking the children of the file system’s root directory
• AUFS creates hidden dentrys and inodes in order to mask branches of the stacked filesystem
• Need to carefully interact between AUFS and tmpfs structures
88
![Page 89: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/89.jpg)
Directory Enumeration Algorithm1) Walk the super blocks list until the “aufs”
filesystem is found• This contains a pointer to the root dentry
2) For each child dentry, test if it represents a directory
If the child is a directory:• Obtain the hidden directory entry (next slide)
• Record metadata and recurse into directory
If the child is a regular file:• Obtain the hidden inode and record metadata
89
![Page 90: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/90.jpg)
Getting the Superblock
# get a reference the list of super blocks
sbptr = obj.Object("Pointer", offset = self.smap["super_blocks"], …)
# the list
sb_list = obj.Object("list_head", offset = sbptr.v(),…)
# walk the list and find “aufs”
for sb in walk_list_head("super_block","s_list", sb_list,
if sb.s_id.startswith("aufs"):
return sb.s_root
90
![Page 91: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/91.jpg)
The Initial Code
# walk the directory and get the hidden dentry
def calculate(self):
# get the superblock
root_dentry = self.get_aufs_sb()
# walk every dentry
for dentry in walk_list_head("dentry", "d_u", root_dentry.d_subdirs, ...):
h_dentry = self.get_h_dentry(dentry, 0)
91
![Page 92: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/92.jpg)
Obtaining a Hidden Directory
struct dentry{
d_inoded_named_subdirsd_fsdata
}
struct au_dinfo{
au_hdentry}
Branch
0
1 Pointer
Pointer
Dentry
92
• Each kernel dentry stores a pointer to an au_dinfostructure inside its d_fsdata member
• The di_hdentry member of au_dinfo is an array of au_hdentry structures that embed regular kernel dentrys
![Page 93: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/93.jpg)
get_h_dentry(dentry)
# the hidden “fsdata” pointerdinfo = obj.Object("au_dinfo", offset=dentry.d_fsdata,
# get the array of hidden dentrysdentry_array = obj.Object("Array", offset=dinfo.di_hdentry, "au_hdentry", count=2)
# grab the hidden dentry at branch ‘index’h_dentry = dentry_array[index].hd_dentry
93
![Page 94: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/94.jpg)
Obtaining Metadata
• All useful metadata such as MAC times, file size, file owner, etc is contained in the hidden inode
• This information is used to fill the stat command and istat functionality of the Sleuthkit
• Timelining becomes possible again
94
![Page 95: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/95.jpg)
Obtaining a Hidden Inode
struct aufs_icntnr{
iinfoinode
}
struct au_iinfo{
ii_hinode}
Branch
0
1 Pointer
Pointer
struct inode
95
• Each aufs controlled inode gets embedded in an aufs_icntnr
• This structure also embeds an array of au_hinodestructures which can be indexed by branch number to find the hidden inode of an exposed inode
![Page 96: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/96.jpg)
Demo/Hands-On
• Will recover the filesystem structure and metadata
• Will examine with a few FS analysis tools
• Will discuss how to keep forensically sound and discuss limitations
96
![Page 97: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/97.jpg)
Goal 2: Recovering File Contents
• The size of a file is kept in its inode’s i_size member
• An inode’s page_tree member is the root of the radix tree of its physical pages
• In order to recover file contents this tree needs to be searched for each page of a file
• The lookup function returns a struct pagewhich leads to the backing physical page
97
![Page 98: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/98.jpg)
Recovering File Contents Cont.• Indexing the tree in order and gathering of
each page will lead to accurate recovery of a whole file
• This algorithm assumes that swap isn’t being used
– Using swap would defeat much of the purpose of anonymous live CDs
• Tmpfs analysis is useful for every distribution
– Many distros mount /tmp using tmpfs, shmem, etc
98
![Page 99: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/99.jpg)
• Discussion:
1. Formulate Approach
2. Discuss the kmem_cache and how it relates to recovery
3. Attempt to recover previously deleted file and directory names, metadata, and file contents
99
Goal 3: Recovering Deleted Info
![Page 100: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/100.jpg)
Approach
• We want orderly recovery
• To accomplish this, information about deleted files and directories needs to be found in a non-standard way
– All regular lists, hash tables, and so on lose track of structures as they are deleted
• Need a way to gather these structures in an orderly manner— kmem_cache analysis to the rescue!
100
![Page 101: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/101.jpg)
Recovery though kmem_cache analysis
• A kmem_cache holds all structures of the same type in an organized manner
– Allows for instant allocations & deallocations
– Used for handling of process, memory mappings, open files, and many other structures
• Implementation controlled by allocator in use
– SLAB and SLUB are the two main ones
101
![Page 102: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/102.jpg)
kmem_cache Internals• Both allocators keep track of allocated and
previously de-allocated objects on three lists:
– full, in which all objects are allocated
– partial, a mix of allocated and de-allocated objects
– free, previously freed objects*
• The free lists are cleared in an allocator dependent manner
– SLAB leaves free lists in-tact for long periods of time
– SLUB is more aggressive
102
![Page 103: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/103.jpg)
kmem_cache Illustrated• /proc/slabinfo contains information about
each current kmem_cache
• Example output:
# name <active_objs> <num_objs>
task_struct 101 154
mm_struct 76 99
filp 901 1420
103
The difference between num_objs and active_objs is how many free objects are being tracked by the kernel
![Page 104: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/104.jpg)
Recovery Using kmem_cache Analysis
• Enumeration of the lists with free entries reveals previous objects still being tracked by the kernel
– The kernel does not clear the memory of these objects
• Our previous work has demonstrated that much previously de-allocated, forensically interesting information can be leveraged from these caches [4]
104
![Page 105: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/105.jpg)
Recovering Deleted Filesystem Structure
• Both Linux kernel and aufs directory entries are backed by the kmem_cache
• Recovery of these structures reveals names of previous files and directories
– If d_parent member is still in-tact, can place entries within file system
105
![Page 106: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/106.jpg)
Recovering Previous Metadata
• Inodes are also backed by the kmem_cache
• Recovery means we can timeline again
• Also, the dentry list of the AUFS inodes still have entries (strange)
– This allows us to link inodes and dentrys together
– Now we can reconstruct previously deleted file information with not only file names & paths, but also MAC times, sizes, inode numbers, and more
106
![Page 107: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/107.jpg)
Recovering File Contents – Bad News• Again, inodes are kept in the kmem_cache
• Unfortunately, page cache entries are removed upon deallocation, making lookup impossible– A large number of pointers would need to stay in-
tact for this to work
• This removes the ability to recover file contents in an orderly manner
• Other ways may be possible, but will require more research
107
![Page 108: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/108.jpg)
Summary of File System Analysis
• Can completely recover the in-memory filesystem, its associated metadata, and all file contents
• Ordered, partial recovery of deleted file names and their metadata is also possible
• Traditional forensics techniques can be made possible against live CDs
– Making such analysis accessible to all investigators
108
![Page 109: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/109.jpg)
Questions/Comments?
• Please fill out the feedback forms!
• Contact:
– @attrc
109
![Page 110: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/110.jpg)
Linux Memory Analysis Workshop – Session 3
Andrew Case
![Page 111: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/111.jpg)
Who Am I?
• Security Analyst at Digital Forensics Solutions
Also perform wide ranging forensics investigations
• Volatility Developer
• Former Blackhat, SOURCE, and DFRWS speaker
• Computer Science degree from UNO
• GIAC Certified Forensics Analyst (GCFA)
111
![Page 112: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/112.jpg)
Format of this Workshop
• I will be presenting the Linux kernel memory analysis capabilities of Volatility
• Along the way we will be seeing numerous examples of Linux kernel source code as well as Volatility’s plugins source code
• Following along with me while I use Volatility to recover data will get you the most out of this workshop
112
![Page 113: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/113.jpg)
Agenda for Today’s Workshop1. Recovering Vital Runtime Information
2. Investigating Live CDs Through Memory Analysis
3. Detecting Kernel Rootkits
113
![Page 114: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/114.jpg)
Agenda for This Hour
• This session will be a walkthrough of kernel-mode rootkits under Linux
• We will discussing the techniques used by rootkits to stay hidden and how the Volatility modules uncover them
• I will also be presenting previously never disclosed rootkit techniques developed for this workshop
• Q&A / Comments
114
![Page 115: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/115.jpg)
115
Introducing Volatility
![Page 116: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/116.jpg)
Volatility• Most popular memory analysis framework
– Written in Python
– Open Source
– Supports Windows {XP, Vista, 7, 2003, 2008}
– Support Linux 2.6.9 to 2.6.3x on Intel and ARM
• Allows for analysis plugins to be easily written
• Used daily in real forensics investigations
• Will be the framework used in this workshop
116
![Page 117: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/117.jpg)
Volatility Object Manager
• Once we have a model of a kernel’s data structures (profiles) we can then just rely on Volatility
• Its object manager takes care of parsing the struct definitions, including types, and then providing them as requested
– Example on next slide
117
![Page 118: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/118.jpg)
Example Plugin Code
• Accessing a structure is as simple as knowing the type and offset
intval = obj.Object(“int”, offset=intOffset, ..)
• Volatility code to access ‘descriptor’ of an ‘Object’:
o = obj.Object("Object", offset=objectAddress, ..)
c = obj.Object("ClassObject", offset=o.clazz, …)
desc = linux_common.get_string(c.descriptor)
118
![Page 119: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/119.jpg)
Volatility Address Spaces
• Address spaces are used to translate virtual addresses to offsets within a memory capture
– Same process used to translate to physical addresses on a running OS
• Plugin developers simply need to pass the given address space to functions that need it
– Manual change only required to access userland(will see an example in a bit)
119
![Page 120: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/120.jpg)
Current Address Spaces
• x86 / x64
• Arm (Android)
• Firewire
• Windows Hibernation Files
• Crash Dumps
• EWF Files
120
![Page 121: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/121.jpg)
121
Linux Kernel-Mode Rootkits
![Page 122: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/122.jpg)
Introduction
• I promise not to bore you with information from ~2002 Phrack articles…
• Rootkits target two types of data:
1. Static
Easy to implement and easy to detect
2. Dynamic
Harder to implement and harder to detect
122
![Page 123: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/123.jpg)
Static-Data Altering Rootkits
• These rootkits target data structures that are easy to modify, but are also effective at hiding activity
• Common technique types include:
– Directly overwriting instructions in memory (.text)
– Overwriting the system call & interrupt descriptor tables
– Overwriting members of global data structures
123
![Page 124: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/124.jpg)
Type 1: Overwriting .text
• Very popular as its easy to implement and makes hiding data easy
• Rootkits alter running instructions for a few reasons:
– To gain control flow
– To filter data (add, modify, delete) to stay hidden
– To implement “triggers” so that userland code can make requests
124
![Page 125: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/125.jpg)
Detecting Code Overwrites
• The compiled code of the kernel is static– One exception is covered next
• The compiled kernel (vmlinux) is an ELF file– All functions, including their name, instructions,
and size can be gathered from debug information
• This information can then be compared to what is in memory
• Any alteration points to malicious (or broken) software
125
![Page 126: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/126.jpg)
SMP Alternatives• There is one circumstance when runtime
modifications happen in the Linux kernel
• When the computer first boots and only one processor is active, all multi-core synchronization primitives are NOP’ed out
• When more than one CPU comes online, the kernel then has to rewrite these instructions with their SMP-safe counterparts to maintain concurrency
126
![Page 127: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/127.jpg)
SMP Alts. Cont.
• These alternative instructions are kept for performance reasons
– No reason to get, set, and check SMP locks if only one CPU is active
• The alternative instructions and their target location are stored within the vmlinux file
• We can gather this information and use it for accurate .text modification checking
127
![Page 128: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/128.jpg)
Type 2: System Call & IDT Overwriting
• To avoid being detected when overwriting .text, rootkits started modifying the tables used to service system calls and interrupts
• This allows for a rootkit’s code to easily filter the data received and returned by native kernel functions
128
![Page 129: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/129.jpg)
Attack Examples
• Overwrite the read system call and filter out the rootkit’s logging data unless a specific register contains a magic value
• Overwrite the stat system call to hide files from userland anti-rootkit applications
• Many more possibilities…
129
![Page 130: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/130.jpg)
Detecting These Attacks
• The IDT and the system call table are simply C arrays
• They can be copied from the clean vmlinux file and then compared to the values in memory
• Will easily detect that the table has been altered and which entries were modified
130
![Page 131: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/131.jpg)
Type 3: Overwriting Data Structures
• Popularized by the adore[1] rootkit, this attack overwrites function pointers of global data structures to filter information
• Adore overwrites the readdir member of the file_operations structure for the proc and rootfilesystems
– The replacement function filters out files on a pattern used by the rootkit, effectively hiding them from userland
131
![Page 132: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/132.jpg)
Other Common Attacks• Overwriting structure members used to
display information through /proc
– Info files in /proc use the seq_operations interface
– Hijacking the show member of this structure allows for trivial filtering of information
• Possible targets
– Loaded modules list
– Networking connections (netstat)
– Open files (lsof)
132
![Page 133: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/133.jpg)
Detecting these Attacks
• We take a generic approach
• During the profile creation stage, we filter for a number of commonly targeted structure types
– For variables found, we then copy the statically set values of each member that may be hijacked
• This ensures that all instances of those structures are checked for malicious tampering
133
![Page 134: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/134.jpg)
Targeted Structure Types
• file_operations
• inode_operations
• vm_operations_struct
• address_space_operations
• Operation structures used by /proc
134
![Page 135: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/135.jpg)
Hands On
• We will look at a memory image infected with a rootkit that uses a number of static-data altering techniques
• Volatility will show us the data structures infected
135
![Page 136: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/136.jpg)
Dynamic-Data Modification Rootkits
• Rootkits that modify dynamic data are much more interesting than those that alter static data
– Require more skill on part of the rootkit developer
– Require more complicated analysis and detection capabilities on the detector
• Cannot be detected by using System.map or vmlinux
– Need deep parsing of in-kernel data structures
136
![Page 137: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/137.jpg)
Attacks & Defenses
• The rest of this session will cover attacks and defense related to dynamic data altering
– Most of these attacks are new (developed for this workshop) to highlight the stealth ability of these types of attacks
• But first, we need to learn about the kmem_cache
– Will be used extensively by our detection mechanisms
137
![Page 138: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/138.jpg)
The kmem_cache
• The kmem_cache is a facility that provides a consistent and fast interface to allocate/de-allocate objects (C structures) of the same size
• The implementation of each cache is provided by the system allocator
– SLAB and SLUB are the two main ones
138
![Page 139: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/139.jpg)
kmem_cache Internals• Both allocators keep track of allocated and
previously de-allocated objects on three lists:
– full, in which all objects are allocated
– partial, a mix of allocated and de-allocated objects
– free, previously freed objects*
• The free lists are cleared in an allocator dependent manner
– SLAB leaves free lists in-tact for long periods of time
– SLUB is more aggressive
139
![Page 140: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/140.jpg)
kmem_cache Illustrated• /proc/slabinfo contains information about
each current kmem_cache
• Example output:
# name <active_objs> <num_objs>
task_struct 101 154
mm_struct 76 99
filp 901 1420
140
The difference between num_objs and active_objs is how many free objects are being tracked by the kernel
![Page 141: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/141.jpg)
Utilizing the kmem_cache
• All of the allocated objects backed by a particular cache can be found on the full and partial lists
– The one caveat is SLUB without debugging on
– Every distro checked enables SLUB debugging
– Might be possible to find all references even with debugging off
141
![Page 142: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/142.jpg)
The Idea Behind the Detection• Dynamic-data rootkit methods work by
removing structures from lists, hash tables, and other data structures
• To detect this tampering, we can take a particular cache instance and use this as a cross-reference to other stores
• Any structure in the kmem_cache list, but not in another, is hidden
– Inverse holds as well
![Page 143: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/143.jpg)
Why the Detection Works• All instances of a structure must be backed by
the caches
• These caches work similar to an immutable store:
– Structures of the specific type cannot be hidden from it
• A few possible attack scenarios exist, but will not work undetected
![Page 144: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/144.jpg)
Detection Subversion Scenarios1. Allocating outside the cache
• Will be detected by the inverse comparisons
2. Allocating in the cache and then removing from it• Very difficult to do and will result in detection as
with scenario #1
3. Allocating in the cache and then setting the entry as free• The structure will be overwritten on next
allocation
![Page 145: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/145.jpg)
Our First New Attack• The first developed attack was hiding
processes from /proc
• A number of rootkit detection systems work by trying to enumerate /proc/[1-65535] and then compare the output to ps
• The numbered proc directories are backed by their respective PID namespace and number
145
![Page 146: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/146.jpg)
Process Background• All tasks are allocated from the
task_struct_cachep
• Tasks are represented by a task_struct
• It is possible to remove a task_struct from the structures /proc uses to fill /proc/<fd>/ while still allowing it to run
146
![Page 147: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/147.jpg)
The Attack
• As simple as removing from the namespace
• Code, where p is the task_struct we want to hide:
pid = p->pids[PIDTYPE_PID].pid; // get the pid ref
detach_pid(p, PIDTYPE_PID); // take out of PID // group
• The process will no longer show up in /proc/<pid> lookups
147
![Page 148: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/148.jpg)
Detection
• We gather processes from a number of places before comparing to those in the cache– Implemented in linux_check_processes.py
1) Each task_struct holds a pointer into the tasks list
2) The run queue, where scheduled processes wait to execute
3) The PID cache, where we just removed our process from
148
![Page 149: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/149.jpg)
Hands-on/Demo
• We will now investigate hidden processes and look at the corresponding Volatility detection code
149
![Page 150: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/150.jpg)
Next Attack: Memory Maps
• The next attack hides memory maps from /proc/<pid>/maps
– This file is used to list every mapped address range in a process
• Each mapping is represented by a vm_area_struct and they are kept in two places:
– The mmap list of the processes’ mm_struct
– The mm_rb tree of the mm_struct
150
![Page 151: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/151.jpg)
mm_struct and vm_area_struct
• mm_struct
– Used to represent the memory management structures of a process
• vm_area_struct
– Represents a single memory mapping within a process
151
![Page 152: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/152.jpg)
The Attack
• Inspection of a maps file makes attacks such as shared library injection very noticeable
– The full path of the mapped binary plus its data and code sections will be visible
• To hide maps, we need to:
– Remove the vma from the mmap lists
– Fixup the structures that account for paging
• This will hide the map and allow for the targeted process to exit cleanly
152
![Page 153: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/153.jpg)
Detection
• Implemented in: linux_check_mappings.py
• The first step is to gather all active VMAs for a process so they can be compared against those in the cache
• The problem is that the VMAs are anonymous
– No immediate linkage to a specific process
153
![Page 154: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/154.jpg)
Detection Cont.• To work around this, we rely on the fact that
vmas keep a back pointer to their owning mm_struct in the vm_mm member
• Using this, we can gather all the vmas for a specific process and then compare against the cache
• Can you think of a bypass in this detection?
154
![Page 155: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/155.jpg)
Preventing Malware Tampering
• Since we rely on vm_mm, malware could try to avoid this detection by changing vm_mm
• Possible attempts:
1. Set vm_mm to some invalid value (NULL, etc)
2. Set vm_mm to another processes’s mm_struct
• Will still be detected:
All mm_structs are also in a kmem_cache
Comparing the list of vm_mm values to this cache will reveal avoidance attempts
155
![Page 156: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/156.jpg)
Next Attack: Open files• The /proc/<pid>/fd directory contains a
symlink per open file
– The symlink name is the file descriptor number
• Used by a number of utilities (lsof) and anti-rootkit applications to detect files being accessed
• To remain stealthy, this directory listing needs to be filtered
156
![Page 157: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/157.jpg)
The Attack• A processes’ file descriptors are stored in an
array of file structures indexed by file descriptor number
• All non-null indexes are treated as open files
– NULL entries are skipped
157
![Page 158: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/158.jpg)
The Hiding Code
idx = loop_counter; // the file desc to test
file = p->files->fdt->fd[idx]; // the file struct
if (file)
{
fn = d_path(…); // get the full path of file
if(!IS_ERR(fn) && strcmp(fn,"/tmp/hidefile.txt"))
fdt->fd[i] = NULL;
}
158
![Page 159: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/159.jpg)
Detection
• As with the process detection algo, finding all open files requires gathering from a number of sources:
– The (non-hidden) open files per-process
– The vm_file structures used to memory map files
– All swap files
• We then compare these against the filp_cachep kmem_cache
159
![Page 160: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/160.jpg)
Demo
• Will demo showing detection of dynamic modification of:
– Mappings
– Open Files
160
![Page 161: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/161.jpg)
Questions/Comments?
• Please fill out the feedback forms!
• Contact:
– @attrc
161
![Page 162: Linux Memory Analysis Workshop Session 1 Andrew Casedfir.org/research/blackhat-workshop-full-presentation.pdf–Supports Windows {XP, Vista, 7, 2003, 2008} –Support Linux 2.6.9 to](https://reader034.vdocuments.us/reader034/viewer/2022052018/6030d1a06fddf44d2a7c5ea8/html5/thumbnails/162.jpg)
References
[1] http://lwn.net/Articles/75991/
162