anomaly detection using call stack information security reading group july 2, 2004 henry feng, oleg...
Post on 20-Dec-2015
214 views
TRANSCRIPT
Anomaly Detection Using Call Stack Information
Security Reading GroupJuly 2, 2004
Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong
Presenter: Jonathan McCune
2
Overview
• Introduction• Related work• VtPath• Experiments• Exploits• Comparisons• Conclusion
3
Introduction
• Runtime, training-phase based, anomaly detection system
• Uses call stack and program counter in addition to system call hooks
• Novel features based on virtual paths
• Compared experimentally and analytically with many other approaches
4
Related work
• Static analysis– Wagner, et al. (abstract stack)– Callgraph
• Run-time, training based:– This– Sekar, et al. (FSA) (last week’s srg)– N-gram, Var-gram
5
“Intrusion detection via static analysis.” Wagner, et al.
• Static analysis of program source code• NDFA from control-flow graph
– Cannot predict branches statically
• Impossible path problem• Abstract stack model
– Pushdown automaton
• Large automata can be resource intensive• Claimed zero false positives
6
“Detecting Manipulated Remote Call Streams.” Giffin, et al.
• Static analysis of binary executables
• Tied to platform, not programming language
• Insertion of “null calls” helps impossible path problem
• Unusual performance analysis
7
“A fast automaton-based method for detecting anomalous program
behavior.” Sekar, et al.
• Compact deterministic FSA from system call analysis of running programs
• Suffers from false positives, but not as a result of non-determinism
• Impossible paths not addressed
• DLLs not adequately addressed
8
Virtual Path (VtPath)
• Run-time analysis• Learns correct behavior via training
executions • Utilizes return address information• Generates abstract path and compares
with learned behavior• Similar to abstract stack of Wagner, et
al., but avoid pushdown automaton
9
Virtual Path Example
main()
foo1()
foo()
foo2()
foo3()
main()
fooA()
foo()
fooB()
fooC()
10
VtPath Training Phase
• Two hash tables:– RA (return address) table– VP (virtual path) table
• RAs and VPs gradually added during normal program execution
• Use NULL entries at beginning and end of paths
11
VtPath Online Detection Phase
• Stack anomaly – if virtual stack list unavailable (common during buffer overflow attacks)
• Return address anomaly – if virtual stack list {a0, a1, … an} contains ai not in RA table
• System call anomaly – if an does not have the correct system call
• Virtual path anomaly – if virtual path at current system call is not in VP table
12
VtPath – Impossible Path Problem
• Return addresses are part of virtual path information
• Modifying return address saved on the stack will cause the program to return to a different location
• The next system call is likely to trigger a virtual path anomaly
13
VtPath - Implementation Issues
• Non-standard control flows– Signals (sigreturn system call)
• Treat each one like a separate program invocation– setjmp()/longjmp()and function pointers
• Hard to handle statically; handled at runtime if trained properly
• Dynamically linked libraries– Relative loading positions can change, invalidating PC
values from training runs– Use a “block” model during training to capture file name and
block length, ignoring start address
• Block anomaly – block lookup fails because attacker is trying to load a malicious DLL
14
Experiments – VtPath vs FSA
• Convergence times similar, but – FSA generates more transitions than VtPath (less
efficient, less precise)– In practice, multiple levels of DLL functions are
called quite frequently – VtPath more effective
• False Positives nearly identical• VtPath executes faster, uses more memory• Common exploits detected by both
15
Exploits
• Authors developed two masked mimicry attacks detectable only by VtPath
• Impossible Path Execution (IPE) Attack 1– Exploits parallel structure of privileged /
unprivileged code within same function
• IPE Attack 2:– F() called twice from within same function with
different arguments– Overflow local variable to change option passed in
16
Attack 1
17
Attack 2
18
Comparison of SysCall-based Anomaly Detection Schemes
1. State-based / Information captured• Data/heap values least useful (transient)• Code segment of some value• Syscalls and callstack most useful• More information = runtime overhead
2. False positives• How well is normal behavior captured?• Only relevant for dynamic systems• Proportional to resolution of program analysis
3. Detection capability• More granularity = better detection capability• SysCalls from invalid points vs. statistical regularity of
training data vs. attacks’ deviation from perceived normal
19
Comparison of SysCall-based Anomaly Detection Schemes
4. Space requirement• System call sequences• Number of NDFA transitions (Wagner)• Transitions in automaton
5. Convergence (training) time• Cover most possible states / transitions• VtPath requires more data than FSA• Static techniques have big advantage here
6. Runtime overheads• SysCall interception• Hash lookup for valid state / return address / virtual path
20
Conclusions
• Using call stack (VtPath) has value
• There is no magic bullet– Everything is “complementary”
• Impossible path execution (IPE) is an important class of attack
• Use of FSA to analyze more than two consecutive SysCalls could improve VtPath
21
Discussion!