omniumpack: fast, generic, and safe unpacking of malware

30
1 OmniUmpack: Fast, Generic, and Safe Unpacking of Malware Authors: Lerenzo Martignoni, Mihai Christodorescu and Somesh Jha Computer Security Applications Conference 2007 (ACSAC’07) Presented by LIU Limin

Upload: agatha

Post on 23-Jan-2016

56 views

Category:

Documents


0 download

DESCRIPTION

OmniUmpack: Fast, Generic, and Safe Unpacking of Malware. Authors: Lerenzo Martignoni, Mihai Christodorescu and Somesh Jha Computer Security Applications Conference 2007 (ACSAC’07) Presented by LIU Limin. Agenda. Introduction Design Implementation Evaluation Conclusion. Agenda. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

1

OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

Authors: Lerenzo Martignoni, Mihai Christodorescu and Somesh JhaComputer Security Applications Conference 2007 (ACSAC’07)

Presented by LIU Limin

Page 2: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

2

Agenda

• Introduction• Design• Implementation• Evaluation • Conclusion

Page 3: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

3

Agenda

• Introduction• Design• Implementation• Evaluation • Conclusion

Page 4: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

4

Packed malware

• Term packed refers to malware whose payload is either compressed or encrypted. – E.g. polymorphic malware

• An unpacking routine that restores the original code from memory and then executes it during runtime.– Self-decompression or self-decryption– Several layers of unpacking

Page 5: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

5

Why malware packed?

• Anti-analysis– Reverse engineering– Experts can make the malware hard to

analyze

• Anti-virus tools evasion– Existing packers are modified to evade anti-

virus tools at the rate of 10-15 per month.– 80% new malware are packed with various

packers.– 50% new malware samples are repacked

versions of existing malware.How to effectively recover the real malicious code?

Page 6: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

6

Traditional unpacking approaches

• Algorithmic unpacking– Using specific unpacking routines to recover the

original program, one routine per packing algorithms.• ClamAV

– Limited to a fixed set of known packers.

Page 7: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

7

Traditional unpacking approaches (Cont.)

• Generic unpacking– Emulation/tracing of the execution until the

unpacking routine terminates. • PolyUnpack ([ACSAC 06]), Renovo ([Worm 07])

– Limitations• Unpacking is slow because of instruction-level

tracing.• Effectiveness depends on the fidelity of the

emulation environment.• The detection of the termination of the unpacking

routine is undecidable.

Page 8: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

8

Goals of OmniUnpack

• Generic unpacking with low-overhead by tracking memory access at the page level.

• Precise unpacking by running the program on native OS.

• An in-memory malware detection strategy independent of packing and self-modification.

Page 9: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

9

Agenda

• Introduction• Design• Implementation• Evaluation • Conclusion

Page 10: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

10

OmniUnpack: simple strategy

• Tracking all memory writes and the program counter.

• The execution of a previously written memory locations indicates the end of an unpacking stage.

• All written-then-executed (or written and about to be executed) memory locations should then be analyzed by a malware detector.Starting from this strategy, designing an unpacking algorithm that achieves low overhead yet does not compromise the security of the system.

Page 11: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

11

Execution traces for a packed executable

• Ideally

– The detection of the termination of the unpacking routine is undecidable.

• Coarse-grained memory access tracking

– using hardware mechanism at page level.– Written-then-executed pages are indicative of unpacking

but not indicative of the end of unpacking.– Increasing the chance to detect spurious unpacking stages

(up to hundreds of thousands stages) . – The overhead introduced by invoking the malware detector

every time a written memory page is executed is prohibitive.

Page 12: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

12

OmniUnpack-enhanced detection• Heuristic: An unpacking stage is completed when the execution of previous written page is followed by a dangerous

system call.

• Choice of dangerous system calls– A dangerous system call is a system call whose execution can leave the system in an unsafe state.

Page 13: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

13

Continuous monitoring of the execution

• OmniUnpack implements a continuous monitoring approach, where the execution is observed in its entirety.– Multiple unpacking stages– Approximation of the end of an unpacking

stage

• A solution for the undecidability of the problem of detecting the real end of the unpacking stage.

• The low overhead of OmniUnpack allows for continuous monitoring in an end-user environment.

Page 14: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

14

Algorithm

Page 15: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

15

Example

• Execution Trace : <x(0), w(1), s1, w(2), x(1), s2, x(2), s3, …>

– x(0): W = Ø, WX = Ø– w(1): W= {1}, WX = Ø– s1 (NtOpenFile): W = {1}, WX = Ø– w(2): W = {1,2}, WX= Ø– x(1): W = {1,2}, WX= {1}– s2 (NtOpenKey): W = {1,2}, WX= {1}– x(2): W = {1,2), WX = {1,2}– s3 (NtDeleteFile): The system call is dangerous, malware

detector is invoked to scan all the memory pages in W.

s1 is not dangerous and WX is empty.

s2 is not dangerous.

Page 16: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

16

Agenda

• Introduction• Design• Implementation• Evaluation • Conclusion

Page 17: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

17

Architecture

• Kernel driver– Tracking memory

access– Detecting when

the dangerous system call occurs

– Triggering malware detection

• User-space Component– Malware detector

responsible for on-demand scanning of memory of the suspicious program.

Page 18: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

18

Monitoring memory accesses

• The W⊕X policy is enforced on the memory pages of the suspicious program.– A memory page can be either writable or executable.

• Page-fault exceptions are trapped by OmniUnpack.– When an unpacking routine writes the unpacked code

to memory, the destination page is marked writable but not executable.

– When an unpacking stage ends, the access to this page for execution will cause a protection exception because of the lack of execution permission.

– Intercept and process such exceptions.

• Non-executable pages can be emulated via software mechanism with a minimal overhead.– PaX PAGEEXEC

Page 19: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

19

Monitoring system calls

• Intercepting the system calls of interest (dangerous ones) through interposition on the system-call dispatch table.

• If newly generated code is present in the program memory, the code is executed, and the execution is followed by a dangerous system call, then consider the unpacking stage concluded and trigger malware detection.

• At the end of scanning, write permissions are revoked from all scanned pages.

Page 20: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

20

User-space malware detector

• Any malware detection strategy can be used to scan the code generated during the previous stage.

• OmniUnpack only provides a detector with data to analyze and then authorizes, or denies, the execution of such data according to the response of the detector.

• A bug in the malware detector will not compromise the stability of the whole system since the detector is running in user-space.

Page 21: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

21

Agenda

• Introduction• Design• Implementation• Evaluation • Conclusion

Page 22: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

22

Experimental evaluation

• Successfully monitored all the analyzed samples and detected the end of each unpacking stage.

• Handled 80% of packed malware, while ClamAV handled only about 15%.

• Armadillo and CExe unpack the original payload on disk and then execute this executable.

• N/A: programs that did not execute correctly after packing, with or without OmniUnpack.

• IMPL: a limitation of the current OmniUnpack implementation.

Page 23: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

23

Time to unpack

• OmniUnpack was significantly faster than PolyUnpack. (e.g., with UPX OmniUnpack was 20.56 times faster than PolyUnpack.)

• In 42% of the cases PolyUnpack timeout (300 secs).

• OmniUnpack was within one order of magnitude of ClamAV (e.g., with UPX OmniUnpack was only 5 times slower).

• In 75% cases, ClamAV failed to unpack.

Page 24: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

24

Overhead for benign programs

• About 6% average overhead on non-packed applications.

• About 11% average overhead on packed applications (with UPX).

Page 25: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

25

Agenda

• Introduction• Design• Implementation• Evaluation • Conclusion

Page 26: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

26

Conclusion

• A new unpacking technique that addresses the shortcomings of previous unpackers.– Generic unpacker

• Able to deal with known and unknown unpacking algorithms.• resilient to anti-debugging, anti-VM and anti-emulation

techniques.• Integrated with any operating system and any malware

detector without trading off efficiency or unpacking capability.

– Suitable for use on end-user environment. • Minimizing the overhead by using hardware mechanism and

working directly in the OS.– Safe in the presence of multiple unpacking stages.– Improved detection rate.

• Any malware detector can delegate to OmniUnpack the recovery of the packed code.

Page 27: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

27

Limitations

• Imprecision of page-level tracking.– The memory monitoring at page level caused

the undecidability of the specific addresses which are really accessed on the page.

– The assumption that memory accesses potentially affect all addresses on a page will cause the identification of spurious unpacking stages.

• Incomplete extracted instructions.– The content of written-then-executed page is

modified before invoking a dangerous system call.

Page 28: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

28

Thank you!Questions?

Page 29: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

29

PolyUnpack ([ACSAC 06])

• Performing static analysis over malware instance to acquire its static code model.

• The statically derived model and malware instance are fed into the dynamic analysis component.

• The execution of malware paused after each instruction and its context is compared with the model.

• When the first instruction of a sequence not found in the static model is detected, representations of that unknown instructions sequences are written out and the malware’s execution is halted.

Page 30: OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

30

Renovo ([Worm 07])

• Heuristic: the original program code and data should be present in memory to be executed, and also the instruction point should jump to the OEP of the restored program code which has been written in memory at runtime.

• A technique dynamically extract the hidden original code and the OEP from the packed executable by examining whether the current instruction has been generated at run-time, after the program binary was loaded.

• Instruction level.