eurosec 2011 gábor pék , boldizsár bencsáth and levente buttyán

17
EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth and Levente Buttyán Laboratory of Cryptography and Systems Security Budapest University of Technology and Economics nEther: IN-GUEST DETECTION OF OUT-OF-THE-GUEST MALWARE ANALYSERS

Upload: amable

Post on 23-Jan-2016

29 views

Category:

Documents


0 download

DESCRIPTION

EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth and Levente Buttyán Laboratory of Cryptography and Systems Security Budapest University of Technology and Economics. nEther : IN-GUEST DETECTION OF OUT-OF-THE-GUEST MALWARE ANALYSERS. Short Summary. We successfully achieved - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

EUROSEC 2011

Gábor Pék , Boldizsár Bencsáth and Levente Buttyán

Laboratory of Cryptography and Systems Security

Budapest University of Technology and Economics

nEther: IN-GUEST DETECTION OF OUT-OF-THE-GUEST MALWARE ANALYSERS

Page 2: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Short Summary

We successfully achieved In-guest detection of an out-of-the-guest

malware analysis framework (Ether) In-guest timing attack Detection based on CPUID information

Detecting hardware assisted virtualization (can be a bit of information for analysis ) Detection based on errata in Intel CPUs

04/21/232

Page 3: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Goals in Malware Analysis Analyser: dissecting and figuring out

the operations of the analysed program

Author of the malware: thwarting the analysis of the code and hiding its real intents, operations, execution

04/21/233

Page 4: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

What is Malware Analysis? Analysing malware

Static (entire program, thwarting disassemblers)

Dynamic (one control path) we focus on this Two types of dynamic analysis: Native and

Virtualization based Main tricks of detecting dynamic analyzers

Timing information Special data structures, e.g., PEB Single-step debugging (trap flag) Exception handling

04/21/234

Page 5: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

HW Assisted Virtualization New and higher CPU privilege level

(Ring -1) Native instruction execution Intel VT

VMX root mode for VMM/Hypervisor VMX non-root mode for guest OS VMX transitions: VM Exit / VM Entry

Rich feature set and control of operation Xen, KVM

04/21/235

Page 6: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Ether – Malware analysis via HW Virtualization Extensions Transparent, out-of-the-guest malware

analysis platform based on Xen and Intel VT Transparency of Ether: the malware cannot

detect Ether Transparency requirements as of the Ether

paper: Higher privilege of analyser environment No non-privileged side effects Same instruction execution semantics X Identical exception handling Identical notion of time X

04/21/236

Page 7: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Advantages of Ether and challenges to detect it

No in-guest memory presence Hide of changes made on CPU registers Memory protection: modifies only

shadow page tables Privileged instruction handling No instruction emulation Controlling timing (e.g., RDTSC

instruction)

04/21/237

Page 8: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Contributions

04/21/238

Design and implementation of an application framework to detect Ether based on multiple feature tests

Feature tests for Ether and Intel VT A practical in-guest timing attack

against Ether Detecting Ether via CPUID information Detection of HW assisted

virtualization utilizing CPU errata

Page 9: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

System Overview

04/21/239

Page 10: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Various uses of RDTSC

04/21/2310

Different behaviour of sensitive instructions (e.g., RDTSC) in VMX non-root mode

Guest OS

rdtsc

Normal operation

TSC

rdtsc faketime (FT)

Operation of Ether

rdtsc FT+Y

VM EntryVM Exit

…CPUVirtualCPU

VirtualCPU

Guest OS Guest OS

Page 11: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Facts about Ether

Alleged operation: FT = TSC, Y = TSC_OFFSET but TSC_OFFSET is disabled

Real operation: Monotonic increase of FT for every RDTSC call (Y =1)

There can be external RDTSC calls during an analysis The TSC difference between two RDTSCs

of the analysed program = # of RDTSCs of the Guest during analysis (~9-171)

04/21/2311

Page 12: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Practical implementation of in-guest timing attack Call an RDTSC and store it Create a loop of non-sensitive

instructions (e.g., nop) Call an RDTSC and compare it with

the stored value (diff)

04/21/2312

if (diff < length of the loop)Ether is present

elseEther is not present

Page 13: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

CPUID information

CPUID instruction: processor identifcation and feature information

Allegedly: Ether has no in-memory presence

Reality: The TSC bit returned by CPUID is unset under Ether

Other bits of information PAE and PSE are disabled

04/21/2313

Page 14: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

CPU Errata Design deficiencies of CPUs Some of them are unpredictable Cause unexpected system behaviour Several have ”No Fix ” status Xen creates virtualized CPUs for privileged

instructions We have an erratum using MSRs (AH4) The access of MSRs are privileged VM exit Errata are not emulated by virtual CPUs Bingo, we have a new feature test

04/21/2314

Page 15: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Detecting Intel VT

04/21/2315

Erratum AH4

Number of updates

# of tests Native Xen Xen + Ether

100 59 0 0

1000 650 0 0

10000 4232 0 0

100000 20870 0 0

Page 16: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Future Work

Fundamentality of these problems Updating the theoretical model and

practical implementation of Ether Finding more feature tests against

other out-of-the-guest approaches (e.g., Azure) Proving that perfect transparency has

practical limitations

04/21/2316

Page 17: EUROSEC 2011 Gábor Pék , Boldizsár Bencsáth  and   Levente Buttyán

Gábor Pék, CrySyS Lab.

Thanks for Your Attention!Questions?

[email protected]@[email protected]

CrySyS Lab. http://www.crysys.huBudapest University of Technology and

Economics04/21/23

17