An Out-of-the-Box Approach to High Assurance Computer System Monitoring and Integrity Protection

Cyber Defense Conference, Rome, NY, May 12-14, 2008

Assistant ProfessorDept. of Computer Science George Mason University

Xuxian Jiang

Associate ProfessorCERIAS and Dept. of Computer

Science Purdue University

Dongyan Xu

Outline

Motivation “Out-of-the-box” for high assurance

New VMM component: OBSERV New capabilities enabled

High assurance system monitoring Stealth malware detection External run of COTS anti-virus software OS integrity protection against kernel rootkits

Planned work Summary

Malware remains a top concern in cyber defense Malware: viruses, worms, rootkits, spyware,

bots…

Motivation

Motivation

Rootkit attack trend

Source: McAfee Avert Lab Report (April 2006)

400% growth

400% growth

Q1 of 2005

700% growth

700% growth

Viruses, worms, bots, …

State-of-the-art: Running high-assurance modules (e.g., anti-virus systems) inside the monitored system Advantage: They can see everything (e.g., files,

processes…) Disadvantage:

VirusScan

FirefoxIE

OS Kernel

…

Why Going “Out-of-the-Box”?

They cannot see anything!

Why Going “Out-of-the-Box”?

Fundamental flaw in current practice Malware and malware defense running in the same

system space at the same privileged level No clear winner in this “arms race”

Solution: Going “out-of-the-box”

FirefoxIE

OS Kernel

…

VirusScan

Virtual Machine Monitor (VMM)

Semantic Gap

The “Semantic-Gap” Challenge

What we get: Low-level states

Memory pages, disk blocks…

Low-level events Privileged instructions, Interrupts, I/O…

What we want: High-level semantic states

Files, processes… high-level semantic events

System calls, context switches…

Virtual Machine Monitor (e.g., VMware, Xen)

Guest OS

VirusScan

Our Solution: OBSERV

OBSERV: “Out-of-the-Box” with SEmantically Reconstructed View A new component missing in current VMMs

FirefoxIE

OS Kernel

…

Virtual Machine Monitor (VMM)

OBSERV

New Capabilities

Capability II: Malware detection by

view comparison

Capability II: Malware detection by

view comparison

Capability I: High-assurance system

logging

Capability I: High-assurance system

logging

FirefoxIE

OS Kernel

…

Virtual Machine Monitor (VMM)

OBSERV

Capability III: External run of COTS anti-virus software

Capability III: External run of COTS anti-virus software

OBSERV View In-the-boxView Diff

Capability IV: OS kernel integrity

protection

Capability IV: OS kernel integrity

protection

OBSERV: Bridging the Semantic Gap

Step 1: Procuring low-level VM states and events Disk blocks, memory pages, registers… Traps, interrupts…

Step 2: Reconstructing high-level semantic view Files, directories, processes, and kernel

modules… System calls, context switches…

VM Introspection

Guest View Casting

Step 1: VM Introspection

Raw VMM Observations

Virtual Machines (VMs)

VMware Academic Program

VM disk image

VM hardware state (e.g., registers)

VM physical memory

VM-related low-level events (e.g., interrupts)

Step 2: Guest View Casting

Virtual Machine Monitor (VMM)

Guest OS

Key observation: The guest OS provides all semantic “templates” of data structures and functions to reconstruct VM’s

semantic view

OBSERVSemantic

Gap

Guest View Casting

Raw VMM Observations

Casted Guest Functions & Data

Structures

Reconstructed Semantic View

Device drivers, file system drivers

Memory translation,task_struct, mm_struct

CR3, MSR_SYSENTER_CS,MSR_SYSENTER_EIP/ESP

Event semanticsSyscalls,

context switches, ....

Event-specific arguments…

VM disk image

VM hardware state (e.g., registers)

VM physical memory

VM-related low-level events (e.g., interrupts)

Guest View Casting on Memory StateProcess List

Process Memory Layout

OBSERV Capability I

Capability I: High-assurance system

logging

Capability I: High-assurance system

logging

FirefoxIE

OS Kernel

…

Virtual Machine Monitor (VMM)

OBSERV

X. Jiang, X. Wang, "'Out-of-the-Box' Monitoring of VM-Based High-Interaction Honeypots", International Symposium on Recent Advances in Intrusion Detection (RAID 2007)

OBSERV Capabilities II and III

Capability II: Stealth malware

detection by view comparison

Capability II: Stealth malware

detection by view comparison

FirefoxIE

OS Kernel

…

Virtual Machine Monitor (VMM)

OBSERV

Capability III: External run of COTS anti-virus software

Capability III: External run of COTS anti-virus software

OBSERV View In-the-boxView Diff

X. Jiang, X. Wang, D. Xu, "Stealthy Malware Detection Through VMM-Based 'Out-of-the-Box' Semantic View Reconstruction", ACM Conference on Computer and Communications Security (CCS 2007)

View Comparison for Malware Detection Experiment setup

Both guest OS and host OS run Windows XP (SP2) VMM: VMware Server 1.0.1

Running Symantec AntiVirus twice Inside Outside

Hacker Defender

NTRootkit

External Scanning

Result

Internal Scanning

Result

Diff

OBSERV Capability IV: OS Kernel Integrity Protection

High-assurance OS kernel No malicious kernel code No kernel rootkit attacks

Two main tasks: Tracking run-time kernel code layout Enforcing the following properties

Only loading authenticated kernel code Only executing authenticated kernel code

R. Riley, X. Jiang, D. Xu, "Guest-Transparent Prevention of Kernel Rootkits with VMM-based Memory Shadowing", CERIAS Technical Report TR2001-146, Purdue University, 2008

OBSERV

NICKLE: “No Instruction Creeping into Kernel Level Executed”

NICKLE

Standard memory

Kernel Code

Shadow memory

VMM

Guest OS

Step 1: Create two memory spaces Standard memory Shadow memory

Step 2: Authenticate and copy kernel code to shadow memory

Step 3: Memory access dispatch Kernel code fetch ->

shadow memory All other accesses ->

standard memory

Kernel Code

Demonstration of Effectiveness

Successfully preventing 23 real-world kernel rootkits!

Planned Work

Porting OBSERV to hardware FPGA, multicore, PCI card…

Research problems Software/hardware function division Hardware primitives/policies for high

assurance Formal verification of OBSERV capabilities Performance optimization

Summary

OBSERV enables “out-of-the-box” malware defense paradigm, bringing high assurance to System logging and monitoring Malware detection and prevention OS kernel (against kernel rootkits)

We are looking for Applications in Cyber Defense activities Collaboration/deployment/funding opportunities

Process Coloring: An Information Flow-Preserving Approach to Malware Investigation

Eugene Spafford, Dongyan Xu, Ryan RileyDepartment of Computer Science and

Center for Education and Research in Information Assurance and Security (CERIAS)

Purdue University

Xuxian JiangDepartment of Computer Science

George Mason University

Part of NICIAR Program

A related project funded by IARPA through AFRL

Thank you!

For more information:

xjiang@gmu.edu, dxu@cs.purdue.eduhttp://www.cs.gmu.edu/~xjianghttp://friends.cs.purdue.edu