IntroductionDesign
Architecture & Evaluation
Scalability, Fidelity, and Containment in thePotemkin Virtual Honeyfarm
Michael Vrable, Justin Ma, Jay Chen, David Moore,Erik Vandekieft, Alex C. Snoeren, Geoffrey M. Voelker,
Stefan Savage
Collaborative Center for Internet Epidemiology andDefenses (CCIED)
University of California, San Diego
The Potemkin Virtual Honeyfarm 1 / 20
IntroductionDesign
Architecture & Evaluation
Background
I Large-scale host exploitation a serious problemI Worms, viruses, bots, spyware. . .I Supports an emerging economic criminal enterprise
I SPAM, DDoS, phishing, piracy, ID theft. . .I Two weeks ago, one group arrested—controlled 1.5 M hosts!
I Quality and sophistication of malware increasing rapidly
The Potemkin Virtual Honeyfarm 2 / 20
IntroductionDesign
Architecture & Evaluation
Motivation
I Intelligence about new threats is critical for defendersI Principal tool is the network honeypot
I Monitored system deployed for the purpose of being attacked
I Honeyfarm: Collection of honeypotsI Provide early warning, accurate inference of global activity,
cover wide range of software
I Design issuesI Scalability: How many honeypots can be deployedI Fidelity: How accurately systems are emulatedI Containment: How well innocent third parties are protected
I Challenge: tension between scalability and fidelity
The Potemkin Virtual Honeyfarm 3 / 20
IntroductionDesign
Architecture & Evaluation
Honeyfarm Scalability/Fidelity Tradeoff
High Scalability High Fidelity
The Potemkin Virtual Honeyfarm 4 / 20
IntroductionDesign
Architecture & Evaluation
Honeyfarm Scalability/Fidelity Tradeoff
High Scalability High Fidelity
VM-based Honeyfarms
(Collapsar, Symantec)
Physical Honeypots(Honeynet Project)
Execute real code
The Potemkin Virtual Honeyfarm 4 / 20
IntroductionDesign
Architecture & Evaluation
Honeyfarm Scalability/Fidelity Tradeoff
High Scalability High Fidelity
Network Telescopes
Lightweight Responders
(iSink, IMS, honeyd)
Millions of addresses
VM-based Honeyfarms
(Collapsar, Symantec)
Physical Honeypots(Honeynet Project)
Execute real code
The Potemkin Virtual Honeyfarm 4 / 20
IntroductionDesign
Architecture & Evaluation
Honeyfarm Scalability/Fidelity Tradeoff
High Scalability High Fidelity
Network Telescopes
Lightweight Responders
(iSink, IMS, honeyd)
Millions of addresses
VM-based Honeyfarms
(Collapsar, Symantec)
Physical Honeypots(Honeynet Project)
Execute real code
Our Goal: Both
The Potemkin Virtual Honeyfarm 4 / 20
IntroductionDesign
Architecture & Evaluation
ApproachNetwork-Level MultiplexingHost-Level Multiplexing
Approach
I Dedicated honeypot systems are overkill
I Can provide the illusion of dedicated systems via aggressiveresource multiplexing at network and host levels
The Potemkin Virtual Honeyfarm 5 / 20
IntroductionDesign
Architecture & Evaluation
ApproachNetwork-Level MultiplexingHost-Level Multiplexing
Network-Level Multiplexing
I Most addresses don’t receive traffic most of the time
⇒ Apply late binding of IP addresses to honeypots
I Most traffic that is received causes no interesting effects
⇒ Allocate honeypots only long enough to identify interestingbehavior
⇒ Recycle honeypots as soon as possible
I How many honeypots are required?I For a given request rate, depends upon recycling rate
The Potemkin Virtual Honeyfarm 6 / 20
IntroductionDesign
Architecture & Evaluation
ApproachNetwork-Level MultiplexingHost-Level Multiplexing
Effectiveness of Network-Level Multiplexing
0.001
0.01
0.1
1
1 10 100
Act
ive
Mac
hine
s / T
otal
Add
ress
es
Recycling Time (s)
2–3 orders of magnitude improvement!
The Potemkin Virtual Honeyfarm 7 / 20
IntroductionDesign
Architecture & Evaluation
ApproachNetwork-Level MultiplexingHost-Level Multiplexing
Effectiveness of Network-Level Multiplexing
0.001
0.01
0.1
1
1 10 100
Act
ive
Mac
hine
s / T
otal
Add
ress
es
Recycling Time (s)
2–3 orders of magnitude improvement!
The Potemkin Virtual Honeyfarm 7 / 20
IntroductionDesign
Architecture & Evaluation
ApproachNetwork-Level MultiplexingHost-Level Multiplexing
Host-Level Multiplexing
I CPU utilization in each honeypot quite low (milliseconds toprocess traffic)
⇒ Use VMM to multiplex honeypots on a single physical machine
I Few memory pages actually modified when handling networkdata
⇒ Share unmodified pages among honeypots within a machine
I How many virtual machines can we support?I Limited by unique memory required per VM
The Potemkin Virtual Honeyfarm 8 / 20
IntroductionDesign
Architecture & Evaluation
ApproachNetwork-Level MultiplexingHost-Level Multiplexing
Effectiveness of Host-Level Multiplexing
0
0.5
1
1.5
2
2.5
3
3.5
4
0 20 40 60 80 100
Mem
ory
Pag
es M
odifi
ed (
MB
)
Time (s)
HTTP
Telnet
Ping
Further 2–3 orders of magnitude improvement
The Potemkin Virtual Honeyfarm 9 / 20
IntroductionDesign
Architecture & Evaluation
ApproachNetwork-Level MultiplexingHost-Level Multiplexing
Effectiveness of Host-Level Multiplexing
0
0.5
1
1.5
2
2.5
3
3.5
4
0 20 40 60 80 100
Mem
ory
Pag
es M
odifi
ed (
MB
)
Time (s)
HTTP
Telnet
Ping
Further 2–3 orders of magnitude improvement
The Potemkin Virtual Honeyfarm 9 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
The Potemkin Honeyfarm Architecture
I Two components:I GatewayI VMM
I Basic operation:
I Packet received bygateway
I Dispatched tohoneyfarm server
I VM instantiated
I Adopts IPaddress
The Potemkin Virtual Honeyfarm 10 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
The Potemkin Honeyfarm Architecture
I Two components:I GatewayI VMM
I Basic operation:I Packet received by
gateway
I Dispatched tohoneyfarm server
I VM instantiated
I Adopts IPaddress
The Potemkin Virtual Honeyfarm 10 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
The Potemkin Honeyfarm Architecture
I Two components:I GatewayI VMM
I Basic operation:I Packet received by
gatewayI Dispatched to
honeyfarm server
I VM instantiated
I Adopts IPaddress
The Potemkin Virtual Honeyfarm 10 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
The Potemkin Honeyfarm Architecture
I Two components:I GatewayI VMM
I Basic operation:I Packet received by
gatewayI Dispatched to
honeyfarm serverI VM instantiated
I Adopts IPaddress
The Potemkin Virtual Honeyfarm 10 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Potemkin VMM Requirements
I VMs created ondemand
I VM creation mustbe fast enough tomaintain illusion
I Many VMs created
I Must beresource-efficient
The Potemkin Virtual Honeyfarm 11 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Potemkin VMM Requirements
I VMs created ondemand
I VM creation mustbe fast enough tomaintain illusion
I Many VMs createdI Must be
resource-efficient
The Potemkin Virtual Honeyfarm 11 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Potemkin VMM Overview
I Modified version of Xen 3.0 (pre-release)I Flash cloning
I Fork copies from a reference honeypot VMI Reduces VM creation time—no need to bootI Applications all ready to run
I Delta virtualizationI Copy-on-write sharing (between VMs)I Reduces per-VM state—only stores unique dataI Further reduces VM creation time
The Potemkin Virtual Honeyfarm 12 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Flash Cloning Performance
Time required to clone a 128 MB honeypot:
Control tools overhead 124 msLow-level clone 11 msDevice setup 149 msOther management overhead 79 msNetworking setup & overhead 158 ms
Total 521 ms
0.5 s already imperceptible to external observers unless looking fordelay, but we can do even better
The Potemkin Virtual Honeyfarm 13 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Flash Cloning Performance
Time required to clone a 128 MB honeypot:
Control tools overhead 124 msLow-level clone 11 msDevice setup 149 msOther management overhead 79 msNetworking setup & overhead 158 ms
Total 521 ms
0.5 s already imperceptible to external observers unless looking fordelay, but we can do even better
The Potemkin Virtual Honeyfarm 13 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Delta Virtualization Performance
I Deployed using 128 MB Linux honeypots
I Using servers with 2 GB RAM, have memory available tosupport ≈ 1000 VMs per physical host
I Currently tested with ≈ 100 VMs per hostI Hits artificial resource limit in Xen, but this can be fixed
The Potemkin Virtual Honeyfarm 14 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Containment Policies
I Must also care about traffic going out
I We deliberately run unpatched, insecure software in honeypots
I Containment: Should not permit attacks on third parties
I As with scalability, there is a tension between containmentand fidelity
I Various containment policies we support:I Allow no traffic outI Allow traffic over established connectionsI Allow traffic back to original hostI . . .
The Potemkin Virtual Honeyfarm 15 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Containment Implementation in Gateway
I Containment policies implemented in network gateway
I Tracks mappings between IP addresses, honeypots, and pastconnections
I Modular implementation in Click
I Gateway adds insignificant overhead (� 1 ms)
The Potemkin Virtual Honeyfarm 16 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Traffic Reflection
Example gateway policy:Redirect traffic back tohoneyfarm
I Packets sent out tothird parties. . .
I . . . may be redirectedback into honeyfarm
Reuses honeypot creationfunctionality
The Potemkin Virtual Honeyfarm 17 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Traffic Reflection
Example gateway policy:Redirect traffic back tohoneyfarm
I Packets sent out tothird parties. . .
I . . . may be redirectedback into honeyfarm
Reuses honeypot creationfunctionality
The Potemkin Virtual Honeyfarm 17 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Traffic Reflection
Example gateway policy:Redirect traffic back tohoneyfarm
I Packets sent out tothird parties. . .
I . . . may be redirectedback into honeyfarm
Reuses honeypot creationfunctionality
The Potemkin Virtual Honeyfarm 17 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Challenges
I Honeypot detectionI If malware detects it is in a honeypot, may act differently
I How easy it is to detect virtualization?I VMware detection code used in the wild
I Open arms race between honeypot detection and camouflage
I Resource exhaustionI Under high load, difficult to maintain accurate illusion
I Large-scale outbreakI Honeypot denial-of-service
I Challenge is intelligently shedding load
The Potemkin Virtual Honeyfarm 18 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
Summary
I Can achieve both high fidelity and scalabilityI Sufficient to provide the illusion of scale
I Potemkin prototype: 65k addresses → 10 physical hostsI Largest high-fidelity honeypot that we are aware of
I Provides important tool for study of and defenses againstmalware
The Potemkin Virtual Honeyfarm 19 / 20
IntroductionDesign
Architecture & Evaluation
OverviewPotemkin VMMContainmentChallenges
For more information:http://www.ccied.org/
The Potemkin Virtual Honeyfarm 20 / 20
Windows on XenCamouflageHoneypot Monitoring
Windows on Xen
The Potemkin Virtual Honeyfarm 21 / 20
Windows on XenCamouflageHoneypot Monitoring
Camouflage
Malware may detect honeypot environment in various ways:I Detect virtualization
I Via incomplete x86 virtualizationI Searching for characteristic hardware configurationsI More complete virtualization can mitigate these leaks
I Detect monitoring toolsI Network, VM-instrospection tools harder to detect
I Detect network environmentI Containment requirement places some limits on camouflage
effectivenessI Network security trends may be in our favor here
The Potemkin Virtual Honeyfarm 22 / 20
Windows on XenCamouflageHoneypot Monitoring
Honeypot Monitoring
Various means to monitor honeypots for interesting activity
I Network-level monitoring: Network intrusion detectionsystems, Earlybird-like detectors, . . .
I Host-level intrusion detection
I Virtual machine introspection
The Potemkin Virtual Honeyfarm 23 / 20