![Page 1: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/1.jpg)
Fuzzing CowsThe “No Bull” Talk on Fuzzing
Security B-Sides OttawaNovember 13, 2010
Mike Sues (Rigel Kent)Karim Nathoo (Inverse Labs)
![Page 2: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/2.jpg)
Objectives• We can’t cover fuzzing in-depth in 50 minutes• Raise awareness of fuzzing as an option in higher
assurance/product evaluations/more focused assessments
• Go over challenges/experiences from the field• Provide real examples• Get you thinking about how you can start fuzzing• Expose the scan monkey• Collect free chicken wings honorarium
![Page 3: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/3.jpg)
What’s With The Title
• An inside joke that went wrong• It is Mike’s fault
![Page 4: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/4.jpg)
WTF is Fuzzing• Pass malicious input to interfaces• Interfaces to target are attacker accessible
ones (either direct or indirect)• Detect anomalous conditions that might be
exploitable• Usually there is some form of automation• All the kewl people are doing it
![Page 5: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/5.jpg)
Fuzzing History
• Manual & custom scripts– Unintelligent
• i.e. cat /dev/random | service to 0wn– It worked!
• A bit more intelligent– Modeling protocols
• Block-based modeling– Frameworks
![Page 6: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/6.jpg)
Fuzzing History
• Tool integration– Inline fuzzing– Fuzzing and root cause analysis
• Process stalking– Fuzzing and code coverage– Commercialization
• Fuzzing support– Reverse engineering of protocols and code
![Page 7: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/7.jpg)
Limits of the Scan Monkey• The Scan Monkey uses nmap and Nessus without discrimination in a failed
attempt at world domination• Good Stuff:
– Tools determine presence of known vulnerabilities– Audit configurations– Verify patches– Highly automatable– You can get co-op students to do this– For some situations this is perfectly fine (low assurance environments,
operational audits, time constrained etc.)– Co-op students will work for Twizzlers
• Bad Stuff:– For new technologies, Scan Monkey tools don’t have signatures– Aside from getting lucky on occasion, effectiveness limited for product or new
technology evaluation– It is boring, contributions to the human condition are limited
![Page 8: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/8.jpg)
When to Fuzz
• New product/technology• Old product but a high level of assurance is
required• Internal QA as part of SDLC if you are a
product vendor• If you are a bug hunter• If you don’t really have a lot going on in your
life
![Page 9: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/9.jpg)
When Not to Fuzz
• If you actually have a life• When you’re testing systems/products in
production– THIS IS NOT A VULNERABILITY ASSESSMENT!!
![Page 10: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/10.jpg)
Different Types of Fuzzing
• Network– Server perspective (example: fuzz web server)– Client perspective (example: fuzz web browser)– Protocol (example: fuzz IPv6 stack)
• Local– File format– API– Driver
![Page 11: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/11.jpg)
Different Types of Fuzzing
• Wireless– 802.11x– Bluetooth– IR– Zigbee– RFID
![Page 12: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/12.jpg)
Generating Payloads/Tests• Generation Based
– Reverse engineer,• Protocol• API• Field encoding
– MIME/BER …
– Manually• Your brain and many test communications
– Wireshark– Strace
• Time-intensive
![Page 13: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/13.jpg)
Generating Payloads/Tests• Generation Based
– Semi-automatic protocol analysis• Proprietary and open protocols
– Open protocols still have grey areas• Analyze or proxy network communications
– Wireshark• Research & tools
– Discoverer– PI (Protocol Informatics)– PDB (Protocol Debugger)
![Page 14: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/14.jpg)
Generating Payloads/Tests• Generation Based
– Modeling input to generate test cases in their entirety• Block-based modeling
s_string ("USER "); s_string_variable("bob"); s_string("\r\n"); s_string("PASS "); s_string_variable("bob"); s_string("\r\n");
![Page 15: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/15.jpg)
Generating Payloads/Tests• Mutation Based
– Use existing valid payload and perturb it– Re-writing proxy
• PDB (Protocol Debugger)• Taof (The Art of Fuzzing)
– Modify stock client if you have source code (ex: openSSL)
![Page 16: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/16.jpg)
Target Observability and Traceability
• Need to be able to observe anomalies as the target is being stressed
• Not only detect an anomalous condition/state but CORRELATE to test case
• Absolutely key to effective fuzzing• If you do it wrong you will waste lots of time
and FAIL
![Page 17: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/17.jpg)
Methods for Target Observability
• Process monitoring (Debugger)– Usually the best way
• Network Heartbeats • Log Files• Test Case Timing
![Page 18: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/18.jpg)
Beware the State Machine
• If you don’t setup protocols properly, all you do is fuzz the crap out of the error state
• perl –e ‘print “A” x 41’ is not always enough• You may also just fuzz decoder code
– MIME/BER encoded fields
![Page 19: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/19.jpg)
Beware the State Machine
Protocol Handshake AuthenticationTransport
Crypto Decoder Payload Processing
Error State
Fuzzing Ninja
![Page 20: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/20.jpg)
Fuzzing Work Flow• Rough methodology,
– Reverse/research target– Prioritize areas/inputs to stress
• Code coverage– Model inputs– Create test cases– Automate– Analyze results– Root cause analysis
• Determine exploitability– Develop proof of concept/full exploit
• Iterate!
![Page 21: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/21.jpg)
Prioritizing• Fuzzing takes a long time, might not be able to cover
everything within engagement scope• Lots of ways to approach, lots of tradeoffs
– Obscure versus common functionality (commercial development experience teaches not everything is QA’d)
– Level of access (ex: kernel mode versus user mode)– May be trade off in terms of level of access or probability
of finding a bug versus affected user base (ex: bug in IE versus Safari)
![Page 22: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/22.jpg)
Prioritizing Cont’d
• Embedded RTOS as an example:– Servers – probably best vendor coverage– Setuid programs - privilege escalation– Regular user programs -limited privileges– Drivers – very target specific– System call API – might find bug that is not
attacker accessible
![Page 23: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/23.jpg)
Root Cause Analysis Challenges• Difficulties:
– Black box: all you have is raw crash data and assembly code
– Bug could be triggered before it becomes apparent using fault detection technique, examples:• simple stack based overflow triggered early in function but not
raise exception till function return. • heap overflow: corrupted memory location might not be used
until well after function return, making it even harder– Analyst needs knowledge of different vulnerability classes
(stack overflows, heap overflows, integer overflows, format string, etc.) to do thorough RCA
![Page 24: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/24.jpg)
Network Fuzzing Challenges
• Binary protocols• Checksums/verifiers, state machine challenges• Closed systems (appliances)
– Limited debug support– Target side instrumentation difficult or impossible
• Multi-threaded/multi-process servers• Test case throughput limited by network
![Page 25: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/25.jpg)
Network Fuzzing Demo 1
• The traditional FTP server example
![Page 26: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/26.jpg)
Network Fuzzing Demo 1• Summary:
– State machine – needed to properly setup authenticated session to find vulnerability
– Fault detection based on network heart beat works in this example
– Correlating test case to exception avoids search space nightmare
– Needed to switch to target debugger view to determine exact target state and exploitability
– Exception is an access violation, fits pattern standard for stack based buffer overflows
– Demonstrated how some analysis is required to get to root cause and formulate an exploit (quick)
– It’s not always this easy :)
![Page 27: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/27.jpg)
Network Fuzzing Demo 2
• Physical security system• Found in field in a real assessment
![Page 28: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/28.jpg)
Network Fuzzing Demo 2• Summary
– Target observability – relying on a network heartbeat in this case would have resulted in missing the bug
• Multiple threads• Server doesn’t crash when one thread generates exception• We need a debugger/ deployed agent in this case
– Root Cause Analysis – does not appear exploitable for remote code exec, unhandled C++ exception with no opportunity to overwrite exception handler
– We can DoS the crap out of the alarm system console and web server though :)
– Amount of root cause analysis depends on target, in this case alarm DoS as interesting as remote code execution
![Page 29: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/29.jpg)
File Format Fuzzing
• Headers and internal structure– PE– Microsoft Office– PDF– Media files– Images– Anti-virus
• File parsing
![Page 30: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/30.jpg)
File Format Fuzzing
• Software reads and interprets these formats– Client or supporting library (e.g. image library)
• Model input structure and fields• Launch client on fuzzed input file• Look for crash
– Process monitoring• Integration of launch and detection in one tool
![Page 31: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/31.jpg)
File Format Fuzzing
• Issues– File formats are complex and many interesting
ones are closed source– Formats can be embedded
• Down the rabbit hole– Many test cases
• Fuzz till the cows come home– File formats can change radically between
software versions
![Page 32: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/32.jpg)
File Format Fuzzing
• Tools– FileFuzzer– FuzzyWuzzy– SPIKEfile– notSPIKEfile
• Distributed fuzzing ….
![Page 33: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/33.jpg)
Client-side Fuzzing
• Why do we like clients?– They pay my bills– They are fun to work with– They have interesting work– Exploiting them gets me right on an internal
workstation• Mike is happy
![Page 34: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/34.jpg)
Client-side Fuzzing
• Coordinated approach– Fuzzing server and test client– Fuzzing model resides on server– Client connects– Server delivers fuzzed input– Client goes boom
![Page 35: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/35.jpg)
Client-side Fuzzing
• Issues– Server maintains state of fuzzing cases
• Distributed fuzzing considerations– Maintaining state across clients
– Client must be activated and pointed to fuzzing server
– Detection of client crash• Process monitoring on client machine• Client or support library?
![Page 36: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/36.jpg)
Client-side Fuzzing
• Issues– Complex client inputs
• Client inputs• Support library inputs
– Many test cases• Distributed fuzzing!
![Page 37: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/37.jpg)
Client-side Fuzzing
• Tools– Peach– Sulley– Condenomicon– COM and ActiveX fuzzers
![Page 38: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/38.jpg)
Driver Fuzzing
• Diving into Ring0
• Different approaches– Remote protocol fuzzing (e.g. stack fuzzing)– Local API fuzzing
![Page 39: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/39.jpg)
Driver Fuzzing
• Local API fuzzing– User mode -> kernel mode
• Privilege escalation• Important for multi stage attacks
– Application specific• User land components• Driver components
![Page 40: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/40.jpg)
Driver Fuzzing
• Issues– Identify the interface and inputs
• Device name/Link• IOCTL
– Header files– Reversing user-land components
– Identifying a crash• Blue screen in Windows
– Slow down testing
![Page 41: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/41.jpg)
Driver Fuzzing
• Tools– Immunity Debugger
• Driverlib– Discover driver names/links
• pyCommand script– Proxy IOCTL calls
• Mutation-based fuzzer– Direct fuzzing
• Generation-based fuzzer
– Kartoffel
![Page 42: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/42.jpg)
Developing Exploits• You don’t go from crash -> 0day in a few minutes• Generating crashes is easy, analysis is hard part• Difficulties:
– It’s not 2001 anymore– Memory corruption mitigations in modern OS’s
• DEP• ASLR• EMET
– 3rd party support libraries– Specific setup conditions– Analyst often needs expert knowledge
![Page 43: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/43.jpg)
Developing Exploits• Goal of engagement
– Exploit development might not be in scope• Working with developers/vendor
• Clients might not want to fund you to develop an exploit– Customers paying for gaps in vendor development
practices?• Smells like a buck is being passed
![Page 44: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/44.jpg)
The Evolution of Cows• Driver fuzzing tools/techniques continuing to improve
and becoming more accessible• Continued integration of fuzzers and RCA tools• File format fuzzing continuing to increase and a
blurring of file-format and client-side fuzzing• More device fuzzing (e.g. smart device stuff)• Better automated tools for developing our models• Distributed fuzzing frameworks and tools
![Page 45: Fuzzing Cows The “No Bull” Talk on Fuzzing Security B-Sides Ottawa November 13, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062315/568163e5550346895dd546b2/html5/thumbnails/45.jpg)
Fuzzing Cows• Questions?