simplify your software testing process · the performance is nearly the same compared to soc's...
TRANSCRIPT
Simplify your Software Testing Process
Fuzzing of Embedded Systems
Table of content
● What the Hack is Fuzzing ?
● Approaches to fuzz embedded Projects
● Improve Fuzzer Inputs with Grammar Mutation
2
The speakers
3
Sirko HöerB.Sc.
Christian HartlageB.Sc.
● Vulnerability Research in Embedded
Systems at Code Intelligence GmbH
● More than 10 years of experience in
the area of Cyber Security
● Development of new methods to
improve fuzzing coverage at
Code Intelligence GmbH
● More than 3 years of experience with
automated malware analysis at FKIE
Overview
4
What is Fuzzing ?
Overview
5
What is Fuzzing in theory ?
Fuzzing or fuzz testing is an automated software testing technique that involves random data as input to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks
Overview
6
What is Fuzzing in theory ?
Fuzzing isor fuzz testing is an automated stuffing testing technique that involves random data into a computer program until ogram is then monitored until it has crashes, failing built-in code assertions, or potential memory leaks
Overview
7
What is Fuzzing in theory ?
Fuzz-Target
Overview
8
What is Fuzzing in theory ?
Fuzz-TargetFuzz-Engine
Overview
9
What is Fuzzing in theory ?
Fuzz-TargetFuzz-EngineRandom Input
● seed ● mutation / mixing / permutation
Monitoring
Overview
10
What is Fuzzing in theory ?
Fuzz-TargetFuzz-Engine CRASHRandom Input
● Memory Leak ● Crashes● Failing built-in code assertions● Racing Conditions
Monitoring
Overview
11
Fuzzing
Blackbox Whitebox / Greybox
Out-of-Process FuzzingIn-Process Fuzzing...Protocol Fuzzer
Types of Fuzzing techniques
Overview
12
Fuzzing
Blackbox Whitebox / Greybox
Out-of-Process FuzzingIn-Process Fuzzing...Protocol Fuzzer
Types of Fuzzing techniques
What is coverage-guided Fuzzing ?
13
What does coverage mean?
Engine
libFuzzer
MutatorFeedback
handler
Fuzz-Target
IO API Modul A
Modul BModul CSeed
What is coverage-guided Fuzzing ?
14
What does coverage mean?
Engine
libFuzzer
MutatorFeedback
handler
Fuzz-Target
IO API Modul A
Modul BModul C
“s1t3\x8f\x44”
Seed
What is coverage-guided Fuzzing ?
15
What does coverage mean?
Engine
libFuzzer
MutatorFeedback
handler
Fuzz-Target
IO API Modul A
Modul BModul C
“s1t3\x8f\x44”
Seed
What is coverage-guided Fuzzing ?
16
What does coverage mean?
Engine
libFuzzer
MutatorFeedback
handler
Fuzz-Target
IO API Modul A
Modul BModul CSeed
Coverage information
mutated Inputs
What is coverage-guided Fuzzing ?
17
Which coverage types are available ?
● Line coverage
● Function coverage
● Data Flow coverage
● Custom Definitions (eg. malloc / free)
● Edge coverage
VUzzer AFL libFuzzer honggfuzz
Assembler Rewrite ✓ ✓
Compile Time ✓ ✓ ✓
Hardware (Intel BTS/PT) ✓
i.e. : -fsanitize-coverage=edge,trace-pc,trace-cmp
Challenges to make Fuzzing possible with libFuzzer
18
● needs llvm, clang and the compiler-rt
● needs knowledge, how to write a Fuzz-Target
● it is complicated to embed it into a build environment
What do you need to run libFuzzer
[Source]: https://en.wikipedia.org/wiki/LLVM
Challenges to make Fuzzing possible with libFuzzer
19
● compile target project with specific options
like -fsanitize-coverage=*, -fsanitize=address, ...
● write a Fuzz-Target like a Unit Test
● compile the Fuzz-Target and link against the
target project
● then we get a executable
Steps to run fuzz-tests with libFuzzer
FUZZ-Target
Target Project
compiler-rt
FuzzerSeed
Approaches to fuzz embedded Projects
20
What is the Problem ?
Problem statement
● Need to cross-compile the code
● Hardware limitations
● Many IoT devices have their own operating system
Fuzz Engine
21[1] Sources: https://tr1.cbsistatic.com/hub/i/2016/04/18/34fe8362-ee13-4785-9e2c-0aa26740dcda/iothero.jpg
[1]
Problem statement
● Need to cross-compile the code
● Hardware limitations
● Many IoT devices have their own operating system
Limitation of available tools
● Hard to setup the environment
● libFuzzer on Linux for ARM-Architecture (32 / 64 Bit)
not officially supported and available ● Sanitizer on Linux for ARM-Architecture (32 / 64 Bit)
not officially supported and available ● also for MIPS (32 / 64 Bit)
[1] Sources: https://tr1.cbsistatic.com/hub/i/2016/04/18/34fe8362-ee13-4785-9e2c-0aa26740dcda/iothero.jpg
Fuzz Engine
22
[1]
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer23
Modul Tests Integration Tests System Tests
Modul 1 Modul n
API
Modul 1 Modul n
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer24
Modul Tests Integration Tests System Tests
Modul 1 Modul n
Fuzzing
API
Modul 1 Modul n
Fuzzing Fuzzing Fuzzing
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer25
Modul Tests Integration Tests System Tests
Modul 1 Modul n
Fuzzing
API
Modul 1 Modul n
FuzzingFuzzing Fuzzing Fuzzing
used as seed
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer26
Modul Tests Integration Tests System Tests
Modul 1 Modul n
Fuzzing
API
Modul 1 Modul n
FuzzingFuzzing Fuzzing Fuzzing
used as seed used as seed
Black box Fuzzing
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer27
Modul Tests Integration Tests System Tests
Modul 1 Modul n
Fuzzing
API
Modul 1 Modul n
FuzzingFuzzing Fuzzing Fuzzing
Black box Fuzzing
used as seed used as seed
Virtualization with QEMU Hardware
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer28
Modul Tests Integration Tests System Tests
Modul 1 Modul n
Fuzzing
API
Modul 1 Modul n
FuzzingFuzzing Fuzzing Fuzzing
used as seed used as seed
Virtualization with QEMU Hardware
Black box Fuzzing
What is QEMU ?
29
● Quick Emulation (QEMU)
● Developed by Fabrice Bellard in 2003
● System level emulation
● Binary translation
● is an open source project
Features :
- CPU emulator
- Emulated devices (e.g. VGA Display, serial port,...)
- Generic devices (e.g. block devices, network devices,...)
- Machine descriptions (e.g. PC, PowerMac,...)
- Debugger
- User interface
Some notations
● Host architecture : where you run QEMU
● Target architecture : the emulated architecture
[Source]: https://wiki.qemu.org/Logo
Features :
● it can be used for unit - and integration tests
● has a high performance
● very good scalability !
but …
● Needs a cross compile environment
● Needs to adapt libFuzzer and clang
Fuzzing with QEMU User Mode
30
Fuzzing with QEMU User Mode
31
Workflow to make fuzzing possible in QEMU User Mode
● build llvm, clang and compiler-rt
● build Fuzz-Target● be aware of all
dependencies
● install and setup Cross-Compile environment
● available on all frequently used linux distributions
● i.e. Debian/Ubuntu, CentOS, Arch Linux
Get X-Compile Env Build llvm toolchain Build Fuzz-Target Run fuzzer
● run Fuzz-Target ● use QEMU binary
translation
● Fuzzer run natively on the (X86-)Host
Modify libFuzzer for embedded Projects
Modification of compiler-rt sources for arm :
1. change some files of build configs
2. Add new options for libFuzzer
a. --qemu_user_mode
b. --qemu_toolchain_lib
3. Implement the functionality
32
Example :
[user@host] $ qemu-arm -L /usr/arm-none-gnueabihf ./fuzz_target \
--qemu_user_mode=1 --worker=4 --jobs=4 corpus
Generate command and execute it multiple times depending the number of jobs “qemu-arm -L <path> fuzz_target corpus > fuzz-0.log 2>&1”
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-0.log 2>&1
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-1.log 2>&1
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-2.log 2>&1
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-3.log 2>&1
Modify libFuzzer for ARM
Modification of compiler-rt sources for arm :
1. change some files of build configs
2. Add new options for libFuzzer
a. --qemu_user_mode
b. --qemu_toolchain_lib
3. Implement the functionality
33
Example :
[user@host] $ qemu-arm -L /usr/arm-none-gnueabihf ./fuzz_target \
--qemu_user_mode=1 --worker=4 --jobs=4 corpus
Generate command and execute it multiple times depending the number of jobs “qemu-arm -L <path> fuzz_target corpus > fuzz-0.log 2>&1”
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-0.log 2>&1
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-1.log 2>&1
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-2.log 2>&1
qemu-arm -L /usr/arm-linux-gnueabihf fuzz_target corpus >fuzz-3.log 2>&1
[Source]: https://i.imgflip.com/x2cxg.jpg
Modify libFuzzer for ARM
Build example :
[user@host] $ clang --target arm-none-gnueabihf -L /usr/arm-none-gnueabihf/lib \
--sysroot= /usr/arm-none-gnueabihf -I /usr/arm-none-gnueabihf/include \
--resources /usr/arm-none-gnueabihf/lib/clang/7.0.0/linux/lib example.c
Steps to get llvm for arm :
● Setup X-Compile environment (arm-none-eabi, arm-linux-eabi, …)
● Modify libFuzzer Sources, such that will compile into ARM architecture
● Install the new binary to the specific path
● specify the path with compile option --resources
34
Advantages :
● You don’t need specific devices
● Don’t need to install a full system (it saves time)
● The performance is nearly the same compared to SoC's thanks to X86 architecture
● Don’t need to specify the hardware like the board or peripherals
● Easy to automate the build process for example with docker
● no RAM limitations, scalable
Fuzzing with QEMU User Mode
Disadvantages :
● Need a cross compile environment, but only once !
● performance weakness (?)
35
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer36
Modul Tests Integration Tests System Tests
Modul 1 Modul n
Fuzzing
API
Modul 1 Modul n
FuzzingFuzzing Fuzzing Fuzzing
Fuzzing
used as seed used as seed
Virtualization with QEMU Hardware
Features :
● complete system test of the project
● simulation of external interfaces like bluetooth, CAN-Bus or others
but …
● testing has slow performance (low executions per seconds)
● only a limited number of boards available !!
● implementing new or custom boards needs a lot of time
Fuzzing with QEMU full system emulation
37
How does it work :
● You have to figure out which board you use for fuzzing
● Depending on the board, there are specific processors available
○ Example : versatileab
- arm1026
- arm1136 ...
● You have to choose peripheral like the network card or a hard disk drive
● You have to download the right (netinstall)-kernel to start the system and maybe install a new OS
● Compiling projects inside the virtual machine is the same as on a real machine [2]
Fuzzing with QEMU full system emulation
Example :
[user@host] $ qemu-system-arm -M versatileab -cpu arm1136 -hda disk.vdi -append "root=/dev/sda2 rootfstype=ext4 rw" -serial stdio
38
Advantages :
● Can be used for acceptance test
● QEMU covers for almost all main architectures
● Changing hardware settings of the virtual board is easy (add CPU cores, change RAM-size, ...)
● Opportunity to fuzz kernel code
Fuzzing with QEMU full system emulation
Disadvantages :
● Hard to setup a virtual environment, it takes time
● Very hard to automate the setup process as well as the fuzzing process
● performance weakness (more in the evaluation)
39
[Source]: https://wiki.qemu.org/Logo
Corpus
Our approach
[1] Sources: https://github.com/google/sanitizers/wiki/AddressSanitizer40
Modul Tests Integration Tests System Tests
Modul 1 Modul n
Fuzzing
API
Modul 1 Modul n
FuzzingFuzzing Fuzzing Fuzzing
Fuzzing
used as seed used as seed
Virtualization with QEMU Hardware
Advantages :
● Easy to setup the system environment
● Applications and binaries runs on a nativ system / real interfaces
Disadvantages :
● Uses black box fuzzing on protocols
● Doesn’t have the feedback technique like coverage-guided fuzzing
● there are more than thousands devices devices on the market available
● limitations of the SoC boards (performance is weak)
Fuzzing on SoC devices
41
[Source]: https://cdn-reichelt.de/bilder/web/xxl_ws/A300/RASP_03_01.png
42
What does the performance look like ?
Test setup and projects
● Compare the performance between :○ SoC devices○ Virtual machines on two boards○ QEMU user mode on my laptop
● Metrics : ○ Executions per seconds○ Memory usage
SystemSoC
Board(RPi)
Vexpress-A9 virt Laptop
Architecture ARMv7 ARMv7 ARMv7 X86_64
CPU-Cores4
(Cortex-A9)
1(Cortex-
A9)
4(Cortex-
A53)
8(Core i7)
RAM 1 BG 1 GB 4 GB 32 GB
OS ArchLinux Debian Debian ArchLinux
Projects :
● RE2 - RegEx parser written in c++
● woff2 - WOFF File Format 2.0
● These projects belong to the fuzzer-test-suite
from google[https://github.com/google/fuzzer-test-suite]
43
Evaluation of RE2
System SoC Board(RPi) Vexpress-A9 virt QEMU mode
exec / s(after 1 min) ~6500 ~980 ~850 ~4000
exec / s(after 5 min) ~5000 ~1010 ~980 ~2900
exec / s(after 10 min) ~3200 ~830 ~800 ~1700
exec / s(after 60 min) aborted aborted ~180 ~460
time
44
Evaluation of WOFF
System SoC Board(RPi) Vexpress-A9 virt QEMU mode
exec / s(after 1 min) ~37000 ~6000 ~17000 ~35000
exec / s(after 5 min) ~33000 ~5800 ~13000 ~30000
exec / s(after 10 min) ~9000 ~5000 ~2500 ~15000
exec / s(after 60 min) ~5000 ~2000 ~2000 bug found
time
45
● QEMU-Mode has the best long term performance
● Full system emulation is nearly 80 % slower than the SoC
● The longer the test goes the slower the execution speed is
● Fuzzing on SoC is limited by the memory and the processor
Summary of performance evaluation
46
47
Structure Aware FuzzingOr how to stop wasting 99.9% of your Fuzzer
Runs
48
Why make the fuzzer structure-aware?
49
50
51
{ "name": "John Wick", "job": "Assassin", "pets": ["Buster"]}
52
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John Wick", "job": "Assassin", "pets": ["Buster"}}
53
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John", "pets": ["Buster"}}
{ "name": "John Wick", "job": "Assassin", "pets": ["Buster"}}
54
How to make the fuzzer structure-aware?
55
“We at Google have a big Hammer. It’s called Protobuf”
-- Kostya Serebryany, Google LibFuzzer Team
Protobuf Example
56
message Person { string name = 1; string profession = 2;
message Pet { string name = 1; }
repeated Pet pets = 3;}
person.proto
#include <stdio.h>#include "person.pb.h"
int main() { Person p; p.set_name("John Wick"); p.set_profession("Assassin"); p.add_pets().set_name("Buster"); p.add_pets().set_name("Spot"); std::cerr << p.DebugString();}
person.cpp
name: "John Wick"profession: "Assassin"
pets { name: "Buster"}
pets { name: "Spot"}
stdout
57
message JsonValue { oneof value_oneof { JsonObject object = 1; JsonArray array = 2; bytes string = 3; Number number = 4; bool boolean = 5; }}message JsonObject { repeated JsonKeyValPair keyvalpairs = 1;}message JsonArray { repeated JsonValue value = 1;}message JsonKeyValPair { required bytes key = 1;}message Number { required bool scientific = 1 [default = false]; oneof number_oneof { sint32 number = 2; double float = 3; }}
...[Source]: https://www.json.org/
Protobuf Conversion
58
{ "name": "John Wick", "profession": "Assassin", "pets": ["Buster"]}
object { keyvalpairs { key: "name" value { string: "John Wick" } } keyvalpairs { key: "profession" value { string: "Assassin" } } keyvalpairs { key: "pets" value { array { value { string: "Buster" } } } }}
ProtoToJson()
JsonToProto()
Protobuf Conversion
59
name: John Wickprofession: Assassinpets: - Buster - Spot
object { keyvalpairs { key: "name" value { string: "John Wick" } } keyvalpairs { key: "profession" value { string: "Assassin" } } keyvalpairs { key: "pets" value { array { value { string: "Buster" } } } }}
ProtoToYaml()
YamlToProto()
Protobuf Mutation
60
https://github.com/google/libprotobuf-mutator
[Source]:https://www.reddit.com/r/utterlyuseless/
Protobuf Mutation
● Protobuf is strict -> every mutation is again valid protobuf
● Every valid Protobuf can be converted to valid JSON/YAML/XML/…
● Mutation algorithm can be customized with little effort
● Integration into libFuzzer (!)
● Adding a new structured language is only 3 steps
○ Writing a Proto Message Definition
○ Write a Converter to Proto
○ Write a Converter to Structured Language
● You’re ready to fuzz only in the language you defined!
61
Protobuf Mutation
● Message Definition in Proto is higher level abstraction
●
62
Real-life Evaluation
● We took a closer look at GnuPG Made Easy (GPGME)
● GPGME ships own JSON parser
63
● Sergej Dechand and Sirko Höer (both Code Intelligence) found 12 bugs
● 2 Stackoverflows
● also integer overflows and memory leaks
Conclusion
64
● Three different approaches for embedded fuzzing are shown
● Modify compiler-RT to make libFuzzer available for ARM32/ ARM64 / MIPS32 / MIPS64
● The performance of QEMU-User-Mode is pretty impressive (partly much faster than SoC)
● Scalable : it can be executed in a docker container on a server
● Structure Aware Fuzzing is able to minimize Fuzzer Runs on parsing engines
● Bypasses Parsers, enables fuzzing of the business logic
65
Are there any questions left ?
66
Thank you for your attention