restler: stateful rest api fuzzing
TRANSCRIPT
![Page 1: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/1.jpg)
RESTler: Stateful REST API Fuzzing
Vaggelis Atlidakis (Columbia University), Patrice Godefroid (Microsoft Research), and Marina Polishchuk (Microsoft Research)
![Page 2: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/2.jpg)
Over the past decade❖ Explosion of cloud services (in Azure and AWS)
❖ Rapidly evolving ecosystem
❖ REST APIs is the standard way to use cloud services
RESTler: Stateful REST API Fuzzing
![Page 3: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/3.jpg)
Over the past decade❖ Explosion of cloud services (in Azure and AWS)
❖ Rapidly evolving ecosystem
❖ REST APIs is the standard way to use cloud services
➢ What about testing?
RESTler: Stateful REST API Fuzzing
![Page 4: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/4.jpg)
Testing REST APIs
RESTler: Stateful REST API Fuzzing
![Page 5: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/5.jpg)
Testing REST APIs❖ Grammar-based fuzzing (e.g., Peach, SPIKE, ...)
➢ Requires manual effort➢ New grammar for every new service
RESTler: Stateful REST API Fuzzing
![Page 6: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/6.jpg)
Testing REST APIs❖ Grammar-based fuzzing (e.g., Peach, SPIKE, ...)
➢ Requires manual effort➢ New grammar for every new service
❖ HTTP fuzzers (e.g., Sulley, Burp, ...)➢ Requires live traffic➢ Not Stateful
RESTler: Stateful REST API Fuzzing
![Page 7: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/7.jpg)
Testing REST APIs❖ Grammar-based fuzzing (e.g., Peach, SPIKE, ...)
➢ Requires manual effort➢ New grammar for every new service
❖ HTTP fuzzers (e.g., Sulley, Burp, ...)➢ Requires live traffic➢ Not Stateful
❖ Custom tools for specific APIs ➢ Labour intensive➢ High maintenance
RESTler: Stateful REST API Fuzzing
![Page 8: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/8.jpg)
Our solution➢ RESTler: A stateful REST API fuzzer
RESTler: Stateful REST API Fuzzing
![Page 9: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/9.jpg)
Our solution➢ RESTler: A stateful REST API fuzzer
Key techniques for stateful REST API fuzzing
1. Dependency analysis between request types
RESTler: Stateful REST API Fuzzing
![Page 10: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/10.jpg)
Our solution➢ RESTler: A stateful REST API fuzzer
Key techniques for stateful REST API fuzzing
1. Dependency analysis between request types
2. Dynamic feedback loop that learns from past tests
RESTler: Stateful REST API Fuzzing
![Page 11: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/11.jpg)
Our solution➢ RESTler: A stateful REST API fuzzer
Kinds of bugs RESTler can find
➢ “500 Internal Server Error” (unhandled exceptions) after executing a sequence of API requests
RESTler: Stateful REST API Fuzzing
![Page 12: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/12.jpg)
Outline❖ Limitations of existing solutions
❖ System overview
❖ Evaluation & bugs found
❖ Experiences with public cloud services
❖ Conclusions
RESTler: Stateful REST API Fuzzing
![Page 13: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/13.jpg)
System overview
REST APIspecification
(e.g., Swagger)
RESTler: Stateful REST API Fuzzing
![Page 14: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/14.jpg)
System overview
REST APIspecification
(e.g., Swagger)
RESTler compiler
❖ Describe how to fuzz each request type
❖ Identify producer/consumer dependencies
❖ Generate code to parse responses
RESTler: Stateful REST API Fuzzing
RESTler grammar
(currently in Python)
![Page 15: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/15.jpg)
System overview
REST APIspecification
(e.g., Swagger)
RESTler compiler
RESTler test engine
❖ Generate and execute tests: sequences of requests
❖ Systematic state-space exploration (breadth first search and others)
❖ Analyze test results: Dynamic feedback loop learns from service responses in past tests
Tests & bugs
RESTler: Stateful REST API Fuzzing
RESTler grammar
(currently in Python)
❖ Describe how to fuzz each request type
❖ Identify producer/consumer dependencies
❖ Generate code to parse responses
![Page 16: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/16.jpg)
Example
Sample Swagger specification RESTler grammar fragment
Sample test (request and response)
RESTler: Stateful REST API Fuzzing
...
![Page 17: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/17.jpg)
Outline❖ Limitations of existing solutions
❖ System overview
❖ Evaluation & bugs found
❖ Experiences with public cloud services
❖ Conclusions
RESTler: Stateful REST API Fuzzing
![Page 18: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/18.jpg)
Questions➢ Q1: Are tests generated by RESTler exercising deeper
service-side logic over time?
➢ Q2: Can RESTler find bugs in large-scale production services?
RESTler: Stateful REST API Fuzzing
![Page 19: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/19.jpg)
Questions➢ Q1: Are tests generated by RESTler exercising deeper
service-side logic over time?
➢ Q2: Can RESTler find bugs in large-scale production services?
Case study: Gitlab❖ Open-source self-hosted GIT service (millions of users)
❖ ~376 kLOC (Ruby + native libraries)
❖ Complex REST API
RESTler: Stateful REST API Fuzzing
![Page 20: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/20.jpg)
Deeper service exploration (Q1)RESTler: Stateful REST API Fuzzing
Testing GitLab APIs with RESTler (5h per API family)
API Family
Total requests
Seq. len.
Cumulative code coverage
(lines of code)
Tests
Commits 11 1 598 12 1108 73 1196 2504 1760 22205 1760 3667
Branches 7 1 598 12 1089 83 1172 584 1182 5765 1185 3644
Issues 22 1 816 372 1163 24443 1163 4156
Repos 10 1 598 12 1117 973 1181 5153
![Page 21: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/21.jpg)
Deeper service exploration (Q1)RESTler: Stateful REST API Fuzzing
Testing GitLab APIs with RESTler (5h per API family)
API Family
Total requests
Seq. len.
Cumulative code coverage
(lines of code)
Tests
Commits 11 1 598 12 1108 73 1196 2504 1760 22205 1760 3667
Branches 7 1 598 12 1089 83 1172 584 1182 5765 1185 3644
Issues 22 1 816 372 1163 24443 1163 4156
Repos 10 1 598 12 1117 973 1181 5153
❖ Longer sequences increase service-side code coverage
![Page 22: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/22.jpg)
Deeper service exploration (Q1)RESTler: Stateful REST API Fuzzing
Testing GitLab APIs with RESTler (5h per API family)
API Family
Total requests
Seq. len.
Cumulative code coverage
(lines of code)
Tests
Commits 11 1 598 12 1108 7
3 1196 2504 1760 22205 1760 3667
Branches 7 1 598 12 1089 8
3 1172 584 1182 5765 1185 3644
Issues 22 1 816 372 1163 2444
3 1163 4156Repos 10 1 598 1
2 1117 97
3 1181 5153
❖ Longer sequences increase service-side code coverage
❖ Sequences of 3 requests (at least)
![Page 23: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/23.jpg)
Deeper service exploration (Q1)RESTler: Stateful REST API Fuzzing
Testing GitLab APIs with RESTler (5h per API family)
API Family
Total requests
Seq. len.
Cumulative code coverage
(lines of code)
Tests
Commits 11 1 598 1
2 1108 73 1196 2504 1760 22205 1760 3667
Branches 7 1 598 12 1089 83 1172 584 1182 5765 1185 3644
Issues 22 1 816 372 1163 24443 1163 4156
Repos 10 1 598 12 1117 973 1181 5153
❖ Longer sequences increase service-side code coverage
❖ Sequences of 3 requests (at least)
❖ Progress in a huge search space Testing Commits API (5 hours)
➢ Brute-force: 11 request types / 4 renderings on avg /(11*4)^3 = 85k feasible sequences of length 3
![Page 24: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/24.jpg)
Deeper service exploration (Q1)RESTler: Stateful REST API Fuzzing
Testing GitLab APIs with RESTler (5h per API family)
❖ Longer sequences increase service-side code coverage
❖ Sequences of 3 requests (at least)
❖ Progress in a huge search space Testing Commits API (5 hours)
➢ Brute-force: 11 request types / 4 renderings on avg /(11*4)^3 = 85k feasible sequences of length 3
➢ RESTler: Seq. Len. 3 / Test generated 250
(feedback + dependencies!)
API Family
Total requests
Seq. len.
Cumulative code coverage
(lines of code)
Tests
Commits 11 1 598 12 1108 73 1196 2504 1760 22205 1760 3667
Branches 7 1 598 12 1089 83 1172 584 1182 5765 1185 3644
Issues 22 1 816 372 1163 24443 1163 4156
Repos 10 1 598 12 1117 973 1181 5153
![Page 25: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/25.jpg)
New bugs found in GitLab (Q2)
RESTler: Stateful REST API Fuzzing
Testing GitLab APIs with RESTler (5h per API family)
API Family BFS BFS-
FastRandom-
Walk ⋂ UCommits 5 1 5 1 5
Branches 7 7 7 5 8
Issues 0 1 1 0 1
Repos 2 3 3 2 3
Groups 0 0 2 0 2
Projects 2 1 3 1 3
Total 16 13 21 9 22
![Page 26: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/26.jpg)
New bugs found in GitLab (Q2)
❖ 22 new bugs found on Aug. ’18 (+6 bugs found on Apr. ‘18)
RESTler: Stateful REST API Fuzzing
API Family BFS BFS-
FastRandom-
Walk ⋂ UCommits 5 1 5 1 5
Branches 7 7 7 5 8
Issues 0 1 1 0 1
Repos 2 3 3 2 3
Groups 0 0 2 0 2
Projects 2 1 3 1 3
Total 16 13 21 9 22
Testing GitLab APIs with RESTler (5h per API family)
![Page 27: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/27.jpg)
New bugs found in GitLab (Q2)
❖ 22 new bugs found on Aug. ’18 (+6 bugs found on Apr. ‘18)
❖ All bugs were disclosed to Gitlab developers
RESTler: Stateful REST API Fuzzing
API Family BFS BFS-
FastRandom-
Walk ⋂ UCommits 5 1 5 1 5
Branches 7 7 7 5 8
Issues 0 1 1 0 1
Repos 2 3 3 2 3
Groups 0 0 2 0 2
Projects 2 1 3 1 3
Total 16 13 21 9 22
Testing GitLab APIs with RESTler (5h per API family)
![Page 28: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/28.jpg)
New bugs found in GitLab (Q2)
❖ 22 new bugs found on Aug. ’18 (+6 bugs found on Apr. ‘18)
❖ All bugs were disclosed to Gitlab developers
❖ All bugs were easily reproducible, confirmed, and fixed!
RESTler: Stateful REST API Fuzzing
API Family BFS BFS-
FastRandom-
Walk ⋂ UCommits 5 1 5 1 5
Branches 7 7 7 5 8
Issues 0 1 1 0 1
Repos 2 3 3 2 3
Groups 0 0 2 0 2
Projects 2 1 3 1 3
Total 16 13 21 9 22
Testing GitLab APIs with RESTler (5h per API family)
![Page 29: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/29.jpg)
New bugs found in GitLab (Q2)
❖ Example Bug [#50268]1. Create a gitlab project2. Create a repository file with a
proper commit message3. Delete the repository file with an
empty commit message
RESTler: Stateful REST API Fuzzing
API Family BFS BFS-
FastRandom-
Walk ⋂ UCommits 5 1 5 1 5
Branches 7 7 7 5 8
Issues 0 1 1 0 1
Repos 2 3 3 2 3
Groups 0 0 2 0 2
Projects 2 1 3 1 3
Total 16 13 21 9 22
Testing GitLab APIs with RESTler (5h per API family)
![Page 30: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/30.jpg)
New bugs found in GitLab (Q2)
❖ Example Bug [#50268]1. Create a gitlab project2. Create a repository file with a
proper commit message3. Delete the repository file with an
empty commit message➢ “500 Internal Server Error”
RESTler: Stateful REST API Fuzzing
API Family BFS BFS-
FastRandom-
Walk ⋂ UCommits 5 1 5 1 5
Branches 7 7 7 5 8
Issues 0 1 1 0 1
Repos 2 3 3 2 3
Groups 0 0 2 0 2
Projects 2 1 3 1 3
Total 16 13 21 9 22
Testing GitLab APIs with RESTler (5h per API family)
![Page 31: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/31.jpg)
Outline❖ Limitations of existing solutions
❖ System overview
❖ Evaluation & bugs found
❖ Experiences with public cloud services
❖ Conclusions
RESTler: Stateful REST API Fuzzing
![Page 32: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/32.jpg)
Experiences with Azure and Office 365❖ Four production cloud services with open-source specs
➢ Resource management Azure services➢ Real-time messaging Office 365 service
RESTler: Stateful REST API Fuzzing
![Page 33: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/33.jpg)
Experiences with Azure and Office 365❖ Four production cloud services with open-source specs
➢ Resource management Azure services➢ Real-time messaging Office 365 service
❖ Needed new features➢ Garbage Collection (resource quotas)➢ Authentication Hooks (short-lived access tokens)➢ Resource-specific mutations (exotic naming schemes)
RESTler: Stateful REST API Fuzzing
![Page 34: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/34.jpg)
Experiences with Azure and Office 365❖ Four production cloud services with open-source specs
➢ Resource management Azure services➢ Real-time messaging Office 365 service
❖ Needed new features➢ Garbage Collection (resource quotas)➢ Authentication Hooks (short-lived access tokens)➢ Resource-specific mutations (exotic naming schemes)
➢ RESTler found bugs in all services tested so far!
RESTler: Stateful REST API Fuzzing
![Page 35: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/35.jpg)
Conclusions❖ Build the first stateful REST API fuzzer!
❖ Found bugs in Azure and Office 365 cloud services!
❖ Found 28 new bugs in Gitlab!
RESTler: Stateful REST API Fuzzing
![Page 36: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/36.jpg)
Conclusions❖ Build the first stateful REST API fuzzer!
❖ Found bugs in Azure and Office 365 cloud services!
❖ Found 28 new bugs in Gitlab!
➢ Developers are fixing the bugs
found with RESTler!
RESTler: Stateful REST API Fuzzing
![Page 37: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/37.jpg)
Thank you!RESTler: Stateful REST API Fuzzing
Paper linkhttps://tinyurl.com/yyg5a8je
![Page 38: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/38.jpg)
Thank you!RESTler: Stateful REST API Fuzzing
Paper linkhttps://tinyurl.com/yyg5a8je
![Page 39: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/39.jpg)
Scalability of state-space exploration strategies
RESTler: Stateful REST API Fuzzing
![Page 40: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/40.jpg)
Impact of the two key techniques
RESTler: Stateful REST API Fuzzing
![Page 41: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/41.jpg)
Extending sequences in Randoop
RESTler: Stateful REST API Fuzzing
![Page 42: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/42.jpg)
Sample bugfix in GitlabRESTler: Stateful REST API Fuzzing
![Page 43: RESTler: Stateful REST API Fuzzing](https://reader031.vdocuments.us/reader031/viewer/2022012013/6158403ffde8212b9d705769/html5/thumbnails/43.jpg)
Developers’ Responses
#50276
#50272
#50677
RESTler: Stateful REST API Fuzzing