why johnny can't pentest: an analysis of black-box web vulnerability scanners
DESCRIPTION
Presentation at DIMVA 2010 of the paper "Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners"Full paper:http://cs.ucsb.edu/~adoupe/static/black-box-scanners-dimva2010.pdfTRANSCRIPT
![Page 1: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/1.jpg)
Why Johnny Can’t Pentest:An Analysis of Black-box
Web Vulnerability ScannersAdam Doupé, Marco Cova and Giovanni
VignaUniversity of California, Santa Barbara
DIMVA 2010 - 7/8/10
![Page 2: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/2.jpg)
Introduction to black box web vulnerability scanners
Design of custom vulnerable website – WackoPicko
Results Analysis
Outline
![Page 3: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/3.jpg)
Describe the design of a testing web application
Identify a number of challenges that scanners need to overcome when testing modern web applications
Test the performance of eleven real-world scanners and identify areas that need further work
Contributions
![Page 4: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/4.jpg)
Web Application Vulnerability Scanners
Server
Crawler Attack Analysis
![Page 5: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/5.jpg)
Authentication Upload Pictures Comment on Pictures “Purchase” Pictures Tag Search Guestbook Admin Area
Vulnerable Web Application – WackoPicko: Design
![Page 6: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/6.jpg)
XSS◦ Reflected, Stored, and Reflected behind JavaScript
Session ID Weak Password Reflected SQL Injection Command Line Injection File Inclusion File Exposure Parameter Manipulation
Vulnerable Web Application – WackoPicko: Publicly Accessible
![Page 7: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/7.jpg)
Reflected XSS behind Flash Stored SQL Injection Directory Traversal Multi-step Stored XSS Forceful Browsing Logic Flaw
Vulnerable Web Application – WackoPicko: Authentication
![Page 8: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/8.jpg)
HTML Parsing Multi-Step Process / State Infinite Website Authentication Client-side Code
◦ Web Input Vector Extractor Teaser (WIVET)
Crawling Challenges
![Page 9: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/9.jpg)
Name Price
Acunetix $4,995 - $6,350
AppScan $12,550 - $32,500
Burp £125 ($190.82)
Grendel-Scan Open source
Hailstorm $10,000
Milescan $495 - $1,495
N-Stalker $899 - $6,299
NTOSpider $10,000
Paros Open source
w3af Open source
Webinspect $6,000 - $30,000
Scanners
![Page 10: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/10.jpg)
Each scanner run four times:◦ WackoPicko
Initial – No configuration (point and click) Config – Given valid Username/Password Manual – Used proxy to thoroughly browse site.
◦ WIVET – Testing JavaScript capabilities
Limitations
Experiment
![Page 11: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/11.jpg)
ResultsName Reflected
XSSStored XSS
SQL Injection
Command line Injection
File Inclusion
File Exposure
XSS via JavaScript
XSS via Flash
Acunetix Initial Initial Initial Initial Initial Initial
AppScan Initial Initial Initial Initial Initial
Burp Initial Manual Initial Initial Initial Manual
Grendel-Scan
Manual Config
Hailstorm Initial Config Config Manual
Milescan Initial Manual Config
N-Stalker Initial Manual Manual Initial Initial Manual
NTOSpider Initial Initial Initial
Paros Initial Initial Config Manual
w3af Initial Manual Initial Initial Manual
Webinspect Initial Initial Initial Initial Initial Manual
![Page 12: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/12.jpg)
Results
![Page 13: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/13.jpg)
Missed by all scanners◦ Session ID◦ Weak Password◦ Parameter Manipulation◦ Forceful Browsing◦ Logic Flaw
Will discuss later◦ Stored SQL Injection◦ Directory Traversal◦ Stored XSS Behind Login
Missed Vulnerabilities
![Page 14: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/14.jpg)
Ranged from 0 to 200+ ◦ Average was ~25
Why?◦ Server Path Disclosure
“Actual” False Positives◦ Hailstorm
XSS, 2 Code Injection◦ NTOSpider
3 XSS◦ w3af
PHP eval() Injection
False Positives
![Page 15: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/15.jpg)
Strictly Dominates
Measuring and Comparing Detection Capabilities
![Page 16: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/16.jpg)
Dominates GraphMore Dominant
Less Dominant
![Page 17: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/17.jpg)
Default values XSS attacks Command-line Injection SQL Injection File Exposure Remote Code Execution
Attack and Analysis Capabilities
![Page 18: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/18.jpg)
Number of Accesses◦ Range from ~50 per page to ~3,000 per page◦ Hailstorm accessed vulnerable pages that required an
account on INITIAL scan!
HTML◦ Burp and N-Stalker
<TEXTAREA>◦ Milescan and Grendel-Scan
POST◦ Hailstorm
No-Injection◦ w3af
No Default
Crawling Capabilities
![Page 19: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/19.jpg)
Uploading a Picture◦ 2 Scanners uploaded without help
◦ 3 Scanners unable to upload one!
Crawling Capabilities
![Page 20: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/20.jpg)
WIVET◦ 3 Scanners couldn’t complete
Paros and Burp - <base> N-Stalker – Frame?
◦ Dynamic JavaScript Webinspect, Acunetix, NTOSpider, Hailstorm
◦ JavaScript library◦ No Flash
Crawling Capabilities - Client-side Code
![Page 21: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/21.jpg)
Created an account successfully◦ 4 Scanners
Hailstorm N-Stalker NTOSpider WebInspect
Crawling Capabilities - Authentication
![Page 22: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/22.jpg)
Incorporate lots of logging in the application Two versions of the site
◦ No vulnerabilities◦ All vulnerabilities
Script running the tests Include:
◦ File upload forms◦ AJAX◦ Several JavaScript UI Libraries
Lessons Learned: Want to make your own benchmark?
![Page 23: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/23.jpg)
Ability to crawl as important as detection
Many vulnerabilities cannot be detected
Cost not directly proportional to functionality
Conclusions
![Page 24: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/24.jpg)
Questions?
![Page 25: Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners](https://reader036.vdocuments.us/reader036/viewer/2022062405/554f4b3fb4c905b9508b48e3/html5/thumbnails/25.jpg)
Thanks!