the owasp foundation owasp nova october 4 th 2012 benchmarking web application scanners for your...
TRANSCRIPT
![Page 1: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/1.jpg)
The OWASP Foundationhttp://www.owasp.org
OWASP NoVAOctober 4th 2012
Benchmarking Web Application Scanners
for YOUR Organization
Dan CornellFounder and CTO, Denim Group
OWASP San Antonio ChapterOWASP Global Membership Committee
[email protected]: @danielcornell
![Page 2: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/2.jpg)
The OWASP Foundationhttp://www.owasp.org
2
My Background
Dan Cornell, founder and CTO of Denim Group
Software developer by background (Java, .NET, etc)
OWASP San Antonio, Global Membership Committee
![Page 3: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/3.jpg)
The OWASP Foundationhttp://www.owasp.org
What Do You Want From a Scanner?
Coverage
Low False Positives
Low False Negatives
3
![Page 4: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/4.jpg)
The OWASP Foundationhttp://www.owasp.org
Scanner CoverageYou can’t test what you can’t see
How effective is the scanner’s crawler?
How are URLs mapped to functionality?• RESTful• Parameters
Possible issues:• Login routines• Multi-step processes• Anti-CSRF protection
4
![Page 5: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/5.jpg)
The OWASP Foundationhttp://www.owasp.org
Are You Getting a Good Scan?Large financial firm: “Our 500 page website is secure because the scanner did not find any vulnerabilities!”
Me: “Did you teach the scanner to log in so that it can see more than just the homepage?”
Large financial firm: “…”
5
![Page 6: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/6.jpg)
The OWASP Foundationhttp://www.owasp.org
Can Your Scanner Do This?
Two-step login procedure:• Enter username / password (pretty standard)• Enter answer to one of several arbitrary questions
Challenge was that the parameter indicating the question was dynamic• Question_1, Question_2, Question_3, and so on• Makes standard login recording ineffective
6
![Page 7: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/7.jpg)
The OWASP Foundationhttp://www.owasp.org
It All Started With A Simple Blog Post…
Ran into an application with a complicated login procedure and wrote blog post about the toolchain used to solve the problemhttp://blog.denimgroup.com/denim_group/2012/04/automated-application-scanning-handling-complicated-logins-with-appscan-and-burp-suite.html
Other scanner teams responded:• IBM Rational AppScan http://blog.denimgroup.com/denim_group/2012/04/automated-application-scanning-handling-complicated-logins-with-appscan-only.html • HP WebInspecthttp://blog.denimgroup.com/denim_group/2012/05/handling-challengeresponse-logins-in-hp-webinspect.html• Mavituna Security Netsparkerhttp://blog.denimgroup.com/denim_group/2012/05/handling-challengeresponse-logins-in-mavituna-netsparker.html• NTObjectives NTOSpiderhttp://blog.denimgroup.com/denim_group/2012/05/handling-challengeresponse-logins-in-ntospider.html
7
![Page 8: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/8.jpg)
The OWASP Foundationhttp://www.owasp.org
Scanner Authentication Scenario Examples
Built as a response to the previously-mentioned blog conversation
Example implementations of different login routinesHow can different scanners be configured to successfully scan?
GitHub site:https://github.com/denimgroup/authexamples
8
![Page 9: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/9.jpg)
The OWASP Foundationhttp://www.owasp.org
Did I Get a Good Scan?Scanner training is really important• Read the Larry Suto reports…
Must sanity-check the results of your scans
What URLs were accessed?• If only two URLs were accessed on a 500 page site, you probably have a bad
scan• If 5000 URLs were accessed on a five page site, you probably have a bad scan
What vulnerabilities were found and not found?• Scan with no vulnerabilities – probably not a good scan• Scan with excessive vulnerabilities – possibly a lot of false positives
9
![Page 10: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/10.jpg)
The OWASP Foundationhttp://www.owasp.org
Low False Positives
Reports of vulnerabilities that do not actually exist
How “touchy” is the scanner’s testing engine?
Why are they bad?Take time to manually review and filter outCan lead to wasted remediation time
10
![Page 11: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/11.jpg)
The OWASP Foundationhttp://www.owasp.org
Low False Negatives
Scanner failing to report vulnerabilities that do exist
How effective is the scanner’s testing engine?
Why are they bad?You are exposed to risks you do not know aboutYou expect that the scanner would have found certain classes of vulnerabilities
What vulnerability classes do you think scanners will find?
11
![Page 12: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/12.jpg)
The OWASP Foundationhttp://www.owasp.org
Other Benchmarking EffortsLarry Suto’s 2007 and 2010 reports• Analyzing the Accuracy and Time Costs of Web Application Security Standardshttp://ha.ckers.org/files/Accuracy_and_Time_Costs_of_Web_App_Scanners.pdf• Vendor reactions were … varied[Ofer Shezaf attended this talk at AppSecEU 2012 and had some great questions and comments. See his reactions to the latest Larry Suto scanner report here :http://www.xiom.com/2010/02/09/wafs-are-not-perfect-any-security-tool-perfect ]
Shay Chen’s Blog and Site• http://sectooladdict.blogspot.com/• http://www.sectoolmarket.com/• http://www.infosecisland.com/blogview/21926-A-Step-by-Step-Guide-for-Choosing-the-Best-
Scanner.html
Web Application Vulnerability Scanner Evaluation Project (wavsep)• http://code.google.com/p/wavsep/
12
![Page 13: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/13.jpg)
The OWASP Foundationhttp://www.owasp.org
So I Should Just Buy the Best Scanner, Right?
Or the cheapest?
Well…What do you mean by “best”?
Follow-on questions• How well do the scanners work on your
organization’s applications?• How many false positives are you willing to
deal with?• What depth and breadth of coverage do
you need?
13
![Page 14: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/14.jpg)
The OWASP Foundationhttp://www.owasp.org
ThreadFix - Overview
ThreadFix is a software vulnerability aggregation and management system that helps organizations aggregate vulnerability data, generate virtual patches, and interact with software defect tracking systems.
Freely available under the Mozilla Public License (MPL)
Hosted at Google Code: http://code.google.com/p/threadfix/
14
![Page 15: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/15.jpg)
The OWASP Foundationhttp://www.owasp.org
What is a Unique Vulnerability?
(CWE, Relative URL)Predictable resource locationDirectory listing misconfiguration
(CWE, Relative URL, Injection Point)SQL injectionCross-site Scripting (XSS)
Injection pointsParameters – GET/POSTCookiesOther headers
15
![Page 16: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/16.jpg)
The OWASP Foundationhttp://www.owasp.org
What Do The Scanner Results Look Like?
Usually XMLSkipfish uses JSON and gets packaged as a ZIP
Scanners have different concepts of what a “vulnerability” isWe normalize to the (CWE, location, [injection point]) noted before
Look at some example files
Several vendors have been really helpful adding additional data to their APIs and file formats to accommodate requests (thanks!)
16
![Page 17: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/17.jpg)
The OWASP Foundationhttp://www.owasp.org
Let’s Look at Some Example Files
AppScan
Netsparker
w3af
Arachni
![Page 18: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/18.jpg)
The OWASP Foundationhttp://www.owasp.org
Why Common Weakness Enumeration (CWE)?
Every tool has their own “spin” on naming vulnerabilitiesOWASP Top 10 / WASC XX are helpful but not
comprehensive
We tried to create our own vulnerability classification schemeProprietaryNot sustainableStupid
CWE is pretty exhaustiveReasonably well-adopted standardMany tools have mappings to CWE for their results
Main site: http://cwe.mitre.org/
18
![Page 19: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/19.jpg)
The OWASP Foundationhttp://www.owasp.org
Challenges Using the CWE
• It is pretty big (909 nodes, 693 actual weaknesses)But it kind of has to be to be comprehensive…
• Many tools provide mappingsAnd sometimes they’re even kind of accurate!
• Some tools provide more than one CWE category for a vulnerability
So in ThreadFix we make a best guess
• Some tools provide “junk” resultsSo in ThreadFix we collapse those into a single
vulnerability
• Some organizations have their own classification schemes
![Page 20: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/20.jpg)
The OWASP Foundationhttp://www.owasp.org
Demo
• Unpack and install ThreadFix• Use ThreadFix to normalize and report on the use of
multiple scanning technologies on a given application• Import multiple scans and de-duplicate the results
These screenshots are based on UNTUNED scans and are NOT meant to show a real benchmark of these scanners – only the process.
20
![Page 21: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/21.jpg)
The OWASP Foundationhttp://www.owasp.org
Unzip the ThreadFix Package (like WebGoat!)
21
![Page 22: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/22.jpg)
The OWASP Foundationhttp://www.owasp.org
Make threadfix.sh Executable
22
![Page 23: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/23.jpg)
The OWASP Foundationhttp://www.owasp.org
Run ThreadFix Pre-Configured Tomcat Server
23
![Page 24: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/24.jpg)
The OWASP Foundationhttp://www.owasp.org
Login to ThreadFix (“user” and “password”)
24
![Page 25: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/25.jpg)
The OWASP Foundationhttp://www.owasp.org
Upload Various Scan Results Files
25
![Page 26: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/26.jpg)
The OWASP Foundationhttp://www.owasp.org
This Vulnerability Found By Three Scanners
26
![Page 27: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/27.jpg)
The OWASP Foundationhttp://www.owasp.org
Mark False Positives(wavsep Uses
“FalsePositives” In the URL…)
27
![Page 28: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/28.jpg)
The OWASP Foundationhttp://www.owasp.org
Summary Report – Found, Not Found, False Positives (Again – NOT
Based on Tuned Scans)
28
![Page 29: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/29.jpg)
The OWASP Foundationhttp://www.owasp.org
Report By Vulnerability Type
29
![Page 30: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/30.jpg)
The OWASP Foundationhttp://www.owasp.org
Detail Report Can Be Used To Error-Check Merge Process
30
![Page 31: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/31.jpg)
The OWASP Foundationhttp://www.owasp.org
Current LimitationsVulnerability importers are not currently formally vendor-supported• Though a number have helped us test and refine them
(thanks!)• After you get a good scan make sure you also got a good
import
Summary report should show data by severity rating• Make it easier to focus on vulnerabilities you probably
care more about• But you can look at the data by vulnerability type
31
![Page 32: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/32.jpg)
The OWASP Foundationhttp://www.owasp.org
Try This At Home, Kids
Pick some applications to test• Representative sample for your organization• Common languages, frameworks
Run scans with the targeted scanning technologies• Make sure you get good scans: login, other state-based issues• If you train the scans (always a good idea) be consistent
Import the scans into ThreadFix• Make sure you’re happy with the import• Complain vigorously to me if you run into import issues
[email protected] orhttps://code.google.com/p/threadfix/issues/list (better)
Run some reports
32
![Page 33: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/33.jpg)
The OWASP Foundationhttp://www.owasp.org
You Know What Would Make All This Way Easier?
Common data standards for scanning tools!
Current efforts:
• MITRE Software Assurance Findings Expression Schema (SAFES)
http://www.mitre.org/work/tech_papers/2012/11_3671/
• OWASP Data Exchange Format Projecthttps://www.owasp.org/index.php/OWASP_Data_Exchange_Format_Project
33
![Page 34: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/34.jpg)
The OWASP Foundationhttp://www.owasp.org
Simple Software Vulnerability Language (SSVL)
Common way to represent static and dynamic scanner findingsBased on our experience building importers for ThreadFix• It “works” for real-world applications because we are essentially using it.
Love to hear feedback• Send me a request and I can share the document for editing/annotation.
Online: • https
://docs.google.com/document/d/1H5hWUdj925TtoZ7ZvnfHdFABe7hBCGuZtLUas29yBGI/edit?pli=1
• Or http://tinyurl.com/cslqv47
34
![Page 35: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/35.jpg)
The OWASP Foundationhttp://www.owasp.org
Simple Software Vulnerability Language (SSVL)
35
![Page 36: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/36.jpg)
The OWASP Foundationhttp://www.owasp.org
Why Use Multiple Scanners?
• Better / different coverageOpen source scanners – no licensing issuesCommercial scanners – limited licensesSaaS services – portfolio coverage / economic questions
• Static and dynamic scannersSupported technologies and URL construction schemesUsed at different points in the processDifferent insight into the software system – internal vs.
external
• Sometimes you have no choiceSecurity licenses scanner ABC, IT Audit subscribes to
service XYZYou have to deal with the results of both scanners…
![Page 37: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/37.jpg)
The OWASP Foundationhttp://www.owasp.org
And Once You Have a Scanner…
…you need to use it
But you alreadyknew that
Right?
![Page 38: The OWASP Foundation OWASP NoVA October 4 th 2012 Benchmarking Web Application Scanners for YOUR Organization Dan Cornell Founder](https://reader038.vdocuments.us/reader038/viewer/2022110208/56649dd45503460f94acbbcf/html5/thumbnails/38.jpg)
The OWASP Foundationhttp://www.owasp.org
38
QuestionsDan [email protected]: @danielcornell
code.google.com/p/threadfix(210) 572-4400