dysfunctional testing - blue sky elearn · pdf filedysfunctional testing is thinking...
TRANSCRIPT
Dysfunctional Testing
Chris Romeo @edgeroute
• Chief Security Advocate,
Cisco Systems • Core CSDL Team Member
• Cisco Security Black Belt
(CSBB), CISSP
• Co-Founded the Cisco Security Awareness Program with Tony Vargas and Lisa McDonald in 2012
Tony Vargas @tvargasciodb
- Co-Founder & CEO - Security Together, LLC - CSSLP, CISSP-ISSAP - (ISC)2 President’s Award Winner – 2014 - Chair, Application Security Advisory Council - (ISC)2 - Cisco Security Champion of the Year – 2013 (Inaugural) - Cisco CIIP Mentorship Award Winner – 2013 - Cisco Security Champion (2012) - Cisco Security Black Belt - Cisco Client Partnership Award Winner - Co-Founder, Chairman & President - (ISC)2 Sacramento
Chapter
Agenda
• Define Dysfunctional Testing • Tools and Vehicles for Dysfunctional Testing
• The Process of Dysfunctional Testing
• Dysfunctional Call to Action
(ISC)2 e-Symposium 4
Trusting What We Test
VS
Product Testing • Check features for functionality
– Features that improve product or meet customer need
• Management does not prioritize security, but manages by bug counts
• Hard to drive security in a group
Customers expect both functional and application security testing
Functional Test Process
Functional Test - Does the feature or product work according to the written specification?
Definition: Dysfunctional Test
• Test cases that behave outside social norms – What the developer never
expected or imagined
• Creating a test case to uncover a malfunctioning part or element
• Not performing testing normally
Ted
Functional Testing: The process of checking if something works as designed
Functional Testing: The process of checking if something works as designed Dysfunctional Testing is the opposite
Dysfunctional Testing is thinking differently about how testers approach their products
Four Keys to Effective Dysfunctional Test
1. Goal oriented – what will be achieved? 1. Full product is in scope 1. Simulate realistic compromise patterns if production pieces
are in scope 1. Break testing into iterations
Source: Zane Lackey, @zlackey, DevOps Security Presentation; concept modified for dysfunctional.
Dysfunctional is Not…
• A call to turn all testers into white hat hackers (not possible)
• Just a negative test – negative
test is constrained to unexpected input
A Search for Motivation
• Misuse case – pinpointing the bad in the good • Agile user stories • Functional Spec – Anti-functional Spec • Threat Modeling • Mindset • DevOps • Rugged Software
A pattern of #dysfunction in your test process is good; a virtual hunt for vulnerabilities.
Tools of Dysfunction
• Testers brain!
• Teamwork • Codenomicon for fuzz testing
• Dynamic / Static Analysis • Root cause analysis
Dysfunctional Testing Vehicles
- Agile - DevOps
- Hack-a-Thon - Internal Bug Bounty
Dysfunctional is Difficult
• It can’t be done by one person -- requires a community • Persistence
• High level of knowledge • Knowledge ++ Experience ++ Industry Relationships
Six Steps of Dysfunctional Testing
1. Build a Tool Chest
2. Search for Motivation
3. Compute the Attack Surface
4. Plan the Attack
5. Execute the Attack
6. Document and Present Results/Demonstration to Others
1. Building Your Tool Chest
+ =
Kali tools for product security testing 1.Nmap 2.Nikto 3.W3AF 4.ZAP 5. SQLmap 6.Wireshark 7.Metasploit 8.Your Brain
2. Search for Motivation • Not traditional skim and approve; hunt for
attack vectors in the artifacts • Review the artifacts with YOUR
SECURITY HAT ON (critically) • Functional specifications (design,
software, hardware) • Test Plans • Agile User Stories
Being Critical for Security • Ask questions about the ideas and information
presented in the document • Take Notes: document your questions to ensure you define
answers or mark as unanswered
• Make judgments about the validity or relevance of the document to the security features and architecture of the product
Writing an Attack User Story
• As an attacker, I want to break <function> so that I can receive this <benefit>.
• As an attacker, I want to break the web administrative interface so that I can dump a list of passwords.
• As an attacker, I want to break the DNS daemon so that I can get elevated privilege on the product.
3. Attack Surface • Information gathering before an attack • Attack Surface -- collection of all entry points that could
be used to attack the product (software or hardware)
4. Planning the Attack • Refine the attack user stories by adding
prioritization – Figure of Merit {High, Medium, or Low}
• Probability of success for an attack • Impact of an attack • Ease of detecting the attack and attacker • Tool Available to Assist?
• Write test procedures to ensure repeatability
5. Executing the Attack • Automated tools?
• Simulate the attack manually?
– Netcat – Browser – Tools to create ip packets – Ruby or perl script
6. Documenting Results • Create an official defect / bug; the issue must be tracked
• Identified issues must be understandable by the
developer stand alone
• Share how the bug was found, and the issue becomes a learning tool for others
Dysfunctional Call to Action Will you commit? A Commitment to
Break Your Product
Conclusion • Release a portion of the testers from the binding of
functional test – Functional test is required; security testing is mandatory
• Expand your mind and tool sets
• Dysfunctional testing is not hard, the skills can be
learned
• Dysfunctional test is good for products
