a comparison of three bug-finding techniques and their relative effectiveness
TRANSCRIPT
Paraso& Proprietary and Confiden1al 1
2/26/15
A Comparison of Three Bug-‐finding Techniques and Their Rela:ve
Effec:veness
Mark Lambert -‐ Paraso& Corp
Paraso& Proprietary and Confiden1al 2 Paraso& Proprietary and Confiden1al 2
Agenda
§ Overview and Comparison 1. PaDern-‐based Analysis 2. Flow Analysis 3. Unit Tes1ng
§ Applica1on and Demonstra1on § Desktop analysis/tes1ng vs. on target § Using Applica1on Monitoring to uncover run1me problems
§ Combining Coverage
Paraso& Proprietary and Confiden1al 3 Paraso& Proprietary and Confiden1al 3
Sta1c Code Analysis
Pa?ern-‐Based Sta:c Analysis
Preven1on technique
Analyzes code structure (parse tree) to apply best
prac1ces
Flow-‐Based Analysis
Detec1on technique
Analyzes code flow to determine
“dangerous paths”
Metric Threshold Analysis
Advisory technique
Finds complex/hard-‐to-‐test code prone to errors
Paraso& Proprietary and Confiden1al 4 Paraso& Proprietary and Confiden1al 4
Code Analysis
§ Well understood o&en under valued § Define the goal of the analysis and the Policy for compliance § Policy defines reduc1on of business risk not pursuit of
perfec1on § E.g. Security (CWE, CERT, …) , Safety Cri1cal (ISO26262, DO178B, …)
§ Start small to promote adop1on and monitor for areas of improvement
§ Apply con1nuously through the SDLC § Balance desktop interac1ve feedback (pre-‐check-‐in) with
server-‐side depth of analysis (post-‐check-‐in)
Paraso& Proprietary and Confiden1al 5 Paraso& Proprietary and Confiden1al 5
Peer Code Reviews
§ Highly valuable in finding REAL bugs; Algorithms/Design
§ Use carefully § Only apply a&er Sta1c Code Analysis § Only apply where there is RISK
Image: http://www.jasonawesome.com/2010/06/01/executing-a-php-code-review/
Paraso& Proprietary and Confiden1al 6 Paraso& Proprietary and Confiden1al 6
Unit Tes1ng
§ Unit Tes1ng § Code focused valida1on § Test components of the system in isola1on (stubs) § Code needs to be built to be testable § Host-‐based and on-‐target § Test before hardware available
§ Where is the ROI? § Did we design it properly § Does it func1on correctly? § Have we mi1gated the business risk?
§ How much is enough? § Code Coverage + Peer Review
Paraso& Proprietary and Confiden1al 7 Paraso& Proprietary and Confiden1al 7
Explora1ve Tes1ng
§ Ad-‐hock/Unstructured Tes1ng of func1onal areas
§ Important part of Agile QA/feedback process
§ Requires traceability to user-‐stories and code
§ Should be ‘reinforced’ with automated tests
Paraso& Proprietary and Confiden1al 8 Paraso& Proprietary and Confiden1al 8
Applica1on and Demonstra1on
§ Code Analysis from within IDE § PaDern-‐Based Analysis § Use Flow Analysis to find poten1al memory issues
§ Dynamic Analysis § Using Applica1on Monitoring to uncover run1me problems
§ Using Coverage to determine what was tested § Using Unit Tests to increase coverage § Combining Coverage from Manual and Unit Tes1ng
Paraso& Proprietary and Confiden1al 9 Paraso& Proprietary and Confiden1al 9
Host-‐based and On-‐Target Unit Tes1ng
Host Development Environment
Simulator or Target Device
Host Based Flow
Create/Extend Tests
Execute Tests (Op1onal)
Review results and coverage
Cross compile test executable
Execute Tests
Paraso& Proprietary and Confiden1al 10 Paraso& Proprietary and Confiden1al 10
Conclusions
1. Capture Business Expecta1ons in a Policy 2. Apply early and con1nuously 3. Use a workflow for remedia1on 4. Leverage both desktop + server based analysis 5. Translate to Business Impact and Monitor for
improvements
Paraso& Proprietary and Confiden1al 11
2/26/15
Thank you Mark Lambert, Rich Newman, Miroslaw Zielinski, Adam Trujillo
Mark.lambert@paraso&.com