metricon 2.0 correlating automated static analysis alert density to reported vulnerabilities in...
TRANSCRIPT
MetriCon 2.0
Correlating Automated Static Analysis Alert Density to Reported Vulnerabilities in
Sendmail
Michael Gegick, Laurie Williams
North Carolina State University
7 August 2007
2
Introducing Security Parallels
Fault-prone component
Likely to contain faults
Failure-prone component
Likely to have failures in field
Component – any logical part of the software system [1]
Make informed risk management decisions and prioritize redesign, inspection, and testing efforts on components.
Reliability context(well-established)
Security context (new)
Vulnerability-prone component
Likely to contain vulnerabilities
Attack-prone component
Likely to be exploited in the field
[1] IEEE, "ANSI/IEEE Standard Glossary of Software Engineering Terminology (IEEE Std 610.12-1990)," Los Alamitos, CA: IEEE Computer Society Press, 1990.
3
Early Reliability Metrics• Static analysis
– N. Nagappan and T. Ball, "Static Analysis Tools as Early Indicators of Pre-release Defect Density," in International Conference on Software Engineering, St. Louis, MO, 2005, pp. 580-586.
– J. Zheng, L. Williams, W. Snipes, N. Nagappan, J. Hudepohl, and M. Vouk, "On the Value of Static Analysis Tools for Fault Detection," IEEE Transactions on Software Engineering, vol. 32, pp. 240-253, 2006.
• Complexity metrics– J. Munson and T. Khoshgoftaar, "The Detection of Fault-Prone Programs," IEEE
Transactions on Software Engineering, vol. 18, pp. 423-433, 1992.– T. Khoshgoftaar and J. Munson, "Predicting Software Development Errors using Software
Complexity Metrics," IEEE Journal on Selected Areas in Communications, vol. 8, pp. 253-261, 1990.
• Historical (failure)– N. Nagappan, T. Ball, and A. Zeller, "Mining metrics to predict component failures," in
International Conference on Software Engineering, Shanghai, China, 2006.– T. J. Ostrand, E. J. Weyuker, and R. M. Bell, "Where the bugs are," in International
Symposium on Software Testing and Analysis, Boston, Massachusetts, 2004, pp. 86-96• Object-Oriented metrics
– V. Basili, L. Briand, and W. Melo, "A Validation of Object Oriented Design Metrics as Quality Indicators," IEEE Transactions on Software Engineering, vol. 21, 1996.
– Y. Zhou and L. Hareton, "Empirical Analysis of Object-Oriented Design Metrics for Predicting High and Low Severity Faults," IEEE Transactions on Software Engineering, vol. 32, no. 10, 2006, pp. 771-789.
4
Research Objective
Build and validate models for predicting vulnerability-
and attack-prone components based upon security-based automated static analyzer (ASA) alerts
– Metric: ASA alert density and severity – early in the development phase
– ASA cannot find all types of security vulnerabilities• Are ASA alerts a good predictor?
– Implementation bugs, design flaws, operational vulnerabilities
– Software engineers plug the number of security alerts into the predictive models to determine which components are vulnerability- and attack-prone.
5
Building the Initial Predictive Model
Generalized linear model (data are not normally distributed)Poisson distribution?
mean number vulnerabilities in component
estimated intercept
estimated slope
value of random variable – alert density of component1x
y
0
1
0 1 1ˆ ˆˆlog( )y x
6
Feasibility Study
• Fortify Software’s Source Code Analyzer (SCA)
• Scanned ten releases of Sendmail– 8.12.2-8.12.11– 996 total files scanned
• 21 potential vulnerabilities– Vulnerabilities reported in RELEASE_NOTES
• Nine vulnerabilities with known exploits
7
Feasibility Study – vulnerability-prone
• Poisson distribution – Models the response data
• Reported vulnerability
• Association between Hot alert density and number of vulnerabilities per reported per file– Positive slope positive association between alerts and reported
vulnerabilities– p-value high significance in association
• Standard error substantial overdispersion – Few data points
Slope p-value Chi-Square/df
Goodness-of-
fit measure
Standard error
294.8069 0.0016 1.1939 93.3422
0 1 1ˆ ˆˆlog( )y x
8
Feasibility Study – attack-prone• Poisson distribution
– Models the response data• Number of known exploits (nine) for a Sendmail file
• Association between Hot alert density and number of known exploits– Slope positive association between alerts and exploits
• p-value low significance– Standard error substantial overdispersion
» Few data points
Slope p-value Chi-Square/df
Goodness-of-
fit measure
Standard error
140.4334 0.4980 1.2099 207.2419
0 1 1ˆ ˆˆlog( )y x
