tao xie automated software engineering group department of computer science north carolina state...
TRANSCRIPT
Trustworthy Software Everywhere
Tao Xie
Automated Software Engineering GroupDepartment of Computer ScienceNorth Carolina State University
https://sites.google.com/site/asergrp/
Trustworthy Software Everywhere
Static Verification Problem: API properties are not available▪ E.g., fopen’s return needs to be NULL-CHECK
Solution: mining API properties from client code
Dynamic Verification (a.k.a. Software Testing)
3
Searching + Mining
Supported by NSF CSR and ARO
4
Development Assisted by Searching+Mining
PARSEWeb [ASE 07]
PARSEWeb Source object type &
Destination object type
Method-invocation sequence
MAPO [ECOOP 09] API method Frequent subsequences of API methods SpotWeb [ASE 08] Framework hotspots/coldspots
5
Verification Assisted by Searching+Mining
Detect deviant behavior as bugs in programs
• Neglected-condition bugs [ASE 09]• Exception-handling bugs [ICSE 09]• Error-handling bugs [FASE 09]• API-sequencing bugs [ESEC/FSE 07]
6
Mining Text in Software Engineering• Detect duplicate bug reports [ICSE 08]• Identify security bug reports [MSR 10]• Mine resource specifications from Javadoc
[ASE 09, Best Paper Award, SIGSOFT Distinguished Paper]
javax.resource.cci.ConnectioncreateInteraction():“Creates an interaction associated with this connection.”getMetaData():“Gets the information on the underlying EIS instance represented through an active connection.”close():“Initiates close of the connection handle at the application level.”
Supported by IBM Jazz Award
Software Testing Problems
=?
Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Generating high-quality test inputs (e.g.,
achieving high code coverage)
Test Oracles Specifying high-quality test oracles (e.g.,
guarding against various faults)
Test Generation
Human Expensive, incomplete, …
Brute Force Pairwise, predefined data, etc…
Random: Cheap, Fast “It passed a thousand tests” feeling
Dynamic Symbolic Execution: Pex, CUTE,EXE Automated white-box Not random – Constraint Solving
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Data
null
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition
Loops Fitnex [DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [ASE 09-sp]
Method sequences MSeqGen [ESEC/FSE 09]
Environments e.g., file systems, network, db, … Parameterized Mock Objects [AST 09, ASE 10-sp]
Opportunities Regression testing [ICSE 09-nier] Developer guidance (cooperative developer testing)
Challenges of DSE
Supported by NSF SoD, NSF SHF, NSF CAREER, Microsoft Research Award
Loops Fitnex [DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [ASE 09-sp]
Method sequences MSeqGen [ESEC/FSE 09]
Environments e.g., file systems, network, db, … Parameterized Mock Objects [AST 09, ASE 10-sp]
Applications Test network app @Army division, Fort Hood, Texas Test DB app of hand-held medical assistant device@FDA Test .NET base libraries @Microsoft
NCSU Industry Tech Transfer
Pex Visual Studio Power Tool
Download counts (20 months)(Feb. 2008 - Oct. 2009 )
Academic: 17,366 Devlabs: 13,022 Total: 30,388
Pex for Fun (Coding Duels)http://www.pexforfun.com/
Various countries/regions Software internationalization▪ Locating constant strings to translate [ICSE 09,
FSE 10]▪ E.g., translating Megamek (a realtime strategy
game)
Various programming languages PL translation▪ E.g., translating Java to C# [ICSE 10]
Trustworthy Software Everywhere
Various types of software Database applications [ASE 10-sp] Network/file-system applications [AST 09] Game applications [ICSE 09] Cyber-physical systems (power grid,
medical device software, …) Could applications Social network applications …
Trustworthy Software Everywhere
Supported by NSF SHF, NSF CAREER
Various types of quality attributes Functional correctness Security (NIST/Fermi Lab collaboration)▪ Testing/verification of access control policies [WWW
07, ACSAC 08, SIGMETRICS 08, IEEE TC 10]▪ Testing/verification of firewall policies [SRDS 08/09,
LISA 10 Best Student Paper]▪ Identification of security bug reports [MSR 10]▪ Attack generation
Performance▪ IBM RTP collaboration on Rational Performance Tester
…
Trustworthy Software Everywhere
Supported by NSF CyberTrust, NIST, IBM Faculty Awards
Trustworthy Software Everywhere
http://people.engr.ncsu.edu/txie/https://sites.google.com/site/asergrp/