introduction to software testing (paul degrandis)
DESCRIPTION
Introduction to Software Testing (Paul deGrandis). [Reading assignment: Chapter 15, pp. 231-252 and notes by Paul deGrandis]. Overview. Terms and Definitions Tool Overview Static Checkers Unit Testing Coverage and Profiling Memory Analysis Bug Reporting and Tracking - PowerPoint PPT PresentationTRANSCRIPT
Introduction to Software Testing
(Paul deGrandis)
[Reading assignment: Chapter 15, pp. 231-252 and notes by Paul deGrandis]
Overview
• Terms and Definitions
• Tool Overview
• Static Checkers
• Unit Testing
• Coverage and Profiling
• Memory Analysis
• Bug Reporting and Tracking
• Continuous Integration
NOTE
•All the material and examples from here on are generated using a single makefile.
•Log into tux, then:wget http://www.pauldee.org/vandvmakefile
mv vandvmakefile Makefile •Export your classpath:
export CLASSPATH=`pwd`/junit/junit-4.4.jar:`pwd`/emma/emma.jar:.
•To make and download all tools and examples:make
Tool Overview
•Verification• Static Checking
• Unit Tests for code correctness
• Memory and performance specifications
•Validation• Unit Tests for behavior and requirements
Static Checkers
•Does not execute the code
•Finds suspicious code or security vulnerabilities
•Tools
•C - SPLINT,GCC Warnings
•Java - FindBugs, PMD
Cyclone - a safe C
GNU Compiler Collection
FindBugs - Find Bugs in Java
Static Checkers - CScreenshots
Static Checkers - CDemo
• Installation of SPLINT
•Compiling using GCC Warning
•Running Splint
Static Checkers - CDemo
• Installation of SPLINT• Download
wget http://www.splint.org/downloads/splint-3.1.2.src.tgz
• Untartar -xzvf splint-3.1.2.src.tgz
• Make (during make, splint checks itself)cd splint-3.1.2/./configuremake
Static Checkers - CDemo
•Compiling using GCC Warnings
• see the screenshot three slides back
Static Checkers - CDemo
•Running SPLINT
• Simple and Basicsplint -preproc +weak +show-summary +stats ../src/*.c
• Dependable Softwaresplint -preproc +checks +show-summary +stats ../src/*.c
Static Checkers - JavaScreenshots
Static Checkers - JavaDemo
• Installation of FindBugs
•Running FindBugs
Static Checkers - JavaDemo
• Installation of FindBugs• Download
wget http://prdownloads.sourceforge.net/findbugs/findbugs-1.2.1.tar.gz
• Untartar -xvzf findbugs-1.2.1.tar.gz cd findbugs-1.2.1
Static Checkers - JavaDemo
•Running FindBugs
• Run It./findbugs
Unit Tests
• Validate a unit of code; smallest testable part
• Executes the code in a sandboxed environment
• Testing mostly for functional requirements• can also test some non-functional requirements
• Many approaches and schools of thought• History Based, Risk Based, Data Path, DOE
Unit Tests Frameworks-CScreenshots
Check Unit Test Framework
http://check.sourceforge.net
Demo
Unit Tests Frameworks-C
• Installation of Check
•Writing a Check unit test
•Compiling your test(s)
•Running your test(s)
http://check.sourceforge.net/doc/check.html/index.html
Demo
Unit Tests Frameworks-C
• Installation of Check• Download
wget http://downloads.sourceforge.net/check/check-0.9.5.tar.gz
• Untartar -xvzf check-0.9.5.tar.gz
• Make (notice the GCC warnings passed)cd check-0.9.5./configuremake
Demo
Unit Tests Frameworks-C
•Writing a Check unit test
• Write your test fixtures
• Write your test(s)
• Write your test suite
• Write your main method (execute your suite)
Demo
Unit Tests Frameworks-C
•Writing a Check unit test
• Write your test fixtures
Demo
Unit Tests Frameworks-C
•Writing a Check unit test
• Write your test(s)
Demo
Unit Tests Frameworks-C
•Writing a Check unit test
• Write your test suite
Demo
Unit Tests Frameworks-C
•Writing a Check unit test
• Write your main method (execute your suite)
Demo
Unit Tests Frameworks-C
•Compiling your test(s)
• See README in check/check-example/square
Demo
Unit Tests Frameworks-C
•Running your test(s)
• See README in check/check-example/square
Unit Tests Frameworks-JavaScreenshots
Demo
Unit Tests Frameworks-Java
• Installation of Junit
•Writing a Junit unit test
•Compiling your test(s)• Just use javac
•Running your test(s)• just use java
http://junit.sourceforge.net/ (look at the cookbook)
Demo
Unit Tests Frameworks-Java
• Installation of Junit• Download
wget http://downloads.sourceforge.net/junit/junit-4.4.jar
• Add the jar to your classpathexport CLASSPATH=$CLASSPATH:`pwd`/junit-4.4.jar:.
Demo
Unit Tests Frameworks-Java
•Writing a Junit unit test
•Write your test fixtures
•Write your test(s)
•Write your test suite
•Write your main method (execute your suite)
Demo
Unit Tests Frameworks-Java
•Writing a Junit unit test
• Write your test fixtures
Demo
Unit Tests Frameworks-Java
•Writing a Junit unit test
• Write your test(s)
Demo
Unit Tests Frameworks-Java
•Writing a Junit unit test
• Write your test suite(s)
Demo
Unit Tests Frameworks-Java
•Writing a Junit unit test
• Write your main method
Code Coverage
•Degree to how much code was tested (E.g., How confident are we in our tests and code)
•Criteria:• Statement - Has each line been executed
• Condition - Every evaluation (if/else, try/catch, switch, loop)
• Path - Every route of code
Code Coverage - CScreenshots
GCC - GCOV
DemoCode Coverage - C
See the Screenshot one slide back
Code Coverage - JavaScreenshots
EMMAhttp://emma.sourceforge.net/intro.html
DemoCode Coverage - Java
• Installation of Emma
• Downloadwget http://downloads.sourceforge.net/emma/emma-2.0.5312-lib.zip
• Unzipunzip emma-2.0.5312-lib.zip
• Instrument Your Codejava -cp emma.jar:./sample:../junit/junit-4.4.jar emma instr -outdir ./sampleout -cp ./sample/ SimpleTest
• Run Your Code and Generate a reportjava SimpleTestjava -cp emma.jar emma report -r html -in sampleout/coverage.em,sampleout/coverage.ec
Memory Analysis
•Sandboxes an application in a VM
•Uses:• Memory Leak - allocate memory, but don’t free
• Buffer Overflow - access memory outside of a fixed buffer
• Profiling - performance analysis
Memory Analysis - CScreenshots
Valgrind
DemoMemory Analysis - C
• Installation of Valgrind• Download
wget http://downloads.sourceforge.net/check/check-0.9.5.tar.gz
• Untartar -xvjf valgrind-3.2.3.tar.bz2
• Make (notice the GCC warnings passed)cd valgrind-3.2.3./configuremake
DemoMemory Analysis - C
•Running of Valgrind• Compile with debugging enabled
gcc -g myprog.c
• Run Valgrindvalgrind --leak-check=yes myprog arg1 arg2
• Full tutorial and infoformationhttp://valgrind.org/docs/manual/quick-start.html#quick-start.intro
Memory Analysis - JavaScreenshots
JConsole
Issue and Bug Tracking
•Organizes and tracks issues, enhancements, features, and bugs in a software system
•Usually contains a database backend and a web frontend
•Generate report of project and team metrics and productivity
Issue and Bug Tracking
•Trac (http://trac.edgewall.org/)
•Roundup (http://roundup.sourceforge.net/)
•Tracker (http://tracker.rubyforge.org/)
Continuous Integration
•Committing changes early and often
•Building and testing at least nightly
• If code breaks, revert
• Integration issues discovered early as developers work with the latest code
You now know …
• … static checkers• … unit testing tools• … coverage and profiling tools • … memory analysis tools• … Bug reporting and tracking tools