csc 395 – software engineering lecture 9: testing -or- how i stopped worrying and learned to love...
Post on 21-Dec-2015
219 views
TRANSCRIPT
CSC 395 –Software Engineering
Lecture 9:
Testing -or-
How I Stopped Worrying and Learned to Love the Bug
Today’s Lecture
Discusses testing in early workflows Starts to answer important questions:
When and how should we do testing? Who should be responsible for performing tests? How do we know something is correct? What faith should we place in testing?
What Do We Test For
Obviously, need to test for correctness What is correct? Who decides?
Important to consider other issues, too Utility - is it easy to use, cost effective, & useful? Reliability - how often will it not work? Robustness - do we rely on specific environment? Performance - is it within space & time
constraints?
How Do We Do This Testing?
There are two basic types of testing Execution-based testing Non-execution-based testing
Testing performs two important actions Verification - Was workflow completed correctly? Validation - Does product satisfies requirements?
Terminology Warning: “Verify” also used for non-execution-based testing
Testing Team
Very useful to have independent testing team Experts on process of testing Insure developers & analysts deliver high-quality Be involved over entire life-cycle
Critical to have managerial independence Bugs, faults, and errors are part of development Testing team must be free to make decisions Arguments with your Boss are hard to win Final decisions should not be made by developer
or tester
Walkthroughs
Walkthrough team ideally has 4 - 6 people Member of team performing current workflow Member of team performing next workflow Member of SQA team Often includes a client Should NOT contain manager, partner, or boss
Walkthroughs must be thoroughly prepared Provide items not understood Provide items that appear to be incorrect
Walkthrough Meeting
Chaired by SQA member Only interest is in finding faults Practiced in running these meetings
Meeting detects any possible faults Should not last longer than 2 hours
“Committee is a group that keeps minutes and loses hours” -- Milton Berle
Some flagged items may be correct Cost of meeting is high, but good if faults found
“A camel is a horse designed by committee” -- Unknown
Running Walkthrough
Document-driven & not participant-driven Focus on determining if documents are correct Relies on knowledge brought by participants Eases transfer of responsibilities between teams
Faults often found describing documents Often discover problems by talking things through Many times bugs discovered
Inspections
Follows five formal steps1. Current workflow member provides overview
2. Participants prepare by developing understanding of document and look for faults
3. Inspection led by one participant through entire document. Written report completed with 1 day
4. Document goes through rework to fix faults
5. Moderator performs follow-up checking that all faults are fixed and documentation (including reports) are updated
Inspection Team
Team consists of 3 - 6 members, including: Moderator Member of team performing current workflow Member of team performing next workflow SQA team member
One person will serve as reader Leads team through the document
One person will serve as recorder Produces report of all possible faults team finds
Metrics Maintained
During inspection, detailed metrics kept Is each fault a minor fault or a major fault
Is the fault from not meeting the specification or that actual & formal arguments do not match?
Where in the project did the fault arise? How many faults were found per document/LOC? How long did it take to find a fault?
Using Metrics
Can compare between products Are we getting better at a workflow? Are the value correlated with something
Take actions if artifact is overly buggy Suggests design may need rethinking
Consider fault statistics from next workflow Some faults may not be detected until later Important to perform quality check on quality team
Testing Results
JPL found savings of $25,000 per inspection Fault counts decreased exponentially by phase
Important to maintain testing independence Benefits tied to faith in testing team Testing metrics should not be used as
performance evaluation Need to consider metrics scientifically
One or two results is not a sufficient sample Statistics can always be bent to tell a story
Inspections vs. Walkthroughs
Walkthrough Two-step, informal process
Inspection Five-step, formal process
Both good for finding faults early in process But rely on having reliable, complete process Requires thorough documentation be available Lastly, needs all teams to buy-in to approach
Correctness Proofs
Halfway between execution-based & non-execution-based testing Examines if algorithm/code matches specification But does so by developing mathematical proof
Relies on creating & understanding proof Also assumes specification is correct
Correctness Proof
Add I/O specification Loop invariants Assertions
Developingthese can beas much art asscience
Three Myths
Software Engineers stink at math Relies on basic algebra; even I passed that class
Costs way, WAY, WAY too much Might be worth it in some situation; use sparingly
Proofs are too darn hard Nontrivial products have been successfully proven Theorem provers can help automate process
Are specification, compiler, & tools correct? Oops, this is real concern