teaching about software testing that is not for defect detection w. morven gentleman dalhousie...
TRANSCRIPT
![Page 1: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/1.jpg)
Teaching about Software Testing that is NOT for Defect Detection
W. Morven Gentleman
Dalhousie University
![Page 2: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/2.jpg)
Conventional Perspective
• Software is tested on behalf of developer
• Purpose of testing is to find (and fix) bugs
This is an important perspective, but not the only one
![Page 3: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/3.jpg)
Other perspectives:how unusual are they?
• Testing to meet regulatory or contractual requirements
• Testing a software artifact to measure and model quantitative attributes of the software
• Testing to measure human behaviour when interacting with the software under test
• Testing where the investigation is open-ended, to find questions to ask
![Page 4: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/4.jpg)
Measuring quantitative attributes
• Benchmarking
• Sizing and capacity planning
• Tuning performance to observed load
![Page 5: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/5.jpg)
Measuring human behaviour
• Usability testing
• Assessing training requirements
• Assessing change management activities
• Studying tool adoption
![Page 6: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/6.jpg)
Open-ended investigation
• Validating a system model or specification
• Studying competitors’ products
• Interoperability testing
• Identifying ways to use a product
![Page 7: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/7.jpg)
Observations on teaching
• Students have great difficulty assimilating experimental techniques and experimental evidence. They cannot– conceive of interesting questions.– propose experiments.to answer such questions– suggest analyses of experimental data.– interpret experimental results.– recognize assumptions contradicted empirically– notice anomalies that indicate the unsuspected .– manage an investment of experimental effort.
![Page 8: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/8.jpg)
Diagnosis
• CS/SE/IS students get no exposure to the culture of empirical science– the fundamental precept that all asserted truth must
ultimately be founded on experimental observation– theory should be seen as an approximation
summarizing experimental evidence
• They are given a mathematical model of a problem, and they are unable to distinguish that model from physical reality
• Remedy?
![Page 9: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/9.jpg)
Some topics students find novel
• Practical instrumentation– Choosing what to measure and what scale to use
– Characterizing load
– Systematic sources of measurement artifact
• Practical application of statistics– Statistical vs.. practical significance
– Exploratory data analysis
– Modeling
– Experimental design
![Page 10: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/10.jpg)
Real Industrial Example
![Page 11: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/11.jpg)
Time to update in-memory Oracle Database – 1 try
• Specification requires < 0.5 seconds
• Vendor demonstration 0.472 seconds
![Page 12: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/12.jpg)
Time to update in-memory Oracle Database – 100 tries
0.00E+00
1.00E-01
2.00E-01
3.00E-01
4.00E-01
5.00E-01
6.00E-01
0 20 40 60 80 100 120
Series1
![Page 13: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/13.jpg)
Time to update in-memory Oracle Database – 5000 tries
0.00E+00
1.00E-01
2.00E-01
3.00E-01
4.00E-01
5.00E-01
6.00E-01
7.00E-01
0 1000 2000 3000 4000 5000 6000
Series1
![Page 14: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/14.jpg)
Another real example
![Page 15: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/15.jpg)
Time to open an Excel file
![Page 16: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/16.jpg)
View of all software testing as searching for defects
• Does it help in designing experiments and analysis?
• Are defect detection strategies appropriate in other situations?
![Page 17: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/17.jpg)
Definition of defect
• “Operationally, it is useful to work with 2 definitions of a defect:– From the producers viewpoint: a product
requirement that has not been met or an attribute of a product that is not in the statement of requirements that define the product.
– From the customers viewpoint: anything that causes customer dissatisfaction, whether in the statement of requirements or not”
![Page 18: Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca](https://reader035.vdocuments.us/reader035/viewer/2022062722/56649f2b5503460f94c45dff/html5/thumbnails/18.jpg)
Conclusion
A broader perspective on software testing can be applicable to CS/SE/IS students in careers with soft6ware developers and software customers