![Page 1: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/1.jpg)
/GU
Testing, Debugging, and VerificationTesting, Part I
Srinivas Pinisetty 1
08 November 2018
1Slides based on material from Wolfgang Aherndt,...
![Page 2: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/2.jpg)
/GU
Let’s review the course contents
I Testing and Debugging (lab)I Formal Specification (lab)I Formal Program Verification (lab)I Property Based testingI Loop Invariant Generation
![Page 3: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/3.jpg)
/GU
Some practical things
I Make sure you are registered for the course (see StudentOffice). Otherwise your marks cannot be recorded.I Even if you are repeating the course, only taking labs or exam.I If in doubt, contact the student office to double check.
I Please sign up for the google group (follow News link)I Exam date: 15 January 2019.I About labs in general
I Use Fire systemI Working in pairs is mandatoryI If there are any problems, notify us immediately (don’t wait
until the day of the deadline...).
![Page 4: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/4.jpg)
/GU
Some practical things
Testing block consists ofI 4 lectures (2 lectures this week + 2 lecture next week)I 1 exercise (Thursday next week)I 1 lab assignment
![Page 5: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/5.jpg)
/GU
Motivation for Course Unit on Testing
Ideas and techniques of testing have become essential knowledgefor all software developers.
Expect to use the concepts presented here many times in yourcareer.
Testing is not the only, but the primary method that industry usesto evaluate software under development.
![Page 6: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/6.jpg)
/GU
Motivation for Course Unit on Testing
Ideas and techniques of testing have become essential knowledgefor all software developers.
Expect to use the concepts presented here many times in yourcareer.
Testing is not the only, but the primary method that industry usesto evaluate software under development.
![Page 7: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/7.jpg)
/GU
Motivation for Course Unit on Testing
Ideas and techniques of testing have become essential knowledgefor all software developers.
Expect to use the concepts presented here many times in yourcareer.
Testing is not the only, but the primary method that industry usesto evaluate software under development.
![Page 8: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/8.jpg)
/GU
Motivation for Course Unit on Testing (cont’d)
I The field of testing is largeI This course (unit) is rather small
I Does it make sense to get started even?
Cost of SW faults gets higher and higher. If the course will help todetect a few more faults in real applications, it was a goodinvestment.
A few basic software testing concepts can be used to design testsfor a large variety of software applications.
The testing techniques present in the literature have much more incommon than is obvious at first glance.
![Page 9: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/9.jpg)
/GU
Motivation for Course Unit on Testing (cont’d)
I The field of testing is largeI This course (unit) is rather small
I Does it make sense to get started even?
Cost of SW faults gets higher and higher. If the course will help todetect a few more faults in real applications, it was a goodinvestment.
A few basic software testing concepts can be used to design testsfor a large variety of software applications.
The testing techniques present in the literature have much more incommon than is obvious at first glance.
![Page 10: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/10.jpg)
/GU
Motivation for Course Unit on Testing (cont’d)
I The field of testing is largeI This course (unit) is rather small
I Does it make sense to get started even?
Cost of SW faults gets higher and higher. If the course will help todetect a few more faults in real applications, it was a goodinvestment.
A few basic software testing concepts can be used to design testsfor a large variety of software applications.
The testing techniques present in the literature have much more incommon than is obvious at first glance.
![Page 11: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/11.jpg)
/GU
Motivation for Course Unit on Testing (cont’d)
I The field of testing is largeI This course (unit) is rather small
I Does it make sense to get started even?
Cost of SW faults gets higher and higher. If the course will help todetect a few more faults in real applications, it was a goodinvestment.
A few basic software testing concepts can be used to design testsfor a large variety of software applications.
The testing techniques present in the literature have much more incommon than is obvious at first glance.
![Page 12: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/12.jpg)
/GU
Motivation for Course Unit on Testing (cont’d)
![Page 13: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/13.jpg)
/GU
A Quiz
A simple program
InputRead three integer values from the command line.The three values represent the lengths of the sides of a triangle.
OutputTells whether the triangle is
Scalene: no two sides are equalIsosceles: exactly two sides are equal
Equilateral: all sides are equal
Create a Set of at least 15 Test Cases for this Program
![Page 14: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/14.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 1: An invalid triangle? e.g. (4,1,2)
4
12
Why not a valid triangle? (a,b,c) with a > b + c
![Page 15: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/15.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 1: An invalid triangle? e.g. (4,1,2)
4
12
Why not a valid triangle?
(a,b,c) with a > b + c
![Page 16: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/16.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 1: An invalid triangle? e.g. (4,1,2)
4
12
Why not a valid triangle? (a,b,c) with a > b + c
![Page 17: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/17.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 2: Some permutations of previous? e.g., (1,2,4), (2,1,4)are still invalid.
![Page 18: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/18.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 3: An invalid triangle with equal sum? e.g., (4,2,2)
4
2 2
![Page 19: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/19.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 4: Some permutations of previous? e.g., (2,2,4), (2,4,2)
![Page 20: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/20.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 5: A valid scalene triangle? e.g., (3,4,5)
4
53
![Page 21: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/21.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 6: An equilateral triangle? e.g., (3,3,3)
3
3
3
![Page 22: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/22.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 7: A valid isosceles triangle? e.g., (3,4,3)
3 3
4
![Page 23: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/23.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 8: All permutations of valid isosceles triangle?
(3,4,3), (3,3,4), (4,3,3)
![Page 24: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/24.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 9: One side with zero value? e.g., (0,4,3)
![Page 25: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/25.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 10: One side with negative value? e.g., (-1,4,3)
![Page 26: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/26.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 11: All sides zero? e.g., (0,0,0)
![Page 27: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/27.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 12: At least one value is non-integer? e.g., (1,3,2.5)
![Page 28: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/28.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 13: wrong number of arguments, e.g., (2,4) or (1,2,3,3)
![Page 29: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/29.jpg)
/GU
Solution — 1 Point for each Correct Answer
Q 14 (the most important one):
Did you specify the expected output in each case?
![Page 30: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/30.jpg)
/GU
About the Quiz
I Q 1–13 correspond to failures that have actually occurred inimplementations of the program
I How many questions did you answer?< 5? 5− 7? 8− 10? > 10? All?
I Highly qualified, experienced programmers score 7.8 onaverage
![Page 31: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/31.jpg)
/GU
About the Quiz
I Q 1–13 correspond to failures that have actually occurred inimplementations of the program
I How many questions did you answer?< 5? 5− 7? 8− 10? > 10? All?
I Highly qualified, experienced programmers score 7.8 onaverage
![Page 32: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/32.jpg)
/GU
First Conclusions
I Finding good and sufficiently many test cases is difficult
I Even a good set of test cases cannot exclude all failuresI Without a specification, it is not clear even what a failure is
The discipline of Testing is all about Test Cases
well, almost ...
Remark: At Ericsson, ca. 35% of code is test cases!
![Page 33: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/33.jpg)
/GU
First Conclusions
I Finding good and sufficiently many test cases is difficultI Even a good set of test cases cannot exclude all failures
I Without a specification, it is not clear even what a failure is
The discipline of Testing is all about Test Cases
well, almost ...
Remark: At Ericsson, ca. 35% of code is test cases!
![Page 34: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/34.jpg)
/GU
First Conclusions
I Finding good and sufficiently many test cases is difficultI Even a good set of test cases cannot exclude all failuresI Without a specification, it is not clear even what a failure is
The discipline of Testing is all about Test Cases
well, almost ...
Remark: At Ericsson, ca. 35% of code is test cases!
![Page 35: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/35.jpg)
/GU
First Conclusions
I Finding good and sufficiently many test cases is difficultI Even a good set of test cases cannot exclude all failuresI Without a specification, it is not clear even what a failure is
The discipline of Testing is all about Test Cases
well, almost ...
Remark: At Ericsson, ca. 35% of code is test cases!
![Page 36: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/36.jpg)
/GU
First Conclusions
I Finding good and sufficiently many test cases is difficultI Even a good set of test cases cannot exclude all failuresI Without a specification, it is not clear even what a failure is
The discipline of Testing is all about Test Cases
well, almost ...
Remark: At Ericsson, ca. 35% of code is test cases!
![Page 37: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/37.jpg)
/GU
Brainstorming
I What is the purpose of testing?
...
![Page 38: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/38.jpg)
/GU
Test Process Maturity Level in an Organisation
(adapted from [Beizer] and [AmmannOffutt])
Level 0 There is no difference between testing and debugging.
Level 1 Purpose of testing: show correctness.Level 2 Purpose of testing: show that the software does not
work.Level 3 Purpose of testing: reduce the risk of using the
software.Level 4 Testing is a mental discipline helping IT professionals
to develop higher quality software.
![Page 39: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/39.jpg)
/GU
Test Process Maturity Level in an Organisation
(adapted from [Beizer] and [AmmannOffutt])
Level 0 There is no difference between testing and debugging.Level 1 Purpose of testing: show correctness.
Level 2 Purpose of testing: show that the software does notwork.
Level 3 Purpose of testing: reduce the risk of using thesoftware.
Level 4 Testing is a mental discipline helping IT professionalsto develop higher quality software.
![Page 40: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/40.jpg)
/GU
Test Process Maturity Level in an Organisation
(adapted from [Beizer] and [AmmannOffutt])
Level 0 There is no difference between testing and debugging.Level 1 Purpose of testing: show correctness.Level 2 Purpose of testing: show that the software does not
work.
Level 3 Purpose of testing: reduce the risk of using thesoftware.
Level 4 Testing is a mental discipline helping IT professionalsto develop higher quality software.
![Page 41: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/41.jpg)
/GU
Test Process Maturity Level in an Organisation
(adapted from [Beizer] and [AmmannOffutt])
Level 0 There is no difference between testing and debugging.Level 1 Purpose of testing: show correctness.Level 2 Purpose of testing: show that the software does not
work.Level 3 Purpose of testing: reduce the risk of using the
software.
Level 4 Testing is a mental discipline helping IT professionalsto develop higher quality software.
![Page 42: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/42.jpg)
/GU
Test Process Maturity Level in an Organisation
(adapted from [Beizer] and [AmmannOffutt])
Level 0 There is no difference between testing and debugging.Level 1 Purpose of testing: show correctness.Level 2 Purpose of testing: show that the software does not
work.Level 3 Purpose of testing: reduce the risk of using the
software.Level 4 Testing is a mental discipline helping IT professionals
to develop higher quality software.
![Page 43: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/43.jpg)
/GU
Level 0 Thinking
Testing is the same as debuggingI Does not distinguish between incorrect behaviour and defects
in the programI Does not help develop software that is reliable or safe
![Page 44: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/44.jpg)
/GU
Level 1 Thinking
Purpose: showing correctnessI Correctness is (almost) impossible to achieveI Danger: you are subconsciously steered towards tests likely to
not fail the program.I What do we know if no failures?
good software? or bad tests?I Test engineers have:
I no strict goalI no real stopping ruleI no formal test technique
![Page 45: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/45.jpg)
/GU
Level 1 Thinking
Purpose: showing correctnessI Correctness is (almost) impossible to achieveI Danger: you are subconsciously steered towards tests likely to
not fail the program.I What do we know if no failures?
good software?
or bad tests?I Test engineers have:
I no strict goalI no real stopping ruleI no formal test technique
![Page 46: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/46.jpg)
/GU
Level 1 Thinking
Purpose: showing correctnessI Correctness is (almost) impossible to achieveI Danger: you are subconsciously steered towards tests likely to
not fail the program.I What do we know if no failures?
good software? or bad tests?
I Test engineers have:I no strict goalI no real stopping ruleI no formal test technique
![Page 47: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/47.jpg)
/GU
Level 1 Thinking
Purpose: showing correctnessI Correctness is (almost) impossible to achieveI Danger: you are subconsciously steered towards tests likely to
not fail the program.I What do we know if no failures?
good software? or bad tests?I Test engineers have:
I no strict goalI no real stopping ruleI no formal test technique
![Page 48: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/48.jpg)
/GU
Level 2 Thinking
Purpose: showing failuresI Looking for failures is a negative activityI Puts testers and developers into an adversarial relationshipI What if there are no failures?
This describes most software companies.
![Page 49: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/49.jpg)
/GU
Level 2 Thinking
Purpose: showing failuresI Looking for failures is a negative activityI Puts testers and developers into an adversarial relationshipI What if there are no failures?
This describes most software companies.
![Page 50: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/50.jpg)
/GU
Level 3 Thinking
Purpose: reduce riskI Whenever we use software, we incur some riskI Risk may be small and consequences unimportantI Risk may be great and the consequences catastrophicI Testers and developers work together to reduce risk
This describes a few “enlightened” software companies.
![Page 51: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/51.jpg)
/GU
Level 3 Thinking
Purpose: reduce riskI Whenever we use software, we incur some riskI Risk may be small and consequences unimportantI Risk may be great and the consequences catastrophicI Testers and developers work together to reduce risk
This describes a few “enlightened” software companies.
![Page 52: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/52.jpg)
/GU
Level 4 Thinking
A mental discipline that increases qualityI Testing only one way to increase qualityI Test engineers can become technical leaders of the project
I Primary responsibility to measure and improve software qualityI Their expertise should help developersI Purpose of testing: improve ability of developers to produce
high quality software
![Page 53: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/53.jpg)
/GU
Level 4 Thinking
A mental discipline that increases qualityI Testing only one way to increase qualityI Test engineers can become technical leaders of the projectI Primary responsibility to measure and improve software qualityI Their expertise should help developers
I Purpose of testing: improve ability of developers to producehigh quality software
![Page 54: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/54.jpg)
/GU
Level 4 Thinking
A mental discipline that increases qualityI Testing only one way to increase qualityI Test engineers can become technical leaders of the projectI Primary responsibility to measure and improve software qualityI Their expertise should help developersI Purpose of testing: improve ability of developers to produce
high quality software
![Page 55: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/55.jpg)
/GU
Activities if Test Engineer
Test engineer: IT professional in charge of test activities, including:I designing test inputsI running testsI analysing resultsI reporting results to developers and managers
I automating any of the above
![Page 56: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/56.jpg)
/GU
Activities if Test Engineer
Test engineer: IT professional in charge of test activities, including:I designing test inputsI running testsI analysing resultsI reporting results to developers and managersI automating any of the above
![Page 57: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/57.jpg)
/GU
Developer = or 6= Test Engineer?
Should testing be done by the developers (of the same software)?
different takes on this:
Contra: Test Principles in [Myers]Principle: Programmer should avoid testing his/her own
program.(misunderstanding of specs carry over to testing)
Principle: A programming organisation should not test its ownprograms.
Pro: Extreme Testing (ET) [BeckGamma]Principle: Developers create tests firstPrinciple: Developers re-run test on all incremental changes
Tool: JUnit designed for ET
![Page 58: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/58.jpg)
/GU
Developer = or 6= Test Engineer?
Should testing be done by the developers (of the same software)?
different takes on this:
Contra: Test Principles in [Myers]Principle: Programmer should avoid testing his/her own
program.(misunderstanding of specs carry over to testing)
Principle: A programming organisation should not test its ownprograms.
Pro: Extreme Testing (ET) [BeckGamma]Principle: Developers create tests firstPrinciple: Developers re-run test on all incremental changes
Tool: JUnit designed for ET
![Page 59: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/59.jpg)
/GU
Developer = or 6= Test Engineer?
Should testing be done by the developers (of the same software)?
different takes on this:
Contra: Test Principles in [Myers]Principle: Programmer should avoid testing his/her own
program.(misunderstanding of specs carry over to testing)
Principle: A programming organisation should not test its ownprograms.
Pro: Extreme Testing (ET) [BeckGamma]Principle: Developers create tests firstPrinciple: Developers re-run test on all incremental changes
Tool: JUnit designed for ET
![Page 60: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/60.jpg)
/GU
Testing Levels Based on Software Activity
Acceptance Testingassess software with respect to user requirements
System Testingassess software with respect to system-levelspecification
Integration Testingassess software with respect to high-level design
Unit Testingassess software with respect to low-level unit design
remarks:– terminology, and depth of this hierarchy, varies in literature
![Page 61: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/61.jpg)
/GU
Testing Levels Based on Software Activity
Acceptance Testingassess software with respect to user requirements
System Testingassess software with respect to system-levelspecification
Integration Testingassess software with respect to high-level design
Unit Testingassess software with respect to low-level unit design
remarks:– terminology, and depth of this hierarchy, varies in literature
![Page 62: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/62.jpg)
/GU
Testing Levels Based on Software Activity
Acceptance Testingassess software with respect to user requirements
System Testingassess software with respect to system-levelspecification
Integration Testingassess software with respect to high-level design
Unit Testingassess software with respect to low-level unit design
remarks:– terminology, and depth of this hierarchy, varies in literature
![Page 63: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/63.jpg)
/GU
Testing Levels Based on Software Activity
Acceptance Testingassess software with respect to user requirements
System Testingassess software with respect to system-levelspecification
Integration Testingassess software with respect to high-level design
Unit Testingassess software with respect to low-level unit design
remarks:– terminology, and depth of this hierarchy, varies in literature
![Page 64: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/64.jpg)
/GU
Testing Levels Based on Software Activity
Acceptance Testingassess software with respect to user requirements
System Testingassess software with respect to system-levelspecification
Integration Testingassess software with respect to high-level design
Unit Testingassess software with respect to low-level unit design
remarks:– terminology, and depth of this hierarchy, varies in literature
![Page 65: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/65.jpg)
/GU
V-Model
Integration Testing
Unit TestingImplementation
Subsystem Design
Architectural Design
Acceptance Testing
System Testing
Requirements Analysis
Customer needs
Choose components,connections
Structure, behaviourof subsystem
Code!
User's/client's needs met?
Assembled system meets spec?
Does components work together?
Test individual methods, classes
(many variants!)
![Page 66: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/66.jpg)
/GU
Testing Levels Based on Software Activity (cont’d)
System Testing – testing system against specification of externallyobservable behaviour
Integration Testing – testing interaction between modulesUnit Testing – testing individual units of a system
traditionally: unit = procedurein object-orientation (Java): unit = method
Failures on higher levels less useful for debugging, as propagationfrom defect to failure is difficult to trace.
This course focuses on lower level: unit testing
![Page 67: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/67.jpg)
/GU
Testing Levels Based on Software Activity (cont’d)
System Testing – testing system against specification of externallyobservable behaviour
Integration Testing – testing interaction between modules
Unit Testing – testing individual units of a systemtraditionally: unit = procedurein object-orientation (Java): unit = method
Failures on higher levels less useful for debugging, as propagationfrom defect to failure is difficult to trace.
This course focuses on lower level: unit testing
![Page 68: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/68.jpg)
/GU
Testing Levels Based on Software Activity (cont’d)
System Testing – testing system against specification of externallyobservable behaviour
Integration Testing – testing interaction between modulesUnit Testing – testing individual units of a system
traditionally: unit = procedurein object-orientation (Java): unit = method
Failures on higher levels less useful for debugging, as propagationfrom defect to failure is difficult to trace.
This course focuses on lower level: unit testing
![Page 69: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/69.jpg)
/GU
Testing Levels Based on Software Activity (cont’d)
System Testing – testing system against specification of externallyobservable behaviour
Integration Testing – testing interaction between modulesUnit Testing – testing individual units of a system
traditionally: unit = procedurein object-orientation (Java): unit = method
Failures on higher levels less useful for debugging, as propagationfrom defect to failure is difficult to trace.
This course focuses on lower level: unit testing
![Page 70: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/70.jpg)
/GU
Testing Levels Based on Software Activity (cont’d)
System Testing – testing system against specification of externallyobservable behaviour
Integration Testing – testing interaction between modulesUnit Testing – testing individual units of a system
traditionally: unit = procedurein object-orientation (Java): unit = method
Failures on higher levels less useful for debugging, as propagationfrom defect to failure is difficult to trace.
This course focuses on lower level: unit testing
![Page 71: Testing, Debugging, and Verification · Testing, Debugging, and Verification Testing, Part I Srinivas Pinisetty 1 08 November 2018 1Slides based on material from Wolfgang Aherndt,](https://reader034.vdocuments.us/reader034/viewer/2022052518/5f0b5e3a7e708231d4302c42/html5/thumbnails/71.jpg)
/GU
Literature related to this lecture
I Introduction to Software Testing - by Paul Ammann, JeffOffuttI Testing levels (Chapter 1)