Ethics of Software Testing
Thomas LaToza
CS 210 Final Presentation
12 / 2 / 2002
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Overview
Background - facts of testing and bugs Education, training, best practices, and a
commitment to reliability All help to improve reliability Don’t solve the problem and still leave
Research How much to spend on testing When to release software Most organizations should spend more on testing
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Facts of Testing Testing is an activity to evaluate quality and find bugs
Testing only proves presence of bugs Testing NEVER proves absence of bugs
(unless you could run every line of code for all possible inputs for all possible states in all possible environments with an oracle that knows the correct output and state)
Testing is underappreciated None of the top 5 CS departments offer an undergraduate
course on testing or building software systems for reliability (other than formal verification) – only taught as part of a software engineering course
Often not perceived as challenging and fun Admission of failure – people make mistakes
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Finding Bugs
Not all bugs are equal.
Bugs may be Easier to find than other bugs More important than other bugs (bugs that
crash or destroy data vs. bugs that make the interface look bad)
Invisible until other bugs are fixed
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Fixing Bugs
Fixing a bug always has a chance of introducing new bug(s) which may be worse
Overall effect of testing depends on ratio of bugs fixed to bugs introduced
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Bugs aren’t cheap
For users Bugs and glitches cost the economy $59.5
Billion a year (NIST estimate)
For software projects Testing cost can easily be 25 – 50% cost of
project and can be much more for certified systems
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Testing bug curve
Bugs
Time / Effort / Testing resources
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Overview
Background - facts of testing and bugs Education, training, best practices, and a
commitment to reliability All help to improve reliability Don’t solve the problem and still leave
Research How much to spend on testing When to release software Most organizations should spend more on testing
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Engineering practices can help prevent bugs Code reviews – highly effective
Team members proof read the code and discuss
Writing for readability There are two ways of constructing a software design; one way is to make it so simple
that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. - C. A. R. Hoare
Design by test Write the interface and test case first, then write the functionality
Statistical testing Commercial off the shelf software / Open Source /
Components Separate Quality Assurance team Betas
But none of these replace testing…
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Research
Problem has (recently) sparked great research interest
Recovery Oriented Computing Formal Methods Program Comprehension Software Engineering
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Testing Review
It’s expensive and time consuming
Doesn’t guarantee lack of bugs
Developers often don’t like doing it
But it’s really important…
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Overview
Background - facts of testing and bugs Education, training, best practices, and a
commitment to reliability All help to improve reliability Don’t solve the problem and still leave
Research How much to spend on testing When to release software Most organizations should spend more on testing
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Review – Applying Ethical Theories
1. Evaluate information2. Consider how decision may affect stakeholders3. Consider what ethics [ethical values] are relevant to
situation4. Determine the best course of action that takes into
account relevant values and stakeholders’ interests.
(from Lockheed Martin Decision Making Model)
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
How much to spend on testing (1. Evaluate Information)
Most organizations should spend more on testing
But have to stop somewhere – can’t spend all of your budget on testing
Question of where on the curve is best There are advantages for being further left
and advantages for being further right
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
More testing / Less bugs(2. Consider how decision may affect stakeholders)
Advantages Developers / Customers – less time working around bugs Customers – less frustration dealing with software Managers – know more about bugs in product and can deliver a higher quality
product Company – quality makes software more valuable to customer Company – sooner bugs can be fixed the cheaper they are to fix
Disadvantages Customers – more expensive products Company – less features, less compelling reasons for customer to spend money
upgrading
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
How much to spend on testing(3. Consider what ethics [ethical values] are relevant to situation)(4. Determine best action based on relevant values and stakeholders)
Utilitarian – maximize cost / benefit for company and / or customers
Value – have a responsibility to provide a quality product by setting a quality standard and sticking to it
Libertarian – just enough that it helps company the most (customer fends for himself)
Altruism – so that it is best for the customer
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
When to release (1. Evaluate Information)
Would like to have a higher quality product But have to stop somewhere – may not ever
find all of the bugs Again question of where on the curve is best There are advantages for being further left
and advantages for being further right
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Release later / Less bugs(2. Consider how decision may affect stakeholders)
Advantages All of the advantages of more testing / less bugs Managers – know more about quality of the product being delivered Company / customers – higher quality product
Disadvantages All of the disadvantages of less testing / more bugs Company – may lose marketshare to competitors Company – no income from product Customer – can’t use product at all Company – may lose credibility for missing deadline
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
When to release(3. Consider what ethics [ethical values] are relevant to situation)(4. Determine best action based on relevant values and stakeholders)
Utilitarian – maximize cost / benefit for company and / or customers
Value – set quality standard vs. keeping a ship date vs. money for company
Libertarian – whenever the most customers will buy it
Altruism – so that customer will have optimum balance of most time to use it and the best product
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Conclusions
Many other ethical questions Should a bug be fixed? How much information about testing to release? Do developers and testers have professional
standards to uphold? Expectations for quality – too high or too low?
© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign
Further Reading
Economics of Software Verification http://cm.bell-labs.com/cm/cs/who/gerard/gz/paste01.pdf