bug debug keynote - present problems and future solutions
DESCRIPTION
TRANSCRIPT
![Page 1: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/1.jpg)
1
Software Testing – Present Problems and Future Solutions
Vipul Kocher
![Page 2: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/2.jpg)
2
Jules Verne’s “Twenty Thousand leagues under the sea” (1869) – Simon Lake built “Argonaut” first submarine to operate in the open seas (1897)
Arthur C. Clark Geosynchronous satellites (1945) – Actual satellites (1963)
Cloning - Dolly
Science Fiction – Fiction today, Reality tomorrow
![Page 3: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/3.jpg)
3
What About These?
Androids Time Teleportation
How about self testing code?
May that never become a reality as
long as I live…
What about ? travel? ?
![Page 4: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/4.jpg)
4
assumptions
![Page 5: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/5.jpg)
5
Assumptions
• All future solutions are based on today’s technologies
• Disruptive innovations like new class of operating systems, development paradigms, languages may significantly alter the landscapes
![Page 6: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/6.jpg)
6
Assumptions
I have a few ideas on
• new class of Operating Systems
• new class of programming languages and paradigms to support that operating system
• ultimately supported by new class of hardware
I have no clue about it !
![Page 7: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/7.jpg)
7
problems in testing
![Page 8: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/8.jpg)
8
Problems in Testing
Are there problems in testing? We all seem to be doing fine!
Probably the biggest problem in testing is- “we do not have a problem”
![Page 9: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/9.jpg)
9
Problems in Testing
Why is this a problem? Is there a solution in future?
![Page 10: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/10.jpg)
10
challenges in testing
![Page 11: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/11.jpg)
11
Challenges in Testing
• What does testing mean for different stakeholders?• Have we tested enough?• How much time do I need to test it?• How many test cases should I write?• Oracle problem• How do I choose right set of regression tests?• How do I get developers to do better testing?• Do I need coding skills? What will I do with those?• How do I teaching testing to my testers?
![Page 12: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/12.jpg)
12
What does testing mean for different stakeholders
![Page 13: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/13.jpg)
13
What does testing mean for different stakeholders?
Let us face itNobody wants testing…
![Page 14: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/14.jpg)
14
What does testing mean for different stakeholders?
If one assumes that “testing is necessary” one is likely to try to improve the way testing gets done
If one believes that testing is unnecessary then one is more likely to find ways to minimize or
eliminate testing
Mindset Matters !
![Page 15: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/15.jpg)
15
testing world’s biggest problem
![Page 16: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/16.jpg)
16
Testing World’s biggest problem
• Where is the innovation in testing? • Innovation as in Breakthrough Innovation?
![Page 17: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/17.jpg)
17
Testing World’s biggest problem
Can you name three breakthrough innovations in testing? At least one?
![Page 18: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/18.jpg)
18
HITE
H I T EHave I Tested Enough
![Page 19: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/19.jpg)
19
HITE
First define “ENOUGH” – • Today
- No catastrophic or critical or major failure for users
• Tomorrow - ?
![Page 20: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/20.jpg)
20
HITE
• Integrated software status monitor - Explained later (Patent Pending)
• New software technology - Self monitoring (Patent Pending), Self healing systems
• Software testing as a distributed, automated service - you pay software companies to install software on your system and test it (Patent Pending)
![Page 21: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/21.jpg)
21
How much time do I need to test it
![Page 22: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/22.jpg)
22
How much time do I need to test it
What are the problems you face in answering this
question?What are the solutions today?
• Estimation techniques? Work
Breakdown Structure? Function Points? Test Points? Use Case Points?
• Historical Data? Rule of
thumb? Gut feel?Are these accurate? What about
• Different levels of quality, workmanship (developers and testers both), project constraints such as budget, time and Subsequent Regression Cycles?
Future solution
• A mechanism to define quality level and a mechanism to measure that level (Patent Pending)
• Collection and analysis of data in automated manner and predictions based on current state of the project (Patent Pending)
![Page 23: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/23.jpg)
23
Bug – The Only Perfect Being
![Page 24: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/24.jpg)
24
Bug – The Only Perfect Being
Most bugs are because imperfect requirements are imperfectly translated into imperfect design which is imperfectly translated into imperfect code
![Page 25: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/25.jpg)
25
Whole Is More Than The Sum Of Parts
![Page 26: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/26.jpg)
26
Whole Is More Than The Sum Of Parts
Requirements are captured in
a granular mode, risk
prioritized and tracked and
checked automatically
for consistency, ambiguity,
incompleteness
Automated/assisted impact analysis is done
on other requirements/d
esign/code
Requirements are mapped to individual elements of design/code
Design is also assisted along
with code generation and
reverse engineering
Unit and integration tests
are automatically generated for
code
System tests of various
granularity are automatically generated and assisted based on some new and old test
design techniques
An integrated environment where…
![Page 27: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/27.jpg)
27
Whole Is More Than The Sum Of Parts
The tests are executed in
an automated
manner and results
compared
The whole stack trace is maintained
and the entire
application can be
replayed line by source
line
All log/user messages are bubbled up by the tool
and test written to
execute those
conditions and all un-displayed
error messages are
flagged as holes
Performance and coverage
data is maintained
The map of code
executed has statistics on
requirements related to
that piece of code, tests,
testers, who wrote that code, bugs, modules,
risks, performance
etc
Each code fix is bubbled up to impact on other related
modules, features, GUI
elements, performance
etc
Regression tests are
suggested and run and
coverage, performance, security etc.
are all checked
again
An integrated environment where…
![Page 28: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/28.jpg)
28
Integrated Environment
Requirements Visual Prioritized Self-checking Testable Constrained
Design – Mapped to Requirement
Code – Mapped to Requirements/DesignTests – Mapped to Requirements/Design/Code
![Page 29: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/29.jpg)
29
Regression Test Selection Tool
![Page 30: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/30.jpg)
30
Test Design Tools for automated / assisted creation of test cases – Q-Patterns, Unified
test design method, Noun and
Verb
Requirem ents
Requirements
Requirem ent based m odels
Requirement based models
D esign
Design
D esign based m odels: U M L
and future
Design based models: UML
and future
Code
Code
Code based m odels: U M L
and future
Code based models: UML
and future
D efect database
Defect database
Executi on tools – CA SE,
CA ST, A utom ati on
Execution tools – CASE,
CAST, Automation
Tool-Chain
![Page 31: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/31.jpg)
31
User Messages and API based test cases
Code and Repository of “user” messages Log messages generated
Where is this message called?
How? Has it been called? Has this
log been generated?
API tests. Have all APIs been
covered? With what data?
![Page 32: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/32.jpg)
32
Coverage & Data – Integral Part of Tool-Chain
Executable
External Interface coverage
Requirement Coverage
Risk Coverage
Test Coverage
Non-functional Coverage
Code Coverage
Code profiling
Code Security check
Customer care –
database, emails, blogs
Defect database,
data-mining for bug-
taxonomy
Data from application
monitoring in production
Internet security
advisories
StandardConformance
![Page 33: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/33.jpg)
33
New format for writing test cases: visual format
![Page 34: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/34.jpg)
34
An example of Visual format (http://www.exampler.com/software.html)
![Page 35: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/35.jpg)
35
Ability to replay code
Crash
![Page 36: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/36.jpg)
36
Testing as a Framework
![Page 37: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/37.jpg)
37
![Page 38: Bug debug keynote - Present problems and future solutions](https://reader031.vdocuments.us/reader031/viewer/2022013003/54bdb3854a7959e46d8b45f6/html5/thumbnails/38.jpg)
38
Far Future – Where Testing?
• Tool driven
• Integral part of development
• Better ways to tie all development artifacts together
• Integration with various sources of information • Better analysis and action