new testing standards are on the horizon: what will be their impact?
TRANSCRIPT
T24 Special Topics
5/2/2013 3:00:00 PM
New Testing Standards Are on the
Horizon: What Will Be Their Impact?
Presented by:
Claire Lohr
Lohr Systems
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Claire Lohr
Claire Lohr has been an active professional in the computer field for thirty years, with the last twenty years emphasizing software process improvement and testing. Claire provides training - design, authoring, and instruction - and consulting services for a wide variety of both government and commercial clients. She is a Lloyd’s Register trained ISO 9000 Lead Auditor and has been trained to perform Software Capability Evaluations for the SW-CMM. Claire was the chair of the Working Group for the IEEE Std 829-2008 Software and System Test Documentation, and has served on both the IEEE Computer Society’s Software and Systems Engineering Standards Committee and the IEEE Computer Society’s Standards Advisory Board.
4/16/2013
1
1
New Testing Standards on the Horizon: What Will Be
Their Impact?
Claire L. LohrLohr SystemsP.O. Box 2998
Reston, VA 20195703.472.5457
2
Topics
• Why bother with standards?
• Traditional objections
• What’s available (free first)
• Impact of ISO 29119
• Examples
• How to start
• Additional resources
• ?’s
4/16/2013
2
3
Why bother with standards?
• Common problems have common solutions
• Standards are reviewed and modified by groups of very experienced ($$$$) practitioners
• You can’t search the Internet for information unless you know what it is commonly called (or if you have never heard of it)
4
Traditional ObjectionsPast Present
Inconsistent and incomplete
Merging together
Overkill Includes high integrity and lower integrity choices
Just theoretical Includes examples
Expensive Increasingly free
Time consuming ROI of 1:5-6
4/16/2013
3
5
What’s available: free
1. nist.gov
Special Publication 800-115 Technical Guide to Information Security Testing
and Assessment
2. open-stand.org
Commitment for the future by IEEE, IETF, IAB,
Internet Society and W3C.
6
What’s available: free
3. Certification bodies of knowledge
• swebok.org Summary of testing (& rest of SW engineering)
• pmi.org The Software Extension to
the PMBOK® Guide
• istqb.org Glossary of testing terms & multiple tester certifications
• buildsecurityin.us-cert.gov Software Assurance (SwA) Competency Model
4/16/2013
4
7
4. Other free resources
• http://pascal.computer.org/sev_display/index.action SEVOCAB definitions
• http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html free ISO standards
What’s available: free
8
What’s available: free (example)
SWEBOK testing section
Fundamentals
Levels
Techniques
Measures
Process
Tools
4/16/2013
5
9
What’s available: not free
7925-1 Vocabulary7925-2 Component testing
829 Doc’s1008 Unit test1012 V&V1028 Reviews1044 Defect categories
12207 Software life cycle15026 Integrity levels25010 Quality requirements25051 Quality reqm’ts for COTS
10
What’s coming
29119 Systems and software engineering—Software testing—• Part 1: Concepts and definitions
• Part 2: Test processes• Part 3: Test documentation• Part 4: Test techniques
4/16/2013
6
11
ISO 29119-1 Concepts and definitions
4. Definitions
6. Approaches
7. Automation
8. Defects
5. Concepts
12
ISO 29119-2 Test processes
6. Organizational
8. Dynamic8.1 Introduction8.2 Design & implementation8.3 Environment setup & maintenance8.4 Incident reporting
7. Management
4/16/2013
7
13
ISO 29119-3 Test documentation
5. Organizational5.2 Policy NEW5.3 Strategy
6. Management6.2 Plan6.3 Status Report6.4 Completion Report
14
ISO 29119-3 Test documentation
7. Dynamic7.2 Design7.3 Case7.4 Procedure7.5 Data requirements NEW7.6 Environment requirements NEW7.7 Data readiness report NEW7.8 Environment readiness report NEW7.9 Actual result7.10 Test Result7.11 Execution Log7.12 Incident report
4/16/2013
8
15
ISO 29119-4 Test techniques
• Equivalence partitioning• Classification tree• Boundary value analysis• Syntax testing• Combinatorial• Decision table• Cause-effect graphing• State transition• Use case• Scenario• Error guessing• Random
5.2
16
D
A B
E
ISO 29119-4 Test techniques
5.3 • Statement• Branch• Decision• Condition• Data flow
6. CoverageAnnex B Specification-based examplesAnnex C Structure-based examples
4/16/2013
9
17
Impact of ISO 29119Past Future
Inconsistent and incomplete
Consistent and complete in one place
Overkill Includes high integrity and lower integrity choices
Just theoretical Includes examples
Expensive Increasingly free
Time consuming ROI of 1:5-6
18
Example #1
I want to improve our testing processes
Test Manager
29119-3 5.3 Strategy (for focus)• Risk management• Test selection and prioritization• Test documentation and reporting• Test automation and tools• Configuration management• Incident management
4/16/2013
10
19
Example #1 (test manager)
29119-2 8.3 Incident reporting processa) Test results are analyzedb) New incidents are confirmed, if anyc) New incident report details are createdd) The status and details of previously-
raised incidents are determined
e) Previously-raised incident report
details are updated as appropriatef) New and/or updated incident reports are communicated to the relevant
stakeholders
20
29119-3 7.12 Incident report 1. Timing information2. Originator3. Context4. Description of the incident
5. Originator’s assessment of severity6. Originator’s assessment of priority7. Risk8. Status of the incident
Example #1 (test manager)
Add related
incidents
4/16/2013
11
21
Example #2
I want to document better so I can get another job
Tester
29119-2 Annex A Example Test Design*29119-3 Annex H Example Test Design* Chose this one – more
complete and more
summary
22
How to start
Suggested steps for transition
1. Survey your current baseline
2. Choose most valuable “next steps” (Kaizen)
3. Implement a pilot
4. Share what works
5. Continue to improve
4/16/2013
12
23
Additional Resources
www.iso.org
www.ieee.org
www. global.ihs.com
The Roadmap to Software Engineering
Standards: a Standards Based Guide, James W. Moore, 2006
24
Questions?