9) sigists13 - stuart reid - iso 29119 the new standard€¦ · bs 7925-1 bs 7925-2 ieee 829...
TRANSCRIPT
![Page 1: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/1.jpg)
ISO/IEC/IEEE 29119
The New International Software Testing
Standards
Stuart Reid
Testing Solutions Group
117 Houndsditch
London EC3
UKTel: 0207 469 1500Fax: 0207 623 8459
www.testing-solutions.com
©Stuart Reid, 2013
1
![Page 2: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/2.jpg)
Scope
• The purpose of standards
• Motivation for software testing standards
• Development of standards
• Overview of ISO 29119
• Applicability
• Timeline
• Future involvement
2
![Page 3: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/3.jpg)
What are standards?
– Guidelines documents as they are not compulsory unless mandated by an individual or an organization
– Agreements because they should reflect a certain level of consensus
“Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationallyrecognized industrial, professional, trade associations or governmental bodies”
- ISO
3
![Page 4: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/4.jpg)
Why use standards?
• Consumers
– Confidence in compliant products
– Authors provide expertise in standards
• Manufacturers
– Conformance and Marketing
– ‘Safety’ from liability
– Guidelines on production
• But not ‘Best Practice’…
5
![Page 5: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/5.jpg)
Quality and Standards
6
![Page 6: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/6.jpg)
What use are standards?
• Basis for:
– Communication – common terminology
– Professional qualifications
– Certification/compliance schemes
– Benchmark of ‘good industry practice’
– Contracts
– Interoperability and consistency……
Standards describe a current ‘body of knowledge’ that provides the basis for a professional discipline
7
![Page 7: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/7.jpg)
A Case in Point
8
![Page 8: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/8.jpg)
Motivation for ISO 29119
• Demand for existing 'standards’
• Conflicts in current definitions and processes
• Gaps in the current standards provision
• A Baseline for the Testing Discipline
• Current industry practice is lacking
• Buyers unclear on what is 'good test practice'
9
![Page 9: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/9.jpg)
Standardization Bodies
10
![Page 10: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/10.jpg)
ISO/IE
C Softw
are Standards
0
20
40
60
80
100
120
140
160
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
Published
Maintained
11
![Page 11: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/11.jpg)
ISO/IEC SC7 – WG26
ISO
TC176
Quality
JTC1
Information Technology
SC7
Software & Systems
Engineering
WG2 ..................
WG26
Software Testing
Representatives of National
Standards Bodies
SC22
Programming
Languages
SC27
IT Security
SC32
Data Management & Interchange
IEC
TC56
Dependability
12
![Page 12: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/12.jpg)
ISO/IEC 29119 – Structure
BS 7925-1
BS 7925-2 IEEE 829
Concepts & Vocabulary
Part 1
ProcessAssessment
TestingTechniques
Part 4
Documentation
Part 3Part 2
Processes
Keyword-Driven Testing
Part 5 ISO/IEC 33063
13
![Page 13: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/13.jpg)
SOFTWARE TESTING CONCEPTS
Scope, Conformance, Normative References
TESTING IN DIFFERENT LIFE CYCLE MODELS
ROLES AND RESPONSIBILITIES IN TESTING
ANNEXES – Metrics, Examples, Bibliography
DEFINITIONS
Part 1: Concepts & Vocabulary
14
![Page 14: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/14.jpg)
Part 2: Testing Processes
TEST MANAGEMENT PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST PROCESSES
15
![Page 15: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/15.jpg)
TEST MANAGEMENT PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST PROCESSES
Instantiating Testing Processes
16
![Page 16: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/16.jpg)
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
TEST
PLANNING
TEST
MONITORING &
CONTROL
TEST
COMPLETION
ORGANIZATIONAL TEST
DOCUMENTATION
FEEDBACK ONORGANIZATIONAL TEST DOCUMENTATION
TEST PLAN UPDATES
TESTPLAN
TESTCOMPLETIONREPORT
DYNAMIC TEST
PROCESSES
TEST
MANAGEMENT
PROCESSES
TEST PLAN,TEST COMPLETION REPORT,TEST MEASURES
TEST MEASURES
TEST PLAN,CONTROL DIRECTIVES
TEST PLAN,CONTROL DIRECTIVES
Test Management Processes
17
![Page 17: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/17.jpg)
OrganiseTest Plan
Development
Identify & Estimate Risks
Design Test Strategy
Determine Staffing and Scheduling
Document Test Plan
Schedule, Staffing Profile
Test Strategy
Analyzed Risks
Scope
Identify Risk Treatment Approaches
Gain Consensus on Test Plan
ApprovedTest Plan
DraftTest Plan
Test Plan Publish
Test Plan
Understand Context
TreatmentApproaches
Test Planning Process
18
![Page 18: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/18.jpg)
DYNAMIC TEST PROCESSES
TESTSPECIFICATION
[NO ISSUESNOTICED]
TEST ENVIRONMENT READINESS REPORT
TESTENV’T
REQUIREMENTS
INCIDENTREPORT
TEST DESIGN &
IMPLEMENTATION
TEST
ENVIRONMENT
SET-UP
TEST
EXECUTION
TEST
INCIDENT
REPORTING
(LEVEL) TEST MANAGEMENT PROCESS
[ISSUE NOTICED orRETEST RESULT]
TEST RESULTS
CONTROL DIRECTIVES TEST
MEASURES
(LEVEL)TEST PLAN
Dynamic Test Processes
19
![Page 19: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/19.jpg)
Test Process Descriptions Example- Organizational Test Process
• Each testing process is described using a standard template (following ISO 24774), for example:
• Purpose
– The purpose of the Organizational Test Process is to develop and maintain organizational test specifications, such as the Test Policy and Organizational Test Strategy.
• Outcomes
– As a result of the successful implementation of the Organizational Test Process:
• The organizational test specification is developed based on stakeholder requirements;
• The organizational test specification is agreed by stakeholders;
• The organizational test specification is published and accessible to stakeholders;
• Conformance of stakeholders with the organizational test specification is managed;
• The organizational test specification is reviewed on a periodic basis;
• Minor updates to the organizational test specification are made as necessary.
• Activities and tasks
– The tester shall implement the following activities and tasks in accordance with applicable organization policies and procedures with respect to the Organizational Test Process.
� Develop Test Specification
• This activity consists of the following tasks:
– Analyze any relevant source documents and the current testing practices within the organization to identify requirements for the organizational test specification.
– etc.
� Gain Consensus on Test Specification
� etc.
• Information items
– The output of the Organizational Test Process is the organizational test specification. The Organizational Test Policy and Organizational Test Strategy are typical examples of organizational test specifications.
20
![Page 20: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/20.jpg)
Part 3 – Test Documentation
TEST DOCUMENTATION
ANNEXES - EXAMPLES
Scope, Conformance, Normative References
21
![Page 21: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/21.jpg)
Test Documentation Types
• Organizational test documentation– Test policy– Test strategy
• Project test documentation – Project test plan– Test project completion report
• Test level documentation– Test plan– Test specification – Test results– Anomaly reports– Level test status report– Test environment report – Test level completion report 22
![Page 22: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/22.jpg)
Part 4 – Test Techniques
TEST COVERAGE MEASUREMENT
Scope, Conformance, Normative References
ANNEXE – TESTING OF QUALITY CHARACTERISTICS
ANNEXE – SELECTION OF TECHNIQUES
ANNEXE – TEST TECHNIQUE EFFECTIVENESS
TEST DESIGN TECHNIQUES
BLACK BOX WHITE BOX
23
![Page 23: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/23.jpg)
Are they applicable to you?
• Intended to be a generic standards
• Full vs. tailored conformance
• Small and large organizations
• Agile and Traditional
• Financial, Safety-Critical, Games...
• Following Regulatory Standards?
• Can you afford not to?
24
![Page 24: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/24.jpg)
Current Status
May 10
May 11
May 12
May 13
…
Working Draft (WD)Committee Draft (CD)Draft International Standard (DIS)Final Draft International Standard (FDIS)Final International Standard (FIS)
Parts 1, 2 & 3Parts 1, 2 & 3
Part 4Part 4
WDWDCD1CD1
DISDISFDISFDISWDWD
CD1CD1
DISDIS
FDISFDISFIS!!!FIS!!!
CD2CD2CD3CD3
CD2CD2CD3CD3
Part 5Part 5
CD4CD4
DIS-2DIS-2
WDWDCDCD
25
![Page 25: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/25.jpg)
Current status
• Drafts are out for review– Unsurprisingly (for a testing standard) we get plenty of comments (1000s)
• Countries attending meetings include:– Argentina, Australia, Brazil, Canada, China, Colombia, Denmark, Finland, France, Germany, Hong Kong, India, Japan, Korea, Malaysia, New Zealand, Peru, Poland, Russian Federation, South Africa, Spain, Sweden, Turkey, UK, USA
• Last meeting – Montreal, May 2013
• Next meeting – Edinburgh, Nov 201326
![Page 26: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/26.jpg)
What is most likely to stopISO 29119?
• Fear of change
• 'Not invented here' syndrome
• Perceived as competition
• Lack of required use
• Anti-standardization – craftsmen
• No evidence of efficacy
• Cost of use
• Complexity
• Lack of responsiveness
• Lack of inclusiveness…27
![Page 27: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/27.jpg)
Conclusions
• Parts 1, 2 & 3 published September 2013
– My thanks to the editors and reviewers ☺
• International standard will provide practitioners with guidelines for testing that cover all aspects of the life cycle
– Provides a consistent set of definitions, processes, procedures & techniques for software testing
• Will be adopted by IEEE, BSI, ISO and other national standards bodies
28
![Page 28: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/28.jpg)
Do you want to be involved?
• Join the working group
– representing your national standards body
– 6 day meetings, every 6 months
– contribute between meetings
• Join a WG26 mirror group
– for your national standards body
• Contribute materials
• Review drafts
29
![Page 29: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/29.jpg)
Any Questions?
Any Volunteers?
30
![Page 30: 9) SIGISTS13 - Stuart Reid - ISO 29119 The New Standard€¦ · BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Process Assessment Testing Techniques Part 4 Documentation](https://reader030.vdocuments.us/reader030/viewer/2022040605/5eacdf8e402fa44fc311bb51/html5/thumbnails/30.jpg)
Finally…
– if you have any questions on the standards
– if you are interested in trialling the standard on a project, reviewing drafts or writing examples
• http://softwaretestingstandard.org/
– WG26 website
• http://www.jtc1-sc7.org/
– access to official documents released by WG 26
31