deployment of a test management solution for a defence project using an integrated tool chain
Post on 14-Jun-2015
261 Views
Preview:
DESCRIPTION
TRANSCRIPT
© 2014 IBM Corporation
Deployment of a test management solution for a defence project using an integrated tool chain
Code 1183
Dr. Einar Karlsen (IBM, Co-Speaker)
Thomas Schöning (Airbus DS)
1
Introduction
Dr. Einar Karlsen (IBM)-Technical Solution Architect
Thomas Schöning (Airbus Defence and Space)- Integration and Test Manager
Thomas Südbröcker (IBM)- Technical Sales and Solutions + Community of Practices Leader Quality Management (Germany)
2
• Subcontractor in a NATO Project with requirement based development, test and formal verification of a complex system entity
• Find a proper tooling solution for test and verification process using the IBM Rational tooling chain (Jazz Platform)
– Support for relevant standards ISO/IEE 15288, ISO/IEEE 12207, ISO 9001, ISO/IEC 9126
– Support for relevant process: Waterfall (V-Model) with incremental SW Development
– Integration with IBM Rational DOORS to accommodate for requirement traceability
– Automatic document generation for requirements, design and test artefacts (plans, procedures, reports)
– Insight into status and trend of the testing efforts (metrics)
Setting the Scene – Situation
3
• Challenge
– Unknown effort to adequately install, integrate and customize the whole tooling chain to perform modelling, implementation, and test management
– Appropriate test management support (planning, execution, reporting, document generation, etc) according to required process?
• Improvements
– Decrease of costs, risks, inconsistencies and time-to-market through automatic generation of project deliverables
– Collection of metrics to measure test design progress, requirements coverage
– Online/Offline determination of test results while performing the test
Setting the Scene – Risk and Opportunity
4
• Profound orientation for configuration and customization of IBM Rational Tools for a tool-aided test management solution (planning, execution, reporting, documentation) for similar test activities
1. Start early to implement test management prototyping to better estimate the effort to adapt the tools to the project needs
2. Invite technical professionals to have guidance and avoid pitfalls
3. Perform team training
4. Look at the deliverables before customizing your tool chain
Setting the Scene – Position and Benefits
5
• Introduction
– Programme Overview
– Test and Evaluation Strategy
• Test management
– Solution Overview
– Solution Architecture
– Document Generation
– Metrics and Dashboards
• Summary
Agenda
6
• Confidential, multinational NATO C4ISR Project (C4ISR=Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance)
– Unmanned Air, Command and Control Entities
– Transportable and Mobile Ground Stations for data exception, processing, and archiving facilities
– Communication links to other NATO and interoperable national ground stations, and to NATO and national Command and Control systems
– Exchange of standardized data via different interfaces and communication lines
• Airbus DS responsible for development, test and formal verification of a system entity consisting of several Subsystems
Programme Overview
Test and Evaluation Strategy
Verification
Development
Testing
Entity
Subsystems
CSCIsHWCIs
ServicesComponents
Development
Testing
Development
TestingTest
Test
Test
Verification
Integration
8
• Requirements of System Entity, Subsystems, Computer Software Configuration Items (CSCIs), and Hardware Configuration Items (HWCIs) are kept in DOORS database with traceable entries
• Testing/Verification is classified either to be
– Informal Tests
• conduct informal tests throughout the system development
– Subcontractor Development Tests
• For CSCI, HWCI, HW/CSCI groupings, Subsystem level, Entities and Integration testing
– Formal Verification
• Verification of allocated customer requirements based on proof of derived requirements (Entity, Subsystem, HW/CSCI groupings)
Test and Evaluation Strategy
Test Management – Solution Overview
9
10
Test Management – Solution Overview
• Test Plan
– consists of a number of Test Cases used to proof requirements
– is linked with one or more Requirements Modules (Test Plan views) � coverage and traceability (verification level)
• Test Case
– is linked to one or more particular requirements belonging to the requirements view(s) proofing requirements on different levels
• Test Script
– keeps the Test Procedure with single steps for test execution
• Test Case Execution Record/Test Case Result
– container to keep the Test Case results depending on iteration (integration, dry-run, acceptance) and test environment
• PTR (=Problem Trouble Report)
– keeps the defects occurred during Test Case execution
Test Management – Solution Architecture
11
IBM Rational DOORS
IBM Rational Insight
IBM Rational Publishing Engine
REST, ETL REST, ETL
IBM Rational Quality Manager
IBM Rational Team Concert
REST, ETL
Dashboards, Status and trend reports
Test Plans, Procedures and Reports(Project Deliverables)
RESTREST REST
OSLClinks
OSLClinks
12
Test Management – Test Document Generation
Test Document Generation – Master Test List
Test Document Generation – Test Case Details
15
• Metrics: What you can measure that can be done!
• Metric types: Project, Product, Process, and Personnel
• Definition of metrics for monitoring and control test planning, design, development and execution
– test planning
• Percentage of requirements having a test case
– test design (analysis and concept)
• Percentage of test case with different status (draft, ready for review, approved)
– test development
• Percentage of test scripts with different status (draft, ready for review, approved)
– test execution
• Percentage of planned test cases being executed (passed, passed with condition, failed, blocked, error)
Test Management – Metrics and Dashboards
16
Metrics and Dashboards – Test Case Design
17
Metrics and Dashboards – Test Case Execution
18
• Tooling Solution (Jazz platform) for test management process with combined DT&E strategy (Informal, Development Tests, Formal Verification) applied for system entity (NATO Project)
• Test Management Solution (Overview and Architecture)
– Test Plans linked with requirements view
– Test Cases linked with requirements
– Test Execution Records keeping the results for each Iteration
– Definition of timelines, iterations to support test planning andexecution
• Test Document Generation
– Tooling adaption for artefact templates (Test Plan, Test Case)
– Examples (Master Test List, Test Case Details)
• Metrics and Dashboards
– metrics for monitoring and control test planning, design, development and execution
Summary
Articles on developerWorks
• RQM Template authoring articles on developer Works.
– Using IBM Rational Publishing Engine to generate compliance
documents: Report generation for Rational Quality Manager,
introducing REST service
URLs, importing schemas, and
basic template authoring
– Using IBM Rational Publishing
Engine to generate compliance
documents: Part 2. Report generation for Rational Quality
Manager, showing how to
extract information from grid sections and report on test
results and related defects
Thank You!
Your Feedback is Important!
Access the Innovate agenda tool to complete your session surveys from your smartphone, laptop or
conference kiosk.
21
• Test Plan:
– Artifact Templates (keep Test Cases and TCERs only)
– Artifact Categories (manufacturer, part number, type) to distinguish between different plans (system, software,etc)
• Test Case:
– Artifact Template has been customized to the documentation format needs (Test Plan, Test Procedure)
– Artifact Categories (test area, test method, test event, test type) and RQM section (description, test objective, requirement links,estimation) are used to appropriately keep Test Case data
• Timelines (Development and Production Phase)
• Iteration Types (Milestone, Software Test, System Test)
• Iteration
– directly corresponds with incremental software development (integration tests, dry run, acceptance test)
– support generation of TCER
Test Management – Tooling adaption
22
• Administration of project requirements and test cases as well as the creation of bi-directional dependencies between them
• Planning test effort and milestones
• Defining test goals, acceptance and quality criteria
• Deriving test cases derived from requirement specifications
• Estimation of the effort of the test activities
• Definition of a time plan for each test iteration with finish dates of key activities
• Collection of different test environments and the generation of test configurations
• Creation and administration of test cases (manual and automated tests)
• Recording test results, creating defect and evaluating reports
• Support review and release process for the test plan and individual test cases.
IBM Rational Tools – Usage
top related