deploying code coverage analysis to improve software quality
TRANSCRIPT
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
© Vector Software, Inc. © Vector Software, Inc. © Vector Software, Inc. © Vector Software, Inc. © Vector Software, Inc. © Vector Software, Inc.
Deploying Code Coverage Analysis to Improve Software Quality
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > About Vector Software
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Vector Software develops embedded software testing products > U.S. Based company founded in 1990 by embedded developers
> First product released in 1994 for Lockheed Martin’s C-130J “Super” Hercules
> Corporate headquarters in Rhode Island
> Sales and support office worldwide
> Providence
> Boston
> Jacksonville
> Denver
> Phoenix
> London
> Düsseldorf
> Shanghai
Our Company
sol id reference accounts in mult iple industr ies
DO-178B | ED-12B
Aerospace
FDA | IEC 62304
Medical
ISO 26262
Automotive
CENELEC | EN 50128
Railway
IEC 61508
Industrial
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Our Clients
Autoliv Autoeversystems AIV Beko Bosch Chang'an Automotive Delphi Dongfeng Automotive ENSO Fujitsu Hyundai Magna Powertrain Magneti Marelli MOTONIC Siemens TRW Valeo Yura
Abbott Laboratories Acist Medical Baxter Healthcare Beckman Coulter Boston Scientific Biosafe Cardinal Health Covidien Enteromedics GE Healthcare Hoana Medical Hospira Medical Impact Instrumentation Medtronic Philips Medical Sirona Dental Smiths Medical Terumo Heart Varian
ABB Allied Telesis Alcatel Ansaldo Signal Alstom Brunel Bombardier CAP Gemini Elin EBG Electro-Motive Diesels GE Transportation Invensys Rail ŠKODA ELECTRIC Safetran Systems Siemens Triconex Thales Union Switch & Signal Verocel
Amazone Analog Devices Arçelik Bechtel Bettis Berthold Technologies British Energy Brunel Caterpillar Changzhou Scale Ltd. Claas Endress+Hauser Flextronics Foster Miller General Electric HAWE Hydraulik ICS Tripex Invensys Ingersoll Rand Itron Krauss-Maffei Mettler Toledo AG NetApp Oak Systems Sanmina Sunways ViaSat Wilo ZIEHL
AEL Sistemas Astrium Boeing Bell Helicopter Belcan BAE Systems Cessna Aircraft EADS Electric Boat Goodrich Aerospace General Dynamics Hamilton Sundstrand Honeywell ITT Communications India Space Research Korean Aerospace Lockheed Martin L-3 Communications Northrop Grumman Pratt & Whitney Raytheon Rockwell Collins Rolls Royce Saab Group Thales Turkish Aerospace U.S. Air Force XMobots Robotic
DO-178B | ED-12B
Aerospace
FDA | IEC 62304
Medical
ISO 26262
Automotive
CENELEC | EN 50128
Railway
IEC 61508
Industrial
Companies who… • have safety or business critical applications • have processes which specify rigorous testing • have unit and integration testing requirements • need the ability to automate regression testing • have requirements to prove code coverage • realize that NOT testing… is NOT an option
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > Understanding Code Coverage Analysis
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Data gathered during execution of an application
> Highlights what lines of code have been executed
> More important is which lines were NOT executed > Indicates poorly tested areas of the application
> The more unexecuted code, the more risk of bugs
> Can be deployed during all phases of testing > Unit testing
> Module integration testing
> Software integration testing
> Functional testing
> System testing
> Especially important for legacy or fielded code which “just works”
What is Code Coverage Analysis?
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Levels of Code Coverage
Statement
Decision or Branch (If Statements, loops, case statements)
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Levels of Code Coverage (Cont.)
Modified Condition/Decision Coverage
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Levels of Code Coverage (Cont.)
Basis Path Coverage (unique paths through each function)
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Required by many of the industry standards as a way to prove that the testing is sufficient
> Helps projects answer critical questions: > Am I done testing?
> Have I identified all the requirements for this project? (if all of the requirements are tested and you still have uncovered code – you may need more requirements)
> Is there any dead code in my application? (code that cannot be reached)
> Helps organizations focus their testing resources on areas that will yield the greatest benefit
> Provides critical information on how well legacy code is being tested
Why is Code Coverage Analysis Important?
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > Industry-Specific Software Testing Guidelines
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Safety Critical Industries
Avionics > DO-178B ▪ DO-178C ▪ DO-278
Aerospace | Defense > Military Standards
Automotive > MISRA ▪ ISO 26262
Medical > IEC 62304 ▪ FDA Guidance
Industrial Controls > IEC 61508
Railway > CENELEC
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Structural Testing > Testing requirements and demonstrating associated code coverage
> Satisfied with a combination of system, integration, and unit testing
> Level C (Statement) > Prove that every executable statement has been invoked at least once
> Level B (Decision) > Above, plus, prove that every point of entry and exit, and each decision point outcome, have
been invoked at least once
> Level A (MC/DC) > Above, plus prove that each component of a complex conditional independently drives the
outcome of the conditional
DO-178B Testing Requirements
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> A. Probabilistic Testing > Probability of error reduced by testing with several independently chosen values for tests
> B. Dynamic Analysis and Testing > Section covers a variety of tests, many of which can be achieved through a mix of testing on
the entire software build (system testing) and unit/module testing
IEC 61508 – Part 3: Software Requirements
Technique / Measurement SIL
1 2 3 4
1. Test case execution from boundary value analysis R HR HR HR
2. Test case execution from error guessing R R R R
3. Test case execution from error seeding - R R R
4. Test case execution from model-based test case generation R R HR HR
5. Performance modeling R R R HR
6. Equivalence classes and input partition testing R R R HR
7a. Structural coverage (entry points) 100% HR HR HR HR
7b. Structural coverage (statements) 100% R HR HR HR
7c. Structural coverage (branches) 100% R R HR HR
7d. Structural coverage (MC/DC) 100% R R R HR
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
FDA Developer Testing Guidelines
“A software products testing
can be organized into unit,
integration, and system
testing”
“Testing starts with unit test
and concludes with system
testing”
“Testing entails running
software products under
known conditions with
defined inputs and
documented outcomes”
“Essential element of test is
the expected result”
“The amount of structural
coverage is commensurate
with the level of risk posed”
“Regression analysis and
testing are employed to
provide assurance that a
change has not created a
problem”
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
IEC 62304 Safety Classification Effects
Software Documentation
Class A
Class B
Class C
Software development plan
Must contain contents to sections 5.1 IEC 62304:2006. The plan's content list increases as the class increases, but a plan is required for all classes.
Software requirements specification
Software requirements specification conforming to 5.2 IEC 62304:2006. The content list for the software requirements specification increases as the class increases, but a document is required for all classes.
Software architecture
Not required. Software architecture to 5.3 IEC 62304:2006. Refined to software unit level for Class C.
Software detailed design
Not required. Document detailed design for software units. (5.4).
Software unit implementation
All units are implemented, documented and source controlled (5.5.1).
Software unit verification
Not required.
Define process, tests and acceptance criteria (5.5.2, 5.5.3). Carry out verification (5.5.5)
Define additional tests and acceptance criteria (5.5.2, 5.5.3, 5.5.4). Carry out verification (5.5.5).
Software integration and integration testing
Not required. Integration testing to 5.6 IEC 62304:2006.
Software system testing
Not required.
System testing to 5.7 IEC 62304:2006.
Software release Document the version of the software product that is being released (5.8.4).
List of remaining software anomalies, annotated with an explanation of the impact on safety or effectiveness, including operator usage and human factors.
Summary of safety classification effects on the code development documentation and process
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
ISO 26262 Automotive Software Standards
> Adaptation of IEC 61508 generic standard
> Deals with electronic software elements related to safety
> Detailed references to required testing
> Risk-management approach
> Automotive Safety Integrity Levels (ASIL) > Similar to IEC 61508 Safety Integrity Levels (SIL)
> Four levels (A-D) to specify necessary safety measures for avoiding risk
ISO 26262 Table 14 – Structural Coverage Metrics at the Software Unit Testing
Methods ASIL Supported by
VectorCAST A B C D
1a. Statement Coverage HR HR HR HR Yes
1b. Branch Coverage R HR HR HR Yes
1c. MC/DC (Modified Condition / Decision Coverage) R R R HR Yes
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > Deploying Code Coverage Analysis on Legacy Applications
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Testing Challenges of Legacy Code
> How do I know where the code might break? > Implicit “trust” in the code since it has been fielded, but still a risk
> Changing code in or around it could cause latent bugs to emerge
> It is too expensive to go back and fully unit test > Requires at least 1 Test Line of Code (LOC) per Deliverable LOC
> Can Consume 50% of Development Budget
> Would eliminate any advantages of code reuse
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> System / Functional Testing > Execution of entire application to ensure conformity to overall system requirements
> System testing cannot guarantee full code coverage
> Generally performed by a dedicated test group
> 60-70% coverage is common
> Regression Testing > Re-runs previously developed tests against changed software
> Identifies whether changed software behaves correctly
> Compares expected against actual test results
> Measures code coverage against previous coverage levels
Testing Legacy Code
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Performed by QA Group or a dedicated system test team
> Goal is to prove functional requirements
> Challenges with system testing > Completeness of testing is difficult to judge
> Completeness of requirements is not known
> “Extra” functionality might be deployed
> “Dead Code” might be deployed
> Code coverage analysis addresses these challenges!
System Testing Facts
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Easy to deploy > Able to instrument one file or many files
> Takes advantage of the testing you are already doing!
> Gives you insight for good decisions based on current testing activities
> Depending on the tool, typically easy-to-use and easy-to-deploy
> Minimal impact on budget and schedule
> Can be used for both legacy code and new development
> Still have risk! > Can’t achieve 100% code coverage during system testing
> Error handling
> Defensive programming
> Hard to reach code
> Need to utilize other testing approaches to further reduce risk
Deploying Code Coverage Analysis During System Testing
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > Integrating Code Coverage Analysis with Other Types of Software Testing
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Static Analysis > Compliance with coding standards like MISRA
> MISRA focuses on language restrictions to mitigate reliability faults
> Finding errors in code statically, by analyzing use of memory for instance
> Dynamic Analysis > Unit Testing
> Testing smallest piece of testable software
> Unit is C/C++ source file or Ada package
> Requires generation of test stubs and drivers
> Driver simulates calling unit and stub(s) simulates called unit
> Enables generation of tests to ensure complete code coverage
> Integration Testing
> Logical extension of unit testing
> Allows testing of all units that make up a functional process
> Requires multiple units under test at the same time
> Identifies problems when units are combined
Other Levels of Testing
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Performed by Developers not QA Groups
> Goal is to prove low-level requirements (correctness)
> Secondary goal is to prove completeness of testing
> Challenges of Unit Testing > Unit testing is expensive (extremely labor intensive if done manually)
> Unit testing is tedious for developers
> Unit testing is not easily repeatable
Unit and Integration Testing Facts
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Unit Test Harness
Source Code
Units (.c, .cpp)
Unit(s) under test
Stubbed dependents
Real dependents
Test Driver
Unit and Integration Testing
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Achieving 100% Structural Coverage
System Testing (always) Unit Test (sometimes)
Structural Coverage 100% Achieved
Coverage 30% - 40% Achieved
Driver Stubs
Coverage 60% - 70% Achieved
Units
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Method 1: New Development
Method 2: Legacy Application
Testing Methodologies
Functional Test Unit Test Coverage
Structural Coverage 100% Achieved
Coverage 30% - 40% Achieved
Driver Stubs
Coverage 60% - 70% Achieved
Units
Driver Units Stubs
High-Level Requirements Coverage 100% Low-Level Requirements Functional Requirements
Unit Test Integration Test Functional Test
Driver Units Stubs Full Application
Coverage
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Newly developed code
> Code that has a high complexity > Use static metrics like McCabe Cyclomatic Complexity
> Code that historically has been fragile
> Goal: Be smart about deploying unit test resources! > Instead of blindly just unit testing everything – or nothing!
> Result: Building up a suite of unit and integration tests over time > Supports the other key organizational goal of continuous integration and test
How To Determine When to Unit Test?
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Testing multiple source code configurations > Application deployed on different physical targets
> PowerPC, ARM, x86 ...
> Application built with multiple tool chains
> Green Hills, GNU, VisualStudio …
> Application built with multiple compile options
> Macro values, optimization flags
> Testing multiple source code versions > Production release code
> Beta release code
> R&D code
> Testing across multiple machines or architectures > Test suite may take weeks to run on one machine
> Application may be deployed on multiple operating systems (Windows, Linux)
> Testing what has changed > Interim testing of only those source files that have changed
> Based on file checksums, or change dates
Why Regression Testing (Repeatability) is Important
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > How Automation Improves Software Quality and Compliance to Standards
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Allows stand-alone code coverage to support testing legacy code
> Fully automates harness generation to support low-level unit testing
> Supports and fully automates integration with a wide array of embedded targets
> Integrates with your existing requirements-tracking tool to support requirements traceability
> Provides both static and dynamic analysis
> Integrates with best of breed solutions across development lifecycle
Characteristics of an Ideal Automated Software Test Tool
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Software Development V-Model
Requirements Specification
Architecture
System Design
Module Design
Integration Module Testing
System Testing
Integration Testing
Unit Testing
Coding
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Regression Testing leads to Better Quality!
An ideal tool should… • Provide a full command line interface to support test repeatability and integration with
existing testing infrastructure
• Encourages continuous integration and test repeatability
“79% of all software errors in medical devices
arise from the addition of new functionality and
regression testing issues.” -- FDA
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > Gathering Code Coverage Metrics from an Embedded Target
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Target needs enough resources to support overhead > Executable size growth due to source code instrumentation
> Memory for data capture
> Target needs communication mechanism back to host > Coverage data needs to be captured from memory for analysis on host
> Streaming in real-time or buffering and dumping memory are common options
> Different tools have optimized to varying degrees > Product evaluations are critical to ensure a specific tool will work
Code Coverage Analysis on Embedded Targets
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> > > > VectorCAST Overview
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Stand-alone Code Coverage Analysis
> System / Functional Test Verification > Running your existing tests to see what code is being
executed
> System tests can be executed different ways:
> from scripts, manually through GUI, simulator,…
> Generally performed by a dedicated test group
VectorCAST/Cover
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Unit Testing > Testing smallest piece of testable software
> Unit is C/C++ source file or Ada package
> Requires generation of test stubs and drivers
> Driver simulates calling unit and stub(s) simulates called unit
> Enables generation of tests to ensure complete code coverage
> Integration Testing > Logical extension of unit testing
> Allows testing of all units that make up a functional process
> Requires multiple units under test at the same time
> Identifies problems when units are combined
VectorCAST/C++ and VectorCAST/Ada
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
> Static Source Code Analysis > Uses Industry Standard Lint Analysis Engine for C/C++
> Enforce coding standard compliance > Out-of-the-box MISRA support for C and C++
> Automate code inspection process > Custom compliance rules
> Highlight language insecurities and dangerous usage
> Safety Critical Profile
> Deliver defect prevention > Highlight code complexity metrics
> Visualize code structure to aid code comprehension
VectorCAST/Lint
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
BAE Systems
BAE Systems Developer of the HAWK Advanced Jet Trainer
BAE Systems use VectorCAST for their
continuous build and integration process. The off-the-shelf Tool Qualification Documents
provided by Vector Software, were used to quickly and easily allow BAE Systems to certify
their system to DO-178B.
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Boston Scientific Corporation
Boston Scientific Corporation Developer of the COGNIS® Cardiac Resynchronization Therapy Defibrillator Use VectorCAST/C++ with MC/DC for the most safety critical aspects of their application.
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Bombardier Transportation
Bombardier Transportation Developer of high-speed rail control solutions
Bombardier development teams uses
VectorCAST for testing European Rail Traffic Management System/European Train Control
System (ERTMS/ETCS), Automatic Train Protection (ATP) and Specific Transmission
Modules (STM).
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Hyundai Motor Company
Hyundai Motor Company Automotive Manufacturer Used VectorCAST/C++ and VectorCAST/RSP to perform low-level requirements testing, and structural coverage analysis.
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Endress + Hauser
Endress + Hauser Industrial measurement and automation
VectorCAST automated embedded software
testing and code coverage tools allows E+H to identify software quality issues early in the
development cycle and to provide the necessary IEC 61508 SIL 3 certification evidences.
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
How the VectorCAST Solution is Different
FREE AND OTHER MANUAL TOOLS
NO AUTOMATION
SEMI-AUTOMATED TOOLS
True Embedded Test Automation
> “Easiest to Use” test tool on the market
> Easy continuous integration and test for repeatability
> Integration with best of breed static analysis tools
Full Embedded Software Testing
> Unit and Integration testing
> Code coverage analysis through lifecycle
> Automated regression testing
> Testing on embedded targets and simulators
> Qualification materials for FAA, FDA, and others
World-Class Service and Support
> Our engineers are our support staff
> Local support via phone and email
> Consulting and Technical Training
Benefits of using VectorCAST
> Reduced Time-To-Market
> Easier Certification / Qualification
> Verifiable savings of time and money
© V
ect
or
Soft
war
e,
Inc.
©
Ve
cto
r So
ftw
are
, In
c.
Connect With Us
vectorcast .c o m
Proven Solutions for Reliable Software
Americas 1351 South County Trail Suite 310 East Greenwich, RI 02818 United States of America T: 401 398 7185 F: 401 398 7186 E: [email protected]
EMEA 33 Glasshouse Street Suite 3.08 London W1B 5DG United Kingdom T: +44 203 178 6149 F: +44 20 7022 1651 E: [email protected]
Germany St. Töniser Str 2a 47906 Kempen Germany T: +49 2152 8088808 F: +49 2152 8088888 E: [email protected]
Asia Pacific Rm 403, Building 6 No.88 Daerwen Rd Zhangjiang Hi-tech Park Pudong New Area Shanghai 201203 China T: 21- 3126 8126 F: 21-5132 8526 E: [email protected]