software testing and analysis tools for productivity and quality

Software Testingand Analysis Tools

for Productivity and Quality

2

SummarySummary

● Verifysoft and its partners

● Cost of software errors● Testing tools in the software development process● Code complexity / Static Analysis.● Code Coverage● Functional Test

3

Verifysoft Technology GmbHVerifysoft Technology GmbH● Verifysoft Technology GmbH

● Created in June 2003 ● Offenburg (Germany)● 200+ customers in Europe (2009)

● Distributor and developper of Software testing tools● distribution, consulting and support

4

ActivitiesActivities

● Tools ● Seminars

5

our pour partnersartners

● Testwell, Tampere (Finland)

● Conformiq, Saratoga (USA)

● Coverity, San Francisco (USA)

6

our customersour customers

Main sectors of application:Development of critical software:

– Aerospace– Automotive– Medical– Nuclear

7

ReferencesReferences

and many more...

8

Cost of software errorsCost of software errors● The losses due to errors of programming

increase every year

Estimation : ???

9

Cost of software errorsCost of software errors

● Software bugs cost in Europe100-150.000.000.000 Euros/year

→ Tendancy increasing !

The complexity of Software doubles every 18 months.

To face it: the "know-how" of developers!!!

(Les Hatton, Kingston University London)

10

Cost of software errorsCost of software errors● Software bugs cost up to

14.000.000 Euros per company and per year

50% of the companies found in the first year of utilisation up to 10 critical errors.

IDC-Survey in 2008 „Improving Software Quality to Drive Business Agility“, within American companies with 250 -10.000 employees www.idc.com

11

Cost of software errorsCost of software errors

40-60% of the global budget is usually used for test and correction

The purpose of testing is to obtain a product „without errors“

12

Software testSoftware test

Testing is necessary for obtaining certifications

IEC 61508 EN 50128 (Railway) IEC 62304 (Medical) EN 62138 (Nuclear) DO-178B (Aerospace)

13

Software testSoftware test

Test: is essential But ... expensive … repetitive activity

→ Test-Automatisation → use of testing tools !

14

Software development processSoftware development processRequirements

Analysis

SystemDesign

ArchitectureDesign

Detailed Specifications

Coding

SoftwareValidation

System Testing

IntegratingTesting

Unit Testing

Black box tests

White box testsTestwell CMT++

CMTJavaComplexity

Measures

Testwell CTA++Unittest

Testwell CTC++Code Coverage

Conformiq Qtronic

Functional Test

15


Analysis

SystemDesign

ArchitectureDesign


Coding

SoftwareValidation

System Testing

IntegratingTesting

Unit Testing

Black box tests


CMTJavaComplexity

Measures



Conformiq Qtronic

Functional Test

16

Testwell CMT++ / CMTJavaTestwell CMT++ / CMTJava

Testwell CMT++ Testwell CMTJava

Code Complexity

Measures Tools

for C/C++ and Java

Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)

17



18

Testwell CMT++ / CMTJavaTestwell CMT++ / CMTJava● Why analysing the code complexity?

● Code complexity correlates with the defect rate and robustness of the application

● Complex code is difficult to test → more errors in the final application

● Complex code is difficult to maintain● Unnecessary complex code is often the reason for bad

code quality and erroneous programs.


19


Testwell CMT++ and Testwell CMTJava analyses your applications for the following metrics:

● Lines-of-code (LOC) metrics● Halstead's metrics● McCabe cyclomatic number v(G) ● Maintainability Index


20


Lines-of-code (LOC) metrics:● LOCphy

number of physical lines● LOCpro

number of lines with program code● LOCbl

number of blanc lines● LOCcom

number of lines with comments


21


Lines-of-code (LOC) metrics:

● Recommanded standard for fonctions: between 4 and 40 program lines

A function definition contains at least a prototype, one line of code, and a pair of braces, which makes 4 lines. A function longer than 40 program lines probably implements many functions.


22


Lines-of-code (LOC) metrics:

● Recommanded standard for files: between 4 and 400 program lines

The smallest entity that may reasonably occupy a whole source file is a function, and the minimum length of a function is 4 lines.

Files longer than 400 program lines (10..40 functions) are usually too long to be understood as a whole.

Comments : 30 to 75%


23


McCabe Cyclomatic Number v(G) describes the complexity of thecontrol flow of a program.

The greater the cyclomatic number is the more execution paths there are through the function, and the harder it is to understand.


24


McCabe Cyclomatic Number v(G)

● recommanded standard for fonctions: <15If a function has a cyclomatic number of 15, there are at least 15 (but probably more) execution paths through it.More than 15 paths are hard to identify and test.

● recommanded standard for files: <100


25

Testwell CMT++ / CMTJavaTestwell CMT++ / CMTJavaHalstead's metrics

● B estimated number of bugs

● D difficulty level, error proneness

● E effort to implement

● L program level (abstraction level of the program)

● N program length

● N1 number of operators

● N2 numbre of operands

● n vocabulary size (unique operators + unique operands)

● n1 number of unique operators

● n2 number of unique operands

● T implementation time / time to understand

● V volume: size of the implementation of an algorithm


26


Maintainability Index (MI)

indicates when it becomes cheaper and less risky to rewrite the code instead to change it.

● 85 and more good maintainability● 65-85 moderate maintainability● < 65 difficult to maintain

→ with really bad pieces of code (big, uncommented, unstructured) the MI value can be even negative


27



28


Development ViewManagement View

Controlling View

Pdf-reports

29


Analysis

SystemDesign

ArchitectureDesign


Coding

SoftwareValidation

System Testing

IntegratingTesting

Unit Testing

Black box tests


CMTJavaComplexity

Measures



Conformiq Qtronic

Functional Test

30

Testwell CTC++ Test Coverage Testwell CTC++ Test Coverage AnalyserAnalyser


Testwell CTC++ Test Coverage Analyser

for C/C++

CTC add-on for Java and C#

31Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)

Code coverage

shows the parts of the code● executed / not executed ● tested / not tested



Why measuring the code coverage?● helps to write better (more adapted) tests/test cases

● helps to avoid that you spend time on writing redundant test cases

● you know when you can stop testing

● you can proof to your customers that your code is tested according to their requirements

● you can be sure that your outsourcing/development partner delivers quality according to your requirements

● ensures high quality with high code coverage

● helps to find "dead code"

● required to obtain certifications.



Testwell CTC++ for all coverage levels:

● Function Coverage● Decision Coverage● Condition/Branch Coverage● Modified Condition/Decision Coverage (MC/DC)● Multicondition Coverage (MCC)



Testwell CTC++ can be used to obtain certifications in automotive, avionics and medical industries

i.e. DO-178B - all levels: A, B, C - of the Federal Aviation Administration, FAA or for EN 61508



Testwell CTC++ is very easy to use● no need of code modifications● support of existing makefiles● GUI integration in several IDEs

– Microsoft Visual Studio

– WindRiver Tornado

– Borland C++ 5.02

– Metrowerks CodeWarrior

– Eclipse

– ...



Reports in Text, XML or HTML● shows the untested code parts ● shows how many time each

code part has been executed ● different coverage reports

– Summary-Levels– Untested Code– Execution Profile Listing



Testwell CTC++ ideal for embedded targets

● very low instrumentation overhead● performs code coverage in all targets

→ “host target add-on” is provided in source code→ can be easily adapted to new targets

● works even with smallest targets and microcontrollers● works with all compilers / cross-compilers



„CTC++ Add-on for Java and C#“

extension of Testwell CTC++ for Java and C#

→ You only need one code coverage tool for C, C++, Java, C#, ...


39


Analysis

SystemDesign

ArchitectureDesign


Coding

SoftwareValidation

System Testing

IntegratingTesting

Unit Testing

Black box tests


CMTJavaComplexity

Measures



Conformiq Qtronic

Functional Test

40Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)

Model-Based Automated Test Model-Based Automated Test DesignDesign

for functional testsfor functional tests

(black box tests)

for software and systems

Conformiq QtronicConformiq Qtronic


Manual tests takes time...and leads to risks

incorrect tests

missed tests

redundant tests

maintenance of tests



Our solution: Automated Test Design

→ model driven testing, model based testing, specification based testing,

specification driven testing,

...


43

Automatic test generation and execution based on your design models

instead of writting the tests manually

Manual vs. AutomaticManual vs. Automatic

Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)

X


The tool automates the design of functional tests (black box tests) for software and systems

Model Driven TestingModel Driven Testing


Manual vs. AutomaticManual vs. Automatic

11

QQQQSystem Model

Test Plan

Manual test script

Automatic test generation Script execution

Script execution


Model Driven TestingModel Driven Testing

Benefits:Benefits:

● enforces the quality of requirements enforces the quality of requirements documentation documentation

● creates a common platform for designers creates a common platform for designers and testersand testers

● self documentedself documented


● Textually in Java (with elements in C#)→ „Qtronic Modelling Language“ (QML)

● Grafically in UML State Charts (optional)

● The model can be created with:– Text editor („Java“)– Qtronic Modeller (UML State Charts)– or Third Party Modeling (UML) tools

ModellingModelling


Process:Creation of the model Import of the model Selection of the test coverage requirements and formats for the test scripts. Automatic test generation with Conformiq Qtronic

Test execution in your environment.

Test generationTest generation


Formats for the test scripts generation:

Python TCL TTCN-3C, C++ Visual Basic JavaJunit Perl ExcelHTML Word Shell Scripts

Test generationTest generation


Test scripts generationTest scripts generation



Test scripts Generation progress

List of test scripts





Requirement Tracebilityof the test scripts





Requirement-TracebilityOf the test scripts

Message Sequence Chartfor a test script

Test Step Viewwith detailled information

about contents





Requirement Tracebilityof the test scriptsMessage Sequence Chart

for a test script Test Step Viewwith detailled information

about contents

Execution Trace View:shows the execution's

trace in the model



Model

Test scripts generation


Test scripts executionin your existing evironment(manually or automatically)

Test scripts executionTest scripts execution

Model

Test ScriptsTest Execution

PlatformSystem Under Test

Test Harnes s

Reports

Model

57

ReviewReview

→ Cost of software errors

→ Testing tools in the software development process

→ Code Complexity / Static Analysis

→ Code Coverage→ Automatic Test Generator

58

more Informationmore Information

Tel: +49 781 63 92 [email protected]

Thank you!Thank you!

software testing and analysis tools for productivity and quality

Documents

cmtjava testwell cmt

partners testwell

cmtjava lines

products of testwell

comments20 testwell

cmtjava17 testwell ctc

complexity of software

software test testing