webinar: test engineering and test data management

V1.0 | 2020-05-11

PREEvision v9.0 Service Pack 10

Webinar: Test Engineering and Test Data Management

2

u Basics - Overview

Test Project Preparation (Test Planning)

Test Specification

Test Implementation

Setting up the Test Project (Test Planning)

Preparing the Test Execution (Test Planning)

Performing the Test Execution

Test Analysis and Reporting

Toolchain: PREEvision - vTESTstudio - CANoe

Agenda

3

Test Engineering & Test Management Workflow

Basics - Overview

4

TE/TM Data Model (focus: manual testing)

Basics - Overview

5

Basics - Overview

u Test Project Preparation (Test Planning)

Test Specification

Test Implementation






Agenda

6

1) Preparation of the Test Project

u Recommended Perspective:

u Test Engineering and Test Management

u Model View:

u Create these artifacts below a Product Line:

u Test container

u Test Data> Test Data Package

> Test Specification

u Test Execution Package> Test Project

u Test Environment Package> Test Environment(s)

> e.g. Test vehicles, test benches, HIL, FMU…

u Test Object Package> Device under Test (ECUs, Subsystems, …)

> Subject under Test (SW-Versions)

Preparation (1/2)


7

2) Prepare test team

u Create Resources

u Administration section

u Optional: assign test team to the Test Project

u Precondition: Test Project exists> 1. Select a Test Project in the Model View.

> 2. in PE, open the Test Team page.

> 3. Drag the required Resources from the Artifact Picker and drop them into the Tester table.

u Advantage: When creating new Test Execution Increments, Test Tasks are automatically created> One Task created for each assigned Resource

Preparation (2/2)


8

Basics - Overview


u Test Specification

Test Implementation






Agenda

9

u Creating Test Specification artifacts:> Test Headings, Test Information

> Test Items, Test Item Steps (init, run, shutdown)

> Test Conditions (Pre/Post-Conditions)

> Test Specification Library (Test Items, Test Conditions)

u Linking to REQ layer (traceability)

u Column Req/CF Links

u Linking between Test Artifacts

u Linking to Test Specification Library> E.g. Link Test Conditions with Test Items

u Linking to test design demands> E.g. Link between Test Information and Test

Item (column Test Links)

u arranging artifacts

u Change order, change hierarchy

u Add comments

Test Specification Editor

Test Specification

10

Test Item:

u Test Items describes the test focus/test idea

u Attributes:

u Pre/Postcondition, Description…

u Later, on Test Execution level:

u Same verdict for precondition, action,

postcondition

u In case of failed:> No differentiation whether precondition or action

failed…

Test Items vs. Test items steps

Test Specification

Test item steps:

u Devide tests into test steps > ☺ verdict for each test step

u 3 roles available: Init/Run/Shutdown

u Example of a Test Item with Steps:> Init: preparation of the test

> initialization of test platform

> variant coding of device under test

> Run: the actual test…

> Only Run steps relevant for testing of REQ

> Shutdown: Cleaning up

> reset of variant coding

u Traceability: only Run Step linked to REQ

u ☺ evaluate verdict depending on test focus

> Especially important for test automation

u E.g. failed at “Init” step> verdict not relevant for test focus / REQ

> “Run” step can be skipped

11

Tests must be linked with REQ

u Traceability, Test coverage analysis

How to set links:

u In TSE, select test artifact to be linked

u e.g. Test Item

u Drag REQ artifact from Artifact Picker

u Drop it onto a test item / Test Item Step

u Into column Req/CF Links

u How to open Test Specification Editor:

u In Model View, double click on Test Specification artifact

u Or open Test Explorer, category Test Specification> Double click on Product Line

Traceability: Linking Tests with Requirements/Customer Features

Test Specification

12

Basics - Overview


Test Specification

u Test Implementation






Agenda

13

u Test Implementation:

u Derived from Test Specification

u Technical refinement of Test Specification

u Example: > Test Specification: „engine running“

> Test Implementation: „CAN signal: IGN_On“

u Specific „implementation“ for:

u Different test platforms> Test vehicles

> Test benches (FMU, HIL)

u Variant specific implementation> ECU variants

> Supplier specific

u Implementation for manual or automatic testing:

u Manual Test Sequence

u Automatic Test Script

Meaning of Test Implementation

Test Implementation

14

Automatically create manual Test Sequences

u By manual TC synthesis:

u On Test Specification artifact

u Context menu ➔ Refactoring ➔ Manual Test

Sequence Synthesis

u Test Headings result into Test Sequences

u New creation or update of existing Manual Test Sequences

Manual Test Sequence Synthesis

Test Implementation

Synthesis

Test SpecificationTest Implementation

15

3 Editors available:

u Minimal View

u Rough Overview only – Test Data Packages

u Test Implementation Editor

u Manual Test Sequence Editor

How to open:

u In Test Explorer ➔ category Test Implementation

Test Implementation Layer Editors

Test Implementation

16

u Overview of implementation of Test Specification

u No Test Steps listed

u Used for creation of…

u A) Manual Test Sequences (for manual tests)

u B) Test Scripts (for automatic tests)

u Alternatively…

u A) use Manual Test Sequence Synthesis

u B) import Test Scripts from Test Design Tool (vTESTstudio)

u Assigning Test Environment

Test Implementation Editor

Test Implementation

17

u Used to refine Manual Test Sequences

u Create Manual Test Cases

u Assigning Test Items to Manual Test Cases> Alternative: Manual Test Sequence Synthesis

u Description of Test Steps: Precondition, Action, Expected Result

u Manual Test Sequence Synthesis:

u Test Step details from Test Specification layer copied to Test Implementation layer> Action, …

u Technical implementation defined in Manual Test Sequence Editor

u Note: Manual Test Sequence Synthesis Update> Update overwrites changes in Action, Expected…

> Use Action Comment, Expected Comment fields for implementation description

Manual Test Sequence Editor

Test Implementation

18

Basics - Overview


Test Specification

Test Implementation

u Setting up the Test Project (Test Planning)





Agenda

19

Associate test data with the Test Project.

u How to open Test Project Editor:

u In Model View ➔ select a Test Execution Package

u Open with -> Test Project Editor> All Test Projects of the Execution Package are listed

u Set Scope of Requirements Package and Test Data

u Assign by Drag/Drop into corresponding columns

u Pre-selection for Artifact Picker tables used in later steps…

u Calculation basis for Analysis results

u Define Start/End Date

Test Project


20

Basics - Overview


Test Specification

Test Implementation


u Preparing the Test Execution (Test Planning)




Agenda

21

Set up Test Execution Increments

u E.g.: for each SW build one Increment

Open the Test Increment Editor:

u In the Test Explorer ➔ click on the ”Test

Execution” category

u Select a Test Project

u Select Tab “Test Increment Editor”

How to add a Test Increment:

u Context menu => New

u Select Subject Under Test

u Defining the test scope:

u Column “Planned Increment Content”: drag/drop from Artifact Picker> Test Specification scope

Creating Test Execution Increments


22

Precondition: Resources assigned as test team

u In Test Explorer ➔ category Test Execution

u Click on an existing Test Execution Increment

u Inside Category list

u Test Tasks are created as children of a Test Execution Increment

u When creating new Test Execution Increments, Test Tasks are automatically created

u One Task created for each assigned Resource

u “Responsible” automatically assigned to the Resource

> See Test Task Editor

Assigning Test Tasks - Test Tasks Editor


23

2 ways of planning the test execution

u Test Task Editor: Column “Test Implementation to Execute”

u Case a) Test manager assigns tests to task

u Case b) “Picking Method”

u Each tester selects Test Sequences on his/her own> Drag/drop from Artifact Picker

> Artifact Picker offers only unassigned tests

Assigning Tests to be Executed


24

Picking Test Cases (not Test Sequences)

u From Artifact Picker

u Drop into “Test Volume” column

Test Sequences: Complete or in Parts:

u In Test Task Editor, “Test Implementation to Execute” column

u click on the […] button of a Manual Test Sequence

u Completely or in parts

u In parts: Select Test Cases for test execution

Test Volume


25

Basics - Overview


Test Specification

Test Implementation



u Performing the Test Execution



Agenda

26

How to open Test Execution Editor:

u in Test Explorer ➔ Category “Test Execution”

u Tab “Test Task Editor”

u Click on 2nd column ➔ open “Test Execution

Editor”

u Add Test Step Run for manual test

u Icon in 2nd column

u if not already done…

u Verdicts can only be added to Test Step Runs

u Enter test results into Column “Verdicts”

u Verdicts entered on “lowest level”> A) test steps exist ➔ verdicts on test steps level

> B) no Test steps ➔ verdicts on test case run level

> See greyed / white cells

u Double click opens verdicts drop down list

Test Execution / Setting verdicts


27

u In the editor, verdicts are aggregated from the lowest to the next level

u Verdicts are liable to a hierarchy

u the lowest verdict is aggregated

u verdict Failed lower than Passed

u ➔Failed “wins”

Verdict Logic


28

Basics - Overview


Test Specification

Test Implementation




u Test Analysis and Reporting


Agenda

29

u coverage of REQ by test items / Test Item Steps

How to open:

u A) main menu ➔ Test ➔ Open Test Analyzer |

Specification Analyzer

u Or B) Select a Requirement Package

u context menu ➔ select Open with | Test Specification

Analyzer

Coverage:

u REQ covered ➔ blue bullet

u REQ not covered ➔ white bullet

u Coverage information is aggregated up to all parent artifacts

u Aggregated Covered ➔ blue bar graph

u Aggregated not covered ➔ number on white

background

Test Specification Analyzer


30

u Coverage of Test Specification by Test Sequences / Test Scripts

How to open:

u A) main menu ➔ Test | Open Test Analyzer |

Implementation Analyzer

u Or B) Select e.g. Test Data artifact ➔ context menu ➔ Open with… | Test Implementation

Analyzer

Test Implementation Analyzer


31

u Test execution coverage

u Generate Report

How to open:

u A) main menu ➔ Test | Open Test Analyzer |

Execution Analyzer | flat or structured.

u Or B) Select a Test Project, open the context menu and select Open with | …

u Test Project Execution Analyzer (flat) or Test Project Execution Analyzer (structured)

Test Project Execution Analyzer


32

Perspective: Cockpit Test Engineering / Test Management


Select Test Project

different calculation

basis

33

Basics - Overview


Test Specification

Test Implementation





u Toolchain: PREEvision - vTESTstudio - CANoe

Agenda

34

Test Engineering & Test Management Workflow


35

u PREEvision:

u Create Test Specification

u Linking to Requirements

u vTESTstudio:

u Test Implementation Design

u PREEvision:

u Planning> Test Project, Increments, Test Task

u CANoe:

u Test Execution

u PREEvision:

u Analysis, Reporting

Workflow Overview


36

u In Test Explorer, Category Test Specification

u Select Test Specification to export

u Open tab Test Specification Editor

u Click on toolbar button Export to Test Design Tool…> .vti-tso file

u Currently, only vTESTstudio export format supported

PREEvision: Export Test Items to vTESTstudio


37

u Create Test Cases

u Import Test Items from PV

u .vti-tso file

u Traceability

u Map vTS Test Cases to PV Test Items

u 2 „exports“ for toolchain:

u 1) for PREEvision:

u Export Traceability Matrix

u .vtc-tso file

u 2) for CANoe:

u Compile Test Cases

u .vtuexe file

vTESTstudio: Actions in vTESTstudio


38

In Test Implementation Editor

u Toolbar button: Import from Test Design Tool

u Traceability Matrix exported from vTESTstudio(.vtc-tso)

u Added artifacts after import:

u Test Scripts > Automatic Test Case

PREEvision: Import into Test Implementation Editor


u Optional step

u Toolchain can be used without import of Test Implementation

39

u Import compiled test cases from vTESTstudio

u .vtuexe

u Execute…

u Export CANoe Test Report

u .xml file

Actions in CANoe


40

u Open Test Task editor

u Select Test Task

u Toolbar button: Import from Test Execution Tool…

u Load Test Protocol(s) ➔Select CANoe report

u CANoe report contains traceability information

u Therefore import process offers to import Test Scripts:> Drop-down List

> Select Plan Test Script + import Test Run

PREEvision: Import CANoe Test Report


41 © 2019. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.0 | 2020-05-11

Author:Thomas TeegenVector Germany

For more information about Vectorand our products please visit

www.vector.com

http://www.vector.com/

webinar: test engineering and test data management

Documents