front loaded test approach verhaert

Integrated hardIntegrated hard-- & software testing for first& software testing for first--time right developmenttime right development

Slide 1

ww

w.m

aste

rsin

inno

vatio

n.co

m

Koen Puimege

[email protected]

Frederik Wouters

[email protected]

www.verhaert.com

www.mastersininnovation.com

Commercially confidence – This presentation contains ideas and information which are proprietary of VERHAERT, Masters in Innovation®*, it is given in confidence. You are authorized to open and view the electronic copy of this document and to print a single copy. Otherwise, the material may not in whole or in part be copied, stored electronically or communicated to third parties without prior agreement of VERHAERT, Masters in Innovation®*.

* VERHAERT, Masters in Innovation is a registered trade name of Verhaert New Products & Services NV.

FIRST-TIME RIGHT DEVELOPMENT ... A DIFFERENT TEST APPROACH


Slide 2

ww

w.m

aste

rsin

inno

vatio

n.co

m

AgendaIntroduction

Software & Hardware in the loop

Front loaded test approach of Verhaert

Case study: Proba Satellite


Slide 3

ww

w.m

aste

rsin

inno

vatio

n.co

m

Verhaert mission, strategy and activitiesActivities:

Space & Security•Instrumentation

•Small satellites

•HALE UAV’s

Industry•Marine

•Finance & Retail (ICT)

•Life Science & Care

•Automotive & Transport

•Materials & Machines

Mission:

Innovative product development services

Strategy:

• Act as system developer in key markets

• Offer unique knowledge in industrial product development


Slide 4

ww

w.m

aste

rsin

inno

vatio

n.co

m

Introduction

Good development practices

Focus on early decisions & on specific design alternatives on a sound basis

Analysis and specification freeze before initial physical prototyping

Feedback on design trade off and optimisations


Slide 5

ww

w.m

aste

rsin

inno

vatio

n.co

m

Requirements

SystemSpecifications

DetailSpecifications

D&D

Low-levelTesting

FunctionalTesting

PerformanceTestingIntroduction

Practices of testing

Requirements based testing

Integrated development & test approach

Front loaded testing


Slide 6

ww

w.m

aste

rsin

inno

vatio

n.co

m

IntroductionWhen can a large-scale test methodology be cost effective ?

Cost driven:Critical development schedule High burden rate i.e. expensive equipment (crashtesting)Usability - Man in the loop

Duration

Safety Critical


Slide 7

ww

w.m

aste

rsin

inno

vatio

n.co

m

IntroductionDebugging tools don’t determine what your bug is, but where it is.


Slide 8

ww

w.m

aste

rsin

inno

vatio

n.co

m

SIL - Software in the loopVirtual simulation of embedded system on host

MicroprocessorApplication

Used for debugging and functional verificationAlow debugging before HW is availableCode validation on simulated hardware


Slide 9

ww

w.m

aste

rsin

inno

vatio

n.co

m

SIL - AdvantagesProvide feedback & selection for HW design

Enable testing/simulation of off-nominal behavior (error conditions, fault recovery)

Concurrent HW & SW developmentNo HW resource conflict when developing/testing with >1 person

Cross validation of environment (interfaces, payloads) through behavioral models


Slide 10

ww

w.m

aste

rsin

inno

vatio

n.co

m

SIL - DisadvantagesDisadvantage: Accuracy of model

Need for code validation on real hardware (HIL)

No Possibility to test system in real-timeVerification SW timing and performance


Slide 11

ww

w.m

aste

rsin

inno

vatio

n.co

m

HIL - Hardware in the loopHighly realistic simulation of equipment in an operational real time virtual environment.

Replacing plant interaction by RT simulations.


Slide 12

ww

w.m

aste

rsin

inno

vatio

n.co

m

HIL - AdvantagesTesting of extreme conditions upto the limits

Performance, Real time behaviour, Environmental...

Anomaly testing & Isolate deficiencies

Testing with/without subsystems & environment

Reducing costEquipment & manpower Time to market


Slide 13

ww

w.m

aste

rsin

inno

vatio

n.co

m

PlatformHIL & Complex Systems

BoardComputer

PayloadComputer

Reactionwheels

PowerControlSystem

StarTracker

AOCS i/f

Payload#2

Payload#3

Payload#4

Magneto-torquers

Magneto-meters

GPS

Payload#1

Batteries SolarArray

Multi-processor

Multi-unit

Redundant system

Many interfaces


Slide 14

ww

w.m

aste

rsin

inno

vatio

n.co

m

HIL - DisadvantagesDifficulty to emulate REAL environment (crash, satellite launch, ...)

Doesn’t debug your embedded system, No breakpoint or pause-on-commandNo info of erroneously codeNo info on bug location


Slide 15

ww

w.m

aste

rsin

inno

vatio

n.co

m

Front loaded Test Approach

Requirements based testing

Software Validation Facility (SIL)

Integrated Test Environment (Extended HIL)


Slide 16

ww

w.m

aste

rsin

inno

vatio

n.co

m

Requirements Based TestingIdentify Requirements related to eSW

Check verification method (Analysis / Design / Test)

Provide list of tests

Identify which tests to do in which project phase(s) and on which level

Simulation / EmulationUnit test / System test

Result is test matrix used forTest progressTraceability

«Test plan must be established from the beginning together with the architectural design»


Slide 17

ww

w.m

aste

rsin

inno

vatio

n.co

m

TEST

-ID

TEST

-ID S

UFF

IX

TEST DEFINITIONPROFILE NAME TEST ITEMS C

OR

E D

ESIG

N(v

erifi

ed b

y th

e de

sign

er)

PASS

/FA

IL

MA

TUR

E C

OR

E

PASS

/FA

IL

VHD

L M

OD

ULE

PASS

/FA

IL

VHD

L SY

STEM

PASS

/FA

IL

ELEC

TRIC

AL

TEST

S

PASS

/FA

IL

SUB

SYST

EM T

ESTS

(test

sin

gle

func

tions

of a

boa

rd)

PASS

/FA

IL

EQ

M V

HD

L TE

ST R

ESU

LTS

(PR

IOR

TO

EQ

M B

UR

N)

SYST

EM T

ESTS

(test

boa

rd fu

nctio

ns a

t sys

tem

leve

l)

PASS

/FA

IL

SYST

EM T

ESTS

(func

tiona

l sys

tem

per

form

ance

by

para

llel e

xecu

tion

of s

elec

ted

test

s)

PASS

/FA

IL

TTM-50 01 TME functional tests registers power-up defaults X PASS X PASS X (*)02 register write/readback X PASS X PASS X (*)03 VC7 generation of idle data / VC0 OBET

time source packet insertion X PASS X PASS X PASS04 VC0 HKM data integrity and handshaking X PASS X PASS X PASS05 VC3 VC4 External PacketWire data

initegrity of packetized data X PASS X PASS X PASS06 VC3 VC4 External PacketWire data

initegrity of non-packetized data X PASS X PASS X (*)07 VC1 Internal PacketWire data integrity of

non-packetized-data X PASS X PASS08 VC2 Internal PacketWire data integrity of

non-packetized-data X PASS X PASS09 VC1 VC2 Internal PacketWire data

integrity of non-packetized-data X PASS X PASS X (*)10 VC1 VC2-Internal PacketWire data

integrity of packetized data X PASS X PASS X PASS X PASS11 Time-source-packet X PASS X PASS X (*)12 TX ringbuffer empty interrupt X PASS X (*)13 Core reset via global reset register X PASS X PASS X PASS X14 VC1 VC2 channel reset X PASS X PASS15 FECW function X PASS X PASS X PASS16 Reed Solomon function X PASS X PASS X (*)17 Convolutional function X PASS X PASS X (*)18 FECW, RS and CONV enabled X PASS X PASS X (*)19 Mark-NRZ-M, SPLIT-PHASE, PSEUDO-

RANDOM functions X PASS X PASS X (*)20 VC1 VC2-Internal-PacketWire data

integrity of packetized data (PD-flag=0) X PASS X PASS21 CLCW-bit-16-17 option X PASS X PASS22 VC bandwidth allocation (dynamic based) X PASS X PASS X FAIL23 VC bandwidth allocation (priority based) X PASS X PASS X (*)

6 7

VALIDATION LEVELSVHDL TEST BENCH AVIONICS TEST BENCH (SCOS)

0 1 2 3 4 5


Slide 18

ww

w.m

aste

rsin

inno

vatio

n.co

m

SVF - Key dataSoftware in the loop system

TSIM - Leon Sparc simulation model used from Gaisler Research

Behavioral models developed in-house in C

eSWWritten in CSVF fully transparant to applicationBinary code runs on TSIM


Slide 19

ww

w.m

aste

rsin

inno

vatio

n.co

m

Environmental ModelDUT Computer Model

Software Validation Facility (SVF)

Processor simulation(TSIM)

eSW (C)

I/F Behavioral model (C)

I/F

I/F

Behavioral model (C)

Behavioral model (C)

Stimuli

Response


Slide 20

ww

w.m

aste

rsin

inno

vatio

n.co

m


SOFTWARESOFTWARE

IL

BOARDBOARD

PROCESSOR

BOARDBOARD

FPGA

IPCORE

IPCORE

IPCORE

IPCORE

HWDRIVER

HWDRIVER

HWDRIVER

HWDRIVER

MEMORY

PROCESSOR

TEST APPLICATION

DEVICEDRIVER

DEVICEDRIVER

DEVICEDRIVER

DEVICEDRIVER

BINAIRECODE

No model Required

Behavioural model (simulation)

Synthesisable model (real)Executable code (binary)


Slide 21

ww

w.m

aste

rsin

inno

vatio

n.co

m


SOFTWARESOFTWARE

BOARDBOARD

PROCESSOR

BOARDBOARD

FPGA

IPCORE

IPCORE

IPCORE

IPCORE

HWDRIVER

HWDRIVER

HWDRIVER

HWDRIVER

MEMORY

FUNCTIONAL APPLICATION

DEVICEDRIVER

DEVICEDRIVER

DEVICEDRIVER

DEVICEDRIVER

BINARYCODE

No model Required

Behavioural model (simulation)



Slide 22

ww

w.m

aste

rsin

inno

vatio

n.co

m


SOFTWARESOFTWARE

BOARDBOARD BOARDBOARD

FPGA

IPCORE

IPCORE

IPCORE

IPCORE

HWDRIVER

HWDRIVER

HWDRIVER

HWDRIVER

MEMORY

PROCESSOR

FUNCTIONAL/TEST APPLICATION

DEVICEDRIVER

DEVICEDRIVER

DEVICEDRIVER

DEVICEDRIVER

BINARYCODE

No model Required

Behavioural model (simulatie)


Hardware


Slide 23

ww

w.m

aste

rsin

inno

vatio

n.co

m Test Bench (TB)DUT computer

Leon SparcRTEMS

eSW (C)

2x embeddedcPCI PIIIRT Linux

Behavioralmodel (C)

I/F

I/FBehavioralmodel (C)Behavioralmodel (C)

Stimuli

LAN

I/F

Test IF Bridge (TIFB)

DebugI/F

PC-104 486RT Linux

Monitoring & Control

Pentium IV PCSuSe Linux

DCA (C)

Response

Integrated Test Environment (ITE)


Slide 24

ww

w.m

aste

rsin

inno

vatio

n.co

m ITE features integrated debugging functionalities

DUT , HIL & DUT – HIL interactionI.e. Specific test software on DUT

Custom interface cards to validate off–nominal behaviour;

Baud rate deviation, ...

Space specific HW cardsCustom HWCustom Drivers for test software

Re-use of SVF SW

ITE - Differences with HIL systems?


Slide 25

ww

w.m

aste

rsin

inno

vatio

n.co

m Functional and performance testing

ITE ComponentsSimulates HW environment of sub-system or whole satelliteSame/similar behavioral models can be used through hardware abstractionDebug interfaceProvides access to embedded system via Ethernet

Monitoring & ControlUser i/fSet of tools (compile, run, debug, test, log, scripting, ...)

ITE – Key data

DUT (Proba Satellite)Real hardware (Leon Sparc,

I/F, PCB)

RTEMS

Application software

Device Control Application


Slide 26

ww

w.m

aste

rsin

inno

vatio

n.co

m

Device Control Application (DCA)

Debug interface

Scripting

Monitoring & Control

Transparency

Independent testing

ITE – Techniques and Tools


Slide 27

ww

w.m

aste

rsin

inno

vatio

n.co

m

ITE - Device Control Application (DCA)Dedicated applicationSeparate from application SWAimed to test and validate all sub-units linked to Board computer (payloads)Implemented according to agreed ICD (Interface Control Document)Enables black-box testing of sub-units from within ITESupports integrated testing of whole systemInherent cross-validation

Sub-unit made by sub-contractor/other teamDCA i/f made by system integrator

«The DCA facilitates integration testing on system level»


Slide 28

ww

w.m

aste

rsin

inno

vatio

n.co

m

ITE - Debug InterfaceSupports debugging and testing of the HW

Debug interface of Board computer allowsdeep control of whole system,not only processor

Two front-end applications usedGDB (open source)Dedicated

«An adequate debug interface is imperative to test the eSW application»


Slide 29

ww

w.m

aste

rsin

inno

vatio

n.co

m

ITE - ScriptingAutomatic/repeated running of tests

Management of ITE

Top-level

Prevents human errors

Same tests must be run in different project phases (Engineering/Qualification/Flight Model)

Advantage of Tcl scripting languageEasy to read/understandInterpreted -> flexible (no compilation)


Slide 30

ww

w.m

aste

rsin

inno

vatio

n.co

m

ITE - Monitoring & ControlVisual representation of system state

Change system state

LAN basedNo keyboard/display i/f on Board computerEnables remote testing and debugging Good support for TCP/IP in most operating systemsReliable connection

Platform independent GUI (Qt - Trolltech)


Slide 31

ww

w.m

aste

rsin

inno

vatio

n.co

m

ITE - TransparencyLayered SW model

Transparency towardsHW i/f (Hardware abstraction)Host platform (x86, Leon Sparc)Operating system

AdvantagesRe-use of SWTest SW in appropriate environmentEarly development/testing of SW

«Transparency leads to more effective eSW development and testing»

Hardware

POSIX compliant OS + Device drivers

POSIX

Application


Slide 32

ww

w.m

aste

rsin

inno

vatio

n.co

m

ITE - Independent TestingeSW Developer

Focused on applicationHighly involved, dedicated to implementationBiasing

eSW TesterFocused on requirements/functionalityDoes not know implementation

AdvantagesBugs are found earlierMore bugs are found

«The developer should not validate his own code»


Slide 33

ww

w.m

aste

rsin

inno

vatio

n.co

m

ConclusionsIntegrated testing, an essential part of eSW development

Good testplan, testmethodology & enhanced testtools will save your time

Simulation enables early testing and provides valuable input for the development

PROBA 1 mission successfulLaunched 22.10.2001Commissioned in spacePlanned to operate for 2 years & still operational


Slide 34

ww

w.m

aste

rsin

inno

vatio

n.co

m

PRoject for On-Board AutonomyPlatform for small satellites

Demonstration ofOperational autonomyNew technologies (batteries, positioning)Payloads

Create wider path for use of small satellitesAutonomy (low cost of ownership)Powerful on-board computer (advanced data processing)Advanced Attitude & Orbit Control System (AOCS)


Slide 35

ww

w.m

aste

rsin

inno

vatio

n.co

m

PROBA – MissionMain task

PROBA 1: Observation of the EarthPROBA 2: Observation of the Sun

SchedulePROBA 1: launched 22.10.2001PROBA 2: scheduled for mid 2007PROBA 3: on the roadmap

Low earth orbit (LEO) – 600km


Slide 36

ww

w.m

aste

rsin

inno

vatio

n.co

m

PROBA – Key dataMicro satellite

120kg, 80cm x 60cm x 60cm40W to 120W power

Modular & FlexibleLauncher interfaceAccommodation of off-the-shelf payloadsSoftware modularity

Powerful processors: ERC32,LEON SPARC

OS : VxWorks, RTEMS


Slide 37

ww

w.m

aste

rsin

inno

vatio

n.co

m


Slide 38

ww

w.m

aste

rsin

inno

vatio

n.co

m

Verhaert New Products & Services nvHogenakkerhoekstraat 219150 KruibekeBelgiumTel +32 (0)3 250 19 00Fax +32 (0)3 254 10 [email protected]

www.mastersininnovation.com

front loaded test approach verhaert

Technology