automated testing environment
DESCRIPTION
Automated Testing Environment. Concepts required for testing embedded systems adopted in this course (quizzes, assignments and laboratories). To be tackled today. Why test, and what kinds of tests are there? Difference between TLD – Test Last Development - PowerPoint PPT PresentationTRANSCRIPT
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
1
Automated Testing Environment
Concepts required for testing embedded systems adopted in this course
(quizzes, assignments and laboratories)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
2 / 28
To be tackled today
Why test, and what kinds of tests are there? Difference between
TLD – Test Last Development TDD – Test Driven Development
(Test First Development) How do you add tests? More details on the testing process E-TDD and
the testing toll E-UNIT Other kinds of available tests – time and access Design of custom TESTs and ASSERTS
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
3 / 28
Why test?
Unit test I AM CURRENTLY BUILDING A NEW FUNCTION –
does it work the way I expect? Testing my and my partner’s understanding of the
Design WE ARE DESIGNING A NEW PRODUCT
-- If we can’t explain (to ourselves via a test) the product ideas – how do we know when product is working?
Regression testing MY PARTNER ADDED A NEW FEATURE to a product –
how do I know it did not “break something” I wrote? System testing
PROOF TO THE CUSTOMER that the product (Lab. or assignment) works the way that was requested.
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
4 / 28
What are the problems with this sort of test?
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
5 / 28
VisualDSP development environment(IDDE)
DebuggingInterface
“JTAG” boundary scan operation
Processor
Peripherals
BF533 Evaluation Board
VisualDSP++
program
Debugging Interface
HPPCIce
Lab. station (ICT318 / 320)
Each time we want to send a message (via printf( ) or cout) we must “stop” the processor from doing what we want it to do, and then send messages over the interface – Slow, plus the testing can impact on peripheral performance.
E.g. no sound while printing in Assignment 2 and Lab. 1
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
6 / 28
More issues with this sort of test!
Tests are mixed up “with production code” Must remove tests before release to customer
Unreliable – may result in test messages at wrong time in released code
Difficult to “add” and “remove tests” Can’t be automated – therefore unlikely to have tests run
often during development – leads to unreliable product
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
7 / 28
E-TDD Tool
E-TDD – embedded test driven development Build-the-tests-first (during design) ideas for product
development has been made popular by the “Agile” approach
Requires an automated testing approach In this class, we are going to use the E-Unit tool for
its automated behaviour Will use in both “test last” development (what most people
currently do) and in “test first” development. I (the customer) will provide tests so that you can check that
you have developed the “right” stuff at the “high” level You will build additional tests to check your own code at the
“low level”
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
8 / 28
Test Environment
Can download “LabTestEnvironment.zip” file from the web (Check the September webpages for exact name of file)
LabX directory iswhere you keepyour “final” code
LabXTests is whereyou keep the testsfor LabX
E-UNITIncludes directory contains “everything” needed to make test environment work
E-UNITIncludes
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
9 / 28
Build a new VDSP project “Assignment1Test” and place in “AssignmentTests” directory
Now add certain standard files to the project – same set of files for all assignments and laboratory
All found in E-UNITIncludes directory
07.h
07.h
07.dlb
07.dlb
Expect minor changes (names and dates) as we “improve” E-UNIT as part of our own going research work on embedded systems
E-UNIT name replaces
E-TDD name
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
10 / 28
Now add your Assignment1 subroutines files to the Assignment1Test project
Don’t move or copy the files into the test directory – just “add” them to the project
Note the use of “Assignment1.h” which has the prototypes for all “C++” and “assembly code” functions used (from Assignment dir.)
This files are “added” to the“Test” project and NOT copiedfrom the Assignment1 directory
07.dlb
07.dlbE-UNIT name replaces
E-TDD name
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
11 / 28
Now we need to develop and add “Testmain.cpp”
E-UNIT name replaces
E-TDD name
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
12 / 28
Now we need to develop and add Assignment1Tests.cpp”
1
2
3
4
5
6
7
8
9
“../E-UNITIncludes/E-UNIT_Tests1Sept07.h”
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
13 / 28
Tests have a standard format
1
3
4
8
9
TESTGROUP NAME
TESTGROUP NAME, TEST TYPE
TEST TYPETEST NAME
Add further tests at this location
“../E-UNITIncludes/E-UNIT_Tests1Sept07.h”
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
14 / 28
Tests then need to be customized
1
2
3
4
5
6
7
8
9
07h“../E-UNITIncludes/E-UNIT_Tests1Sept07.h”
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
15 / 28
Test Components
The ACTUAL Test (This is a TEST MACRO)CHECK(result1 == result2) Would provided better test messages if we
“self-documented the code”CHECK(result_CPP == result_ASM);
Test CONTROL statements (ALL MACROS) TEST_CONNECT( test_name) LINK_TEST( test_name, test_type) TEST_LEVEL( which_test_level)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
16 / 28
Stages to “automating” the tests
1) Activate the E-TDD Gui (E-TDDIncludes directory)
This causes the GUI to appear, and adds new “menus” to VisualDSP
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
17 / 28
Stages to “automating” the tests
2) Select E-UNIT Connection | Refresh Connection List
This automatically builds the E-Unit Test Info file
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
18 / 28
Now “build and run” the testsQuick test failure
visualization
Detailed test failure information
Clicking on the “detailed test info”automatically takes you to the “test” that failed
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
19 / 28
Test information modes arecontrolled via “ActivateTests.cpp” Show Failures only
Show Failures and Successes
| BTC_CONNECTION
| BTC_CONNECTION
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
20 / 28
Add a separate test to show that result2(from the ASM code) is equal to 7
Add further tests at this location
“../E-UNITIncludes/E-UNIT_Tests1Sept07.h”
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
21 / 28
Write the tests for “self-documenting” messages(What does “result1 == result2” mean if 200 tests present?)
HAVE
WANT
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
22 / 28
Write the “CPP” code to get to workbut “Write the ASM for speed”
FAIL
SUCCESS
SUCCESS
“../E-UNITIncludes/E-UNIT_Tests1Sept07.h”
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
23 / 28
Did the ASM code “destroy” or even “use” the CPParray[ ] defined in CPPassignment1.cpp?
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
24 / 28
Available ASSERTS
LONG INT tests – 32 bit – typical for “C++” CHECK(condition) – Success if passes CHECK_EQUAL(expected, actual) EXPECTED_FALSE(condition) – Success if fails ARRAYS_EQUAL(expected, actual, length) CHECK_VALUES(expectedTemp, condition, actualTemp)
DOUBLES_EQUAL(expected, actual, threshold)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
25 / 28
ASSERTS for embedded systems
Short int tests – 16 bit – typical for embedded systems SHORTS_EQUAL(expected, actual) SHORTS_CHECK(expected, condition, actual) USHORTS_EQUAL(expected, actual) USHORTS_CHECK(expected, condition, actual)
Timing of routine – Blackfin Specific MAXTIME_ASSERT(max_time, function) Useful utility
MEASURE_EXECUTION_TIME(timetaken, function)
Examples available in the tests associated with Lab. 1
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
26 / 28
New ASSERTS can be added to customize the tests or speed the testsASSERT FORMAT
#define CHECK_VALUES(expectedTemp, condition, actualTemp)\if (!((expectedTemp) condition (actualTemp))) \{\
FAILURE1(#expectedTemp##" !"#condition##" "#actualTemp);\}\else {\ SUCCESS1(#expectedTemp##”""#condition##" "#actualTemp);\}
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
27 / 28
Practice – Assert Design
Design an assert that will determine whether the lower 8 bits of a 32-bit variable is identical to the lower 8 bits of a second 32-bitvariable.
The two variables should remain unchanged by the assert (since they may be reused later)
The upper 24 bits of each variable MUST be ignored in the assert
Does it matter if the variables were defined in the .data section of an assembly code routine or in an C++ code routine?
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
28 / 28
Handled today – some easy “non-programming” quiz and midterm questions
Why test, and what kinds of tests are there? Difference between
TLD – Test Last Development TDD – Test Driven Development
(Test First Development) How do you add tests for Assignment 1? More details on the testing process E-TDD and
the testing tool E-UNIT Other kinds of available tests – time and access Design of custom TESTs and ASSERTS