automated regression testing for embedded systems in action
DESCRIPTION
This presentation shows a real world example of streamlining the software development for a medical device system, using continuous integration, Behavior Driven Development, and even robotics!These ideas may be applied to any software project, regardless of budget or technologies.TRANSCRIPT
Automated Regression Testing for Embedded
Systems in Action
by Steve Peterson
AAND Tech, Inc
Situation
Client manufactures a Class II* medical device Produces reliable software. Productivity is OK All the T s are crossed and I s are dotted in terms of
process and documentation.
However …
* - See this site for an explanation of the FDA’s medical classifications: http://www.fda.gov/CDRH/devadvice/3132.html
Inefficiencies
Efficiency-wise we were getting killed by:
Requirements (traceability, maintaining) Test Plans (writing, maintaining, executing)
Old Way
Requirements written in Word documents. Manual Updates. Brute Force Traceability.
Test Plans written in Word documents. Execution by personnel.
Let’s Step Back in order to Solve This
What are the “requirements” for the process? We started with defining “project complete”
All Requirements are written. All Requirements are (successfully) tested. All Requirements are documented.
Next, Define the Deliverables*
Software/Firmware Requirements Documents Test Plans Test Results
*- This is far from a complete list, and doesn’t include items such as reviews and project plans.
Define Some Characteristics
Software and its process are Testable Changeable Traceable
Now something we would really like
For the 4 deliverables from a couple slides ago: Software/Firmware Requirements Documents Test Plans Test Results
We would like this to be in sync for the entire duration of the project!
Our Plan
Previously we did all the basics with revision controlling the documents, source code, binary images, etc.
We added: A method of automatically linking
requirements to tests/implementation A framework that will build the code (from
revision control) deploy it to the target, and perform automated regression testing.
Our Implementation*
These items are covered in detail later. Requirements remain in Word documents. However
now we have automated requirements traceability (to tests/implementation)
Automated Regression Tests Continuous Integration**
* - Many 3rd party solutions for requirements tracking and testing exist. Due to cost and limits of customization, we chose to implement our own.
** - True Continuous Integration runs the entire regression test suite after each check in. In our case we stretch the term “continuous” to mean nightly. This is due to the abbreviated test suite requiring 3 hours to run and several days for the formal V&V test suite.
Requirements Tracking Tool Dynamic and Static Real Time
generation of requirements traceability*.
Requirements based Testing. Finds untested requirements,
orphaned tests. PDF report generation Coming soon, changed
requirements, triggers for changed code.
PDF report generated by the tool tracing the requirements to test/implementation.
Static coverage occurs from parsing the requirements documents and using Ruby’s introspection on the classes to indentify coverage
Dynamic coverage occurs from running the regression test. Each assert statement lists the requirements that it’s testing. We generate coverage matrices both for assert successes and failures. For you UML fans, the relationship between asserts and requirements is many-many (an assert can test more than one requirement and a requirement can be tested by more than one assert).
Tri-State Coverage
Green Light – Test item passesRed Light – Test item failsYellow Light – No test coverage for this item. Red Light items get immediate attention. Yellow Light items are given some slack (but not too
much!) The idea is to continuously add features and keep
the code, tests, and documents in sync. No “big bang” testing or documentation effort at the
end!
Automated Testing Framework
Components Revision Control System (we used
Subversion) The Scheduler The Build engine The “Deployer” – installs the correct software
under test on the device under test, whether a web or embedded device.
Peripheral Measurement Devices.
Embedded Device
Want as much of the testing to be as non-invasive as possible.
Next revision will have robot button presses and a camera to read the display
Until then, the firmware allows simulated button presses and reading the LCD using commands via the serial port. Also a couple generate commands to read values.
Software safeguards exist to mitigate against these commands interfering with device operation.
Chassis of the DUT (Device Under Test)
with the serial port clip used to program the DUT,
Issue commands and read status
Build Engine
Performs a complete source code check out from the SVN* (Subversion) repository. The default is the trunk, but any branch, tag or SVN ID may be used. Launched from a cron job** or manual initiation.
The build may be production code base, a special developer code base, or for the embedded code, unit test code.
Applicable to both the embedded and web based technologies.
* - Subversion (SVN) is an open source revision control system: http://subversion.tigris.org/
** - A Cron job executes commands or scripts (groups of commands) automatically at a specified time/date.
The Scheduler (Page 1)
DUT (Device Under Test)
Peripheral Test Module<Programmable Power Supply,
Breathing Simulator,Manometer,
X10, power switch>
Test Run
Job
Software Version< Release Candidate,
Increment,Nightly Build,
Developer Produced>
1..*
1
1
1 1
1..* 1
UML diagram showing the relationship between a Job, Test Runs, a DUT, Peripheral Test Modules and installed Software version.
The Scheduler (Page 2)
A DUT (Device Under Test) may be a single embedded device, a web component or a system DUT consisting of both.
A test module consists of peripheral devices under script control such as a programmable power supply, breathing simulator, manometer.
A run consists of a DUT (see above), and 0 or more peripheral test modules running a single script
Web Interface for monitoring and controlling the testing “jobs”
The Scheduler (Page 3)
A job consists of 1 or more runs. May span numerous DUTs Noteworthy jobs are the formal V&V job (may require up to 3
days to run), the nightly (the full V&V run trimmed down to about 6 hours), and developer submitted jobs.
The job “manifest” specifies either a specific version of code to test (as produced by the build engine or a developer produced version).
The scheduler also acts as a traffic cop. A database tracks the attributes of various DUT units and queues up job/run requests based on run requirements.
Flash Memory Programmer (The Deployer)
Under script control, this programs the embedded device’s flash memory with the specified binary image.
Peripheral Test Modules (Page 1)
MacGyver Manometer*
* The term MacGyver Manometer is a tongue in cheek reference to the 80/s TV show, where MacGyver could create anything using a paper clip and rubber band (see photo above). In this case the DUT, which contains a pressure sensor more accurate than most handheld manometers, is outfitted with special firmware that displays the current pressure, and reports the pressure up to 16 samples/second over the serial connector.
Programmable Power Supply
Peripheral Test Modules (Page 2)
Hans Rudolph Breathing Simulator
A couple more pictures
More testing power per sq foot of space anywhere!
X10 Power Control
More Pictures
Robotic Arm Pressure Control Valve
Benefits Able to give embedded devices the same
automated regression testing benefits available to web and PC applications
Process accommodates change. It is still not free, but much less painful.
Sanity check. No more being 90% done for 50% of the project.
Much less Stress! Bugs are fixed immediately. Formal V&V testing took just a couple days, instead
of weeks. Over 97% of testing was automated!
Additional Benefits
Because we used the real devices (day after day, night after night), this doubled as life testing. Non-software items were validated as well.
Due to the easy command set for controlling the DUT and capturing information from the measuring equipment, this was used for research applications as well.
Ruby has a rich set of Microsoft Office plugins so automated formatting of data in Excel and Word is straight forward.
ChallengesAutomated Testing is not the same as a user
We found that in actuality it is (or very close). We tested the Behavior (read button presses tangible user responses). In addition we constantly had real people performing unstructured testing. If this located a problem, additional items were added to the automated testing.
How can you be sure that the tests work (how do you test the tests?)The test suites incorporate counter test cases – inject out of bounds conditions and verify that the test correctly catches it.
Contact
Steve Peterson
AAND Tech, Inc.
http://www.aandtech.com
http://www.jruby.net