agile testing days
DESCRIPTION
This is the presentation we gave in 2009 during Agile Testing Days in Berlin. Even though it is already more than 2 years old, many things we said during the talk are very valid today. Some things did not change at all.TRANSCRIPT
![Page 1: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/1.jpg)
Test Driven Development of
Embedded SystemsMarcin CzenkoMartijn Gelens
Wim van de Goor
Agile Testing Days, 14 october 2009, Berlin
![Page 2: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/2.jpg)
• Test Consultancy at VIIQ.
• Agile Test Team leader at Philips CareServant.
• 10 years of experience in the Testing Field (7 years V-Model, 3 years Agile).
• Some experience with coding.
• Bachelor degree in Information Technics.
• Married to Jeanne, has a dog (Sasha) and 2 cats.
Martijn Gelens
![Page 3: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/3.jpg)
• Currently, Philips Software Engineering Services, MiPlaza, Software Designer.
• Ph.D. in Computer Security from University of Twente, Enschede, The Netherlands.
• M.Sc. in Software Engineering at Warsaw University of Technology.
• 7 years of experience as an embedded software developer (SterKom (Poland) and freelancer).
• Modelling Languages: my master thesis + one year internship at Institute National des Télécomunications (INT), Evry, Cedex, France.
• Agile, test-invected since one year.
• Married to Beata and also has two cats (Mufasa and Ursus).
Marcin Czenko, Ph.D.
![Page 4: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/4.jpg)
Wim van de Goor• Agile Software Team Leader at Philips Software
Engineering Services, Philips MiPlaza.
• 8 years experience with eXtreme Programming.
• Agile mentor and Coach.
• Advised us on Agile Principles.
![Page 5: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/5.jpg)
Contents• Agile Testing
What Do we want to do?• Constraints
What can we do?• Agile Embedded Testing
How do we test?• Conclusions
![Page 6: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/6.jpg)
What do we want ?
• Prerequisites • Test Strategy • Acceptance Criteria • Continuous Integration • Testing
![Page 7: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/7.jpg)
Prerequisites
• Testing is integrated in the team's Way of Working.
• Acceptance Criteria are defined, discussed and explored beforehand.
• Test Driven Development.
• Continuous Build and Integration.
• Testing is structured (specify, verify, report).
![Page 8: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/8.jpg)
Test Strategy
• Product risks and mitigation.
• Tester-role.
• Definition of Done.
• Quality gates.
• Testing is structured (specify, verify, report).
![Page 9: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/9.jpg)
Test Strategy (cont.)
• Deliverables from outside (also HW).
• Issue procedure and attendees.
• Reporting (test reports, PMI, ...).
• Emergency scenarios.
![Page 10: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/10.jpg)
Test Strategy (cont.)
The test strategy is build around the iterations and hardware deliveries.
![Page 11: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/11.jpg)
Acceptance Criteria• Defined upfront
• Based upon Quality Attributes; functionality, usability, efficiency, maintainability, reliability, portability, etc...
• Definition of Done:Tested and accepted, Code reviewed, documents written, yellow sticker on the note, ...
• Quality Gate:Acceptance tests passed, smoke test passed, ...
![Page 12: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/12.jpg)
Continuous Integration
Automate your:
• Build procedure.
• Release package creation.
• Deployment to simulator.
• Deployment to Board Support Package.
![Page 13: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/13.jpg)
Continuous Integration
![Page 14: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/14.jpg)
Continuous Integration
![Page 15: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/15.jpg)
Test Driven Development
• Means: “Write unit tests before code”.
• Integrate with your Continuous Integration environment.
• Automate the Acceptance Tests.
![Page 16: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/16.jpg)
This is your framework
![Page 17: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/17.jpg)
Constraints
![Page 18: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/18.jpg)
Light to HeavyLight Heavy
![Page 19: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/19.jpg)
Light• No cross-compilation required.
• Usually mainstream OS (Windows/Linux).
• Wide range of testing/mocking frameworks available.
• Standard hardware - no or very limited hardware level programming required.
• No dependence on a particular vendor (supplier).
![Page 20: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/20.jpg)
Heavy
• Small memory (no OS), limited performance, limited debugging possibilities.
• Limited cross–compilers: often only C, and Embedded (Extended) C++ available.
• Vendor specific.
• Difficult to find a testing/mocking framework.
• Custom hardware (ramp-up).
![Page 21: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/21.jpg)
Hardware design challenges
![Page 22: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/22.jpg)
Using Evaluation Boards
![Page 23: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/23.jpg)
Getting There (1)
![Page 24: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/24.jpg)
Getting There (2)
![Page 25: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/25.jpg)
Getting There (3)
![Page 26: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/26.jpg)
Getting There (4)
![Page 27: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/27.jpg)
Getting There (5)
![Page 28: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/28.jpg)
Getting There (6)
![Page 29: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/29.jpg)
Getting There (7)
![Page 30: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/30.jpg)
Development Board
• Selecting the right board can be challenging (expensive ?).
• Chip selection driven by the availability of the right board.
• The board selection driven by the availability of the BSP and OS.
![Page 31: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/31.jpg)
Is there a more lean solution ?
![Page 32: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/32.jpg)
Are we lean ?
![Page 33: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/33.jpg)
Queue 1
![Page 34: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/34.jpg)
Queue 2
![Page 35: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/35.jpg)
Queue 3
![Page 36: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/36.jpg)
The effect on Agile Principles
• Longer ramp-up time.
• Resistance to modify hardware (introduces up-front design).
• Limited response to changes.
• Higher risk.
![Page 37: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/37.jpg)
How to proceed ?
• Often we cannot remove queues & batches in HW development (are we going to be able doing so in any predictable future ?).
• Reducing the queue size is also often not an option.
• Is there a lean solution ?
![Page 38: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/38.jpg)
Getting There (7)
![Page 39: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/39.jpg)
Fix in between...
![Page 40: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/40.jpg)
Fix in between...
![Page 41: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/41.jpg)
How do we test ?
Our experiences
![Page 42: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/42.jpg)
What do we need
• Testing Strategy.
• Hardware to work with.
• Tool chain (compiler).
• Testing Framework.
• Mocking Framework.
![Page 43: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/43.jpg)
Compiler
ISO C++
ANSI C
![Page 44: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/44.jpg)
Testing Framework
Keep it simple !
Do-It-Yourself !
![Page 45: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/45.jpg)
Testing Framework (C)
• Many frameworks are simple ports of the frameworks for PC-based development.
• Increased stack consumption.
• Dynamic memory allocation.
![Page 46: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/46.jpg)
CMock
• Easy to understand. Easy to customise. Lightweight.
• Comes with Supporting Ruby-based Mocking Framework.
• Ready For “Heavy Embedded” - tests executed in batches.
![Page 47: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/47.jpg)
Testing Frameworks (C++)• Run-Time Type
Information (RTTI).
• Exceptions.
• ISO C++ compiler needed.
• Gnu or Microsoft are preferred.
![Page 48: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/48.jpg)
Testing Frameworks (C++)
• We could not find a framework that compiles on GreenHills and WindRiver C++ compilers (forget IAR Extended Embedded C++).
• It was cheaper and more effective to come with your own simple testing frameworks.
![Page 49: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/49.jpg)
yaffut
• Our choice for unit testing in C++.
• Just one header file.
• Not meant for embedded: needs RTTI, and C++ exceptions.
• Easy to understand and customise. We made an RTTI and Exception-free version.
![Page 50: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/50.jpg)
Mocking
• We did not succeed in using existing frameworks. Our best candidate GoogleMock does not even compile and it is quite complex.
• Does it mean that no one is doing TDD on embedded ? Probably not.
![Page 51: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/51.jpg)
Mocking - way to go
• Think of your own framework.
• We needed three “evenings” to create our own mocking framework.
![Page 52: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/52.jpg)
Educate your customer
![Page 53: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/53.jpg)
Educate your customer
![Page 54: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/54.jpg)
Conclusions• Agile in Heavy Embedded is a challenge: we cannot change it, but
we can understand it and try to reduce its impact on agile software development.
• The tester should be experienced in working with hardware, perhaps even more than a developer.
• There is no one way: what you can do depends on the constraints you have (e.g. light to heavy).
• Hardware development is far from being lean - and there is not that much we can change.
• Development and support tools are far behind the needs of the agile teams.
• Let your agile testing framework grow with your code.
![Page 55: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/55.jpg)
Conclusions• Agile in Heavy Embedded is a challenge: we cannot change it, but
we can understand it and try to reduce its impact on agile software development.
• The tester should be experienced in working with hardware, perhaps even more than a developer.
• There is no one way: what you can do depends on the constraints you have (e.g. light to heavy).
• Hardware development is far from being lean - and there is not that much we can change.
• Development and support tools are far behind the needs of the agile teams.
• Let your agile testing framework grow with your code.
Be agile.
![Page 56: Agile Testing Days](https://reader030.vdocuments.us/reader030/viewer/2022020217/54b537104a79594f358b4610/html5/thumbnails/56.jpg)
Questions