test driven development for embedded c, why debug? esc-411 … · 2011-09-23 · test driven...

Test-Driven DevelopmentESC-411

[email protected]

Copyright © 2008-2011 James W. GrenningAll Rights Reserved. For use by training attendees.

Test Driven Developmentfor Embedded C, Why Debug?

ESC-411Boston 2011

James W [email protected]

www.jamesgrenning.comwww.twitter.com/jwgrenning

You can find my book athttp://pragprog.com/titles/jgade/

1


[email protected]


Questions I’ll Answer

• Why should I care about TDD?• What is TDD?• How is TDD adapted to embedded development? • How do I test code with dependencies?• Why are Unit Tests critical?• Why is Automation Critical?

2


[email protected]


Don’t be Late, Don’t Have Bugs

MissedCustomer

ExpectationsLate Project

Poor Quality3


[email protected]


We Make Our Problems

Vague Requirements

MissedCustomer

ExpectationsLate ProjectUnrealistic Plan

Unplanned Activities

Unpredictable Activities

Poor Quality4


[email protected]


Positive Feedback - Unstable System

Unrealistic Plan Late ProjectMissed

Customer Expectations

Schedule Pressure

Long Hours

Bugs Poor Quality

Vague Requirements

Unplanned Activities

Unpredictable Activities

5


[email protected]


This Work Flow is Designed to Allow Defects

6

Development

Test

Defects


[email protected]


7

Your program will have bugs. And they will surprise you when you find them.


[email protected]


The Physics of Debug Later Programming (DLP)

• As Td increases, Tfind increases dramatically• Tfix is usually short, but can increase with Td

8

Bug discoveryMistake made(bug injection)

Bug found Bug fixed

Td Tfind T fix

Time


[email protected]


A Bug’s Life

9From http://www.softwaretestinghelp.com/bug-life-cycle/


[email protected]


Edsger DijkstraThose who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result, the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.

10


[email protected]


Can we Realize Dijkstra’s Dream andPrevent Defects with

Test Driven Development?

11


[email protected]


Development and Test Work TogetherPreventing Defects

12

Development

Test


[email protected]


13

A Complex system that works is invariably found to have evolved from a simple system that worked.


[email protected]


The Physics of Test Driven Development

• When Td approaches zero, Tfind approaches zero• In many cases, bugs are not around long enough to be considered

bugs.• See: http://www.renaissancesoftware.net/blog/archives/16

14

Mistake discovery

Mistake made

Root cause found

Mistake fixed

T d Tfind T fix

Time


[email protected]


Zune 30G

• On December 31, 2008 this function went into an infinite loop.

• Ruining New Years Eve Parties around the world.

15

– It accepts the number of days since January 1, 1980 and figures out: day, date month and year.


[email protected]


16

Zune 30G Bug tracked to this Function

BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime)


[email protected]


17

One the Got Awaystatic void SetYearAndDayOfYear(RtcTime* time){ int days = time->daysSince1980; int year = STARTING_YEAR; while (days > 365) { if (IsLeapYear(year)) { if (days > 366) { days -= 366; year += 1; } } else { days -= 365; year += 1; } }

time->dayOfYear = days; time->year = year;}


[email protected]


18

This Test Could Have Prevented it

TEST(Rtc, check20081231){ days = daysSince1980(2008, 366); CHECK(ConvertDays(days, &time)); assertDate(WED, 2008, 12, 31);}


[email protected]


What Tests to Write

• One Test Would Have Prevented Zune Bug• If we knew where the bugs are, we could just write

those tests (or fix the bug).• Bugs can be anywhere, so we have to write tests for

everything.

19


[email protected]


What is TDD?

20


[email protected]


TDD Micro Cycle

• Write a test• Watch it not compile• Make it compile, but fail• Make it pass• Refactor (clean up any mess)• Repeat until done

21


[email protected]


TDD State Machine in C/C++

22

Write the test

Make the test link

Make the test compile

Compilation error

Link error

New test failsMake the test pass

Refactor(Make it right)

All tests pass

All tests pass No more tests

Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


Test Driving a Circular Buffer (FIFO)

23

Out-Index In-Index

23 7 66 12 99 16 90

Out-Index In-Index

42 -6 23 7 66 12 99 16 90

Out-Index In-Index

99


[email protected]


Create a Test-List(avoid analysis paralysis)

24

Circular Buffer Tests

Initially Empty Transition to empty Transition to Full Transition from Full Put-Get FIFO Put to full Get from empty Filled but not wrapped around Wrap around


[email protected]


Setup a Test Fixture

25

TEST_GROUP(CircularBuffer){ CircularBuffer* buffer;

void setup() { buffer = CircularBuffer_Create(); }

void teardown() { CircularBuffer_Destroy(buffer); }};

TEST(CircularBuffer, TestName){}


[email protected]


Make a Partial Skeleton of the Production Code Header File

26

#ifndef D_CircularBuffer_H#define D_CircularBuffer_H

typedef struct CircularBuffer CircularBuffer;

CircularBuffer * CircularBuffer_Create();void CircularBuffer_Destroy(CircularBuffer *);

#endif // D_CircularBuffer_H


[email protected]


Make a Partial Skeleton of the Production Code Source File

27

#include "CircularBuffer.h"#include <stdlib.h>

struct CircularBuffer{ int dummy;} ;

CircularBuffer* CircularBuffer_Create(){ CircularBuffer* self = calloc(1, sizeof(CircularBuffer)); return self;}

void CircularBuffer_Destroy(CircularBuffer* self){ free(self);}


[email protected]


Choose a Test

28

Write the test

Make the test link


Compilation error

Link error



All tests pass


Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


Which Test?

29

Out-Index In-Index

41 59 14 33 7 31

In-Index Out-Index


[email protected]


Write a Test

30

Write the test

Make the test link


Compilation error

Link error



All tests pass


Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


Start with a Test That is Easy to Get to Pass -- Designing the Interface

31

TEST_GROUP(CircularBuffer){ CircularBuffer* buffer;

void setup() { buffer = CircularBuffer_Create(); }

void teardown() { CircularBuffer_Destroy(buffer); }};

TEST(CircularBuffer, ShouldBeEmptyAfterCreate){ CHECK(CircularBuffer_IsEmpty(buffer));}

Write the test

Make the test link


Compilation error

Link error



All tests pass


Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


Make the Test Compile

32


[email protected]


Make the Test Compile

33

#ifndef D_CircularBuffer_H#define D_CircularBuffer_H

typedef struct CircularBuffer CircularBuffer;

CircularBuffer * CircularBuffer_Create();void CircularBuffer_Destroy(CircularBuffer *);int CircularBuffer_IsEmpty(CircularBuffer *);

#endif // D_CircularBuffer_H

Write the test

Make the test link


Compilation error

Link error



All tests pass


Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


Make the Test Link

34


[email protected]


Make the Test Linkand Intentionally Fail

35



... not all code shown ...

int CircularBuffer_IsEmpty(CircularBuffer* self){ return 0;}


[email protected]


Watch it Fail

36

CircularBufferTest.cpp:48: error: Failure in TEST(CircularBuffer, ShouldBeEmptyAfterCreate)

CHECK(CircularBuffer_IsEmpty(buffer)) failed

Write the test

Make the test link


Compilation error

Link error



All tests pass


Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


Make the Test Pass

37


[email protected]


Write no More Code than is Necessary to Pass the Current Test

• Hard code behavior or partial implementations are OK and often preferred.

38


[email protected]


Make the Test Pass

39



... not all code shown ...

int CircularBuffer_IsEmpty(CircularBuffer* self){ return TRUE;}


[email protected]


Repeat until You Run Out of Tests

• The first tests drive the interface definition• Later tests complete the behavior• You will think of more tests as you go

– Write the new test– or Add it to the test list

40

Code without

new feature

Code withnew tested

feature

Half the way through, half the tests are passing and the code is

partially complete


[email protected]


But that is so Inefficient!So Slow!

Show us the fast way!

41


[email protected]


42


[email protected]


Get good at being careful and you can go fast.

43

The Careful way is the Fast Way!


[email protected]


Sustainability

44

Speculate Code Test DebugTraditional development

Time developers do not notice nor plan for

Test-driven development

Speculate Test and Code Debug

Feels slower


[email protected]


45

A Complex system that works is invariably found to have evolved from a simple system that worked.


[email protected]


Adaptation for Embedded Software

What is scarce during embedded development?

46


[email protected]


Hardware is Scarce!

• It does not exist.• It is being used by

someone else.• It has bugs of its

own.

47


[email protected]


Minimize DoH!

Debug

on

Hardware48


[email protected]


Use Your Development Systemfor a Test Bed

• Multi-targeted code.• Must beware of hardware and OS dependencies.• Object Oriented approach to Dependency

Management.• Avoid inefficiencies

– Waiting for hardware.– Waiting for restart.– Waiting for downloads.– Waiting for long compiles.– Debugging on the target.

49


[email protected]


But There are Risks with Development System Tests

• Architecture differences– Word size– Big-endian Little-endian– Alignment

• Compiler differences• Library differences• Execution differences

50

Write a TestMake it Pass

Refactor

Stage 1

Compile for TargetProcessor

Stage 2

Run Tests in the Eval Hardware

Stage 3

Run Testsin Target Hardware

Stage 4

AcceptanceTests

Stage 5

More Frequent Less Frequent


[email protected]


TDD Adaptation for Embedded Development

51


[email protected]


TDD Adaptation for Embedded Development

52

See : http://renaissancesoftware.net/files/articles/ProgressBeforeHardware.pdf

Write a TestMake it Pass

Refactor

Stage 1

Compile for TargetProcessor

Stage 2

Run Tests in the Eval Hardware

Stage 3

Run Testsin Target Hardware

Stage 4

AcceptanceTests

Stage 5

More Frequent Less Frequent


[email protected]


Problems Found at Appropriate Stage

53

Stage Problems Likely to Find

1 Logic, design, modularity, interface, boundary conditions

2 Compiler compatibility (language features)Library compatibility (header files, prototypes)

3 Processor executions problems (bugs in compiler and standard libraries)Portability problems (word size, alignment, endian)

4 Ditto stage 3Hardware integration problemsMisunderstood hardware specifications

5 Ditto stage 4Misunderstood feature specification


[email protected]


How do I test Code with Dependencies?

54


[email protected]


A Module with Collaborators

• Every minute, the RTOS wakes up the Light Scheduler.

• If it is time for one of the lights to be controlled, the LightController is told to turn on/off the light.

55

Time Service

+ GetTime()+ SetPeriodicAlarm()

LightScheduler

+ScheduleTurnOn()+RemoveSchedule()+WakeUp()

Light Controller

+ On(id)+ Off(id)

Hardware RTOS

<<anonymous callback>>

Admin Console


[email protected]


Program to Interfaces

• Separate interface and implementation as separate entities.

• This design has good separation of responsibilities

56

<<interface>>Time Service


LightScheduler

+ ScheduleTurnOn()+ RemoveSchedule()+WakeUp()

<<interface>>Light Controller

+ On(id)+ Off(id)

Model 42 Hardware RTOS

<<anonymous callback>>

Model 42Light Controller

RTOS Time Service

<<implements>> <<implements>>

Admin Console


[email protected]


Testing a Module with Collaborators

• Use the real collaborators if you can.

• Use fakes when you must.

57

<<interface>>Time Service


LightScheduler

Test

LightScheduler

+ ScheduleTurnOn()+ RemoveSchedule()+wakeUp()

<<interface>>Light Controller

+ On(id)+ Off(id)

Light Controller Spy

FakeTime Service

<<implements>> <<implements>>


[email protected]


Testing the Scheduler

58

TEST(LightScheduler, ScheduleOnTodayNotTimeYet){ LightScheduler_ScheduleTurnOn(3, EVERYDAY, 1000); FakeTimeSource_SetMinute(999);

LightScheduler_Wakeup();

LONGS_EQUAL(LIGHT_NA, LightControllerSpy_GetState(3));}

TEST(LightScheduler, ScheduleOnTodayItsTime){ LightScheduler_ScheduleTurnOn(3, EVERYDAY, 1000); FakeTimeSource_SetMinute(1000);

LightScheduler_Wakeup();

LONGS_EQUAL(LIGHT_ON, LightControllerSpy_GetState(3));}


[email protected]


Partial Skeleton Lets You Try the Design and Test Ideas Early

59


[email protected]


Right Next to the Siliconwith

Mock Objects

60


[email protected]


Message Flow for Flash Memory

Block Erase with Error

61

FlashDriver FlashDevice

IOWrite(CommandRegister, 0x40)

IOWrite(offset, data)

IORead(StatusRegister)

b7 == 0IORead(StatusRegister)

b7 == 1, other bits == 0

Repeats while b7 == 0

IORead(offset)

data

Flash_Program(offset, data)

FlashSuccess

b7 == 1

Start

Program CommandWrite 0x40 to 0x0

Start/Stop

Write data to address

Read status register

b3 == 0

b4 == 0

b1 == 0

YES

NO

YES

YES

YES

Vpp Error

Program Error

Protected Block Error

NO

NO

NO

Clear status Write 0xFF to 0x0

Wai

t for

read

y lo

op


[email protected]


Flash Program Flow Chart

62

How do you test these errors in the target?

How do you test these errors in the target?

How Many Tests are Needed?


[email protected]


Test List

63

Flash Memory Write - Test List Successful operationInvalid programming voltageProgram errorAttempt to program protected blockRead back failsDevice timeout during write operation


[email protected]


Flash Driver Test Fixture

64

<<implementation>>IO

<<implements>>

+ Read(addr) : data+ Write(addr, data)

<<interface>IO

+ Expect_Read(addr, data)+ Expect_Write(addr, data)

MockIO<<hardware>>

+ Program(addr, data)+ ProtectBlock(block)+ EraseBlock(block)//etc

FlashDriver

FlashDriverTest


[email protected]


Flash Driver Test

65

TEST(Flash, ProgramSucceedsReadyImmediately1){ int result = 0; MockIO_Expect_IOWrite(0, 0x40); MockIO_Expect_IOWrite(0x1000, 0xBEEF); MockIO_Expect_IORead(0, 1<<7); MockIO_Expect_IORead(0x1000, 0xBEEF);

result = Flash_Program(0x1000, 0xBEEF);

LONGS_EQUAL(0, result); MockIO_Assert_AllExpectationsMet();}


[email protected]


Mock Flash Read/Write

66

//MockIO.h#include "IOReadWrite.h"

void MockIO_Create(int maxExpectations);void MockIO_Destroy();void MockIO_Expect_IOWrite(ioAddress_t offset, uint16_t data);void MockIO_Expect_IORead(ioAddress_t offset, uint16_t data);void MockIO_Assert_AllExpectationsMet();

//IOReadWrite.h#include <stdint.h>

typedef uint32_t ioAddress_t;typedef uint16_t ioData_t;

ioData_t IORead(ioAddress_t offset );void IOWrite(ioAddress_t offset, ioData_t data);


[email protected]


Mock Flash Read/Write

67

//Snipvoid MockIO_Create(int maxExpectations){ expectations = malloc(sizeof(Expectation)*maxExpectations); memset(expectations, 0, sizeof(expectations)); setExpectationCount = 0; getExpectationCount = 0; maxExpectationCount = maxExpectations; failureAlreadyReported = 0;}

//MockIO.ctypedef struct Expectation{ int kind; ioAddress_t addr; ioData_t value;} Expectation;

static Expectation* expectations = 0;static int setExpectationCount;static int getExpectationCount;static int maxExpectationCount;static int failureAlreadyReported = 0;


[email protected]


Capturing Flash Read/Write Expectations

68

void MockIO_Expect_IOWrite(ioAddress_t addr, ioAddress_t value){ failWhenNoMoreRoomForExpectations(report_too_many_write_expectations); recordExpectation(FLASH_WRITE, addr, value);}

void MockIO_Expect_IORead(ioAddress_t addr, ioAddress_t value){ failWhenNoMoreRoomForExpectations(report_too_many_read_expectations); recordExpectation(FLASH_READ, addr, value);}


[email protected]


Mock Flash Read

69

ioData_t IORead(ioAddress_t addr){ setExpectedAndActual(addr, -1); failWhenNotInitialized(); failWhenNoUnusedExpectations(report_read_but_out_of_expectations); failWhen(expectationIsNot(FLASH_READ), report_expect_write_was_read); failWhen(expectedAddressIsNot(addr), report_read_wrong_address);

return expectations[getExpectationCount++].value;}


[email protected]


Mock Flash Write

70

void IOWrite(ioAddress_t addr, ioData_t value){ setExpectedAndActual(addr, value); failWhenNotInitialized(); failWhenNoUnusedExpectations(report_write_but_out_of_expectations); failWhen(expectationIsNot(FLASH_WRITE), report_expect_read_was_write); failWhen(expectedAddressIsNot(addr), report_write_does_not_match); failWhen(expectedDataIsNot(value), report_write_does_not_match); getExpectationCount++;}


[email protected]


All Expectations Must Be Met

71

void MockIO_Assert_AllExpectationsMet(){ if (failureAlreadyReported) return; failWhenNotAllExpectationsUsed();}


[email protected]


72

Some of the Things Professional Developers Need to Know


[email protected]


Unit Tests Are Critical

Though Not Enoung

73


[email protected]


System Level Tests Cannot be Thorough

• 1000 (or more) tests are needed to test this simple system

74

10 states

10 states

10 states

5 interactions

5 interactions

5 interactions

Tests


[email protected]


TDD unit tests can be thorough

• As few as 30 unit tests and 15 integration test when tested as units

75

10 states

10 states

10 states

5 interactions

5 interactions

5 interactions

Tests Tests


[email protected]


TDD is About Design

76


[email protected]


Refactoring

77

Structured code transformation to prepare the code

for change

Key points:• Doesn’t adjust functionality• Small and disciplined• Well defined• Keeps code healthy


[email protected]


Refactoring’s Three Critical Skills

78

Recognize what is wrong, fix it!Keep the code working the whole time!


[email protected]


Modularity == Testability

79

Focus on tests first ensures modularity and other good design principles

It’s about design, stupid!


[email protected]


80

Design from the perspective of userather than implementation


[email protected]


Tests Must Be Automated

81

test


[email protected]


Assume Test Effort is Proportional to Development Effort

82

Effort

Iterations

dev

5

10

15

20

25


[email protected]


• 25% of all defects are introduced while changing and fixing code

• [R.B Grady, Software Process Improvement]

83

If a system is working, leave it alone. Don't change anything

test


[email protected]


Manual Test is Unsustainable

84

Effort

Iterations

Unsustainable growth

dev

5

10

15

20

25

test


[email protected]


Risk Accumulates in the Untested Code Gap

85

Effort

Iterations

Unsustainable growth

dev

5

10

15

20

25 Untested Code Gap


[email protected]


Why Don’t We Just Test at the End, Save all that Retesting?

Requirements

Code

Design

Test

Time

NovemberSeptemberJulyMayMarch

Test and FixTest and Fix



Test and Fix






86

Test


[email protected]


FBI Case-File System - Never Delivered

87


[email protected]


Denver Baggage SystemAlmost One Year Late

88


[email protected]


Exercise

89


[email protected]


Why is it Worth Changing How You Program

• Defects kill predictability:–Cost of fixing is not predictable–When they materialize is not predictable

• Test-driven is predictable:–Working at a steady pace–Results in fewer bugs–More productive than “debug-later programming”

• Test-driven programmers rarely need the debugger

90


[email protected]


References and Resources• [XP] Kent Beck, Extreme Programming Explained, 1999• [REF] Martin Fowler. Refactoring. Improving the Design of Existing Code. 1999• [WELC] Michael Feathers, Working Effectively with Legacy Code• [TDD] Kent Beck, Test-Driven Development, 2003• [XUNIT] Gerard Meszaros, xUnit Testing Patterns, 2008• [PRAG] Andy Hunt, Dave Thomas, The Pragmatic Programmer• [SLAD] Craig Larman and Bas Voode, Scaling Lean & Agile Development• [POP] Mary Poppendieck and Tom Poppendieck, Implementing Lean Software Development:

From Concept to Cash, 2006• [AGILE] Robert C. Martin, Agile Software Development: Principles, Patterns, and Practices, 2002• [CLEAN] Robert C. Martin, Clean Code, 2008• [LESSONS] Cem Kaner, et. al. Lessons learned in Software Testing• [TD] Lasse Koskela, Test Driven, 2007

• Test harnesses– [CPPTEST] www.sourceforge.org, project CppUTest– [FITNESSE] www.fitnesse.org

• Groups– http://groups.yahoo.com/group/testdrivendevelopment– http://groups.yahoo.com/group/AgileEmbedded

91


[email protected]


See Embedded TDD and Related Blogs and Papers

http://www.renaissanceSoftware.net http://www.renaissancesoftware.net/blog/

• Embedded TDD• Zune Bug: Test Driven Bug Fix• Learning Tests are Free!

• TDD as a Design Rot Prevention System• Crashing Your Way to Great Legacy C Tests • TDD and the Big Framework Part• Bug Fixes and TDD• Physics of Test Driven Development• Tests vs. Short Term Cache Between Your Ears• Embedded Systems Conference FAQ • I miss constructors • Who says you can’t test drive a device driver?

92

• Why are You Still Using C? • Planing Poker• Agile Embedded Software Development (ESC)

• Launching Extreme Programming at a Process Intensive Company (IEEE)

• Test Driven Development for Embedded Software• Progress Before Hardware • Agile Times - Containing Progress Before

Hardware• Test-Driven Development for Embedded C++

Programmers


[email protected]


http://pragprog.com/titles/jgade/

93

test driven development for embedded c, why debug? esc-411 … · 2011-09-23 · test driven...

Documents