what, why and how? johnny bigert, 2012-10-25 [email protected]
TRANSCRIPT
My View on Unit TestingPart of something bigger – testing on many
levels1. Fast and early feedback!
Tight feedback loop, results within seconds Early implies lower cost
2. Confidence! Less fear of refactoring
3. Better code! Testability requires decoupling and usable APIs
4. Reliability! Possible to remove all non-determinism
Definitions and TerminologyUnit test, functional test, system test – what does it mean?
Definitions and TerminologyThere are no definitive answersDifferent dimensions of automated tests:
Level – Unit, component, integration, systemAPI – Developer test, external testCharacteristic – Functional, non-functional
Acceptance testing, regression testing
What Level Are We Testing On?Unit – The smallest unit of program code, normally
a single function, class with methods or data structure with functions
Component – A set of units integrated together, normally a module or subsystem comprising an isolated set of classes or files.
Integration – Several layers of a stack integrated, normally a stand-alone application (e.g. client or server)
System – Several stand-alone applications integrated (e.g. clients and servers), normally the full system
What API Are We Using For Testing?Developer tests – Function calls to an internal
or external interface, normally test and production code running in the same process
External tests – Invokation of a non-programming interface, normally running in separate processes or machines. For example, UI (simulated user input), network (HTTP etc), IPC.
What Aspect Are We Testing?Functional – The behavior of the system,
normally expressed as a step-by-step instruction with some alternative paths
Non-functional – The characteristics of the system, normally expressed as measurements and numbers. For example, latency, round-trip time, memory/CPU/bandwidth consumption, scalability, availability.
FunctionalDeveloper External
Unit “unit test” N/A
Component “component test”
Integration “functional test”
System N/A “system test”
“Component test” – personal favorite. Provides stability just like a “functional test” (test code less brittle than unit tests). Provides flexibility, reliability and fast feedback just like a “unit test” (quick to write, easy to mock out problematic parts, results within milliseconds).
Non-FunctionalDeveloper External
Unit N/A
Component e.g. test threading model of a component
Integration e.g. measure CPU consumption
System N/A e.g. “load test”
Choose How to TestWhen choosing, consider
Cost (of writing, maintaining, execution resources etc.)
Feedback loop (time)Reliability
Trade-off example: Mocking or using system x?Mocking: cost of writing and maintaining, risk
of modeling wrong behavior.Using: cost/risk of non-determinism, cost of test
resources and longer execution times.
Regression And Acceptance TestingDefinition: “regression” or “regression bug”
Something that was once considered “done” but does not fulfill requirements anymore
Acceptance testing: the use of a test to determine if a feature fulfills the requirements
Acceptance tests becomes part of the regression test suite once a feature is “done”
How to Unit TestNecessities of effective unit and component testing
DependenciesAssumption: we want to achieve fast
feedback and reliable testsNote, there are other kinds of tests that also
have meritRemove non-determinism: we need to be in
controlRandomness, time, file system, network,
databases, multi-threadingRemove dependencies to non-deterministic
components
DependenciesWhat kind of problematic dependencies to
you have in your code base?
Getting Rid of Dependencies, C++Example: Randomnessclass A {
public: void foo() { bool coinflip = rand() % 2 == 0; if(coinflip) … }};
Problem: dependency to non-deterministic component (randomness)
Getting Rid of Dependencies, C++C++ solution: introduce interface and use
dependency injection
class IRandomness { virtual bool coinflip() = 0;};
Getting Rid of Dependencies, C++class A {
public: A(IRandomness &r) : randomness(r) {} // ctor inject void foo() { if(randomness.coinflip()) … }private: IRandomness &randomness;};
What Is a Mock Object?Conforms to interfaceProgrammable behavior
For example, “if someone calls your send function with argument ‘hello’, return 7”
Can verify function invocations
Mock frameworks availableE.g. gmock (Google)Much more elegant in reflective languages
Getting Rid of Dependencies, C++Test code:void testFoo() {
RandomnessMock random; // inherits IRandomness A a(random); a.foo(); …}
Getting Rid of Dependencies, C++Production code:void main() {
Randomness random; // inherits IRandomness A a(random); …}
Bi-Directional Dependencies
sendData
(inherits)onDataReceived
(inherits)Application
Network
INetworkReceiver INetworkSender
Getting Rid of Dependencies, CFunction pointers = hard-to-read codeLink-time tricks
Link with other translation unit defining e.g. rand()
Getting Rid of Dependencies, CRandMock m; // mixed C/C++ test code
RandMock &getRandMock() { return m; // used by test case to access mock}int rand() { m.rand(); // forward to mock object}
Drawbacks: hard to understand test case code, one test executable per component to test = harder to maintain
Or clean C solution
Code CoverageDon’t focus on high levels of code coverage
per seNecessary, but not sufficient
Instead, focus on covering the most important user scenarios
After that, use code coverage to determine which code has not been covered
Analyze to understand what user scenarios are missing
Unit Test FrameworksTest runners/frameworksGTestUnitTest++CppUnitC Unit Testing Framework
Mock frameworksGMockHippoMockppAmop
VisionAgree on what to aim for
Fast Feedback ExampleFrom a previous job:Proprietary hardware/software/OS platformVery expensive hardware = bottleneck in
testingEmulator to test software locally on Linux
machineRan unit tests in emulator, turn-around time
= 15 minsUnacceptable
Fast Feedback ExampleMain problem: emulator start-up timeChallenge: proprietary types used (e.g. string
classes)Solution: fake proprietary types (string with
std::string, many others just empty implementations)
Ran unit tests on local machine,turn-around time < 15 seconds (from 15 mins)
Fast Feedback ExampleInvest to speed up feedbackInvest to remove testability obstaclesUnit testing should be easy, fast and fun
Development ExcellenceDefect prevention
Continuous improvement - feedback from bugs in order to improve software development process
Relentless testingRegression detection
Continuous integration running all regression tests often
Focus on both functional and non-functional requirements
Always releasableKeep CI builds greenKeep the product working
Continuous IntegrationStaged
Stage 1 – build, unit tests, code coverage and functional smoke tests – 10 mins
Stage 2 – extensive functional tests, smoke non-functional tests – 1 hour
Stage 3 – full functional and non-functional regression tests , static and dynamic analysis – 12 hours
Run stages perCommit (fine-grained tracability, expensive),
orRegularly (1. commit, 2. hourly, 3. nightly)
Code Analysis ToolsStatic analysis (run on source code)
Lint – free, local analysis only, lots of false positivesKlocwork – commercial, global analysisCoverity – expensive, global analysis, good
precisionDynamic analysis (need representative data)
IBM Rational Purify – commercial, memory leaks, double frees, buffer overflow, use of uninitialized memory. Instrumentation.
Valgrind – free, memory problems, heap, usage, cache usage, call graphs, threading problems. No instrumentation.