software testing 13. integration daniel riegelhaupt - thomas de vylder - gertjan loots - tim saquet...
Post on 21-Dec-2015
219 Views
Preview:
TRANSCRIPT
Software Testing13. Integration
Daniel Riegelhaupt - Thomas De Vylder - Gertjan Loots - Tim Saquet - Philip De Smedt
2
13. Integration
• Integration in Object-oriented Development
• Integration Patterns
3
13.1.1 Definitions
A System is composed out of Components whoare themselves systems of smaller components
Component(Focus of integration)
System(Scope of integration)
Typical intercomponent interface(Location of integration faults)
Method Class Instance variablesIntraclass messages
Class Cluster Interclass messages
Cluster Subsystem Interclass messagesInterpackage messages
Subsystem System Interprocces communicationRemote procedure callORB servicesOS services
4
13.1.2 Integration testing is essential
3 basic kinds of testing needed to show that components areminimally interoperable:
• Tests of individual components• Tests of the system resulting from the federation of components• Test of components interoperation
Integration testing = a search for component faults that cause intercomponent failures
5
Integration testing is essential
In contrast:• Component scope testing = search for intracomponent
faults• System scope testing = search for faults that lead to a
failure to meet a system scope responsibility
6
Integration testing is essential
• Antidecomposition axiom:A test suite that achieves coverage at system scope will not necessarily result in the same coverage for its components
In other words: even if the program works as a whole this doesn’t mean we test every component fully (some code might never be reached)
• Anticomposition axiom:Adequate testing of components (in isolation) ≠ adequate testing of a system of components
7
Integration testing is essential
Integrating most or all components in a system at the same time is usually problematic :
- Debugging is difficult (bug can be in any interface)- Last minute patches may be necessary (=> low quality, not
adequately tested )- No systematic testing can lead to undetected bugs
Incremental testing !
8
Integration testing is essential
• Advantages of incremental testing:- Interfaces are systematically exercised and proven to
be stable before exercising unproven interfaces- Buggy interface code will be reached; if not triggered- Observe failures are most likely to come from most
recently added component• Incremental testing requires discipline and commitment:
choose a sequence of components by analyzing dependencies (=> architecture must be known
or planned!). Next manage plan and automate
9
10
• Integration testing takes place at all scopes and is repeated in each
development increment
11
Integration testing is essential
Integration test plan must answer:• what are the major and minor development increments ?• which components interfaces / subsystems will be the focus
of integration ?• In what sequence will components and their interfaces be
exercised ?• Which stubs and drivers must be developed ?• Which test design patterns should be used ?• When can integration testing be considered complete ?
12
Integration testing is essential
Key considerations:• Scope of an increment and dependencies of components• Testability ( # components testability )• Component stability (postpone integration of volatile
components, start with stable ones)• Test environment stability
13
13.1.3 Dependency Analysis
Class/cluster scope dependencies:• Composition and aggregation• Inheritance • Global variables …• pointers to object used as message parameters • …
Intercomponent dependencies:• Explicit:
• Message exchange • Remote procedure calls• Interprocess communication
• Implicit: Not considered in Integration testing
14
Dependency Analysis
Explicit dependencies:• often dictate the sequence of testing• typically correspond to interfaces that should be exercised by an
integration TS
15
Dependency Analysis
Make a dependency tree:• by hand (if small)• Use topological search ( Unix: tsort; commercial tools)• Can have multiple roots, cyclical dependencies
Note that tools might have problems finding dependenciesover: Networks, VM , OS …
16
17
18
13.1.4 Faults Model
• Configuration/version control problems• Missing, overlapping or conflicting functions• Problems with file or DB ( incorrect/ inconsistent
representation; conflicting view /usage; data integrity ...)• Wrong method called due to coding or binding error• Client sends illegal message to server (violates pre
condition or sequential constraints)• Wrong object bound to message• Wrong parameter or incorrect parameter value
19
Faults Model• Failures due to incorrect memory management
(de)allocation• Incorrect usage of VM , ORB or OS services• Attempt by the IUT to use target environment services
that are obsolete / not supported in the currant version/ release
• Intercomponent conflicts (e.g. thread X will crash when process Y is running)
• Resource contention: target env. can’t allocate resources needed for nominal load (e.g. Need to open 6 windows but IUT crashes when fifth is opened)
IUT = implementation under test
20
13. Integration
• Integration in Object-oriented Development• Integration Patterns
21
Integration patterns
• Big Bang Integration. Attempt to demonstrate system stability by testing all components at the same time. - Useful in a few limited circumstances.
• Bottom-up Integration. Interleave component and integration testing by following usage dependencies. - Works well for small to medium systems.
22
Integration patterns
• Top-down Integration. Interleave component and integration testing by following the application control hierarchy. - Works for almost any scope or architecture.
• Collaboration Integration. Choose an order of integration according to collaborations and their dependencies. Exercise interfaces by testing one collaboration at a time. - Works for almost any scope or architecture.
23
Integration patterns
• Backbone Integration. Combine Top-down Integration, Bottom-up integration, and Big Bang Integration to reach a stable system that will support iterative development. - Well suited to small to medium systems and
especially useful for embedded applications.
• Layer Integration. Incrementally exercise interfaces and components in a layered architecture. - Applies to layered architectures.
24
Integration patterns
• Client/Server Integration. Exercise a loosely coupled network of components, all of which use a common server component. - Appropriate for client/server architectures.
• Distributed Services Integration. Exercise a loosely coupled network of peer-to-peer components. - Appropriate for decentralized networks containing
peer-to-peer nodes.
25
Integration patterns
• High-frequency Integration. Develop and rerun an integration test suite test hourly, daily, or weekly. - Can be applied at almost any scope or
architecture.
For complete scope specific considerations See book 13.2.1
26
Big Bang Integration
• Intent- Demonstrate stability with a few test runs
• Context- SUT is stabilized and only a few components have
been added since the last time- SUT is small and testable- Only feasible for monolithic systems
• Tightly coupled system• Bad design
27
Big Bang Integration
• Fault Model- Does the system run? Demonstrate that system
testing can begin
• Strategy- No incremental integration testing- Entire system is built and a test suite is applied
28
Big Bang Integration
29
Big Bang Integration
• Entry Criteria- All components have passed component scope
testing- The VM to be used is stable- No anomalies found that interfere with integration
testing
30
Big Bang Integration
• Exit Criteria- The test suite passes
• Consequences- Many interface bugs: where to look?
• Debugging is HARD: every component is suspect• Even if SUT passes: many hidden interface faults?
31
Big Bang Integration
• Known Uses- Subsystem Big Bang Integration is used in
Backbone Integration.
32
Bottom-Up Integration
• Intent- Add components to the SUT in uses-dependency
order. Components with fewest dependencies first.
• Context- Stepwise verification between tightly coupled
components- Test each component and then integrate it with
already-tested components
33
Bottom-Up Integration
• Context (2)- Useful when you can defer testing until all
components have been developed• Components developed by teams
- Early assessment of a component that must implement a critical requirement
• Fault Model- See the Integration Faults
34
Bottom-Up Integration
• Strategy- Test Model
• Dependency tree• For each component: responsibility test suite
- Do not only test responsibilities» Also exercise subcomponents
- Test Procedure• Go from the leaves of the dep tree to the root
- n levels -> n stages• Major steps (see next slide)
35
Bottom-Up Integration
1. Leaf-level components are coded. Test drivers are coded for the leaf-level components and the leaf level is tested by these drivers.
2. Components on the next higher level are coded. These components use object tested in a previous stage. Iterate for each level.
3. The entire system is exercised using the root-level component.
36
Bottom-Up Integration
37
Bottom-Up Integration
38
Bottom-Up Integration
39
Bottom-Up Integration
• Strategy- Automation
• One driver for each component• Bottom-up driver corresponds to
- Independent component- Set of components
• Testing several components with a single driver may be possible
- But you should avoid this- Reuse of drivers!- Reduce test driver maintenance
40
Bottom-Up Integration
• Entry Criteria- VM is stable- Components in each stage meet exit criteria- No anomalies interfere with integration testing
• Exit Criteria- Each component meets its exit criteria- Interface (per subcomponent) has been tested
once or more- All root-level components pass their test suites
41
Bottom-Up Integration
• Disadvantages - Driver development (huge code size)
• Promote reuse of drivers and test components
- Changes to a tested component• Revise test configuration
- No direct testing of intercomponent interfaces• Limited to component that is root of the subtree under
test
- Getting values or exceptions from lower-level components = difficult
- Critical sections are tested last• Highest level of control in root -> tested last• Unstable component indications will come when
component is exercised and not when it is integrated.
42
Bottom-Up Integration
• Advantages - Start testing as soon as any leaf-level component
is ready- Work in parallel
• Using subtrees
- Well suited to responsibility-based design• Only component interfaces are exercised.
- Find mistakes in requirements
43
Bottom-Up Integration
• Known Uses- Used with clusters- Integration by Contract
• Test design based on the relationship between client and server classes
- Object Relations• “To test a class, what other classes must be tested?”
• Read “Testing Object-Oriented Systems” for more information
44
Top-down Integration
• Intent- Add components to the SUT in control hierarchy
order, beginning with top-level control objects
• Context- Iterative and incremental approach- Control structure of the system
• Modeled as a tree• Top-level components have control responsibilities
- Control objects
45
Top-down Integration
• Context- Control objects
• Essential control strategies: high risk
- Focus on control object components first
- Incremental development• Add layer under control layer on each increment
- Concurrent hw/sw development• Develop hardware interfaces and control components at
the same time
- Concurrent sw development• Develop VM and control components at the same time
46
Top-down Integration
47
Top-down Integration
48
Top-down Integration
49
Top-down Integration
• Disadvantages - Stub development
• Large number of stubs coded• Unforeseen requirements in a lower-level component may
change many top-level components, breaking (part of) the test suite
• Stubs are implementation-specific
- Interoperability of all components in the SUT is not tested until the last component replaces its stub and the test suite passes
50
Top-down Integration
• Advantages - Early testing and integration- Reduced cost of driver development
• Only one driver instead of a driver per subtree
- Upper-level components are tested multiple times• Each time a lower-level component is added• Regression testing!
- Parallel development• Different levels of components and stubs
51
Collaboration Integration
• Intent- Add sets of components to the SUT that are
required to support a particular collaboration
• Context- Test interfaces between participants of a
collaboration- Organize integration according to collaborations- Not every collaboration must necessarily be
tested• Completed when all components and interfaces have
been exercised
52
Collaboration Integration
• Context- Indicators (or when to use collaboration
integration)• Use it when SUT has clearly defined collaborations that can
cover all components and interfaces in its scope• To demonstrate a working collaboration asap. High
technical risk or critical collaboration.• Some system require more testing than just a pass through
the top of the control hierarchy.- Example: search function with critical performance objectives
• Fault model- See Integration Faults
53
Collaboration Integration
• Test Procedure1. Develop a dependency tree
• Map collaborations onto the tree
2. Choose a sequence in which to apply the collaborations• Simplest collaborations first, complex later• Start with the collaboration that requires fewest stubs• Use a state machine that models sequential constraints on
collaborations• Test in order of risk of disruption for system testing
3. Develop the test suite for the first collaboration4. Run the test suite. Debug until the first
collaboration passes. Revise
54
Collaboration Integration
55
Collaboration Integration
56
Collaboration Integration
57
Collaboration Integration
• Automation- Single driver coupled to the interface for the
collaboration- Stubs required
• For untested components that do not participate in a collaboration. Relevant stubs are replaced with each collaboration
• Entry Criteria- VM is stable- Components in collaboration are operable- No anomalies interfere with integration testing
58
Collaboration Integration
• Exit Criteria- Each tested collaboration meets the exit criteria- All component and intercomponent messages in
the system have been exercised at least once
59
Collaboration Integration
• Disadvantages - Intercollaboration dependencies may be subtle- Participants in a collaboration are not tested
separately• Remember Big Bang Integration?
- Requirement of stubs- Testing lower-level components may be difficult- Collaborations may be incomplete
60
Collaboration Integration
• Advantages - Interface coverage in a few test runs
• Possibly without testing all collaborations
- Useful for system scope testing• Focus on end-to-end functionality
- Minimal coupling due to a single test interface- Testing and integration may begin when
components in the first collaboration are coded• Between top-down and bottom-up integration
- Minimize driver development costs
61
Backbone Integration
• Intent- Combine elements of Top-down Integration,
Bottom-up Integration and Big Bang Integration to verify interoperability among tightly coupled subsystems
• Context- Used e.g. by embedded system applications.
• Backbone provides services essential for running tests and applications
• Creating doubles for entire backbone would be impractical
62
Backbone Integration
• Fault model- See Integration Faults
• Strategy- Test model
• The top and possibly the second level are tested top-down• Application subsystems are developed bottom-up• Backbone components tested in isolation• The tested components are then big banged.
63
Backbone Integration
- Test Procedure1. Each backbone component is tested first in isolation,
bottom-up. Drivers and stubs are needed. (Fig 1)2. Application control is tested next, top-down. Stubs are
needed. (Fig 2)3. Backbone components are big-banged together and
tested under a driver. (Fig 3)4. Big Bang control components and backbone. (Fig 4)5. Use top-down on the remaining leaves. (Fig 5)
64
Backbone Integration
Figuur 1
65
Backbone Integration
Figuur 2
66
Backbone Integration
Figuur 3
67
Backbone Integration
Figuur 4
68
Backbone Integration
Figuur 5
69
Backbone Integration
• Automation- Stubs and drivers are required.
• Entry criteria- System architecture and component interfaces
have been designed and passed careful review- Backbone and application of roughly same size and
complexity.
70
Backbone Integration
• Exit criteria- All components meet exit criteria for applied test pattern- Each intercomponent interface exercised at least once.- Complete increment passed system scope test suite.
• Advantages - Mitigates disadvantages of Top-Down and Bottom-Up
(curtails their use at the point they lose effectiveness)
• Disadvantages - Requires careful analysis of system structure and
dependencies- Requires stubs and drivers
71
Backbone Integration
• Known uses- Development of Windows NT can be characterized
as a Backbone Integration followed by High-Frequency Integration.
• Related patterns- Sandwich integration:
• Combination of Top-Down and Bottom-Up• Final configuration meeting in the middle• But Backbone requires more testing of components in
isolation.
72
Layer Integration
• Intent- Uses an incremental approach to verify stability in
a layered architecture.
• Context- Verifies interoperability in systems that can be
modeled as an hierarchy that only allows interfaces between adjacent layers.
• Top layer doesn’t communicate with bottom layer• Instead, top layer communicates with middle layer and the
middle layer communicates with bottom layer
73
Layer Integration
• Fault model- See integration faults- Propagation through layers must meet hard, real-
time deadline Inadequate performance is often a critical failure
• Strategy- Test Model
• Top layer and possible second layer tested top-down.• Middle layer developed bottom-up.• Primary components bottom layer tested in isolation
74
Layer Integration
- Test procedure• May be Top-Down or Bottom-Up
1. Test each layer in isolation using drivers where needed. (Fig 1)2. Perform Top-Down/Bottom-Up Integration of layers.
• Figure 2 shows the test configurations for a Top-Down Integration, Bottom-Up would reverse the order.
• Automation- Requires stubs and drivers for each layer- Driver for top layer must be designed so it can
exercise entire system
75
Layer Integration
Figuur 1
76
Layer Integration
Figuur 2
77
Layer Integration
• Entry criteria (same as Collaboration Integration)- VM is stable, else consider Backbone Integration- Components in collaboration are operable- No anomalies interfere with integration testing
• Exit criteria- Each driven component meets exit criteria for its
test pattern- Each collaboration that traverses two or more
layers has been exercised once.
78
Layer Integration
• Advantages & Disadvantages- Similar to those of Top-Down (for the top-down
variant) and Bottom-Up (for the bottom-up variant)
• Known uses- Protocol stacks are a widely used layered application
79
Client/Server Integration
• Intent- Demonstrate stability of interactions among clients
and servers. - Initially test clients and servers in isolation, then
controlled increases in scope
• Context- Client/Server Integration achieves stepwise
verification between client components that are loosely coupled to a single server component
- Concurrent development/testing is possible
80
Client/Server Integration
• Fault model- See Integration Faults
• Strategy- Test model
• Test plan must identify clients and servers• In two-tier star basic integration strategy is client-server • In three-tier star both client-server and server-server
configurations, followed by client-server-server configuration
81
Client/Server Integration
- Test procedure• Each client is tested with stub from server• Server is tested with stubs from all client types• Next, pairs of clients tested with actual server (server may
retain stubs for other clients)• Finally, all stubs are removed and individual use case tests
are replayed.
• Two-tier client/server star integration follows a three-step procedure for each client group:
1. Representative client + server stub2. Server + client stub3. Representative client + server
82
Client/Server Integration
Figuur 1
83
Client/Server Integration
• Automation- Test script for a stage should be designed to support
two kinds of use• Should be repeatable to support regression testing upon
integration of more components• Should be extensible so system scope tests may be added
after final integration testing is completed
- Debugging/Testing tools mostly platform-specific• Difficult to perform controlled testing and debugging across
multiple platforms
- Replicate an isomorphic test environment to replicate the structure of the architecture
84
Client/Server Integration
• Entry criteria- VM is stable, else consider Backbone Integration- Components in collaboration are operable- No anomalies interfere with integration testing- Multiplatform test suite installed and working- Multiplatform version control system
• Exit criteria- Each driven component meets exit criteria for its test
pattern- Interface to each subcomponent exercised at least once- Test suite passes in test environment that is isomorphic
with respect to the target environment
85
Client/Server Integration
• Advantages - Avoids problems of Big Bang Integration- Order of client/server integration usually has few
constraints integration sequenced by priority/risk
• Disadvantages - Cost of driver and stub development- Testers cannot exercise end-to-end use cases until
midway or late in development cycle.
• Known uses- Read book for extensive example
86
Client/Server Integration
• Entry criteria- VM is stable, else consider Backbone Integration- Components in collaboration are operable- No anomalies interfere with integration testing- Multiplatform test suite installed and working- Multiplatform version control system
• Exit criteria- Each driven component meets exit criteria for its test
pattern- Interface to each subcomponent exercised at least once- Test suite passes in test environment that is isomorphic
with respect to the target environment
87
Integration Patterns
• Distributed Services Integration. Exercise a loosely coupled network of peer-to-peer components. Appropriate for decentralized networks containing peer-to-peer nodes.- See book for more info
• High-frequency Integration. Develop and rerun an integration test suite test hourly, daily, or weekly. Can be applied at almost any scope or architecture.- See book for more info
top related