Enabling Agility with Continuous

Integration Testing

Monica Luke September 18, 2012

WELCOME

IBM’s statements regarding its plans, directions, and intent are subject to change or

withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product

direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise,

or legal obligation to deliver any material, code or functionality. Information about potential

future products may not be incorporated into any contract. The development, release, and

timing of any future features or functionality described for our products remains at our sole

discretion.

Performance is based on measurements and projections using standard IBM benchmarks

in a controlled environment. The actual throughput or performance that any user will

experience will vary depending upon many factors, including considerations such as the

amount of multiprogramming in the user’s job stream, the I/O configuration, the storage

configuration, and the workload processed. Therefore, no assurance can be given that an

individual user will achieve results similar to those stated here.

Agenda

• Integration Testing Challenges

• Continuous Integration Testing

• Agile Integration Testing

Taking a new look at integration testing

.94

.92

.98

x3

.97

Given high

individual

systems’

reliability

(>.92)

Unreliability

increases

exponentially:

R = .79

Do you know

what’s

underneath the

surface?

“In practice, integration has proven to be the most enlightening activity in

exposing the risks and uncertainties in software development.”**

**Walker Royce, Chief Software Economist at IBM

4

Balancing

Quality and Speed

Increasing

Cost of Quality

Increasing

Development Complexity

An estimated 60 - 80 percent of the cost of software development is in rework*

* Source:http://www.sei.cmu.edu/about/message/

Outsourcing labor is no longer

a sustainable model as global wages

are increasing

Product and application

complexity and size

are increasing

Productivity is inhibited as test

teams can no longer keep up with

agile development

Heterogeneous Environments

Public Cloud Private Cloud

Data Warehouse Mainframe Enterprise

Service Bus

Directory Identity

File systems

Collaboration

Web/ Internet

Routing Service

Third-party Services

Portals

Content Providers

EJB

Shared Services Archives

Business Partners

Messaging Services

Traditional

Testing

Other costs

Testing costs

In-house Development

Service-oriented Architecture

(SOA)

Cloud / SaaS End-user Web Application

Mobile Device Access

Third-party IP

Partner Code

Commercial Off-the-shelf

Directory Service

Outsourced and Globally Distributed

Open Source

Main Frame

Heterogeneous systems are complex

And… hard to configure for integration tests

Data Warehouse Mainframe

Directory Identity

File systems

Collaboration

Web/ Internet

Routing Service

Third-party Services

Portals

Content Providers

Shared Services

EJB

Archives

Business Partners

Messaging Services

Enterprise Service Bus

Application

Does your organization look like this?

Tester

Setup integration test environment

Developer

Developer

Developer

Developer

Developer

Developer

N days to install & configure

I just spent 2 weeks setting this up &

the build is no good?$%^??

N nightly builds are piling up

What if we could change it? Takes up to 50% of test cycles

Test Setup is minutes Time spent on meaningful work expands

Package ready to release

Iteration 1… n

Days Days Days Days

Nightly build

Fix Defects Run Test Find Defects

Develop

Agile

QA

Independent Test

Stage

Production

Check ins

Deliver App Package into Production

Test & Verify

Integration Test

Test & Verify

Delay

Manual setup of Test Env

Delay

Manual setup of Test Env

Delay

Manual setup of Test Env

Delay

Manual setup of Test Env

Agenda

• Integration Testing Challenges

• Continuous Integration Testing

• Agile Integration Testing

11

What is Service Virtualization?

•Virtual Services simulate the behavior of an entire application or system

during testing

•Virtual Services can run on commodity hardware, private cloud, public

cloud

•Each developer, tester can easily have their own test environment

•Developer and testers continue to use their testing tools (Manual, Web

performance, UI test automation)

Cap

ture

& M

od

el

Heterogeneous Environments

Public

Cloud Private Cloud

Data Warehouse

Mainframe Enterprise

Service Bus

Directory

Identity

File systems

Collaboration

App Under Test Routing Service

Third-party

Services Portals

Content Providers

EJB

Shared Services Archives

Business Partners

Messaging Services

Databases Mainframe applications

App Under Test

Third-party Services

Packaged apps, messaging services, etc.

Virtual Services

Incremental Integration Testing

Actual Service/App

Virtual Service/App

Services, applications, systems are

introduced into the continuous integration

cycle in a prioritized, controlled fashion

Testing integrations as an evolution

12

Continuous integration Testing Example

C1 C2 C3 ERP WSDL 3rd party UDDI Pass/Fail

Real V V V Test my

own piece

• Test C1 with three virtualized services.

• Can use simple or complex integration scenarios.

• Quick to setup and low-cost.

13

Time

Real V V V R

Real V V V V

Real V V V V V V

Real V V V V R R

Test my

own piece

Integrate

w/another

This won’t

slow me down!

First Complete

System Test

All components

Integrated!

• Integrate C2 and C3… with virtualized backend systems

• Use enterprise integration scenarios for Testing, inc BVT!

• Quick to setup and low-cost.

14

Continuous integration Testing Example

C1 C2 C3 ERP WSDL 3rd party UDDI Pass/Fail

Real V V V

Pass/Fail Time

Real V V V

Real V V V R

Real V V V V

Real V V V V V V

Real V V V V R R

Real R R V V R R

Real R R R R R R

Test my

own piece

Integrate

w/another

This won’t

slow me down!

First Complete

System Test

All components –

Integrated!

The expensive

backend

All Systems UP!

• Gradually integrate with expensive backend

systems.

• Minimize use of backend systems for testing

purposes

• Process offers easy and cost effective defect

isolation

C1 C2 C3 ERP WSDL 3rd party UDDI

15

Continuous integration Testing Example

Continuous Integration testing for builds

Green” builds have

passed meaningful

integration tests

Compile Unit Test Install & Configure

Integration Test

Environment

Publish Results Run Tests

Rational Quality

Manager

Agenda

• Integration Testing Challenges

• Continuous Integration Testing

• Agile Integration Testing

Complex systems Staged agility

Iteration 1 Design, Code, Test Weeks 2 - 6

Integration test

iteration #1

Iteration planning

Week 1

Iteration 2 Design, Code, Test Weeks 8 -12

Integration test

iteration #2

“Stable, working code”?

• Are integration tests pushed later and later in the release cycle?

• Collaboration challenge: Expectation is now that test is aligned with

development

• Are developers doing integration validations?

• Are you compromising on the

definition of stable, working code?

End-to-end Lifecycle Management

RM DM SCM Build Stage QM Monitor

Capabili

ties

CM

Planning

Design Develop Stage Production

Life

cycle

A

cce

lera

tors

Agility@Scale Continuous Delivery

Agile Agile

with Service Virtualization

• Whole team collaboration

• Dev & Test are working on the same code base at the same time

• Change the perspective on team metrics

• Time to fix defects vs number of defects

• Integration testing is happening on every milestone and build

• Whole system testing and verification

• In Agile terms, you are really “done, done, done” on every milestone

• Alignment builds trust delivering on agile vision of self-directed teams

Whole team alignment

22

Rational Test Workbench

Enable functional, performance and integration testing

throughout your project lifecycle

With a scriptless, wizard-driven test authoring

environment and support for more than 70 technologies

and protocols

Rational Test Virtualization Server

Model real system behavior to eliminate test

dependencies and simulate production,

Accelerate testing and reduce the setup and

infrastructure costs of traditional testing environments.

Rational Performance Test Server

Reuse your functional integration tests for performance

testing

Combine your tests to model real-world scenarios and

assess the impact of load on your integrated

infrastructure.

Test

Workbench

Functional Testing

Integration Testing

Performance Testing

Regression Testing

Test

Servers

Load Agents

Virtualization Agents

QA staff

Test Lab

Infrastructure

Introducing IBM Rational’s Service Virtualization and

Test Automation Solutions

23

IBM Rational Quality Management Offerings

Storage

Collaboration

Search & Query Discovery

Administration: Users, projects, process

Presentation: Mashups

Best Practice Processes

Manage

Test Lab

Create

Plan

Build

Tests

Report

Results

Execute

Tests

IBM Collaborative Application Lifecycle Management

Test Management

Rational Quality Manager

Quality Dashboard

Requirements Management Defect

Management

Open Lifecycle Service Integrations

Performance Testing

Functional Testing

Unit Testing Test Data

Management Integration Testing

Test Virtualization

Rational Integration

Tester Rational

Functional Tester Rational

Performance Tester

Rational Test

Virtualization

Optim

TDM

www.ibm.com

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

www.ibm.com