Tackling Software Development Bottlenecks Continuous Test and Continuous Delivery

Presented by:

Speaker: Steve Weaver

Role: IBM Rational Solution Marketing Manager, Cloud & DevOps

IBM’s statements regarding its plans, directions, and intent are subject to change or

withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product

direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise,

or legal obligation to deliver any material, code or functionality. Information about potential

future products may not be incorporated into any contract. The development, release, and

timing of any future features or functionality described for our products remains at our sole

discretion.

Performance is based on measurements and projections using standard IBM benchmarks

in a controlled environment. The actual throughput or performance that any user will

experience will vary depending upon many factors, including considerations such as the

amount of multiprogramming in the user’s job stream, the I/O configuration, the storage

configuration, and the workload processed. Therefore, no assurance can be given that an

individual user will achieve results similar to those stated here.

Software quality is a major problem across all industries

• Software is blamed for more major

business problems than any other

man-made product.

• Poor software quality has become one

of the most expensive topics in

human history

–$150+ billion per year in U.S.

–$500+ billion per year worldwide.

• Projects cancelled due to poor quality

are 15% more costly than

successful projects of the same size

and type.

Source: Capers Jones, 2011

Based on 675 companies, 35 government/military groups, 13,500 projects, 50-75 new projects/month, 24 countries, 15 lawsuits

4

Breaking down the costs

Tools & labor to prevent and

identify defects (reviews, training, process, testing, etc.)

(1) IDC - Worldwide and U.S. Discrete Testing Services 2012–2016

(2) Ovum worldwide market for software and systems testing services

(3) Capers Jones – Software Quality in 2012: A survey of the state of the art

(4) NIST - The Economic Impacts of Inadequate Infrastructure for Software Testing

Downtime, recalls, etc.

(e.g. $390M Knight Capital,

$3B Toyota Prius antilock system)

Debugging, rework,

regression testing

Adding more Testers is not a sustainable model Labor costs continue to increase

Testers

Testers

Testers

Development

Team

Testers

hybrid

Network Latency

Security/Privacy

Availability

Cost

Data Integrity

Adopting technology only increases testing complexity Test environment setup is harder and more expensive than ever before!

external

users internal

users

private

Business Partner

Services

public

External 3rd Party

Services

Application

Layer

Mainframe ERP

Systems

Custom

Applications Databases

Enterprise Service Bus

Functionality Usability

Performance/Scalability

Reliability Consumability

Faster and more frequent builds are not enough The transformation to Agile isn’t benefitting Testers

Tester

Setup test environment

Developer

Write & deliver code

Nightly build(s) Compile, unit test, publish

Developer

Developer

Developer

Developer

Developer

“N” days to install & configure

“N” nightly builds are piling up

Quality

Gate BVT

Multiple

Test Environments

FVT

Perf Test

Security Test

SIT

UAT

I just spent 2 weeks setting this up &

the build is no good?

When agility stops at testing, the entire

organization hits the wall

8

Development Velocity Testing Labor Systems Complexity

What’s causing the spend increase?

Outsourcing labor is no longer

a sustainable model as global

wages are increasing

Product and application

complexity are driving

test lab cost increases

Test teams can no longer

keep up with agile

development

Heterogeneous Environments

Public Cloud Private Cloud

Data Warehouse Mainframe Enterprise

Service Bus

Directory Identity

File systems

Collaboration

Web/ Internet

Routing Service

Third-party Services Portals

Content Providers

EJB

Shared Services Archives

Business Partners

Messaging Services

Traditional

Testing

Other costs

Testing costs

Impact of evolving customer and market expectations

User Experience Today Emerging

Primary Workload Type Systems of Record (Transactional)

Systems of Engagement (+ Record) Big Data, Analytics, Mobile/Social Channels

Delivery Model Planned Incremental (DevOps)

Development and Operations Team Sizes

100s and costly 10s with built-in DevOps automation

Release Frequency Months to years Days to weeks, based on business opportunity

Integration Frequency Weeks Continuous

Infrastructure Deployment Days Minutes

Time to Value Planned Opportunistic

Operational Model Systems management Built into application, recovery-oriented computing, continuous availability

Service Sourcing Develop Consume and assemble (public and private)

Feedback Loop Periodic Continuous interaction with clients

9

10

What’s going wrong?

Failures due to differences in dev and

ops environments

Time to market pressure for more frequent releases

Manual (tribal) processes for release

lack repeatability/speed

Major releases take days, 100 people and are

managed by a spreadsheet

Daily

Build

Release

Who did

this last

time?

Dave…

Dave’s

not here

man…

Dev

Prod

I’ll order

breakfast

11

Solution: A Continuous Delivery Pipeline

Development Testing Staging Production Business

Owners Customers

Validate on more production-like

conditions earlier

Automate hand-offs/promotions to

increase velocity through the

different stages

Standardization on processes and

assets between Dev and Ops

Automated monitoring and

dashboarding of quality and

performance against service level

agreements at multiple stages

Ensure applications are production-ready throughout the lifecycle

and can be released at any time while minimizing rollback due to quality issues

IBM Release and Deploy

• Reduce errors: Automated software

release and deployment

• Improve productivity: Push-button

deployments for developer and

operations

• Faster time-to-market: Automated

release and deployment with built-in best

practices provides

• Compliance and auditability: Enforced

Security and traceability

Versioned Artifacts

QA Environment

Production Environment

Development Environment

Artifacts Artifacts Artifacts

Execute Against Execute Against Execute Against

QA Release

Product Release

Development Release

Application Blueprint

UrbanCode complements IBM’s DevOps solution enabling clients to more rapidly deliver software

12

IBM UrbanCode Deploy

Deployment Automation

• Orchestrates and automates the

deployment of applications, middleware

configuration, and database changes into

development, test, and production

environments,

• Accelerates time to market and reduces

cost by shrinking manual efforts from days

to minutes with automation, while reducing

risk by cutting deployment errors by up to

90 percent.

13

Features/Business Value

• Automated application deployment and orchestration

• Environment and configuration management

• Store deployment artifacts more securely to enable traceability

• Middleware configuration management

• Security, approvals and notifications

14

IBM UrbanCode Release

Release Automation

An intelligent collaborative release management solution that replaces error-prone, manual spreadsheets and streamlines release activities for application and infrastructure changes, allowing clients to:

Efficiently manage environments

Track both the application and infrastructure changes in the release

Orchestrate the release of interdependent apps using application- and team-centric processes

Features and Benefits

Enforce release quality gates for environments.

Orchestrate changes across multiple applications.

Manage infrastructure in addition to application changes.

Provide real-time status for long production releases.

Track the allocation of environments to releases.

Dev

Dev

Providing Integrated Test Environments is hard

Challenges • Lack of Availability and access of

dependent components & systems

• Managing complexity multitude of

environments and configurations

• Mismatch in speed of dev changes

and ability to provide environments

• Cost, Skills and Complexity to setup

Impact: Fewer environments available limiting integrated

testing to late in the lifecycle

Week 8

Week 4 1 hour / day

Build 10 times / day

PVT Dev

FVT

FVT FVT UAT

Continuous Testing Complements

Release and Deployment

What Testers really need! Validating today’s complex applications

Continuous Testing

DevOps Foundation

Test Automation

• Run unattended

• Validate business flows

across channels

• Durable, maintainable,

resilient to change

Faster

Orchestration from build to production deployment

• Reliable • Repeatable • Traceable Cheaper

Test Environments

• Realistic & complete

- (production like)

• Available on demand

• Inexpensive to setup

Better

Solving the Test Environment dilemma Making production-like test environments a reality

Traditional test environments are both underutilized and insufficient – Environments are expensive: hardware, software, and labor

– Tester and developer time wasted waiting for appropriate environment

Agile test environments require four enabling capabilities

Dynamic Infrastructure

Deployment

Automation

Test Environments

DevOps Foundation

Service

Virtualization

Test Data

Management

Organization Process Technology

an agile test organization and testers need continuous access to test data

with agile development you have continuous integration and delivery and have to test often

test data management software needs to support agile method by streamlining access to test data and having insight into test data

Agile testing needs continuous

access to test data

Without Test

Data Management

With Test

Data Management

Sits in queue for days

and take several days

to create

Submits request for test data

Sits in queue for days

Create test data

Tester

DBA

Takes several days to

create

DBA Create or Refresh

test data

Tester -Use test data in testing

-Request data refresh

Takes hours to create

Tester -Use test data in testing

Refresh test data -

Submits request for test data

Sits in queue for days

Create test data

Tester

DBA

Automate creation and refresh of test data

What is Service Virtualization? Making the unavailable available for testing

Virtual components simulate the behavior of a service or application during testing

– System dependencies are a key challenge in setting up test environments

– Unavailable/inaccessible services slow down the testing process

– Costly 3rd party access fees can significantly impact testing availability

Virtual components run on commodity hardware, private cloud, public cloud

Each developer and tester can easily have their own test environment

Developers and testers can continue to use current testing procedures and tools

Streamline creation of production-like environments and negative testing

Heterogeneous Environments

Public Cloud Private Cloud

Data Warehouse Mainframe Enterprise

Service Bus

Directory

Identity

File

systems

Collaboration

App Under Test Routing

Service

Third-party

Services Portals

Content

Providers EJB

Shared

Services Archives

Business

Partners

Messaging

Services

Databases Mainframe

applications

App Under Test

Third-party

Services

Packaged apps, messaging services, etc.

Virtual Components

Continuous Testing improves software quality

Actual Service/App

Virtual Service/App

Service Virtualization is an enabler for continuous testing

Services, applications, systems are introduced into the continuous integration cycle in a prioritized, controlled fashion.

Controlled integration helps isolate defects for faster resolution.

Using virtual components Enables testers to continuously test what they want, when they want,

how they want

Reducing the cost of testing Deploy what has changed, virtualize the rest

Scenario: The team is making changes to an application component adding

integration with an existing 3rd party service to implement new required

functionality. To validate this minor change, the time to stand up a complete

test lab combined with the 3rd party service access fees adds substantial

project cost – cost which is unnecessary.

Service virtualization lowers the cost of testing and eliminates delays.

Create and deploy a virtual

component to simulate the 3rd party

service functionality and performance

without the fees.

Quickly stand up test labs using virtual

services to simulate dependencies

which have not changed.

Test applications end to end with confidence

but at a lower cost and without delay.

Testers

Business Partner

Services

3rd Party

Services

Enterprise Service Bus

Mainframe Database ERP

System

Custom

Application

Virtual

Service

Virtual

Service

App Layer

Virtual

Service

new

The ROI of Service Virtualization The results are real!

3 Year Benefits:

• Increased project delivery capacity 100% (40 projects to 80 projects)

• Avoided hiring three additional FTE's, saving almost $600k.

• Reduced incidents found in production from 2.5 to .3 per project, saving $3 million

• Identified defects earlier and freed the team up to focus on new capabilities, saving over $560k

• Reduced consulting and third party testing fees with better test automation, saving $408k

• Eliminated the need for additional hardware and labor to set up test environments, saving $700k

• Generated new revenue for the business – estimated at tens of millions!

A major European bank implemented service virtualization and automated integration

testing as part of an Agile transformation to enable continuous testing. As a result of

IBM capabilities, their system integration test (SIT) process was reduced from 3

weeks down to four hours!

The Total Economic Impact™ of IBM Rational Service Virtualization and Test Automation Solutions, a July 2013 commissioned study conducted by Forrester Consulting on behalf of IBM

IBM UrbanCode Deploy orchestrates and automates the deployment of

applications, middleware configuration, and database changes into

development, test and production environments.

Databases Internal Messages

Third-party Services

virtual components

Simultaneously

test across

multiple test

stages

Dev QA

IBM Rational Test

Virtualization

Server

Orchestration from build to production Reliable, repeatable, & traceable deployment automation

IBM UrbanCode

Deploy

IBM Rational

Test Workbench

integrated with

Test Environments

Dynamic Infrastructure

Deploy what is ready,

virtualize the rest

Continuously test in

production-like env.

Test using real world

network conditions n SHUNRA Network

Virtualization

What Testers really need! Validating today’s complex applications

Continuous Testing

DevOps Foundation

Orchestration from build to production deployment

• Reliable • Repeatable • Traceable

Test Environments

• Realistic & complete

- (production like)

• Available on demand

• Fast & inexpensive to

standup & setup

Test Automation

• Unattended execution

• Validation of business

flows across channels

• Durable, maintainable,

resilient to change

Develop and Test solutions

Measureable results and business value

29

Communications Service Provider

Revamp manual approach to decrease

testing time and increase quality

Solution: Rational Test Workbench

with IBM Green Hat

Results: Increased team efficiency by 300%,

and decreased test time 25-40%

A leading global financial services

firm with assets of over $2 trillion

Many disparate legacy format to be

migrated to next gen payments system

Solution: Virtualized third-party systems

otherwise unavailable for testing

Results: Reduced manual testing from 10 days

to 10 minutes, saving over $7 million to date

Major U.S. insurer

Recognized manual test efforts insufficient

for complex SOA and web services

Solution: Agile middleware solution created

to match the legacy systems’ functionality

Results: Reduced testing by 95% to 2 hours;

reduced ‘rate filing’ validation by 94%

to 320 hours

Global manufacturer acquires

competitor

Migrate off rented infrastructure onto

company’s standardized middleware platform

Solution: Virtualized critical, unavailable systems

during migration

Results: Fully integrated in 6 months – 2 months

ahead of schedule; saved significant rental costs

and dependencies on third-party systems

Release and Deployment solutions

Deliver measureable business results

30

International Investment Firm Driving Down Costs

Release process required considerable effort

and delayed by manually-introduced errors

Solution: Automated release process

Results: Cost avoidance of over $2.3M/year,

reduced release time from 2-3 days to

1-2 hours and virtually eliminated test team

“down-time”

Higher Education Speeding Time-to-Market

Agile development teams constrained by

slow deployment to dev. and test environments

Solution: Accelerate deployment by enabling

development teams to self deploy with

automation

Results: Deployments cut from hours to

minutes and a greater number of servers

with fewer resources

SaaS Software Provider Reducing Risk

Difficulty managing multiple customer

configurations and versions of software

deployed across servers

Solution: Automate managing configuration

and version deployment

Results: Execute customer specific releases,

reduced deployment outages by over 90%

Online Retailer Speeding Time-to-Market

Significant delays getting application

changes to production

Solution: Scaled up continuous deployment

Results: Deployment time reduced by

over 95% with easy scale and deploying

to over 250 servers within 2 months

of implementation

www.ibm.com/ibm/devops/us/en

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

Line of Business

Delivery pipeline for mobile and mainframe Incrementally adopt when/if needed

Rational Team Concert

Rational Focal Point

Rational Requirements Composer

SmartCloud Orchestrator

IBM UrbanCode Deploy

IBM UrbanCode Release

Rational Quality Manager

Rational Test Virtualization Server

Builder

Studio

Rational Developer for System z

Rational Development & Test

Environment for System z

IBM Mobile Quality Assurance

Automate deployment to IBM MobileFirst

Platform (Worklight)

SCM Build Artifact Repository

Worklight

Application Center

Console

Process

.wlapp

adapter Binary (IPA/APK)

Automatically deploy Worklight applications as part of your deployment process

The plugin supports:

•deploying Worklight adapter and application components to the Worklight Server

•deploying Worklight mobile application binaries to the Worklight Application Center for distribution and management

IBM UrbanCode Deploy Development

Continuous Delivery with Cloud

IBM UrbanCode Deploy SmartCloud Orchestrator

IBM Pure Systems Build Artifact Library

Application Resource Template

Build Deploy Provision

Application binaries (versioned)

Environment configurations (versioned)

•Automate provisioning of environments as part of the end-to-end delivery process

• How? Establish and automate the deployment of an “Application Blueprint” with “Resource Templates” imported from “Cloud Patterns”.

•Rapid application deployment for quicker feedback and testing to ensure high quality and faster releases

• How? Using a seamless process flow for incremental, full stack provisioning and application deployment automation

Application lifecycle management with cloud

SCM Build Automation

IBM Endpoint Manager

DEV QA PROD

Provision Platform

Patch/update

Publish build

Pull changes

IBM UrbanCode Deploy SmartCloud Orchestrator

Platform Config Management

Deploy early and often to ensure high quality and faster releases

using repeatable, reliable, and

managed automation

Execute application deployment and manage settings

across environments