© 2013 ibm corporation continuous delivery implementation internal adoption and devops proceses and...

© 2013 IBM Corporation

Continuous Delivery ImplementationInternal adoption and DevOps proceses and tools within enterprise projects

Robbie MinshallIBM Software, RationalJune 20 2013


Agenda

https://jazz.net/wiki/bin/view/Main/ContinuousDeliveryImplementationMeetings

Purpose:

Describe and discuss the scope, initial thoughts and working environment for Continuous Delivery Implementation

Agenda:

5 min: [Frank] Welcome and why this is important

15 min: [Robbie] Project overview and strategy

25 min: [All] Discussion. Open for all to state a perspective, describe problems or opportunities that they are excited about. 5 min limit per topic.

10 min: [Robbie] Next steps

2


33Legend: *: Contractor^: Co-op~: Supplemental

Dibbe Edwards Rational Development VP RTP, USA

Dibbe Edwards Rational Development VP RTP, USA

Frank Varone Manager Continuous Delivery Infrastructure RTP, USA

Frank Varone Manager Continuous Delivery Infrastructure RTP, USA

Robbie Minshall – Chief Architect

Mary Yost – Project Manager

Kevin Williams – Cloud/pattern Architect Brad Herrin – Pattern / AutomationJennifer Liu – Pattern / AutomationJohn Gambino^ - Automation

UK (PfMgr: Paul Strachan)Kenneth Thomson – Pattern / Automation

Canada (PfMgr: Rodolfo Napoles)Tanya Wolff – Pattern / Automation

Maneesh Mehra - Pipeline Automation leadThomas Picoli - SSE / Pipeline AutomationThomas Neal - Pipeline Christophe Elk – JAZZ IHD

Extended Team Members:Jason Vasilew – IWD/ SCP / TSAM

Martha DasSarma – Build Lead keeping the clm builds running

India (PfMgr: Pradeep Balachandran)Divya Vidyadharan – Build

India (PfMgr: Krishna Kishore)Sanket Jain – Build

Canada (PfMgr: Judy Keen)Li Zhu – Build

Mexico (PfMgr: Guadalupe Torres)Dainer Zavala Priego – Build

Chris Daly – Install LeadCorinna Schultz – Install

UK (PfMgr: Paul Strachan)Mark Anderson – Install Alan Forbes – Build

Zhi Cheng LiuXiang Yue GaoXue Po WangTing Hui TH LITBH 2 in July

TBH 3 to do next version work

Yi Ming W Wang^ Sheng Rong SR^

Xue Ming Zuo (Eric)Dev Mgr; Workload Patterns(Beijing, China)

Xue Ming Zuo (Eric)Dev Mgr; Workload Patterns(Beijing, China)

Extended Team Members: Rajeev Sikka – Install Allen Zhou – Install Christopher Maguire - Build Peter Steinfeld - Build Architect Chad D Montgomery - new open source prototype (Maven based) - new build tooling strategy … Nathan Bak – Jazz Platform Guadalupe Torres – non clm builds

???Bulid & Install ManagerRTP, USA

???Bulid & Install ManagerRTP, USA


Market shifts are fundamentally changing the way businesses approach software driven innovation

Macro Business EnvironmentIncreasing regulatory environments drive

the need to reduce risk

Macro Business EnvironmentIncreasing regulatory environments drive

the need to reduce riskEmpowered UsersConsumerization of IT drivesthe need for apps with high quality customer experience

Empowered UsersConsumerization of IT drivesthe need for apps with high quality customer experience

Technology TrendsMobile, social, big data, cloud, intelligent/connected systems

drive the need for agility

Technology TrendsMobile, social, big data, cloud, intelligent/connected systems

drive the need for agility

Mobile

Cloud

Intelligent/Connected Systems

Social

Big Data

1. Clients are trying to rapidly innovate and become agile with systems of engagementAt the same time need to evolve their systems of records and manage regulatory compliance, reduce costs by outsourcing and provide world class customer experience

2. The above factors have resulted in shift in purchase dynamicsDevelopers influencing LOB buyers for entire delivery stack (cloud, mobile, middleware, tools)

Multi-sourcingIncreasing outsourcing drive

the need for governance

Multi-sourcingIncreasing outsourcing drive

the need for governance

4


Organizations that effectively leverage software innovation outperform their competitors... yet few are able to deliver it effectively

5

86%

of companies believe software delivery is important or critical

25%

leverage software delivery effectively today

But only…

Source: “The Software Edge: How effective software development drives competitive advantage,” IBM Institute of Business Value, March 2013

69%

outperformthose who don’t

of those wholeverage software

delivery today


Why is this important

This is really really hard

Continuous Integration, Test and Delivery of Software within non trivial landscape

CLM Relevance

– 4th Major release

– Significant Legacy

– Historically this is the point where products struggle to delivery incremental value and innovation

Rational

– Customers recognize that innovation is essential to their business and that software is the important part of that (first time that it has ranked top of the list)

– Customers are looking for ways to increase speed of software innovation, the developer is king

– Rational should be as well positioned in this space. We are not.

6


Executive Perspective Rational Continuous Delivery adoption

Lifecycle Measurements 2008 2010 2012 – 2013 Total Improvement

Project Initiation 30 days 10 days 2 days 28 days

Groomed Backlog 90 days 45 days On-going 89 days

Overall TTD 120 days 55 days 3 days 117 days

Iteration Length 6 weeks 4 weeks 4 weeks 4 weeks

Number of Iterations 6 8 3 N / A

Composite Build Time 36 hours 12 hours 8 hours 400 %

BVT Availability N / A 18 hours < 1hour 17 hours

Iteration Test Time 5 days 2 days 4 hours 4 days

Total Deployment Time 2 days 8 hours 2 hours 2 days

Overall TTP 9 days 3 days 15 hours 8 days

Time Between Releases 12 Months 12 Months 3 Months 9 Months

Drive adoption

Measure and Optimize Value

External Deliverables

DevOps Transformation ProjectDevOps Transformation Project

Consume and Drive

Processes and Culture

© 2013 IBM Corporation88

Continuous Business Planning

Level 3 Level 4 Level 5Level 2Level 1

Continuous Release and Deployment

• Quality-gated automatic promotions

• Deployment of all application artifacts (e.g. database updates)

• Coordinated SOA/multi-tier deployments

• Ad-hoc deployment scripts

• Self-service deploy to test environments

• Mostly standardized deployments

• Self-service deployment to test and production environments

• Fully standardardized deployments

• Continuous deployment to production

Continuous Monitoring

• Automated performance tests as part of quality gates

• Reactive performance resolution (e.g. outage alerts)

• Performance instrumentation and monitoring

• Real-time performance dashboards

• Continuous performance optimization

Continuous Customer

Feedback and Optimization

• Advanced business impact analysis

• Big-data analysis• Automatic crash reporting

• User session recording and reporting

• In-app customer feedback

• Log analysis

• Measuring customer experience KPIs

• A/B, Split and Multivariate testing

• Voice of the customer investigation

• Self-evolving application based on A/B testing

• Real-time dashboard of business value attainment

• Integration of business and customer metrics

• Business objectives documented in silo

• Business objectives linked to development work items

• Ability to objectively trade off requirements

• Visibility to development status

• Full implementation of lean principles

• Real-time integration entire lifecycle spanning from business objectives to customer sentiments

Collaborative Development

• Build verification tests on production-like environments

• Advanced SCM (efficient personal branch/merge)

• Lifecycle traceability

• Ad-hoc build scripts

• Siloed developers

• Bare-bone SCM (e.g. non-atomic commits)

• Nightly builds

• Usage of methodologies (Agile, Scrum…)

• Basic SCM (basic tagging, merging)

• Build on commit

• Publish to asset repository

• Real-time work item planning

• Development intelligence

• Continuous process improvement

Continuous Testing

• Some test automation • Use quality management tools

• Continuous integration testing w/service virtualization

• Static code analysis

• Security testing

• Fully automatic acceptance tests

• Coverage analysis

• Risk-based testing

Industry norm

Target attainment

Continuous Delivery Example Maturity Model

Will be updated!!!!!


Back to the future: WAS Adoption of WCA

Background

Realized Results

Projected Results

Background and ChallengesShorten time delivery time via Agile Processes Provide meaningful dev & test environmentsIncrease Hardware Utilization Decrease system admin costs

WCA SolutionConstruct Private Cloud leveraging WCA Seed Cloud with 6% of InfrastructureAdopt and scale to 50% over 4 years

Realized Benefits Deployment reduced from 3+ hours to 20 minutesHardware Utilization increased from 6% to 60%User Deployment failures down from 50% to 5%Complete Performance Environment as PatternsIncreased system admin efficiency 16 fold

Realized Results 1Q 2009 – 2Q2010 355 K in Labor efficiency gains160 K in Infrastructure savingsEnabled Results due to Process Changes2.1 million Labor efficiency gains780 K / year downstream regression test savings

Projected CloudBurst Adoption Rate : 2010 = 6%,2011=20%,2012=40%,2013/14=50%

Benefit Summary:Setup = 24K Training = 36K 5 year return on investment = 10.4 Million


WebSphere deploys private cloud driving Agile Adoption and gain efficiencies throughout delivery

Client Pains Enable rapid access to complete testing

topologies to improve quality and decrease costs

Drive down costs and increase utilization of infrastructure

Back to the future

Real Results 1,700 K in direct savings, 2.1 Million/year in enabled efficiency.

Reduced topology install time from 3 Hours to 20 Minutes. Increased hardware utilization by 54% . Increased System Admin efficiency by 1600% compared with bare metal.

Interact


Back to the future: Recipes for success and lessons

Improve developer experience and functional stability

Realize internal efficiencies

Adopt and drive IBM DevOps technologies

Produce a functioning DevOps Reference Implementation

Contribute to IBM thought leadership


To be honest we either never quite bought in, or were never successful with Agile Development. Frankly, while our development teams may have been agile that did not really translate as we had hoped into faster application delivery. We really struggled with how to engage our operations team and to meet the speed requirements for social and mobile applications. There was always this tension between the development of those next generation applications and how that related to our traditional systems. Once you introduce public and private cloud technologies into the equation we were lost and were looking for a model based on real results. I was sick of hearing the words Agile, DevOps and Cloud, help me know how to realize some results.

The Rational DevOps Transformation project helped us make sense of this mess and I loved the transparency. Think of it as a DevOps Reference Architecture, Adoption Path, Business Value Assessment and Deployable solution that not only tells the story of IBMs transformation but provided a blueprint for ours.

It was first brought to our attention by our Executives who pretty much wanted to duplicate the results. We were skeptical but their transition story was transparent and helped us understand a set of best practices and lessons.

One of the key takeaways was that we needed to invest in this transition. Ultimately there was of course exciting new technologies to adopt but also changes in our organization and lines of business were necessary. The business value assessments created a model that helped us with these adoption challenges.

From a technology standpoint we were able to pilot DevOps solution very quickly because we were able to take what IBM used internally and deploy it quickly in order to demonstrate the value we were looking for.

From there it was a matter of integrating the functioning solution with out existing systems and creating a strategy for realizing value in our Systems of Record as well as our emerging applications.

If you are looking for validation that DevOps is achievable based upon real results in a non trivial environment then check out the DevOps Transformation Project and IBM DevOps technologies.

Desired Customer Perspective for Transformation Effort


App Scan

Integration Test

Usability

~ 1 week

Developer Interaction

Deployable Solution on Private Cloud Production

Jazz.net Production

JazzHub Production

Release Dashboards

Development RepositoryUnit Test

Smarter Pipes Service

Function Test Performance Test

System Test

Integration Test

Production like

uBuild

Production like

Manual Tests

Rational Collaborative

Application Lifecycle

Management

Drive adoption




Consume and Drive



Project Structure

14


Emerging timeline

15


Ways of working

T skills

– Deep technical knowledge

– Broad awareness and contribution

Deliver

– 90% of what YOU do should have a known customer or developer deliverable when you start

Collaborate

– Engage the organization

Win

– We will be measured by the developer experience

16


Discussion

17

5 minute topics

Open for all to state a perspective, describe problems or opportunities that they are excited about. 5 min limit per topic.

No Whining


Discussion Point: Smart Pipes

Problem: static test pipe of increasing complexity/value is too slow for continuous integration

Constantly quantify the value of a test case in terms of it's reliability, it's history of finding defects, the variety of systems that it finds defects in, and whether or not it is related to change sets. For various build/integration purposes have a defined regression window. Within that window dynamically organize the pipeline in the most optimal way to ensure the driver is stable, to find regressions as quickly as possible and to begin expensive or time consuming tests cases such as long runs earlier enough to matter.

Said another way run the best tests to ensure stability, don't run stuff that never matters, run the most relevant tests, run stuff concurrently not linearly, start valuable long running tests as soon as stability is determined. Do all of this within a relevant execution window and provide data and an assurance metric throughout the process.

A smarter pipeline would become an important part of a best of breed Application Release System that is necessary to differentiate IBM from open source ALM and emerging devOps solutions.

Question: With what | Who | When

18

Drive adoption




Consume and Drive



Discussion Point: Delivery mechanisms to Production

Jazz.net

JazzHub

Internal adopters (Hursley, and Rational)

Workload Patterns

DevOps developerWorks portal

Traditional Deploy Repositories

19

Drive adoption




Consume and Drive



Discussion Point: Applying service

How should upgrades and service be applied in JazzHub and Jazz.net environments?

– Swim-lanes / Hold Cold

– Patch

– Rolling upgrades

How are the same mechanisms used within the our delivery pipeline

Do all new project requests get placed on latest available (JazzHub)

How many topology deployments at a time (JazzHub)

20

Drive adoption




Consume and Drive



Next Steps

21

What are the next steps


Next Steps

June – July (Learning, evaluation and communicating a perspective)

Form mini projects with dedicated resources

Create some initial horizontal teams

Merge various efforts and understand what is not going to be done

Establish baseline business value assessments

22

© 2013 ibm corporation continuous delivery implementation internal adoption and devops proceses and...

Documents