lessons learned from large scale adoption of devops for ibm z systems software
TRANSCRIPT
Lessons Learned from Large-Scale Adoption of DevOps for IBM z Systems Software
Per KrollDirector Offerings Mgmt, Development & Support DevOps for Enterprise Systems
1
My journey…
Creator and Chief Architect
Extended Core Team Transforming +20,000 developers Practice library for agility@scale Deployment assets… Initiator and Project lead
Product Manager and Development Manager
Rational Unified Process
Development and Support ExecutiveDevOps for Enterprise Systems
Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
2
Why transform
• Innovation – exceed customer expectations and beat competition
• Delight customers through an iconic user experience
• Quality
• Speed
• Cost
• Job satisfaction
3
Complexities we face as a product organizationCommon in the industry• Extremely distributed• Varying need for business agility and investment levels• Mixture of tools, including homegrown• Dispersed cultures, many acquisitions
Specific for our product development organization• Multi platform (multiple OSs, DBs, App Server, browsers, ….) – 100s of permutations to test• Translations to dozens of languages• Legal and packaging constraints
Specific for mainframe development• Many commercial and open source tools do not support mainframes• Attitude: ‘Mainframe is different’, average age is higher 4
Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
5
Continuous Improvement: An Agile Non-negotiable
6
• There is no such thing as “100% Agile”
• Stop living with pain - leverage known practices to alleviate your pain• Automated build and deployment• Shift left testing: test early, test often• Loosely coupled architecture• ….
“If you can’t tell me you’re better this month than you were last month, don’t tell me you’re ‘Agile!’”
All things agile
UserExperience
Development
Business
All things agile
UserExperience
Development
Business
Design Thinking Offerings Management
DevOps
Innovation
Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
9
Problem: Business Process
• Our business process, IPD, was too heavy– A lot of overhead– Encouraged over specification– Discouraged early experimentation– Forced too early commitment to a specific solution– Expensive business change process
• Solution: Introduce Offerings Management Discipline– A lighter-weight business process – Centered around the principles for the LeanStartup
Offerings Management Discipline: Inspired by the Lean StartUpLight-weight artifacts, with focus on learning and pivoting
11
Learn and Pivot
Light-weight artifacts
Focus on markets and business outcomes
Design Thinking
Design Thinking
Understand
Explore
Prototype
Evaluate
HillsInvest for market outcomes
Sponsor UsersUnderstand the user experience
PlaybacksCollaborate. Align. Engage!
IBM is increasingly recognized for design leadership
Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
15
Adopting DevOps: 2015 Focus Areas
• Build and deployment automation: Fast and extremely high ROI– Weeks or months, not quarters or years
– Key products: Rational Team Concert Enterprise Edition, Jenkins, and Urban Code Deploy
• Test automation: Crucial, but labor intensive– Incremental adoption, targeted investments: default priority order
1. Build Verification Test – Automate most frequently run tests
2. New features – Stop digging a deeper hole…
3. Components with many defects – Immediate quality benefits
4. Components with low test coverage – Incrementally bite the bullet
– If tests never fail and component not updated => Do you need to automate?
– Key products: RDz zUnit, Rational Test Workbench, Application Delivery Intelligence
• Not a focus: Already mature areas – Collaborative Development: Rational Team Concert (RTC)
– Modern Development: Modern IDEs / Problem Determination tools (RDz / PD Tools) 16
• Cross discipline feature teams• Establishing rhythm, continuous
integration, definition of done
• Shift left testing
• Complex testing run continuously
• Automation part of development
• Anything done manually analyzed for elimination or automation
• Retrospectives, eliminate friction points
17
Plan Develop Develop Develop Test/Fix/Test4 weeks 4 weeks 4 weeks
Warmup Sprint 1 Sprint 2 End Game…
Develop Develop Test/FixTest
Plan
retrospectiveSprint 1 Sprint 2 End Game
4 weeks4 weeks
Quarterly Releases
Ship ready with each Sprint
Create a culture committed to automating everything…
Continuously Test Using Automation and VirtualizationExample: Collaborative Lifecycle Management Team
Example: Increasing Test Automation in IBM Rational Developer for z
Problems• Long release cycles for both new releases and service• Development process more waterfall than agile• Too much effort spent on manual testing
Progress made in 2015H2• Educate team (JUnit and Rational Functional Tester)• Ensure JUnits are added for all new capabilities• Optimize BVT suite to run in 15 min or less, triggered by a build. Publish results.• Analyze components to understand where to invest: Quality issues, test coverage, churn• Apply swat team for rapid increase in test automation• Adopt strategic automation tooling: UrbanCode Deploy• Incorporate data-driven frameworks to automate complex areas
18
Example: Increasing Test Automation in IBM Rational Developer for z
Results so far (work in progress)• Automated test coverage increased from 20% to 50%• Daily BVT uncovered defects earlier in the cycle, resulting in quality improvement• Reduced regression testing for each deliverable from 3 weeks to 2 weeks• Shortened time to test while growing number of test cases and improving quality• Test team able to handle more deliverables and cover more integrations
19
Executes 10000s of Junits & security scans
Develop Test (continuous) Interactive Testing
Usability Test
Security Pen testing
Unit Test
Production
Jazz.net
System Test
Performance Test Integration Test
Function Test
Build
Production-Like Environment Production-Like Environment
Global dev/test/ops team ~100 daily commits/deploys using our tools to build our tools
191,000 jazz.net community members, 50K sandbox instances, >105K page views
DeploymentsDeploy Golden Topology
IBM BluemixDevOps Services
20
Developers launch test from their
workspace Rapid install and smoke TestsFunctional and systems tests on Golden Topologies
BVT
“Simply put, things always had to be in a production-ready state: if you wrote it, you darn well had to be there to get it running!”
- Mike Miller, Co-founder and Chief Scientist at Cloudant,an IBM Company
Continuous Integration and Test
Build a Delivery Pipeline Leveraging Tools-As-a-Service Example: IBM Collaborative Lifecycle Management Team
Continuous Delivery Pipeline DashboardExample: IBM Collaborative Lifecycle Management Team
21
• Automated quality gates• Single view of quality• Ability to quickly drill down on problems• View the live dashboard at: jazz.net
IBM CICS Tools (Ant / RTC Build)
IBM CICS Explorer (Ant / RTC Build)
Example: IBM z Systems DevOps TransformationKey Challenges (2015, Q2):• Products get built, tested & delivered
separately• Inconsistent build technologies &
infrastructure• Lack of synergy in driving product
compatibility• Lots of manual interventions (e.g.
testing, deployment)
Impact to customers:• Difficult for customers to figure out what
version of which products work together• Late discovery of defects, especially
related to cross-product integrations• Some products may not work well
together 22
DevIESCEXDB2IES Dev SCM Build Test
Feedback
IBM Rational Developer for z Systems (Eclipse PDE Build)
IBM Problem Determination Tools (Maven / Jenkins Build)
IBM z/OS Explorer (Ant / RTC Build)
IM Download site(RDz)
IM Download site(PD Tools)
Eclipse p2 Update Site (CICS Explorer)
Eclipse p2 Update Site (z/OS Explorer)
IM Download site(CICS Tools)
“Internal” DevOps (Phase 1) delivers IBM ADF for z Systems
23
Eclipse 4.4 Luna
z/OS Explorer V3.0 Aqua
Rational Developer for z Systems
Application Delivery Foundation for z SystemsApp Performance
AnalyzerDebug Tool
File Manager
Fault Analyzer
CICS Explorer
z/OS Connect EE
RTC
CICS ToolsSingle IM
Composite Site
Single Eclipse p2 Update
Site
IBM Explorer for z/OS Aqua Release Highlights
A single “Release Train” delivery pipeline (more than a dozen products on z/OS Explorer) Common build technology & infrastructure Common software stack & dependencies Improved cumulative build time
Improved integrated testing across products Improved overall product compatibility & quality A single download site for IM and Eclipse p2 installs Improved user experience with a single “MainframeDev” site on IBM DeveloperWorks for all products
IBM Application Delivery Foundation for z Systems V1.2
24
• A comprehensive set of z/OS tools with a modern, integrated & consistent user experience
• Best of breed capabilities in a single bundle enables the delivery of high quality z/OS applications quickly & efficiently
– Accelerates development, test & maintenance of z/OS applications
– Increases debugging efficiencies & reduces application development lifecycle
– Helps isolate system constraints quickly & maximize application response time
– Expedites the identification & analysis of system and application failures
– Simplifies manipulation of data in data stores such as DB2, IMS, CICS, MQ, data sets & HFS files
• A single & attractive pricing model for the complete bundle
• Accelerates the DevOps transformation of z/OS application delivery
RDz
DT
FA
APA
FM
IBM z Systems DevOps Transformation – Q2 2016 (Phase 2)
Phase 1 – Dec 2015 Phase 2 – Q2 2016
Theme – “Extend DevOps through automation”• Extend existing Continuous Integration pipeline to support Test & Deploy• On-board & streamline product delivery to be on the common pipeline • Increase focus & investment on test case automation (e.g. BVT, smoke test, IVT)• Integrate with IBM UrbanCode Deploy to orchestrate test automation & deployment
Expected Outcome & ROI• Reduce TCO (total cost of ownership) by replacing manual work with automation• Improve speed of product delivery across our portfolio• Further improve product quality & compatibility through continuous/automated testing
Single Eclipse p2 Update Site
Single IM Composite Site
ADF: Business Value of our DevOps Transformation
26
Our adoption of DevOps improves speed, reduces waste & enhances quality
Improvement Area Q1, 2015 Q2, 2016 Improvement Benefits
Release frequency 12‐18 months 1 month 90‐95%
‐ Accelerates innovation‐ Improves Time‐To‐Market (TTM)‐ Continually drive higher product quality‐ Enables continuous delivery
Cumulative build time ‐‐ ‐‐ 30%‐ Earlier detection of issues‐ Shortens development lifecycle
WASTE REDUCTION Build & delivery process
5 different processes
1 consolidated process 80%
‐ Eliminates duplications of work‐ Deflects customer reported issues
Automation: BVT & Integration Test 0% 100% N/A‐ Earlier detection of issues‐ Enables continuous testing & delivery
Automation: Deployment 0% 100% N/A‐ Reduced labour‐ Enables continuous testing & delivery
Dependency Management Ad‐hocCommon repository N/A
‐ Earlier detection of issues‐ Enhances product compatibility
SPEED
QUALITY
Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
27
Driving Large-Scale Organizational Change
Key factors • Provide clarity in objectives• Drive the right culture• Gain executive buy-in• Establish bottoms-up communities• Encourage self-assessments• Drive tools standardization while allowing appropriate levels of freedom• Automate everything: Build, Test, Deploy• Govern through transparency: Aid, remove roadblocks, and establish accountability
28
Becoming agile is a journey. Enjoy your journey.
Drive the right culture: Values and PrinciplesHelp install values through social contracts
29
Principles
Make self-organization work: Clarity of Outcome
30
Establish bottom-up communities
31
+10,000 members
Several events every week
Easy access to experts
Learn from, teach and interact with people in your area of interest
Line of Business
Rational Doors Next Generation (RDNG)Rational Team Concert (RTC)Request for Enhancement (RFE)
Rational Quality Manager (RQM)Rational Test Workbench (RTW)Rational Test Virtualization Server (RTVS)jUnit Automation Tool (JAT), JazzSM-Dash
Jenkins
RTC EE, RTC, ANT, Jenkins
Rational Team Concert (RTC)Rational Developer for the Enterprise
(RDz, RDi, RDp, RD A&L)Git
Pipeline automationRational Team Concert (RTC)Jenkins
UrbanCode Deploy (UCD)
zService, SPoRT
Golden Topology: Driving tools standardizationwhile allowing appropriate levels of freedom
15
Self Assessments and Roadmaps to Address Gaps
Project Management
Project Tracking Design & Design Thinking
Builds Technical Debt Automated Testing
WholeTeams
Provisioning Automated Deployments
Monitoring of Environments
Collaboration Customer Focus Monitoring Customer Usage
1 1 1.5 1.25 1.5 3 0.5 5 6 2 0 1.75 0.5PLANNING BASIC SOFTWARE ENGINEERING PRACTICES DEPLOYMENTS TO TEST AND
PRODUCTION (as applicable)DEV & OPS WORKING TOGETHER
CUSTOMERS
3.5 11.25 8 0 2.25
Area: XXXContact: XXXModel assessment results – as of 04/09/2015
No actions PlannedFuture actions TBD
Areas being addressed
Area Summary of actions Target date Target Metric
Automated Testing • Building test center of competency • Adopting test driven development• Created detailed action deck already being executed on
• 06/30/15• 11/01/15
1
Automated Deployments • Investigate UCD usage• Utilize for new offerings• Adopt for existing products and include provisioning
• 07/01/15• 12/1/15• 06/30/16
3
Dependency • Requires ability to staff Mexico team
Risks • Continued escalations and test fixes taking resource from these efforts
Issues • None
Effective governance through transparency: Aid, remove roadblocks, and establish accountability
Central dashboards providing visibility into• Improvement actions• Metrics• InhibitorsView by organization• Product• Area• ExecutiveUsed to • Identify common problems so we can address them strategically• Help teams to structure their improvement efforts• Provide accountability
34
35
The wrong metrics can incent negative behavior
Leverage Outcome and In-Process Metrics
Outcome Metrics• Determines whether you achieve your
end goals or not• Responds slowly to change• Examples from my organization:
– Delivery time of defect fix release– Delivery time of major release– Customer-reported quality issues– Customer satisfaction (survey)– Employee satisfaction (survey)
In-Process Metrics• Determines whether you are effectively
adopting the right practices• Responds rapidly to change• Examples from my organization:
– Code quality at sprint end– Point of customer feedback for release
(e.g. 20% into a project)– % of test automation– % of build verification test automation– % of regression test automation– Duration of full regression testing (manual
as well as automated)
36
Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
37
Driving Large-Scale Organizational ChangeWe are here to help!
• DevOps requires an end-to-end integrated tool set– We are the only vendor providing an end-to-end, cross platform DevOps solution
• Driving real change requires a cultural transformation– We have many experts and assets to help you in your transformation
• Start with our Deployment Project Office– Work with your IBM account team, and contact Jon Sayles ([email protected]) if
uncertain who to talk to– Example: We can offer ongoing advisory sessions between our and your executives and
senior leaders
38
Becoming agile is a journey. Enjoy your journey.
How to Get Started IBM DevOps Workshop
Wasted effortLegend:
Think Product-based
Agile
Automated
Collaborative
Optimizing
Morepredictable
Moretransparent
Morecontinuous
Process-based
Process-heavy
Manual
Silo-ed
Code/Test
Run
Manage
Inefficient LeanerLeaner and
Smarter
Productive effort
Workshop Objectives
– Define business drivers for DevOps, and see what is currently possible for your organization
– Identify existing or planned DevOps initiatives, and map them to the business drivers
– Determine the top inhibitors within the current software lifecycle
– Create an adoption roadmap for these DevOps practices, including milestones, time frames and known adoption challenges.
Overview
– No charge!– Led by IBM DevOps Solution Architects– Audience : For senior IT and LOB executives and
managers in application development and IT Operations
Sign up for a free DevOps Innovation & Optimzation
workshop
39
Notices and Disclaimers
40
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
Notices and Disclaimers Con’t.
41
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank YouYour Feedback is Important!
Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your
smartphone, laptop or conference kiosk.