devops approach (point of view by ravi tadwalkar)
TRANSCRIPT
DevOps Approach-People, Process, Tools, Governance and Metrics
(From Ravi Tadwalkar’s Point of View)
Source: “2015 Annual State of DevOps A GLOBAL PERSPECTIVE ON THE EVOLUTION OF DEVOPS & A BENCHMARK FOR SUCCESS”, Delphix and Gleanster
DevOps ApproachPeople
Process
Tools
Governance
Metrics
We do lot of Lean-Agile Coaching and DevOps assessment work with clients around US, and based on that, we have come to the conclusion that implementing DevOps is more than implementing deployment pipeline for a technology stack (such as IBM, Microsoft, SAP, Oracle or Atlassian) based end-to-end tool chain. In fact, tool chain is only one of the 4 critical success factors, the other 3 being people (changing roles), process (waterjile/agilefall (hybrid scrum), pure scrum, scrum at scale, scrumban or Kanban?) and lean governance model (specially around upstream Portfolio Kanban system coupled with downstream Release Engineering Kanban system).What kind of metrics matter for assessing success of DevOps implementation?
We will describe these concepts, in this brief slide-deck.
Step 1: Intake & Planning(upto 1- 2 weeks)
•Define Goals-e.g. 1. Visibility at all levels2. Frequent releases
for TTM with quality
•Define Scope:early adopters- programs/products
•Identify key stakeholders:from Business, Dev/Engg, SQA/Test, Release Engg and/or Operations
• Identify landscape:Value chain, initiatives, priorities, expectations, impediments, and known constraints
Step 2: Discovery phase(upto 3-4 weeks)
•Apply "learning to see” approach to create value by eliminating muda(non-value-add aka NVA waste).Narrow down and validate current state value stream map (VSM)–identify current process workflows, analyze timings and identify NVA waster (impediments) causing delays in flow. Visualize future state VSM- “single piece flow” in ideal DevOps world.
•Workshops- to identify gaps in key DevOps tenets related to people, process, tools, governance & metrics
•Assessment of DevOps Capability maturity
•Formal Report- Submit assessment worksheet highlighting gaps
Step 3: Roadmap phase(upto 4-6 weeks)
• Recommendations for Agile& Lean adoption with DevOps-
•People- Agile and DevOps adoption related new roles
•Process -Practices and ceremonies for chosen process
•Tool chain for Deployment Pipeline for selected engineering stack
•Governance & Metrics
• 30-60-90 plan specific to identified gaps in order to improve DevOps Capability Maturity level
#Step 4:Pilot Phase
• Identify people for Agile & DevOps related Roles from selected teams
• Training & Coaching- Process
Option 1- “At Cadence” DeliveryOption 2- “Continuous” Delivery
and - People
- changing roles andresponsibilities
- Tools- tool chain synchronized with
changing roles and processes
• Build-measure-learn-learn to either pivot or persevere
• Validated learning -socialize outcomesand communication plan of continuous improvement
#Step 5:Wider Rollout
•Move the needle
•Create Community of Practice (CoP)-core team & charter
•Add champions, evangelist to CoP
•Cadence of events
•Execute Continuous improvement plan of Agile & DevOps at scale, apply process evaluator tool with a centralized or autonomous governance model:Option 1-revolutionary (scaled) framework e.g. SAFe, DAD, LeSS,…Option 2-evolutionary (phased) framework e.g. Kanban and/or Scrumban
DevOps Approach: 5 steps for DevOps Journey
#Time line is context dependent after step 3
3
Tenets Parameters Context
Lean-Agile practices
Business Requirements & Change Management
Agility to accommodate changing requirements
Sprint Model Avoid Big Requirements Up Front
Self organized teams Needs Leader, not commander
Continuous integration
(CI)
Build automation Stabilize Release Management
Code quality metricsQuality Built-In rather than
afterthought
SW Regression Coverage Reduced Regression ~ Predictability
New Test & Automation Automate for Faster Time to Value
CI/CD pipeline Management
Deployment pipeline ~ Faster Time to Market, with Quality
Continuous delivery
(CD)
Code Promotion ModelNo more “But it works on my
system”
Application Deployment Enabler for Business Agility
Predictive Monitoring & Analytics
Amplify Feedback Loops amongst Business, Dev, QA & Operations
CD pipeline managementEnabler for Environment
Provisioning
Agile Testing
Test Automation (Unit, Functional, Regression, Performance, Security)
Avoid manual testing in order to increase Time to Market, with
Quality
TDD & RefactoringProductivity requires Reduced
Technical Debt
Tenets Parameters Context
EnvironmentManagement
Environment provisioning
Predictability requires reduced environmental inconsistencies
Service virtualization for Dynamic scaling
Scalability for Upstream and Downstream “customers”
Test data management Reproducible & consistent behavior
Prod – non prod environment closeness
No surprises for environment refreshes
Lean Architecture for DevOps
Modular release strategy Frequent Releases
SW deployment modularity
Reliable Release Calendar
Design for resiliency and serviceability
Deal with Non-Functional Requirements Early
Culture & Org
Alignment
Org alignmentBusiness Embracing Culture of
Collaboration
Dev & Ops Collaboration Model
Dev, QA & Ops Embracing Culture of Collaboration
Team profileIncrease Flow with Cross-Functional
Teams
Operations
Resilience & Disaster Recovery
Robustness
Performance Scalability Testing
Enabler for High Availability
Preliminary Assessment Requires Holistic Template4
DevOps maturity model (Preliminary Assessment Report)
Key DevOps Tenets
Parameters Context
"As Is" state:Consolidate w/
consensus (crawl/walk/run/sprint)
Crawl Walk Run Sprint
Culture & Alignment
Org Alignment Business Embracing Culture of Collaboration C
Dev & Ops Collaboration Model Dev, QA & Ops Embracing Culture of Collaboration C
Team Profile Increase Flow with Cross-Functional Teams C+
Lean-Agile Practices
Biz Requirements & Change Management Agility to accommodate changing requirements W
Scrum/Kanban/Scrumban Model Avoid Big Requirements Up Front C+
Self organized teams Needs Leader, not commander W
CI (Continuous Integration)
Build Automation Stabilize Release Management R
Code Quality Metrics Quality Built-In rather than afterthought W
SW Regression Coverage Reduced Regression ~ Predictability C-
New Test & Automation Automate for Faster Time to Value W
CI/CD pipeline Management Deployment pipeline ~ Faster Time to Market, with Quality C
CD (Continuous
Delivery)
Code Promotion Model No more “But it works on my system” C
Application Deployment Enabler for Business Agility C
Predictive Monitoring & Analytics Amplify Feedback Loops amongst Business, Dev, QA & Operations W
CD Pipeline Management Enabler for Environment Provisioning C-
Agile TestingTest Automation Avoid manual testing in order to increase TTM, with Quality C
TDD & Refactoring Productivity requires Reduced Technical Debt C
Environment Management
Environment Provisioning Predictability requires reduced environmental inconsistencies C+
Service Visualization for Dynamic Scaling Scalability for Upstream and Downstream “customers” W
Test Data Management Reproducible & consistent behavior C+
Prod & non-prod Environments Closeness No surprises for environment refreshes W
Lean Architecture for DevOps
Modular Release Strategy Frequent Releases C
SW Deployment Modularity Reliable Release Calendar C
Design for Resiliency & Serviceability Deal with Non-Functional Requirements Early C
OperationsResilience & Disaster Recovery Robustness W
Perf Scalability Testing Enabler for High Availability W
As Is To Be
5
“As-Is” Value Stream Map – Illustrating Kaizen Lightening Bursts
Customer
Plan (Idea / Discovery) Build (Delivery) Run (Operations)
Business Case
PlanningPrioritization
- Ideation
- Biz & Tech
Assessment
- Go/No-go/Defer
- Capacity
Planning
- Project
Planning
- Prioritization
- Project Selection
- RFE Approval
Design(Arch)
Build(Dev)
Test (QA)
- Technical
Design
- WBS
- Integrate
- Test Planning
- Ops Planning
- Get Approval
- Deploy to QA
- Configure Tests
- Manual Testing
- User Acceptance
ProductionGo-Live
Business-As-UsualChange Mgmt
(ITIL)Environment Provisioning- Ops Planning
- Config&Deploy
- Wait for DBA
- Get Approval
Prod Support & Monitoring
Ops Mgmt(DB, Network)
3 to 5 m 4 to 6 m2 to 3 m Sustenance mode
Business
Portfolio:
Governance &
Oversight
Environments & Tools
Time
Ladder
(approx.)
Lead Time ?Value Add Time ?
Non-VA Time ?
Electronic Information Flow
Offshoring
Teams
0.5 m 1 m
1 m 1 m
1-3 m
0.5 m 2 m 0.5 m
0.5 m 0.5 m
1-2 m
1-2 m
1-2 m
1 m
WIP (Inventory)PUSH- scheduled before neededExternal source
LEGEND
Eliminate Waste
Time Analysis
Non-Value Add vs. Total 45%
Value Add vs. Total 55%
Value Add vs. Non-VA ?
Sustenance
6
AS Is State of “Agile and DevOps”
Permit to Build Permit to Operate
No visibility of portfolio, program and teams
Need PO for team
Architect overloaded as Scrum Master; PM role unclear
Dev and Ops work in silos
Agile process practices only for execution
Waterfall still preferred for multi-site planning, Dev, & operations
Atlassian tool chain not fully integrated with Amazon Cloud infrastructure
Big Design Up Front (BDUF) approach
“Agile Testing” is done very late in sprint
Teams “inspect & adapt” via demo feedbacks, but deployment not at sustainable pace
In onsite-offshore model, PO, Dev & Testers mostly work in silos, when scrumming during sprint. Need MVD (min viable documentation) template with Cucumber, Jira and Confluence
Need centralized OMS for “pre-prod” i.e. production-like environment provisioning & data refresh, wherever feasible
Build Automation exists with Jenkins; Test Automation: start w/ Cucumber; Release Automation is in its infancy.
Customer demo and feedback happens frequently, but no interim release cadence hampers that communication.
Environment provisioning and cost is not a challenge with Amazon Cloud; but database setup still a challenge.
Pre-prod not always feasible due to: Distributed digital assets Student privacy concerns
Long Release Cycle: features deployed twice/year (seasonality)
Big-bang release demo for users(True even for pilots, no “fail fast”)
OMS can be challenge to move from 6 weeks internal release to monthly release cadence
Manual deployment- error prone
Seasonality a big factor in maintaining status-quo on environment mismatch
7
DevOps maturity model (Goal Setting for 30-60-90 Plan)
Key DevOps Tenets
Parameters Context
Normalized state transition "As Is" -> “To Be”:
Consolidate w/ consensus (crawl/walk/run/sprint)
Crawl Walk Run Sprint
Culture & Alignment
Org Alignment Business Embracing Culture of Collaboration C -> W
Dev & Ops Collaboration Model Dev, QA & Ops Embracing Culture of Collaboration C -> W
Team Profile Increase Flow with Cross-Functional Teams C+ -> W
Lean-Agile Practices
Biz Requirements & Change Management Agility to accommodate changing requirements W
Scrum/Kanban/Scrumban Model Avoid Big Requirements Up Front C+ -> W
Self organized teams Needs Leader, not commander W
CI (Continuous Integration)
Build Automation Stabilize Release Management R (better!)
Code Quality Metrics Quality Built-In rather than afterthought W
SW Regression Coverage Reduced Regression ~ Predictability C- -> W
New Test & Automation Automate for Faster Time to Value W
CI/CD pipeline Management Deployment pipeline ~ Faster Time to Market, with Quality C -> W
CD (Continuous
Delivery)
Code Promotion Model No more “But it works on my system” C -> W
Application Deployment Enabler for Business Agility C -> W
Predictive Monitoring & Analytics Amplify Feedback Loops amongst Business, Dev, QA & Operations W
CD Pipeline Management Enabler for Environment Provisioning C- -> W
Agile TestingTest Automation Avoid manual testing in order to increase TTM, with Quality C -> W
TDD & Refactoring Productivity requires Reduced Technical Debt C -> W
Environment Management
Environment Provisioning Predictability requires reduced environmental inconsistencies C+ -> W
Service Visualization for Dynamic Scaling Scalability for Upstream and Downstream “customers” W
Test Data Management Reproducible & consistent behavior C+ -> W
Prod & non-prod Environments Closeness No surprises for environment refreshes W
Lean Architecture for DevOps
Modular Release Strategy Frequent Releases C -> W
SW Deployment Modularity Reliable Release Calendar C -> W
Design for Resiliency & Serviceability Deal with Non-Functional Requirements Early C -> W
OperationsResilience & Disaster Recovery Robustness W
Perf Scalability Testing Enabler for High Availability W
As Is To Be
8
“To-Be” Value Stream Map – Seek “Build” Process Improvements
Customer
Plan(A “Release”) Build(Deployables) Run (Operations)
TDADMulti-
Sprint PlanBacklog Prep <Story Map>
- Technical Design - Multi-team
- At cadence
- Ops work
- 1 story map
- per hour
- per epic
Design(Artifacts)
Sprint Plan + Sprint Retro
Iterate(Dev & QA)
- MVD (wiki)
- Test Plans
- UX artifact
- 2 mtgs, 4hr each
- Ops work included- Build w/ Unit Tests
- Execute test cases
- Expedite bug-fixes
- Dev & Test NFRs
DemoBusiness-As-Usual
Change Mgmt(ITIL)
Environment Provisioning- Env on demand
- PO Acceptance
- Production bugs
- Proactive DBAs
Prod Support & Monitoring
Ops Mgmt(DB, Network)
3m 3m (4 sprints) 1m Sustenance mode
Business
Portfolio:
Governance &
Oversight
Environments & Tools
Time
Ladder
(approx.)
Lead Time ~4mValue Add Time ~2m
Non-VA Time ~2m
Time Analysis
Non-Value Add vs. Total 50%
Value Add vs. Total 50%
Value Add vs. Non-VA 1
Electronic Information Flow
Offshoring
Teams
2m 1w
1w .5 w
NVA:1d
1m
0.5 w 1w.5 w 2w
NVA:1d
1w
WIP (Inventory)PUSH- scheduled before neededExternal source
LEGEND
Eliminate Waste
1w
3m (4 sprints)
Sustenance
9
To Be State of “Agile and DevOps”
Permit to Build Permit to Operate
Visualize work with Kanban:plan-build-run
Active Product Management: match funding & seasonality?
Scaled Agile: Conduct multi-sprint planning with user story map & feature planning board
Common Release and sprint cadence for teams
Defined Roles & Responsibilities for SM, PO, Program Mgr & Product Mgr
Evangelize with communities using open space events
Get into Scrumming Kata (e.g. retrospectives ) and Kanban Kata for continuous improvement
PO be accountable for release & sprint goals
TDD & BDD: Use frameworks for development, rather than after-the-fact during testing
Improve cycle time in each stage of tool chain for better overall lead time: e.g. pre-commit level code review rather than post-commit
Version everything: code, metadata, databases, build artifacts…
Test automation implemented & relevant for execution *during* sprint
Integration, Regression, Load, performance, security and acceptance testing is not a hardening event
Build-Measure-Learn: leverage analytics where possible
Validated Learning:Conduct experiments to evangelize “fail fast” mindset: pivot or persevere
Visualize deployment of releases with Release Engineering Kanban
Release Automation:Release and Deployment management is automated
Production Monitoring and incident management automation
Management 3.0 tools for leaders:Knowledge management for team building and developing generalists out of specialists
10
11
People: DevOps extends Product Owner & Scrum/Flow Master Roles
DevOps extends Product Owner & Scrum Master roles with additional responsibilities. DevOps also adds higher level “Agile Product Manger (aka Business Owner)” and “Agile Program Manager (aka Release Manager)” roles.
Role Responsibilities Role Responsibilities
Agile Product Manager
aka
Business Owner
•Org Alignment during Portfolio-level Planning: •Take ownership of delivering business capabilities based on business priorities
•Participate in pre-planning activities e.g. envisioning workshops for business goal, vision & roadmap
•Share business objectives with key stakeholders e.g. PMO, Business Management, and System Architects
•Communicate business capabilities for next release planning aka multi-team or multi-sprint planning
•Org Alignment during Multi-team or Multi-sprint Planning: •Take ownership of delivering business capabilities based on total (multi-)team capacity
•Participate in driving release planning during sprint 0•Participate in management review for scope adjustment, problem-solving & improvisations and compromises
•Participate in leadership level obstacle escalation & removal in order to act on obstacles & dysfunctions
•Org Alignment during Release Management & Deployment: •Participate in providing feedback during demos
Agile Program Manager
aka
Release Manager
•Org Alignment during Portfolio-level Planning: •Collaborate with leadership (from Biz, Dev, QA & Ops) to support release cadence
•Org Alignment during Multi-team or Multi-sprint Planning & Execution: •Take ownership and responsibility of Multi-team or Multi-sprint release planning•Facilitate release planning aka multi-team or multi-sprint planning event•Define & maintain release plans which include content, dates & success criteria•Manage release dependencies across applications, products and/or components•Define & measure metrics to “inspect & adapt” for continuous improvement•Facilitate change management
•Org Alignment during Release Management & Deployment: •Take ownership and responsibility of release management•Lead cross-functional teams (Biz, Dev, QA & Ops) to support release variability•Coordinate with business owners, product owners, scrum masters & Ops team•Define and manage release readiness•Ensure deployment automation / continuous delivery setup•Execute a low-fidelity obstacle escalation & removal process•Define release types, associated workflows & approval requirements for artifacts•Provide oversight for deployments into production for Service Delivery •Guide the completion of root cause analysis to ensure issues do not recur/regress
Product Owner
•Responsible for aligning Operations & Development on business goal, vision & roadmap
•Consider operations needs as input to prioritization•Create product backlog during sprint 0; maintain it thereafter
Scrum Masteraka Flow Master
•Facilitate collaboration (Dev<->Ops)•Ensure impediment free execution by Dev and Operations team •Ensure adoption of agile practices & ceremonies, DevOps processes & tools•Facilitate cross-training of team members and rotating the roles of team members
12
People: DevOps Extends Agile Team Member RoleAspect of Role DevOps Specific Core Responsibilities
Dev • Collaborate with Operations to decide on Sprint goal and commit to deliverables• Collaborate with Operations to identify environment configuration & provisioning requirements• Resolving recurring Ops issues• Understanding and developing as per Ops requirements• Understand Ops tools & processes• Feature development and testing
+• Continuous testing• Test automation• Test data management • Data deploy automation• Performing Ops-like testing for Ops-like metrics ( e.g. security)
+• Collaborate with Development to detail out Operations specific needs• Provision & Configure environments as per the needs identified jointly with development team• Understand Dev tools & processes• Maintain continuous delivery setup & help with troubleshooting• Debug & fix environment specific issues• Contribute to development using available bandwidth• Monitor continuous delivery setup and help fix issues slowing down the progress
Test
Ops
From “Plan->Build->Run” perspective, planned releases & sprints are executed using iterative & incremental delivery workflow depicted here. Each delivery cycle consists of these steps. Note that the Sprint cycle (steps 2 to 5) is repeated for all sprints in each release:
1. Release Prep• Portfolio Kanban System:
• Agile Product Manager brings visibility to “product” feature set for initiative(s) identified, by shaping demand via portfolio backlog.• Agile Product Manager brings visibility to features (including tech debt items), by auditing each feature on Portfolio Kanban system.• Product Owner writes user stories based on User Story Maps for each feature.
• Release Planning meeting: • Agile Product Manager drives the Release Planning event that marks the end of “sprint 0”, a time-box used for purpose of planning.• Agile Program Manager facilitates creation of Release Plan, emphasizing that release plan is an estimate/forecast, not commitment.
• Activities: Epic & tech debt intake, story mapping, create user stories & related documents, and create Feature Planning Board.• Agile Program Manager correlates feature-level forecast from the Release Plan with sprint-level commitments during entire release.
2. Sprint Startup• Sprint Planning meeting:
On day 1 of Sprint, scrum master facilitates Spring Planning meeting to create Sprint Plan WIP-limited by available team capacity.
3. Sprint Execution- Delivery• Sprint Execution:
After Sprint Planning meeting, scrum master facilitates Spring execution, by removing impediments faced by team during sprint.• Mid-Sprint Backlog Grooming meeting:
In this meeting, held around mid-point during the sprint duration, Product Owner gives a glimpse of stories & defects for upcoming sprint, so that team can do any course correction needed to attain current and future sprint goals.
4. Sprint End- Acceptance• Sprint Demo and Retrospective meetings:
On the last day of Sprint, scrum master facilitates Sprint Demo or System Demo meeting, followed by team-only Sprint Retrospective.• High performing teams may conduct current-sprint demo and next-sprint planning during same meeting, followed by retro next day.
5. Deploy to Production• Release Engineering Kanban System:
• Agile Program Manager uses feature deployment status on Release Engineering Kanban system to validate done-ness of deployment artifacts related to work-items, and to relay feedback to team(s) on issues during each deployment.
• Agile Product Manager provides feedback so that executive leadership can make go/no-go/defer decision based on that feedback.
Process: Option 1: “At Cadence” Delivery with Frequent ReleasesRelease Prep
(Portfolio Kanban System)
Deploy to Production (Release Engineering Kanban System)
Sprint Execution-Delivery
Sprint End-Acceptance
Sprint Startup
13
From “Plan->Build->Run” perspective, planned releases are executed using iterative and/or incremental delivery workflow depicted here. Each delivery cycle consists of these steps. Note that the Kanban/Scrumban cycle (steps 2 to 5) is repeated for all releases until the ship date:
1. Release Prep• Portfolio Kanban System:
• Agile Product Manager brings visibility to “product” feature set for initiative(s) identified, by shaping demand via portfolio backlog.• Agile Product Manager brings visibility to features (including tech debt items), by auditing each feature on Portfolio Kanban system.• Product Owner writes user stories based on User Story Maps for each feature.
2. Release Planning meeting• Agile Product Manager drives the Release Planning event that marks the end of “sprint 0”, a time-box used for purpose of planning.• Agile Program Manager facilitates creation of Release Plan, emphasizing that release plan is an estimate/forecast, not commitment.
• Activities: Epic & tech debt intake, story mapping, create user stories & related documents, and create Feature Planning Board.• Agile Program Manager correlates feature-level forecast from the Release Plan with sprint-level commitments during entire release.
3. Kanban/Scrumban in the middle• Execution: “Focus on the work, not the worker!”
After Planning meeting, “Flow Master” facilitates execution for single-piece flow, removing impediments faced by team during release.• System Capability Review (aka Input Queue Replenishment) meeting:
In this meeting, held at cadence based on lead time stats, Product Manager and Product Owner – both - give a glimpse of work remaining, so that team can do any course correction needed to attain current and future release goals.
4. Continuous Improvement (4 Kaizen events)• Feedback Loops:
At regular cadence, based on increments of continuous delivery- at least once a month- management facilitates Kaizen meeting, using Kanban Depth framework. Over 4 such Kaizen events, team & management observe areas for continuous improvement in structured way.
• High performing teams may conduct such Kaizen events right after incremental or end-to-end system demo & Scrum-style retrospectives.
5. Deploy to Production• Release Engineering Kanban System:
• Agile Program Manager uses feature deployment status on Release Engineering Kanban system to validate done-ness of deployment artifacts related to work-items, and to relay feedback to team(s) on issues during each deployment.
• Agile Product Manager provides feedback so that executive leadership can make go/no-go/defer decision based on that feedback.
Process: Option 2: “Continuous” Delivery with Frequent ReleasesRelease Prep
(Portfolio Kanban System)
Deploy to Production (Release Engineering Kanban System)
Kanban/Scrumban in the middle
Continuous Improvement
Release Planning
14
Agile and DevOps RACI matrix*
* This is only an example. Modify based on your org structure and needs!
4 main roles:•Product Owner (PO)•Scrum Master (SM)•Scrum team member•Ops team member
Other pertinent roles: •Proxy-PO•User Experience Lead•Architect•Agile Program Manger•Functional Manager•Agile Product Manager
Process Step ->-----------------------------------
Existing\New Role
Portfolio Kanban System
Release Planning
Sprint Start-
Kick-off
Sprint Execution- Delivery
Sprint End-Acceptance
Release Engineering
Kanban System
Agile Product Manager R R I C C I
Product Owner C R R R A I
Systems Architect A C C C C C
Team Lead C C C R R C
Functional Manager C C I C I I
Agile Program Manager I A R C I I
Scrum Master I R A C I I
Scrum Team Member I C C A C C
Ops Team Member I C C R C A
Ops Manager I I I C C R
15
15
16
Process: Scrumban Process Framework for DevOps• Scrumban is our recommended process framework for DevOps. • Dev and Ops are part of a single team following Scrumban and effectively handling Dev and Ops user stories.
• Dev and Ops are on a single team.• Alternatively Dev and Ops will be on one team, supported by (horizontal) Central Ops team.• Team has single product owner who owns both Dev and Ops requirements in single product backlog
and has authority to prioritize between Dev and Ops work items
• Based on history of unplanned work items, effort division is considered. E.g. 80% planned work (including Dev and Ops) and 20% unplanned work is expected. In this example, let us assume-• 70% Dev efforts would be required• 30% Ops efforts would be required• Out of 30% Ops efforts, 10% may be planned and 20% would be unplanned work.
• Team will plan for 70% of planned Dev work as part of scrum (it could be less, depending on buffer).• Team will plan for 10% planned Ops work as part of scrum.• Planned work (both Dev and Ops work items) will be stack ranked i.e. prioritized for sequencing.• Remaining 20% bandwidth is reserved for Kanban– WIP limits to funnel the flow of unplanned work
e.g. WIP limiting the expedited severity 1 issues
17
Process: How to use Scrumban Process Framework for DevOps
• From Day 1 of sprint until unplanned high priority incident is raised, team works on prioritized user stories.• As soon as an unplanned severity 1 incident is raised, priority is set as 0 for this incident. • A designated or team member having expertise in that area takes up this work item, while deprioritizing the
current planned work item.• Once the unplanned incident is completed and delivered for release, this team member resumes work on
the planned work item.
• If the volume of unplanned work items becomes higher than 20%, PO ensures that some stories are deprioritized and impact on planned work is acceptable by the key stakeholders.
• Similarly, if there are less unplanned incidents, the available bandwidth is used to complete any planned Dev and Ops user stories.
• It’s a continuous process to observe and control flow of unplanned work (with WIP limits) and accordingly adjust the planned vs unplanned distribution for upcoming sprints to take appropriate number of work items in scope.
• Due to single team and single product backlog, it is easier to ensure both Dev and Ops work on agreed priority and have common goal of delivering business value during each sprint.
• Team can initially start with separate Dev and Ops resources; and slowly learn complementary skills.• Single PO helps in reducing Dev and Ops conflicts.• Following scrum ceremonies ensures that best practices like retrospectives are followed in DevOps.• Following Kanban ensures that due attention is given to high priority items and are released with no
constraints of aligning with remaining Dev items.
Execution
Exceptions
Benefits
18
Process: Key Considerations for Adopting Process Frameworks with DevOpsDev teams need to start thinking beyond feature development. Ops teams need to start thinking beyond infra support.
What are availability, capacity, security, service levels and service continuity requirements? What is the plan to meet those? What infra capacity is required to achieve operational goals and sustain it? What is the impact on ongoing operations? What constraints does it put on design and development? What are the requirements to manage operational issues & support? (scripts, alerts, logs, trails…) How are knowledge sharing requirements? How to enable it through documentations, cross-silo and trainings? How to do incremental validation of build quality for production release?
How to ensure production release readiness? Is there plan for proactive detection and escalation of possible failures? How to implement plan to enable continuous monitoring of applications with metrics, alerts, etc.? How to avoid future issues by quick resolution of service interruptions and root cause analysis? How to provide feedback into development for improvement in specific operational aspects?
How to manage changes in requirements for impact on operational goals? How to ensure that appropriate runtime infra setup is available? How to validate operational goals against evolving design? How to standardize release & deployment plan? How and what deployment & release artifacts should be made available?
• Adopt a correct team structure like single DevOps team or single team with Central Ops
• Adopt a right process which can cater to both Dev and Ops needs, e.g. Scrumban
• Have an integrated product backlog for better prioritization among Dev and Ops work items
• Have a single product owner (PO) to manage integrated or otherwise both Dev and Ops backlog. PO should have authority to resolve dependencies and conflicts
• Ensuring PO has clear visibility of both Dev and Ops work, and is accountable for business value delivered
• Understand the need of DevOps teams to work on high severity items which demands them to stop work on existing work items. Such context switching may impact throughput. DevOps teams should be measured on reduced change lead time, not velocity/productivity
• Focus on tools and technologies which enable easier release management considering needs of DevOps teams to deploy multiple times while still continuing development of planned work
19
Process- Key Success Criteria
End to End Atlassian Tool Chain for DevOps with SAFe Constructs
Development ToolsAtlassian BitBucket (GIT)
Developer writes code and check-in
Associate JIRA Issue Types(Tasks / Bugs)
Build gets triggered
Microsoft Visual Studio (For .NET) Eclipse (For JAVA)
Atlassian BAMBOO for CI
Build Definitions for: Database .NET Java
Atlassian BAMBOO for Continuous Deployment (CD)
Approve / RejectBuild
DIT1
Functional Testing
• Continuous Integration• Scheduled Build
Approve / RejectBuild
DIT2
Approve / RejectBuild
SIT
Regression Testing
Selenium
Approve / RejectBuild
UAT
UAT Testing
Approve / RejectBuild
PROD
Functional Testing
DEVELOPMENT BranchQA BranchMASTER BranchFEATURE Branch
Service & Data Mock
Epicse.g. Ship to Home Pre-pay
(JIRA Issue Type)
User Stories(JIRA Issue Type)
Monitoring
Ngaios
Initiativese.g. Sports on Spectrum
(JIRA Issue Type)
Release Stream aka Agile Release Train (ART)
Themes(Strategic Goals)
Test Planning Test Execution Traceability & Reporting Defect Creation in JIRA Track Coverage & Quality Metrics
UX Team DB Team ART Engineer Product Managers
Note: Integrate Atlassian Bamboo with “Chef” for non-windows deployments
Develo
pm
ent
Ta
sks
Visualize Portfolio Timeline by Releases
EasyBI plugin(Reporting & Charting)On top of existing JQL queries
Visualize Investment by Themes
PMO-level
Status Reporting
Testing Tasks
20
Kanplan: backlog for Kanban and Scrumban teams
20
During planning, create org alignment to improve flow (1st way of DevOps), using Portfolio Kanban system.
21
Lean Governance Model- Applying 3 ways of DevOps
Enable managers in shaping various communities of practice: create culture of continual experimentation & learning (3rd way of DevOps).
• Begin with succinctly defined portfolio hierarchy or swim lanes that consist
of strategic (CAPEX) and tactical (OPEX) investments:
• Themes
• Initiatives
• Features
• Create single-piece flow of portfolio items with Portfolio Kanban board.
Apply lean metrics for continuous improvement.
• Apply first way of DevOps: emphasize on performance of entire system by
analyzing value streams, WIP, lead times & due date performance.
During execution, enable continuous feedback loops (2nd
way of DevOps) with obstacle boards to act on
impediments & dysfunctions.
• This requires low-fidelity obstacle escalation process.
This can be a 2-level (team->management) process, or
a 3-level (team->program>portfolio) process.
• Example of obstacle board and related workflow:
Upstream Kanban:
Downstream Kanban:
22
• Begin with the end in mind!
• Use Portfolio Kanban board across Org Design (Team of Teams)
• Use a simple Kanban board with “ToDo -> Doing -> Done”, “Plan->Build->Run” or similar workflow
• Visualize portfolio feature level bottlenecks to "Stop Starting, Start Finishing"!
• Supplement Portfolio Kanban system (“Upstream Kanban”) with Release Engineering Kanban system (“Downstream Kanban”)
Run
F
E
I
Idea Deploy to Prod
G
D
GY
PB
MN
2∞
AB
Biz DesignPlan Build (Dev & QA)
Architecture, resource plan Core team OMS …
3 3
DE
Explicit policy: Feature is “done” only when last team working on it has completed “Deploy to Prod” Lead time ends at
1st infinite queue! There could be additional queues with no WIP limit.
Lead time “clock” starts at 1st queue i.e. WIP limitedinput queue!
3 3Deployed
(Note touch time)
NPD (New Prod Dev) (60%)
Defect&Tech Debt
(30%)
Innovation(10%)
5
2
1
Expedite 1
3
Fixed Date
Standard
Intangible
2
2
3
WIP Limitsbased on classes of service(Release Engineering)
WIP Limitsbased on work types(feature teams)
Lean Portfolio Management and Release Engineering
0
20
40
60
80
100
120
140
160
48 51 4 6 8 10 12 14 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
2012 2013 2014
Throughput(# of resolved tickets / week)
Average Lead Time / week
Average Flow Efficiency % / week
Metrics: Lean Metrics for DevOps Teams
* Lead Time Distribution indicates Predictability* Average Lead Time impacts SLA Adherence* High Productivity means better “Flow Efficiency”* Monte-Carlo forecasting for planning fixed bid projects
How do you coach a kanban team on adjusting their "Total WIP“? By observing trends in throughput, lead time & flow efficiencies!
Apply Little’s law:Total WIP (for each workday) = Avg. delivery rate (per workday)* Avg. lead time (in workdays)
23
Key Capabilities- Begin with End in Mind
Desired Capabilities
(Work Streams)Description
Reduced Change Lead Time
DevOps Ecosystem Visibility -people, process & tools
Quality Built-In
Build & Test AutomationZero
Manual Intervention (single click deploys)
Release Automation Zero config/env level
dependencies
Agile Testing: Code commit should include TDD, BDD & regression tests. Enable a fully integrated deployment pipeline, integrating code & tests with quality metrics.
Infrastructure as a Code: Ability to virtualize services and third party systems to enable testing without any dependency on config/env. Multiple environments can spin up and down to support multiple deployments.
CI & CD Implementation:
Code commit in CI-CD pipeline should automatically trigger test scripts & notify results for each build and deployment.Mechanism to automated deploy the successful build in System Test (aka pre-prod) environment.
End to end requirements traceability
Socialize DevOps approach (with training & coaching) :Approach that visualizes DevOps Ecosystem with tool chain synchronized from portfolio to deployment. It integrates People (changing roles) into Process (“no” 1-size-fits-all). It deals with kaizen bursts from “To Be” value stream map.
Our experience
says
• At least 10~15% reduction in configuration management defects and ensures traceability
• 30~40 reduction in effort & time for deployment and re-deployment thereby allowing more time to work on more user stories
• Elapsed time from code check-in to testing to reduce by 35% ~ 45%
• Improve early detection of defects at least by 30%leading to Reduction in cost of rework by 20%
DevOps Goal #1: Visibility at all levels
DevOps Goal #2: Frequent Releases
24
Prioritization of Work Streams: Value Management
Ease of Implementation
HighMedLow
Low
Me
dH
igh
Val
ue
to
Bu
sin
ess
ID Work Streams
WS01 DevOps Ecosystem Visibility-people, process and tools
WS02 Quality Built-In
WS03 Build & Test Automation
WS04 Release Automation
ID Quick Wins
QW01 Lean/Agile/DevOps Training (WS01)
QW02 Lean Portfolio & Operations (WS01)
QW03 Migrate from manual tests to BDD (WS02)
QW04 MVD & (proactive) Code Reviews (WS02)
QW05 Builds integrated with xUnit & BDD (WS03)
QW06 Experiment with “Infra as Code” (WS04)
Desired Capabilities (Work Streams)WS02
QW01
WS01
WS03
QW05
WS04
QW04
Top 6 Quick Wins
QW03
QW06
QW02
25
DevOps implementation roadmap
Quality Built-In
DevOps Ecosystem Visibility
Release Automation
Build Automation Test Automation
Training: Release Planning (Mock Training)
Visibility
3~4 Months 4~5 Months
Tools Standardization
& Rationalization
Build CI Pipeline (Proof of Concept)
Build CD Pipeline (Proof of Concept)
Integrate Builds with CI
Integrate Dev Testing with CI
Automate Deployments to Dev & QA Environments
Deploy To System-Test/Pre-Prod
Decide on tools & plugins for Agile Testing: Unit test frameworks, Integration testing w/ UX &
service layers, Behavioral & Regression testing
Define Test Automation Strategy Integrate Existing Test Cases with pre-defined “Acceptance Criteria” within CI/CD Pipeline
Define “Infra as Code” StrategyAllocate infra & implement tools
Prepare configuration files (e.g. Puppet manifests) and Pilot for a Project
Integrate With CI/CD Pipeline
QW01 QW02
QW04
Monitor, Review & Improve
Test Automation Framework: POC & Rollout
Extend to Program
WS02
Integrated tool chain for Functional and NFR (Non-Functional Requirements)
5~6 Months
Training: Kanban basics(GetKanban Game)
Lean/Agile/DevOps Training for teams, leadership,
product, project, program, & functional management
Training: Advanced Kanban(2 workshops- team process
selection & lean metrics)
Training: CI, Agile Testing-Concepts, Tools & Plugins
Training: CD,Automating Infra w/ Code,
Atlassian tool chain, Amazon cloud infrastructure
Are We
There Yet?
About to Walk
Ready?
You Are Here!
Crawling now
Inspect & Adapt
#1
Inspect & Adapt
#2
Inspect & Adapt
#3
Tactical Strategic
Review current landscape around People, Process & Tools using formal assessments, leadership team feedback & informal input via open spaces for course correction
Start Release Planning (Scrum In the middle)
Create Kanban teams (Kanban In the middle)
Scrumban for DevOps (Scrumban In middle)Start Operations (Downstream) Kanban
Manage Portfolio w/ Kanban
Manage Operations w/ Kanban
Start Portfolio (Upstream) Kanban
Review current Governance & Metrics
Enable Changing Roles & Responsibilities (multi-site)
Training
Enable Lean Governance Model w/ Lean Metrics
1~2 Months
QW03
QualityWS03
QW05
Build & Test
Automation
WS04
QW06
Release
Automation
2~3 Months
26
Creating DevOps Culture with Agile, Lean, Lean Startup and TQM
Top down approach:
30-60-90 strategic plan for continuous improvement
with baseline assessment of DevOps capability maturity
Bottom up approach:
Crowd sourcing tactical plan for continuous improvement
with assessment based open space events
Inside out approach:
PDCA Kanban board for feedback driven visibility
with Lean Startup method (build-measure-learn loop)
and TQM (PDCA loop)
Implementing DevOps requires mindset & culture that is open to all 3 approaches!