‘agile on steroids’: devops led enterprise agiledocuments.grenadine.co/pmi bangalore india...
TRANSCRIPT
‘Agile on Steroids’: DevOps led Enterprise Agile
Transformation
A Case Study
Sub-Theme: Strategies to transform organisation
Abstract ID: PMIBC-17-1-002
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 2
CONTENTS
Abstract ............................................................................................................................................................................................................... 3
Introduction.......................................................................................................................................................................................................... 3
Details of the paper ............................................................................................................................................................................................. 4
Conclusion ........................................................................................................................................................................................................ 19
References ........................................................................................................................................................................................................ 20
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 3
ABSTRACT
Meeting the demands of an evolving marketplace is a constant challenge for most organizations. Product
development teams are transitioning to Agile development after decades of Waterfall. These teams are pushing the
operations groups to move even faster; are consuming more and more resources; and stepping across traditional
boundaries once exclusively the domain of IT Operations.
The struggle between achieving business goals (Dev!) and maintaining stable and secure infrastructure (Ops!) can
bring these two ends of any organization to instability. How do we tightly align these two loosely coupled groups?
DevOps is quite a buzzword these days. Everyone is starting DevOps teams, hiring DevOps engineers, and buying
DevOps tools. However, hiring engineers and buying tools constitutes only a portion of the effort.
This paper is based on our experience of using DevOps as fail fast approach to bring in a DevOps cadence, quick
error detection, faster automated testing and overall delivery efficiency through a Deployment pipeline. Further, the
paper will discuss a case study in detail with the 4 pillars of DevOps strategy to achieve the desired agile
transformation - Culture and Mindset, Micro Services based architecture, Built in Quality and Tools &
Automation.
INTRODUCTION
With technology disrupting every industry, products and processes that we used in the past, no longer fits in the
solution framework for the future. Organizations are looking for a new, faster and innovative service delivery structure
to serve their customers. As organisation started evolving, silos started to build up.
Agile software development has broken down some of the silos between requirements analysis, testing and
development. Deployment, operations and maintenance are other activities which have suffered a similar separation
from the rest of the software development process. The DevOps movement is aimed at removing these silos and
encouraging collaboration between development and operations.
DevOps has become possible largely due to a combination of new operations tools and established agile engineering
practices, but these are not enough to realize the benefits of DevOps. Even with the best tools, DevOps is just
another buzzword if you don't have the right culture.
The primary characteristic of DevOps culture is increased collaboration between the roles of development and
operations. There are some important cultural shifts, within teams and at an organizational level, that support this
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 4
collaboration, aimed at increasing collaboration and cooperation among different functions of an organization to
deliver faster, better, reliable and high-quality software solutions.
As per Gartner and Riley, 2017 is the year of DevOps transformation in majority of organizations, where we stop
talking about DevOps and start doing it:
According to 2017 DevOps State Report, in 2017, high-performing DevOps teams deploy 46x faster, enjoy 440x
faster lead time for changes, recover on average 96x faster, and suffer 5x fewer change failures. These organizations
were more than twice as likely to benefit from higher quality and quantity of products and services, better operating
efficiency and higher customer satisfaction, among other business impact goals.
DETAILS OF THE PAPER
HOW IT ALL STARTED – Silos started building across the organisation
The DevOps movement started to coalesce sometime between 2007 and 2008, when IT operations and software
development communities got vocal about what they felt was a fatal level of dysfunction in the industry.
They railed against the traditional software development model, which called for those who write the code to be
organizationally and functionally apart from those who deploy and support that code.
Developers and IT/Ops professionals had separate (and often competing) objectives, separate department
leadership, separate key performance indicators by which they were judged, and often worked on separate floors or
even separate buildings. The result was siloed teams concerned only with their own fiefdoms, long hours, botched
releases, and unhappy customers.
Surely there’s a better way, they said.
THEN…… AGILE HAPPENNED
Delivering products faster with better quality has led to agile practices that are dramatically impacting how
organizations deliver software. As agile delivers shorter and smoother development cycles - known as sprints -
SILOS STARTED BUILDING ACROSS…
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 5
organizations are enjoying less risk by deploying smaller development packages with shorter delivery cycles. There
is also a greater involvement of the part of development with understanding business needs and requirements.
Operations teams however, are finding themselves struggling to deal with releases on a much more accelerated
basis. This has created many software release challenges.
While agile has helped improve development productivity and quality, the next natural step of the software delivery
evolution was to take agile to production - linking development with operations - which gave birth to DevOps. DevOps
is a set of practices and principles intended to help development and operations work more effectively together. The
challenge is how to effectively implement DevOps in a demanding business setting.
• Focusing on faster development cycles packed with features…
• Documentation to working software each iteration
• Waterfall releases to Incremental high value feature releases
• Dev + Test – one agile team with cross functional skills
LEADING TO… ‘WALL OF CONFUSION’
Developers always want to deliver changes as soon as possible. Operations want reliability and stability.
Lee Thomson describes this as a wall of confusion between development and operations. This wall of confusion
not only exists between the mind sets of the two teams but also with the tools they use. Development uses some
tools and operation uses some other tools to perform the same stuff. Some of the paradoxical dilemma that exists
between Dev and Ops are as below:
Development Team Operations Team
Fast Innovation System Stability
Code Streaming Waterfall release
Easy Access Locked Security
Agile Delivery Project Planning
More features More simplicity
BUT SILOS CONTINUED DEV + TEST …….. OPS
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 6
Smoothen the development cycle Smoothen the delivery cycle
This wall of confusion has the following impact to the delivery and development cycles and ultimately to the customer:
• Too costly defects found in the production
• Delay in Speed time to market (production)
• Dev and ops with different mindsets
• Traceability and feedback becomes difficult
• Blame game starts due to Human errors leading to havoc in the system
• Fail to adapt to changing requirements
• Ultimately. Dissatisfied customers
DEVOPS ADOPTION – BREAKING DOWN THE WALLS OF CONFUSION
DevOps is a cultural shift that breaks the silos between operations, development, tools and business to deliver a
competitive edge to the business.
Patrick Debois, who’s often called “the father of DevOps”, coined the word “DevOps” in 2009. As the word depicts,
it was formed by combining two words: “development” and “operations”. DevOps is a collaborative way of developing
and deploying software. DevOps (a portmanteau of development and operations) is a software development method
that stresses communication, collaboration and integration between software developers and information technology
(IT) operation professionals.
• DevOps is an approach based on agile and lean principles in which business owners, development,
operations, and quality assurance team collaborate to deliver software in a continuous stable manner
• DevOps is an environment that promotes cross practicality, shared business tasks and belief
• DevOps is a movement that improves IT service delivery agility
• DevOps is a culture that promotes better working relationship within the company
• DevOps is a set of practices that provides rapid, reliable software delivery
DevOps break down the walls of confusion between development and operations team, unifying development to
operations for better, faster outcomes.
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 7
AGILE + DevOps – The new enterprise agility
Agile and DevOps is the new enterprise agility. While Agile is about building quality products with shorter and
smoother development cycles, Dev Ops is about shorter and smoother delivery cycles to take agile to production in
order to achieve the following results:
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 8
• Develop and test against production-like systems
• Speed time to market ---> speed time to production
• Shift left development and operation concerns
• Deploy with repeatable, reliable process
• continuously dev, test -- Continuously dev, test, build, and release
• Independent testable and deployable unit
• Reduces defects cycle time – ability to reproduce and fix defects faster
• Fast Feedback (including visual)
• Consumer-centric and Value driven
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 9
• Optimizing on speed and quality
• Minimizes Rollbacks
According to McKinsey. companies can reduce the average number of days required to complete code development
and move it into live production from 89 days to 15 days, a mere 17 percent of the original time (Exhibit 1).
Exhibit 1
Case Study – Leading Industrial Automation Company
HARMAN partnered with a leading Industrial Automation company to support their end-to-end PDLC services for
mission critical projects. The engagement spanned across Business Consulting, Agile Transformation, DevOps,
Development, QA/Automation, Release Management, Sustenance to accelerate product development, test and
deploy with discipline and rigor. Product was Envisioned to Re-platform and Integrate over 20 different products
developed across 6 different geographies from a Monolith Architecture to a Cloud First service driven platform
enabling Cross sell / Up sell features across products. They intended to increase their global foot print thereby
expanding their customer base into Commercial, Residential and Enterprise Space with an aim to reduce release
cycle times from 6 – 8 months to 2 – 3 weeks and stay current in the market
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 10
ENGAGEMENT SUMMARY
- Team size – 200+
- Scrum teams – 14 scrum teams + 5 supporting teams (HUI, DB, API, DEVOPS and Architectural Runway)
- Release trains - 2 (Software and Firmware)
- Program duration – 3 yrs.+ roadmap
- Complexity includes Software and Firmware development teams.
CHALLENGES
• Monolithic code base posing challenges to build, test and deploy for every changes
• 25 + Services with manual Deployment, Manual Testing, 500 odd config changes ---errors leading to
unstable build
• Turnaround time to handover a refreshed build for QA = >1 week
• Environment creation was manual and tedious ---lack of automation
• No Built in quality like Unit tests, Static code checkers etc
• Dev doing the ops with very limited knowledge of tools, release management, dependencies, config.
Changes
• Tools – Just Bamboo and Manual Deployment
• Limited Safety Nets no API and UI functional tests
SOLUTION TENETS
HARMAN leveraged DevOps and Automation to achieve Fail fast approach, seamless integration and release for
large teams bringing in efficiency and productivity gain of up to 30%. HARMAN created a DevOps roadmap that
covered people, process and technology. We designed a unique operating model that leveraged SAFE Agile to drive
frequent releases and Synergies. HARMAN support DevOps culture and defines 4 essential pillars that will help you
to map out a DevOps journey for your company.
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 11
Agile Transformation:
SAFe was implemented across globally distributed teams involving software, firmware and hardware. The teams
were distributed across the following locations:
• Bangalore (Harman-India)
• Bangalore (Customer-India)
• Madurai (India)
• Shanghai (China)
• Bruno
Culture and Mindset:
DevOps isn’t just another software development methodology. It is a culture; an intersection of people, products and
processes to develop a performance-driven team structure. It isn’t for specific people or team, but for everyone who
is involved in delivering value to end users.
For DevOps to create an effective and seamless collaboration, the entire organization needs to embrace the mindset
and culture of DevOps. For E.g. The Entire infrastructure of Amazon is set up to do things faster than anywhere else
with the basic principle ‘you build it, you run it’. Following were some of the implementations:
- Train the team and make them aware about the impact of their mistakes
- Build the dev-ops team with App experts and tool experts
- Integrate code daily (Continuous Integration)
- Run Continuous Delivery (CD- complimenting CI)
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 12
- Promote builds on various environments
- Set up DevOps Cadence (Sample)
- Avoid name and shame but educate the team on implications
- In certain organizations, there is a penalty or made them wear a cap for breaking the builds
etc. should be avoided)
- Honor and respect the Master branch and always maintain the sanctity of it
For successful adoption and transition to DevOps, one needs massive business transformation initiatives along with
transparent communication across teams to ensure proper alignment. Such DevOps practices will help organizations
to provide faster, stable and continuous deliveries to survive dynamic market conditions.
Adapting Micro Services Architecture:
Organizations have been using monolithic architecture for long now. Monolithic architecture is like your big container
that has all the business services tightly assembled and coupled together in a single unit.
Due to its tight packaging of software applications in a single unit, the deployment of services become difficult
because even if a small component in the application needs a change, the whole application needs to be re-packaged
and re-assembled. It leads to high cost, reduced agility and scalability.
But, with time, companies realized the importance of micro services over monolithic architecture. Organizations
started looking micro services as an enabler of faster and agile application delivery. Micro services architecture
breaks down massive software projects into smaller, independent modules which are then defined for specific tasks
which can communicate with each other through simple APIs.
Micro services gave the concept of ‘fail fast and fix faster’. With micro services, parts of applications with errors can
be rectified independently and can be deployed easily and separately without touching the entire application. Smaller
codebases can be easily maintained by smaller teams which results in increased agility, scalability and reliability.
Thus, micro services make the organization nimble and agile in their development efforts by improving the uptime
and speed of execution. Following were the implementations:
- Monolithic to Micro-services
- Windows Services
- Easily deployable for changes
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 13
Building Quality-driven Mindset:
Organization needs to maintain synergy between quality and speed for the successful business delivery.
Quality assurance in DevOps organization is not limited to finding bugs but to prevent them from arising in the first
place. Software Development Life Cycle (SDLC) in DevOps culture requires rapid software development followed by
continuous testing, continuous deployment and continuous monitoring. In such a process, manual testing has to be
simply ruled out and test automation has to be deployed.
Automating the deployment process in DevOps environment leads to faster time to market, higher quality software
solution and increased organizational effectiveness.
DevOps practices accelerate the process of software delivery using Continuous Integration (CI) and Continuous
Delivery (CD) which allow business units to collaborate and function faster.
Few more important methods for Quality improvement are:
• Developing a Quality Improvement Plan based on customer feedback and internal defects
• UI, API, UT should be made part of Sprint development process so that automation becomes integral part
of delivery which would otherwise lead to rise of technical debt
• Address the technical debt incrementally (first by removing blockers then by performing code refactoring
wherever possible and lastly by resolving critical issues)
• Design & develop quality gates (UT >=90%, Code Complexity <=15, Code duplication <=5%) and strictly
adhere to it
• Infuse code checks (SonarQube, PCLint, KlockWorks, Coverity, Fortify etc.)
Tools and Automation:
As the companies are rushing to take their software early-to-the-market, the entire software development life cycle
(SDLC) is going through a paradigm shift. In the DevOps environment, you need to maintain a continuous feedback
system and with a lot of information moving back and forth, logging everything becomes critical.
- Therefore, integrating a tool system that lets you collaborate across development, testing and deployment
would save considerable amount of time which otherwise would take days during the DevOps
implementation.
- Tool selection is also important as it should reflect and adhere to a common set of business objectives
while providing seamless collaboration and integration between tools.
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 14
- Tools can be used to capture request and feedback. They can be integrated with survey platforms and can
provide customer feedback while social media platforms can be integrated to provide the team with real-
time feedback. It can later help in optimizing the products and spotting the gaps, issues and inefficiencies
among the team.
- In DevOps environment, deploying the right set of tools to the right users isn’t enough. A suitable
connectivity between the DevOps tools is imperative to create a collaborative work environment among
the team.
- Bring redundancy in tool chain (especially on Web - Azure, AWS, Verizon). Even a minor interruption in
network can cause build/deployment failure. So, manual intervention is required. For ex. If build failed to get
a NuGet package, re-trigger. Unable to push package to Octopus (Deployment tool) due to network error,
re-push..
- Choose always minimum tool set or otherwise, providing stable cadence is challenge (DevOps cadence is
end-to-end delivery, so, breakage between any tool set would break the entire chain and hence, failure)
BENEFITS TO THE ORGANISATION AND CUSTOMER
Before After
• Multiple, inorganically grown product bases
• Large amount of feature overlap among products
• Micro Service based architecture
• Standardized repository layout
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 15
• Monolithic code base posing challenges to build,
test and deploy for every changes
• 25 + Services with manual Deployment, Manual
Testing, 500 odd config changes ---errors
leading to unstable build
• Build times in excess of an hour
• Tools used in silos
• Ad-hoc library distribution/non-reproducible builds
• Complex and manual release process
• Migration from SVN to modular Git repositories
initiation
• Any new environment creation was a hectic task
and usually took almost a week time with lot of
stabilization issues
• Replicating the same environments was a
daunting task
• Turnaround time to handover a refreshed build
for QA = >1 week
• Environment creation was manual and tedious ---
lack of automation
• Limited Built in quality like Unit tests, Static code
checkers etc
• Dev doing the ops with very limited knowledge of
tools, release management, dependencies, config.
Changes
• Tools – Just Bamboo and Manual Deployment
• Limited Safety Nets no API and UI functional
tests
• Faster CI build times (takes ~ 5 mins currently)
• Sane branching and streamlined code reviews
with PRs (Feature enabled by using Git – Code
reviews become an online discussion that are
always in context of the change)
• Nuget for package distribution along with proper
versioning
• Nightly build and automated deployment prototype
• Code refactored with dependency injection
• Guided CI adoption across ~9 teams
• Modular MSI installers for each service
• ~Zero manual installation steps
• Feature toggling – Any feature can enable/disable
just by toggling
• New infra creation time has almost come down to
1 day
• Replicating environments is a mouse click
Practice Benefits
Sanity of master build - Dev CI
after every commit
- Highly stable build consistently across all environments
- Proper branching strategy, code review & merging process
- Faster CI build times (takes ~ 5 mins currently)
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 16
Scalable deployment model using
Micro services
- Capability of deploying any micro-service with a single mouse
click
- Supported multi environment, geographically distributed
DevOps Cadence - Disciplined DevOps Delivery (DDD)
Automated and more frequent
deployments
- Overall cycle time to hand over build to QA has come down from
5 days to 3.30hrs.
Daily CI and CD Build - Better stability
- Fail fast mechanism and prompt feedback via mails on CD
- CI builds for rapid feedback on every push
Faster recovery from failures - Daily visibility
Built in quality and safety nets - Shift left quality
- Integrated Built-In-Quality (UT, Static Code Check), if
parameters doesn’t meet goal, build will fail
- Involvement of Code Committers as part of review for every Pull
Request
- Growing set of unit tests that provide fail-fast behavior
- SonarQube for transparency on Code Quality
- Code Coverage/Unit Test reports [SonarQube]
- Code analysis report from SonarQube
- Nightly Builds with automated E2E test execution - Always in a
‘KNOWN STATE’
- API test suite integrated and runs as part of CD
- Selenium integrated and runs as part of CD
More (environments) with less
with latest code base
- Automation
No weekend or late night shifts
during release times
- Better Work life balance
Single view of current state –
JIRA, Stash, Bamboo and
Octopus tools integrated
- Integrated tools for better E-E- traceability
Release velocity over the past 10
releases
While, few builds appear to be failed, in actual scenario, it involves
whole build creation of all 16 micro services, UT, Static code check,
MSI creation and push the package to Octopus for deployment and all
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 17
are successful. However, below builds are connected to one more
build plan called ISP. Deploy. Master which triggers the Octopus
Deployment, where it failed because of low API test percentage. But,
end-to-end deployment is successful.
Average velocity of past 10 release is around 208 minutes which
includes Compiling+ UT+ Static Code Check+ MSI creation+ pushing
the package to Octopus for further Deployment
CRITICAL SUCCESS FACTORS:
• Average time after a check-in to build, test and deploy to an environment
Average Build time: 208 minutes
API Test execution time: 45 minutes (Report doesn’t have time stamp)
Redeploying the database: 15 minutes (We are compelled to redeploy DB because some of API test cases
do direct transaction with DB and populates lot of data. Currently tearing down is not happening.
Run Selenium Test suite: 6hrs (There are around 1000+ test cases that are distributed in 5 production grade
servers which fires through batch scripts in parallel)
TOTAL TIME TAKEN: 10hrs 28 minutes is overall cycle time
• Build promotion process & automation
Inferences: Build from Dev to Functional QA environment once API test percentage is greater than 70%. From
FQA to BETA once Selenium is greater than 80%.
• Build traceability back to issue (JIRA) and commit (Version Control)
• Build reproducibility i.e. ability to re-generate any past build with 100% exactness
Octopus has a provision of restoring any package back to date and choose the package and deploy. However,
as per retention policy, we are keeping only 10 packages. Even said that, restoring single package alone may
DEV FQA BETA
API > 70% Selenium >
80%
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 18
lead to service disruption given the context of dependent services also should be deployed. The dependency
matrix is already available and based on that the Services will be chosen for rollback plan.
• Code review process: For eg. Is it possible to revisit an old ticket, see the code that was written and the code
review feedback for the said code
Before merging the code from Future branch to Master branch, developer has to raise the Pull Request (PR).
This Pull Request has to be approved by a role called Code Committer (CC). The review comments given by
CC has to be fixed and once CC approves PR, the commit is authorized to merge to Master
• Current system to manage code quality (static analysis, code duplications)
• DevOps Cadence
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 19
LESSONS LEARNT
Business today is compelled to deliver business technology in cycles of hours, or days. Faster cycles render
not only tradition “waterfall” processes and silo based IT obsolete, it also renders traditional metrics ineffective!
These arcane metrics no longer deliver the visibility and granularity tech pros need to fine-tune their delivery
capability. The mission has transitioned to RAPIDLY deliver high quality, high value solutions. For all, this is a
significant shift from the past, when the main points of focus were schedule, cost, and efficiency. Modern
software metrics — speed, quality, and value — are based on continuous feedback from business partners
and customers. Some of the key takeaways, listed and briefly described below.
CONCLUSION
Waterfall gave way to V-Model which in turn was replaced by Agile as the preferred choice for software
development. DevOps is a natural evolution of the software industry, enabling organizations to do more with fewer
resources while differentiating leading and successful companies from their competitors. DevOps aim is aligning
development and operations roles and processes in the context of shared business objectives, and organizing them
into principles and practices.
Big visible dashboards
Teams across the organization get real-time information on the state of the software system, including build status, customer metrics, and availability.
Colocation Teams are in physical proximity to one another to enhance communication.
Continuous integration
You build your software (environments, applications, and so on) with every change.
Cross-functional teams
Software-delivery teams are composed of experts across various disciplines, including programmers, testers, analysts, and operations.
Polyskilled experts
Reduce specialist silos by expanding skill sets on cross-functional teams.
Scripted deployments
Deploying software to environments is entirely scripted so that it can be run from a single command.
Scripted environments
Creation of environments is entirely scripted so that it can be run from a single command.
Self-service releases ("You build it, you run it.")
Any authorized person on the team can and does perform deployments to production.
Stop the line Anyone can and should stop the continuous-integration system when necessary.
Test-driven everything
Write automated tests for everything: application, infrastructure, everything. This might include writing unit, acceptance, load, and performance tests.
Version everything
Version all artifacts: infrastructure, configuration, application code, and data.
Project Management Practitioners’ Conference 2017
www.pmibanga lorechapter .o rg
Page 20
Successful DevOps requires a symbiotic blend of culture, tools and methods. These pillars will allow organizations
to achieve a 360-degree view on how to build a DevOps strategy for their business to function faster and reduce risk
during SDLC through automation or shorter development iterations. It’s a continuous improvement cycle that
software development models undergo from time-to-time. You need to embrace, understand and inculcate it. You
need to master the various automation and continuous integration tools, so that your automation efforts add value
to the chain and are lean enough to quickly adapt to changes.
DevOps is the future….
REFERENCES
[1] Tech Insights Report: What Smart Businesses Know About DevOps, September 2013
[2] https://www.linkedin.com/pulse/devops-creating-agile-steroids-culture-stefano-la-porta
[3] DevOps for Dummies – Sanjeev Sharma
[4] https://devops.com/devops-2017-reflections-predictions/
[5] http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/beyond-agile-reorganizing-it-for-
faster-software-delivery
[6] https://puppet.com/resources/whitepaper/state-of-devops-report
[7] http://www.gartner.com/newsroom/id/2999017
[8] http://dev2ops.org/wp-content/uploads/2010/02/WallOfConfusion.png
[9] http://www.collab.net/solutions/devops http://www.collab.net/solutions/devops