from dev and ops to devops - reconfiguring the plane in flight

From Dev and Ops to DevOps

“reconfiguring the plane while flying”

Mike Wessling

Lead Nerd @Bitbrains

Who is Bitbrains

• Bitbrains provides managed hosting for medium to large enterprises on our own shared platform. – Mainly for financials

– Often HPC platforms and other odd ducks.

• From Day Zero to the Bitter End™– We get involved as early as possible in projects.

– Prefer iterative design approach.• Start building early

• Think, build, test, review, repeat.

• 35 bitbrainers.

Old Organization

Fairly traditional along life cycle steps:• Project engineering (customer+platform)

– Presales consultancy– Proof-of-Concepts– Design– Implementation– Big Changes

• Operations (customer+platform)– Maintaining status Quo– Changes– Maintenance– User requests

• Separated by the big Ops Hand-over– 47 checks long. Some checkboxes are big..

Why Change?

• External influences:– Customers using agile project approaches.

• Even the big ones. • Traditional Waterfall doesn’t deliver.

– Customers want “try before you buy”,– Customers want “pay as you go”.– Applications getting bigger and more complex. – Growth!!

• Internal influences:– The organization didn’t reflect how things worked,– Projects rarely end… – And never get handed-over completely– Customer like the TLC they get in the project phase.

• Responsive and flexible. • SLAs and processes are less important• Willing to pay for it.

– 1 project = 1 engineer is vulnerable • Especially when running at 100%+ capacity

Result: Out of control in 2013

• Still delivering happy customers but– Project engineers overstretched

• Handovers suffered. • No time to make life easier.

– Customers stuck in project phase– Operations struggling with

• Partially handed-over customers, • Complex customers without history and experience• Customers not used to more rigorous processes• Out of date documentation.

– Internal improvements not happing– Platform support stuck between engineering groups.

• No time to grow• Organization can’t scale any further this way. • Engineers working on 20 or more projects

So Now What??

• Need to scale the teams

• Need to reduce pressure

• Need to get the process back in control

• Need to create space to grow.

• Need to get rid of the handovers

• Need to spend time on internal improvements

Lets try this DevOps Thingy

• Seems to solve key problems

– No to hand-overs

– Deals with the ‘Never ending’ projects

– Shared responsibility between Project and Ops.

• We are already doing it

– But Implicit, unstructured & out-of-process

• In our DNA – it is how we started

– So how to scale this….

DevOps? Ops ok But Dev?

In our enterprisey world:

• Read “Dev” as the team ”Specifying, designing and building the service.”

– Not just the “code-monkeys”

– “Coding the stack”

• Read “Ops” as the team “Running and delivering the service”

1st step: Creating Time and Space

• Nothing is going to change otherwise– Customer projects are always priority– Customer questions and requests are second

Approach:• Take work of the plates

– Outsource well defined projects. – Verify/renegotiate target dates.. How hard are they?– Teach Sales not to say yes to all dates. – Cancel/Delay projects if needed.

Enter KanBan

• Start with project engineering – Mature self-aware group – Role model for other engineers

• Start with a good coach (!)– External/neutral – Group will rebel or demotivated at some point

• Requires somebody who stands above the process

• KanBan will expose problems– Be prepared to deal.

• Look for any benefit– Being able to share the work across the engineers for example

• DON’T GET HUNG UP ON TOOLS..– Not important. – Distract – Force a way of working

Result

• Project Engineers happier– In control of work

– Less context switching

– Able to share the workload• New engineer productive after 3 days.

• Project Manager happier– Better overview

– No need to ask all the time

– Aware of the scary pile of work in the queues

• Operations eager to switch to KanBan

Next: Operations

• Bigger challenge– Vast majority of work are tickets

– Any projects are individuals doing their best

– More junior group

• First Results:– Insight:

• many tickets were parked– Returned to queue.

– Bad stats – Service management unhappy

• Lots of little side projects

Operations continued

• Insight:

– Need to add long term and short term engineers

• Short term to clear the heap.

• Long term to stay on top and have room.

• Creation of Platform team

– Different beast with different focus

– Big projects and operational role. (very DevOpsy)

towards: Dev+Ops = DevOps

• Working towards close alignment of processes

– Same Queues

– Same Definitions of Work

• Already work can travel across teams

• Coordinated Set of priorities across teams

– Service management plays key role

– Currently 1 week window, extending to 2 and 4 weeks.

The Hand-over Elephant

• The big divider between Dev and Ops

• Big To-Do list.

– Big block of work after the interesting bits are finished

• Used by Project engineering as buffer time.

• If Ops accept they are stuck with the customer

The Hand-over needs TO GO

But how?

• Engineers took as step back and looked.

• Hand-over must be continuous process.

– Stuff gets added and changed all the time

– Different bits of the environment are in different stages

• Integrated in the definitions of work.

Switch to DevOps Mode

• Flip the engineering teams from a Life Cycle organization to DevOps teams

– Won’t be using DevOps as name

– Teams are end-to-end responsible for a set of environments grouped by common factor and single board.

– Should be a small natural event.

– Number of teams: depends

The End

Questions??