deploying software is hard

Deploying Software is Hard

@masteinhauser / Myles Steinhauser - IBM Blue Box 1

Boston DevOps Meetup - September 21st, 2016

Who is Blue Box?● OpenStack as a service● Dedicated● On-Premises● Started OpenStack in Oct, 2013● Myles joined in June, 2014● Acquired by IBM Cloud in June, 2015

blueboxcloud.com

2@masteinhauser / Myles Steinhauser

http://blueboxcloud.com

http://blueboxcloud.com

What’s this really about?

Reliable People

Solid Process

(Always) Improving Tools

3

People

4

Chris Wheeler / OpenStack Ops Team Lead

● Transplant from IBM Ops● Has the connections● Knows the knowledge● Always delivers

@masteinhauser / Myles Steinhauser 5

Ulysses Kanigel / OpenStack Ops Team Lead


● From Blue Box Ops● Has the history● Knows the knowledge● Always delivers● Expert at deep-diving on problems

Sina Sadeghi / OpenStack Ops Team Lead

● New hire to IBM Blue Box● Our man “Down under”● Runs 2nd shift● Knows the knowledge● OpenStack Expert● Always delivers


Jesse Keating / OpenStack Eng Team Lead

● From Blue Box Eng● Ansible Expert● OpenStack Expert● Organizes all the things

(This is his PR reviewing face)


Paul Czarkowski / SiteController Eng Team Lead

● From Blue Box Eng● (Really loves to cook)● Container Expert● Ships POCs faster than Myles can cry● SiteController’s ☔


… and oh so many more!

< insert collage of all team members here ;) >


Process

11

gonna

● Compliance is hard● Silos make Auditors happy● “Change Control”● Escalate all the problems● Firefight all the deployments

… It wasn’t working.


Screw. That.

● Overcome Silos with Communication

● Engineer <-> Engineer● Engineer <-> Operator● Product -> Engineer● Support -> Engineer● Support -> Product

Do not be afraid to speak up. Share your perspective!


OpenStack

● Get features from Product● Get defects from Ops● Cut releases Quarterly (cry inside)

● Monitoring / Alerting Checks● Logging configurations


SiteController

● Proxy and Mirror everything● Monitoring pipeline● Alerting pipeline● Logging pipeline● SSH Bastion● Single Pane of Glass


Ops

● UPDATE ALL THE THINGS○ When deployments break pull in Engineering

● Tier 1 & 2 for all alerts● Escalate to OpenStack & Sitecontroller On-call● Fix the issues together

○ Fix the code or document the failure so Ops can handle in the future.

● Ops has knowledge of hardware and recurring problems Eng doesn’t.

“DevOps” doesn’t mean the same thing in every context.Sometimes limited by what is organizationally possible.


Tools

17

Tools

1. Slack2. Github Enterprise3. Github4. Ursula5. Ansible6. Phone Calls7. Trello8. Video Conferencing / Whiteboarding9. “Off hours”, remembering to take time away from work

1000. IBM Notes


Really, what I’m trying to say is...

19

Above all else,the teammatters most.

20

So have fun,And let’s ship something great.

21

Want to chat more?Let’s grab ☕
