from 0 to 1000 apps: the first year of cloud foundry at the home depot

From 0 to 1000 AppsThe first year of Pivotal Cloud Foundry @ The Home

Depot from a platform operator’s perspective.

From 0 to 1000 AppsThe first year of Pivotal Cloud Foundry @ The Home

Depot from a platform operator’s perspective.

3100

introductionssetting expectationsreminisce & learnaha momentquestions

Agenda

Well, hi!

15 years

Anthony McCulleyManager, Application PlatformsThe Home Depot anthonymcculley amcculley@gmail.com

Startups

Platform operator perspective

Being a platform operator

A year-long retrospective

A surprise reveal at the end…

A surprise reveal at the end…

We ready?

Jun 2015

June 2015

?PCF installed on-prem

Jul 2015

July 2015

?

July 2015

Lesson #1

Be empathetic to the people, processes, and technology already in place.

July 2015

2 teams using Pivotal Cloud Foundry (Pivotal Labs)3 foundations (Sandbox, Nonprod, Prod)? App instances

Aug 2015

August 2015

Dev Forum: Introducing PCF

Lesson #2

Let your community know that a thing exists… and real.

August 2015 – PCF is here but has caveats

Platform requires permission to use itPlatform isn’t self-serviceNo idea how to charge customers for it…. so we don’t (yet)1st team goes to production… takes 4.5 hours!

Sep 2015

September 2015

Gave prod access to devs

Prod

Lesson #3

Use new things as an excuse to question everything and do things differently.

September 2015

Enabled dev teams to be Org Managers in ProdDecided on no chargebacks… PCF is “free” to useSeveral teams using their own Slack channels

Oct 2015

October 2015

October 2015

Not much going on…Data center incident impacts Nonprod for 5 daysGreat blameless retro as a resultAttend our first “Cloud Foundry Workshop” with Pivotal

Nov 2015

November 2015

Dev Forum: GitHub, Slack

Lesson #4

Establish a reliable cadence for communicating to community.

November 2015

Slack doubles overnight to 300 peopleWe pilot self-provisioning in Non ProdPivotal (Josh Kruck) asks me to consider running workshopsI give a videotaped talk at a Cloud Foundry meetup

This spawns internal discussions – “Orange Tape”

Dec 2015

December 2015

Make it accessible

December 2015

Josh asks (again) if we considered running workshopsSelf-provisioning added for ProductionWillItConnect madness beginsRemove need to request Org Manager

Lesson #5

Remove all barriers to entry – human or otherwise.

Jan 2016

January 2016

Pupil becomes the master

January 2016

We start running several PCF workshops2x a week, every week for next 4 months800 developers will eventually do workshop

Lesson #6

Go beyond operating the platform – be an advocate, a teacher, and a thought leader.

Feb 2016

February 2016

February 2016

Workshop tweaked and rebrandedRemoved deeper CF topicsAdded organizational change topicsGit, Slack, Full Stack, Paired Programming

My team adopts 100% paired “programming”

Lesson #7

Establish a scalable customer service and support model.

Customer Support

Encourage pair programmingDevs: Ask your pair -> then slack -> then the internet

We dedicate a pair each iteration to 100% customer serviceConversational support.

No ticketsNo queues

Mar 2016

March 2016

March 2016

250 devs go through Modern Development WorkshopHighest single month

We create #pcf-operators slack channelPublic channel for matrixed teams to engageInfrastructure, networking, Pivotal, platform, security

Apr 2016

April 2016

?

April 2016

Production is running out of capacity just as…

”PCF is the strategic platform of choice”

May 2016

May 2016

May 2016

Prod runs out of capacity

April 2016

We turn off self-provisioning in Production

Lesson #8

Don’t run out of capacity.

Lesson #8 (again)

No. Seriously. Don’t run out of capacity.

Get really good at monitoring capacity.

Lesson #9

Be careful about your choice of words.

“Use PCF” changed to “Build 12 Factor Apps”.

Jun 2016

June 2016

June 2016

We stand up 2nd production foundationWe increase capacity on existing foundationAnd then…

Red Alert: 1st Major Incident

Red Alert: 1st Major Incident

Its okay. We are all learning.

Lesson #10

Don’t over react to first incident. Avoid operational scarring.

Takes great leaders!

Lesson #11

Have a dedicated team… dedicated to PCF.

Lesson #12

Don’t hand “support” of platform off to another team.

Jul 2016

July 2016

Today

By the numbers

3000 Apps4000 App Instances5800 people in Slack – over 1100 of them in #cloudfoundry1300 unique platform users6 foundations (more planned EOY)14000 cf push – 1500 to Production

in June alone> 2 billion log messages a month

Lesson #12

Don’t police developers.

Study them with user analytics.

Lesson #13

Based on analytics, single out people and teams to have discussions with.

What’s next?

Motivate teams to be lean with capacity Showback modelquota vs. utilization ratios

Remove all matrixed dependencies for teamsAutomate network and security requests

Community-built smoke testsAdd app tests to our tests on changes

I promised an aha moment!

We don’t “operate a platform”.We enable developers.We are a product team.Developers are our customers.

Thanks! Any questions?