Transcript
Page 1: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Effecting a DevOps Transformation at Salesforce.comAgile 2013

Dave MangotArchitect, Infrastructure Engineering

@davemangot

in/dmangot

Karthik RajanVP, Infrastructure Engineering

@kabbyr

in/karthikrajan

Page 2: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Safe Harbor

Safe harbor statement under the Private Securities Litigation Reform Act of 1995:

This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services.

The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site.

Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

Page 3: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Karthik Rajan

Responsible for the infrastructure platform for running the salesforce.com service -

“Infrastructure as Code”

5 years @ salesforce.com, in various roles - developer to management

Photographer. Petrolhead.

Collector of many many gadgets

Page 4: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Dave Mangot

• Mainly focused on automation, monitoring, orchestration and lastly...cultural change!

• Systems Engineer 15+ years, a variety of companies big (Cable & Wireless, Sun Microsystems) and small (Tagged.com, Terracotta)

Page 5: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

The 5 Why’s:

• Why are we the same? Different?• Why didn’t we transform TechOps on the first try?

• Why are we succeeding?• Why we still have challenges

• Why are you here?

Page 6: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Why are we the same? Why are we different?

http://flic.kr/p/eeWbP8 http://flic.kr/p/eeWbGp

Page 7: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

A little story...

Page 8: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

It was the year 2000

Page 9: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Number of people

Page 10: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

fast innovativesmart

Page 11: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Number of Major Releases per year

Page 12: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

7 years later

Page 13: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

rapid success

Page 14: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

59,300+ Customers

10 B transactions/Quarter

tons of people

Page 15: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

it was getting more difficult to deliver

Page 16: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Features Delivered per Team

Days between Major Releases

Page 17: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Number of Major Releases in 2006

Page 18: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

2007 – Birth of ADM

Page 19: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Major R&D wide Agile Transformation to ADM

Page 20: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

On time delivery?

Last waterfall release

Page 21: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

But what about infrastructure?

Page 22: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

More importantly, here’s a story of a typical start-up

We’ve got an app…

Page 23: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

We’ve got a customer or two…

Web

DB

LogsConf

Data

Stat

Hlth

ID

App

Othr

AppLogsConf

Data

Stat

Hlth

ID

App

Othr

LogsConf

Data

Stat

Hlth

ID

App

Othr

Customer

Admin

Page 24: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

I’ve got a few more customers…

DB LogsConf

Data

Stat

Hlth

ID

App

Othr

FW

LBLB

FW

Page 25: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Now we’re making money…

DB LogsConf

Data

Stat

Hlth

ID

App

Othr

FW

LBLB

FWFW

FWFW

FW

LBLB

LBLB

DB LogsConf

Data

Stat

Hlth

ID

App

Othr

Page 26: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Salesforce.com Architecture

Page 27: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Why didn’t we transform TechOps on the first try?

Page 28: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Delivering on business priorities

Scaling through hiring

Still a relatively small team

Page 29: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

But thenRapid success continues

Page 30: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

100,000+ Customers

1 B transactions/day

tons and tons and tons people

Page 31: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

So can we continue to

innovate at scale?

Page 32: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Lack of visibility

http://upload.wikimedia.org/wikipedia/commons/thumb/4/46/Morning_Fog_at_GGB.JPG/448px-Morning_Fog_at_GGB.JPG

Page 33: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Resource Bottlenecks

Page 34: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Lack of responsiveness, lack of team alignment on priorities

Page 35: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Unpredictable completion of projects and initiatives

Page 36: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

2012 – Infrastructure Take 2 ADM + DevOps

Page 37: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Take 2a

Developers supporting Operations (part time)

Page 38: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Take 2b

Infrastructure Engineering

TechOps splits into

Operations

Page 39: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Take 2c

Clouds

Embedded teams

Page 40: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

How to use DevOps?

Page 41: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

No Crisis

Page 42: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

The 3 Ways – Gene Kim

The First Way emphasizes the performance of the entire system

The Second Way is about creating the right to left feedback loops.

The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure;

Page 43: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

C A M S

Page 44: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Culture

Page 45: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Automation

http://upload.wikimedia.org/wikipedia/commons/6/61/Differential_free.png

Page 46: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Metrics

http://www.aosabook.org/images/graphite/composer-ui.png

Page 47: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Sharing

Page 48: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Why are we succeeding?

Page 49: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

The Toyota Way - Kata

http://upload.wikimedia.org/wikipedia/commons/7/70/Iaido2.jpg

Page 50: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

ADM for Cultural Change - DevOps Kata

•Daily Standup - #8 “Encourage effective two way communication and other means to drive out fear throughout the organization so that everybody may work effectively and more productively for the company.”

•Sprint Retrospective - # 13 “Institute a vigorous program of education and self-improvement”

•Sprint Demo - #9 “Break down barriers between departments. People in research, design, sales, and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.”

Page 51: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Cloud Realignment

http://www.freefoto.com/images/46/01/46_01_49---Clouds_web.jpg

Page 52: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Theory of Constraints as per Goldratt

http://images4.wikia.nocookie.net/__cb20080623095705/starwars/it/images/thumb/d/d4/JawaNEGAS.jpg/331px-JawaNEGAS.jpg

Page 53: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Hack Day

Page 54: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Internal DevOps Mini-Conference

http://www.flickr.com/photos/gazeronly/7645165306/

Page 55: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Infrastructure Development Lifecycle

http://www.freefoto.com/images/46/01/46_01_49---Clouds_web.jpg

Artifact Y Deployed to Environment X from release store

Test Results marked in ReleaseDB

Eligible forpromotionto next environment

Environment specific tests are Run(increasing clock time)

PASS?

Ejected fromprocessFAIL?

Testing and Deployment

Page 56: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Virtualization

Page 57: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Embeds

http://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Embedded_Artillery_Shell_at_Fort_Sumter_%287639234388%29.jpg/800px-Embedded_Artillery_Shell_at_Fort_Sumter_%287639234388%29.jpg

Page 58: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Scrum Master Training

Page 59: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Failure? Still?

http://theintentionalwriter.com/wp-content/uploads/2013/01/head_in_hands.jpg

Page 60: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Front Door Process

http://upload.wikimedia.org/wikipedia/commons/thumb/e/ef/Alnwick_Castle_a_frontal_door.JPG/450px-Alnwick_Castle_a_frontal_door.JPG

Page 61: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Bugs

http://upload.wikimedia.org/wikipedia/commons/thumb/9/9b/Cotton_Harlequin_Bugs.jpg/757px-Cotton_Harlequin_Bugs.jpg

Page 62: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Why we still have challenges

Page 63: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Why are you here?

Page 64: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Present and Future Challenges

• Bringing Agile into traditional IT Ops

• Bringing IT Ops in with Infra Eng and R&D

• Re-educating workforce

• Recruiting

• Scaling Securely

Page 65: Agile.2013.effecting.a.dev ops.transformation.at.salesforce

Top Related