take the red pill: how criteo revamped its software development process

40
TAKE THE RED PILL DR. ADRIAN PERREAU DE PINNINCK

Upload: adrian-perreau-de-pinninck

Post on 29-Nov-2014

68 views

Category:

Engineering


2 download

DESCRIPTION

These are the slides of the talk I gave at the Global Scrum Gathering Berlin in 2014. It describes how Criteo managed to change the software engineering culture of its R&D department from 'Testing is cheating' to 22K+ tests being run every half hour.

TRANSCRIPT

Page 1: Take the Red Pill: How Criteo revamped its software development process

TA K E T H E R E D P I L LD R . A D R I A N P E R R E A U D E P I N N I N C K

Page 2: Take the Red Pill: How Criteo revamped its software development process

T R E M E N D O U S G R O W T H . C R I T E O : G O G O G O ! ! !

• From 20 to 160 engineers in less than 5 years

• 160 code repositories

• 7 million lines of code

• Took months to release

Page 3: Take the Red Pill: How Criteo revamped its software development process

W E M U S T S E E R E A L I T Y T O P U T T H E M A C H I N E S U N D E R C O N T R O L

Page 4: Take the Red Pill: How Criteo revamped its software development process

W H AT W A S R E A L I T Y ?

• Average lead time for new features = 60 days

• Average time to merge commits = 20 days

• The release pipeline was blocked for months

• It was getting worse each year

Page 5: Take the Red Pill: How Criteo revamped its software development process

S O M E T H I N G H A D T O C H A N G EI T W A S G O I N G T O TA K E T O P M A N A G E M E N T T O A G R E E O N S O M E T H I N G L I K E T H I S

Page 6: Take the Red Pill: How Criteo revamped its software development process

H O W W O U L D Y O U S E L L T H I S I N I T I AT I V E ?

PA I R D I S C U S S I O N :

Page 7: Take the Red Pill: How Criteo revamped its software development process

O B J E C T I V E S

• Reduce commit merge time to hours instead of days

• Reduce time from commit to Prod to < 4 days

• Reduce number of cancelled release candidates

Page 8: Take the Red Pill: How Criteo revamped its software development process

T E C H N I C A L S O L U T I O N

• Tailor-made Build System

• Mandatory Code Reviews

• Continuous Integration Pipeline

• Sandbox to execute tests

Page 9: Take the Red Pill: How Criteo revamped its software development process

P R O C E S S F O L L O W E D

Page 10: Take the Red Pill: How Criteo revamped its software development process

E S TA B L I S H S E N S E O F U R G E N C Y

• Code freeze in 2012 crippled the team during months

• The release pipeline had been blocked for months

• Things weren’t getting better for 2013’s code freeze

Page 11: Take the Red Pill: How Criteo revamped its software development process

E N E R G Y W I L L D I S S I PAT EW I T H O U T A S T R O N G G U I D I N G C O A L I T I O N

Page 12: Take the Red Pill: How Criteo revamped its software development process

S T R O N G G U I D I N G C O A L I T I O N

• Executive Vice President of Engineering

• Vice President of QA

• 10 Software Engineers, high level of seniority

• 2 Senior Engineering Program Managers

Page 13: Take the Red Pill: How Criteo revamped its software development process

D E V E L O P I N G A C H A N G E V I S I O N

• Brainstormed for the key elements

• Prepared a document explaining the vision

• Shared the document with Key players

• Modified the document to take into account feedback

• Shared the document with Dev Leads

• Modified the document to take into account feedback

• Gave presentation explaining the project

• Sent document to all

Page 14: Take the Red Pill: How Criteo revamped its software development process

C H A N G E V I S I O N

• 26 page document

• Took over a month to get alignment in version 0.1

• Sent to 20 senior engineers for feedback

• Last version took into account all feedback

Page 15: Take the Red Pill: How Criteo revamped its software development process

C O M M U N I C AT E T H E V I S I O N

• Monthly updates at all hands meetings

• Weekly emails with progress reports

• Events with food

Page 16: Take the Red Pill: How Criteo revamped its software development process

E M P O W E R B R O A D - B A S E D A C T I O N

Page 17: Take the Red Pill: How Criteo revamped its software development process

E M P O W E R A C T I O N

• The improvement backlog was shared with everyone

• Some items prioritised by voting

• Voluntary-based work encouraged

• Test-a-thons and other wide range initiatives

Page 18: Take the Red Pill: How Criteo revamped its software development process

S H O R T- T E R M W I N S

• Used Scrum to guide the change process

• Developed usable tools from the beginning

• Developed one tool at a time

• Pushed adoption slowly until tipping point

• Forced rest to adopt once tipping point passed

Page 19: Take the Red Pill: How Criteo revamped its software development process

I N T E G R AT I O N T I M E S

Page 20: Take the Red Pill: How Criteo revamped its software development process

0.00%$

10.00%$

20.00%$

30.00%$

40.00%$

50.00%$

60.00%$

70.00%$

80.00%$

90.00%$

100.00%$

12/11/13$

12/18/13$

12/25/13$

1/1/14$

1/8/14$

1/15/14$

1/22/14$

1/29/14$

2/5/14$

2/12/14$

2/19/14$

2/26/14$

3/5/14$

3/12/14$

3/19/14$

T E S T P L A N

Page 21: Take the Red Pill: How Criteo revamped its software development process

P R O J E C T B U R N D O W N

Page 22: Take the Red Pill: How Criteo revamped its software development process

0"

10"

20"

30"

40"

50"

60"

70"

80"

90"

100"

5/13/13" 6/13/13" 7/13/13" 8/13/13" 9/13/13" 10/13/13" 11/13/13" 12/13/13" 1/13/14" 2/13/14" 3/13/14" 4/13/14" 5/13/14"

Commit&to&Prod&Lag&C O M M I T A G EF R O M 3 5 D A Y S T O 1 5 D A Y S

Page 23: Take the Red Pill: How Criteo revamped its software development process

D O N ’ T L E T G OI T ’ S A LW A Y S H A R D E R T H A N Y O U T H O U G H T

Page 24: Take the Red Pill: How Criteo revamped its software development process

L A R G E S T I M P E D I M E N T S

• Technical

• Eat your own dog food

• Replicating the Sandbox

• Human

• People added to the team not sharing vision

• Getting teams to use the tools

• Managing expectations and communication

Page 25: Take the Red Pill: How Criteo revamped its software development process

D O N ’ T L E T G O

• Fully dedicated team for a whole year.

• After the project ended the team remained as a new unit.

• Even through the hardest times management gave full support.

Page 26: Take the Red Pill: How Criteo revamped its software development process

I N C O R P O R AT E C H A N G E I N T O T H E C U LT U R E

Page 27: Take the Red Pill: How Criteo revamped its software development process

I N C O R P O R AT E C H A N G E I N T O T H E C U LT U R E

• Training program for new recruits

• Engineering partners are expected to conform to new way of working

• Training program for engineering partners

• Tools in place become the cultural gates

Page 28: Take the Red Pill: How Criteo revamped its software development process

T H E R E I S M O R E T O C H A N G E T H A N K O T T E R

O T H E R T E C H N I Q U E S W E U S E D

Page 29: Take the Red Pill: How Criteo revamped its software development process

B R I D G E B U I L D E RW E A S K E D T H O S E A L R E A D Y O N B O A R D T O H E L P O T H E R S D O T H E S A M E

Page 30: Take the Red Pill: How Criteo revamped its software development process

C H A M P I O N S K E P T I C

Find someone senior in the organisation:

• That wants the goal to be reached

• That doesn’t fully support your technical solution

• Talk with him regularly

Page 31: Take the Red Pill: How Criteo revamped its software development process

D E D I C AT E D C H A M P I O N

Y O U W O N ’ T G O FA R W I T H O U T O N E

Page 32: Take the Red Pill: How Criteo revamped its software development process

D O F O O DM A K E S U R E T O B R I N G S N A C K S T O G E T T O G E T H E R S

Page 33: Take the Red Pill: How Criteo revamped its software development process

E A R LY A D O P T E R SW I L L I N G T O T R Y T H I N G S B E F O R E T H E Y A R E R E A D Y

Page 34: Take the Red Pill: How Criteo revamped its software development process

E X T E R N A L VA L I D AT I O N

L I N K E D I N W E N T T H R O U G H T H E S A M E P R O C E S S

Page 35: Take the Red Pill: How Criteo revamped its software development process

G R O U P I D E N T I T Y

M A K E P E O P L E P R O U D O F B E I N G PA R T O F T H E C H A N G E

Page 36: Take the Red Pill: How Criteo revamped its software development process

I N Y O U R S PA C EW E U S E D L A R G E S C R E E N S T O S H O W P R O G R E S S

Page 37: Take the Red Pill: How Criteo revamped its software development process

I N V O LV E E V E R Y O N EI F Y O U W A N T L A S T I N G C H A N G E …

Page 38: Take the Red Pill: How Criteo revamped its software development process

U S I N G S C R U MA N D W E D I D I T A L L

Page 39: Take the Red Pill: How Criteo revamped its software development process

- A G E N T S M I T H

“Never send a human to do a machine’s job”

Page 40: Take the Red Pill: How Criteo revamped its software development process

R E F E R E N C E S

• Leading Change - John P. Kotter

• Fearless Change - Linda Rising, PhD

• Continous Delivery - Jez Humble, David Farley