#atlassian

NOTES:

• Your main title goes in the large blue

font.

• If you have a title that naturally splits

into a subtitle, use the smaller green

font for the subtitle. If not, delete the

subtitle

PHOTO

1. Place your photo at around the

same size as the example photo

2. (Keynote users:) Move your photo

onto the blue shape below Select both

photo and shape and then choose

“Mask with selected shape” from the

menu. Double click the photo to edit

the scale and crop position.

ZANE SALIM • SENIOR TECHNICAL PM • TWITTER • @ZANES

Scaling Product Experimentation with JIRA and Confluence

Technology is wiping out companies faster than ever

Companies that stop innovating die… fast

NOTES:

• If you want to divide your talk

into chapters, use this slide for

Chapter titles

How are you innovating?

1. Our approach @twitter

2. Challenges at scale 3. How we scaled 4. What got better

5. Final takeaways

1. Our approach @Twitter

NOTES:

• If you want to divide your talk

into chapters, use this slide for

Chapter titles

@Twitter we innovate through experimentation

Anyone can launch an experiment

Unleashes creativity

Drives user insights at scale

100s every quarter

Experimentation is the new black

… and the old black

“Negative results are just what I want. They’re just as valuable to me as positive results. I can never find the thing that does the job best until I find the ones that don’t.”

― Thomas Edison, Harper’s Magazine (Vol. 165, p 405)

“We must conduct research and then accept the results. If they don't stand up to experimentation, Buddha's own words must be rejected.”

― Dalai Lama XIV, A Policy of Kindness (p 31)

“There must be some ability to experiment to advance the state of the art… In the early days of aviation there was a great deal of experimentation and a high death rate. We don't want that … but by the same token we can't have a situation where no deaths are ever allowed, because that would put innovation in a coffin too.”

– Elon Musk, L.A. Times interview (August 1, 2012)

“There’s a wonderful thing about learning as a group. A fundamental part of that is making mistakes together. There’s no learning without trying lots of ideas and failing lots of times.”

― Jony Ive, Telegraph interview (May 23, 2012)

Both refinement and exploration

Example: which drives more clicks?

Example: which drives more retention?

2. Challenges at scale

Visibility

Learning

Collaboration

Rigor

3. How we scaled

COMMON JIRA WORKFLOW FOR COLLABORATION

Lifecycle of an Experiment

Lifecycle of an Experiment in JIRA

Lifecycle of an Experiment in JIRA

KNOWLEDGE MANAGEMENT IN

CONFLUENCE

Confluence: Knowledge Management

Confluence: Experiment Brief Templates

Confluence: Experiment Brief Templates

Experiment Brief Experiment Write-up Retrospective Analysis

Describe the problem Describe the experiment Present retrospective analysis

Define a hypothesis Present the results Present the results

Design the experiment Offer take-away’s Offer take-away’s

Confluence: Reviewed Experiments & Learning

HYPOTHESIS TREES

Visualizing Focus Areas & Insights

4. What got better

Doubled completed experiments

Reduced cycle time up to 50%

Learning org à smarter experiments

5. Final takeaways

Experimentation drives innovation

Collaboration & learning essential

Smart use of tools can make you 4x more effective

Thank you!

ZANE SALIM • SENIOR TECHNICAL PM • TWITTER • @ZANES

Example: <made up example experiment>