devopsdays amsterdam - monitoring at service provider scale

1

MONITORING AT SERVICE PROVIDER SCALE June 23, 2014

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

INTRODUCTIONS

CHRIS JACKSON Rackspace CTO (DevOps Practice)

chris.jackson@rackspace.co.uk

@chriswiggy

+44 (0) 7712 854462

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 3

Image courtesy of http://www.toastmasters.krakow.pl/

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4

THE DEVOPS RACK APPLIANCE…

ALL OF THE THINGS IN 3-CLICKS!

Image courtesy of Axiomtek

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 5

HOW DO WE APPLY DEVOPS TO SCALE SERVICE?

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

THE DEVOPS PRACTICE AT RACKSPACE

6

THE FIRST WAY – CREATE A DELIVERY SYSTEM

THE SECOND WAY – AMPLIFY FEEDBACK LOOPS

THE THIRD WAY – LEARN & EXPERIMENT CONTINUOUSLY

This is PRODUCT & SUPPORT delivering great service powered by automation

!

Our DevOps practice is a way to accelerate feedback by removing complexity

!

The goal is to never hold our customers up, we must always be faster…

!

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

SCALING TRANSFORMATION

7 SCALING TRANSFORMATIONS

Image courtesy of Wordpress

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 8

SO WHAT ABOUT MONITORING?

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 9

WHY #MONITORINGSUCKS FOR RACKSPACE

We generate over 40,000 events per second on just our Cloud Monitoring platform.

If a customer is Managed, we troubleshoot ANY alarm that is raised.

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 10

WHY #MONITORINGSUCKS FOR RACKSPACE

Image courtesy of Spot On Lists

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11

WHY #MONITORINGSUCKS FOR RACKSPACE

Image courtesy of Cepolina

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 12

WHY #MONITORINGSUCKS FOR RACKSPACE

Image courtesy of New Line Cinema

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 13

WHAT ARE WE DOING TO FIX THIS?

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

SCALE MONITORING TUNING

14

WHEN TUNING IS MANUAL: EFFORT=tN

Where t = time to configure a single node and N = number of nodes

0 100 200 300 400 500

Effo

rt

# Nodes

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

SCALE MONITORING TUNING

15

WHEN TUNING IS INTELLIGENT: EFFORT=t1-N

Where t = time to configure a single node and N = number of nodes Ef

fort

# Nodes

Shrink this!

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

TOOLS FOR INTELLIGENT CONFIGURATION

16

Configuration Management If you know what you want to measure!If your constraint is just the number of nodes!Gets complicated when you’re thresholding all of the things…!

Configuration Discovery We call it Waldo, you know it as Sartori in Openstack!Uses ohai-solo to find information about your configuration!Compare to best practice and make changes accordingly…!

Image courtesy of Walker Books Ltd

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

MONITORING DATA ANALYTICS

17

EVENT GENERATION EVENT PROCESSING

EVENT HANDLING

EVENT ANALYTICS

TUNING INSIGHT

CREATE FEEDBACK LOOPS INSIDE MONITORING REMOVE WORK BY TUNING ALARMS

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

INTELLIGENT ALERTS

18

ANOMALY MONITORING PATTERN RECOGNITION

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

WHAT ARE WE USING?

19

statsd

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

TOOLCHAIN INTEGRATION

20

EVENT GENERATION EVENT PROCESSING

EVENT HANDLING

EVENT ANALYTICS

TUNING INSIGHT

statsd

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

HELP US TO DO THE HARD WORK

21

YOU

US

Things We Want to Know™!How do you use monitoring?!

How intelligent is the tuning?!

What use of anomaly or threshold?!

Approach for creating tool chains?!

What do you dislike doing?!

We’ll build it and open source it!!

Image courtesy of Dave Hogan/Getty Images

Twitter - @chriswiggy http://www.rackspace.com/devops http://devops.rackspace.com

22

THANK YOU

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

MORE THAN A CLOUD COMPANY

WE PROVIDE

SUPPORT FOR EVERY STEP OF YOUR JOURNEY

JUST STARTING? RACKSPACE ADVISORY

SERVICES

NEED SERVICES? RACKSPACE HYBRID

CLOUD

NEED SUPPORT? RACKSPACE DEVOPS

AUTOMATION SERVICE