Delivering Business Value Overnight with Ruby

Michael Pellon and Brad Jewell

What does Datafiniti do?

We target a set of domains for specific content (e.g., product details, business contact information, etc).

What does Datafiniti do?

We then use our globally distributed network of tens of thousands of desktop computers to crawl those domains and extract the relevant content.

What does Datafiniti do?

Our in house cluster turns all that raw HTML content into clean structured business, product or people records that are available for sale on our website.

What does Datafiniti do?

Our customers run queries to locate the data they are most interested in, purchase and then download that data instantly in a format of their choosing.

What does Datafiniti do?

They use this data to provide a wide variety of services ranging from:

Collecting Online Reviews

What does Datafiniti do?

They use this data to provide a wide variety of services ranging from:

Pricing Intelligence

What does Datafiniti use Ruby for?

➔ Managing offsite backups to AWS S3

➔ Payment processing (Sintra)

➔ Datafiniti.net (Rails)

➔ Web-based deployment console (Rails)

➔ Company-wide metrics (Sinatra)

Why Ruby?

➔ Best of Perl’s string processing and Unix integration

➔ Best of list processing from Lisp

➔ Best of object orientation from Smalltalk

➔ Best of iterators from CLU

➔ Consistent, practical, beautiful and fun

➔ Large, stable and mature community

A note about performance . . .

➔ Ruby’s performance is good on today’s fastest hardware. Not great. Good.

➔ Ruby is nothing but upside though from a future performance perspective ( a lot of low hanging performance fruit in ruby-land -- e.g., garbage collection maturity ).

Who are we?

We write code.

Who are we?

We write code.

But that is not our job.

Who are we?

We write code.

But that is not our job.

Our job is creating business value.

Business Value

Business value is anything that makes it more likely for people to give Datafiniti money.

Business Value

Our code generates business value when it runs not when we write it.

Business Value

So what does it do when it runs?

Business Value

Measure it.

Measurement

perception != reality

Measurement

Measuring makes our decisions better.

Measurement

But only if we are measuring the right thing.

Picking the right metrics!

Not all metrics are created equal.

We don’t have systems just to keep CPUs busy.

The purpose of a server is to do work so the metrics that measure the work being done are

more important than others.

Picking the right metrics!

Fault

the system is asked to perform work and it is not getting done

Don’t forget about the human!

Our monitoring system is made for humans to use.(easy to forget!)

OODA Loop

Observe Orient Decide Act

OODA Loop

Observe Orient Decide Act

OODA Loop

Observe

MetricsMonitoringAlertingAlarming

OODA Loop

AnalysisVisualizationCorrelation

Orient

OODA Loop

What shall I do?

Decide

OODA Loop

Decide

Recognition Primed Decisions

Rule-based Decisions

Choice Decisions

Creative Decisions

OODA Loop

Decreasing cognitive effort

Decreasing effects of stress

RPD Rule-Based Choice Creative

Increasing cognitive effort

Increasing effects of stress

“Information is not a scarce resource. Attention is.”

Herb Simon

Descartes

Graphite Carbon

App

App

App

Whisper

Metrics App

Metrics App Architecture

Demo

➔ Install 134 packages

➔ Modify 27 config files

➔ Create 3 users

➔ Start 8 processes

➔ Change 31 system settings

➔ 2.5 weeks

The Process

➔ Add more servers

➔ Different configurations

➔ Make a system-wide change

➔ Split up responsibilities

How does this architecture scale?

Chef!

What is Chef?

Ruby DSL

(Domain Specific Language)

Rails, Sinatra

➔ Run 1 command

➔ 11 minutes

The Process with Chef

What is Chef?

Figure out what needs to be done

Write it once

Use it repeatedly

Chef Server

CookbookCookbook Cookbook

RecipeRecipe Recipe

Chef Architecture

Server

Server