Analytics and Witch Doctoring:A Cure for the Black Box

Mentality

February 1, 2011

O’Reilly Strata Conference

J.C. Herz, Batchtags LLC

jc@tripledex.com

Analytics: An Intervention

• Original SinEnterprisePathologies Critical Questions

Analytics: Occult Phenomenon

Very powerful

Don’t understand it

Practitioners possessarcane knowledge

Alchemy

SecretAlgorithms

Greek Letters Are YourKryptonite

High Status Helplessness

• If you understood the technology, you’dbe one of those people whose job it isto make technology work.

• You know, underlings

Executive ADD - Kaching!

• Re-starts are whereconsulting shopsmake their money

Mid-Life Crisis

Shiny Pebble Syndrome• Infoviz Porn: Visualization with no

use case– Ex: Social Network visualization. Why?– START with a use case and work

forward

• Demo Envy: just because it looksslick doesn’t mean it’s possible, oreven advisable, to pipe your datainto it.

A Ballad of SpectacularInformation Display

• Time Magazine 1976• Telex text routing:

information off the wiregoes to terminals, properlyfoldered

• Z8 terminal display awesexecutives

• Pneumatic system noteliminated

Technical Reality vs.Leadership Attention Span

Half Ass Syndrome

• Halfway into the project, jump off intothe next problem.

• Haven’t refined results or hypothesis

• Failure blamed on technology, but it’sreally loss of interest and desire forinstant gratification

Shelfware Syndrome• The guy who was driving the program left...• Approach-Avoidance conflict --> pilot-itis• A US agency has $30M of software that

hasnʼt been installed…some of it withmaintenance contracts.

• Base Model vs. Fully Loaded– One enterprise bought $12M worth of

Autonomy before figuring out that the add-onsthey needed would be another $22M.

Customization Before Testing

Critical Question: What is theValidation Test?

• Formulating the validation test keepsboth the customer and the developerfocused - and honest

• Suggest pay for performance, and see ifthe developer or vendor freaks out.

• Make sure validation is ongoing - incase the ground is shifting

Data Due Diligence & Auditing

ShameShame

Ugly Babies

Critical Question: Data Quality

• How complete is it?– Ex: 600 custom fields, only two have more than 50%

coverage

• How accurate is it? How do you know?

• How consistent is it?

– Good test: make three calls to different partsof the company, to get an answer to a factualquestion that doesn’t require calculation.

Critical Question: Half-Life of Data

• How long is thedata accurate?

• How long is thedata useful?

Critical Question: Real World Context

• Without real world data, “behavioral” metricsare misleading

• Where is the transactional data that validatesinsights from non-transactional data?

• How would you prove the magic analyticsWRONG?

• Are you prepared to spend painfulamounts of money cleaning up your data?

• Crack heads if people don’t share data?• Make business units accountable for their

data?• Play hardball to make sure data is not

stored in single-application proprietaryformats?

Data: Gut Check

Critical Questions: Workflow• What workflow changes will this

proposed capability require?• People hate changing their workflow,

even if it’s an improvement• Never attribute to stupidity what can be

attributed to laziness• What is your plan for changing

workflow? How do you enforce it?

The perfect application thatno-one uses is still worthless

Process

Politics

What are you prepared to do?

Critical Question: Consequences

• What actions are you willing to take onthe basis of validated analytic insight?– Change your product?– Change your marketing budget?– Change people’s job descriptions?– Re-allocate R&D budgets?

• What actions are you not willing to take?

Stakes

Critical Question: Tempo

• How fast will a decision be made on thebasis of analytic insight?

• Quarterly?• Daily?• Within seconds?• Milliseconds?• Never?• Realtime vs. Continuous vs. Batch

Precision vs. Accuracy

When precision exceeds accuracy, you’resetting yourself up for analytic failure

OODA Loop

Which of these does an analytic tool/technology do?

Before You Rip ‘n’ Replace:

What is the exit cost of thistechnology?

Does “turnkey” meanmonoculture?

Business Payoff vs.Intellectual Appeal

Social NetworkAnalysis

OperationsResearch

MarketSegmentation

CompetitiveIntelligence

Pilots to test newanalyst tools withtiny amounts ofgeneric data

360ºLeadScoring

Validate MarketingEffectiveness

Questions?

• J.C. Herz jc@tripledex.com (202) 213-3151

Base Model vs. Fully Loaded

“Never attribute to stupiditywhat can be explained by

laziness.”

Ugly Babies & Pretty Babies