Dynamics Day 2015

Systems of Intelligence in Action

Experimentation!=

Learning From Mistakes

1MALLEABLE: It must be cost effective to change the way that the system or process behaves. Ideally for just a subset of usage or users.

WHAT WE NEED

2

3

OBSERVABLE: The system or process must be sufficiently well instrumented such that we have the information available to test the hypothesis.ENGAGED: The system or process must be being used at sufficient scale and in a sufficiently ‘real’ environment for any results to be significant.

An experiment is a procedure carried out to verify, refute, or establish the validity of a hypothesis… Experiments vary greatly in goal and scale, but always rely on repeatable procedure and logical analysis of the results.

SYSTEMS OF EXPERIMENTATION

MALLEABLE

OBSERVABLEENGAGED

Experiments

MALLEABLE

OBSERVABLE

ENGAGED

“arguments about whether or not a feature idea is worth doing or

not generally get resolved by just spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users.”

There are over 500 different K-Cups

There Are Cartridge Loaded Software Systems

Microsoft Azure Demand Center

Internal Systems & Product/Platforms

Dynamics CRMMarketo

Web Email Call Center

Data Warehouse

PowerBI

SystemsOf

Record

SystemsOf

Experimentation

SystemsOf

Engagement

MALLEABLE

OBSERVABLE

ENGAGED

Storage is Cheap…

…Data Is Not

Smart Appliances (Think Printers + Ink)

Existing ‘Dumb’ Product

SensorTagIoT Suite

Smart Screen Mobile Apps(s)

Raw Data Store

???

Personalization Platform

PowerBI

SystemsOf

Experimentation

SystemsOf

Engagement

MALLEABLE

OBSERVABLE

ENGAGED

“…Our tool works best when you’ve got at least 1 million or so

app installs to work with…”

Towards Systems

of Intelligence

Flipping The Model

Inputs OutputsIT System

ExpertiseInterpretationExperimentation

Flipping The Model

Inputs OutputsIT System

ExplanationExperimentation

Automation

Models

Machine Learning

Example:Building SystemsProvider (~HVAC)

Much of the Tooling is Free

Is Becoming More Widely

Available

Is Becoming Easier to Use

And Is Entering The Public Consciousness

And can be crowd-sourced

In Boots & All

Use Cases We Are Seeing

Big Data Use Cases, BARC Research 2015

STOREOPTIMIZATIO

N

TRACKING, TARGETING

SEGMENT OF 1SMART

PRODUCTSSMARTASSETS

PRODUCT IMPROVEMEN

T

CUSTOMERACQUISITION &

RETENTION

USER ANALYTICS

&OPTIMIZATiON

SERVICE/B2B RETAIL SUPPLY CHAIN

FRAUD&

COMPLIANCE

• Embrace Experimentation• Malleable, Agile, Cartridges• Observable; Capture More Data• Engaged; Ship Early & Learn

• Systems of Intelligence• Doing Data Science <- Easier• Knowing What We Can Do <-

Hard• Analytics Inside Your

Organization• Survey for Opportunities• Most CRM Systems• Customer Facing Apps/Sites• IoT