strata hadoop world 2015 context computing - jonas keynote - final

Context ComputingStrata + Hadoop World 2015

Jeff Jonas, IBM FellowChief Scientist, Context Computinghttp://www.twitter.com/jeffjonaswww.jeffjonas.typepad.com

Jeff JonasIBM FellowChief Scientist, Context Computing

Founded Systems Research & Development (SRD) in 1985

Architected, designed, developed roughly 100 systems over the last three decades

– Financial services– Defense, intelligence– Manufacturing – Humanitarian efforts

Acquired by IBM in 2005

Currently focused on Context Computing, Sensemaking and Privacy by Design

No Context

NewsletterSubscriber

Context

“Better understanding something by taking into account the things around it.”

I ducked as the bat flew my way.

Another exciting baseball game.

In Context

SocialMedia

Influencer

NewsletterSubscriber

LoyaltyClub Member

High ValueCustomer

JobApplicant Watch

Listed Party

Context Accumulating

ContextAccumulation

ContextualizedObservations

Observation(Any kind of data from

any kind of sensor)

Context Informs Decisioning

ContextAccumulation

ContextualizedObservations

ObservationIn Context

Decisioning

Data Finds Data Relevance Finds You

Observation(Any kind of data from

any kind of sensor)

The Puzzle Metaphor

Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes, colors

What it represents is unknown – there is no picture on hand

Is it one puzzle, 15 puzzles, or 1,500 different puzzles?

Some pieces are duplicates, missing, incomplete or have errors

Some pieces may even be professionally fabricated lies

Until you take the pieces to the table, it is nearly impossible to assess the scene

270 pieces90%

200 pieces66%

150 pieces50%

6 pieces2%

30 pieces10% (duplicates)

First Discovery

More Data Finds Data

Duplicates in Front Of Your Eyes

First Duplicate Found Here

Incremental Context – Incremental Discovery

6:40pm START

22min “Hey, this one is a duplicate!”

35min “I think some pieces are missing.”

37min “Looks like a bunch of hillbillies on a porch.”

44min “Hillbillies, playing guitars, sitting on a porch, near a barber sign and a banjo!”

150 pieces50%

Incremental Context – Incremental Discovery

47min “We should take the sky and grass off the table.”

2hr “Let’s switch sides, and see if we can make sense of this from

different perspectives.”

2hr10m “Wait, there are three … no, four puzzles.”

2hr18m “I think you threw in a few random pieces.”

How Context Accumulates

With each new observation one asserts: 1) unrelated; 2) related; or 3) connected

Must favor the false negative

New observations sometimes reverse earlier assertions

Some observations produce novel discovery

The emerging picture helps focus collection interests

Big Data [in context]. New Physics.

More data: better the predictions– Lower false positives– Lower false negatives

More data: bad data good– Suddenly glad your data is not perfect

More data: less compute

Big Data

Pile of ______ Information In Context

One Essential Form of Context: “Entity Resolution”

Is it 5 people each with 1 account or is it 1 person with 5 accounts?

Is it 20 cases of SARS in 20 cities or one case reported 20 times?

If one cannot count, one cannot estimate vector or velocity (direction, speed).

Without vector and velocity prediction is nearly impossible.

Who is Fang Wong?

Fang WongTop 100 Customer

F A WongSeattle, DOB: 6/12/82

Former Customer

@FangWong2.5M Followers

FangWong@Email.comNewsletter Subscriber

Fang WongFangWong@Email.comMarketing Department’s

Prospect List

Resolving the Fang Wong

Fang WongTop 100 Customer

F A WongSeattle, DOB: 6/12/82

Former Customer

@FangWong2.5M Followers

FangWong@Email.comNewsletter Subscriber

Fang WongFangWong@Email.comMarketing Department’s

Prospect List

Resolving the Fang Wong

Fang WongTop 100 Customer2.5M Followers

Newsletter Subscriber

Graphing the (resolved) Fang Wong

Bill SmithMember of the Board

Employee

Customer

FraudsterFang Wong

Top 100 Customer2.5M Followers

Contextualizing Sandy Maden

Sandy MadenNew Account

Employee

Lives With

Co-signer

FormerCustomer

Customer Customer

Customer

FraudsterFang Wong

“Entities”

Lives With

Co-signer

Sandy MadenNew AccountFormer

Customer

Employee

Customer Customer

Customer

FraudsterFang Wong

Company

Asteroid

Asteroid Hunting

Single Detection

Image courtesy of: Eva Lilly, Institute of Astronomy, University of Hawaii

From Orphans to Orbits

Single Detections(trash)

TrackletteTrackOrbitForecasting

Named entity: S100ZUtza

Single Detection (orphan)

Anticipation

"We have directly observed a collision between asteroids for the first time, instead of having to infer that they happened from million-year-old remains."

Colin SnodgrassPlanetary Scientist

Max Planck Institute for Solar System Research

Geospatial Context via “Space Time Boxes”

Detecting Colocation

TIME1 day

CE 1 hour

Determine encounter distance and time

0.05 A

Space Time Boxes

Computing 600k Asteroid Interactions over 25 Years

4-5 orders of magnitude improvement

Initial Analysis

Adding 1 New Trajectory

Space-Time Box Method

2,880 CPU hours

15 CPU minutes

N-body Simulation Method

10,000,000 CPU hours

4,000 CPU hours

Asteroid vs. Asteroid Encounters

Encounter Distance

Asteroid 1

Size Asteroid 2

May 1, 2032 299km 00A9170 2-4km 0008758 4-9km

Nov 24, 2016 449km 00P5634 1-2km 0055711 2-5km

Jan 11, 2018 449km K08E88J 530-1200m

00N0062

June 12th, 2015

Hi Jeff & the gang,

I have great news! On Tuesday I happened to observe a close encounter you guys predicted - one 1 km and the other one 2 km in diameter!

To my knowledge this is the first case ever of direct observation of a close encounter in the small main belt asteroids. The closest point of encounter unfortunately happened during bright daylight in Hawaii, so I missed it …

Cheers!Eva -

[Theatrical Pause]

Action

Red Analytics

Green Analytics

Blue Analytics

ObservationSpace

Old School: Isolated Analytics

ObservationSpace

ActionInformationIn Context

Next: General Purpose Context Computing

Context Computing

ObservationSpace

ActionInformationIn Context

Context Computing

Helping Focusing Human Attention

General Purpose • Marketing• Customer service• Fraud detection• Asteroid hunting

Simultaneously!

Making Data Work: Recommendations

Widen the observation space

Accumulate context to improve understanding

Deliver significantly higher quality outcomes … everywhere– Life sciences– Financial services– Public safety

Leverage Hadoop/Spark to accelerate innovation

Blog: www.jeffjonas.typepad.com

Email: JeffJonas@us.ibm.com

Next: San Francisco, Nov 10-12, @Datapalooza

Context ComputingStrata + Hadoop World 2015

Jeff Jonas, IBM FellowChief Scientist, Context Computinghttp://www.twitter.com/jeffjonaswww.jeffjonas.typepad.com

strata hadoop world 2015 context computing - jonas keynote - final

Data & Analytics

strata+hadoop 2015 keynote: impacting business as it happens

presto: distributed sql on anything - strata hadoop 2017...

heatmaps best practices strata hadoop

strata + hadoop world 2012: taming the elephant - learn how...

apache atlas. data governance for hadoop. strata london 2015

securing the hadoop ecosystem atm (cloudera) & shreepadma...

adversarial analytics - 2013 strata & hadoop world talk

sqoop2 refactoring for generic data transfer - hadoop strata...

availability and integrity in hadoop (strata eu edition)

hadoop strata talk - uber, your hadoop has arrived

strata + hadoop world: jump into the data lake with...

linkedin's logical data access layer for hadoop -- strata...

presto strata hadoop sj 2016 short talk

behavioral analytics with smartphone data. talk at strata +...

cloudy with a chance of on-prem - jim scott, strata hadoop...

model workers juan mateos-garcia, nesta strata hadoop 2014,...

apache flink@ strata & hadoop world london

alluxio keynote at strata+hadoop world beijing 2016

tamr | strata hadoop 2014 michael stonebraker

unlocking the data in paper - strata+hadoop world 2015