Download - IBM IOD Conference 2011 Opening Keynote Deck
© 2011 IBM Corporation
Big Data. Deep Analytics. New Physics.The Journey from Enterprise Amnesia
to Enterprise Intelligence
Jeff Jonas, IBM Distinguished EngineerChief Scientist, IBM Entity Analytics
Email: [email protected]: www.jeffjonas.typepad.com
Twitter: http://www.twitter.com/jeffjonas
© 2011 IBM Corporation
State of the Union:Enterprise Amnesia
© 2011 IBM Corporation
Amnesia, definition
A defect in memory, especially resulting from brain damage.
© 2011 IBM Corporation
Enterprise Amnesia, definition
A defect in memory, resulting in missed opportunity, wasted resources, lower revenues, unnecessary fraud losses, and other bad news.
© 2011 IBM Corporation
Time
Trend: Organizations Are Getting Dumber
Sensemaking Algorithms
Available Observation
Space
Structured data Unstructured data Social media Cyber audit logs Geospatial data
Com
pu
tin
g P
ow
er
Gro
wth
EnterpriseAmnesia
© 2011 IBM Corporation
Data Volumes Exploding
“Every two days now we create as much information as we did from the dawn of civilization up until 2003.”
~ Eric Schmidt, CEO Google
© 2011 IBM Corporation
WHY?
Trend: Organizations Are Getting Dumber
Time
Sensemaking Algorithms
Available Observation
Space
Com
pu
tin
g P
ow
er
Gro
wth
© 2011 IBM Corporation
Algorithms at Dead End.
You Can’t Squeeze Knowledge
Out of a Pixel.
© 2011 IBM Corporation
Context, definition
Better understanding something by taking into account the things around it.
© 2011 IBM Corporation
Information in Context … and Accumulating
Top 200Customer
Job Applicant
IdentityThief
CriminalInvestigation
© 2011 IBM Corporation
The Puzzle Metaphor
Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes and colors
What it represents is unknown – there is no picture on hand
Is it one puzzle, 15 puzzles, or 1,500 different puzzles?
Some pieces are duplicates, missing, incomplete, low quality, or have been misinterpreted
Some pieces may even be professionally fabricated lies
Until you take the pieces to the table and attempt assembly, you don’t know what you are dealing with
© 2011 IBM Corporation
Puzzling
270 pieces90%
200 pieces66%
150 pieces50%
6 pieces2%(pure noise)
30 pieces10% (duplicates)
© 2011 IBM Corporation
© 2011 IBM Corporation
© 2011 IBM Corporation
First Discovery
© 2011 IBM Corporation
More Data Finds Data
© 2011 IBM Corporation
Duplicates in Front Of Your Eyes
© 2011 IBM Corporation
First Duplicate Found Here
© 2011 IBM Corporation
© 2011 IBM Corporation
© 2011 IBM Corporation
Incremental Context – Incremental Discovery
6:40pm START
22min “Hey, this one is a duplicate!”
35min “I think some pieces are missing.”
37min “Looks like a bunch of hillbillies on a porch.”
44min “Hillbillies, playing guitars, sitting on a porch, near a barber sign … and a banjo!”
© 2011 IBM Corporation
150 pieces50%
© 2011 IBM Corporation
Incremental Context – Incremental Discovery
47min “We should take the sky and grass off the table.”
2hr “Let’s switch sides, and see if we can make sense
of this from different perspectives.”
2hr10m “Wait, there are three … no, four puzzles.”
2hr17m “We need a bigger table.”
2hr18m “I think you threw in a few random pieces.”
© 2011 IBM Corporation
© 2011 IBM Corporation
© 2011 IBM Corporation
© 2011 IBM Corporation
How Context Accumulates
With each new observation … one of three assertions are made: 1) Un-associated; 2) placed near like neighbors; or 3) connected
Must favor the false negative
New observations sometimes reverse earlier assertions
Some observations produce new discovery
As the working space expands, computational effort increases
Given sufficient observations, there can come a tipping point. Thereafter, confidence improves while computational effort decreases!
© 2011 IBM Corporation
Big Data [in context]. New Physics.
More data: better the predictions– Lower false positives
– Lower false negatives
More data: bad data … good– Suddenly glad your data was not perfect
More data: less compute
© 2011 IBM Corporation
Enterprise IntelligenceOne Plausible Journey
Enterprise IntelligenceHow to Get From Here to There
© 2011 IBM Corporation
ObservationSpace
Sense and Respond
What you know
New Observations
© 2011 IBM Corporation
ObservationSpace
Decide
?Relevance
Finds the Sensor(<200ms)
Data Finds Data
Sense and Respond
© 2011 IBM Corporation
Explore and Reflect
ObservationSpace
Decide
?DirectedAttention
Relevance Find You
DeepReflection
CuratedData
PatternDiscovery
RelevanceFinds the Sensor
(<200ms)
Data Finds Data
Sense and Respond
© 2011 IBM Corporation
ObservationSpace
Decide
?DirectedAttentionNEW
INTERESTS
DeepReflection
CuratedData
PatternDiscovery
RelevanceFinds the Sensor
(<200ms)
Data Finds Data
Explore and ReflectSense and Respond
© 2011 IBM Corporation
ObservationSpace
Decide
?
DeepReflection
CuratedData
PatternDiscovery
RelevanceFinds the Sensor
(<200ms)
Data Finds Data
ILog
NetezzaBigInsights
DirectedAttention
Cognos
Explore and ReflectSense and Respond
InfoSphere Streams
NEWINTERESTS
© 2011 IBM Corporation
ObservationSpace
Decide
?DirectedAttentionNEW
INTERESTS
DeepReflection
CuratedData
PatternDiscovery
RelevanceFinds the Sensor
(<200ms)
Data Finds Data
Report and Manage
Explore and ReflectSense and Respond
© 2011 IBM Corporation
Decide
?DirectedAttentionNEW
INTERESTS
CuratedData
PatternDiscovery
RelevanceFinds the Sensor
(<200ms)
Data Finds Data
Info Management Systems
Content ManagementCase Management Data Warehousing
Report and Manage
© 2011 IBM Corporation
ObservationSpace
Decide
?
DeepReflection
CuratedData
PatternDiscovery
RelevanceFinds the Sensor
(<200ms)
Data Finds DataIdentity Insight/Sensemaking
SPSS
ILog
NetezzaBigInsights
DirectedAttention
Cognos
Explore and ReflectSense and Respond
InfoSphere Streams
NEWINTERESTS
© 2011 IBM Corporation
Closing Thoughts
© 2011 IBM Corporation
The most competitive organizations
are going to make sense of what they are observing
fast enough to do something about it
while they are observing it.
© 2011 IBM Corporation
Wish This On The Competition
Time
Sensemaking Algorithms
Available Observation
Space
Com
pu
tin
g P
ow
er
Gro
wth
EnterpriseAmnesia
© 2011 IBM Corporation
The Way Forward: Enterprise Intelligence
Time
Sensemaking Algorithms
Available Observation
Space
Com
pu
tin
g P
ow
er
Gro
wth
Context Accumulation
© 2011 IBM Corporation
Big Data. Deep Analytics. New Physics.The Journey from Enterprise Amnesia
to Enterprise Intelligence
Jeff Jonas, IBM Distinguished EngineerChief Scientist, IBM Entity Analytics
Email: [email protected]: www.jeffjonas.typepad.com
Twitter: http://www.twitter.com/jeffjonas