background, overview and what’s next - massachusetts...

Background, Overview and What’s Next

Christopher Codella, Ph.D.IBM Distinguished Engineer, CTO Public Sector, Watson Group

© 2014 IBM Corporation

IBM Distinguished Engineer, CTO Public Sector, Watson Group

… learn and interact naturally with people toextend what humans could do on their own.They help us solve problems by penetratingthe complexity of Big Data.

CognitiveSystems Era

ProgrammableSensors& Devices

Cognitive Systems…


ProgrammableSystems Era

TabulatingSystems Era We are

here

VoIP

EnterpriseData

SocialMedia

Informed Decision Making: Search vs. Expert Q&A

Decision Maker

Search Engine

Finds Documents containing Keywords

Delivers Documents based on Popularity

Has Question

Distills to 2-3 Keywords

Reads Documents, FindsAnswers

Finds & Analyzes Evidence


Expert

Understands Question

Produces Possible Answers & Evidence

Delivers Response, Evidence & Confidence

Analyzes Evidence, Computes Confidence

Asks NL Question

Considers Answer & Evidence

Decision Maker

Finds & Analyzes Evidence

Automatic Open-Domain Question AnsweringA Long-Standing Challenge in Artificial Intelligence to emulate human expertise

Given– Rich Natural Language Questions

– Over a Broad Domain of Knowledge

Deliver


Deliver– Precise Answers: Determine what is being asked & give precise response

– Accurate Confidences: Determine likelihood answer is correct

– Consumable Justifications: Explain why the answer is right

– Fast Response Time: Precision & Confidence in <3 seconds

4

Capture the imagination

– The Next Deep Blue

Engage the scientific community– Envision new ways for computers to impact society & science

– Drive important and measurable scientific advances

A Grand Challenge Opportunity


– Drive important and measurable scientific advances

Be Relevant to IBM Customers– Enable better, faster decision making over unstructured and structured content

– Business Intelligence, Knowledge Discovery andManagement, Government, Compliance, Publishing, Legal, Healthcare, Business Integrity, Customer Relationship Management, Web Self-Service, ProductSupport, etc.

5

Real Language is Real Hard

Chess–A finite, mathematically well-defined search space

–Limited number of moves and states

–Grounded in explicit, unambiguous mathematical rules


Human Language–Ambiguous, contextual and implicit

–Grounded only in human cognition

–Seemingly infinite number of ways to express the same meaning

What It Takes to compete against Top Human Jeopardy! PlayersOur Analysis Reveals the Winner’s Cloud

Winning HumanWinning Human

Top humanplayers areremarkably

good.

Top humanplayers areremarkably

good.

Each dot – actual historical human Jeopardy! games

© 2014 IBM Corporation7

Winning HumanPerformance


2007 QA Computer System2007 QA Computer System

Grand ChampionHuman Performance


What Computers Find Easy (and Hard)(ln(12,546,798 * π)) ^ 2 / 34,567.46 =

Owner Serial Number

David Jones 45322190-AKInvoice # Vendor Payment

INV10895 MyBuy $104.56

0.00885

Select Payment where Owner=“David Jones” and Type(Product)=“Laptop”,

© 2014 IBM Corporation8 IBM Confidential

Serial Number Type Invoice #

45322190-AK LapTop INV10895

INV10895 MyBuy $104.56

David Jones

David Jones =Dave Jones

David Jones≠

What Computers Find HardComputer programs are natively explicit, fast and exacting in theircalculation over numbers and symbols….But Natural Language is implicit,highly contextual, ambiguous and often imprecise.

Where was X born?

Person Birth Place

A. Einstein ULM

Structured

Unstructured


Where was X born?One day, from among his city views of Ulm, Otto chose a water color to

send to Albert Einstein as a remembrance of Einstein´s birthplace.

X ran this?If leadership is an art then surely Jack Welch has proved himself a

master painter during his tenure at GE.

Person Organization

J. Welch GE

Unstructured

Different Types Of Evidence: Keyword Evidence

celebrated

In May1898

400th

In May

celebratedKeyword MatchingKeyword Matching

Keyword MatchingKeyword Matching

arrived in

In May, Gary arrived inIndia after he celebrated hisanniversary in Portugal.

In May 1898 Portugal celebratedthe 400th anniversary of thisexplorer’s arrival in India.


India

400thanniversary

arrival in

Portugal

India

Garyexplorer

anniversary

in Portugal




10

Evidence suggests“Gary” is the answerBUT the system mustlearn that keywordmatching may beweak relative to other

types of evidence

On 27th May 1498, Vasco da Gamalanded in Kappad Beach


celebrated

May 1898 400th anniversary

In May 1898 Portugal celebratedthe 400th anniversary of thisexplorer’s arrival in India.

Portugal

landed in

27th May 1498Temporal

Reasoning


On the 27th of May 1498, Vasco daGama landed in Kappad Beach

Search Far and Wide

Explore many hypotheses

Find Judge Evidence

Many inference algorithms

Search Far and Wide

Explore many hypotheses

Find Judge Evidence

Many inference algorithms

Different Types Of Evidence: Deeper Evidence


arrivalin

Vasco da Gama

StatisticalParaphrasing

GeoSpatialReasoning

explorer

Kappad Beach

Para-phrases

Geo-KB

DateMath

11

India

Strongerevidence canbe muchharder to findand score.

The evidence is still not 100% certain.

This fish was thought to be extinct millions of years agountil one was found off South Africa in 1938

Category: ENDS IN "TH" Answer:

When hit by electrons, a phosphor gives off electromagnetic energy

The type of thingbeing asked for isoften indicated but

can go from specificto very vague

coelacanth

Some Basic Jeopardy! Clues


When hit by electrons, a phosphor gives off electromagnetic energyin this form

Category: General Science Answer:

Secy. Chase just submitted this to me for the third time--guesswhat, pal. This time I'm accepting it

Category: Lincoln Blogs Answer:

light (or photons)

his resignation

12

Broad Domain

1.50%

2.00%

2.50%

3.00%

We do NOT attempt to anticipate allquestions and build databases.

In a random sample of 20,000 questions we found2,500 distinct types*. The most frequent occurring <3% of the time.

The distribution has a very long tail.

And for each these types 1000’s of different things may be asked.

We do NOT try to build a formalmodel of the world


0.00%

0.50%

1.00%

1.50%

he

film

gro

up

capital

wom

an

song

sin

ger

show

com

poser

title

fruit

pla

net

there

pers

on

lang

uag

eholid

ay

colo

rpla

ce

son

tree

line

pro

duct

birds

anim

als

site

lady

pro

vince

dog

substa

nce

insect

way

founder

senato

rfo

rmdis

ease

som

eone

maker

fath

er

word

sobje

ct

write

rnove

list

hero

ine

dis

hpost

month

veg

eta

ble

sig

ncountr

ies

hat

bay

Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text.Structured sources (DBs and KBs) provide background knowledge for interpreting the text.

*13% are non-distinct (e.g, it, this, these or NA)

Even going for the head of the tail willbarely make a dent

Automatic Learning For “Reading”

Inventors patent inventions (.8)

Volumes of Text Syntactic Frames Semantic Frames


Officials Submit Resignations (.7)People earn degrees at schools (0.9)

Inventors patent inventions (.8)

Vessels Sink (0.7)People sink 8-balls (0.5) (in pool/0.8)

Fluid is a liquid (.6)Liquid is a fluid (.5)

Evaluating Possibilities And Their Evidence

In cell division, mitosis splits the nucleus & cytokinesissplits this liquid cushioning the nucleus.

Organelle Vacuole Cytoplasm Plasma

Many candidate answers (CAs) are generated from many different searches

Each possibility is evaluated according to different dimensions of evidence.

Just One piece of evidence is if the CA is of the right type. In this case a “liquid”.


Is(“Cytoplasm”, “liquid”) = 0.2

Is(“organelle”, “liquid”) = 0.1

Is(“vacuole”, “liquid”) = 0.2

Is(“plasma”, “liquid”) = 0.7

“Cytoplasm is a fluid surrounding the nucleus…”

Wordnet Is_a(Fluid, Liquid) ?

Learned Is_a(Fluid, Liquid) yes.

↑

Plasma Mitochondria Blood …

Just One piece of evidence is if the CA is of the right type. In this case a “liquid”.

What It Takes to compete against Top Human Jeopardy! PlayersOur Analysis Reveals the Winner’s Cloud

In 2007, we committed to

Each dot – actual historical human Jeopardy! games

Winning HumanWinning Human


2007 QA Computer System2007 QA Computer System

In 2007, we committed tomaking a Huge Leap!

Computers?Not So Good.





DeepQA: The Technology Behind WatsonMassively Parallel Probabilistic Evidence-Based Architecture


60%

70%

80%

90%

100%

5/2009

10/2009

11/2010

12/2008

Deep QA: Incremental Progress in Precision and Confidence6/2007-11/2010

Now Playing in theWinners Cloud



4/2010

Pre

cis

ion


0%

10%

20%

30%

40%

50%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

% Answered

Baseline

12/2007

8/2008

12/2008

5/2008

Pre

cis

ion

One Jeopardy! question can take 2 hours on a single 2.6Ghz CoreOptimized & Scaled out on 2880-Core IBM workload optimized

POWER7 HPC using UIMA-AS,Watson answers in 2-6 seconds.

Question100s Possible

Answers

1000’s ofPieces of Evidence

MultipleInterpretations

100,000’s scores from many simultaneousText Analysis Algorithms100s sources


Interpretations

. . .

HypothesisGeneration

Hypothesis andEvidence Scoring

Final ConfidenceMerging &Ranking

SynthesisQuestion &

TopicAnalysis

QuestionDecomposition

HypothesisGeneration

Hypothesis and EvidenceScoring

Answer &Confidence

90 x IBM Power 7501 servers

2880 POWER7 cores

POWER7 3.55 GHz chip

500 GB per sec on-chip bandwidth

10 Gb Ethernet network

15 Terabytes of memory

Watson – a Workload Optimized System


15 Terabytes of memory

20 Terabytes of disk, clustered

Can operate at 80 Teraflops

Runs IBM DeepQA software

Scales out with and searches vast amounts of unstructuredinformation with UIMA & Hadoop open source components

Linux provides a scalable, open platform, optimizedto exploit POWER7 performance

10 racks include servers, networking, shared disk system,cluster controllers

1 Note that the Power 750 featuring POWER7 is a commercially availableserver that runs AIX, IBM i and Linux and has been in market since Feb 2010

Watson’s real value proposition: Efficient decision support over unstructured(and structured) content

Jeopardy! Challenge

Deeper Understanding,Higher Precision and Broader,

Timely Coverage at lower costs


Unstructured Data Broad, rich in context Rapidly growing, current Invaluable yet under utilized

Structured Data Precise, explicit Narrow, expensive

Deeper Understanding but BrittleHigh Precision at High CostNarrow Limited Coverage

Shallow UnderstandingLow Precision

Broad Coverage

LearningUnderstanding Interacting Explaining

Specific Questions

The type of murmurassociated with this

condition isharsh, systolic, and

increases in intensitywith Valsalva

From specificquestions

Question-In/Answer-Out

Evidenceanalysis and

Batch Training Process

Scale domainlearning and

Precise Answers& Accurate Confidences

Move fromquality answers

Taking Watson beyond Jeopardy!


questionsto

rich, incompleteproblem

scenarios(e.g. EHR)

Rich ProblemScenarios

EntireMedicalRecord

analysis andlook-ahead,

drive interactivedialog to refine

answers andevidence

Interactive DialogTeach Watson

Refined Answers, Follow-upQuestions

Input, Responses

Dialog

learning andadaptation rateand efficiency

Continuous Training& Learning Process

Answers,Corrections, Judgements

Responses, LearningQuestions

quality answersto quality

answers andevidence

ComparativeEvidence Profiles

Teach Watson technology generates questions to enhance understandingand acquire new knowledge – used in dialog or crowd-sourcingopportunities

These disorders can stop theThese disorders can stop thenerves and muscles in youresophagus from workingright. This can cause food to

Patients with preexistingseizure disorder should notuse bupropion due to ahigher-than-proportional

Watson considers…What would have to be

true for seizuredisorder to be correct?

What neurological conditioncontraindicates the use of

bupropion?


Automatically generates LearningQuestions…

Does “contraindicates theuse of” mean “should notuse” in general?

right. This can cause food tomove slowly or even getstuck in the esophagus.

higher-than-proportionalincrease in the possibility ofseizure as the dose isincreased.

ADoes contraindicates theuse of bupropion meanshould not use bupropion?

Q

bupropion?

Q: What diagnosisexplains the patient’scondition?

Present FactorsRed, painful eyeBlurred visionFamily history of arthritis

Behcet’s DiseaseBehcet’s Disease 45%45%

SarcoidosisSarcoidosis 32%32%

Lyme DiseaseLyme Disease 1%1%

Behcet’s Disease 45%

Sarcoidosis 32%

Lyme Disease 1%

Dialoguing to an answer


MedicalRecord

The first symptom of Lyme disease (also calledLyme’s disease) for about 50% of people is asmall, red bull’s-eye rash, called erythemamigrans, at the site of an infected tick bite. …Other early, acute Lyme symptoms are flu-like –fatigue, achy muscles or joints, fever, chills, stiffneck, swollen glands, and a headache.

Absent Factors

Circular rash Fatigue Headache

muscle aches.

Lyme disease can affect different bodysystems, such as the nervoussystem, joints, skin, and heart. Symptoms areoften described as happening in three stages(although not everyone experiences all three):1. A circular rash, typically within 1-2 weeks of

infection, often is the first sign of infection. …2. Along with the rash, a person may have flu-

like symptoms such as swollen lymphnodes, fatigue, headache, and muscle aches.

and is transmitted to humans through theLyme disease is caused by the bacterium Borreliaburgdorferi and is transmitted to humans through thebite of infected blacklegged ticks. Typical symptomsinclude high temperature, headache, fatigue, and acharacteristic skin rash called erythema migrans.Behcet’s DiseaseBehcet’s Disease

SarcoidosisSarcoidosis

Lyme DiseaseLyme Disease

1%1%

63%63%

3%3%

Behcet’s Disease

Sarcoidosis

Lyme Disease

1%

63%

3%

Conclusion

Navigating/Exploring the Information Space (with Watson)

Q

HReasoning

Question

Hypothesis

e Evidence

H

H

Q

Q

Q

Q

H

H

H e

e

e

Q H

H

H e

e

e

Q


H

Q

H e

Q

Q

Q

Line ofInquiry

Q

H

Q H e

Q

H e

Q H e Report

Analytics: the use of data and computation to make smart decisions

Data

Historical Data instances

Decision point Possible outcomes


Historical

Simulated

Text Video, Images Audio

Reports and queries ondata aggregates

Predictive models

Answers and confidence

Feedback and learning

Option 2

Initial Watson Solutions

Decisions

WellPointInteractive Care

Guide & Reviewer

WellPointInteractive Care &

Insights forOncology

Engagement

IBM WatsonEngagement

Advisor

Discovery

IBM WatsonDiscovery Advisor

Cases

IBM WatsonCase Advisor

(Research)


Streamlineauthorization of

procedures

Health Plans &Providers

Improve Diagnosisand Treatment

Clinicians

TransformCustomer

Experiences

Customers

AccelerateResearch

and Insights

Scientists Clinicians

Summarize Casesand identify therelevant data

IBM Watson Is Ushering In A New Era Of Computing

System“Intelligence”

Cognitive


Punch cardsTime card readers

SearchDeterministicEnterprise dataMachine languageSimple outputs

DiscoveryProbabilisticBig DataNatural languageIntelligent options

1900 - ~1940 ~1940 - 2010 2011

Tabulation

Programmatic

THANK YOU


THANK YOU