![Page 1: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/1.jpg)
David Talby
@davidtalby
CTO, Atigeo
S EM A N T IC N AT U R A L L A NGUAGE U N DE RSTA ND IN GW ITH S PA R K , U IM A & M AC H IN E-L EA R NE D O N TO LO GIES
Claudiu Branzan
@melcutz
Principal Lead, Atigeo
![Page 2: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/2.jpg)
2 2
THE PROBLEM
Who needs to be vaccinated?
Who fits this clinical trial?
Who is at risk for sepsis?
Who is getting meds they’re allergic to?
Who on this protocol did not have this
side effect?
![Page 3: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/3.jpg)
3
AT THE BEGINNING, THERE WAS SEARCH
Scalable & robust Indexing pipelineTokenizers & analyzersSynonyms, spellers & Auto-suggestFile formats & header boostingRankers, link & reputation boosting
![Page 4: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/4.jpg)
4
THEN THERE WAS SEMANTIC SEARCH“cheap red prom dresses”“laptops under $500”“italian restaurants near me that deliver”“captain america civil war tonight”“nba scores”
Dictionary Based Attribute Extraction
Dell - XPS 15.6 4K Ultra HD Touch-Screen Laptop - Intel Core i5 - 8GB Memory - 256GB Solid State Drive - Silver
Machine Learned Attribute Extraction
If you go for the ambience, you'll be disappointed. If you go for good, inexpensive and authentic Mexican food, then you're in the right place.
![Page 5: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/5.jpg)
5
AND THEN, YOU NEED TO UNDERSTAND LANGUAGEPrescribing sick days due to diagnosis of influenza. Positive
Jane complains about flu-like symptoms. Speculative
Jane may be experiencing some sort of flu episode. Possible
Jane’s RIDT came back negative for influenza. Negative
Jane is at high risk for flu if she’s not vaccinated.
Conditional
Jane’s older brother had the flu last month. Family history
Jane had a severe case of flu last year. Patient history
![Page 6: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/6.jpg)
6
LANGUAGE GETS COMPLEX & DOMAIN SPECIFICJoe expressed concerns about the risks of bird flu. Nothing
Joe shows no signs of stroke, except for numbness. Double Negative
Nausea, vomiting and ankle swelling negative. Compound
(it gets worse – in reality a lot of text isn’t valid English)
Patient denies alcohol abuse. Speculative
Allergies: Penicillin, Dust, Sneezing. Compound
![Page 7: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/7.jpg)
7 7
ASSERTIONS
ASSERTIONS
LET’S BUILD THIS!
The input(patient records)
The processingframework
The output The query engines
![Page 8: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/8.jpg)
8 8
SENTENCE DETECTION
SECTION DETECTION
TOKENIZER LEMMATIZER
STOPWORD REMOVAL
NEGATION DETECTION
CONDITIONAL SCOPE
SPECULATIVE SCOPE
DATE NUMBER UNIT QUANITITY
CONCEPT EXTRACTION
![Page 9: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/9.jpg)
9 9
First Demo: Annotators & Assertions
![Page 10: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/10.jpg)
1 0 10
MACHINE LEARNED ANNOTATORS
Grammatical Patterns
If … then …
Direct Inferences
Age < 18 ==> Child
Lookups
RIDT (lab test)
Under-diagnosed conditions
Flu Depression
Implied by Context
relevant labs normal
Sometimes, it’s easier to just code an annotation’s business logic
But sometimes it’s easier to learn it from examples:
![Page 11: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/11.jpg)
1 1 11
Second Demo: Machine Learned Annotator
![Page 12: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/12.jpg)
1 2
![Page 13: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/13.jpg)
1 3 13
WHAT ABOUT EXPANDING & UPDATING ONTOLOGIES?
Word2Vec
![Page 14: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/14.jpg)
1 4 14
ONTOLOGY
LET’S BUILD THIS TOO!
![Page 15: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/15.jpg)
1 5 15
Third Demo: Ontology Enrichment
![Page 16: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/16.jpg)
1 6 16
SUMMARY & APPLICATIONS
Who needs to be vaccinated?
Who fits this clinical trial?
Who is at risk for sepsis?
Who is getting meds they’re allergic to?
Who on this protocol did not have this
side effect?
![Page 17: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/17.jpg)
1 7 17
@Atigeo@melcutz
@davidtalby
![Page 18: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/18.jpg)
© 2015 Atigeo, Corporation. All rights reserved. Atigeo and the xPatterns logo are trademarks of Atigeo. The information herein is for informational purposes only and represents the current view of Atigeo as of the date of this presentation. Because Atigeo must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Atigeo, and Atigeo cannot guarantee the accuracy of any information provided after the date of this presentation. ATIGEO MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
![Page 19: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/19.jpg)
APPENDIXIn case the live demo gets cold feet on stage
1 9
![Page 20: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/20.jpg)
2 0
![Page 21: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/21.jpg)
2 1
![Page 22: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/22.jpg)
2 2
![Page 23: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/23.jpg)
2 3
![Page 24: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/24.jpg)
2 4
![Page 25: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/25.jpg)
2 5
![Page 26: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/26.jpg)
2 6
![Page 27: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/27.jpg)
2 7
![Page 28: Semantic Natural Language Understanding with Spark, UIMA & Machine Learned Ontologies](https://reader030.vdocuments.us/reader030/viewer/2022020410/5885bbca1a28ab6f168b5bc9/html5/thumbnails/28.jpg)
2 8