![Page 1: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/1.jpg)
Building a Voice Assistant for Enterprise
@vmanjuQConSF, Nov 2018
Manju Vijayakumar Lead Software Engineer, Salesforce
![Page 2: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/2.jpg)
Agenda
● Why Voice?● Demo of Einstein Voice Assistant● Conversational AI
○ Ecosystem○ Natural Language Understanding (NLU)
● Challenges● Future
○ Considerations○ What’s next for NLP and AI
![Page 3: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/3.jpg)
Voice Recognition - A Story in 3 pictures
Source on Twitter
![Page 4: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/4.jpg)
![Page 5: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/5.jpg)
![Page 6: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/6.jpg)
From programmatic to natural interactionsComputing is Evolving
Point & Click
Command Line
Touch
Voice
![Page 7: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/7.jpg)
Deliver an intelligent assistant that leverages Voice and NLU capabilities to
understand, and support users in accomplishing their goals
![Page 8: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/8.jpg)
EINSTEIN VOICE DEMO
Pilot
![Page 9: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/9.jpg)
Meet Amy, a busy salesperson
Amy needs to update Salesforce
![Page 11: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/11.jpg)
How did Voice Assistant help Amy?
Accuracy & timeliness of data capture
Visible to the team
Unstructured data -> Structured data● Productive● No system expertise
![Page 12: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/12.jpg)
Building Blocks of Voice Assistant
![Page 13: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/13.jpg)
ASRAutomatic Speech
Recognition
![Page 14: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/14.jpg)
ASRAutomatic Speech
Recognition
NLUNatural Language
Understanding
![Page 15: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/15.jpg)
ASRAutomatic Speech
Recognition
NLUNatural Language
Understanding
CRMIntegration
![Page 16: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/16.jpg)
Conversational AI Ecosystem
![Page 17: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/17.jpg)
Conversational AI Ecosystem
Einstein PlatformAutomatic Speech
RecognitionModels
Natural Language Understanding
Models
SalesforceCRM Metadata
![Page 18: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/18.jpg)
Conversational AI Ecosystem
Einstein Platform
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
Context Management
Automatic Speech Recognition
Models
Natural Language Understanding
Models
SalesforceCRM Metadata
Slot Filling
![Page 19: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/19.jpg)
Conversational AI Ecosystem
Einstein Platform
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
Context Management
Automatic Speech Recognition
Models
Natural Language Understanding
Models
SalesforceCRM Metadata
Slot Filling
Einstein Voice Assistant Einstein Voice Bots Smart Speakers* Voice Navigation*
![Page 20: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/20.jpg)
Conversational AI Service
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
Context Management
Slot Filling
![Page 21: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/21.jpg)
Conversational AI Service
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
Context Management Slot Filling
![Page 22: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/22.jpg)
Named Entity Recognition (NER)
The committee of 30 government and university scientists and engineers, led by McCleese, was asked to recommend to the space agency by the end of this month a rationale and strategy for precursor flights and the sample-return missions.
The ‘O’
committee ‘O’
of ‘O’
... ‘O’
McCleese ‘PER’
the ‘DATE’
end ‘DATE’
of ‘DATE’
this ‘DATE’
month ‘DATE’
*CoNLL format
![Page 23: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/23.jpg)
Named Entity Recognition (NER)
The committee of 30 government and university scientists and engineers, led by McCleese, was asked to recommend to the space agency by the end of this month a rationale and strategy for precursor flights and the sample-return missions.
NER7 model recognizes 7 entities:Person, Organization, Location, Date, Time, Money, Percentage
The ‘O’
committee ‘O’
of ‘O’
... ‘O’
McCleese ‘PER’
the ‘DATE’
end ‘DATE’
of ‘DATE’
this ‘DATE’
month ‘DATE’
*CoNLL format
![Page 24: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/24.jpg)
What are the entities in the text ?
PERSON
ORGANIZATION
DATE
MONEY
Follow up call with Chris in two weeks DATE (two weeks is normalized to 2018/07/15)
![Page 25: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/25.jpg)
Conversational AI Service
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
Context Management Slot Filling
![Page 26: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/26.jpg)
Entity Resolution - Is this entity in my CRM ?
Salesforce CRM DB
Records matched for ‘Acme’
Send records to user to disambiguate
![Page 27: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/27.jpg)
Conversational AI Service
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
Context Management Slot Filling
![Page 28: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/28.jpg)
Context Management - What data do we have so far ?
{ "context": { "Organization": { "id": "001XXXX", "name": "Acme Corp" }, }, ... }
Do we have organization
in the context?
![Page 29: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/29.jpg)
Conversational AI Service
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
Context Management Slot Filling
![Page 30: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/30.jpg)
{
"probabilities": [
{
"label": "CREATE",
"probability": 0.9904295
},
{
"label": "UPDATE",
"probability": 0.009345241
},
...
]
}
Text Classification - What are the intents ?
Acme Corp’s timeline for purchasingMarketing software is set for July 1st and may purchase up to $250K of product
Follow up call with Chris in two weeks
Language API Intent model
Prediction request
JSON
![Page 31: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/31.jpg)
Conversational AI Service
Conversational API Named Entity Recognition
Entity Resolution
Text Classification
State Management Slot Filling
![Page 32: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/32.jpg)
Slot Filling - What are the slots for each action item ?
Fill in the date and money slots for Update action
Fill in the date slot and person slot for Create Task action.
Here, date is normalized:In 2 weeks => 10/7/18
![Page 33: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/33.jpg)
Challenges
![Page 34: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/34.jpg)
Data challenges
Heterogenous database
How do you make it work for every customer schema ?
AccountID Name Phone
AccountID Name Phone BankAccount
- Customers can define custom schemas- Schemas are not consistent
![Page 35: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/35.jpg)
Data challenges
Inconsistent data
Which Acme Corp. did you mean ?
- Lots of duplicates
- Identify the most relevant ‘Acme’
- Affects user experience
![Page 36: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/36.jpg)
Automatic Speech Recognition is not perfect
DOMAIN SPECIFICJARGON
AUDIO ENVIRONMENT
ACCENTS & LINGUISTIC PROFILES
![Page 37: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/37.jpg)
Named Entity Recognition is not perfect
Named Entity Recognition is easy for humans but hard for machines
![Page 38: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/38.jpg)
Named Entity Recognition is not perfect
Today, JP Morgan and I spoke about...
..the san juan center is led by a team of scientists..
..Man joy and I met today at Starbucks to discuss..
Cannot identify san juan as a location due to case sensitivity
“Manju” misspelled as “Man Joy”. Misspelled pronouns are hard to catch
Is JP Morgan a company or a person ?
![Page 39: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/39.jpg)
Future Considerations
![Page 40: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/40.jpg)
Optimized Models
- Configurable- Normalized
Feedback
- Capture Feedback- Retrain Models
Voice
- Guided user experience- Multi channel
![Page 41: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/41.jpg)
Deep learning
What’s next for NLP and AI?
?Architecture engineering
for single tasksMachine learning with feature engineering
![Page 42: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/42.jpg)
Deep learning
What’s next for NLP and AI?
Single multitask model
Architecture engineering
for single tasksMachine learning with feature engineering
To learn more: decaNLP.com
![Page 43: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/43.jpg)
Voice Recognition - A Story in 3 pictures
Source on Twitter
![Page 44: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/44.jpg)
Voice Recognition - The Complete story
Source on Twitter
![Page 45: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/45.jpg)
![Page 46: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/46.jpg)
Key Takeaways
Language understanding is AI Complete.
Focus on solving customer pain points
in your domain.
Voice will become the new User Interface.
![Page 47: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/47.jpg)
![Page 48: Building a Voice Assistant for Enterprise · Conversational AI Ecosystem Einstein Platform Conversational API Named Entity Recognition Entity Resolution Text Classification Context](https://reader034.vdocuments.us/reader034/viewer/2022052613/5f1b1c51d1873a303b60f457/html5/thumbnails/48.jpg)
Resources
Einstein.ai - published papers, research etc.
Einstein.ai/careers - We are hiring!
@vmanju