development of an automated language interpreting tool for ... · language interpreting tool for...
TRANSCRIPT
![Page 1: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/1.jpg)
Development of an Automated Language Interpreting Tool
for Hispanic Mental Health Patients
The Way Forward:Advancing Mental Health Equity in California
March 30, 2017
![Page 2: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/2.jpg)
Behavioral Health Center for Excellence
Colaborators
• Steven Chan• Alberto Odor• Alvaro Gonzalez• Andrés Sciolla• Glen Xiong• Michelle Burke Parish• Shang Wei• Ana-Maria Iosif• Peter Yellowlees (PI)
Support / Grants
• Blue Shield of California• Virtual Collaborative
Health Network• Behavioral Health Center
for Excellence• AHRQ – 2 R01 Grants
![Page 3: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/3.jpg)
Behavioral Health Center for Excellence
Research• 2006 – 2020
• Transcultural Telepsychiatry Across Language Barriers (AHRQ – R01)
• 2013 – 2020• Automated Translation System for Transcultural Telepsychiatry (AHRQ - RO1)
• 2017 –• Automated Facial Expression Recognition in Telepsychiatry (Pilot)
• 2017 –• Augmented Reality to Show Visual and Auditory Hallucinations to Non-Psychotics
• Need to buy some FTE
![Page 4: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/4.jpg)
Behavioral Health Center for Excellence
The Problem
4.7%of the population(over 5 years of age)had limited English-speaking abilities.
This is nearly the size of Illinois.
![Page 5: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/5.jpg)
Behavioral Health Center for Excellence
The Problem
79%16%
3%2%
21%
English only English OK + Other
English "not well" + Other English "not at all" + Other
2010 U.S. CensusLanguage Use
![Page 6: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/6.jpg)
Behavioral Health Center for Excellence
The Problem
Sacramento County languages
90+non-English languages, like
SpanishMiaoChineseVietnameseTagalog
RussianUkrainianHindiGermanHmong
![Page 7: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/7.jpg)
Behavioral Health Center for Excellence
Old Practice
Patient
Provider
Ad hoc
![Page 8: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/8.jpg)
Behavioral Health Center for Excellence
Current PracticePatient
Provider
![Page 9: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/9.jpg)
Behavioral Health Center for Excellence
Current Practice
Patient
Provider
Live interpreter
![Page 10: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/10.jpg)
Behavioral Health Center for Excellence
Current Practice Problem:
•Unfunded mandate.
•Additional $30 - $50/hour
•Most private insurers do not reimburse interpretation. Except in
California. (Wilson 2013, Guntzel 2012)
•35% difference in interpreter salary
Money, Time Quality
![Page 11: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/11.jpg)
Behavioral Health Center for Excellence
Current Practice Problem
Money, Time, Quality
• Increased encounter time by 57% (live vs phone interpreters) (Locatis 2010).
•Travel time delays (Masland 2010)
•Perceived time constraints (Diamond 2009)
![Page 12: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/12.jpg)
Behavioral Health Center for Excellence
Current Practice Problem
Money, Time, Quality
• Health systems determine what’s “reasonable” for quality of language interpretation (The Joint Commission, 2010)
• Hospitals often don’t test staff competency, utilization, or effectiveness of interpreting services(Tschurtz et al., 2011)
• Only 100 of 3,500 medical interpreters in Oregon have proper qualifications (Foden-Vencil, 2014)
![Page 13: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/13.jpg)
Behavioral Health Center for Excellence
Current Solutions
Live interpretersPhone interpretersVideoconferencing interpreters
• Little information about accuracy of live interpretation
• Individual variabilities
• Hypothesis: Machine translation has similar levels of accuracy in a clinical setting.
![Page 14: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/14.jpg)
Behavioral Health Center for Excellence
Ultimate Goal
• Analyze language as data.
• Create new clinical processes for in-person
care and telemedicine.
• Create a better standard of care with
improved interpreting, more efficiently, and
less expensively.
![Page 15: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/15.jpg)
Behavioral Health Center for Excellence
Asynchronous S&F Telepsychiatry Workflow
![Page 16: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/16.jpg)
History (10 years ago)
Visalia
Tulare
![Page 17: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/17.jpg)
History (10 years ago)
![Page 18: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/18.jpg)
Behavioral Health Center for Excellence
Current State of Automated Translation
![Page 19: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/19.jpg)
Behavioral Health Center for Excellence
SDKs Assessment
Se
rvic
e
Ac
ce
ss
Wo
rks
wit
h a
ny
us
er
wit
ho
ut
pre
tra
inin
g
HIP
AA
Tra
ns
cri
pti
on
Tra
ns
lati
on
Ha
nd
les
str
ea
min
ga
ud
io
Un
de
rsta
nd
s
me
dic
al
term
ino
log
y
Co
ns
um
er
ap
p
ava
ila
ble
to
de
mo
?
Google Translate SaaS API ✓ – – ✓ ✓ ? ✓ ? Android, iOS
Nuance online SaaS API ✓
Nuance Dragon NaturallySpeaking SDK
Client DSClocal software –
✓—
local✓ – – ✓
Nuance Dragon NaturallySpeaking SDK
Server DSSWindows local ✓ ✓— ✓ – ✓ – ?
Nuance Dragon Dictate local software –✓—
local✓ ✓
Nuance 360 SaaS API ✓ ✓ – ✓ ✓ third-party apps
Nuance NDEV SaaS API ✓ – ✓ – ✓ Android, iOS
Mac OS X
Client & Apple-
operated
servers✓ – ✓ – – ✓
Microsoft Bing Voice Recognition &
Microsoft Translator on AzureSaaS API ✓ ✓ ✓ ✓ ✓
Windows Phone, web
translation
IBM BlueMix SaaS API –✓ English
only✓ ✓ – web translation
SRIlocal & client-
server✓ ✓ ✓ ✓ ✓ –
SysTran 7 Business Translatorlocal, server, &
SaaS✓ ✓ – ✓ N/A ✓ iPhone, web
![Page 20: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/20.jpg)
Behavioral Health Center for Excellence
Transcripts
Target
language (HUMAN)
Original
language
Target
language (COMPUTER)
![Page 21: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/21.jpg)
Behavioral Health Center for Excellence
Error Types
omission, in which the interpreter (human or machine) did not interpret a word/phrase uttered;
addition, in which the interpreter added a word/phrase not uttered by the patient;
substitution, in which the interpreter substituted a word/phrase for a different word/phrase; and
false fluency, in which the interpreter used an incorrect word/phrase that does not exist in that particular language
![Page 22: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/22.jpg)
Behavioral Health Center for Excellence
Errors in a PTSD Transcript
Inte
rpre
ter
1
Aggregate counts Transl #Data types J L K M NGrand Totaltotal_added_word_count 2 11 1 0 1 15total_drastic_word_count 214 225 277 189 265 1170total_unusual_word_count 25 12 21 33 31 122total_word_count 1202 1303 1181 1189 1253 6128Grand Total 1443 1551 1480 1411 1550 7435
Aggregate counts Transl #Data types J L K M NGrand Totaltotal_added_word_count 4 3 0 6 2 15total_drastic_word_count 244 343 398 278 477 1740total_unusual_word_count 19 68 74 96 71 328total_word_count 1214 1305 1186 1192 1261 6158Grand Total 1481 1719 1658 1572 1811 8241
Inte
rpre
ter
2
![Page 23: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/23.jpg)
Behavioral Health Center for Excellence
Automated Evaluation of Languages
BLEU (Bilingual Evaluation Understudy), an error-scoring algorithm that has found to have high correlation with human error quality with a score between 0 to 1.
Higher scores indicate a higher degree of similarity between human-translated and computer-translated texts.
The BLEU formula is the most widely used automated metric for testing MT apps.
![Page 24: Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show](https://reader035.vdocuments.us/reader035/viewer/2022081513/5f0ef0d87e708231d441b129/html5/thumbnails/24.jpg)
Behavioral Health Center for Excellence
Implications
• Language apps can be incorporated in both asynchronous and synchronous telemedicine video encounters and workflows.
• Not all translation apps are alike: significant differences in the quality between apps.
• These apps are already being used in commercial industries to replace or augment language interpreters. Expect increased reliability and accuracy in the future.