Center for Language and Speech ProcessingThe Johns Hopkins University

Language and Speech Processing at Johns Hopkins University

March 5, 2010

The JHU Center for Language and Speech Processing

CLSP was established in 1992 with outside support to promote research and education in the science and technology of speech and language.

CLSP

Applied Math & StatisticsBiomedical Engineering

Cognitive Science

Electrical and ComputerEngineering

Computer Science

Human Language Tech.Center of Excellence

Speech and Language Faculty at JHU(does not list senior research staff, postdocs, students, …)

Electrical & Computer Eng Andreas Andreou Mounya Elhilali Hynek Hermansky Frederick Jelinek (Director) Damianos Karakos Sanjeev Khudanpur

Computer Science Chris Callison-Burch Jason Eisner David Yarowsky

Applied Math & Statistics Carey Priebe

Cognitive Science / Psychology Justin Halberda Geraldine Legendre Kyle Rawlins Paul Smolensky (Asst Dir) Colin Wilson

Biomedical Engineering Eric Young

Applied Physics Laboratory James Mayfield Christine Piatko

HLT Center of Excellence Kenneth Church Mark Dredze`

Aren Jansen Ben Van Durmé

CLSP Vision Statement

Understand how human language is used to communicate ideas/thoughts/information.

Develop technology for machine analysis, translation, and transformation of multilingual speech and text.

CLSP Mission Statement

1. Research Advance state of the art in our interdisciplinary field Focus on developing key algorithms and statistical models Focus on strategic languages, including low-resource languages

2. Education Attract the best students and train them to be leaders Offer full spectrum of courses Conduct annual international summer school at JHU

3. Outreach Be responsive to government and industry problems Serve as a “hub” for the HLT community Organize international summer research workshop at JHU Welcome short- and long-term visitors

Research: Primary Areas

Speech Recognition Acoustic processing Acoustic-phonetic modeling Pronunciation modeling Language modeling

Speech Applications Keyword spotting Spoken term detection Speaker verification Language identification

Speech Science Auditory physiology Neuromorphic signal processing

Natural Language Processing Morphological analysis Syntactic analysis (parsing) Information extraction Co-reference resolution

Machine Translation Low-resource languages Arabic and Chinese

Knowledge-Base Population Automatic content extraction Inference and learning

Machine Learning Small-sample learning Structured prediction Minimally supervised learning

Sponsored Research in Speech & Language in WSE is ≈ $2.5M/year

P Investigator Project Title (Granting Agency) Period Amount

Jelinek Investigation of Meaning Representation in Language Understanding (NSF) 10/05-01/10 $2.5 M

Jelinek Cross Cutting Research Workshops in Intelligent Information Systems (NSF) 09/07-08/11 $830 K

Eisner Finite-State Machine Learning on Strings and Sequences (NSF) 02/04-01/10 $500 K

Khudanpur Rosetta: An Analyst Co-pilot (DARPA/IBM) 10/05-04/11 $3.4 M

Eisner Learned Dynamic Prioritization (NSF) 09/10-08/14 $1.2 M

Hager Gesture Induction for Manipulative and Interactive Tasks (NSF) 02/06-01/10 $490 K

Smolensky* Unifying the Science of Language (NSF: IGERT) 05/06-04/11 $3.0* M

Vice Provost* Human Language Technology Center of Excellence (MPO) 01/07-01/17 $50* M

Andreou Energy Efficient Organic Semiconductor Circuits (DOE) 05/07-04/10 $660 K

Yarowsky Multi-Level Modeling of Language and Translation (NSF) 06/07-06/10 $400 K

Karakos Novel Approaches to Unsupervised Classification via ISPDTs (NSF) 09/07-08/10 $300 K

Callison-Burch DARPA Computer Science Study Group (DARPA) 02/08-02/09 $93 K

Jelinek Research Workshops in Intelligent Information Systems (Google) 06/08-05/10 $270 K

Khudanpur Self-Supervised Discriminative Training of Statistical Language Models (NSF)

09/08-08/09 $137 K

Khudanpur Self-Training for ASR in Low Resource Languages (BBN) 09/09-08/10 $101 K

CLSP Mission Statement

1. Research Advance state of the art in our interdisciplinary field Focus on developing key algorithms and statistical models Focus on strategic languages, including low-resource languages

2. Education Attract the best students and train them to be leaders Offer full spectrum of courses Conduct annual international summer school at JHU

3. Outreach Be responsive to government and industry problems Serve as a “hub” for the HLT community Organize international summer research workshop at JHU Welcome short- and long-term visitors

Education: Interdisciplinary Environment

Who and where PhD, MSE, and BS students from multiple depts. Shared interdisciplinary offices in CSEB Shared technical perspective and computing infrastructure

Coursework Interdisciplinary core curriculum (extends dept. requirements) Variety of other relevant courses (growing list, new plans) International 2-week summer school

Research Students do research from the start Students work with faculty from multiple departments and HLTCOE

Other learning Distinguished outside speaker every week Student speaker and town meeting every week Reading groups and conference travel

Sample Courses for an MSE in Human Language Technology

Course Number Course Title Instructor

CS 600.465 Natural Language Processing Eisner

CS 600.466 Information Retrieval and Web Agents Yarowsky

CS 600.425 Declarative Methods Eisner

AMS 550.732 Pattern Recognition Priebe

COG 050.320 Introduction to the Syntax of Natural Language Legendre

COG 050.325 Sound Structure in Natural Language Burzio

COG 050.825 Optimality Theory Smolensky

ECE 520.445 Introduction to Speech and Audio Processing Elhilali

ECE 520.447 Introduction to Information Theory and Coding Jelinek

ECE 520.651 Random Signal Analysis Khudanpur

ECE 520.666 Information Extraction from Speech and Text Jelinek

ECE 520.674 Information Theoretic Methods in Statistics Khudanpur

ECE 520.682 Computational Systems Neuroscience Elhilali

ECE 520.735 Sensory Information Processing Andreou

Education: Track Record

WSE has the 2nd largest university group in the U.S. working on Human Language Technology

38 PhDs awarded, many more MSEs

CLSP PhDs presently hold research/faculty positions at

Carnegie Mellon University U. of Massachusetts, Amherst Swarthmore College Michigan State University Hong Kong Polytechnic Univ. Bogazici University (Turkey) U. of Karlsruhe (Germany) Saarland University (Germany)

CLSP PhDs presently hold senior technical/research positions at

Apptek BBN Convergys e-Scription Fair Isaac Google (several) Microsoft (several) MITRE IBM (several) NSA (several) Nuance (several) SRI International

CLSP Mission Statement

1. Research Advance state of the art in our interdisciplinary field Focus on developing key algorithms and statistical models Focus on strategic languages, including low-resource languages

2. Education Attract the best students and train them to be leaders Offer full spectrum of courses Conduct annual international summer school at JHU

3. Outreach Serve as a “hub” for the HLT community Be responsive to government and industry problems Organize international summer research workshop at JHU Welcome short- and long-term visitors

JHU Summer Workshops in HLT:Integrating Research and Education

Organized by JHU on behalf of the Human Language Technology field 3 teams per summer (since 1995)

selected & refined from 25 proposals by “interactive peer review” each team comes to JHU for 8 weeks of intense collaborative research

Mixed teams of senior and student researchers Team ≈ 3 academics, 1 industry, 1 govt, 2-3 grad students, 2 undergrads 30+ participants 8 weeks 15 years More than 160 star students trained in HLT research (1998—2007)

Outcomes Numerous research breakthroughs New, long-term collaborations, tangible knowledge transfer Diverse expertise, research infrastructure, data resources

15

A Few of ManyWorkshop Accomplishments

A small sample of research results and their wider impact Statistical Machine Translation (1999)

GIZA++ is extensively used to build SMT systems even today MEAD Multilingual Multi-document Summarization (2001)

100s of worldwide users, active developers in the community SuperSID: High-level information for Speaker-ID (2002)

Major breakthrough in speaker recognition technology Factored Language Models (2002)

Improved ASR technology for conversational Arabic Moses Machine Translation Repository (2006)

The de facto standard in statistical machine translation More than 100 refereed publications

Detailed technical reports also available on CLSP web-site

Human Language TechnologyCenter of Excellence at JHU

• Long-term research mission: Automatically analyze a wide range of speech, text, and document images in multiple languages.

• Founded with government support in 2007• Has brought many new researchers and

research challenges into the CLSP community• Aggressively hiring the top new Ph.D.s nationally

Human Language TechnologyCenter of Excellence at JHU

Sponsor RDLeadership

ExecutiveDirector

AdministrativeStaff

Director ofResearch

Researchers

JHU Provost

SponsorResearchers

Whiting School of Engineering

Sponsor Technical Board

Center forLanguage and

Speech Processing

SecurityStaff

Prof. Andreas G. Andreou: Sensory Information Processing in Natural and Synthetic Systems

Research• Principles of sensory information

processing in biology.• Sensory communication.• Algorithms and processor architecture

design for energy efficient acoustic, speech and vision processing.

• Physics of sensing and computation.

ApplicationsAlgorithms for robust ASR• Robust acoustic feature

representation and dimensionality reduction

• Algorithms and architecture optimization for Chip Multi Processors (CMP) in Exascale systems

Multimodal scene analysis• Active and passive processing for

scene analysis (visual & auditory)• Acoustic and EM micro-Doppler

imagingBio-inspired systems• Energy efficient microsystems for

processing what and where in natural environments.

Prof. Chris Callison-Burch: Statistical Machine Translation

Research• Statistical machine translation• Syntactic translation models• Low resource languages• Data-driven paraphrasing• Evaluation measures, creation

of shared data resources

Prof. Kenneth Church:Human Language Technology (HLT) at Scale

Applications• Web search• Cloud computing• Language modeling• Text analysis• Spelling correction• Word-sense disambiguation• Terminology• Translation• Lexicography• Compression• Speech recognition and synthesis• OCR

Research• Speech Processing at Scale• Language Processing at Scale• Web Search at Scale• Mining Speech/Language with

Zero Linguistic Resources

Prof. Mark Dredze: Applications of Machine Learning to Real-World Text Processing

ApplicationsDomain adaptation• Extending NLP models to new

datasetsCross-domain learning• Applying NLP techniques to

languages with few resourcesKnowledge base population• Building large high precision

knowledge bases from textIntelligent email• Improved email clients by aiding

the user with artificial intelligence

Research• Adaptation of machine

learning algorithms between text domains

• Large scale information processing and learning

• Intelligent user interfaces for information management

Prof. Jason Eisner: Algorithms and Models for Language Processing

ApplicationsParsing sentence structure• Faster and more accurate algorithms• Unsupervised or cross-lingual

learningMachine translation• Model syntax, structure, word order• Combinatorial methods for

translation and for training modelsMorphology / phonology• Word spelling and pronunciation• Variant word forms (conjugation,

transliteration, misspelling, …)Information integration• Truth maintenance • Deductive databases• Reasoning from facts in text

Research• Novel algorithms for NLP • Bayesian statistical models

of linguistic structure• Machine learning

(structured prediction, novel training objectives)

• Declarative formalisms for grammars and algorithms

Prof. Mounya Elhilali: Reverse Engineering the Neurobiology of Speech and Audio Processing

Applications• Speech intelligibility in noise and

distortions• Auditory scene analysis and

speaker segregation• Speech enhancement• Hearing prostheses• Adaptive audio systems • Robotics and autonomous

systems• Object tracking in sensor

networks• Communication channels • Microphone Design

Research Goals• Information representation

and computational strategies employed by the brain

• Sound perception in distorted or complex acoustic environments

Prof. Hynek Hermansky: Robust Acoustic Speech Processing

ApplicationsSpeech recognition • what has been said?Speaker identification• who is speaking?Speaker verification• is the talker the one claimed to be?Language identification• which language is being used?Speech and audio coding• how to store/transmit the signal

efficiently?Enhancement of degraded speech• how to make noise or reverberated

speech easier listening to?

Technology• Proprietary techniques

based on temporal cues in the signal and on artificial neural net post-processing

• Emulations of auditory processing in biology

Prof. Aren Jansen:Knowledge-based Approaches to Speech Processing

ApplicationsNoise-Robust Speech Recognition• Invariance and efficiency through sparsityLow-Resource Speech Recognition• What can be done with little or no

transcribed training data?Spoken Term Detection and Discovery• “Google” for speech documents• Query-by-example vs. text queriesLarge-Scale Speech Processing • Scaling speech technology to massive

problem sizes

Research• Pursuit of more invariant representations of

speech• Unsupervised/semi-supervised learning of

speech units• Sparse representations and models• Computational models of human speech

perception

Prof. Frederick Jelinek:Statistical Speech Recognition and Machine Translation

ResearchStatistical aspects of Automatic Speech

Recognition (ASR)Language Modeling• Predicting next word given the pastReconstruction of ASR output• Create a grammatical sentence

preserving the speaker’s intended meaning

Rescoring of ASR output alternativesSearch algorithms for ASR and

Machine Translation

Interests• Statistical grammar and

parsing• Signals and systems• ASR treatment of out-of-

vocabulary words and phrases

• Machine translation

Prof. Damianos Karakos: Statistical Aspects of Speech and Language

ApplicationsSpeech recognition • Adaptation to the speech topic• Error corrective techniquesMachine translation• System combination• Language modelingDocument categorization• Automatic clustering into

meaningful categories• Detection of topics of interest

Technology• Data fusion and

dimensionality reduction for improved inference in text classification.

• Novel language modeling techniques for speech recognition.

Prof. Sanjeev Khudanpur:Statistical Modeling for Information Processing

ApplicationsAutomatic speech recognition• Domain and genre adaptation• Pronunciation variability modelingMachine translation (text & speech)• Output language word ordering• Context dependent translationMultimedia search and retrieval• Searching large speech archives• Content-based image/video searchRobotic minimally invasive surgery• Automated skill assessment• Automated surgical training

Basic Research• Stochastic Modeling of Signals

and Systems• Parameter Estimation• Model Structure Estimation• Information Theory and

Statistics

Prof. Benjamin Van Durme: Computational Semantics and Large-Scale Text Processing

ApplicationsKnowledge Acquisition• Enable “everyday” reasoning• Formal interpretation of generic

sentences (e.g., dictionary definitions)

“Deep” Information Extraction• Infer implicit relations• Semantic language modeling• Recognize higher order

modification of factoidsOrganizing Social Media• Dynamic clustering of authors,

documents, feeds

Research• Application of theoretical

semantics to problems in language technology

• Streaming algorithms for efficient processing of large text collections

Prof. David Yarowsky:Minimally Supervised Learning for Low-Resource Languages

ApplicationsMachine Translation• Translation discovery without

aligned bilingual text• Exploiting language universals and

language family relationshipsNatural Language Processing• Word sense disambiguation• Inflectional and derivational

morphologyInformation Extraction• Biographic fact extraction• Characterizing communicants• Informal genres

Basic Research• Cross-language information

projection• Cross-domain knowledge transfer• Co-training• Active learning and human

computation• Creative bootstrapping from multiple

knowledge sources

Prof. Kyle Rawlins: Formal & Computational Semantics

Prof. Paul Smolensky: Architecture of Universal Grammar

Prof. Colin Wilson: Theoretical, Experimental, & Computational Phonology

Prof. Geraldine Legendre: Syntax, Morphology, Acquisition

Linguistics and Human Language Processing

Prof. Justin Halberda: Word Learning in Children + a new professor …

Human Sentence Processing

Lots of Great Ph.D. Students:The Next Big Things!