multilingual access to large spoken archives douglas w. oard university of maryland, college park,...
TRANSCRIPT
![Page 1: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/1.jpg)
Multilingual Access to Large Spoken Archives
Douglas W. OardUniversity of Maryland, College Park, MD, USA
![Page 2: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/2.jpg)
MALACH Project’s Goal
Dramatically improve access to large multilingual spoken word collections
… by capitalizing on the unique characteristics of the Survivors of the Shoah Visual History Foundation's collection of videotaped oral history
interviews.
![Page 3: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/3.jpg)
Spoken Word Collections
• Broadcast programming– News, interview, talk radio, sports, entertainment
• Scripted stories– Books on tape, poetry reading, theater
• Spontaneous storytelling– Oral history, folklore
• Incidental recording– Speeches, oral arguments, meetings, phone calls
![Page 4: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/4.jpg)
Some Statistics
• 2,000 U.S. radio stations webcasting
• 250,000 hours of oral history in British Library
• 35 million audio streams indexed by SingingFish– Over 1 million searches per day
• ~100 billion hours of phone calls each year
![Page 5: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/5.jpg)
Economics of the Web in 1995• Affordable storage
– 300,000 words/$
• Adequate backbone capacity– 25,000 simultaneous transfers
• Adequate “last mile” bandwidth– 1 second/screen
• Display capability– 10% of US population
• Effective search capabilities– Lycos, Yahoo
![Page 6: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/6.jpg)
Spoken Word Collections Today• Affordable storage
– 300,000 words/$
• Adequate backbone capacity– 25,000 simultaneous transfers
• Adequate “last mile” bandwidth– 1 second/screen
• Display capability– 10% of US population
• Effective search capabilities– Lycos, Yahoo
1.5 million words/$
30 million
20% of capacity
38% recent use
![Page 7: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/7.jpg)
Research Issues
• Acquisition
• Segmentation
• Description
• Synchronization
• Rights management
• Preservation
MALACH
![Page 8: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/8.jpg)
Description Strategies• Transcription
– Manual transcription (with optional post-editing)
• Annotation– Manually assign descriptors to points in a recording– Recommender systems (ratings, link analysis, …)
• Associated materials– Interviewer’s notes, speech scripts, producer’s logs
• Automatic– Create access points with automatic speech processing
![Page 9: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/9.jpg)
Key Results from TREC/TDT
• Recognition and retrieval can be decomposed– Word recognition/retrieval works well in English
• Retrieval is robust with recognition errors– Up to 40% word error rate is tolerable
• Retrieval is robust with segmentation errors– Vocabulary shift/pauses provide strong cues
![Page 10: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/10.jpg)
Supporting Information Access
SourceSelection
Search
Query
Selection
Ranked List
Examination
Recording
Delivery
Recording
QueryFormulation
Search System
Query Reformulation and
Relevance Feedback
SourceReselection
![Page 11: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/11.jpg)
![Page 12: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/12.jpg)
![Page 13: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/13.jpg)
Broadcast News Retrieval Study
• NPR OnlineManually prepared transcripts
Human cataloging
• SpeechBotAutomatic Speech Recognition
Automatic indexing
![Page 14: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/14.jpg)
NPR Online
![Page 15: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/15.jpg)
![Page 16: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/16.jpg)
SpeechBot
![Page 17: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/17.jpg)
![Page 18: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/18.jpg)
Study Design• Seminar on visual and sound materials
– Recruited 5 students
• After training, we provided 2 topics– 3 searched NPR Online, 2 searched SpeechBot
• All then tried both systems with a 3rd topic– Each choosing their own topic
• Rich data collection– Observation, think aloud, semi-structured interview
• Model-guided inductive analysis– Coded to the model with QSR NVivo
![Page 19: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/19.jpg)
Criterion-Attribute Framework
Relevance Criteria
Associated Attributes
NPR Online SpeechBot
Topicality
Story Type
Authority
Story title
Brief summary
Audio
Detailed summary
Speaker name
Audio
Detailed summary
Short summary
Story title
Program title
Speaker name
Speaker’s affiliation
Detailed summary
Brief summary
Audio
Highlighted terms
Audio
Program title
![Page 20: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/20.jpg)
Some Useful Insights
• Recognition errors may not bother the system, but they do bother the user!
• Segment-level indexing can be useful
![Page 21: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/21.jpg)
Shoah Foundation’s Collection• Enormous scale
– 116,000 hours; 52,000 interviews; 180 TB
• Grand challenges– 32 languages, accents, elderly, emotional, …
• Accessible– $100 million collection and digitization investment
• Annotated– 10,000 hours (~200,000 segments) fully described
• Users– A department working full time on dissemination
![Page 22: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/22.jpg)
Example Video
![Page 23: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/23.jpg)
Existing Annotations• 72 million untranscribed words
– From ~4,000 speakers
• Interview-level ground truth– Pre-interview questionnaire (names, locations, …)– Free-text summary
• Segment-level ground truth– Topic boundaries: average ~3 min/segment– Labels: Names, topic, locations, year(s)– Descriptions: summary + cataloguer’s scratchpad
![Page 24: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/24.jpg)
Annotated Data Example
Subject PersonLocation-Time
Berlin-1939 Employment Josef Stein
Berlin-1939 Family life Gretchen Stein Anna Stein
Dresden-1939 Schooling Gunter Wendt Maria
Dresden-1939 Relocation Transportation-rail inte
rvie
w ti
me
![Page 25: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/25.jpg)
MALACH Overview
AutomaticSearch
BoundaryDetection
InteractiveSelection
ContentTagging
SpeechRecognition
QueryFormulation
ASR SpontaneousAccentedLanguage switching
NLPComponents Multi-scale segmentation
Multilingual classificationEntity normalization Prototype
Evidence integrationTranslingual searchSpatial/temporal
UserNeeds
Observational studiesFormative evaluationSummative evaluation
![Page 26: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/26.jpg)
MALACH Overview
AutomaticSearch
BoundaryDetection
InteractiveSelection
ContentTagging
SpeechRecognition
QueryFormulation
ASR SpontaneousAccentedLanguage switching
![Page 27: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/27.jpg)
ASR Research Focus
• Accuracy– Spontaneous speech– Accented/multilingual/emotional/elderly– Application-specific loss functions
• Affordability– Minimal transcription– Replicable process
![Page 28: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/28.jpg)
Application-Tuned ASR
• Acoustic model– Transcribe short segments from many speakers– Unsupervised adaptation
• Language model– Transcribed segments– Interpolation
![Page 29: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/29.jpg)
ASR Game Plan
Hours Word
Language Transcribed Error Rate
English 200 39.6%
Czech 84 39.4%
Russian 20 (of 100) 66.6%
Polish
Slovak
As of May 2003
![Page 30: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/30.jpg)
English Transcription Time
~2,000 hours to manually transcribe 200 hours from 800 speakers
Hours to transcribe 15 minutes of speech
Inst
ance
s (N
=83
0)
![Page 31: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/31.jpg)
English ASR Error Rate
0
20
40
60
80
100
Wo
rd E
rro
r R
ate
Training: 65 hours (acoustic model)/200 hours (language model)
![Page 32: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/32.jpg)
MALACH Overview
AutomaticSearch
BoundaryDetection
InteractiveSelection
ContentTagging
SpeechRecognition
QueryFormulation
UserNeeds
Observational studiesFormative evaluationSummative evaluation
![Page 33: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/33.jpg)
Who Uses the Collection?
• History• Linguistics• Journalism• Material culture• Education• Psychology• Political science• Law enforcement
• Book• Documentary film• Research paper• CDROM• Study guide• Obituary• Evidence• Personal use
Discipline Products
Based on analysis of 280 access requests
![Page 34: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/34.jpg)
Question Types
• Content– Person, organization– Place, type of place (e.g., camp, ghetto)– Time, time period– Event, subject
• Mode of expression– Language– Displayed artifacts (photographs, objects, …) – Affective reaction (e.g., vivid, moving, …)
• Age appropriateness
![Page 35: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/35.jpg)
Observational Studies
• Four searchers– History/Political Science– Holocaust studies– Holocaust studies– Documentary filmmaker
• Sequential observation• Rich data collection
– Intermediary interaction– Semi-structured interviews– Observational notes– Think-aloud– Screen capture
• Four searchers– Ethnography
– German Studies
– Sociology
– High school teacher
• Simultaneous observation
• Opportunistic data collection– Intermediary interaction
– Semi-structured interviews
– Observational notes
– Focus group discussions
Workshop 1 (June) Workshop 2 (August)
![Page 36: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/36.jpg)
Segment Viewer
![Page 37: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/37.jpg)
Observed Selection Criteria
• Topicality (57%)Judged based on: Person, place, …
• Accessibility (23%)Judged based on: Time to load video
• Comprehensibility (14%)Judged based on: Language, speaking style
![Page 38: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/38.jpg)
References to Named Entities
AttributesMentions
Selection Reformulation
Person
(N=138)
GenderCountry of birthNationalityDate of birthStatus, intervieweeStatus, parents
110101
221513111211
Place
(N=116)
CampCountryGhetto
10 8 7
451612
![Page 39: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/39.jpg)
FunctionalityNeeded Function Boolean Search and Ranked Retrieval (13)
Testimony summary (12)
Pre-Interview Questionnaire search/viewer (9)
Rapid access (7)
Related/Alternative search terms (3)
Adding multiple search terms at once (2)
Keywords linked to segment number for easy access(1)
Multi-tasking (1)
Searching testimonies by places under ‘Experience Search’ (1)
Extensive editing within ‘My Project’ (1)
Desired Function Temporary saving of selected testimonies (4)
Remote access (3)
Integrated user tools for note taking (3)
Map presentation (2)
Reference tool (1)
More repositories (1)
Introductory video of system tutorial (1)
Help (1)
![Page 40: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/40.jpg)
MALACH Overview
AutomaticSearch
BoundaryDetection
InteractiveSelection
ContentTagging
SpeechRecognition
QueryFormulation
NLPComponents Multi-scale segmentation
Multilingual classificationEntity normalization
![Page 41: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/41.jpg)
scratchpad
transcript
“True” segmentation:transcripts aligned with scratchpad-based
boundaries
Hours Words Sentences Segments
Training 177.5 1,555,914 210,497 2,856
Test 7.5 58,913 7,427 168
Topic Segmentation
cataloguer
![Page 42: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/42.jpg)
true
system output
missfalsealarm
Effect of ASR Errors
![Page 43: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/43.jpg)
Rethinking the Problem
• Segment-then-label models planned speech well– Producers assemble stories to create programs– Stories typically have a dominant theme
• The structure of natural speech is different– Creation: digressions, asides, clarification, …– Use: intended use may affect desired granularity
• Documentary film: brief snippet to illustrate a point• Classroom teacher: longer self-contextualizing story
![Page 44: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/44.jpg)
OntoLog: Labeling Unplanned Speech
• Manually assigned labels; start and end at any time– Ontology-based aggregation helps manage complexity
![Page 45: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/45.jpg)
Goal
Use available data to estimate the temporal extent of labels in a way that optimizes the utility of the resulting estimates for interactive searching and browsing
![Page 46: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/46.jpg)
Multi-Scale SegmentationL
abel
s
Time
![Page 47: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/47.jpg)
Characteristics of the Problem
• Clear sequential dependencies– Living in Dresden negates living in Berlin
• Heuristic basis for class models– Persons, based on type of relationship– Date/Time, based on part-whole relationship– Topics, based on a defined hierarchy
• Heuristic basis for guessing without training– Text similarity between labels and spoken words
• Heuristic basis for smoothing– Sub-sentence retrieval granularity is unlikely
![Page 48: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/48.jpg)
Manually Assigned Onset Marks
Subject PersonLocation-Time
Berlin-1939
Dresden-1939
Employment Josef SteinGretchen SteinAnna Stein
RelocationTransportation-rail
SchoolingGunter Wendt
Family Life
Maria
inte
rvie
w ti
me
![Page 49: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/49.jpg)
Some Additional Results
• Named entity recognition– F > 0.8 (on manual transcripts)
• Cross-language ranked retrieval (on news)– Czech/English similar to other language pairs
![Page 50: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/50.jpg)
Looking Forward: 2003
• Component development– ASR, segmentation, classification, retrieval
• Ranked retrieval test collection– 1,000 hours of English recognition– 25 judged topics in English and Czech
• Interactive retrieval– Integrating free text and thesaurus-based search
![Page 51: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/51.jpg)
![Page 52: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/52.jpg)
Relevance Categories
• Overall relevanceAssessment is informed by the assessments for the individual reasons for relevance (categories of relevance), but the relationship is not straightforward
• Provides direct evidence
• Provides indirect / circumstantial evidence
• Provides context(e.g., causes for the phenomenon of interest)
• Provides comparison (similarity or contrast, same phenomenon in different environment, similar phenomenon)
• Provides pointer to source of information
![Page 53: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/53.jpg)
Scale for overall relevance
Strictly from the point of view of finding out about the topic, how useful is this segment for the requester? This judgment is made independently of whether another segment (or 25 other segments) give the same information.
4 Makes an important contribution to the topic, right on target
3 Makes an important contribution to the topic
2 Should be looked at for an exhaustive treatment of the topic
1 Should be looked at if the user wants to leave no stone unturned
0 No need to look at this at all
![Page 54: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/54.jpg)
Direct relevanceDirect evidence for what the user asks for
Directly on topic, direct aboutness. The information describes the events or circumstances asked for or otherwise speaks directly to what the user is looking for. First-hand accounts are preferred, e.g., the testimony contains a report on the interviewee's own experience, or an eye-witness account on what happened, or self-report on how a survivor felt. Second-hand accounts (hearsay) are acceptable, such as a report on what an eyewitness told the interviewee or a report on how somebody else felt.
* Direct Evidence *- Evidence that stands on its own to prove an alleged fact, such as testimony of a witness who says she saw a defendant pointing a gun at a victim during a robbery. Direct proof of a fact, such as testimony by a witness about what that witness personally saw or heard or did. ('Lectric Law Library's Lexicon)
![Page 55: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/55.jpg)
Indirect relevanceProvides indirect evidence on the topic, indirect aboutness (data from which one could infer, with some probability, something about the topic, what in law is known as circumstantial evidence) Such evidence often deals with events or circumstances that could not have happened or would not normally have happened unless the event or circumstance of interest (to be proven) has happened. It may also deal with events or circumstances that precede the events or circumstances of interest, either enabling them (establishing their possibility) or establishing their impossibility. This category takes precedence over context. One could say that provides indirect evidence also provides context (but the reverse is not true).
* Circumstances, Circumstantial Evidence * Circumstantial evidence is best explained by saying what it is not - it is not direct evidence from a witness who saw or heard something. Circumstantial evidence is a fact that can be used to infer another fact.
![Page 56: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/56.jpg)
ContextProvides background / context for topic, sheds additional light on a topic, facilitates understanding that some piece of information is directly on topic.
So this category covers a variety of things. Things that influence, set the stage, or provide the environment for what the user asks for. (To take the law analogy again any things in the history of a person who has committed a crime that might explain why he committed it).
Includes support for or hindrance of an activity that is the topic of the query andactivities or circumstances that immediately follow on the activity or circumstance of interest.
In a way, this category is broader than indirect If a context element can serve as indirect evidence, indirect takes precedence.
![Page 57: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/57.jpg)
Comparison
Provides information on similar / parallel situations or on a contrasting situation for comparison
The basic theme of what the user is interested in, but played out in a different place or time or type of situation.
Comparable segments will be those segments that provide information either on similar/parallel topics, or on contrasting topics. This type of relevance relationship identifies items that can aid understanding of the larger framework, perhaps contributing to identification of query terms or revision of search strategies. An example would be a segment in which an interviewee describes activities like activities described in a topic description, but which occurred at a different place or time than the topic description
![Page 58: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/58.jpg)
Pointer
Provides pointers to a source of more information. This could be a person, group, another segment, etc
•Pointers will be segments that provide suggestions or explicit evidence of where to find more relevant information. An example of a pointer segment would be one in which an interviewee identifies another interviewee who had personal experiences directly associated with the topic. The value of these segments is in identifying other relevant segments, particularly but not limited to segments about a topic.
![Page 59: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/59.jpg)
Quality Assurance
• 20 topics were redone, 10 were reviewed.
• Redo: A second assessor did a topic from scratch
• Review: A second assessor reviewed the first assessors work and did additional searches when needed.
• Assessors would then get together and discuss their interpretation of the topic and resolved differences in relevance judgments.
• Assessors kept notes on the process.
![Page 60: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/60.jpg)
Looking Forward: 2006
• Working systems in five languages– Real users searching real data
• Rich experience beyond broadcast news– Frameworks, components, systems
• Affordable application-tuned systems– Oral history, lectures, speeches, meetings, …
![Page 61: Multilingual Access to Large Spoken Archives Douglas W. Oard University of Maryland, College Park, MD, USA](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649e165503460f94b01723/html5/thumbnails/61.jpg)
For More Information
• The MALACH project– http://www.clsp.jhu.edu/research/malach/
• NSF/EU Spoken Word Access Group– http://www.dcs.shef.ac.uk/spandh/projects/swag/
• Speech-based retrieval– http://www.glue.umd.edu/~dlrg/speech/