multimedia semantic analysis in the prestospace project
DESCRIPTION
Multimedia Semantic Analysis in the PrestoSpace Project. Valentin Tablan, Hamish Cunningham, Cristian Ursu NLP Research Group University of Sheffield Regent Court, 211 Portobello Street, Sheffield, S1 4DP, UK http://nlp.shef.ac.uk , http://gate.ac.uk. Project Mission. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/1.jpg)
Multimedia Semantic Analysis in the PrestoSpace Project
Valentin Tablan, Hamish Cunningham,Cristian Ursu
NLP Research GroupUniversity of Sheffield
Regent Court, 211 Portobello Street,Sheffield, S1 4DP, UK
http://nlp.shef.ac.uk, http://gate.ac.uk
![Page 2: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/2.jpg)
2LREC 2006, Genoa, Italy – Crossing Media Workshop
Project Mission
The 20th Century was the first with an audiovisual record. Audiovisual media became the new form of cultural expression. These historical, cultural and commercial assets are now entirely at risk from deterioration.
PrestoSpace aims to provide technical devices and systems for digital preservation of all types of audio-visual collections.
![Page 3: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/3.jpg)
3LREC 2006, Genoa, Italy – Crossing Media Workshop
The Partners IP, 34 partners Steering Board:
Institut National de l’Audiovisuel INA (France)British Broadcasting Corporation BBC (UK)Radiotelevisione Italiana RAI (Italy)Joanneum Research JRS (Austria)Netherlands Institute for Sound and Vision - Beeld
en Geluid B&G (The Netherlands) Oesterreichischer Rundfunk ORF (Austria)University of Sheffield USFD (UK)
![Page 4: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/4.jpg)
4LREC 2006, Genoa, Italy – Crossing Media Workshop
Project OrganisationPreservation
Playback DevicesRobotics and Automation
Media Condition Assessment ToolsInformation System and Integration
Storage
Technology Information ToolsMass Storage Quality Assurance
Delivery & Access
Turnkey SystemExport System
Restoration
System ToolsVisual & Audio Algorithms
Restoration SubsystemsIntegration and Evaluation
Archive ManagementPreservation and Access - Business Case Planning
Preservation Project-Management
Preservation Quality Assurance
Metadata
Discovery and StructuringPublic Access
Delivery and Exchange
![Page 5: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/5.jpg)
5LREC 2006, Genoa, Italy – Crossing Media Workshop
Semantic Analysis – Motivation Sizeable archives plus new material
produced daily (BBC has 8 TV and 11 radio national channels). Some of this material can be reused in new productions.
Access to archive material can be provided by some form of semantic annotation and indexing, but manual annotation is time consuming (up to 10x real time) and expensive.
Archive budgets alone cannot support digitisation effort.
![Page 6: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/6.jpg)
6LREC 2006, Genoa, Italy – Crossing Media Workshop
English SA - RichNews A prototype addressing the automation of
semantic annotation for multimedia material. Not aiming at reaching performance
comparable to that of human annotators. Fully automatic. Aimed at news material, further extensions
envisaged. TV and radio news broadcasts from the BBC
were used during development and testing.
![Page 7: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/7.jpg)
7LREC 2006, Genoa, Italy – Crossing Media Workshop
Overview Input: multimedia file Output: OWL/RDF descriptions of content
Headline (short summary)List of entities (Person/Location/Organization/…)Related web pagesSegmentation
Multi-source Information Extraction systemAutomatic speech transcriptSubtitles/closed captionsRelated web pagesLegacy metadata
![Page 8: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/8.jpg)
8LREC 2006, Genoa, Italy – Crossing Media Workshop
Using ASR TranscriptsASR is performed by the THISL system. Based on ABBOT connectionist speech
recognizer. Optimized specifically for use on BBC news
broadcasts. Average word error rate of 29%. Error rate of up to 90% for out of studio
recordings. No capitalisation – limited IE capability.
![Page 9: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/9.jpg)
9LREC 2006, Genoa, Italy – Crossing Media Workshop
ASR error exampleshe was suspended after his arrest [SIL] but the process were set never to have lost confidence in him
he was suspended after his arrest [SIL] but the Princess was said never to have lost confidence in him
and other measures weapons inspectors have the first time entered one of saddam hussein's presidential palaces
United Nations weapons inspectors have for the first time entered one of saddam hussein's presidential palaces
![Page 10: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/10.jpg)
10LREC 2006, Genoa, Italy – Crossing Media Workshop
Architecture
THISLSpeech Recogniser
C99 Topical Segmenter
TF.IDFKey Phrase Extraction
Media File
Manual Annotation(Optional)Entity Validation Semantic Index
Web-Search and
Document Matching
KIMInformation Extraction
Degraded TextInformation Extraction
![Page 11: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/11.jpg)
11LREC 2006, Genoa, Italy – Crossing Media Workshop
Search for Related Pages ASR transcript segmented using C99. Key-phrases found for each segment using
TF/IDF. Any sequence of up to three words can be a phrase; up
to four phrases extracted per story. Key-phrases used to search the BBC, Times,
Guardian and Telegraph newspaper websites. Searches are restricted to the day of broadcast, or the day after.
The text of the returned web pages is compared to the text of the transcript to find matching stories.
![Page 12: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/12.jpg)
12LREC 2006, Genoa, Italy – Crossing Media Workshop
Using the Web Pages
The web pages contain: A headline, summary and section for each
story. Good quality text that is readable, and
contains correctly spelt proper names. They give more in depth coverage of the
stories.
![Page 13: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/13.jpg)
13LREC 2006, Genoa, Italy – Crossing Media Workshop
Semantic Annotation The KIM knowledge management system can
semantically annotate the text derived from the web pages: KIM will identify people, organizations, locations etc. KIM performs well on the web page text, but very poorly
when run on the transcripts directly. This allows for semantic ontology-aided searches
for stories about particular people or locations etcetera. So we could search for people called Sydney, which
would be difficult with a text-based search.
![Page 14: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/14.jpg)
14LREC 2006, Genoa, Italy – Crossing Media Workshop
Entity Matching
![Page 15: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/15.jpg)
15LREC 2006, Genoa, Italy – Crossing Media Workshop
Evaluation Evaluation based on 66 news stories from 9
half-hour news broadcasts. Web pages were found for 40% of stories. 7% of pages reported a closely related story,
instead of that in the broadcast. Lenient recall: 47%, precision: 100%. Results are based on earlier version of the
system, only using BBC web pages.
![Page 16: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/16.jpg)
16LREC 2006, Genoa, Italy – Crossing Media Workshop
Future Improvements
Use teletext subtitles (closed captions) when they are available
Better story segmentation through visual cues.
Use for different domains and languages.
![Page 17: Multimedia Semantic Analysis in the PrestoSpace Project](https://reader036.vdocuments.us/reader036/viewer/2022062810/56815dfe550346895dcc3c43/html5/thumbnails/17.jpg)
17LREC 2006, Genoa, Italy – Crossing Media Workshop
Thank you!
More information:
http://www.prestospace.org
http://gate.ac.uk