the automatic generation of formal annotations in a multimedia indexing and searching environment...
TRANSCRIPT
The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment
Thierry DeclerckDFKI GmbH
Annotation Workshop, DI, 15. Februar 2002
The MUMIS Consortium
• CTIT University of Twente, Enschede, NL NLP/IE• TSI University of Nijmegen, Nijmegen, NL
ASR• DFKI Saarbrücken, D
NLP/IE• MPI Nijmegen, NL MM
Archives• DCS University of Sheffield, UK NLP/IE• ESTEAM Gothenburg, SE (location Athens, GR)
Translation Software
• VDA Hilversum, NL Video
Objectives of MUMIS
• Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process)
Find and annotate relevant events, together with the involved entities and relations. Also detect Metadata information.
• Technology development to exploit indexed multimedia archives (on-line process)
Search for interesting scenes and play them via Internet
Test Domain: Soccer Games / UEFA Tournament 2000
Off-line Task
Indexing by
• Automatic Speech Recognition (Radio/TV Broadcasts)
Automatically transforms the speech signals into texts (for 3 languages — Dutch, English and German)
• Natural Language Processing (Information Extraction)
Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts ...), identify and extract interesting entities, relations and events. Also detect Metadata information.
• Merging all the annotations produced so far• Create a database with formal annotations• Use video processing to adjust time marks
Gain
• What gets lost? Is it necessary?• Potential: direct Internet Service, less
dependencies
Current Procedure MUMIS Procedure
Manual Video Annotation Automatic Video Annotation and DB
IntegrationIntegration Central DB
Query via PC Query via PC
Results on PC
Results on PCAnd
Select & Play
Contact Video Archive
Get Video Tapes
Search on Tape on VCR
Segment & Play
The Generation of Formal Annotations
• Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive• Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences• All Formal Annotations available in XML Standard
The Event TableRelated to domain ontology and multilingual terminology. Guiding the generation of formal annotations
Final whistle # 90>t>120
Subj=referee, score etc… Final score
Shot on Goal # 0>t>120
Subj=pl, loc=loc, cons=cons,..
Dribbling # 0>t>120
Subj=pl, loc=loc, …
Substitution # 0>t>120
Subj=pl, I.obj=pl, cause=c, …
Team (adding pl)
Red Card # 0>t>120
Subj=ref, I.obj=pl, cause=c, …
Team (red at t)
Goal # 0>t>pen.
Subj=pl, I.obj=team, score=s,
Order of goal
…
Event ID
Time Subcat/Modification Metadata
Off-line Task
Events indexed in video recording
1:0
60 m25 m25 m
SchollBasler
CampbellMatthäusBaslerNeville
DribblingFreekick
28min24 min18 min17 min
DefensePassGoalFoul
Radio Commenting3 Languages
Radio Commenting3 Languages
Radio Commenting3 Languages
Audio Commenting (TV, Radio)3 Languages
NewspaperText
NewspaperText
NewspaperText
NewspaperTexts
3 Languages
NewspaperText
NewspaperText
NewspaperText
Tickers etc.3 Languages
multilingual IE
=> event tables
Merging of Annotations
Event = goal Player = Basler
Dist. = 25 m Time = 18
Score = 1:0
Event = goal Type = Freekick Player = Basler Dist. = 25 m Time = 17
Score: leading
Event = goal Player= Basler Team = GermanyTime = 18 Score = 1:0 Finalscore = 1:0
Event = goal Type = Freekick
Player = Basler Team = GermanyTime = 18 Score = 1:0 Final score = 1:0 Distance = 25 m
The Role of IE in MUMIS
• Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user.
• The relevant information is typically represented in form of predefined “templates”, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS)
• IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).
Extension of our IE system in MUMIS
• Multilingual and multisource IE. Incremental information building
• Cross-document co-reference resolution• Combine Metadata and event extraction =>
better organisation and dynamic updating of information (KM)
• Multiple presentation of results: Template, Event table, integration in MPEG-7 XML and Hyperlinks (Named Entities, rel. to Knowledge Management)
The DFKI Implementation
• Based on XML output of SPPC (Dev. At DFKI)
• Mapping the XML into a feature structure (the CorpA/schug Program)
• Cascaded grammar descriptions for enriching (or correcting) the SPPC output
• Including agreement processing and detection of grammatical functions
• Adapting the “Paradime triangle” for template generation and filling
Information Extraction
IE is generally subdivided in following tasks:- Named Entity task (NE) - Template Element task (TE)- Template Relation task (TR)- Scenario Template task (ST) - Co-reference task (CO)
Subtasks of IE
• Named Entity task (NE): Mark into the text each string that represents, a person, organization, or location name, or a date or time, or a currency or percentage figure.
• Template Element task (TE): Extract basic information related to organization, person, and artifact entities, drawing evidence from everywhere in the text.
Subtasks of IE (2)
• Template Relation task (TR): Extract relational information on employee_of, manufacture_of, location_of relations etc. (TR expresses domain-independent relationships).
• Scenario Template task (ST): Extract pre-specified event information and relate the event information to particular organization, person, or artifact entities (ST identifies domain and task specific entities and relations).
• Co-reference task (CO): Capture information on co-referring expressions, i.e. all mentions of a given entity, including those marked in NE and TE.
IE applied to soccer
Terms as descriptors for the NE task Team: Titelverteidiger Brasilien, den respektlosen
Außenseiter Schottland Player:Superstar Ronaldo, von Bewacher Calderwood noch
von Abwehrchef Hendry, von Jackson als drittem Stürmer, Torschütze Cesar, von Roberto Carlos (16.),
Referee: vom spanischen Schiedsrichter Garcia ArandaTrainer: Schottlands Trainer Brown, Kapitän Hendry seinen
Keeper LeightonLocation: im Stade de France von St. Denis (more fine-
grained location detection would be: Stadion: im Stade de France and City: von St. Denis )
Attendance: Vor 80000 Zuschauern
IE applied to soccer (2)
Terms for NE TaskTime: in der 73. Minute, nach gerade einmal 3:50 Minuten,
von Roberto Carlos (16.), nach einer knappen halben Stunde, scheiterte Rivaldo (49./52.) jeweils nur knapp, das vor der Pause Versäumte versuchten die Brasilianer nach Wiederbeginn, ...
Date: am Mittwoch, der Turnierstart (?), im WM-Eröffnungsspiel (?)
Score/Result: Brasilien besiegt Schottland 2:1, einen 2:1 (1:1)-Sieg, der zwischenzeitliche Ausgleich, in der 4. Minute in Führung gebracht, köpfte zum 1:0 ein
IE applied to soccer (3)
Relations for TR TaskOpponents: Brasilien besiegt Schottland, feierte der Top-
Favorit ... einen glücklichen 2:1 (1:1)-Sieg über den respektlosen Außenseiter Schottland,
Player_of: hatte Cesar Sampaio den vierfachen Weltmeister ... in Führung gebracht, Collins gelang ... der zwischenzeitliche Ausgleich für die Schotten, der Keeper des FC Aberdeen, Brasiliens Keeper Taffarel
Trainer_of: Schottlands Trainer Brown...
IE applied to soccer (4)
Events for ST task:Goal: in der 4. Minute in Führung gebracht, das schnellste
Tor ... markiert, Cesar Sampaio köpfte zum 1:0 ein, Collins (38.) verwandelte den Strafstoß, hätte Kapitän Hendry seinen Keeper Leighton um ein Haar zum zweiten Mal bezwungen, von dem der Ball ins Tor prallte
Foul: als er den durchlaufenden Gallacher im Strafraum allzu energisch am Trikot zog
Substitution: und mußte in der 59. Minute für Crespo Platz machen...
IE applied to soccer (5)
Description of the Templates: Teamteam-templateTACTIC [ ] SCORE [ ]NAME [ ]PLAYER [ ]TRAINER [ ]
goal-templateTIME [ ]SCORE [S]PLAYER [P]TEAM [team-templ ]TYPE [ ]SUCCESS [ ]
team-templateTACTIC [ ] SCORE [S]NAME [ ]PLAYER [P]TRAINER [ ]
Merging Component
• Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those
• Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology
• Completing the information with domain knowledge, inference Machine
Use of Standards
• XML as the annotation language and data interchange format
• MPEG-7: standard for the description of features of multimedia content, XML compliant (for content description), with a slot for textual annotations
More about MPEG (Moving Picture Coding Experts Group)
• MPEG-1: For the storage and retrieval of movie pictures and audio on storage media
• MPEG-2: For digital television• MPEG-4: Codes content as objects and
enables those objects to be manipulated
• MPEG-7: Where 1,2 and 4 make content available, MPEG-7 allows to find the content one needs
On-line Tasks
Searching and Displaying
• Search for interesting events with formal queriesGive me all goals from Overmars shot with his head in 1.
Half.Event=Goal; Player=Overmars; Time<=45; Previous-
Event=Headball
• Indicate hits by thumbnails & let user select scene
• Play scene via the Internet & allow scrollingOf course: slow motion, fast play, start/stop, etc
On-line Tasks
Searching and Displaying
• Search for interesting events with formal queriesGive me all goals from Overmars shot with his head in 1.
Half.Event=Goal; Player=Overmars; Time<=45; Previous-
Event=Headball
• Indicate hits by thumbnails & let user select scene
• Play scene via the Internet & allow scrollingOf course: slow motion, fast play, start/stop, etc
On-line Tasks
Knowledge GuidedUser Interface
&Search Engine
München - Ajax1998
München - Porto1996
Deutschland - Brasilien1998
PlayMovie
Fragmentof that Game
Freekick Goal Pass Defense
17 min 18 min 24 min 28min
Foul Freekick Dribbling
Kohler Basler Matthäus Wörns
Basler Bierhoff
25 m 25 m 60 m
On-line SW Architecture
ClientApplet
JMF
WWW ServerJava Server
MediaServer MPEG1
MediaServer MPEG1
MediaServer MPEG1
DBServer rDBMS
MediaServer MPEG1
FileServer
HTTPRMI
RMI (RTP, RTSP)
JDBC
Client Objects
Hit Rendering Objects
Media Server Objects
Query Engine Objects
MetadataAnnotations
KeyframesMPEG Movies
Lexica
Ontology
Query interface:• pre-selection• guided by domain knowledge• interactive, visual feedback
Server structure:• fully distributed• JMF media presentation • RMI-based interaction
On-line HW Architecture
• efficient & reliable storage management (near-line capacity, media change, 2. Location)
• high storage capacity (n TB, 1 h MPEG1 = 1 GB)• powerful media servers / powerful network
RAID
TapeLibrary
FC Switch
Media Server
Media Server
GB Switch
Internet
1GbpsGb-Switch
Router
Acknowledgements
• UEFA
• DFB, FA, KNVB
• EBU, WDR, NOS, SWR