WP5.4/3.1/4.2/5.5 meeting

29th of November 2007, DFKI

Agenda

9:00 - 9:15        welcome, intro and agenda 9:15 - 10:00      Jan Nemrava: current work in WP5.4/3.1

– video analysis, extended tickers coverage 10:00  - 10:30   Nikos Simou: FiRE reasoner 10:30 - 11:00    discussion on connection WP4.2 (FiRE

reasoner) with cross-wp work on 5.4/3.1 11:00 - 11:30    coffee break 11:30 - 13:00    wp5.5 - IPTC manipulator, KAT

integration, discussion about WP5.4 demo

13:00 - 14:00    lunch

Current work in WP5.4/WP3.1

• Textual Sources

• Video Analysis

• OCR

• Reasoning (FiRE)

• KAMC paper

WP5.4

• semantically annotate complementary resources

• extract semantically organized ‘cross-media features’from the aligned text and video data sets

• use in event type classification of video segments by use of fuzzy reasoning

Textual Resources

• Structured– Players names, numbers, attendance, place, date etc.

• Unstructured– 3 German and 3 English minute by minute reports for

all World Cup matches– SPRouT grammar differs for each language– Ita-Fra events extracted from text

shortest longest german english allplayeraction 31/31 104/112 76/89 230/315 308/404player 68/73 134/157 176/259 186/310 286/569

without duplicate concepts / all extracted concepts

OCR (with TUB and JRS)

• Time synchronization

• Additional primary resource

• Merging 16 frames to recognize moving vs. Static objects in the image

Text region detection example

audio-video analysis

• S

audio-video analysis (DCU)

• Crowd image detector• Speech-Band Audio Activity• On-Screen Graphics Tracking• Motion activity measure• Field Line orientation• Close-up

AUDIO3NEXT

0

20

40

60

80

100

120

0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

27 0 0 0 0 0 0

AUDIO3NEXT

audio-video analysis (DCU)

• Experiments with machine learning– Linear regression used to classify

• Midfield shot• Endzone shot• Other (replays, shot on crowd, etc)

– K-Nearest Neighbour to classify • Cornerkicks

– KL-Miner• Part of LispMiner developed at UEP• Time Series analysis

• By considering seconds as isolated instances we loose a lot of information

IPTC manipulator (UEP)• Extraction of semantic concepts from

unstructured texts using DFKI ontology based information extraction tool

IPTC and extraction problems

• Inability to insert multiple values into certain fields (in case they are duplicate) - city, country, date

• SPRouT identifies concepts that are in the text but are not depicted on the picture

• Which information are really relevant for annotation (information about future, past?)(possibility to use sentence syntax analysis to solve this problem)

• Domain dependency

ITPC manipulator demo

ITPC manipulator

• Swiss forward Alexander Frei (R) goes airborne challenged by Ukrainian midfielder Oleg Gusev (L) during the World Cup 2006 round of 16 football game Switzerland vs. Ukraine, 26 June 2006 at Cologne stadium. AFP PHOTO / ODD ANDERSEN

• A boy shows a football autographed by French midfielder Zinedine Zidane at the entrace of at Weserbergland Stadium in Hameln, 03 July 2006, where the French national team are training ahead of their next match. France on 01 July defeated Brazil 1-0 and will play Portugal 05 July in Munchen in a 2006 World Cup semifinal showdown. AFP PHOTO / DANIEL GARCIA

• SCHUG output??

Japan's Keisuke Tsuboi, left, and Singapore's Indra Sahdan Daud battle

for the ball during the first half in their Asian zone first round of

the World Cup soccer soccer qualifying match at Saitama Stadium in

Saitama, near Tokyo, Wednesday, Nov. 17, 2004. Japan leads Singapore 1-0 at half-time.

Discussion

• Demo during the K-Space review (Berlin)

• Complementary Resources Architecture– KAT implementation– COMM ontology with descriptors to express

results of comp. resources analysis.

• IPTC manipulator– Future plans

WP5.4 demo