1

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

JOANNEUM RESEARCH

a TRADITION of INNOVATION

Herwig Rehatschek

Institute of Information Systems & Information Management

Cross Media Aspects in the Areas of Media Monitoring

and Content Production

2

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Contents

Cross media aspects in media production (NM2 project) in media monitoring (DIRECT-INFO project)

Conclusions

Outlook

3

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

NM2 (New Media for a New Millennium) - MotivationTraditional distribution channels have developed their specific compelling media forms art, science and technology of production, direction and editing in

non-linear narrative are poorly formed

NM2 investigates on non-linear, interactive narratives tools for production and delivery

Main Objectives of NM2 proof of concept to produce compelling non-linear narratives production of compelling media

aesthetically pleasing truly engaging (30+ minutes) plausible from a business perspective

4

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Basic Technical Concepts

Narrative Objects (NO) MPEG-7 to describe audiovisual features OWL to describe semantics of NO

Narrative Structure Language internal representation of a „story world“ basic narrative structures

5

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Media Semantic Mapping Utility (Monsoon)

Tabs for objects, concepts and relations

Definition of an object by low level features

Definition of an object by using other objects

6

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

System Architecture

metadata essence stories

Database interface

Middleware API

user info

Management Realisation Engine Profile Learning

Inge

stio

nDescription•Manual•Automated•Ontology-based

AuthoringUsing narrative

structure language

Session ManagementFor high volumes

End-user ApplicationIncluding application preview

tools delivery

middleware

JRS

7

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Main Project Results

Definition of a Narrative Structure Language describing multimedia objects in narrative context

Bridging semantic gap MPEG-7 & OWL linking lower level features to higher level concepts

Six NL productions with compelling media

Targeted for three end-user devices Windows XP Media Centre and TV card Game consoles Mobile phones

8

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

DIRECT-INFO - Background Media Monitoring Companies

Measurement of advertisement expenditure

Find out how much a company has spent for advertising on a product level over different media

Typical business cases advertisement information creativity (new campaigns) (targeted in MediaCampaign) sponsorship tracking (targeted in DIRECT-INFO)

9

18/0

4/23

ISO

9001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Sponsorship Tracking

Sport teams need sponsors to finance their activities

Measure efficiency of sport sponsorship broadcasting & print media convince sponsors to invest again currently done manually

Technically very complex task context information needed multimodal analysis necessary

10

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Main Technical Innovations

Identification of logos within visual content

Multimodal event modelling

Genre classification

Text analysis (pos. / neg. mentions)

Com

bin

ati

on

of

Semantic correlation & fusion of multi-modal content analysis tosemantically meaningful reports

11

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Technical Workflow (1/2)

12

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Technical Workflow (2/2)

13

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

BrandDetector – Detection of Logos

14

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

MPEG-7 Result Viewer/Editor

Player

Key framesshots

Basic events

Module parameters

15

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

ConclusionsStandardized metadata (MPEG-7, OWL) is a good choice otherwise a lot of time will be lost with specification and

extensions MPEG-7, OWL already provided the needed structures one expert for MPEG-7, OWL is necessary

Web services not so straightforward as always thought managed vs. unmanaged code not all SOAP protocol features supported

Do not underestimate integration efforts! take security issues into account remote access essential

16

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Outlook: MediaCampaignR&D project, Started: 04/2006

Identification and tracking of new campaigns in different media (Press, TV, Internet) utilizing different modalities (audio, text, image, video)

Semantic cross-market advertis. interlinking relate similar advertisements over different media &

countries

Define Ontology for cross-media modeling of media presence and campaigns Media Presence and Campaign Ontology (MEPCO) includes media presence information

17

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Relevant LinksNM2 website http://www.ist-nm2.org/

DIRECT-INFO website http://www.direct-info.net

MediaCampaign website http://www.media-campaign.eu

JOANNEUM RESEARCH MPEG-7 Library available for free! full support for parts 3, 4 and 5 of

the MPEG-7 standard http://mpeg-7.joanneum.at

Available inJune 2006!

18

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

JOANNEUM RESEARCHInstitute of Information Systems &

Information ManagementSteyrergasse 17

A-8010 Graz / AUSTRIAVoice: +43 316 876-1194

Mobile: +43 699 1876-1194Fax: +43 316 876-1191

E-mail: Herwig.Rehatschek@joanneum.atWeb: http://www.joanneum.at/iis

Dr. Herwig Rehatschek

20

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Delivery

Optimiser

InferenceEngine

Realisation Engine

Delivery SystemRendering

Engine

StreamingServer

Client

MediaPlayer

MediaStorage

playlists

video

21

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

MPEG-7 Automatic Detection Preview (UI)

22

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

System Architecture

23

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Genre Classification

Fully automatic classification of commercials news studio/speaker soccer credits movie

Real-time analysis

Semantic block classification ~90% precision ~90% recall

24

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Multimodal Event Modelling

Event detection in audio and video

Integration of basic events visual shots audio segments overlaid text visible frontal faces

Pseudo-scene/story segmentation

Classified segments

s1 s2 c1 s2a2+s2 s3 c2 c3 s4 u s4 s5

25

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Text AnalysisDetect positive & negative mentions of brand names teams

SCHUG text analysis engine “Shallow and Chunk-based Unification Grammar” robust chunk parser for linguistic analysis

Topic detection

Sources transcripts and text captions of TV audio newspaper texts supported languages: English, Italian

26

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Semantic Data Fusion

Combing data from various analysis subsystems

Based on time-stamped analysis results of all analysis modules

Rule-based derivation of new facts basic appearancescomplex appearances

Two main user interfacesconsolidation of appearancesset-up application for delivery system

27

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Semantic Data FusionDisplay of Results

28

18/

04/2

3IS

O 9

001

cert

ified

Herwig RehatschekLREC Workshop, Genoa23 May 2006

Delivery system for Customers