Deep-linking into Media Assets at the Fragment

Level: Specification, Model and Applications

Raphaël Troncy <raphael.troncy@eurecom.fr>

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 2

TimBL Vision back in 1994

A typical HTML web page

17/12/2013 - - 3 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

What it looks like to a machine

17/12/2013 - - 4 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

Okay, so HTML is not helpful

Maybe we can tell the machine what the

different parts of the text represent?

title

time

speaker

location

abstract

biosketch

host

17/12/2013 - - 5 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

XML to rescue?

XML fans propose creating a XML tag set to use for each

application.

For talks, we can choose <title>, <speaker>, etc.

<title>

<time>

<speaker>

<location>

<abstract>

<biosketch>

<host> </host></biosketch>

</abstract>

</location></time>

</speaker>

</title>

17/12/2013 - - 6 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

XML machine accessible meaning

But, to your machine, the tags

still look like this….

The tag names carry no meaning.

XML DTDs andSchemas have

little or no semantics.

< >title

< >time

< >speaker

< >location

< >abstract

< >biosketch

< >host </ >host</ >biosketch

</ >abstract

</ >location</ >time

</ >speaker</ >title

17/12/2013 - - 7 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

do not readthe following sign

youloose

we interpretmachines don't

Why is it so difficult to find appropriate multimedia content, to

reuse and repurpose content previously published and to present this content in interfaces that vary

with user needs?

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 13

Image/Video indexing

Techniques used by mainstream search engines search term occurs in the filename or in the caption or in

user tags no semantics

Image indexing: main problem an image is not alphabetic: there is no countable discrete

units, that, in combination will provide the meaning of the image

image descriptors are not given with the image: one needs to extract or interpret them

Video indexing: additional problem a video has additionally a temporal dimension to take into

account a video has a priori no discrete units neither (i.e. frames,

shots, sequences cannot be absolutely defined)

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 14

Sounds Familiar?

[Arnold Smeulders, PAMI, 2000]The semantic gap is the lack of coincidence between the information that one can extract from the sensory data and the interpretation that the same data has for a user in a given situation

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 15

The science of labeling

Automatically detecting the presence of a concept in a video stream

Naming visual information

airplane

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 16

A Simple Concept Detector

[Cees Snoek and Marcel Worring, SSMS, 2007]

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 17

Support Vector Machine

[Cees Snoek and Marcel Worring, SSMS, 2007]

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 18

The Computer Vision Approach

Building detectors one-at-the-time

a face detector for frontal faces

a face detector for non-frontal faces

3 years later

One (or more) PhD for every new concept

long waya little drop of semantics goes a

Jim Hendler [1997]

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 20

Once upon a time …

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 21

… leading to sharing Media Fragments Publishing status message containing

a Media Fragment URIUse a ‘#’ !Highlight a

videosequence

Highlight a regionto pay attention to

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 22

W3C Video on the Web Workshop - 2007

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 23

Key topics

Addressing: having global identifiers for identifying spatial and temporal clips (for deep linking, bookmarking, caching and indexing)

Metadata: searching and discovering video is difficult with the volume of online video

Video codec: recommending a baseline (open) video codec for the World Wide Web

Content protection: managing digital rights associated with the media is key: W3C should look into metadata for digital rights

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 24

Making video a "first class citizen"

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 25

Flickr Notes

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 26

http://www.flickr.com/photos/mhausenblas/2883727293/

YouTube Temporal Addressing (Sept 2008)

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 27

Media Fragments Use Cases

Bookmark / Share parts (fragments) of audio/video content

Annotate media fragments

Search for media fragments

Develop Mash-ups/Collage

Conserve bandwidth

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 28

http://www.w3.org/TR/media-frags-reqs/

t0 20 35temporal media fragment

spatial media fragment

track media fragment

named media fragment“Scared Scene”

What are Media Fragments?

17/12/2013 - - 29 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

Media Fragments Dimensions

r01: Temporal fragments: a clipping along the time dimension from a start to an end

time that are within the duration of the media resource

r02: Spatial fragments: a clipping of an image region, only consider rectangular

regions

r03: Track fragments: a track as exposed by a container format of the media

resource

r04: Named fragments: A temporal media fragment that has been given a name

through some sort of annotation mechanism

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 30

Media Fragments (temporal)

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 31

Fragment beginning Fragment endPlayback progress

Original resource length

Media Fragments (spatial)

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 32

semi-opaque overlay

highlighted fragment

http://ninsuna.elis.ugent.be/MFPlayer/html5

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 33

Media Fragment (Semantic) Annotation

Media Fragment creation: localize a region (person)

Media Fragment annotation (tagging) = interpretationWinston Churchill, UK Prime Minister, Allied Forces, WWII

Media Fragment semantic annotation:Reg1 foaf:depicts dbpedia:WinstonChurchill.

dbpedia:Churchill rdfs:label "Winston Churchill"; rdf:type foaf:Person dbprop:order dbpedia:Prime_Minister_(UK).

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 34

The "Big Three" at the Yalta Conference (Wikipedia)

Reg1

Media Fragment (Semantic) Annotation

Media Fragment creation: localize a temporal sequence

Media Fragment annotation (tagging) = interpretationG8 Summit, EU Summit, Heiligendamm, 2007, Gothenburg, 2001

Media Fragment semantic annotation:Seq1 foaf:depicts dbpedia:33rd_G8_Summit.:Seq4 foaf:depicts dbpedia:EU_Summit.

dbpedia:33rd_G8_Summit rdfs:label "33rd G8 summit"@en ; grs:point "54.143055555555556 11.841666666666667".

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 35

A history of G8 violence (video) (© Reuters) Seq1

Seq4

Things, not strings!http://googleblog.blogspot.fr/2012/05/introducing-knowledge-graph-things-not.html

Use knowledge bases (LOD)

Use commonvocabularies (LOV)

Follow the 4 Linked Data principles

Refine the 4 Linked Media principles

- 3617/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

Media Fragment Semantic Annotation

Open Annotation Data Model

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 37

Specification developed in the W3C Open Annotation Community Grouphttp://www.openannotation.org/spec/core/

Core model OWL vocabulary for representing

and sharing annotation of digital resources (and their fragment) … in RDF

A body is related to a target Nature of the annotation changes

according to intention (motivation)

How to annotatethis image?

Semantic Annotation of an Image

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 38

http://www.w3.org/community/openannotation/wiki/SE_Semantically_Tagging_an_Image

Maphub: http://maphub.github.io/

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 39

Open Video: Annotation Project

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 40

http://openvideoannotation.org/

LinkedTV: automatic annotations ...

17/12/2013 - - 41 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

... and enrichment for hypervideos

CubismExpressionism

Fauvism

FACETS / PROPERTIES OF CONCEPT

CONCEPT IN PLAYER

CONTENT ENRICHMENT

17/12/2013 - - 42 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

Media Fragments and Annotations

nerd:Location Cafe Rick

nerd:PersonH. Bogart

nerd:PersonI. Bergman

nerd:Location Casablanca

Media Fragment URI 1.0 Chapters Scenes Shots etc…

http://data.linkedtv.eu/media/e2899e7f#t=840,900

17/12/2013 - - 43

7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)

Enrichment and Hypervideos

nerd:Location Cafe Rick

nerd:PersonH. Bogart

nerd:PersonI. Bergman

nerd:Location Casablanca

Nerd:PersonE. Tierney

nerd:Location China

17/12/2013 - - 44

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 45

Compare performances of NER and NEL tools Understand strengths and weaknesses of different Web APIs Adapt NER processing to different context

(Learn how to) Combine NER (/ NEL) tools

NERD: Named Entity Recognition and Disambiguation

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 46

What is NERD?REST API2ontology1

UI3

1 http://nerd.eurecom.fr/ontology2 http://nerd.eurecom.fr/api/application.wadl

3 http://nerd.eurecom.fr

NERD User Interface

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 47

Locator

MediaResource

MediaFragmentAnnotation

Entity

URL (hyperlink)

Type

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 48

Media Fragment + Open Annotation + NERD

Media Fragment Enricher: http://mfe.synote.org/mfe/

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 49

Linking pieces of knowledge

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 50

Linking pieces of knowledge

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 51

7ème Entretiens du Nouveau Monde Industriel (ENMI 2013)17/12/2013 -

http://linkedtv.project.cwi.nl/news/

- 52

Take Away Summary

Video is a first class citizen on the WebAnnotations: Ontology and API for Media

Resources, Open Annotation Data Model

Access: Media Fragments URINERD platform for extracting key information from

textual resources including video subtitles and microposts

Embrace the Linked Media visionPublish, re-use, re-purpose and remix media

descriptionsDevelop links between (part of) media items via

their descriptions17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 53

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 54

Take Away Summary

Credits

Giuseppe Rizzo, Vuk Milicic, José Luis Redondo Garcia (EURECOM)

Thomas Steiner (Google Inc.), Yunjia Li (University of Southampton)

Marieke van Erp (Free University of Amsterdam)

Erik Mannens, Davy ven Deursen (iMinds, Uni. Ghent)

Paolo Ciccarese, Robert Sanderson, Herbert Van de Sompel and all the members of the W3C Open Annotation Community Group

… and many other students

17/12/2013 - 7ème Entretiens du Nouveau Monde Industriel (ENMI 2013) - 55