convenient discovery of archived video using audiovisual hyperlinking

Convenient Discovery of Archived Video Using Audiovisual Hyperlinking

Convenient Discovery of Archived Video Using Audiovisual Hyperlinking

Roeland J.F. OrdelmanUniversity of Twente &Netherlands Institute for Sound and VisionThe NetherlandsRobin AlyData ManagementUniversity of TwenteThe Netherlands

Maria EskevichEURECOMSophia AntipolisFrance

Benoit HuetEURECOMSophia AntipolisFrance

Gareth J.F. JonesADAPT Centre / CNGLSchool of ComputingDublin City University, Ireland

Audio-Visual ExplosionEU alone hosts 500+ online video platforms42.7m hrs of footage in online archives of broadcasters and producers (61% of archive footage is online)UGC content is soaring: YouTube receives 72 hrs of video/minuteVine and Instagram video messagingInternet video will reach 62 percent by the end of 2015, 75% in 2017(source: CISCO)How to make the content accessible?

01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking2VISUAL WEB[R. Jain 2015]

Typical Research Approach

small data setsynthetic data setsmall data setsmall data set1 class data set1 class data set1 class data set1 class data setso called large scaledata set01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking3

The BIG Data VsNot Video butVOLUME, VARIETY, VELOCITY+ Veracity, + Value, + Variability, + Visualization

01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking4

New technologies to stimulate active appropriation of multimedia contentUser Information NeedExpectationInterestDesireSerendipityStory Telling

Content Accessibility


Content AccessibilityVideo search system evaluation resultsBBC and NISVUsers:Dont know footage availabilityUsually start their search withCelebrityLocation (home town,)Personal Interest (hubby, sport, etc)AND THEN what else might there be in the archive?01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking6

Content Recommendation


Archive Exploration/Browsing

Anchor


YouTube HyperLinking ?


Video HyperLinking Example





... the queen...


Usage ScenarioExploration of additional information (video) sources while accessing content in a linear fashionExploration of an audiovisual archive via a structure of linked video segmentsCreating narratives on the basis of linked video segmentsPersonalisation01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking13

3 types of HyperLinksInside-in (Video to Video Linking)Outside-in(Media to Video)Inside-out(Video to Media)


14

Automatic Video HyperLinking Process01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking15

Anchor IdentificationKey issues:multimodality: anchor can be in speech/audio, visual or both.. Other?unfamiliarity: user evaluations demonstrate that anchors in video are a difficult conceptEvaluation perspective:ask (professional) users to select anchors let task participants automatically identify anchors


Anchor RepresentationFocus on segments: start-time and end-time (media fragment)Extract multimodal featuresContextAnchor representation may be noisy due to analysis errors


Target searchSearch for relevant link targetsWhat is relevant?Working hypothesis:Content about what is represented in the anchor (topically related) context is importantContent that is based upon it, similar, or has identical semantic labels context is less important


content that is about what isrepresented in the anchor { we sometimes refer to this as'topically related' { and not content that is based upon it,which is similar to it, or has identical semantic labels.18

Target presentationA ranked list of search results for each anchor?Depends on scenario, not addressed in current evaluation set-upFor assessment of results we provide assessors with anchor target pairs01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking19

2015 Evaluation CampaignsMediaEval - Search and Anchoring in Video Archives (SAVA) video search using multimodal features (continued)automatic anchor selection (new)

TRECVid Video Hyperlinkingautomatic video-to-video linking


Anchor Identification - Task scenarioPosition yourself in the role of a producer wanting to create a new production, e.g., a news item, report or documentaryS/he is searching for content in the BBC archive for this production and selects clipsImagine that the producer wants to place hyperlinks in the clips that help the end-user that watches the final program better to understand the program or enrich their watching experienceImagine that these links are provided to end-users for example via a second screen (e.g., iPad)01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking21

Topic Description

Describe what you are looking (e.g., I am looking for clips with castles and medieval villages)Provide keywords that could be in the speech of relevant clips (e.g., middle ages, doomsday)Provide keywords for visual content (e.g., castle, bridge, knight)


Search

Here is what you put earlier. You can change description, queries if neededYou can check if a result is relevant for youIf you want to have this clip click the buttonClips you collect will end up here. Provide at least 2 clips01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking23

Anchor Creation

Title of anchor (e.g., Castle)To what would you want to link this (e.g., a documentary on this castle)Was the anchor something visual or in the speech?01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking24


User Study - FindingsAnchor Modality SourceSpoken ContentWhole SceneWhat about Visual Content?

Anchor creation is intention drivenContent producer VS end-user/viewerVS advertisers01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking26

Entity-Based Anchor IdentificationContent Producer use-case for Video HyperlinkingRich interactive TV experienceKeeping the Editor in the loop while automating much of the processEditor ToolsAnchors / Targetshttp://editortool.linkedtv.eu/trial


Outside-In LinkingUsing external content to link into the video archiveStimulate discovery and re-useMultimodal Analysis of the archive speech transcript, visual analysisMetadataMatching with Named Entities from RSS news feeds01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking28


End User EvaluationInterest But Unsatisfactory quality of the hyperlinksLimiting factors:Fine-tuning of the structured query formulation Dataset sparseness (limited to 3000 hours)


Inside-Out LinkingVideo content links to external informationProvide enrichmentMultimodal Analysis of Audio-Visual Broadcastspeech transcript, (visual analysis)Broadcast Metadata (closed caption)Extract Named Entities from the text and use Semantic Web technologies to identify relevant structured content


The concept of Linked Television

meet the viewers information needdirectly associated with the TV program easily accessible for the viewerunder the control of the broadcaster



https://vimeo.com/119107849

ConclusionsInsights on Audio-Visual HyperlinkingGrowing interest from both the research community and industryBenchmark EvaluationWhat NEXT?Collaborations between fields (audio, visual, nlp, semantic web, social media, big data)Intent (anchor and hyperlink level)


Thank you

Special thanks to Jana Eggink and Andy ODwyer

Any questions?

TRECVid (NIST)01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking35

ReferencesR. Aly, K. McGuinness, M. Kleppe, R. Ordelman, N. E. O'Connor, and F. de Jong. Link anchors in images: Is there truth? In Proceedings of the 12th Dutch Belgian Information Retrieval Workshop (DIR 2012), pages 1{4,Ghent, 2012. University Ghent.R. Aly, R. Ordelman, M. Eskevich, G. J. F. Jones, and S. Chen. Linking inside a video collection - what and how to measure? In Proceedings of the 22nd International Conference on World Wide Web Companion, IW3C2 2013, Rio de Janeiro, Brazil, pages 457-460, Brazil, May 2013. ACM.E. Apostolidis, V. Mezaris, M. Sahuguet, B. Huet, B. Cervenkova, D. Stein, S. Eickeler, J. L. Redondo Garcia, R. Troncy, and L. Pikora. Automatic fine-grained hyperlinking of videos within a closed collection using scene segmentation. In ACMMM 2014, 22nd ACM International Conference on Multimedia, Orlando, USA, 11 2014.J. Blom. Deliverable 1.5, linkedtv annotation tool, final release. Public deliverable, LinkedTV Project (FP7-ICT grant agreement no 287911), 2015.M. Bron, B. Huurnink, and M. de Rijke. Linking archives using document enrichment and term selection. In S. Gradmann, F. Borri, C. Meghini, and H. Schuldt, editors, Research and Advanced Technology for Digital Libraries, volume 6966 of Lecture Notes in Computer Science, pages 360-371. Springer Berlin Heidelberg, 2011.L. S. Connaway, T. J. Dickey, and M. L. Radford. "If it is too inconvenient I'm not going after it": Convenience as a critical factor in information-seeking behaviors. Library & Information Science Research, 33(3):179-190, 2011.M. Eskevich, H. Nguyen, M. Sahuguet, and B. Huet. Hypervideo browser: Search and hyperlinking in broadcast media. In ACMMM 2015, 23nd ACM International Conference on Multimedia.P. E. Hart and J. Graham. Query-free information retrieval. IEEE Intelligent Systems, (5):32-37, 1997.M. Kleppe and J. Briggeman. Deliverable 1.8, final use case evaluation report. Public deliverable, AXES Project (FP7-ICT grant agreement no 269980), 2015.R. Mihalcea and A. Csomai. Wikify!: Linking Documents to Encyclopedic Knowledge. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management (CIKM '07), pages 233-242, 2007.J. Morang, R. J. F. Ordelman, F. M. G. de Jong, and A. J. van Hessen. InfoLink: analysis of Dutch broadcast news and cross-media browsing. In Proceedings of IEEE International Conference on Multimedia and Expo (ICME 2005), pages 1582-1585, Los Alamitos, 2005. IEEE Computer Society.D. W. Oard, A. S. Levi, R. L. Punzalan, and R. Warren. Bridging communities of practice: Emerging technologie for content-centered linking. In Museums and the Web, 2014.D. Stein, E. Apostolidis, V. Mezaris, N. de Abreu Pereira, J. Muller, M. Sahuguet, B. Huet, and I. Lasek. Enrichment of news show videos with multimodal semi-automatic Analysis. In NEM-Summit 2012, Networked and Electronic Media, Istanbul, Turkey, 10 2012.D. Stein, A. Oktem, E. Apostolidis, V. Mezaris, J. L. Redondo Garca, R. Troncy, M. Sahuguet, and B. Huet. From raw data to semantically enriched hyperlinking: Recent advances in the LinkedTV analysis workow. In EM Summit 2013, Networked & Electronic Media, Nantes, France, 10 2013.P. Stockinger. Audiovisual Archives. John Wiley & Sons, Inc., 2013.T. Tommasi and R. Aly and K. McGuinness and K. Chateld and R. Arandjelovic and O. Parkhi and R. Ordelman and A. Zisserman and T. Tuytelaars. Beyond metadata: searching your archive based on its audio-visual Content. In IBC 2014, Amsterdam, The Netherlands, 2014.01/11/2015SLAM2015 - Convenient Discovery of Archived Video Using Audiovisual Hyperlinking36