creating media stories, sssw july 13
Post on 20-Sep-2014
12 views
DESCRIPTION
Automatically creating coherent multimedia presentations based on different types of annotations. Examples from CeWe photobook and Vox Populi video sequence generation. Annotation types include discourse, rhetorical, interviewee, film continuity.TRANSCRIPT
Creating stories with media
Lynda Hardmanhttp://www.cwi.nl/~lynda
CWI, Information AccessUvA, Institute for Informatics
http://www.flickr.com/photos/iboy/4528401870/
Video collection:What do you think of war as a
solution?
2http://www.flickr.com/photos/soldiersmediacenter/1183836576
Interactive Media Access
• Users need support for– finding desired content– in one or more media types– for their specific task
• We need to be aware that there is more than the information “expressed'' by the media asset itself, e.g.– when/where the media was captured– the intended purpose of the creator– the context in which the media
asset was created4
We need to enable • the processing of information-
bearing content• of one or more media types• that can be interpreted by end users
End-users are primarily interested in • the meaning conveyed by a
combination of media assets• interacting further with the media
– as part of complex “search” task– passing it on to someone else in
media “chain”
We don’t care about the media!
5http://www.flickr.com/photos/benheine/4687572408/
How can we get this to work?
We need mechanisms– for identifying (part of) an
individual media asset– for associating metadata
with an identified fragment– for agreeing on the
meaning of metadata– that enable larger
meaningful structures to be composed, identified and annotated
6http://www.flickr.com/photos/jannem/3312115991
Outline of talk
• Information processes in which media and metadata play a role– “canonical processes of media production”
• Vox Populi system, demonstrating high level user interaction enabled by media and metadata
• Study of information needs for videos that are able to support processes of opinion-forming and decision-making
7
Workflow for Multimedia Applications
• Identify and define a number of canonical processes of media production
• Community effort
8
– 2005: Dagstuhl seminar– 2005: ACM MM Workshop on
Multimedia for Human Communication
– 2008: Multimedia Systems Journal Special Issue(core model and companion system papers)editors: Frank Nack, Zeljko Obrenovic and Lynda Hardman
http://link.springer.com/journal/530/14/6/
Canonical Reduced to the simplest and most significant form possible, without loss of generality
Example application areas
• multimedia feature extraction systems• professional news production systems (VRT)• new media art• hyper-video production• photo book production (CeWe)• ambient multimedia systems with complex
sensory networks
Overview of Canonical Processes
11
Example: CeWe Color PhotoBook
• Application for authoring digital photo books• Automatic selection, sorting and ordering of photos
– Context analysis methods: e.g., timestamp, annotation
– Content analysis methods:e.g., color histograms, edge detection
• Customized layout and background
12http://www.cewe-photobook.com
CeWe Color PhotoBook Processes
My winter ski holidays with my friends
13
Premeditate
Construct Message
CeWe Color PhotoBook Processes
• Media assets are captured, generated or transformed
Create
CeWe Color PhotoBook Processes
15
Annotate
CeWe Color PhotoBook Processes
16
CeWe Color PhotoBook Processes
17
Organize
Query
landscape
portrait
18
Organise using domain annotations
skiing holiday
Monday
Wednesday
Monday
Tuesday
Tuesday
CeWe Color PhotoBook Processes
19
Publish
Distribute
CeWe Color PhotoBook Processes
20
Canonical Processes
21http://link.springer.com/journal/530/14/6/
Creating Stories with Media
• Long term goal to find and present information to end-users– In a way that is useful to them
• We understand how to design information interfaces by hand. – How can metadata help us in giving more flexible access to
media collections?• We can link media assets to existing linked data, and
use this to improve presentation, e.g. by– Selecting a sub-set– Grouping, ordering and linking media assets– Influencing the (visual) presentation
22
How can annotations help?
What can be expressed explicitly?– the message to be conveyed– objects that are depicted in a media asset– domain information (e.g., art, painter)– human communication roles (discourse)
What can they be used for?– disambiguating query terms– grouping similar items for conveying topic breadth– visualizing items for presentation, e.g. timeline, map– finding similar items– …
23
Vox Populi: Generating video documentaries from annotated media repositories
Stefano Bocconi, Frank Nack (CWI, Amsterdam)
Video Documentaries on the Web
• Traditional video authoring: there is only one final version, what is shown is the choice of the author/editor
• Proposed video sequence creation:– Annotate the video material– Show automatically what the user asks to see,
using presentation forms a film editor would use
25
Video material
• Interview with America video footage with interviews and background material about the opinion of American people after 9-11 www.interviewwithamerica.com
• Filming 27-10-2001 to 01-11-2001 in Stamford (CT), New York (NY), Boston (MA) and Cleveland (OH)
26
Premeditate
Example: What do you think of the war in Afghanistan?
“I am never a fan of military action, in the big picture I
don’t think it is ever a good thing, but I think there are circumstances in which I certainly can’t think of a more effective way to counter this sort of thing...”
27http://www.flickr.com/photos/soldiersmediacenter/1183836576
The annotations
Rhetorical– Rhetorical Statement
(mostly verbal, but visual also possible)– Argumentation model: Toulmin model
Descriptive– Question asked– Interviewee (social)– Filmic next slide
28
Annotate
Filmic annotations
Continuity, e.g.– camera movement
none, pan left/right, shaking, tilt up/down, zoom in/out– framing continuity
close-up, medium shot, long shot– gaze direction of speaker
left, centre, right– lighting conditions– background sound
29
Continuity, e.g.– camera movement
none, pan left/right, shaking, tilt up/down, zoom in/out– framing continuity
close-up, medium shot, long shot– gaze direction of speaker
left, centre, right– lighting conditions– background sound
Annotate
Statement encoding
3-part statement:– <subject> <modifier> <predicate>– E.g. “war best solution”
Thesaurus (pre Wordnet) containing: – Terms (155)– Relations between terms:
similar (72), opposite (108), generalization (10), specialization (10)e.g. war opposite diplomacy
30
Annotate3-part statement:– <subject> <modifier> <predicate>– E.g. “war best solution”
Thesaurus (pre Wordnet) containing: – Terms (155)– Relations between terms:
similar (72), opposite (108), generalization (10), specialization (10)e.g. war opposite diplomacy
Connect statements
• Using the thesaurus, generate related statements and query the repository“war best solution”,“diplomacy best solution”,“war not solution”
• Create a graph of related statements– nodes are the statements
(corresponding to video segments)– edges are either support or contradict
31
Statement Graph
32
= support= contradict
war best solution
war not solution
diplomacy best solution
Toulmin model
33
ClaimData
Qualifier
Warrant
Backing
Condition
Concession
57 Claims, 16 Data, 4 Concessions, 3 Warrants, 1 Condition
Toulmin in example
34
Claim
Concession
Claim contradict
supportClaim
I am not a fanof military action
War has never solved anything
Two billions dollar bombs on tents
I cannot think of amore effective solution
weaken
Annotate
What do you think of the war in Afghanistan?
35
I am not a fan of military actions
War has never solvedanything
I cannot think of a more effective solution
Two billionsdollar bombson tents
Vox Populi interface
36
Query
Construct Message
Vox Populi Processes
Vox Populi Conclusions
• Automatic generation of video interviews augmented with supporting and/or contradicting material
• The user can determine the subject and the bias of the presentation
• The documentarist can add material and let the system generate new documentaries
38
Pointer & Acknowledgments
• More on VoxPopuli at:http://www.cwi.nl/~media/demo/VoxPopuli/
• VoxPopuli was funded by the Dutch national ToKeN I2RP and CHIME projects
39
User information needs forenvironmental opinion-forming and decision-
making inlinked-enriched video
Ana Carina PalumboUniversity of Amsterdam
Technical University of Eindhoven
The message
• Inform citizens about environmental issues• Scenario: users form opinions while watching
videos (from TV or internet)• Goal: specify information that should be
captured in annotations
Method
• Expert interviews– environmental governance– video production and broadcasting
• User survey– information users say they need– 215 participants
• User experiment– information users really select– 6 participants
High-level overview of results
Environmental video conclusions
• We know what to annotate in environmental videos to support opinion-forming
• Some annotations can be done automatically (e.g. subjects and concepts, location)
• Others need to be manually annotated• Challenges:
– level of objectivity and subjectivity– trustworthiness of sources.
EuroITV '13
What are my messages?
• Annotations associated with media assets can be used for different stages of interactive access, not just searching
• Annotations can be added by hand, linked automatically or automatically extracted
• The intended message can be made explicit (more annotations)
• Media content and associated annotations can be passed among systems
• We need community agreement on how to do this(e.g. canonical processes)
• Users can be given much richer and more flexible access to(annotated) media content, but…
• we need to store annotations & media in a reusable way45
45
Acknowledgements
46
Raphaël Troncy
Ana CarinaPalumbo
Jacco van Ossenbruggen
Stefano Bocconi
Frank Nack
Andre Fialho
47
Literature• Special Issue on Canonical Processes of Media Production
http://link.springer.com/journal/530/14/6/http://www.cwi.nl/~media/projects/canonical/
• Lynda Hardman, Zeljko Obrenovic, Frank Nack, Brigitte Kerhervé and Kurt Piersol: Canonical Processes of Semantically Annotated Media Production. Multimedia Systems Journal, 14(6), 2008
• Philipp Sandhaus, Sabine Thieme and Susanne Boll: Canonical Processes in Photo Book Production. Multimedia Systems Journal, 14(6), 2008
• Stefano Bocconi, Frank Nack and Lynda Hardman: Automatic generation of matter-of-opinion video documentaries.Journal of Web Semantics, 6(2), p139-150, 2008.
• Ana Carina Palumbo and Lynda Hardman: User information needs for environmental opinion-forming and decision-making in link-enriched videoIn EuroITV '13, pp 85-88