a study of awareness in multimedia search
DESCRIPTION
A Study of Awareness in Multimedia Search. Robert Villa, Nick Gildea, Joemon Jose Information Retrieval Group April 2008. Overview. Introduction Collaboration and awareness in search Research questions Experimental study Inducing awareness: a game scenario Video retrieval - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/1.jpg)
A Study of Awareness in Multimedia Search
Robert Villa, Nick Gildea, Joemon Jose
Information Retrieval Group
April 2008
![Page 2: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/2.jpg)
Overview
• Introduction• Collaboration and awareness in search• Research questions• Experimental study
– Inducing awareness: a game scenario– Video retrieval – Demo of multimedia retrieval system– Some results
• Conclusions
![Page 3: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/3.jpg)
Information Retrieval
• Deals with practical and theoretical models of searching unstructured collections of documents– Idealised aim: supply the system with a
natural language description of your need– The system returns a ranked list of the
documents relevant to your need• Process is naturally probabilistic and
uncertain
![Page 4: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/4.jpg)
Video retrieval
• Video retrieval systems index and search collections of videos– Like traditional IR, the indexing of the video
data is assumed to be automatic• Extraction of visual or audio features• Use of automatic speech recognition
– Queries are typically textual or by example
![Page 5: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/5.jpg)
Example interface
![Page 6: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/6.jpg)
Video retrieval
• Videos are automatically split into ‘shots’ (shot segmentation)– Shot boundaries are determined using the
visual content of video frames– Each shot is a short element of a video
• in TRECVID 2006, typically 2 to 3 seconds, although there are some much longer shots
• Shots are the element of retrieval – Where a text system retrieves documents,
video retrieval systems retrieve shots
![Page 7: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/7.jpg)
Example of a shot
• Every shot has an associated text transcript: • E.g. “A dramatic arrival”
• Generated by Automatic Speech Recognition (ASR)• Transcript can often be very wrong
![Page 8: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/8.jpg)
Collaborative Retrieval
• Most current search systems assume searching is a solitary activity– Is this always the case, or can
collaborative searching with one or more others be effective?
• Rather than focus on collaboration in general, we decided to look at only one aspect of collaboration - awareness
![Page 9: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/9.jpg)
Awareness
• Awareness enables an “understanding of the activities of others”, an important aspect of collaboration
• Paul Dourish and Victoria Bellotti. “Awareness and Coordination in Shared Workspaces”, CSCW'92
• Scenario:– Two users are searching on the same task
at the same time in different places– Synchronous and remote
![Page 10: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/10.jpg)
Previous work – collaborative search
• Cerchiamo (FXPAL, Xerox)• Adcock et. al., in TRECVID 2007• Two people collaborating, one a “gatherer” and the
other a “reviewer”
• SearchTogether (Microsoft)• Morris, M. R. (2007)• Provides a messaging system, recommending a
web page to another, query awareness, etc.
• Fischlar-DiamondTouch (DCU)• Smeaton et. al. (2007) • Table-top display which allows two people to work
around it
![Page 11: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/11.jpg)
Research question
• Can awareness of another searcher aid a user when carrying out a multimedia search?– Will their performance increase?– Will less effort be needed to reach a given
performance?• Shots played, browsing required
– Will the user’s search behaviour change?• Number of queries executed, shots found
independently
![Page 12: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/12.jpg)
Competitive game scenario
• We wanted to evaluate the effect of awareness in a “best case” scenario– i.e. a situation where there was some
benefit to users in being aware of another’s actions
• A competitive game scenario was used, where pairs of users competed to “win” the search tasks
![Page 13: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/13.jpg)
Aim of the ‘game’
• The aim of the ‘game’ was to find as many relevant shots as possible for the task– Domain was video retrieval, where users
had to search a video collection for ‘shots’– Whoever finds the most shots ‘wins’– A monetary award was given to the winner
![Page 14: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/14.jpg)
System
• Our existing video retrieval system was modified to allow collaboration– Each user could be given a view of the
other user’s search screen– This was designed to work with two
monitors:• The user’s own search interface on one screen• The other screen optionally showing the other
user’s search screen
– We supported 4 different situations
![Page 15: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/15.jpg)
User A User B“Mutually Aware”
A can see B’s screen and B can see A’s screen
![Page 16: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/16.jpg)
User A User B“A aware of B”
A can watch B’s screen while B cannot watch A
![Page 17: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/17.jpg)
User A User B“B aware of A”
B can watch A’s screen while A cannot watch B
![Page 18: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/18.jpg)
User A User B“Independent”
Both A and B cannot watch each other
![Page 19: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/19.jpg)
System interface
![Page 20: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/20.jpg)
Local search interface
![Page 21: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/21.jpg)
Text Query
![Page 22: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/22.jpg)
Search results
![Page 23: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/23.jpg)
Shots to use in relevancefeedback
![Page 24: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/24.jpg)
Result shots for the user
![Page 25: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/25.jpg)
Remote search interface
![Page 26: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/26.jpg)
User cannot see the other user’sfinal results –
only a count of the number of shot currentlymarked by the
user
![Page 27: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/27.jpg)
This screen doesnot update
automatically, theuser must press
the “Refresh”button to update
the screen
![Page 28: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/28.jpg)
Video browser
• Simple video browser pop’s up when the user clicks a keyframe
• Allows the user to view the shot, and move backwards and forwards in the video
![Page 29: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/29.jpg)
Conditions
• From the point of view of an individual user:– Working independently– Cannot watch the other user, and knows
that the other user can watch him/her– Can watch the other user, and knows that
the other user cannot watch him/her– Can watch the other user, and knows that
the other user can watch him/her
![Page 30: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/30.jpg)
TRECVID 2006 Collection
• Almost 260 hours of mostly news data from the end of 2005– CNN, LBC, CCN, etc. – Multilingual (English, Chinese and Arabic)
• Has a standard shot segmentation• ASR transcripts provided
– For Chinese and Arabic video, also automatically translated into English
![Page 31: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/31.jpg)
TRECVID 2006 Topics
• 24 topics, of which we used the 4 worst performing overall from the interactive track– Hoped that these would be a similar
challenge for the user– Adcock et al. (2007) found that users
collaborated better on difficult tasks
![Page 32: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/32.jpg)
Topics
TopicMedian
MAPTopic description
0189 0.038Find shots of a group including least four people dressed in suits, seated, and with at least one flag
0173 0.037Finds shots with one or more emergency vehicles in motion (e.g., ambulance, police car, fire truck, etc.)
0175 0.034Find shots with one or more people leaving or entering a vehicle
0192 0.030Find shots of a greeting by at least one kiss on the cheek
![Page 33: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/33.jpg)
Experimental design
• Within user study was carried out• Latin square design
– 4 tasks– 4 conditions– 24 users (12 pairs)
![Page 34: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/34.jpg)
Procedure
• Users took part in pairs• Users had 15 minutes to find as many
shots as possible• At the end of the 4 tasks, the “winner”
was announced – Each user was paid £10 – Winner got an extra £5, shared if there was
a draw
![Page 35: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/35.jpg)
Results
• 12 competitive runs– 11 wins and 1 draw
• And there was an immediate issue with one of the user’s ...
![Page 36: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/36.jpg)
![Page 37: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/37.jpg)
Independent Watched Watching Mutual
MAP
Mean 0.0163 0.0199 0.0222 0.0243
SD 0.0150 0.0163 0.0165 0.0204
Precision at 10 shots
Mean 0.3083 0.3750 0.4167 0.4667
SD 0.2569 0.2996 0.3158 0.3332
Search performance
![Page 38: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/38.jpg)
Search Performance
• No significant difference found between the level of performance in the four different conditions– Overall performance was very low (typical
in video IR, for these hard topics)– Performance does vary widely across the
four tasks • Tasks 189 and 192 performed worst
![Page 39: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/39.jpg)
Search behaviour: queries
Ind Watched Watching Mutual
Total Queries 603 501 473 570
Queries per task
Mean(SD)
25.13
(14.72)
20.88
(13.61)
19.71
(9.34)
23.75
(13.35)
• Do users execute more queries searching alone?
• Significant variation between watching and Independent was found
![Page 40: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/40.jpg)
Number of shots found independently
Ind Watched Watching Mutual
Total shots found 155 188 244 222
Shots found/task
Mean (SD)
6.46
(4.11)
7.83
(7.17)
10.17
(9.31)
9.25
(8.07)
• Significant interaction was found between Independent and Watching
![Page 41: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/41.jpg)
Changes in search behaviour
• Users searched less when watching someone else– Also less searching executed in the
watched condition (not significant)• Users found more shots themselves
when watching someone else– Also found more shorts in the mutual and
watched conditions, but not significant
![Page 42: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/42.jpg)
Search terms used
• One possible way awareness may help is by providing a user with new terms with which to use in queries
• Did users copy search terms from the remote user?– We could not directly record this in the logs
(terms are easily retyped)
![Page 43: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/43.jpg)
Estimating copied search terms
• Search terms which could have been copied were derived from the logs
• Method:– Found the set of common terms– Found who used that term first– Checked for a click of the “refresh” button
by the user who was second– Assumed that second user could have then
copied that term
![Page 44: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/44.jpg)
Copied terms
Watching Mutual
Total unique terms 355 388
Total terms copied 44 40
% terms copied %12 %10
• Suggests that a user is able to reuse search terms used by the other user
![Page 45: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/45.jpg)
Searcher effortIndependent Watched Watching Mutual
Play events per search task
Mean 187.08 192.21 170.08 184.79
SD 116.51 112.91 101.28 117.52
Next shot in video events per search task
Mean 115.04 121.08 97.04 104.13
SD 106.64 99.23 96.35 107.01
Previous shot in video events per search task
Mean 16.88 20.71 12.33 12.29
SD 23.90 22.20 18.77 15.86
![Page 46: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/46.jpg)
Searcher effort• Recorded three types of events to gauge searcher
effort– Play events when a user clicks a shot– Move to next shot in video– Move to previous shot in video
• Only significant relationship:– Watching and Watched and move to previous shot
![Page 47: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/47.jpg)
Where did a user’s final results come from?
• From the interface, we logged the user dragging and dropping shots between the different parts of the interface– We could record when someone copied a
shot from the other user• Using this, we can estimate where
user’s got their final results– (roughly!)
![Page 48: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/48.jpg)
![Page 49: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/49.jpg)
Conclusions
• Despite the game scenario, users didn’t copy other people’s shots much– This came as something as a surprise
• There’s no significant increase in a user’s performance– Only a trend ...
• There is evidence that user’s do reuse search terms – 10 and 13% of terms are potentially copied
![Page 50: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/50.jpg)
Conclusions
• Results from user effort were unclear– Only significantly less interaction in one
event
![Page 51: A Study of Awareness in Multimedia Search](https://reader036.vdocuments.us/reader036/viewer/2022070403/568139a4550346895da140d9/html5/thumbnails/51.jpg)
The End