![Page 1: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/1.jpg)
Predictive Video RetrievalA Matter of Trust
Bouke Huurnink
MediaMill
![Page 2: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/2.jpg)
The TeamBouke Huurnink Jiyin He Koen van de SandeOrk de RooijCees SnoekMaarten de RijkeJan van GemertJasper UijlingsXirong LiIvo EvertsVladimir Nedovic
Michiel van LiemptRichard van Balen
FeiYanMuhammad Tahir
Krystian MikolajczykJosef Kittler
Jan-Mark GeusebroekTheo Gevers
Marcel WorringArnold Smeulders
Dennis Koelma
![Page 3: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/3.jpg)
Come see our interactive search demo
0
0.05
0.10
0.15
0.20
0
0.05
0.10
0.15
0.20
Now with (inter)active learning!Presented by Ork de Rooij
UvA
![Page 4: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/4.jpg)
Why predictive video retrieval?
• Video retrieval is multichannel problem:• Speech• Detectors• Examples
• Observations• Speech works for named entity topics• Detectors work when closely related to topic• Examples can also work pretty well
• We want to exploit this knowledge
![Page 5: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/5.jpg)
Idea
• Predict which type of search - retrieval channel - we can trust for a topic
• Rerank results from this channel with secondary result information
![Page 6: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/6.jpg)
Outline
• System description
• Result overview
• Analysis
• Conclusion
![Page 7: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/7.jpg)
Our predictive system
Retrieval Channels
Speech Search
Detector Search
Example Search
Predict Trusted Channel
Reranking
Final Results
Information NeedFind shots of pieces
of paper with writing, typing, or printing,
filling more than half of the frame area.
Result Lists
Trusted Results
- Detector results
Secondary Results
- Speech results
Secondary Results
- Example results
![Page 8: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/8.jpg)
Our predictive system
Retrieval Channels
Speech Search
Detector Search
Example Search
Predict Trusted Channel
Reranking
Final Results
Information NeedFind shots of pieces
of paper with writing, typing, or printing,
filling more than half of the frame area.
Result Lists
Trusted Results
- Detector results
Secondary Results
- Speech results
Secondary Results
- Example results
![Page 9: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/9.jpg)
Our predictive system
Retrieval Channels
Speech Search
Detector Search
Example Search
Predict Trusted Channel
Reranking
Final Results
Information NeedFind shots of pieces
of paper with writing, typing, or printing,
filling more than half of the frame area.
Result Lists
Trusted Results
- Detector results
Secondary Results
- Speech results
Secondary Results
- Example results
Distribute ASR and MT over shot neighbourhood, then retrieval using language modelling approach
Pseudo active-learning, with positive examples from topic and 100 random negative examples from collection
Content based selection from 57 learned concepts, followed by unweighted score-based fusion
![Page 10: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/10.jpg)
Our predictive system
Retrieval Channels
Speech Search
Detector Search
Example Search
Predict Trusted Channel
Reranking
Final Results
Information NeedFind shots of pieces
of paper with writing, typing, or printing,
filling more than half of the frame area.
Result Lists
Trusted Results
- Detector results
Secondary Results
- Speech results
Secondary Results
- Example results
![Page 11: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/11.jpg)
Our predictive system
Retrieval Channels
Speech Search
Detector Search
Example Search
Predict Trusted Channel
Reranking
Final Results
Information NeedFind shots of pieces
of paper with writing, typing, or printing,
filling more than half of the frame area.
Result Lists
Trusted Results
- Detector results
Secondary Results
- Speech results
Secondary Results
- Example results
Named entity? Trust speech resultsDetector match? Trust detector resultsElse...trust example results
![Page 12: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/12.jpg)
Our predictive system
Retrieval Channels
Speech Search
Detector Search
Example Search
Predict Trusted Channel
Reranking
Final Results
Information NeedFind shots of pieces
of paper with writing, typing, or printing,
filling more than half of the frame area.
Result Lists
Trusted Results
- Detector results
Secondary Results
- Speech results
Secondary Results
- Example results
![Page 13: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/13.jpg)
Our predictive system
Retrieval Channels
Speech Search
Detector Search
Example Search
Predict Trusted Channel
Reranking
Final Results
Information NeedFind shots of pieces
of paper with writing, typing, or printing,
filling more than half of the frame area.
Result Lists
Trusted Results
- Detector results
Secondary Results
- Speech results
Secondary Results
- Example resultsTruncate result lists to top 1000
Eliminate all results not in trusted listCombine results with (weighted) Borda fusion
![Page 14: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/14.jpg)
Query-class vs Prediction
Query-class Prediction
Query class determines retrieval strategy
Query features determine retrieval strategy
Focus on assigning query-class dependent weights
Focus on identifying trusted retrieval channel
![Page 15: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/15.jpg)
Runs
• Speech channel only UvA-MM-6
• Detector channel only UvA-MM-5
• Example channel only supplementary
• Predictive reranking UvA-MM-4
• Predictive weighted reranking UvA-MM-3
![Page 16: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/16.jpg)
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Overall Automatic Search Performance
Predictive reranking
Detector channel Example channel Speech channel
Predictive weighted reranking
mea
n in
ferr
ed a
vera
ge p
reci
sion
All runs
![Page 17: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/17.jpg)
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Overall Automatic Search Performance
Predictive reranking
Detector channel Example channel Speech channel
Predictive weighted reranking
Predictive reranking outperforms individual channels
mea
n in
ferr
ed a
vera
ge p
reci
sion
All runs
![Page 18: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/18.jpg)
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Overall Automatic Search Performance
Predictive reranking
Detector channel Example channel Speech channel
Predictive weighted reranking
Predictive reranking outperforms individual channels
Weighting did not have big influence
mea
n in
ferr
ed a
vera
ge p
reci
sion
All runs
![Page 19: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/19.jpg)
General findings
• 20 topics > 0.05 inferred average precision
• 1 speech topic
• 11 detector topics
• 8 example topics
• Accurately predicted 15 of 20 topics
![Page 20: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/20.jpg)
A closer look
person opening doora bridge
people with trees and plantsface filling over half the frame
paper with writingpeople with a body of water
a mapvehicle moving away
people looking in microscopeperson watching television
people in a kitchena crowd of people outdoors
a classroom scenean airplane exterior
a plant that is the main objecta street scene at night
people at table with computerpeople in white lab coats
ships or boats in the waterman talking to camera indoors
inferred average precision0.1 0.2 0.3 0.4 0.50
Predictive w. rerankingDetector channelExample channelSpeech channel
![Page 21: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/21.jpg)
A closer look
person opening doora bridge
people with trees and plantsface filling over half the frame
paper with writingpeople with a body of water
a mapvehicle moving away
people looking in microscopeperson watching television
people in a kitchena crowd of people outdoors
a classroom scenean airplane exterior
a plant that is the main objecta street scene at night
people at table with computerpeople in white lab coats
ships or boats in the waterman talking to camera indoors
A lot of variance between channels
inferred average precision0.1 0.2 0.3 0.4 0.50
Predictive w. rerankingDetector channelExample channelSpeech channel
![Page 22: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/22.jpg)
person opening doora bridge
people with trees and plants
paper with writing
a map
people looking in microscope
people in a kitchena crowd of people outdoors
a classroom scenean airplane exterior
a plant that is the main objecta street scene at night
people at table with computerpeople in white lab coats
man talking to camera indoors
When prediction worked
Only trusted channel and reranked performance
shown
Predictive w. rerankingDetector channelExample channelSpeech channel
inferred average precision0.1 0.2 0.3 0.4 0.50
![Page 23: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/23.jpg)
person opening doora bridge
people with trees and plants
paper with writing
a map
people looking in microscope
people in a kitchena crowd of people outdoors
a classroom scenean airplane exterior
a plant that is the main objecta street scene at night
people at table with computerpeople in white lab coats
man talking to camera indoors
When prediction worked
Only trusted channel and reranked performance
shown
Predictive reranking often close to or better than trusted channel
Predictive w. rerankingDetector channelExample channelSpeech channel
inferred average precision0.1 0.2 0.3 0.4 0.50
![Page 24: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/24.jpg)
When prediction didn’t work
face filling over half the frame
people with a body of water
vehicle moving away
person watching television
ships or boats in the waterman talking to camera indoors
Predictive w. rerankingDetector channelExample channelSpeech channel
inferred average precision0.1 0.2 0.3 0.4 0.50
Only trusted channel and reranked performance
shown
![Page 25: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/25.jpg)
When prediction didn’t work
face filling over half the frame
people with a body of water
vehicle moving away
person watching television
ships or boats in the waterman talking to camera indoors
Predictive w. rerankingDetector channelExample channelSpeech channel
Predictive reranking boosts trusted channel results
inferred average precision0.1 0.2 0.3 0.4 0.50
Only trusted channel and reranked performance
shown
![Page 26: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/26.jpg)
Conclusions
![Page 27: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/27.jpg)
Conclusions
Predictive retrieval works, even with simple reranking
![Page 28: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/28.jpg)
Conclusions
Predictive retrieval works, even with simple reranking
Incorrect predictions have limited impact
![Page 29: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/29.jpg)
Conclusions
Predictive retrieval works, even with simple reranking
Incorrect predictions have limited impact
Good ingredients are crucial:garbage in garbage out!
![Page 30: Predictive Video Retrieval · 2008-12-17 · Come see our interactive search demo 0 0.05 0.10 0.15 0.20 Now with (inter)active learning! Presented by Ork de Rooij UvA](https://reader033.vdocuments.us/reader033/viewer/2022050205/5f58b6ea8b5dc047f7719d11/html5/thumbnails/30.jpg)
Beeld en Geluid Searches
20 20 uur 20 uur journaal aartsen afghanistan ajax algemene beschouwingen amsterdam
andere tijden avondjournaal balkenende beatrix buitenhof bush close up
de wereld draait door debat eenvandaag evn feyenoord gemeenteraadsverkiezingen
goedemorgen nederland hirsi ali holland sport holleeder internationale nieuwsuitwisseling irak
iran jeugdjournaal journaal journaal 20 kassa klokhuis
koefnoen koninginnedag kooten kopspijkers kro kruispunt langs de lijn libanon lijst 0 lingo man bijt hond
max catherine maxima mens milosevic miniatuur moszkowicz nederland kiest netwerk nieuwslicht
nioscoop nos nos journaal nova nps arena opsporing verzocht paul de leeuw pauw pauw en
witteman pauw witteman pechtold politie polygoon radar rembrandt rouvoet rutte saddam schepper
co schepper en co schipholbrand sesamstraat sonja spiritus sporen uit het oosten sport sportjournaal
studio sport tegenlicht televisie tros tv show twee vandaag uruzgan vandaag
verdonk verkiezingen voetbal vragenuur vragenuurtje vroege vogels wereld draait door wilders wouter bos
zembla zoekt en gij zult vinden zomergasten
Pondering
• What if we had more variety in query types?
General object queries
Named entity queries