media retrieval (2) - ryerson university · 2018-09-21 · media retrieval (2) prepared by ling...
TRANSCRIPT
20/09/2018
1
Media Retrieval (2)Prepared by
Ling GuanJose Lay
Paisarn MuneesawangNing ZhangRui Zhang
1
Outlines (revisited)• Introduction:
– Intellectual Foundation of Multimedia Information Retrieval– Retrieval Models
• Text Retrieval:– Database, Bibliographic, and Keyword Searches
• Content‐based Retrieval:– Object‐matching and beyond
• Indexing:– Inverted file
• Searching:– Multimodality and Query adaptation
2
20/09/2018
2
Content‐based Retrieval(the conventional approaches)
• Images, Audio, and Video
• Caption Text
• Multimodalities
• Search methods and search engines
• Result visualization/presentation
Let us get start with a tour!
3
Examples of The MMR systems
• Text‐based (and Content‐based) Querying– Images
• Google’s Image Search• Yahoo Image Search (AltaVista’s Image Search)• TinEye Image Search (Toronto Local)
– Audio• Yahoo Audio Search• Compaq’s SpeechBot (no longer exist)
– Video and Multiple modalities• Facebook.com• Lycos Multimedia Search (evolving from a single search engine into a focused network of community and social sites)
• FAST AllTheWeb Search • emediasearch.com (covering the Middle East region)
4
20/09/2018
3
Google’s image search: new feature
5
Google’s image search: new feature
6
20/09/2018
4
Google’s image search: new feature
7
Yahoo (Alta Vista) Image Search
8
Yahoo Image Search
20/09/2018
5
TinEye, developed by Idée, Inc
9
Compaq’s SpeechBot(closed by HP)
10
20/09/2018
6
FAST’s AllTheWeb Search(bought out by Overture)
11
Qwiki(acquired by Yahoo!)
• Traditional search result presentation grid‐based
• It turns static information into dynamic experience
12
20/09/2018
7
MediaMill
• Semantic video search engine
– Cross browser
– Galaxy browser
– Sphere browser
13
3D Visualization/Presentation‐MIT Tangible HCI Lab
14
20/09/2018
8
Non‐general searching sites
Image/Video Hosting sites
• Primarily function as sharing sites for on‐line community.
• Indexed large database provide searching features for users.
• Image hosting– Flickr, Panoramio, Picassa
• Video hosting– YouTube
Vertical Search Engine
• Focus on a specific segment of online contents
• Academic: – IEEE Explore, Google Scholar
• Map: with yellow page info– Google Maps and StreetView,
Bing Maps and StreetSide.
• Food and Shopping– Yummly, Bing shopping, Yelp
• Information– Wikipedia
15
The Extended Text‐based Systems
• Processing textual information to infer on audio‐visual content.
– Often rely on the filename. • koala.jpg
– Using meta information embedded in the web page.• <IMG SRC=“ryerson1234.gif" WIDTH="300" HEIGHT="60" ALT=“koala watching TV">
– Further processing of the container page.• The webpage is called: Trailer for Starwars episode I: Phantom the Menace, and in there you find a link to the phantom.mpg file.
16
20/09/2018
9
Speech Pre‐processing
• In a speech retrieval engine, e.g., SpeechBot, a speech processing mechanism is applied to convert audio data into textual data, then conventional text‐based indexing and searching could be applied.
17
Touring the First Generation CBR Systems
• Visual‐based Querying
– University of California at Irvine’s MARS (Initially developed at Univ. of Illinois)
– IBM’s QBIC
18
20/09/2018
10
MARS
19
QBIC at Hermitage Museum
20
20/09/2018
11
What did you observe?
• What has been extended?
• Which applications may be supported?
• What is the constraint?
21
CBR: a brief history
• In 1970s, image data are commonly archived independently and indexed using text‐based databases.
– Indexing carries subjectivity of annotation.
– Manual and expensive.
• Into 1990s, efforts are taken to remove person indexing and to automate the mechanism.
– Image data themselves are used as indexes.
– Full automation is idealized.
– Content‐Based Retrieval was born.
22
20/09/2018
12
CBR: a brief history ‐2
• The CBR was founded on the computer vision and pattern recognition idealism and their techniques.
– CBR employs object‐matching mechanisms.
– Earlier systems are characterized by Query by Example approach.
– Features commonly used are: Color, Texture, and Shape.
– Motion features in case of videos.
23
CBR: a brief history ‐3
• As Computer vision ideal is yet to achieve, CBR found itself to be constrained in a green‐house.
– Many set‐off for sophisticated feature exploration and computational reasoning adventures. Deep NNs have provided a viable solution.
– A strong call for abandoning the full‐automation idealism for returning to human interaction.
– HCI techniques such as relevant feedbacks started to roll‐out.
24
20/09/2018
13
Between Yesterday & Tomorrow
• An interesting question to ask today may well be: is the object‐matching idea all we want?– Typical current CBR query would mostly read: retrieve all documents that presents a blonde wearing blue dress driving a red porsche.
– Now let us suggest some offsets: How about retrieve all paintings created using chiaroscuro technique? What differences are there between Christmas Cards and Birthday Cards?
• Where do you want to go today?
chiaroscuro – pictorial representation in terms of light and shade without regard to color
25
The Passage
• Go Deeper. Enhancing what we have got.– Human‐centered computing.– Distributed Computing.– Multi‐modalities querying and processing.
• Go Sidewise. Call for better multimedia understanding.– Efforts to find out which information and semantics are useful
and how they could be derived and managed – Concept‐based (or semantic) retrieval.
– Transcending the Object‐Matching Idealism.– Multimedia information mining.– Big data– Deep learning– …
26
20/09/2018
14
Content‐based Retrieval?
Traditional Bibliographic Objectives
Keyword SearchesObject Matching
27
Roadmap
Test 1: Tuesday, October 16, 1 hour Coverage of material: everything studied up to Lecture 5.
The 1st half of the class is normal teaching, and the 2nd is test time.
Open notes (paper copies only). You may consider bringing in a sheet of paper summarizing the contents of the course up to and including Lecture 5.
Material taught on October 16 will not be included in Test 1.
Office hours: Monday, October 15, 2:30‐4:30pm, ENG 315.
Submit your project proposal electronically by Friday, October 12, 11:59pm.
28