A. Frank

Multimedia/Video Search

חיפוש מולטימדיה בדגש חוזי

אריאל פרנקמחלקה למדעי המחשב

אוניברסיטת בר-אילןariel@cs.biu.ac.il

A. Frank2

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

• Video downloads

A. Frank3

Multimedia Resources

• ,Pictures, Photos, Maps, Graphics) תמונות Animations)

• (Audio, Sounds) שמע

• (Music, MP3)מוסיקה

• (Radio, Podcasts, Speeches) הקלטות

• (Video, Flash, Vlogs)חוזי

• (Movies, TV) סרטים

A. Frank4

MM Search & Information Retrieval (MMIR)

• Search & information retrieval of multimedia objects is hard!!

• Using SEs to find relevant, quality MM objects can be difficult and frustrating.

• Sample query: find a movie that has a large mouse chasing a Persian cat with twins watching from above?

A. Frank5

MM Search & Information Retrieval (MMIR)

• Common MM search is mainly text-based; the object text is derived from:– caption of the MM object

– text surrounding the MM object

– entire text of the containing page

– filenames of MM object and the containing page

– manual/automatic annotations

• Recently, tag-based MM search is becoming effective (social computing).

• Fortunately, long due, content-based MM search on the Web is forthcoming.

A. Frank6

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

• Video downloads

A. Frank7

General SEs Sample

• Google (general, video)

• Yahoo (general, audio)

• AOL (general, video)

• MSN/Live (images, video)

• AltaVista (images, audio, video)

A. Frank8

Multimedia: Google

A. Frank9

Multimedia: Google

A. Frank10

Multimedia: Yahoo

A. Frank11

Yahoo Audio Search

A. Frank12

A. Frank13

AOL Video Search

A. Frank14

MSN Live

A. Frank15

Multimedia: MSN Live

A. Frank16

AltaVista Multimedia Search

A. Frank17

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

• Video downloads

A. Frank18

Vertical SEs Sample

• Picsearch (image)

• FindSounds (audio)

• SearchVideo (video)

• Clipblast! (video)

• Flurl (multimedia)

A. Frank19

Images: Picsearch

A. Frank20

Picsearch Advanced Interface

A. Frank21

Audio: FindSounds

A. Frank22

Types of Sounds

A. Frank23

Video: SearchVideo

A. Frank24

Video: Clipblast!

• Organizes and makes the video Web relevant, fast and simple to navigate.

• Video Search and Navigation technology makes it easy to search, browse and personalize the video that viewers want, when they want it.

• Crawls the entire Web for all video content available, indexing more video content providers than any other video search engine.

• Largest video distribution platform ever.

A. Frank25

Video: Clipblast

A. Frank26

Multimedia: Flurl

A. Frank27

Flurl list of sites

A. Frank28

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

• Video downloads

A. Frank29

Tag-based Search Sample

• Flickr (image)

• Panoramio (image, maps)

• YouTube (video)

• Keotag (multimedia)

A. Frank30

Flickr

A. Frank31

Flickr

A. Frank32

Flickr

A. Frank33

Image/Maps: Panoramio

A. Frank34

Image/Maps: Panoramio

A. Frank35

Image/Maps: Panoramio

A. Frank36

Video: YouTube

A. Frank37

Keotag Meta-SE

A. Frank38

Keotag Meta-SE

A. Frank39

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

• Video downloads

A. Frank40

“Content-based“ Image Search/Retrieval

• “Content based” image DB or Web site(s) that contain a large number of images.

• Image analysis techniques used to describe images by their extracted visual features such as color, texture, shape and orientation.

• These objects can be represented by sets of image features and then compared with the sets extracted from other images.

• Feature sets are described by their statistics, and the computer searches for statistically likely matches.

• Queries can be formulated in terms of visual features or by similarity options.

A. Frank41

Content-based Search Sample

• Squid (image)

• retriever (image)

• Melodyhound (audio)

• Musipedia (music)

• Blinkx (video)

A. Frank42

Image: Squid

• Has about 1100 images of marine creatures in its database.

• Each image shows one distinct species on a uniform background.

• Every image is processed to recover the boundary contour, which is then represented by three global shape parameters and the maxima of the curvature zero-crossing contours in its Curvature Scale Space (CSS) image.

A. Frank43

Images: Squid

A. Frank44

Images: Squid

A. Frank45

Images: Squid

A. Frank46

Images: Squid

A. Frank47

Image: retriever

• Experimental service which lets you search and explore in a selection of Flickr images by drawing a rough sketch.

• Matches the most pronounced shapes and slabs of colors.

• The results are usually fairly good, sometimes even stunning,

• But it doesn't do object/face recognition of any kind, so if drawing an outline sketch of a chair, it almost certainly won't get you one back.

A. Frank48

Images: retriever

A. Frank49

Images: retriever

A. Frank50

Images: retriever

A. Frank51

MelodyHound

A. Frank52

MelodyHound

A. Frank53

MelodyHound

A. Frank54

MelodyHound

A. Frank55

Musipedia

A. Frank56

Audio: MusiPedia

A. Frank57

Music: Musipedia

A. Frank58

Musipedia or Web Search?

A. Frank59

Video: Blinkx

• World's largest and most advanced video search engine.

• Use of visual analysis and speech recognition to better understand rich media content.

• Advanced image and audio analysis techniques are used to automatically extract information about a video in real time and to create a rich index about the video content.

• Automatically and implicitly conducts searches based on the content being viewed by each individual user.

• With the advent of blinkx TV, users can now search and access news, movie trailers, popular multimedia segments and other video formats on demand.

A. Frank60

Video: Blinkx

A. Frank61

Video: Blinkx

A. Frank62

Blinkx TV Architecture

A. Frank63

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

• Video downloads

A. Frank64

Video downloads

• Nearly none of the video search/sharing sites provide a simple one-click button to download video clips.

• If video download needed, various workarounds exist (ordered by preferred methods, if some not working):

1. Right-click on link -> “Save Target/Link As”.2. Install/use the Firefox VideoDownloader Extension,

from Mozilla addons site, to download video (and audio).3. Copy video link into KeepVid site and download from

the provided link (usually saving it as FLV file).4. View -> “(Page) Source”, locate video filename/format,

and look in its directory, or search for it (filetype:, contains:). 5. Use Applian Technologies streaming capture applications

to record streaming audio/video into a file.

A. Frank65

VideoDownloader

• A Mozilla Firefox Extension that downloads all embedded MM objects on a page including video.

• Just click the icon to download the watched video.• VideoDownloader

A. Frank66

Keepvid

A. Frank67

“View Source” techniques

How to get playing VideoDownloader Flash file!?

A. Frank68

“View Source” page

A. Frank69

File name/path search & save

A. Frank70

Applian’s A/V Streaming Capture Suite

• Replay A/V & Media Catcher are the simplest and most powerful ways to capture Video and Audio from the Web on your PC.

• Capture Windows Media, Real, Streaming MP3, Streaming Flash Animations and Video.

A. Frank71

References

• Kate Greene, Better, More Accurate Image Search, Technology Review, April 9, 2007, http://www.technologyreview.com/Infotech/18501/

• Kate Greene, Surfing TV on the Internet, Technology Review, April 18, 2007, http://www.technologyreview.com/Infotech/18545/

• Amit Agarwal, 10 Interesting Things You Can Do with YouTube or Google Videos, Digital Inspiration (Blog), July 27, 2006, http://labnol.blogspot.com/2006/07/10-interesting-things-you-can-do-with.html

A. Frank72

Still Up(loaded))?-: