From“Snow White and the Magic

Mirror”

To“Scientist and

Multimedia Retrieval”

Mr. Scientist, one night, was traveling to Fairy-tale land

What was happening there?

Mirror, mirror, on the wall,

Who is the fairest of them all?

O Lady Queen, Snow-White is the fairest of them all

Cool!How the mirrorcan do that?

...

Searching Process

Askingquestion

Sendingquestion with invocation

Searching

Selecting answer

Sending result

Showing result

呪文

Snow-White's stepmother: User who creates the question or the query

Mirror: Interpreter who translates “muggles” language to “wizard” language, and vice versa

Wizard: Seeker who uses his “magic” search engine to find the right person in the fairy-tale land. The answer is called retrieval

白雪姫の継母： 質問（クエリ）をつくる

魔法の鏡： 魔法を使えない人間の言葉を魔法使いの言葉に訳したり、その逆をする

魔法使い： 魔法のサーチエンジンを使っておとぎの国の住人の中から　　　　　　　クエリにあった人を探す。これを検索（retrieval）と呼ぶ

Mr. Scientist started to think about it. He wants to turn the magic mirror into his computer

and turn the wizard into search engine

Remember

Computer only understands “0” and “1”

All data must be transformed to 'digital signal' デジタル信号

Translate from picture to digital image

Translate from sound to digital audio signal

It seems that everything is solved,

but .....

すべて解決したように思うけど・・・

Challenges The question of Snow-White's stepmother is

“who is the fairest of them all?”. It means that, the magic mirror must look for − The most beautiful girl, and− The most benevolent girl

親切な

Challenges How to evaluate “a beauty” ? for example, in

fairy-tale land, a beautiful girl must reach a standard

Hair is as black as ebony → color（色） Body is as slender as willow → shape（形） Skin is as soft as silk → texture（質感） Singing voice is as nightingale‘s → signal（音）

→ Since these information belong to a girl, it is called internal information or internal features

コンピュータはどうやって「美しさ」を評価するのか？

内部情報 内部特徴

Challenges How to evaluate “a benevolence” ? for

example, in fairy-tale land, a benevolent girl must reach a standard

– Be loved by most of inhabitants in fairy-tale land

– Doing a lot of things out of charity→ Since these information does not belong to a

girl, it must be collected from inhabitant who know the girl, it is called external information.

The features could be letters, audio/video records, books, etc.

慈愛心から

外部特徴

Some technical terms Color, texture, shape, and signal are called

low-level features that are extracted directly from objects and the computer very easily understands. 色、質感、形、音は低レベル特徴量と呼ばれる。これは、画像、映像中の「モノ」から直接取り出すことができて、コンピュータでも理解しやすい

Some technical terms Hair, body, skin, and eyes are called high-

level features. In order to let the computer understand these features, Mr. Scientist must prepare a lecture by which the search engine could recognize them.

→ That lecture is called “bridge the semantic gap between low-level and high-level features”

色、質感、形、音は低レベル特徴量と呼ばれる。これは、画像、映像中の「モノ」から直接取り出すことができて、コンピュータでも理解しやすい

「教えてあげる」ことを低レベル特徴量から高レベル特徴量へのセマンティックギャップの橋渡しという

Some technical terms After the computer translate query of users to

features, the search engine will compare these features to features in its database to choose the best match one.

The tool to calculate/compare the similarity between two sets of features is called Distance measure.

コンピュータ（魔法の鏡）が質問を特徴量に変換したら、サーチエンジン（魔法使い）がデータベース内の特徴量と比べて一番似ているものを選ぶ

特徴量同士がどれだけ似ているかを比べるには、距離尺度を使う

Remind … Remind ... One object usually has internal features and

external information Multimedia includes images, audio, video,

and text Query is what users ask computer to search Retrieval is what search engine found

おさらい

Remind … Remind ... Computer does not understand human being

language. Computer easily recognizes low-level features but hardly understand high-level features

The semantic gap between low-level features and high-level features is large, and it is very difficult to bridge this gap

Remind … Remind ... Search engine only understands and works

with features Distance measure is necessary to comparing

two sets of features Browsing is the process of showing results

returning by search engine under human languages

Indexing is the process of re-ordering data in database in order to look for faster

閲覧は人に理解しやすい形で検索の結果を見せること

索引付けは見つけやすくなるようにデータを並び替えること

Query by keyword

Wrong result

Annotation

Cinderalla

Mermaid

OK

FALSEManualTime consuming

AutomaticFast but very difficult

Automatic annotation

Query by concepts

スケッチ

Query by color

It looks good ...

But sometime the results

are not good at all

キーワード

History of Search Engine's development

Conclusions

Beginner Intermediate Expert