Intelligent Information Retrieval and Presentation

with Multimedia Databases

Floris Wiesman (IKAT/UM)

Stefano Bocconi (CWI)

Boban Arsenijevic (ULCL/UL)

Yulia Bachvarova (CWI)

Nico Roos (IKAT/UM)Lambert Schomaker (AI/RUG)

2

Outline

• From search to presentation

• I2RP architecture

• Query processing & presentation generation

• Natural languange generation

• Ontology mappings

• Conclusions

3

From search to presentation

• Standard (multimedia) IR: result of query is ordered list

• Question-answering system: result of query is answer

• Our approach: result of query is multimedia presentation containing the answer

4

I2RP architecture

Presentationgenerator

Ontology agent

Semantic network

MMDB 1

MMDB n

DB 1ontology

DB nontology

Query processorqueryanswergraph Playermultimedia

presentation

Natural languagegenerator

facts text

5

6

Query processing and presentation generation

• Currently simple ‘closed’ queries

• Answers are determined from semantic network

• Rules determine which information to present

• Rules determine which modalities to use

7

Semantic network:Rembrandt’s worldChiaroscuro

Caravaggio

Caravaggist

Italy

PieterLastman

Bible Mythology

Rembrandt Saskia

Rubens

teacher teacher

Bol

is_founded_by

paints paints

inspired_by compared_to

belongs

studies

paintspaints

studies

uses

Bent Birds

Jan Lievens teacher

works_with

operated_in

Portraits

HistoryPaintings

paints

paints

paints

Night Watch

has_genre

Hendrickje

is_married

has_relation

Maria Trip

has_genre ProphetessAnna

has_genre

8

Metadata

• Search in databases by metadata, no content-based retrieval

• Metadata has to be such that:– The system can find what it looks for– The system can assemble the retrieved

information in a meaningful way– The presentation really means what was

intended: a new context is created

9

The information context

• Information items do not exist standalone, they have a context

• A presentation needs to combine the retrieved information items in a new context

• Information sources, from most structured to less structured:– Multimedia Databases (e.g., ARIA)– Digital Library (e.g, Open Archives Initiative)– The Internet

10

Narrative as a context• Example of narrative structure (Greimas):

– 6 Actants: Subject, Object, Helper, Opponent, Destinateur, Receiver

– 4 Narrative Units: Contract, Competence, Performance, Sanction

• Every character in the story plays a role in the narrative (identified by rules)– e.g. Artist biography: roles are main character, family

members, teachers, collaborators, students

• Structuring according to role– e.g. family members are grouped in private life section

11

Multimedia presentation

12

Natural-language generation

• Starts from semantic level• Semantic representation may contain:

– Participating concepts– Event structure– Temporal organization– Quantification– Relevant discourse functions

• Transforms selected meaning to natural language sentence

13

Learning ontology mappings

• The task:– Establish a mapping of concepts in ontology 2

to concepts in ontology 1

• Two steps:– Establish joint attention: which are common

instances of the ontologies?– Establish mapping: which operations map the

concepts best? (wrt joint attention)

14

Two example ontologiesOntology 1

Object

Title = Self portrait as St. Paul

Artist = Rembrandt Harmensz. van Rijn

Materials = oil on canvas

Date = 1661-1662

Ontology 2

Artefact

Title = Self portrait as the apostle St. Paul

Creator

family name = Rijn

given name = Rembrandt

Harmensz.

other name = van

Material = oil on canvas

Period

start = 1661

end = 1662

Mapping can be made by copying, splitting, and merging leaf conceptsMapping can be made by copying, splitting, and merging leaf concepts

15

Establishing joint attention

• do: – agent 1: sends instance of concept to agent 2

– agent 2: returns instance with highest proportion of words in common

• until enough common instances found above threshold

result is joint attention set: concepts of agent 1 with instances known to agent 1 & 2

16

Mapping of concepts• Mapping of concepts consists of operators:

– field n: copy leaf concept– merge s: merge leaf concepts using separator s– split s: split leaf concept at separator s– first: copy first part of split– last: copy last part of split

• Separator: none, space, colon, semicolon (,TC)• Example:artefact.period.end

field object.date, split(-), last

17

Establishing a mapping

• Search space consists of all possible mappings• Value of a mapping: number of correct mappings

on joint attention set• Mapping with highest value wins• Search is guided by proportion of words that

instances have in common• Search space is reduced by ignoring mappings

between leaf concepts with a low proportion of common words

18

Conclusions

• We have shown approach for IR from multimedia databases using:– Knowledge-based query augmentation

– Combination of IR results in a single multimedia presentation

– Natural language generation

– Automatic ontology mapping

• Parts are realized as prototypes;

to be combined in one system