warren hayes ai symposium - working draft

3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions

Pat Hayes, Margaret Warren

A Lightweight Ontology for Linking Images


what we are/and are not doing

• images themselves are our main topic, rather than as illustrations of other topics, linking information to and from them and linking the images themselves

• Interested in using linked data to represent the kinds of things artists themselves say about their work (or the person doing the markup)

• What we don’t do: expect people to agree on artistic interpretations or emotional content of images

• ignore our paper in the proceedings – it was a working draft and we have made significant changes.


foaf:depiction


Snapshot, line drawings, news photos, paintings, fine art


• By Image, we mean a digital image, typically one that can have associated EXIF data.

• So, other kinds of "images" (drawings, paintings, non-digital photographs, stained glass, etc..) have to be classified as Works.

• A work may be a physical thing with dimensions, a location, a time and place of creation, etc. that is separate from those same properties in an image; a digital image can be a work.

• Basic relation between Work and Image is facsimile, a kind of 'visually same As' relation.

Image identity


Broader notion of depicts

All images depict the person: McKenzie Oerting


Depiction can include imaginary, biblical, mythological notions


What is the subject?


shows, depicts, subject, conveys

Shows: trees, grass, house,

window, ventilator

Depicts: particular type of car,

each of the people

Subject: group looking at a car

Shows: head

Depicts: hands

Conveys: grief, sadness


Do all images depict something?

Depict diamonds? Shapes? Or Nothing…


Pictorial elements

Pictorial elements are related by inclusion:

Basic graphical 'unit' of depiction is a pictorial element.

This includes 'boxes' but also lines, regions, areas,

patches of color, texture, etc.. “

elements of art - The basic components used by the artist

when producing works of art. Those elements are color,

value, line, shape, form, texture, and space. The elements

of art are among the literal qualities found in any artwork.

(Artlex, 1996-2010)

http://www.artlex.com/ArtLex/a/artquotations.html

http://www.artlex.com/ArtLex/c/color.html

http://www.artlex.com/ArtLex/uv/value.html

http://www.artlex.com/ArtLex/Li.html

http://www.artlex.com/ArtLex/Sg.html

http://www.artlex.com/ArtLex/f/form.html

http://www.artlex.com/ArtLex/T.html

http://www.artlex.com/ArtLex/So.html


http://www.artlex.com/ArtLex/Art.html


Pictorial Elements:Shape of thing that

looks like a

female torso

Shape of thing that looks like a

representation of a brain

area that looks likethe torso is

sitting astride a seat of some sort

Depicts vs. looksLike

Depicts: egg, a test tube, a photographic transparent layer, a piece of jewelry

Title: Surfing With Psyche


Pictorial Elements can be 'empty'

A reclining nude is depicted by white space defined by an outline. The white space is a pictorial element.

Cataloging pictorial elements is a large task. As a beginning, we distinguish visible/implied and line/area. Here, the nude is depicted by an implied area. All the visible elements are red lines.


backgrounds

hasPictorialBackground

all white, not necessarily

able to define with a region

hasDepictedBackground

sky, clouds, not necessary

to define with a region


Angels in the clouds

depicts:

Blue Angels ,jets,

planes, formation

conveys:

military power?

subject: specific formation?

hasDepictedBackground:

sky, clouds

region contains a shape

(PictorialElement) that

looksLike an Angel

(iconic)

depicts: clouds

depicts: angel

shows: clouds


The ontology so far


future work


http://www.purl.org/net/mw

http://www.carmapro.com/mw/mw.rdf

http://www.purl.org/net/mw

http://www.carmapro.com/mw/mw.rdf


Extra slides follow


Looking like an archetype• Some shapes are best described by what they look like iconically.

• If we can use RDFS or OWL Full, this could be described as 'looking like' a class; but this will break OWL-DL reasoners.

• Or, we could treat a SKOS concept as an archetype, but this is not (yet) a widely accepted convention.


visiblePictorialElements

looksLike iconic heart

hasColor Red

looksLike iconic dancing figures

hasColor yellow orange red

hasPictorialBackground: white


What's in an image?

• We conventionally say that depicted things are "in" an image.

• Is this a philosophical mistake? Maybe. But rather than try to correct it, the ontology tries to avoid it.

• Basic methodological rule for linked data: don't try to repair "bad" intuitions.


shows: person, depicts: face in hands

subject: crying conveys: grief or despair, etc.

shows: broken headlight, turn signal

depicts: front fender of school bus

subject:

conveys: beauty in decay, decline of

education


Can any Thing be depicted? Do we need a class of 'depictable things'?

Things depicted include

(keywords):

• the car (specific type)

• people (specific people)

• a group of 7 people

• a building

• trees

• brown grass

Subject: group studying a car?

one of the people?

the car itself?


Adult, Adults Only, Agriculture, Beautiful People, Beauty, Beauty In Nature,

Biting, Branch, Color Image, Concepts, Custard Apple, Desire, Enjoyment,

Food, Freshness, Fruit, Green, Gripping, Growth, Harvesting, Healthy Eating,

Healthy Lifestyle, Horizontal, Human Face, Human Hand, Human Mouth,

Hungry, Leaf, Lifestyles, Meal, Nutrient, One Person, One Woman Only, One

Young Woman Only, Only Women, Open, Opening, Organic, People,

Photography, Picking, Plant, Southeast Asian Ethnicity, The Human Body,

Tree, Vegetarian Food, Women, Young Adult

http://www.thinkstockphotos.com/Search


















































What can be depicted?

• FOAF: People.

• DBPedia: People, places, events,…

• Umbel (Cyc): VisibleThings

• Visual Resources Association: myths, legends, intentions, …

• We provide a different property conveys for the relationship between an image and "un-depictable" things such as mood, idea, intention, emotion, etc..


http://www.gettyimages.com/Search/Search.aspx?query=z.i.H4

sIAAAAAAAEAO29B2AcSZYlJi9tynt_SvVK1-

B0oQiAYBMk2JBAEOzBiM3mkuwdaUcjKasqgcplVmVdZhZAz

O2dvPfee--

999577733ujudTif33_8_XGZkAWz2zkrayZ4hgKrIHz9-

fB8_In7dfLn91etf49f4NX6PX_dskV3kvyb9mtD_f9N5ns3S9fKqz

larYnnxa_7f9PwaG59f0_xM6qpqX2Z1tmjMZ78W_r-

78xvSj1_v3evr5vSdeePX9P7mltm5Bai_4-

VfOzuf2g78P35tfLDD8Bete3PR_rr6-2_xa3DX9tc99-s99-t99-s-

__rrUB-lhWf_Qne_Lv1RLyw24Z-_Pj7axV-_Hn8-

sTA6f_Nr5yEU-

ydD2bFQzjtQ3N_8Wm1e6_zJUPYslLoDxf1NBM0tDPMH_s_U

3QV1iQL5yr4f_PVr55PGfmP-

4BnJXY_e779W7pp7v__azWplP8cfBqPfyr54OXUvXk5_bf39t8

TfO_a3ffvbLn77dara4zb7F381yf2vzF94fu3Z1dL8_muaP8BSv0

6z9kYb_PVr55nDz_-Dfl_5X9g_ft0mgBb--etM83futeCvX_vds-

dOGugP-

0VWe2h7f_xGzbRa5U_Wy1npUT_41AD8ccB5sHMfv_86xGpr2

zz46_8Bs9pv0zoEAAA.


Linked Image Data

• Linked data…

• Newest manifestation of the semantic web: use RDF to create a web of semantic links between data items.

• Meanings of links are described by lightweight (small, intuitive, useful) ontologies.


Linked data, not data silos


Linked data is link + description

• RDF descriptions use URIs as names to refer to things; linking means using those URIs to also connect to the described thing, or a source of more information about it.

• This can be smoothly integrated into HTML using RDFa.<foaf:knows> <foaf:Person rdf:about="http://danbri.org/foaf.rdf#danbri">

<foaf:mbox_sha1sum>362ce75324396f0aa2d3e5f1246f40bf3bb44401</foaf:mbox_sha1sum>

<foaf:name>Dan Brickley</foaf:name>

<rdfs:seeAlso rdf:resource="http://danbri.org/foaf.rdf"/>

</foaf:Person>

</foaf:knows>

<foaf:based_near rdf:resource="http://dbpedia.org/resource/Milton_Keynes"/>

<PersonalProfileDocument rdf:about="">

<cc:license rdf:resource="http://creativecommons.org/licenses/by-nc/3.0/"/>

<dc:title>Tim Berners-Lee's FOAF file</dc:title>

<maker rdf:resource="http://www.w3.org/People/Berners-Lee/card#i"/>

<primaryTopic rdf:resource="http://www.w3.org/People/Berners-Lee/card#i"/>

</PersonalProfileDocument>


Linked data is link + description

• Description terms defined by lightweight ontologies.

• Designing lightweight ontologies is an art, not a science.

• Often, the formal ontologies are quite minimal, eg FOAF refuses to say anything more than the class People is all people, and Agent is "things that make stuff happen". Still, it does specify that People is an rdfs:subClass of Agent. In practice, meaning is specified by use (in the actual links).

• It is considered good form to re-use existing relations, classes and properties whenever possible. But this can set up some tension with the meaning-as-use observation.


Linked data is here, now

Collectively, the data sets consist of over 13.1 billion RDF triples (dataset statistics),

which are interlinked by around 142 million RDF links (linksset statistics) (November 2009).

http://esw.w3.org/topic/TaskForces/CommunityProjects/LinkingOpenData/DataSets/Statistics

http://esw.w3.org/topic/TaskForces/CommunityProjects/LinkingOpenData/DataSets/LinkStatistics


Linked Image data?• There are many images in existing linked data, but almost always as illustrations of things

(people, places, buildings..), as in FOAF.

• We want to describe images as objects in their own right. Images are our topic. So we need an image-describing vocabulary with its own lightweight ontology.

Earlier work on image description

all focuses either on the things

depicted, or on the cultural setting

of the images, or on the curation

and cataloging of museum

collections.


Very lightweight image ontology

Questions one might ask….

1. Can any Thing be depicted? Do we need a class of 'depictable things'?

2. Do all images depict something?

3. What exactly is depiction anyway?

4. What exactly is an Image? (Is it a file type, for example? Are all images

digital images?)

etc…


• Dense lines of particular colors, splatters, all pictorial elements

• No distinct background, nothing depicted


Pictorial elements

Pictorial elements are related by inclusion:

Basic graphical 'unit' of depiction is a pictorial element.

This includes 'boxes' but also lines, regions, areas,

patches of color, texture, etc.. “

elements of art - The basic components used by the artist

when producing works of art. Those elements are color,

value, line, shape, form, texture, and space. The elements

of art are among the literal qualities found in any artwork.

(Artlex, 1996-2010)

http://www.artlex.com/ArtLex/a/artquotations.html

http://www.artlex.com/ArtLex/c/color.html

http://www.artlex.com/ArtLex/uv/value.html


http://www.artlex.com/ArtLex/Sg.html

http://www.artlex.com/ArtLex/f/form.html

http://www.artlex.com/ArtLex/T.html

http://www.artlex.com/ArtLex/So.html


http://www.artlex.com/ArtLex/Art.html


Ontology can be molded to fit

A depiction of the

torso of

a female mannequin

AND/OR

A Pictorial Element

that looks like a

female torso


Depicting vs. looking like

• Simple depiction (as in FOAF) means something like "is a picture of". Our depiction is wider in scope, and does not imply visual similarity. An Xray image is a depiction, for example. So we also need the notion of looking like.

• This is also important for characterizing pictorial elements.

Depicted female

torso, x-ray

Depicted female

torso

looksLike

female torso*

looksLike

iconic female

torso


If an image can depict, then a PART of an image can depict. if a whole image can depict, so can part of an image.

Images are pictorial elements.


Entire image is a pictorial element

that looks like a female torso

Entire image is a pictorial element

that depicts a female torso


Depiction is in the eye of the …

depicts Marie-Therese Walter depicts female torso


Defining classes of image

• Can now define for example 'line drawing': an image all of whose component visible pictorial elements is a line.

<owl:Class rdf:ID="LineDrawing">

<owl:intersectionOf rdf:parseType="Collection">

<owl:Class rdf:about="#Image" />

<owl:Restriction>

<owl:onProperty rdf:resource="#hasVisualPart" />

<owl:allValuesFrom>

<owl:unionOf rdf:parseType="Collection">

<owl:Class rdf:about="#ImplicitPictorialElement" />

<owl:Class rdf:about="#LinearPictorialElement" />

</owl:unionOf>

</owl:allValuesFrom>

</owl:Restriction>

</owl:intersectionOf>

</owl:Class>

warren hayes ai symposium - working draft

Technology