semantics for visual resources: use cases from e-culture
DESCRIPTION
Keynote Semantic Web Summer School, Cercedilla, Spain, July 2006TRANSCRIPT
![Page 1: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/1.jpg)
Semantics for visual resourcesUse Cases from E-Culture
Guus Schreiber
Free University Amsterdam
![Page 2: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/2.jpg)
2
Purpose Analyze a number of use cases from e-culture
domain– Multimedia plays key role
Required technology– Typically combination of technologies
Relation to state of the art
Acknowledgements: This presentations contains slides and images provided by Laura Hollink, Giang Nguyen and Cees Snoek. Also thanks to the MultimediaN E-Culture team
![Page 3: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/3.jpg)
3
Use case: Asian chairs
User has found an image of an Asian chair
Annotation:ex:image vra:stylePeriod aat:Guangxu .
How can we find images of Asian chairs from the same historical period?
![Page 4: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/4.jpg)
4
AAT info on Guangxu
![Page 5: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/5.jpg)
5
Importance of time and space informationMany queries require time/space
knowledge, either absolute or abstractedFor the chair image we can establish
– Country = China (link Chinese => China)– Period = 1644-1911 (from Qing description)
Technology requirements:– Thesuari relating time/space concepts– NLP for unstructured descriptions– Time/space reasoning techniques
![Page 6: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/6.jpg)
6
![Page 7: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/7.jpg)
7
![Page 8: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/8.jpg)
8
Sample place information in TGN
<tgn:AdministrativePlace rdf:about="&tgn;1000111"
tgn:standardLatitude="35" tgn:standardLongitude="105“> <vp:parentPreferred
rdf:resource="&tgn;1000004"/> ……..</tgn:AdministrativePlace>
![Page 9: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/9.jpg)
9
Issues when searching for “nearby” Asian chairsClose in space:
– Other country in (East) Asia– Latitude/longitude
Close in time:– Links between style periods– Match time periods (and
handle incomplete information)
![Page 10: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/10.jpg)
10
![Page 11: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/11.jpg)
11
Use case: painting style
Find paintings of a similar style
MATISSE, HenriLe bonheur de vivre (The Joy of Life)1905-1906Oil on canvas, 69 1/8 x 94 7/8 in. (175 x 241 cm)Barnes Foundation, Merion, PA
![Page 12: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/12.jpg)
12
How can we find this other Fauve painting?
DERAIN, AndreThe Turning Road, L'Estaque, 1906Oil on canvas, 51 x 76 3/4 in. (129.5 x 195
cm)Museum of Fine Arts, Houston, Texas
![Page 13: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/13.jpg)
13
Issues
Parse annotation to find matches with thesauri terms– E.g. match artists to ULAN individuals
Artists-style links– AAT contains styles; ULAN contains artists, but there
is no link• Learn link from corpora• Derive it from other annotations
– Domain-specific rules/reasoning needed • see example in SWRL doc• Painters may have painted in multiple styles
![Page 14: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/14.jpg)
14
![Page 15: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/15.jpg)
15
![Page 16: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/16.jpg)
16
Search: WordNet patterns that increase recall
without sacrificing precision (Hollink)
![Page 17: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/17.jpg)
17
Issues w.r.t. thesauri
Public availability!RDF/OWL representationLearning/specifying term/concept mapping
– owl:equivalentClass, owl:sameAs, rdf:type, rdfs:subClassOf
– Domain-specific linksManaging the evolution of the thesauri and
the mappings
![Page 18: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/18.jpg)
18
Use case: find images with the same subject
Find another painting which portrays dancing
![Page 19: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/19.jpg)
19
Issues
Same subjects can be visually very different
Subject is often missing from the annotation
Mismatch: users often search for subjects of images
![Page 20: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/20.jpg)
20
Conceptual subject descriptions
85% of the user queries:
General Descriptions of generally known items. Only general, everyday knowledge is necessary. Descriptions are at the level of the Natural categories of E. Rosch (1973), or more general. E.g An ape eating a banana.
Specific Descriptions of objects or scenes that can be identified and named. Specific domain knowledge is necessary to recognize the objects or scenes. E.g. The old male gorilla Kumba, born in Cameroon and now living in Artis, Amsterdam
Abstract Descriptions for which interpretative knowledge is used. This category is subjective. E.g An animal threatened with extinction.
![Page 21: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/21.jpg)
21
Example concepts in image
Specific– Fall of the Berlin Wall
General– People walking at night
Abstract– Fall of the Iron Curtain
![Page 22: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/22.jpg)
22
Use of conceptual categories by people searching for images
Conceptual level: 83%
0%
20%
40%
60%
80%
100%
event time place relation scene object
Characteristics
Nu
ber
of
elem
ents
in
% o
f co
nce
ptu
al e
lem
ents
Abstract
Specific
General
![Page 23: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/23.jpg)
23
Thesauri for scenes: Iconclass
![Page 24: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/24.jpg)
24
![Page 25: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/25.jpg)
25
![Page 26: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/26.jpg)
26
Annotation of image content
Template for subject descriptionAgent Action Object Recipient
Guidelines for manual annotation– Annotate as specific as possible
Default reasoningCBIR support:
– Object identification– Spatial relations
![Page 27: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/27.jpg)
27
![Page 28: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/28.jpg)
28
![Page 29: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/29.jpg)
29
Some forms of image content are well suited to image analysis
Collection of clothesAbstract painting
![Page 30: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/30.jpg)
30
The semantic gap
The distance between Content-Based Image Retrieval and semantics:– Smeulders, Worring, Santini, Gupta, Jain. Content-
based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), December 2000.
Direct links between visual features and semantic concepts become more difficult when the domain is broader / more general
![Page 31: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/31.jpg)
31
Example semantic bridge:microscopic cell images
mpeg7 : StillRegion(region) ^mpeg7x : Dense(region) ^mpeg7 : DominantColor(region, col) ^swrlb : lessThan(col, 100) => mpeg7 : Depicts(region, mesh : MatureGranule)
![Page 32: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/32.jpg)
32
Segmentation often requires user interaction
![Page 33: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/33.jpg)
33
Automatic detection of concepts can be difficult even in “easy” cases
What is the color of this ape?
![Page 34: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/34.jpg)
34
Image analysis useful for collection navigation
![Page 35: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/35.jpg)
35
Bridging the semantic gap:CBIR and ontologies
Visual WordNet (GE paper)– Adding knowledge about visual characteristics
to WordNet: mobility, color, …– Build detectors for the visual features– Use visual data to prune the tree of categories
when analyzing a visual object
![Page 36: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/36.jpg)
36
Sample visual features and their mapping to WordNet
![Page 37: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/37.jpg)
37
Experiment: pruning the search for “conveyance” concepts
6 concepts foundIncluding taxi cab
12 concepts foundIncluding passenger train and commuter train
Three visual features: material, motion, environment Assumption is that these work perfectly
![Page 38: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/38.jpg)
38
Bridging the semantic gap:concept detectorsSnoek et al., TRECVID2004
– 185 hours of news video32 detectors for concepts in news video
– Through machine learningSimilarity detectors based on keywords
and visual analysisQuery interface in which these functions
can be combined
![Page 39: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/39.jpg)
39
“Concepts” for which visual detectors were built
![Page 40: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/40.jpg)
40
LSCOM lexicon: 229 - Weather
Context-specific (i.e. news broadcast) interpretation:
“Weather forecast”
![Page 41: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/41.jpg)
41
LSCOM lexicon: 110 – Female Anchor
Composite concept Alignment needed for
semantic search, e.g. with WordNet
![Page 42: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/42.jpg)
42
Natural-lang proc.automatic annotation
text stings concepts
Distributedcultuurwijzer.nl collections
OAI-based access
Reasoning supporttime/space reasoning
Web interfacesupport for web collections
Presentation facilitiessemantic presentation
device-specific
InteroperabilityXML/RDF/OWL
Scalability> 10,000,000 triples
OntologiesWordNet, AAT, TGN ULAN, Dutch labels
Search strategiessibling searchsemantic distance
Dublin Corespecializationsdumb-down
semantic annotation
DIGITAL HERITAGE COLLECTIONS
semantic search
BASELINEENHANCEDENHANCEDFEATURESFEATURES
NEWNEWFEATURESFEATURES
![Page 43: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/43.jpg)
43
![Page 44: Semantics for visual resources: use cases from e-culture](https://reader033.vdocuments.us/reader033/viewer/2022060115/5578f5ffd8b42a675b8b472a/html5/thumbnails/44.jpg)
44
Main observation
A combination of many different techniques is needed to be able to cope with the complexity of multimedia semantics– NLP, segmentation, CBIR, visual feature
detectors, visual ontologies, publicly available thesauri, thesauri mappings, dedicated reasoning techniques (time, space, default), personalization, presentation generation
Key role for user studies