Download - Descriptive Semantic Image Retrieval
![Page 1: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/1.jpg)
Descriptive Semantic Image Retrieval
David NortonDerral Heath
![Page 2: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/2.jpg)
Motivation Retrieve an image based on a descriptive
query: “Find me an image that is red, dark, scary, and
beautiful”
![Page 3: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/3.jpg)
Content-Based Image Retrieval Retrieve an image strictly from image features
color texture shape
General semantic based image retrieval is hard “Find me a picture of a piranha”
![Page 4: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/4.jpg)
Emotional Semantic Image Retrieval Query images matching emotional words or
word-pairs “Find me a happy picture”
Usually adjectives
Usually a small orthogonal subset of terms
Query via single words (or pairs)
![Page 5: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/5.jpg)
Descriptive Semantic Image Retrieval Open to all descriptive words
Query via any number of words “Find me an image that is red, dark, scary, and
beautiful”
![Page 6: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/6.jpg)
Three Components Extraction of image features
Semantic representation of image
Mapping between visuals and language
![Page 7: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/7.jpg)
Extraction of Image Features Color (12)
Average RGB values Color count
Texture (50) Entropy
Shape (10) Eccentricity
![Page 8: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/8.jpg)
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
114.0887 99.0219 118.648 20 0.1332 8.924632 0.109940296
113.5899 98.5232 118.145 18 0.1411 8.200488 0.101902564
113.6192 98.5358 118.153 20 0.1511 8.015762 0.082569348
![Page 9: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/9.jpg)
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
146.4833 75.8135 1.547 5 0.5259 7.805696 0.064152
145.9641 75.2931 1 4 0.5317 7.654775 0.073264
146.0377 75.3628 1.054 4 0.5496 7.502806 0.054083
![Page 10: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/10.jpg)
Semantic Representation of Image How do we obtain a description?
What is a descriptive word?
What are the features?
![Page 11: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/11.jpg)
User Input Interface
![Page 12: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/12.jpg)
Bright has 11 Senses 1. (17) bright -- (emitting or reflecting light readily or in large amounts; "the sun was bright and hot"; "a
bright sunlit room") 2. (6) bright, brilliant, vivid -- (having striking color; "bright dress"; "brilliant tapestries"; "a bird with vivid
plumage") 3. (5) bright, smart -- (characterized by quickness and ease in learning; "some children are brighter in one
subject than another"; "smart children talk earlier than the average") 4. (3) bright -- (having lots of light either natural or artificial; "the room was bright and airy"; "a stage bright
with spotlights") 5. (1) bright, burnished, lustrous, shining, shiny -- (made smooth and bright by or as if by rubbing;
reflecting a sheen or glow; "bright silver candlesticks"; "a burnished brass knocker"; "she brushed her hair until it fell in lustrous auburn waves"; "rows of shining glasses"; "shiny black patents")
6. (1) bright -- (splendid; "the bright stars of stage and screen"; "a bright moment in history"; "the bright pageantry of court")
7. undimmed, bright -- (not made dim or less bright; "undimmed headlights"; "surprisingly the curtain started to rise while the houselights were still undimmed")
8. bright, brilliant -- (clear and sharp and ringing; "the bright sound of the trumpet section"; "the brilliant sound of the trumpets")
9. bright -- (characterized by happiness or gladness; "bright faces"; "all the world seems bright and gay") 10. bright, shining, shiny, sunshiny, sunny -- (abounding with sunlight; "a bright sunny day"; "one shining
morning"- John Muir; "when it is warm and shiny") 11. bright, promising -- (full or promise; "had a bright future in publishing"; "the scandal threatened an
abrupt end to a promising political career")
![Page 13: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/13.jpg)
Narrowing Down the Feature Space Interface:
Adjectives from WordNet
Restrict characters
Reduce available senses
![Page 14: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/14.jpg)
Narrowing Down the Feature Space Post Processing:
Use Synsets
Frequent synsets
Fit ORM ontology lexicons
![Page 15: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/15.jpg)
Image ORM Ontology
![Page 16: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/16.jpg)
Mapping between visuals and language Series of Neural Networks
Bayes Net
Fuzzy Logic
![Page 17: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/17.jpg)
Evaluation Let machine label images
Let humans label images
Let different humans evaluate machine and human labels
Compare evaluations
![Page 18: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/18.jpg)
Related Work Aesthetic Visual Quality Assessment of
Paintings (2009) Congcong Li and Tsuhan Chen
Labeled impressionistic style landscape paintings as ‘high’ or ‘low’ quality using machine learning.
Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition (October 2008) Ritendra Datta, Jia Li, and James Z. Wang
Overview of research involving predicting the quality class, score, and emotional label of photographs.
![Page 19: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/19.jpg)
Related Work A Survey on Emotional Semantic Image
Retrieval (2008) Weining Wang and Qianhua He
Surveys ongoing Emotional Semantic Image Retrieval research.
Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction (October 2006) Wang Wei-ning, Yu Ying-lin, and Jiang Sheng-ming
Labeled paintings with 12 emotional word pairs. Psychological research involved in choosing word pairs.
![Page 20: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/20.jpg)
Further Motivation Augment the study of human perception and
cognition.
Establish a linguistic-visual foundation for an artificially creative artist.
![Page 21: Descriptive Semantic Image Retrieval](https://reader036.vdocuments.us/reader036/viewer/2022062410/56816014550346895dcf168b/html5/thumbnails/21.jpg)
Questions?