image search: then and now
TRANSCRIPT
![Page 1: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/1.jpg)
Image Search Then and Now
Integrated Knowledge Solutionsiksincyahoocom
sikrishangmailcomiksincwordpresscom
Outlinebull Introductionbull Image = Content + Contextbull Content Based Image Retrieval (CBIR)bull Bridging the Semantic Gapbull Using Social Interactions for Retrievalbull Where do we go from here
What is Image Search
bull Image search means retrieving images from an image database that satisfy the userrsquos need
bull The user need may be expressed in the following waysndash Keywords or text describing the image contentndash An exemplar image
bull Other names for image searchndash Image retrievalndash Image similarity searchndash Content based image retrieval (CBIR)
Document Search Not a New Problem
Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang
The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC
However EarlierFew Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 2: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/2.jpg)
Outlinebull Introductionbull Image = Content + Contextbull Content Based Image Retrieval (CBIR)bull Bridging the Semantic Gapbull Using Social Interactions for Retrievalbull Where do we go from here
What is Image Search
bull Image search means retrieving images from an image database that satisfy the userrsquos need
bull The user need may be expressed in the following waysndash Keywords or text describing the image contentndash An exemplar image
bull Other names for image searchndash Image retrievalndash Image similarity searchndash Content based image retrieval (CBIR)
Document Search Not a New Problem
Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang
The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC
However EarlierFew Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 3: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/3.jpg)
What is Image Search
bull Image search means retrieving images from an image database that satisfy the userrsquos need
bull The user need may be expressed in the following waysndash Keywords or text describing the image contentndash An exemplar image
bull Other names for image searchndash Image retrievalndash Image similarity searchndash Content based image retrieval (CBIR)
Document Search Not a New Problem
Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang
The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC
However EarlierFew Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 4: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/4.jpg)
Document Search Not a New Problem
Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang
The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC
However EarlierFew Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 5: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/5.jpg)
Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang
The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC
However EarlierFew Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 6: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/6.jpg)
The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC
However EarlierFew Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 7: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/7.jpg)
However EarlierFew Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 8: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/8.jpg)
But Now a Days
No Distinction Between Document Producers and Consumers
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 9: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/9.jpg)
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily
Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday
Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday
Instagram has over 20 billion photos About 60 million photos are uploaded everyday
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 10: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/10.jpg)
An image now a days is not just a picture but it is a picture with thousand words
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 11: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/11.jpg)
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 12: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/12.jpg)
So image retrieval should benefit from the contextual component if
present
How
But first let us look at image retrieval from the content
perspective only
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 13: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/13.jpg)
QBICsignal similarity
Concept semantic similarity
Concept plus context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 14: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/14.jpg)
A Typical QBIC Type Image Retrieval System
Feature Extraction
FeaturesMedia Collection
Indexing amp Matching
Query Feature Extraction
Retrieved Results
Relevance Feedback
Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 15: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/15.jpg)
Semantic Gap
Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept
Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000
httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 16: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/16.jpg)
Semantic Gap
Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts
Visually dissimilar images representing the same concept
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 17: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/17.jpg)
Semantic Gap Challenge
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 18: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/18.jpg)
How to Bridge the Semantic GapManual annotation
Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features
Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 19: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/19.jpg)
Crowdsourcing for Manual Annotation
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 20: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/20.jpg)
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 21: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/21.jpg)
Example of Image Search using Keywords
Search result in 2014The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 22: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/22.jpg)
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 23: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/23.jpg)
Example of Image Search using Keywords
Search result in 2014
Again the results are better organized in sub-categories
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 24: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/24.jpg)
Exploiting Context An Example
Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 25: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/25.jpg)
Machine Learning of Image Concepts
bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 26: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/26.jpg)
Feature Extraction Issues
Whole image based features Easy to use but not very effective
Region based features Both regular region structure and segmented regions are popular
Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 27: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/27.jpg)
Scale Invariant Feature Transform (SIFT) Descriptors
SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values
httpwwwvlfeatorg
D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 28: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/28.jpg)
Learning Image Concepts
bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used
bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 29: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/29.jpg)
VQ Based Learning Classifier
TestImage
Best CodebookLabel
Water Codebook
Sky Codebook
Fire Codebook
Mustafa amp Sethi (2004)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 30: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/30.jpg)
httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf
Bag of Words Approach
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 31: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/31.jpg)
Bag of Words Representation of Images
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 32: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/32.jpg)
Co-occurrence of Bag of Words
ImageCollection
Edge AnalysisImages
Collection of Binary Image
Blocks
Clustering
Local Feature
Descriptors(Codewords)
CodewordRepresentation
Of Images
Co-occurrenceMatrices of
Local Features
ComputeDistances
ImageDistanceMatrix
PathfinderNetwork
Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 33: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/33.jpg)
Co-occurrence of BoW
Original image
Representation byfeature indices (cluster membership)
Co-occurrence matrix
)()(max)( ABhBAhBAH
))max(min()( AaBbbaBAh
Hausdorff metric
Manhattan distance
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 34: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/34.jpg)
Notice how similar images are placed together in the graph
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 35: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/35.jpg)
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 36: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/36.jpg)
Project
httplabelmecsailmitedu
Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 37: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/37.jpg)
IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 38: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/38.jpg)
Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities
From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)
Image Classification via Probabilistic Modeling
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 39: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/39.jpg)
Image = Content + Context
TagsCherryblossomJapantownSanFranciscoPeacePagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 40: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/40.jpg)
Tagging
All time most popular tags at Flickr
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 41: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/41.jpg)
About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works
with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 42: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/42.jpg)
Why Not Use Social Tags for Retrieval
Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list
Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 43: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/43.jpg)
Tag Recommendation using TagsCo-occurrences
Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 44: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/44.jpg)
Tag Recommendation using TagsCo-occurrences and Visual Similarity
Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)
Given a target image and initial tags use the existing tagged images to suggest tags for the target image
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 45: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/45.jpg)
Tag Ranking
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 46: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/46.jpg)
Tag Ranking Another Approach
Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 47: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/47.jpg)
How to Compute Tag Similarity
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 48: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/48.jpg)
Tag Recommendation After Tag Ranking
bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 49: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/49.jpg)
Tag Completion
The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure
Wu and Jain IEEE-PAMI JANUARY 2011
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 50: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/50.jpg)
What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations
Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 51: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/51.jpg)
Tag History amp Social Interaction Features
Tag history features are based on the tags the user has used in the past
Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters
X Chen amp H Shin ICDM 2010
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 52: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/52.jpg)
Current Status of Image Searchbull Extensive interest as evident from conferences journals and
special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked
collections are gaining more tractionbull Integration of content and context through tags and
comments is receiving increasing attention to help improve retrieval
bull Killer applications are beginning to emerge as visual search gains prominence
bull Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 53: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/53.jpg)
Performance Evaluation Efforts
ImageCLEF2013 - Annotation Task
- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified
- A lot of label Noise inside the training set due to the automatic label extraction from websites
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 54: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/54.jpg)
Performance Evaluation Efforts
TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 55: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/55.jpg)
Application Examples
Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 56: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/56.jpg)
CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets
represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or
image regions of interestndash Carry our search in real-time via use of scalable
computational architectures
Extraction from Image repositories based upon
spatial information
Analysis of datain the digital domain
hellip001011010111010111
Resultant Surface Map orgallery of matching images
or
Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics
Department of PathologyUniversity of Michigan Health System
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 57: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/57.jpg)
Medical Image Retrieval
Text ldquoFind all the cases in which a tumor decrease in size
for less than three month post treatment then resumed a growth pattern after that periodrdquo
QUERY
Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo
+Medical image
QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED ONTOLOGY
GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS Textual query i - Indexes
MEDICAL ONTOLOGY
TEXT QUERYCONCEPTS EXTRACTION
Verification
Fusion
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 58: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/58.jpg)
Image Search Products
httpwwwpicalikecomproductssimilarity-searchphp
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 59: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/59.jpg)
Image Search Products
httpwwwpcssocom
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 60: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/60.jpg)
Image Search Products
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 61: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/61.jpg)
Image Search Products
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 62: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/62.jpg)
httpviralimagentuagr
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 63: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/63.jpg)
Take Home Message
bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future
bull Multimodalcross-modal retrieval is gaining importance
bull Approaches combining social search and visual search techniques are expected to gain prominence
bull Crowdsourcing is a cheap and effective way of tagging media
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 64: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/64.jpg)
Acknowledgement
bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that
bull Also want to thank my present and past students and collaborators
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-
![Page 65: Image Search: Then and Now](https://reader038.vdocuments.us/reader038/viewer/2022102323/54c689634a7959163d8b4572/html5/thumbnails/65.jpg)
Questions
Email iksincyahoocomEmail sikrishangmailcom
- Image Search Then and Now
- Outline
- What is Image Search
- Document Search Not a New Problem
- Slide 5
- Slide 6
- However Earlier
- But Now a Days
- Slide 9
- Some Relevant Numbers
- Slide 11
- Image = Content + Context
- Slide 13
- History of Image Retrieval
- A Typical QBIC Type Image Retrieval System
- Slide 16
- Slide 17
- Slide 18
- Semantic Gap
- Semantic Gap Challenge
- How to Bridge the Semantic Gap
- Crowdsourcing for Manual Annotation
- Slide 23
- Example of Image Search using Keywords
- Example of Image Search using Keywords (2)
- Example of Image Search using Keywords (3)
- Example of Image Search using Keywords (4)
- Exploiting Context An Example
- Machine Learning of Image Concepts
- Feature Extraction Issues
- Scale Invariant Feature Transform (SIFT) Descriptors
- Learning Image Concepts
- VQ Based Learning Classifier
- Bag of Words Approach
- Bag of Words Representation of Images
- Co-occurrence of Bag of Words
- Co-occurrence of BoW
- Slide 38
- Slide 39
- Slide 40
- Object Detectors for Image Concepts
- Project
- Image Category Classifiers Examples
- Image Classification via Probabilistic Modeling
- Image = Content + Context (2)
- Tagging
- About Tags
- Why Not Use Social Tags for Retrieval
- Tag Recommendation using Tags Co-occurrences
- Tag Recommendation using Tags Co-occurrences and Visual Similar
- Tag Ranking
- Tag Ranking Another Approach
- How to Compute Tag Similarity
- Slide 54
- Tag Recommendation After Tag Ranking
- Tag Completion
- What about Taggers amp Commenters
- Tag History amp Social Interaction Features
- Current Status of Image Search
- Performance Evaluation Efforts
- Performance Evaluation Efforts (2)
- Application Examples
- CBIR for Whole Slide Imageries
- Medical Image Retrieval
- Image Search Products
- Image Search Products (2)
- Image Search Products (3)
- Image Search Products (4)
- Slide 69
- Slide 70
- Take Home Message
- Acknowledgement
- Questions Email iksincyahoocom Email sikrishangmailcom
-