![Page 1: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/1.jpg)
Image Classification over Visual Tree
Jianping FanDept of Computer ScienceUNC-Charlotte, NC 28223
www.cs.uncc.edu/~jfan
![Page 2: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/2.jpg)
1. Research Motivation
Large-Scale Visual Recognition
Inter-concept Visual Correlations rather than independency
![Page 3: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/3.jpg)
1. Research Motivation
Large-Scale Visual Recognition: Challenges
We need to learn large amounts of classifiers for large-scale visual recognition!
Some object classes and image concepts are visually-related and hard to be discriminated!
Some object classes and image concepts may have huge inner-concept visual diversity!
![Page 4: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/4.jpg)
1. Research Motivation
Large-Scale Visual Recognition: Challenges
Huge inner-concept visual diversity ---simple models may not work, but using complex models may overlap with others! Huge inter-concept visual similarity ---training complexity will increase for
distinguishing visually-related concepts! Huge computational cost ---thousands of inter-related classifiers should be trained jointly!
![Page 5: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/5.jpg)
2. Collecting Large-Scale Training Images
Flickr Images & Other Image Sites
Keywords for image crawling
![Page 6: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/6.jpg)
Visual Feature Extraction
2. Collecting Large-Scale Training Images
![Page 7: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/7.jpg)
Synonymous Concepts: Visual Similarity
CVPR2010
![Page 8: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/8.jpg)
Synonymous Concepts: Visual Similarity
![Page 9: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/9.jpg)
Ambiguous Concept: Visual Diversity
2. Collecting Large-Scale Training Images
![Page 10: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/10.jpg)
Ambiguous Concept: Visual Diversity
2. Collecting Large-Scale Training Images
![Page 11: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/11.jpg)
Junk Image Filtering
subject to:
Decision function:
R
OutliersMajority
IEEE Trans. CSVT 2009
2. Collecting Large-Scale Training Images
![Page 12: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/12.jpg)
Junk Image Filtering
2. Collecting Large-Scale Training Images
![Page 13: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/13.jpg)
Junk Image Filtering
2. Collecting Large-Scale Training Images
![Page 14: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/14.jpg)
Most text terms are weakly related or even irrelevant to web images in the same webpage
tigerbig catSoutheast AsiaRussianChineseBengalSiberianIndochineseSouth Chinese….
Noise image
![Page 15: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/15.jpg)
Text-Image Alignment for Web Image Indexing
WWW2010, PR2014
2. Large-Scale Image Preparation
![Page 16: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/16.jpg)
Informative Image Extraction
Noise image
Two simple rules: Aspect ratio (>0.2 or <5) Image size (min(width,
height) > 60 pixel)
Not perfect but produce satisfied results
Unsupervised and computationally efficient
![Page 17: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/17.jpg)
Webpage Segmentation Surrounding Text Extraction
Visual-based algorithm precise but expensive [Cai et al. MSR-TR’03]
DOM (Document Object Model) based method computationally efficient
![Page 18: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/18.jpg)
Webpage Segmentation Surrounding Text Extraction
Visual-based algorithm precise but expensive [Cai et al. MSR-TR’03]
DOM (Document Object Model) based method computationally efficient
![Page 19: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/19.jpg)
Text-Image Alignment for Web Image Indexing
WWW2010, PR2014
Near-duplicates share similar semantics!
![Page 20: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/20.jpg)
Text-Image Alignment for Web Image Indexing
WWW2010, PR2014
![Page 21: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/21.jpg)
Duplicate Detection
CVPR2012
2. Collecting Large-Scale Training Images
Duplicates may mislead classifier training tools!
![Page 22: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/22.jpg)
Duplicate Detection
2. Collecting Large-Scale Training Images
![Page 23: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/23.jpg)
Automatic Tag-Instance Alignment
Missing Tag Prediction
ACM MM 2010
CVPR 2012
2. Collecting Large-Scale Training Images
![Page 24: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/24.jpg)
3. Visual Concept Network
Why we need visual concept network?
---concept ontology, object co-occurrence network, ….
Common space: classifier training & concept detection
---visual feature space rather than label space or concept space
We need to characterize inter-concept visual correlations rather than others!
ACM MM2009
Inter-related learning task determination
![Page 25: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/25.jpg)
3. Visual Concept Network
![Page 26: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/26.jpg)
3. Visual Concept Network
![Page 27: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/27.jpg)
3. Visual Concept Network
![Page 28: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/28.jpg)
3. Visual Concept Network
![Page 29: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/29.jpg)
3. Visual Concept Network
![Page 30: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/30.jpg)
Label Tree for Efficient Classification
1, 2, 3, 4
1
1, 3
2, 4
3 2 4
Label 1: cat Label 2: mini vanLabel 3: dog Label 4: fire truck
It is a fire truck!
Number of dot products needed in the label tree: 1 + 1 = 2
Number of dot product needed in a flat approach:1 + 1 + 1 + 1 = 4
[Bengio et al. NIPS’2010]
![Page 31: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/31.jpg)
Construction of Label Tree
SVM 1
Testing Samples
SVM 2
SVM N
…
Confusion Matrix
1,
2,
…,1
0
1,5,6
2, 4,8,9
3,7,
10
2, 8
4,9
4
9
Training N one-vs-rest SVMs is very expensive
The SVMs could be unreliable Huge sample imbalance Negative samples could
mislead the classifier training
![Page 32: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/32.jpg)
Visual Similarity Matrix
Result is based on ImageNet data set of 1000 categories
![Page 33: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/33.jpg)
4. Visual Tree Construction: Hierarchical Clustering
Root
![Page 34: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/34.jpg)
4. Visual Tree Construction: Hierarchical Clustering
![Page 35: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/35.jpg)
4. Visual Tree Construction: Hierarchical Clustering
![Page 36: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/36.jpg)
4. Visual Tree Construction: Hierarchical Clustering
![Page 37: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/37.jpg)
4. Visual Tree Construction: Hierarchical Clustering
![Page 38: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/38.jpg)
4. Visual Tree Construction: Hierarchical Clustering
![Page 39: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/39.jpg)
Bag-of-Words (BoW)
![Page 40: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/40.jpg)
To distinguish visually-similar categories, dictionaries with strong discrimination is critical
Joint dictionary learning
CVPR 2012TPAMI2013
5. Joint Dictionary Learning for Discriminative Image Representation
![Page 41: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/41.jpg)
7. Large-Scale Classifier Training
IEEE Trans. IP 2011, IEEE Trans. PAMI 2014, PR 2013
![Page 42: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/42.jpg)
Inference Model Selection for Classifier Training
7. Large-Scale Classifier Training
![Page 43: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/43.jpg)
Inference Model Selection for Classifier Training
7. Large-Scale Classifier Training
![Page 44: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/44.jpg)
7. Large-Scale Classifier Training
![Page 45: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/45.jpg)
7. Large-Scale Classifier TrainingHierarchical Organization
![Page 46: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/46.jpg)
7. Large-Scale Classifier TrainingHierarchical Organization
![Page 47: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/47.jpg)
7. Large-Scale Classifier TrainingHierarchical Organization
![Page 48: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/48.jpg)
7. Large-Scale Classifier TrainingFlat Organization
![Page 49: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/49.jpg)
8. Interactive Classifier Assessment
VAST 2011
![Page 50: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/50.jpg)
8. Interactive Classifier Assessment
![Page 51: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/51.jpg)
8. Interactive Classifier Assessment
![Page 52: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/52.jpg)
9. Some Experimental Results
![Page 53: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/53.jpg)
9. Some Experimental Results
![Page 54: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/54.jpg)
9. Some Experimental Results
![Page 55: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/55.jpg)
9. Some Experimental Results
![Page 56: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/56.jpg)
9. Some Experimental Results
![Page 57: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/57.jpg)
9. Some Experimental Results
![Page 58: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/58.jpg)
9. Some Experimental Results
![Page 59: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/59.jpg)
9. Some Experimental Results
![Page 60: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/60.jpg)
10. Conclusions & Current Work
A new image representation approach via discriminative dictionary learning;
A novel approach for semantic gap quantification; A new solution for inter-related classifier training
and large-scale visual recognition.
Performing our experiments on large-scale images !
![Page 61: Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 jfan](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfec1a28abf838cb830f/html5/thumbnails/61.jpg)