synchronizing multi-user photo galleries with mrf
TRANSCRIPT
Synchronizing Multi-User Photo Galleries with MRF
Emanuele Sansone, Giulia Boato Minh-Son Dao DISI, University of Trento UIT-HCM Trento, Italy HCMC,
Viet Nam [email protected], [email protected] [email protected]
Synchronization and Clustering
Multiple Galleries
Feature Extraction
Offset Estimation
Synchronized Galleries
Stereo Matching I
Stereo camera
Acquired Images
Stereo Matching II
Observed node = pixel intensity
Latent node = possible disparity for a given node
Potentials associated to each link in the network
Inference algorithm
Disparity
Stereo Matching: Model
• Pixels Photos• Pixel Intensitities Image features (Timestamp, GPS, SURF, color
histograms)• States: possible diparities States: possible offsets
Stereo Matching vs. Photo Galleries Synchronization
Synchronization: States
Synchronization: Potentials
Synchronization: Potentials
Synchronization: Potentials
Synchronization: Potentials
Synchronization: Potentials
Synchronization: Potentials
Synchronization: Final Model
Synchronization: Inference
Clustering
• K-means algorithm• 3 runs
Clustering
• K-means algorithm• 3 runs
Color Structure Descriptors(CSD - 64 values) + Local Binary Patterns 3x3 (LBP - 18 values each block)
Clustering
• K-means algorithm• 3 runs
Color Structure Descriptors(CSD - 64 values) + Local Binary Patterns 3x3 (LBP - 18 values each block)
PCA Color Structure Descriptors (6 values)
Clustering
• K-means algorithm• 3 runs
Color Structure Descriptors(CSD - 64 values) + Local Binary Patterns 3x3 (LBP - 18 values each block)
PCA Color Structure Descriptors (6 values)
PCA Color Structure Descriptors (6 values) + TIME
Results: Synchronization
Dataset Precision Accuracy
Vancouver 0.35 0.86
London 0.25 0.89
Results: Synchronization
Dataset Precision Accuracy
Vancouver 0.35 0.86
London 0.25 0.89
• Good performance in terms of accuracy
Results: Synchronization
Dataset Precision Accuracy
Vancouver 0.35 0.86
London 0.25 0.89
• Good performance in terms of accuracy• Only 1/4 galleries correctly synchronized
Results: ClusteringDataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885
Results: ClusteringDataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885
• Performance are decreased by using only color descriptors
Results: ClusteringDataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885
• Performance are decreased by using only color descriptors• In the Vancouver dataset time doesn’t help in clustering
Results: ClusteringDataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885
• Performance are decreased by using only color descriptors• In the Vancouver dataset time doesn’t help in clustering• In the London dataset time becomes a more reliable feature
Lessons Learned
Synchronization:Only 1/4 galleries correctly synchronized
Lessons Learned
Synchronization:Only 1/4 galleries correctly synchronized• new features (e.g. text tags, social network data)
Lessons Learned
Synchronization:Only 1/4 galleries correctly synchronized• new features (e.g. text tags, social network data)• “no association” between a pair of images
Lessons Learned
Synchronization:Only 1/4 galleries correctly synchronized• new features (e.g. text tags, social network data)• “no association” between a pair of images
Clustering:Time is important!
Thank you