graph-based multimodal clustering for social event detection in large collections of images
DESCRIPTION
Presentation by my colleague Giorgos Petkos of our paper at the Multimedia Modeling conference (MMM2014) in Dublin.TRANSCRIPT
![Page 1: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/1.jpg)
MMM 2014
Graph-based multimodal clustering for social event
detection in large collections of images
Georgios Petkos, Symeon Papadopoulos, Emmanouil Schinas, Yiannis KompatsiarisInformation Technologies Institute (ITI)Centre for Research & Technologies Hellas (CERTH)
![Page 2: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/2.jpg)
MMM 2014 Georgios Petkos et al.#2
Overview
• The problem of social event detection• Existing approaches• Proposed approach• Evaluation• Summary & future work
![Page 3: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/3.jpg)
MMM 2014 Georgios Petkos et al.
the problem
![Page 4: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/4.jpg)
MMM 2014 Georgios Petkos et al.#4
entertainment
personal
news
wedding / birthday / drinks
concert / play / sports
demonstration / riot / speech
Social events?
Attended by people and represented by multimedia content shared online
![Page 5: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/5.jpg)
MMM 2014 Georgios Petkos et al.#5
Pope Francis
Pope Benedict
2007: iPhone release
2008: Android release
2010: iPad release
http://petapixel.com/2013/03/14/a-starry-sea-of-cameras-at-the-unveiling-of-pope-francis/
![Page 6: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/6.jpg)
MMM 2014 Georgios Petkos et al.
Social event detection
Social event detection involves the automatic organization of a multimedia collection C into groups of items, each (group) of which corresponds to a distinct event. Can be treated as a multimodal clustering problemCOLLECTION
EVENT DETECTION
EVENT SET
E1
E2
EN
#6
![Page 7: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/7.jpg)
MMM 2014 Georgios Petkos et al.
existing approaches
![Page 8: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/8.jpg)
MMM 2014 Georgios Petkos et al.
Supervised event detection
• Rationale: use a large number of “known” event assignments to “learn” how to identify “same event” / “same cluster” relationships
Two variants:• Item-to-item: learn whether two items belong to the same
event cluster or not. – Model Input: the set of per modality distances between two images.
• Item-to-cluster: learn whether a new item belongs to a given event cluster or not. – Model input: the set of per modality distances between an image and
a prototype representation of the event.
#8
![Page 9: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/9.jpg)
MMM 2014 Georgios Petkos et al.
Utilizing the “same event” model for clustering
• Item-to-item: – (Incremental). For each incoming image, average all item-to-item SE
scores for all items in each cluster. Assign to best-matching cluster if average above threshold or create new cluster (Becker et. al.).
– (Batch). Compute all item-item SE scores between each image and all other images and form an indicator vector. Cluster indicator vectors (Petkos et. al.).
• Item-to-cluster: – (Incremental). For each cluster maintain a multimodal representation.
Compute SE score between each incoming item and the existing prototype event representations. Assign to best-matching cluster if above threshold or create new cluster (Becker et. al). Alternatively use a second model for deciding if a new cluster should be added or not (Reuter et. al.).
#9
![Page 10: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/10.jpg)
MMM 2014 Georgios Petkos et al.
proposed approach
![Page 11: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/11.jpg)
MMM 2014 Georgios Petkos et al.
Overview of proposed approach
#11
• Item-to-item SE model utilized.• Candidate neighbours selection step (first appears in (Reuter et. al)) using a set of per modality indexes. • Graph representation.• Community detection on graph. Two variants of the algorithm:
• Batch: SCAN• Incremental: QCA
![Page 12: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/12.jpg)
MMM 2014 Georgios Petkos et al.
Proposed approach: advantages
#12
• Item-to-cluster methods may suffer from incorrect prototype representations (due to averaging). • Candidate neighbours selection step makes the application of the method much more scalable.• Graph representation: in order to introduce a scalable item-to-item approach without averaging.
![Page 13: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/13.jpg)
MMM 2014 Georgios Petkos et al.
evaluation
![Page 14: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/14.jpg)
MMM 2014 Georgios Petkos et al.
Evaluation setup
• Used the dataset of the 2012 SED task of MediaEval• Ground truth: 7,779 photos clustered around 149
events (18 technical, 79 soccer, 52 Indignados)• Assess the following aspects:
– accuracy of same-event classification– compare clustering quality between item-to-cluster and
the two versions of item-to-item (batch & incremental)– measure contributions of different features– study generalization abilities of same event model
![Page 15: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/15.jpg)
MMM 2014 Georgios Petkos et al.
Evaluation setup
Features:• Uploader identity.• Actual image content:
– GIST– SURF, aggregated using the VLAD scheme
• Textual features: title, description and tags. Either a TF-IDF or a BM25 weighting scheme is utilized.
• Time of media creation.• Location, when available (geodesic distance).
Appropriate indices are utilized in order to rapidly fetch the candidate neighbours for each modality.
![Page 16: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/16.jpg)
MMM 2014 Georgios Petkos et al.
Evaluation: SE accuracy & clustering quality
• Same event classification accuracy 98.58% (SVM)– 10K pos/neg training, 10K pos/neg testing (random)
• Clustering quality (NMI): 30/119 training/testing events [10 random splits]– Incremental same or better than batch– Item-to-item better than item-to-cluster (significant at 0.95 confidence)
• When non-event photos enter the dataset, NMI degrades quickly
BATCH INCREMENTAL ITEM-TO-CLUSTER
AVG 0.924 0.934 0.898
STD 0.019 0.021 0.027
NON-EVENT BATCH INCREMENTAL ITEM-TO-CLUSTER
5% 0.4824 0.5164 0.3954
10% 0.3421 0.3683 0.2899
* In the second table, results were obtained using sed2011 for training and sed2012 for testing.
*
![Page 17: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/17.jpg)
MMM 2014 Georgios Petkos et al.
Evaluation: contribution of features
• Same experiments using limited sets of features
• Repeating the same experiments without the use of blocking led to significantly worse results– e.g. 0.030 for visual, 0.7148 for textual
• Time is an extremely important feature
FEATUERS BATCH INCREMENTAL
VISUAL 0.8020 ∓ 0.0193 0.8179 ∓ 0.0151
TEXTUAL 0.7925 ∓ 0.0255 0.7792 ∓ 0.0310
VISUAL+TIME 0.9244 ∓ 0.0195 0.9360 ∓ 0.0183
TEXTUAL+TIME 0.9016 ∓ 0.0173 0.9049 ∓ 0.0209
![Page 18: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/18.jpg)
MMM 2014 Georgios Petkos et al.
Evaluation: generalizing same event model
• Train using one event type > test on a different one• In most cases negative impact• In few cases, performance is very high!
BATCH
soccer technical Indignados
soccer - 0.8658 0.8494
technical 0.7967 - 0.8977
Indignados 0.9645 0.8456 -
INCREMENTAL
soccer technical Indignados
soccer - 0.8892 0.8667
technical 0.7661 - 0.7735
Indignados 0.9845 0.8482 -
![Page 19: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/19.jpg)
MMM 2014 Georgios Petkos et al.
summary & future work
![Page 20: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/20.jpg)
MMM 2014 Georgios Petkos et al.
Summary
• Scalable item-to-item multimodal clustering approach for SED
• Key characteristics:– Item-to-item “same event” model– Candidate neighbor selection – Organization of “same event” relationships to a graph– Efficient graph clustering algorithms: SCAN (batch) / QCA
(incremental)
• In general though, item-to-item approaches are less scalable than item-to-cluster approaches
#20
![Page 21: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/21.jpg)
MMM 2014 Georgios Petkos et al.
Future work
• Extend method so that non-event images are properly handled
• Multiple sources of multimedia
• The MediaEval datasets are somewhat limited. Investigate the effect of crawling / image collection to the quality of results
#21
![Page 22: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/22.jpg)
MMM 2014 Georgios Petkos et al.
thank you!
questions?Acknowledgements
![Page 23: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/23.jpg)
MMM 2014 Georgios Petkos et al.
online clustering of same-event graph
QCA maintains community structure incrementally following graph change operations: node & edge addition (removal operations not applicable in same event graph): based on the concept of community attraction forces
A
B
C
D
X new nodenew edge
Cu
Cw
Cz
force from Cu to Cz
force from Cz to Cu
• Depending on a test (computed based on local graph structure), community structure could remain the same, X assigned to Cu or A to Cz.
• If A is assigned to Cu, all its neighbours will be checked for potential reassignment.
#23
![Page 24: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/24.jpg)
MMM 2014 Georgios Petkos et al.
graph clustering :: SCAN
outlier
hub
(μ,ε)- corestructural similarity
• resilient to spurious links (e.g. visual links that connect unrelated images)
• very fast (scales linearly to the number of edges)• leaves less-/ and over-connected items out of the clustering
#24
![Page 25: Graph-based multimodal clustering for social event detection in large collections of images](https://reader036.vdocuments.us/reader036/viewer/2022070315/5550b45cb4c905ff618b4a2c/html5/thumbnails/25.jpg)
MMM 2014 Georgios Petkos et al.
References
• Reuter, T., & Cimiano, P. (2012, June). Event-based classification of social media streams. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 22). ACM.
• Petkos, G., Papadopoulos, S., & Kompatsiaris, Y. (2012). Social event detection using multimodal clustering and integrating supervisory signals. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 23). ACM.
• Becker, H., Naaman, M. & Gravano, L.. Learning similarity metrics for event identification in social media. In Proceedings of the third ACM International Conference on Web search and Data Mining, WSDM ’10, pages 291–300, New York.
• Nguyen, N., Dinh, T., Xuan, Y., & Thai, M.. Adaptive algorithms for detecting community structure in dynamic social networks. In INFOCOM 2011. 30th IEEE International Conference on Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, 10-15 April 2011, Shanghai, China, pages 2282–2290. IEEE, 2011.
• Xu, X., Yuruk, N., Feng, Z. & Schweiger, T.. SCAN: a structural clustering algorithm for networks. In Proceedings of the 13th ACM SIGKDD, KDD ’07, pages 824–833, NY, USA, 2007. ACM
#25