twitter community extraction by markov clustering

Download Twitter Community Extraction by Markov Clustering

If you can't read please download the document

Upload: kingsbsd

Post on 16-Apr-2017

1.156 views

Category:

Internet

1 download

Report

Download

Embed Size (px):

TRANSCRIPT

Twitter Community Extraction
(Beware of geeks bearing .gifs?)

https://github.com/kingsBSD

http://big-social-data.net/

@kingsBSD

Our Data Ourselves Project:

Giles Greenway, Tobias Blanke, Jennifer Pybus & Mark Cot.Departments of Digital Humanities & Culture, Media & Creative Industries, King's College London

Our aims are to increase our understanding of the nature and role of the data that young people produce when they use platforms and applications on their smartphones.

Social network community extraction is part of this.

Markov Clustering -MCL

Assumptions:There are clusters of Twitter users with densely connected networks of friend/follower relationships.

If you take a random walk around the network, you are likely to stay within the cluster you started in.

http://www.micans.org/mcl/

MCL -A Trivial Example

1: Build an adjacency matrix for the graph.

2: Normalize the columns to produce transition probabilities.

MCL -A Trivial Example

3: Square the matrix to get probabilities after two steps.

MCL -A Trivial Example

4: Element wise square the matrix and re-normalize.

5: Rinse and repeat until convergence.

The matrix entries will be 0 or 1. Interpret rows as: If I'm in this row node, which column nodes are credible start-points?

Does it work?

Gephi's OpenOrd layout is meant to emphasise clusters. Are nodes in the same cluster close together?

Compare with Gephi's own modularity algorithm, the Louvain method.

MCL was applied to two Twitter accounts of digital culture researchers with ~7000 once-removed friend-follower relationships.

Does it work?

MCL

Louvain

Does it work?

Why did Gephi put these two in the same modularity class?

Researchers rated clusters for both methods.

MCLLouvain

Cluster is identifiable and relevant.20%0% !

Cluster is not identifiable, but possibly relevant.37%Cluster is neither identifiable or relevant.43%

Tools

Acquire Twitter data with Twython/Celery/Redis/RabbbitMQ.

Store Twitter data with: Neo4J/Py2Neo.

Perform MCL with NumPy.

Export to Gephi with NetworkX.

Conclusions

The Louvain method works by combining smaller clusters to maximize modularity. Does the very high degree of Twitter networks harm its performance?

MCL produces highly relevant clusterings, albeit rather slowly.

ECAI 2014 poster - Unsupervised semantic clustering of Twitter hashtags

DNA methylation correlation networks in overweight and ...634125/...IFNGR2 - Interferon-Gamma receptor 2 MCL - Markov CLustering algorithm Methylome - Methylation pattern on DNA that

rEMM: Extensible Markov Model for Data Stream Clustering … · rEMM: Extensible Markov Model for ... Although the traditional Markov Chain is an excellent modeling ... Extensible

Hypergraph Partitioning and Clusteringweb.eecs.umich.edu/~imarkov/pubs/book/part_survey.pdfHypergraph Partitioning and Clustering David A. Papa and Igor L. Markov University of Michigan,

AUTHORS FR: MARKOV, G.L. TO: MARKOV, V.A. · Title: AUTHORS FR: MARKOV, G.L. TO: MARKOV, V.A. Subject: AUTHORS FR: MARKOV, G.L. TO: MARKOV, V.A. Keywords: Tho IL-28 kircrtSt (Technical

Probabilistic Space-Time Analysis of Human Mobility Patterns · Key–Words: Human Mobility Model, Space-Time Analysis, Location Clustering, Continuous Time Markov Chain, Individual

Clustering Hidden Markov Models with Variational HEMmixture model, variational approximation, time-series classi cation 1. Introduction The hidden Markov model (HMM) (Rabiner and Juang,

Opinion Mining on Twitter Data using Unsupervised Learning ... · clustering tweets as recommended (thumps up) or not recommended (thumps down) by using a most popular clustering

Optical measurements association using optimized boundary ... · Markov clustering algorithm . 7th European Conference on Space Debris, ESA, Apr 2017, Darmstadt, Germany. hal-01511781

Graph Clustering Algorithms - pages.di.unipi.itpages.di.unipi.it/marino/cluster18.pdf · Markov Clustering Algorithm 1 Normalize the adjacency matrix. 2 Expand by taking the e-th

Clustering Sequences with Hidden Markov Models · Clustering Sequences with Hidden Markov Models Padhraic Smyth Information and Computer Science University of California, Irvine CA

New clustering proposals for Twitter based on the Louvain algorithm · 2017-11-02 · New clustering proposals for Twitter based on the Louvain algorithm Author: Jordi Colomer Matutano

Markov Chains Regular Markov Chains Absorbing Markov Chains

OPTICAL MEASUREMENTS ASSOCIATION USING ......OPTICAL MEASUREMENTS ASSOCIATION USING OPTIMIZED BOUNDARY VALUE INITIAL ORBIT DETERMINATION COUPLED WITH MARKOV CLUSTERING ALGORITHM C

Affinity Clustering: Hierarchical Clustering at Scalepapers.nips.cc/paper/7262-affinity-clustering-hierarchical-clustering-at-scale.pdf · Afﬁnity Clustering: Hierarchical Clustering

Clustering Hidden Markov Models with Variational HEMeceweb.ucsd.edu/~gert/papers/coviello14a.pdf · mixture model, variational approximation, time-series classi cation 1. Introduction

Hypergraph Partitioning and Clusteringimarkov/pubs/book/part_survey.pdfHypergraph Partitioning and Clustering David A. Papa and Igor L. Markov University of Michigan, EECS Department,

Bayesian clustering of replicated time-course gene ..._Publications_files...Bayesian clustering, mixture model, random effects, Dirichlet process, Chinese restaurant process, Markov-chain

Activating Transcription Factor 4 Links Metabolic Stress ... · unique Entrez IDs. Protein networks were built running the Markov clustering algorithm using the functional and physical

Clustering Sequences with Hidden Markov Modelspapers.nips.cc/paper/1217-clustering-sequences-with...Clustering Sequences with Hidden Markov Models Padhraic Smyth Information and Computer

Semantic-based topic detection using Markov decision processescicip.sxu.edu.cn/docs/2019-04/7dc8b2df87ff4899a6cb... · including text classiﬁcation and clustering, information retrieval,

Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System

Deconvolutive Clustering of Markov States › ~axk › ecml_deconv.pdf · model [17]. This was previously used e.g. in language modelling [17,11] and bibliometric analysis [7]. The

G-PCCA: Spectral Clustering for Non-reversible Markov Chains · Abstract Spectral clustering methods are based on solving eigenvalue prob-lems for the identi cation of clusters, e.g.,

rEMM: Extensible Markov Model for Data Stream Clustering in R · The extensible Markov model (EMM) adds the temporal component to data stream clustering by superimposing a dynamically

SCIENTIFIC ABSTRACT MARKOV, P.U. - MARKOV, V.A. · Title: SCIENTIFIC ABSTRACT MARKOV, P.U. - MARKOV, V.A. Subject: SCIENTIFIC ABSTRACT MARKOV, P.U. - MARKOV, V.A. Keywords: 303!5

Markov Systems and Markov Decision Processes...States, Actions, Observations Passive Controlled Fully Observable Markov Model Markov Decision Process (MDP) Hidden State Hidden Markov

Markov Chains. Summary Markov Chains Discrete Time Markov Chains Homogeneous and non-homogeneous Markov chains Transient and steady state Markov chains

Taxi Service Strategy Optimization with Data Clustering ... · with Data Clustering and Markov Decision Processes Kexin Zhang A report submitted for the course COMP4650 Advanced Computing

Research on Online Digital Cultures — Community Extraction from Twitter Networks by Markov Clustering

Hidden Markov Model Clustering of Acoustic Data Matthew Butler

9 Markov chains and Hidden Markov Models - Freie … · 9 Markov chains and Hidden Markov Models We will discuss: Markov chains Hidden Markov Models (HMMs) Algorithms: Viterbi, forward,

Clustering IV. Outline Impossibility theorem for clustering Density-based clustering and subspace clustering Bi-clustering or co-clustering

I-Louvain: An Attributed Graph Clustering Method · techniques like divisive algorithms based on the minimum cut [14], the spectral meth-ods [34] or the Markov Clustering algorithm

SCIENTIFIC ABSTRACT MARKOV, K.K. - MARKOV, K.K. · Title: SCIENTIFIC ABSTRACT MARKOV, K.K. - MARKOV, K.K. Subject: SCIENTIFIC ABSTRACT MARKOV, K.K. - MARKOV, K.K. Keywords: k. a r