Twitter Power-User HOWTO

@steveblackmon

✦Visualize content and connections

✦Ad-hoc queries

✦Exploration

✦Focus on specific profiles and posts

Goals

2

✦Assemble data about your twitter network

✦Analyze and Assess

✦…

✦Profit?

3

Plan

Data Visualization

4

Data Repositories

5

Why ?✦ Handles de-duplication and

merging

✦ Speaks JSON and REST

✦ Scales vertically and horizontally

✦ Because

6

Why ?✦ Handles de-duplication and

merging

✦ Relationships central to data model

✦ Built and optimized for graph traversal queries

✦ Because

7

SupportingTechnologies

8

Why ?

✦Micro-service friendly

✦Works the same way across OS, distribution, cloud provider

✦Build, ship, run anywhere with 3-5 shell commands (build, [tag,] push, [pull,] run)

9

Why ?✦Assemble pipelines of modular data providers and database writers

✦Normalize your social databases to community standards

✦Why waste time hunting for and wrangling new SDKs?

10

11

I exist. Who am I?

Part 1: ID

12

Capture own history

Who am I?

13

Capture userstream

Who am I?

14

Other people do too.

Who are they?

15

Part 2: EGO

Collect friend/follower

network

Who are they?

16

I am but one piece of a vast

society.What is my

place?

Part 3: SUPER-EGO

17

What is my place?Research interesting accounts

DEMO19

20

21

22

23

24

25

HOWTO

26

✦ Using linux? • apt-get / yum install docker.io

✦ Using a mac? ✦ Install Boot2Docker ✦ http://boot2docker.io/

• boot2docker ssh• ifconfig• note the bridged IP (eth1)

27

Docker

✦docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e PLUGINS=mobz/elasticsearch-head,elasticsearch/marvel/latest quay.io/steveblackmon/elasticsearch:1.4.4

✦docker run -d --name kibana -p 5601:5601 --link elasticsearch:elasticsearch quay.io/steveblackmon/kibana:4.0.0

✦docker run -d --name neo4j -p 7474:7474 quay.io/steveblackmon/neo4j:2.2.0.M4

28

Installation

✦ docker run -d --name linkurious -p 3000:3000 —link neo4j:neo4j quay.io/steveblackmon/linkurious:latest

** LINKURIOUS IS NOT OPEN-SOURCE **** PURCHASE A LICENSE AT http://linkurio.us **

** 20% OFF IF YOU MENTION THIS TALK **** 30-DAY MONEY-BACK GUARANTEE **

** ACCESS TO THIS DOCKER REPO IF YOU BUY**

29

Installation (Cont.)

✦developer.twitter.com

✦mytwitterid.com

✦mkdir streams

✦cd streams

Configuration (Part 0)

30

Configuration (Part 1)

twitter.userstream.json1. Add twitter oauth credentials2. Add own userid to twitter.follow3. Set elasticsearch.hosts4. Set graph.hostname

twitter.history.json5.Add twitter oauth credentials6.Add own userid to twitter.info7.Set elasticsearch.hosts

✦docker run -d --name twitter.userstream -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.userstream.json -cp /twitter-0.2-SNAPSHOT.jar twitter.UserstreamPipeline

✦docker run -d --name twitter.history -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline

Execution (Part 1)

32

twitter.friends.json1. Add twitter oauth credentials2. Add own userid to twitter.info3. Set graph.hostname

twitter.followers.json1. Add twitter oauth credentials2. Add own userid to twitter.info3. Set graph.hostname

Configuration (Part 2)

33

✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline

✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.followers.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline

Execution (Part 2)

34

Configuration (Part 3)Export IDs of your connections with Cypher

•MATCH (v:page {handle:'handle'})-[:follow]-(x:page) RETURN DISTINCT SUBSTRING(x.id, 11);

twitter.connections.history.json1.Add twitter oauth credentials2.Add connection ids to twitter.info3.Set elasticsearch.hosts

twitter.connections.friends.json4.Add twitter oauth credentials 5.Add connection ids to twitter.info6.Set graph.hostname

35

✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline

✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline

Execution (Part 3)

36

@steveblackmon37

FIN