![Page 1: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/1.jpg)
Twitter Power-User HOWTO
@steveblackmon
![Page 2: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/2.jpg)
✦Visualize content and connections
✦Ad-hoc queries
✦Exploration
✦Focus on specific profiles and posts
Goals
2
![Page 3: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/3.jpg)
✦Assemble data about your twitter network
✦Analyze and Assess
✦…
✦Profit?
3
Plan
![Page 4: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/4.jpg)
Data Visualization
4
Data Visualization
![Page 5: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/5.jpg)
Data Repositories
5
![Page 6: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/6.jpg)
Why ?✦ Handles de-duplication and merging
✦ Speaks JSON and REST
✦ Scales vertically and horizontally
✦ Because
6
![Page 7: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/7.jpg)
Why ?✦ Handles de-duplication and merging
✦ Relationships central to data model
✦ Built and optimized for graph traversal queries
✦ Because
7
![Page 8: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/8.jpg)
Supporting
Technologies
8
![Page 9: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/9.jpg)
Why ?
✦Micro-service friendly
✦Works the same way across OS, distribution, cloud provider
✦Build, ship, run anywhere with 3-5 shell commands (build, [tag,] push, [pull,] run)
9
![Page 10: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/10.jpg)
Why ?✦Assemble pipelines of modular data providers and database writers
✦Normalize your social databases to community standards
✦Why waste time hunting for and wrangling new SDKs?
10
![Page 11: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/11.jpg)
11
![Page 12: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/12.jpg)
I exist. Who am I?
Part 1: ID
12
![Page 13: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/13.jpg)
Capture own history
Who am I?
13
![Page 14: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/14.jpg)
Capture userstream
Who am I?
14
![Page 15: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/15.jpg)
Other people do too.
Who are they?
15
Part 2: EGO
![Page 16: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/16.jpg)
Collect friend/follower network
Who are they?
16
![Page 17: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/17.jpg)
I am but one piece of a vast
society.
What is my place?
Part 3: SUPER-EGO
17
![Page 18: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/18.jpg)
What is my place?Research interesting accounts
![Page 19: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/19.jpg)
DEMO19
![Page 20: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/20.jpg)
20
![Page 21: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/21.jpg)
21
![Page 22: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/22.jpg)
22
![Page 23: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/23.jpg)
23
![Page 24: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/24.jpg)
24
![Page 25: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/25.jpg)
25
![Page 26: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/26.jpg)
HOWTO26
![Page 27: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/27.jpg)
✦ Using linux?
• apt-get / yum install docker.io ✦ Using a mac?
✦ Install Boot2Docker
✦ http://boot2docker.io/ • boot2docker ssh
• ifconfig
• note the bridged IP (eth1)
27
Docker
![Page 28: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/28.jpg)
✦docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e PLUGINS=mobz/elasticsearch-head,elasticsearch/marvel/latest quay.io/steveblackmon/elasticsearch:1.4.4
✦docker run -d --name kibana -p 5601:5601 --link elasticsearch:elasticsearch quay.io/steveblackmon/kibana:4.0.0
✦docker run -d --name neo4j -p 7474:7474 quay.io/steveblackmon/neo4j:2.2.0.M4
28
Installation
![Page 29: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/29.jpg)
✦ docker run -d --name linkurious -p 3000:3000 —link neo4j:neo4j quay.io/steveblackmon/linkurious:latest
** LINKURIOUS IS NOT OPEN-SOURCE **** PURCHASE A LICENSE AT http://linkurio.us **
** 20% OFF IF YOU MENTION THIS TALK **** 30-DAY MONEY-BACK GUARANTEE **
** ACCESS TO THIS DOCKER REPO IF YOU BUY**
29
Installation (Cont.)
![Page 30: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/30.jpg)
✦developer.twitter.com
✦mytwitterid.com
✦mkdir streams
✦cd streams
Configuration
(Part 0)
30
![Page 31: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/31.jpg)
Configuration (Part 1)
twitter.userstream.json 1. Add twitter oauth credentials
2. Add own userid to twitter.follow
3. Set elasticsearch.hosts
4. Set graph.hostname
twitter.history.json 1.Add twitter oauth credentials
2.Add own userid to twitter.info
3.Set elasticsearch.hosts
![Page 32: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/32.jpg)
✦docker run -d --name twitter.userstream -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.userstream.json -cp /twitter-0.2-SNAPSHOT.jar twitter.UserstreamPipeline
✦docker run -d --name twitter.history -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline
Execution (Part 1)
32
![Page 33: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/33.jpg)
twitter.friends.json 1. Add twitter oauth credentials
2. Add own userid to twitter.info
3. Set graph.hostname
twitter.followers.json 1. Add twitter oauth credentials
2. Add own userid to twitter.info
3. Set graph.hostname
Configuration (Part 2)
33
![Page 34: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/34.jpg)
✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.followers.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
Execution (Part 2)
34
![Page 35: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/35.jpg)
Configuration (Part 3)Export IDs of your connections with Cypher
•MATCH (v:page {handle:'handle'})-[:follow]-(x:page) RETURN DISTINCT SUBSTRING(x.id, 11);
twitter.connections.history.json 1.Add twitter oauth credentials
2.Add connection ids to twitter.info
3.Set elasticsearch.hosts
twitter.connections.friends.json 4.Add twitter oauth credentials
5.Add connection ids to twitter.info
6.Set graph.hostname35
![Page 36: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/36.jpg)
✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline
✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
Execution (Part 3)
36
![Page 37: Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on specific profiles and posts Goals 2 Assemble data about your twitter network](https://reader034.vdocuments.us/reader034/viewer/2022043020/5f3c416c10a98b1e1a3d1e37/html5/thumbnails/37.jpg)
@steveblackmon37
FIN