Collecting Photo Stream in Node.js on Heroku

@stomita (ja)@shinichitomita (en)

Who am I ?

• CEO at Mashmatrix, Inc.

• JavaScript Developer

• Formerly (almost forgot)

• Identity (OpenID/SAML/SSO)

• JavaEE

• Oracle

• Ex-Salesforce.com (2006/12-2007/12)

@stomita

Objective

• Collect and store photos tweeted in the world

• Add metadata to photos manually by crowd

• Category ? (food/people/landscape)

• Venue ? (foursquare)

Twitter Streaming API

https://dev.twitter.com/docs/streaming-api

Photo Stream

• Set tracking keywords to filter tweets with photo uploading services URL

• e.g. tracks=lockerz,plixi,tweetphoto,twitpic,yfrog,photozou,twitgoo,instagr

• Extract photo URLs by pre-given rule

• e.g. http://twitpic.com/7wiwlw => http://twitpic.com/show/thumb/7wiwlw

Architecture

Streaming JSON

Twitter-Node

https://github.com/technoweenie/twitter-node

Visualization

• Present photo stream to users in real-time

• Not our primary goal, but just interesting

• Don’t think, FEEL.

tweet-photo-stream

http://tweet-photo-stream.herokuapp.com/https://github.com/stomita/tweet-photo-stream

socket.io

• A Node.js library, enabling push messaging to browsers

• Stream Transport Supported :

• WebSocket/FlashSocket/XHR-Multipart/XHR-polling/JSONP-polling

• Used as fallback of SocketStream

Architecture

Streaming socket.io

Similar Work - Twitcaps

http://twitcaps.com/

Twitcaps - Not Real-time

Delayed Streaming Ajax Polling

Issue

• Heroku Node.js doesn’t support WebSocket yet (as of Dec 2011)

• Fallbacks to XHR-polling

• c.f. Joyent No.de SmartMachine

End