data hacking and the city: ucl casa conference 2013 workshop

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 1/42

Data Hacking and th

CASA Workshop

Steven GrayThursday 26th September 2013 - 9:30 - 12:30

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 2/42

OrderoftheMorning

9:30-10:00-Showcaseofwhatwedo

10:00-11:00-Coding--Collec8ngtheData11:00-12:00-MoreCoding-VisualisingData

12:00-12:30-Ques8onandWrapup

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 3/42

Letsstartcoding!

Butfirstafewthingsweneedtosortout

SeparateintoGroups

--SocialData(BestforBeginnerstocoding)

--TransportData

--EnvironmentalData

Rules

--Don’tbeafraidtoaskques8onsifyoudon’tunderstan

--HavefunandEnjoyyourself 

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 4/42

QuickShowcaseofmywork

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 5/42

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 6/42

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 7/42

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 8/42

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 9/42

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 10/42

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 11/42

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 12/42

Letsstartcoding!

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 13/42

Mac--InstallXcodeCommandlinetoolsforyourMacOSVersion

--h$ps://developer.apple.com/downloads/ --InstallHomebrew(hp://brew.sh/)

--Runonterminal:

ruby-e"$(curl-fsSLhps://raw.github.com/mxcl/homebrew

--Followhp://madebyhoundstooth.com/blog/install-node-with-homebrew-on-os-x/

--brewinstallnode

NPMwillbeinstalledatthesame8measnode!

InstallNodeJS+NPM

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 14/42

Linux(Ubuntu)

--Followhps://github.com/joyent/node/wiki/Installing-Node.js-via-p

Windows

--Downloadandfollowinstruc8ons

InstallNodeJS+NPM

hp://nodejs.org/download/

OpenNode.jsfromStartMenu-Programs-Node.js-Node.jsCommandP

Orsearchfor“Node.jsCommandPrompt”onyoursystem(Windows,

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 15/42

InstallModulesviaNPM(AllSystems)

UseNPMtoinstallthefollowingmodulesusedbythecollector.

Checkthatyouhavenpminstalledbyrunning“npm-v”(itshouldreport1.3

Runthefollowingtoinstall

npminstalloauthcolors

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 16/42

CollecngSocialMediaData(Twier)-ARecipe

Whatyouwillneed:

1xTwierAccount

1xDeveloperKeyfromhps://dev.twier.com/

Sprinklewithsomepa8enceand8meandin8meyou’llbe

datafromTwier.

Instruc8onscomingupsoonaerasmallexplanta8on.

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 17/42

Understandingwhat’shappening

http://dev.twitter.com/doc

User Pro!les Tweet Database

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 18/42

   H   T   T   P 

  G   E   T

    R  e  q 

  u  e  s   t

R    e   s   u   l    t   s  

{"results":[

{"text":"@twitterap

  "to_user_id":396524

  "to_user":"TwitterA

  "from_user":"jkoum "

  " metadata":{

  "result_type":"pop

  "recent_retweets":

},

  "id":1478555574,

  "from_user_id":1833

  "iso_language_code"

  ....}

}

Returns tweets that:• match speci!ed query•

located in Lat/Lon box• Historical Tweets• User Pro!le Information• Return Public Timeline

Service is rate limited

JSON or A

MakingRequeststoTwier

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 19/42

Authencaon

2MethodsofAuthen8ca8on

--BasicHTTPAuthen8ca8on

--oAuth(MoreComplex

MostmoderndaywebservicesuseoAuth(includingTwierandGoogleSe

Keepsinforma8onsafefromthewrongpeoplebutalsoallowsprovidersto

iftheyareusingtoomanyresourcesormaking.

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 20/42

SohowdoesoAuthwork

Ser

Applica8on

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 21/42

SohowdoesoAuthwork

Ser

Applica8on

RequestaSessionTokenPassaIDandaSecret

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 22/42

SohowdoesoAuthwork

Ser

Applica8on

RequestaSessionTokenPassaIDandaSecret

AsksLoginwithIDServergivesaPinnumberforsession

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 23/42

SohowdoesoAuthwork

Ser

Applica8on

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 24/42

SohowdoesoAuthwork

Ser

Applica8on

Loginissuccessful:)

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 25/42

SohowdoesoAuthwork

Ser

Applica8on

Loginissuccessful:)

SendsAppTokensTheseneedtobesaved!

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 26/42

SohowdoesoAuthwork

Ser

Applica8on

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 27/42

SohowdoesoAuthwork

Ser

Applica8on

RequestusingtokensServerchecksifit’svalid

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 28/42

SohowdoesoAuthwork

Ser

Applica8on

RequestusingtokensServerchecksifit’svalid

DataReturned:)

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 29/42

Gotohps://dev.twier.com/

LoginusingyourcurrentTwieraccount

Clickyourprofilepicture(toprightcorner)andclickMyApplicaons

ClickCreateNewApplicaon

Fillinyourdetailsfortheapp

You’llbepresentedwithyourConsumerKeyandConsumerSecret

Savethesekeysforthenextstep

CollecngSocialMediaData(Twier)CreangyourTwierApplicaonKeys

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 30/42

LetsCollectsomedata

DownloadtheCollector

hps://gist.github.com/sjg/6712144

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 31/42

LetsCollectsomedata

Aerdownloadingthescripts8ckinyourkeysandsecretinthescriptbetwe

quota8onmarksandsavethefile

ConsumerKey

ConsumerSecre

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 32/42

LetsCollectsomedata

Copythescriptintoafolderthatyoucanrunthescriptfromandnavigatet

folderinyoursystemterminal.

Youcanedittheloca8onofthecollectorbychangingthesevaluesinthefi

Torunthecollectortype:

nodetwier_collect.js

“SearchTag”,“La8tude”,

-NB:“”willreturnalltw

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 33/42

LetsCollectsomedata

Ifallgoeswellandeverythingisworkingyoushouldseethefollowingscree

Gotothelinkinabrowser,logintotwierandtypethepinnumberin.Pre

andtweetswillstartcollec8nginafilecalled“tweets.csv”inthesamefold

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 34/42

Visualisingtheoutput

fi i li i h h ll d

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 35/42

Gotohps://mapsengine.google.com

ClickonNewMap

Nameyourmapbyclickingon“UntledMap”Click“Import”anduploadyourCSVfile

ClickBaseMaptochangethemapstyle

CreangyourfirstVisualisaonwiththeDataCollected

C fi Vi li i h h D C ll d

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 36/42

CreangyourfirstVisualisaonwiththeDataCollected

Oncethefilehasuploaded,choosethecorrectLa8tudeandLongitudecolu

thenclickCon8nue.ClicktheIDfield(leover)asthemarker8tleandpre

Theupdatedoutputfilewill,hopefully,autodetectthelat/lngfields.

C fi t Vi li ith th D t C ll t d

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 37/42

CreangyourfirstVisualisaonwiththeDataCollected

Styleyourmarkersbypressingthepaintbucket(highlightedinred)andsel

circleshapeandcolour.Experimentwithstylingyourmap.

C fi t Vi li ith th D t C ll t d

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 38/42

CreangyourfirstVisualisaonwiththeDataCollected

Ifallwentwellyourmapwilllooksomethinglikethis.Whynotshareitwit

world!

C fi t Vi li ith th D t C ll t d

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 39/42

CreangyourfirstVisualisaonwiththeDataCollected

Andyoucanclickthemarkersandseethetweets

Different Sources you can use to Visualise your data

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 40/42

DifferentSourcesyoucanusetoVisualiseyourdata

GoogleFusionTables

-hp://www.google.com/drive/apps.html#fusiontables

GoogleMapEngine

-hps://mapsengine.google.com

GoogleEarth

-hp://www.google.com/earth/index.html

MapTube

-hp://www.maptube.org

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 41/42

hp://download.bigdatatoolkit.org

h k

7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop

http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 42/42

Thanks

Steven Graysteven.gray@ucl.ac.uk

@frogo+StevenGray