data gathering with the web observatory
TRANSCRIPT
![Page 1: Data Gathering with The Web Observatory](https://reader034.vdocuments.us/reader034/viewer/2022051521/587a79371a28abf0468b59cb/html5/thumbnails/1.jpg)
THE WEB OBSERVATORY DATA GATHERING WITH
EUGENE SIOW & XIN WANG
29 JANUARY 2016
![Page 2: Data Gathering with The Web Observatory](https://reader034.vdocuments.us/reader034/viewer/2022051521/587a79371a28abf0468b59cb/html5/thumbnails/2.jpg)
WHAT IS THE WEB OBSERVATORY?
SEARCH + ACCESS
![Page 3: Data Gathering with The Web Observatory](https://reader034.vdocuments.us/reader034/viewer/2022051521/587a79371a28abf0468b59cb/html5/thumbnails/3.jpg)
DATA | WO | APP OPEN
PRIVATE
NoSQL
SQL STREAMS
LINKED DATA
JS
PYTHON NODE
![Page 4: Data Gathering with The Web Observatory](https://reader034.vdocuments.us/reader034/viewer/2022051521/587a79371a28abf0468b59cb/html5/thumbnails/4.jpg)
GATHERING DATA WITH SCRAPING
DATA ON WEBPAGES DATA I CAN USE transform ( )
![Page 5: Data Gathering with The Web Observatory](https://reader034.vdocuments.us/reader034/viewer/2022051521/587a79371a28abf0468b59cb/html5/thumbnails/5.jpg)
THE PROCESS OF SCRAPING
INVESTIGATE THE STRUCTURE OF THE PAGE
CHECK IF THERE IS AN API {APPLICATION PROGRAMMING INTERFACE}
USE CHROME’S INSPECTOR OR FIREBUG
EXTRACT, TRANSFORM, LOAD WHAT IS YOUR DESIRED END FORMAT?
![Page 6: Data Gathering with The Web Observatory](https://reader034.vdocuments.us/reader034/viewer/2022051521/587a79371a28abf0468b59cb/html5/thumbnails/6.jpg)
HANDS-ON RESOURCES
codepen.io/xgfd/pen/wMyQWb
github.com/eugenesiow/datathon2016/wiki
DATA-DRIVEN APPS USING THE WO
DATA GATHERING
webobservatory.soton.ac.uk THE SOTON WEB OBSERVATORY
BACKGROUNDS FROM THE HUBBLE SPACE TELESCOPE