data tools ecosystem for non-programmers
Post on 13-Sep-2014
2.060 views
DESCRIPTION
Presentation by @oscarmarinmiro at Itnig, November 8th 2013, about non-programming resources for getting, analyzing and visualizing dataTRANSCRIPT
Data Tools Ecosystem for non-programmers
Óscar Marín Miró@outliers_es
www.outliers.es
1
Contents at a glance
‣Why mess with Data?‣A Data Pipeline
The Data Process
Data Acquisition Tools‣Data Sources‣Google Forms & Google Docs‣Scraping tools‣Network acquisition tools‣PDF tools‣Open Refine
Data Analysis‣Excel capabilities‣Beyond excel
Data Visualization Tools‣Timeline.js‣Wordle‣Plot.ly‣Google Fusion Tables‣Gephi‣CartoDB‣MapBox‣Tilemill
Developers needed‣Size‣Real-Time‣Analysis beyond Basics‣Interactivity*
Resources
2
The Data Process
http://en.wikipedia.org/wiki/Data-driven_journalism
3
Why mess with data?
Data
Information
Knowledge
Wisdom
•Register observations
•Structured Data•Semantics•Narrative•Spatialized Data
•Personal Absortion•Action: Information application•‘Insights’
•Non transferable•Ethic knowledge application
Cog
nitiv
e C
ompl
exity
4
A Data Pipeline
AnalysisAcquisition Presentation
Data Information
•API’s•Scraping•Files•DB’s•Providers
•Text Mining•Machine Learning•Network Metrics•Stats•Filtering
Data
•Visualization•Story
5
Data Acquisition Tools
6
Data Formats
• JSON (Javascript Object Notation) - Dev
• CSV, XLS - Tabular
• XML - Old, but still used
• Web data
7
Data Acq Techniques
• Ask the user: Forms
• Download from Data Repositories
• Web data: Scrape
• Ask the provider: API access
8
Google Forms
9
Google Forms
10
Google Forms
11
Google Forms
http://www.encuesta15m.cc/subject.html
12
Google Forms
http://www.encuesta15m.cc/subject.html
13
Google Forms
http://data.elperiodico.com/
14
Google Spreadsheet
http://mashe.hawksey.info/2012/10/feeding-google-spreadsheets-exercises-in-import/
15
Google Spreadsheet
16
17
Scrape w/ Chrome Scraper
https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd
18
Scrape w/ Chrome Scraper
19
Scrape w/ Chrome Scraper
20
Scrape w/ Chrome Scraper
http://dataist.wordpress.com/2012/10/12/get-started-with-screenscraping-using-google-chromes-scraper-extension/
21
PDF Extraction: Tabula
http://tabula.nerdpower.org/
23
PDF Extraction: Tabula
http://tabula.nerdpower.org/
24
PDF Extraction: Tabula
http://tabula.nerdpower.org/
25
Network Data: Flocker
http://flocker.outliers.es/
26
Network Data: Flocker
http://flocker.outliers.es/
27
Open Refine: Cleaning Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
28
Open Refine: Cleaning Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
29
Open Refine: Cleaning Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
30
Open Refine: Cleaning Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
31
Open Refine: Cleaning Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
32
Open Refine: Cleaning Data
33
Open Refine: Cleaning Data
https://github.com/OpenRefine/OpenRefine/wiki/GREL-Functions
https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions
34
Data Analysis Tools
35
Data Analysis Tools
• Numeric analysis
• Network analysis
• Language analysis
36
Data Analysis w/ MS Excel
• Sort
• Basic Stats: Mean, Median, Sum...
• Filter
• Charts
• Pivot tables
• What-if
• Data Analysis Tool
37
Data Analysis:MS Excel‘Pivot Tables’
38
Data Analysis:MS Excel‘What-If ’
39
Data Analysis: MS Excel
Correlations, Covariance, Statistic Summary, Regression, Histogram, Factor Analysis, Fourier Analysis....
http://ire.org/media/uploads/car2013_tipsheets/excel_stats_nicar2013.pdf
40
Data Analysis: NLP??
42
Data Analysis: NLP??
43
Data Visualization Tools
http://well-formed.eigenfactor.org/
44
Data Visualization Tools
• Timelines
• Words
• Chart tools
• Networks
• Maps
45
Data Visualization ToolsTimeline JS
http://timeline.knightlab.com/
46
Data Visualization ToolsTimeliner (OKFN)
http://timeliner.okfnlabs.org
47
Data Visualization ToolsWordle
http://www.wordle.net/create
48
Data Visualization ToolsWordle
http://www.wordle.net/create
49
Data Visualization ToolsGoogle Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
54
Data Visualization ToolsGoogle Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
55
Data Visualization ToolsGoogle Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
56
Data Visualization ToolsGoogle Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
57
Data Visualization ToolsGoogle Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
58
Data Visualization Tools
http://www.tableausoftware.com/public/
59
Data Visualization ToolsCartoDB
http://cartodb.com/
http://developers.cartodb.com/tutorials/electoral_map.html
63
Data Visualization ToolsCartoDB
http://cartodb.com/http://developers.cartodb.com/tutorials/electoral_map.html
“Shapefile”
64
Developers needed
73
When you’ll need a developer
‣Size‣Real-Time‣Analysis beyond Basics‣Interactivity‣API access
74
References
75
OKFN
http://okfn.org/http://schoolofdata.org/http://ckan.org/http://2014.okfestival.org/
76
Books
78
Books
80
Other
http://itnig.net/en/events/revealing-the-power-of-data-for-non-developers
http://www.cccb.org/ca/marc-periodisme_de_dades-44600
http://www.visualisingdata.com/index.php/resources/
81
Data Sources
http://ckan.org/http://www.theguardian.com/datahttp://www.google.com/publicdata/http://data.worldbank.org/http://datahub.io/http://www.quora.com/Data/Where-can-I-find-large-datasets-open-to-the-public
82
Any questions?Thanks for your attention!
83