advanced r programming - lecture 5 · lecture 5. 6/ 39 input and outputbasic i/ocloud storageweb...

1/ 39

Input and output Basic I/O Cloud storage web APIs web scraping Shiny Relational Databases

Advanced R Programming - Lecture 5

Krzysztof Bartoszek(slides by Leif Jonsson and Mans Magnusson)

Linkoping University

[email protected]

18 September 2017

K. Bartoszek (STIMA LiU) STIMA LiU

Lecture 5

2/ 39


Today

Input and output

Basic I/O

Cloud storage

web APIs: Lab

web scraping

Shiny

Relational Databases


Lecture 5

3/ 39


Questions since last time?


Lecture 5

4/ 39


Input and output

Format, localization


Lecture 5

5/ 39


Input and output

Format, localization and encoding...... hell!

http://www.joelonsoftware.com/articles/Unicode.html

The Absolute Minimum Every Software Developer Absolutely, Positively

Must Know About Unicode and Character Sets (No Excuses!)

Unicode defines codes for all (?) characters—multiple encodings(for a given language only small fraction of characters used)Content-Type tag for HTMLBUT e–mail, .txt, .csv


Lecture 5

http://www.joelonsoftware.com/articles/Unicode.html

6/ 39


”Formats”


Lecture 5

7/ 39


Localization

own Computer Cloud Storagelocal network web pageslocal database web scraping

web APIsremote database

Table: Local - Remote


Lecture 5

8/ 39


Files on your computer

# Input simple data

read.table ()

read.csv()

read.csv2()

load()

# Output simple data

write.table ()

write.csv()

write.csv2()

save()


Lecture 5

9/ 39


More complex formats

software/data packageExcel XLConnectSAS, SPSS, STATA, ... foreignXML xmlJSON (GeoJSON) rjsonio, RJSONDocuments tmMaps spImages raster

Table: Format - R package


Lecture 5

10/ 39


Cloud storage

Table: Local - Remote


Lecture 5

11/ 39


Why?

Robust

Backups

Cloud computing

can be tricky in the beginning

but

how about safety?

But control on what is going on?

BUT requires internet connection


Lecture 5

11/ 39


Why?

Robust

Backups

Cloud computing


but how about safety?


BUT

requires internet connection


Lecture 5

11/ 39


Why?

Robust

Backups

Cloud computing


but how about safety?


BUT requires internet connection


Lecture 5

12/ 39


Localization

Arbitrary data Structured data


Lecture 5

13/ 39


API Packages

Remote packageGeneral downloaderGitHub repmis, downloaderDropbox rdropAmazon RAmazonS3Google Docs googlesheets


Lecture 5

14/ 39


web APIs

application program interface using http

”contract to ’get data’ online”

more and more common

examples:

github

Riksdagen

Statistics Sweden


Lecture 5

15/ 39


RESTful

Basic principles:

Data is returned (JSON / XML)

Each specific data has its own URI

Communication is based on HTTP verbs


Lecture 5

16/ 39


Hypertext Transfer Protocol (http)


Lecture 5

17/ 39


Hypertext Transfer Protocol (http)


Lecture 5

18/ 39


Verbs

Verb DescriptionGET Get ”data” from server.POST Post ”data” to server (to get something)PUT Update ”data” on serverDELETE Delete resource on server


Lecture 5

19/ 39


Status codes

Code Description1XX Information from server2XX Yay! Gimme’ data!3XX Redirections4XX You failed5XX Server failed


Lecture 5

20/ 39


Example REST API’s


Lecture 5

http://www.linkoping.se/open/data/Luftkvalitet/

Linkoping Luftkvalitet API

https://developers.google.com/maps/documentation/geocoding/intro

Google Map Geocode API

http://www.linkoping.se/open/data/Luftkvalitet/

https://developers.google.com/maps/documentation/geocoding/intro

21/ 39


Common API formats

JavaScript Object Notation (JSON)Think of named lists in R

R Packages: RJSONIO, rjsonlite

Extensible Markup Language (XML)Older format (using nodes)

xpathR Packages: XML


Lecture 5

22/ 39


JSON{

"firstName": "John",

"lastName": "Smith",

"age": 25,

"address": {

"streetAddress": "21 2nd Street",

"city": "New York",

"state": "NY",

"postalCode": "10021"

},

"phoneNumber": [

{ "type": "home", "number": "212 555" },

{ "type": "fax", "number": "646 555" }

],

"newSubscription": false ,

"companyName": null

}K. Bartoszek (STIMA LiU) STIMA LiU

Lecture 5

23/ 39


XML

<?xml version="1.0" encoding="utf -8"?>

<wikimedia >

<projects >

<project name="Wikipedia" launch="2001 -01 -05">

<editions >

<edition language="English">en.wikipedia.org</edition >

<edition language="German">de.wikipedia.org</edition >

<edition language="French">fr.wikipedia.org</edition >

<edition language="Polish">pl.wikipedia.org</edition >

<edition language="Spanish">es.wikipedia.org</edition >

</editions >

</project >

<project name="Wiktionary" launch="2002 -12 -12">

<editions >

<edition language="English">en.wiktionary.org</edition >

<edition language="French">fr.wiktionary.org</edition >

<edition language="Vietnamese">vi.wiktionary.org</edition >

<edition language="Turkish">tr.wiktionary.org</edition >

<edition language="Spanish">es.wiktionary.org</edition >

</editions >

</project >

</projects >

</wikimedia >


Lecture 5

24/ 39


web scraping

Unstructured http(s) data

Often HTML format

Spiders / scraping / web crawlers

Basics behind search engines


Lecture 5

25/ 39


HTML

<!DOCTYPE html>

<html>

<head>

<title >This is a title</title>

</head>

<body>

<p>Hello world!</p>

</body>

</html>


Lecture 5

26/ 39


(har)rvest

JavaScript Object Notation (JSON)Simplify spider activity

Download dataParse dataFollow linksFill out formsStore crawling history


Lecture 5

27/ 39


Difficulties and bad spiders

Scraping is fragile!Difficulties and bad spiderswww.domain.se/robot.txtPoliteness

robot trapsjavascriptdelays


Lecture 5

28/ 39


Shiny?

Interactive dashboards made easy

online or local

R as ”backend”


Lecture 5

29/ 39


Shiny?


Lecture 5

https://www.rstudio.com/products/shiny/shiny-user-showcase/

Shiny Examples

https://www.rstudio.com/products/shiny/shiny-user-showcase/

30/ 39


How it works

Application

Reactive

modify using HTML

MyAppName/server.R

MyAppName/ui.R

server.R define working directory


Lecture 5

31/ 39


Shiny Example

library(shiny)

# Examples with code

runExample("01 _hello")

runExample("03 _reactivity")


Lecture 5

32/ 39


Publish Shiny

locallyzip-file in cloud

github (see runGithub() )


Lecture 5

33/ 39


Publish Shiny

locallyzip-file in cloud

github (see runGithub() )

your own servershinyapps.io


Lecture 5

34/ 39


Relational Databases

Structured database in tables

local or online

query language for I/O

effective for big data

difficult to design


Lecture 5

35/ 39



Lecture 5

36/ 39



Lecture 5

37/ 39


A good database

Can be difficult to design ?

No duplicatesNo redundanciesEasy to update”Normal forms”

Easy to query


Lecture 5

37/ 39


A good database

Can be difficult to design ?No duplicates

No redundanciesEasy to update”Normal forms”

Easy to query


Lecture 5

38/ 39


Using databases from R

Database system R packageODBC (Microsoft Access) RODBCPostgreSQL RPostgresqlOracle ROracleMySQL RMySqlMongoDB rmongodb

Table: Database - R package


Lecture 5

39/ 39


The End... for today.Questions?

See you next time!


Lecture 5