r and collecting internet data - university of california, berkeley

Motivation Tools we need to learn

R and Collecting Internet Data

Luis F. Campos

Department of StatisticsUniversity of California, Berkeley

February 11, 2011 - 4-5 PM - 1011 Evans Hall

Luis F. Campos UC, Berkeley



Outline1 Motivation

Internet Movie Database (IMDb)Music DatabasesYahoo! SportsMain example: Superbowl - Play by Play Data

2 Tools we need to learnXML/HTMLXML Path Query Language (XPath QL)R Programming LanguageR Package: XMLRegular Expressions (regex)Demonstration/Resources




Internet Movie Database (IMDb)

Outline1 Motivation






Internet Movie Database (IMDb)




Music Databases

Outline1 Motivation






Music Databases




Yahoo! Sports

Outline1 Motivation






Yahoo! Sports




Main example: Superbowl - Play by Play Data

Outline1 Motivation







What is actually going on in your browser?




XML/HTML

Outline1 Motivation






XML/HTML

What is XML?

XML stands for eXtensible Markup Language

markup languages: provide annotations for structure(marks) that are distinguishable from text (examples:LaTeX, html, PostScript)extensible: you can create your own marks

In XML, the structure markers are angle brackets: < >HTML could be considered a derivative of XMLHTML (HyperText Markup Language): is currently thepredominant markup language




XML/HTML

What is XML?

XML stands for eXtensible Markup Languagemarkup languages: provide annotations for structure(marks) that are distinguishable from text (examples:LaTeX, html, PostScript)

extensible: you can create your own marks





XML/HTML

What is XML?

XML stands for eXtensible Markup Languagemarkup languages: provide annotations for structure(marks) that are distinguishable from text (examples:LaTeX, html, PostScript)extensible: you can create your own marks





XML/HTML

What is XML?


In XML, the structure markers are angle brackets: < >

HTML could be considered a derivative of XMLHTML (HyperText Markup Language): is currently thepredominant markup language




XML/HTML

What is XML?


In XML, the structure markers are angle brackets: < >HTML could be considered a derivative of XML

HTML (HyperText Markup Language): is currently thepredominant markup language




XML/HTML

What is XML?






XML/HTML

What is HTML? Simply, a set of predetermined structuralmarkers.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><html>

<head><title>The document title</title>

</head><body>

<h1>Main heading</h1><p>A paragraph.</p><a href = "www.stat.berkeley.edu">Statistics Website</a>

</body></html>




XML/HTML

What is HTML? Simply, a set of predetermined structuralmarkers.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><html><head><title>The document title</title>

</head><body><h1>Main heading</h1><p>A paragraph.</p><a href = "www.stat.berkeley.edu">Statistics Website</a>

</body></html>




XML/HTML

Another useful way to view an HTML document is as a tree.




XML/HTML

We call the boxes nodes and the arrows edges

an edge goes from a to b if:<a>

<b></b>

</a>

Note: there is a unique path from the root node to anygiven node.




XML/HTML

We call the boxes nodes and the arrows edgesan edge goes from a to b if:<a>

<b></b>

</a>

Note: there is a unique path from the root node to anygiven node.




XML/HTML

Each node can have the following elements associated to it:

a required name for the node, not necessarily uniqueany number of attributesoptional text<a href = "www.stat.berkeley.edu">

Statistics Website</a>

nodename: aattribute: href with value "www.stat.berkeley.edu"text: "Statistics Website"




XML/HTML

Each node can have the following elements associated to it:a required name for the node, not necessarily unique

any number of attributesoptional text<a href = "www.stat.berkeley.edu">






XML/HTML

Each node can have the following elements associated to it:a required name for the node, not necessarily uniqueany number of attributes

optional text<a href = "www.stat.berkeley.edu">






XML/HTML

Each node can have the following elements associated to it:a required name for the node, not necessarily uniqueany number of attributesoptional text

<a href = "www.stat.berkeley.edu">Statistics Website

</a>





XML/HTML

Each node can have the following elements associated to it:a required name for the node, not necessarily uniqueany number of attributesoptional text<a href = "www.stat.berkeley.edu">Statistics Website

</a>





XML/HTML


</a>

nodename: a

attribute: href with value "www.stat.berkeley.edu"text: "Statistics Website"




XML/HTML


</a>

nodename: aattribute: href with value "www.stat.berkeley.edu"

text: "Statistics Website"




XML/HTML


</a>





XML Path Query Language (XPath QL)

Outline1 Motivation







XPath is a query language that allows you to retrieve specificnodes in a XML/HTML tree. Some of the symbols we can useto represent a path on the tree:

/ finds the root node// selects from anywhere in the tree. selects current node.. selects parent of current node@ selects attributesnodename finds location(s) of the named node






/ finds the root node

// selects from anywhere in the tree. selects current node.. selects parent of current node@ selects attributesnodename finds location(s) of the named node






/ finds the root node// selects from anywhere in the tree

. selects current node

.. selects parent of current node@ selects attributesnodename finds location(s) of the named node






/ finds the root node// selects from anywhere in the tree. selects current node

.. selects parent of current node@ selects attributesnodename finds location(s) of the named node






/ finds the root node// selects from anywhere in the tree. selects current node.. selects parent of current node

@ selects attributesnodename finds location(s) of the named node






/ finds the root node// selects from anywhere in the tree. selects current node.. selects parent of current node@ selects attributesnodename finds location(s) of the named node





To traverse to the only anchor node below, we can use:"/../body/div/a" or "//a"

If we had <a href = "stat.berkeley.edu">stats</a>:"//a[@href = ’stat.berkeley.edu’]" or "//a[text() = ’stats’]"





To traverse to the only anchor node below, we can use:"/../body/div/a" or "//a"If we had <a href = "stat.berkeley.edu">stats</a>:"//a[@href = ’stat.berkeley.edu’]" or "//a[text() = ’stats’]"




R Programming Language

Outline1 Motivation







Why R, in general?

Plenty of packages! So you can extend R to do whateveryou want.It’s free! So you can spend money on other things.

Why R, for collecting internet data?

Collecting data for statistical purposes <–> R is made forstatisticsImplementations of XPath, Regular Expressions are fairlyintuitiveThere are other ways, but they follow similar procedures





Why R, in general?Plenty of packages! So you can extend R to do whateveryou want.

It’s free! So you can spend money on other things.Why R, for collecting internet data?






Why R, in general?Plenty of packages! So you can extend R to do whateveryou want.It’s free! So you can spend money on other things.

Why R, for collecting internet data?







Why R, for collecting internet data?Collecting data for statistical purposes <–> R is made forstatistics

Implementations of XPath, Regular Expressions are fairlyintuitiveThere are other ways, but they follow similar procedures






Why R, for collecting internet data?Collecting data for statistical purposes <–> R is made forstatisticsImplementations of XPath, Regular Expressions are fairlyintuitive

There are other ways, but they follow similar procedures






Why R, for collecting internet data?Collecting data for statistical purposes <–> R is made forstatisticsImplementations of XPath, Regular Expressions are fairlyintuitiveThere are other ways, but they follow similar procedures





R: briefly

R is a command line interpreter language (interpretscommands you type)

We’ll be using functions provided by packages (XML) andthe base package. Example of calling a function below:It’s usually a good idea to name all named arguments whencalling a function (otherwise R will just use them in order)Notice that if fun has any output it will be stored in x

x <- fun(unnamed, arg = named)





R: briefly

R is a command line interpreter language (interpretscommands you type)We’ll be using functions provided by packages (XML) andthe base package. Example of calling a function below:

It’s usually a good idea to name all named arguments whencalling a function (otherwise R will just use them in order)Notice that if fun has any output it will be stored in x






R: briefly

R is a command line interpreter language (interpretscommands you type)We’ll be using functions provided by packages (XML) andthe base package. Example of calling a function below:It’s usually a good idea to name all named arguments whencalling a function (otherwise R will just use them in order)

Notice that if fun has any output it will be stored in x






R: briefly

R is a command line interpreter language (interpretscommands you type)We’ll be using functions provided by packages (XML) andthe base package. Example of calling a function below:It’s usually a good idea to name all named arguments whencalling a function (otherwise R will just use them in order)Notice that if fun has any output it will be stored in x





R Package: XML

Outline1 Motivation






R Package: XML

Created by Prof. Duncan Temple Lang. This package is acollection of functions that will enable us to work withXML/HTML tree structures. Some of the functions we will useare:

htmlTreeParse: gets html page specified and creates aninternal tree structure. We can then use XPath to traversethe tree>url="http://en.wikipedia.org/wiki/Main_Page">doc=htmlTreeParse(url,useInternalNodes = TRUE)

getNodeSet: takes document name and XPathinstructions about the node we’re looking for:> x1 = getNodeSet(doc, "//div[@class = ’bd’]")> x2 = getNodeSet(doc, "//a")




R Package: XML

x1 will contain the location of all the nodes named "div"who have a class attribute with value "bd"

x2 will contain locations of ALL the anchor nodes (containlinks)! Which may or may not be useful (unless you’reGoogle).

xmlParent: gets location of the parent of the specifiednode:> xmlParent(x1[[1]])

xmlChildren, xmlAncestors: gets all children orancestors of a given node.xmlValue: gets the text of a given node.




R Package: XML




xmlChildren, xmlAncestors: gets all children orancestors of a given node.

xmlValue: gets the text of a given node.




R Package: XML




xmlChildren, xmlAncestors: gets all children orancestors of a given node.xmlValue: gets the text of a given node.




R Package: XML

One shortcut:If you know you want to get a table element from an htmlfile

a table is a very specific html element!

readHTMLTable will retrieve all table nodes and transformthem into a workable form for you!So, we’re done right?!Correct! No, not everything is stored in a table!This is why we learned all this!




R Package: XML



readHTMLTable will retrieve all table nodes and transformthem into a workable form for you!

So, we’re done right?!Correct! No, not everything is stored in a table!This is why we learned all this!




R Package: XML



readHTMLTable will retrieve all table nodes and transformthem into a workable form for you!So, we’re done right?!

Correct! No, not everything is stored in a table!This is why we learned all this!




R Package: XML



readHTMLTable will retrieve all table nodes and transformthem into a workable form for you!So, we’re done right?!Correct! No, not everything is stored in a table!

This is why we learned all this!




R Package: XML



readHTMLTable will retrieve all table nodes and transformthem into a workable form for you!So, we’re done right?!Correct! No, not everything is stored in a table!This is why we learned all this!




Regular Expressions (regex)

Outline1 Motivation







Provide a framework for finding specific text in a body oftext for a list of possible texts.

We can specify the string we are looking for with thefollowing quantifiers:

OR operator - | - ex: "gray|grey" finds all instances of either"grey" or "gray"Grouping (parenthesis)- ( ) - ex: "gr(a|e)y". Same statementas aboveQuantifiers (number of), given after the character or group:

question mark - ? - zero or one - ex: "colou?r" finds "color"or "colour"asterisk - * - zero or more - ex: "ab*c" finds "ac", "abc","abbc", ...plus sign - + - one or more - ex: "ab+c" finds all from aboveexcept "ac"





Provide a framework for finding specific text in a body oftext for a list of possible texts.We can specify the string we are looking for with thefollowing quantifiers:








OR operator - | - ex: "gray|grey" finds all instances of either"grey" or "gray"

Grouping (parenthesis)- ( ) - ex: "gr(a|e)y". Same statementas aboveQuantifiers (number of), given after the character or group:







OR operator - | - ex: "gray|grey" finds all instances of either"grey" or "gray"Grouping (parenthesis)- ( ) - ex: "gr(a|e)y". Same statementas above

Quantifiers (number of), given after the character or group:








question mark - ? - zero or one - ex: "colou?r" finds "color"or "colour"

asterisk - * - zero or more - ex: "ab*c" finds "ac", "abc","abbc", ...plus sign - + - one or more - ex: "ab+c" finds all from aboveexcept "ac"







question mark - ? - zero or one - ex: "colou?r" finds "color"or "colour"asterisk - * - zero or more - ex: "ab*c" finds "ac", "abc","abbc", ...

plus sign - + - one or more - ex: "ab+c" finds all from aboveexcept "ac"





POSIX Expressions:. - matches ANY single character - ex: "a.c" finds "aac","abc", "acc",...

[ ] - matches a single character from the brackets - ex:"gr[ea]y" finds "grey" or "gray"These are different from parenthesis since ranges can bespecified in square brackets: a-z means any lower casealphanumeric character.There are a ton more. See reference.





POSIX Expressions:. - matches ANY single character - ex: "a.c" finds "aac","abc", "acc",...[ ] - matches a single character from the brackets - ex:"gr[ea]y" finds "grey" or "gray"

These are different from parenthesis since ranges can bespecified in square brackets: a-z means any lower casealphanumeric character.There are a ton more. See reference.





POSIX Expressions:. - matches ANY single character - ex: "a.c" finds "aac","abc", "acc",...[ ] - matches a single character from the brackets - ex:"gr[ea]y" finds "grey" or "gray"These are different from parenthesis since ranges can bespecified in square brackets: a-z means any lower casealphanumeric character.

There are a ton more. See reference.





POSIX Expressions:. - matches ANY single character - ex: "a.c" finds "aac","abc", "acc",...[ ] - matches a single character from the brackets - ex:"gr[ea]y" finds "grey" or "gray"These are different from parenthesis since ranges can bespecified in square brackets: a-z means any lower casealphanumeric character.There are a ton more. See reference.





Regular Expressions in R are used in the following functions:grep finds a pattern in a list of candidates:>grep("abc",c("abcdsf", "fabcasda", "cba"))[1] 1 2

gsub replaces a pattern with another in a list ofcandidates:>gsub("abc","CANDY",c("abcdsf", "fabca", "cba"))[1] "CANDYdsf" "fCANDYa" "cba"





strsplit>strsplit(c("abcdsf", "fabcasda", "cba"), "abc")[[1]][1] "" "dsf"

[[2]][1] "f" "asda"

[[3]][1] "cba"




Demonstration/Resources

Outline1 Motivation






Demonstration/Resources

We’ll go though a quick Demo! (time permitting)Resources:

R: http://cran.r-project.org/R::XML:http://cran.r-project.org/web/packages/XML/index.htmlDuncan Temple Lang:http://www.stat.ucdavis.edu/ duncan/XPath Tutorial:http://www.w3schools.com/xpath/default.aspRegEx: http://www.regular-expressions.info/reference.html,WikipediaThis Presentation: http://www.stat.berkeley.edu/ luis/



r and collecting internet data - university of california, berkeley

Documents