getting to know your data with r
TRANSCRIPT
dev.Objective() ~ Thursday, 14 May 2015
Getting to Know Your Data with RSteve Withington
01
Steve Withington✤ Blue River / Mura CMS
Application Engineer & Instructor!
✤ dev.Objective() Steering Committee Member since 2010!
✤ Author!
✤ Marathoner & Triathlete!
✤ Amateur Data Enthusiast
Steps in a Data Analysis
✤ Define the question!
✤ Define the ideal data set!
✤ Determine what data you can access!
✤ Obtain the data!
✤ Clean the data!
✤ Exploratory data analysis!
✤ Statistical prediction/modeling!
✤ Interpret results!
✤ Challenge results!
✤ Synthesize/write up results!
✤ Create reproducible code
Steps in a Data Analysis
✤ Define the question!
✤ Define the ideal data set!
✤ Determine what data you can access!
✤ Obtain the data!
✤ Clean the data!
✤ Exploratory data analysis!
✤ Statistical prediction/modeling!
✤ Interpret results!
✤ Challenge results!
✤ Synthesize/write up results!
✤ Create reproducible code
Types of Data Science Questions
✤ In approximate order of difficulty!
✤ Descriptive!
✤ Exploratory!
✤ Inferential!
✤ Predictive!
✤ Causal!
✤ Mechanistic
Descriptive Analysis
✤ Goal: Describe a set of data!
✤ The first kind of data analysis performed!
✤ Commonly applied to census data!
✤ The description and interpretation are different steps!
✤ Descriptions can usually not be generalized without additional statistical modeling
Exploratory Analysis
✤ Goal: Find relationships you didn’t know about!
✤ Exploratory models are good for discovering new connections!
✤ They are also useful for defining future studies!
✤ Exploratory analyses are usually not the final say!
✤ Exploratory analyses alone should not be used for generalizing/predicting!
✤ Correlation does not imply causation
Correlation is Not Causation
✤ Even if you observe two variables are correlated with each other, you have to prove to yourself that they’re not related because of some other variable you didn’t measure!
✤ Example: shoe size and literacy
Inferential Analysis
✤ Goal: Use a relatively small sample of data to say something about a bigger population!
✤ Inference is commonly the goal of statistical models!
✤ Inference involves estimating both the quantity you care about and your uncertainty about your estimate!
✤ Inference depends heavily on both the population and the sampling scheme
Inferential Analysis
– http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3521092/
Predictive Analysis
✤ Goal: To use the data on some objects to predict the values for another object!
✤ If X predicts Y it does not mean that X causes Y!
✤ Accurate prediction depends heavily on measuring the right variables!
✤ Although there are better and worse prediction models, more data and a simple model works really well!
✤ Prediction is very hard, especially about the future references
Predictive Analysis
– http://fivethirtyeight.com/interactives/uk-general-election-predictions/
Predictive Analysis
– http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/
Causal Analysis
✤ Goal: To find out what happens to one variable when you make another variable change!
✤ Usually randomized studies are required to identify causation!
✤ There are approaches to inferring causation in non-randomized studies, but they are complicated and sensitive to assumptions!
✤ Causal relationships are usually identified as average effects, but may not apply to every individual!
✤ Causal models are usually the “gold standard” for data analysis
Causal Analysis
– http://www.nejm.org/doi/full/10.1056/NEJM200007133430201
Mechanistic Analysis
✤ Goal: Understand the exact changes in variables that lead to changes in other variables for individual objects!
✤ Incredibly hard to infer, except in simple situations!
✤ Usually modeled by a deterministic set of equations (physical/engineering science)!
✤ Generally the random component of the data is measurement error!
✤ If the equations are known but the parameters are not, they may be inferred with data analysis
Mechanistic Analysis
– http://www.ellisonfoundation.org/content/mechanistic-analysis-transcriptional-switch-regulating-p53-activity-and-premature-aging
“You can have data without information, but you cannot have information without data.”
–Daniel Keys Moran
01
New Terminology
– http://www.dailymail.co.uk/sciencetech/article-2247081/There-soon-words-data-stored-world.html
Yottabyte
✤ 1000 kB kilobyte1000
2 MB megabyte
10003 GB gigabyte
10004 TB terabyte
10005 PB petabyte
10006 EB exabyte
10007 ZB zettabyte
10008 YB yottabyte!
✤ 1 YB = 1000
8 bytes
= 1024
bytes = 1 000 000 000 000 000 000 000 000 bytes = 1000 zettabytes = 1 trillion terabytes
NSA’s Bumblehive
– http://www.bbc.com/news/business-26383058 (Capable of storing one thousand trillion gigabytes!)
Data
✤ Companies have terabytes of data about the consumers they interact with!
✤ Governmental, academic, and private research institutions have extensive archival and survey data on every manner of research topic
The Challenge
✤ Presenting information in easily accessible and digestible ways
Visualizations
✤ We’re really good at gleaning useful information from visual presentations!
✤ Graphical presentations are an excellent way to convey results and uncover meaning
Visualizing Friendships
– https://www.facebook.com/note.php?note_id=469716398919
Visualizing Friendships
– https://www.facebook.com/note.php?note_id=469716398919
“Data is a set of values of qualitative or quantitative variables; restated,
pieces of data are individual pieces of information.”
– http://en.wikipedia.org/wiki/Data
Definition of Data
“Data is a set of values of qualitative or quantitative variables; restated, pieces of data are individual pieces of information.” !
✤ Qualitative: properties that are observed and can generally not be measured with a numerical result. (Sex, country of origin, etc.)!
✤ Quantitative: properties that can exist as a magnitude or multitude. (Height, weight, blood pressure, etc.)
Definition of Data
“Data is a set of values of qualitative or quantitative variables; restated, pieces of data are individual pieces of information.” !
✤ Variable: a logical set of attributes!
✤ Attribute: a characteristic of an object (person, thing, etc.)
Definition of Data
“Data is a set of values of qualitative or quantitative variables; restated, pieces of data are individual pieces of information.” !
✤ Information: an answer to a question, as well as that from which knowledge and data can be derived (as data represents values attributed to parameters, and knowledge signifies understanding of real things or abstract concepts). As it regards data, the information’s existence is not necessarily coupled to an observer (it exists beyond an event horizon, for example), while in the case of knowledge, information requires a cognitive observer.
“The data may not contain the answer. The combination of some data and an aching desire
for an answer does not ensure that a reasonable answer can be extracted from a given body of data.”
–John Wilder Tukey
What Data Looks Like
– http://brianknaus.com/software/srtoolbox/s_4_1_sequence80.txt
What Data Looks Like
– https://developers.facebook.com/docs/graph-api
What Data Looks Like
– http://www.nytimes.com/2012/06/26/technology/in-a-big-network-of-computers-evidence-of-machine-learning.html
What Data Looks Like
– http://www.baseball-reference.com/leagues/AL/2015-standard-pitching.shtml
Data is Second Most Important
✤ The most important thing in data science is the question!
✤ The second most important is the data!
✤ Often the data will limit or enable the questions!
✤ But having data can’t save you if you don’t have a question
01
“We wanted users to be able to begin in an interactive environment, where they did not consciously think of
themselves as programming. Then as their needs became clearer and their sophistication increased, they should be able to slide gradually into programming, when the language and
system aspects would become more important.”
–John Chambers, “Stages in the Evolution of S”
R
✤ Comprehensive statistical platform (just about any type of data analysis can be done in R)!
✤ Useful for interactive work, and also contains a powerful programming language for developing new tools (user -> programmer)!
✤ Open source & free (see http://www.fsf.org)!
✤ State of the art graphics!
✤ Easily import data from a wide variety of sources!
✤ RStudio IDE!
✤ Developer Ecosystem and active community (mailing lists, StackOverflow, etc.)!
✤ Can be integrated into other languages, including C++, Java, Python, PHP, etc.
Design of the R System
✤ The “base” R system that you download from CRAN!
✤ Everything else
The R System
✤ R functionality is divided into a number of packages!
✤ The “base” R system contains, among other things, the base package which is required to run R and contains the most fundamental functions!
✤ There are several other packages contained in the “base” system as well as several “recommended” packages!
✤ Over 6,000 other packages are available on CRAN developed by users and programmers around the world!
✤ In addition, sites such as http://bioconductor.org (for genomic and biological data analysis) host a plethora of R packages
Where to Get R
✤ Comprehensive R Archive Network (CRAN) http://cran.r-project.org!
✤ Linux, Mac OS X, and Windows
Getting Started
✤ If you think of R as having a programming language instead of being one, you’ll be better off in the long run!
✤ It’s a case-sensitive, interpreted language!
✤ Enter commands one at a time via the command line, or sets of commands from a source file!
✤ Wide variety of data types, including vectors, matrices, data frames (similar to recordsets), and lists (collections of objects)!
✤ An object is basically anything that can be assigned a value
demo()
RStudio
RStudio’s Interface
text-file.R
Some Important R Functions
✤ Access help file
Some Important R Functions
✤ Search help files
Some Important R Functions
✤ Get arguments
✤ See code
R Reference Card
– http://cran.r-project.org/doc/contrib/Short-refcard.pdf
Entering Input
✤ At the R prompt we type expressions. The <- symbol is the assignment operator
✤ The grammar of the language determines whether an expression is complete or not
✤ The # character indicates a comment. Anything to the right of the # (including the # itself) is ignored
Evaluation
✤ When a complete expression is entered at the prompt, it is evaluated and the result of the evaluated expression is returned. The result may be auto-printed.
✤ The [1] indicates that x is a vector and 5 is the first element
Printing
✤ The : operator is used to create integer sequences
Objects
✤ R has five basic or “atomic” classes of objects:!
✤ character!
✤ numeric (real numbers)!
✤ integer!
✤ complex!
✤ logical (True/False)
Objects
✤ The most basic object is a vector!
✤ Vectors are one-dimensional arrays that can hold data!
✤ A vector can only contain objects of the same class!
✤ The one exception is a list, which is represented as a vector but can contain objects of different classes (that’s usually why they’re used)!
✤ Empty vectors can be created with the vector() function
Numbers
✤ Numbers in R are generally treated as numeric objects (i.e., double precision real numbers)!
✤ If you explicitly want an integer, you need to specify the L suffix!
✤ For example, entering 1 gives you a numeric object; entering 1L explicitly gives you an integer!
✤ There is also a special number Inf which represents infinity; e.g., 1 / 0; Inf can be used in ordinary calculations; e.g., 1 / Inf is 0!
✤ The value NaN represents an undefined value (“not a number”); e.g., 0 / 0; NaN can also be thought of as a missing value
Attributes
✤ R objects can have attributes!
✤ names, dimnames!
✤ dimensions (e.g., matrices, arrays)!
✤ class!
✤ length!
✤ other user-defined attributes/metadata!
✤ Attributes of an object can be accessed using the attributes() function
Vectors
✤ Vectors are one-dimensional arrays that can hold numeric data, character data, or logical data. The combine function c() is used to form the vector.!
!
!
✤ You can also use the vector() function
Vectors
✤ You can refer to elements of a vector using a numeric vector of positions within brackets. For example, a[c(2, 4)] refers to the second and fourth elements of vector a
Mixing Objects
✤ What about the following?
✤ When different objects are mixed in a vector, coercion occurs so that every element in the vector is of the same class
Explicit Coercion
✤ Objects can be explicitly coerced from one class to another using the as.* functions, if available
Explicit Coercion
✤ Nonsensical coercion results in NAs
Matrices
✤ A matrix is a two-dimensional array. It has a dimension attribute which is an integer vector of length 2 (nrow, ncol)
Matrices
✤ Matrices are constructed column-wise, so entries can be thought of starting in the “upper left” corner an running down the columns
Matrices
✤ Matrices can also be created directly from vectors by adding a dimension attribute
cbind-ing and rbind-ing
✤ Matrices can also be created by column-binding or row-binding with cbind() and rbind()
Arrays
✤ Arrays are similar to matrices, however they can have more than two dimensions
Factors
✤ Variables can be described as nominal, ordinal, or continuous!
✤ Nominal, categorical, without an implied order (e.g., blood type)!
✤ Ordinal, also categorical and imply order but not amount (e.g., grades)!
✤ Continuous, take on any value within some range, and both order and amount are implied (e.g., age)!
✤ Categorical (nominal) and ordered categorical (ordinal) variables in R are called factors!
✤ Factors are crucial in R because they determine how data is analyzed and presented visually
Factors
✤ The factor() function stores the categorical values as a vector of integers in the range [1… k], (where k is the number of unique values in the nominal variable) and an internal vector of character strings (the original values) mapped to these integers
Factors
✤ By default, factor levels for character vectors are created in alphabetical order and can be overridden by specifying a levels option
Lists
✤ Lists are the most complex of the R data types. Basically, a list is an ordered collection of objects. A list allows you to gather a variety of (possibly unrelated) objects under one name. For example, a list may contain a combination of vectors, matrices, data frames, and even other lists.
Data Frames
✤ A data frame is more general than a matrix in that columns can contain different modes of data (numeric, character, etc.). They store tabular data. These are the most common data types in R.!
✤ They are represented as a special type of list where every element of the list has to have the same length!
✤ Each element of the list can be thought of as a column and the length of each element of the list is the number of rows!
✤ Unlike matrices, data frames can store different classes of objects in each column (just like lists); matrices must have every element be the same class!
✤ Data frames also have a special attribute called row.names!
✤ Data frames are usually created by calling read.table() or read.csv()!
✤ Can be converted to a matrix by calling data.matrix()
Data Frames
✤ Each column must have only one mode, but you can put columns of different modes together
Names
✤ R objects can also have names, which is very useful for writing readable code and self-describing objects
Names
✤ Lists can also have names
Names
✤ And matrices
Missing Values
✤ Missing values are denote by NA or NaN for undefined mathematical operations!
✤ is.na() is used to test objects if they are NA!
✤ is.nan() is used to test objects for NaN!
✤ NA values have a class also, so there are integer NA, character NA, etc.!
✤ A NaN value is also NA but the converse is not true
Missing Values
Data Types Summary
✤ atomic classes: numeric, logical, character, integer, complex!
✤ vectors!
✤ matrices!
✤ arrays!
✤ lists!
✤ data frames!
✤ names!
✤ missing values
Data Types Summary
Oddities
✤ The period (.) has no special significant in object names. The dollar sign ($) has a somewhat similar meaning to the period in other object-oriented languages and can be used to identify the parts of a data frame or list (e.g., mydata$columnname)!
✤ R don’t have multiline or block comments (must use the #)!
✤ Assigning a value to a nonexistent element of a vector, matrix, array, or list expands that structure to accommodate the new value!
✤ R doesn’t have scalar values, they’re represented as one-element vectors!
✤ Indices start at 1, not 0!
✤ Variables can’t be declared, they come into existence on the first assignment
Helpful Methods
✤ getwd()!
✤ setwd(“some/directory”)!
✤ list.files()!
✤ ls() or objects()!
✤ rm(object/objectlist)!
✤ rm(list=ls())!
✤ edit(objectname)!
✤ source(“filename”)!
✤ sink(“filename.ext”, append=TRUE, split=TRUE)!
✤ data() ~ list available data sets!
✤ search() ~ list loaded packages!
✤ q()
Reading Data
✤ There are a few principal functions reading data into R!
✤ read.table, read.csv, for reading tabular data!
✤ readLines, for reading lines of a text file!
✤ source, for reading in R code files (inverse of dump)!
✤ dget, for reading in R code files (inverse of dput)!
✤ load, for reading in saved workspaces!
✤ unserialize, for reading single R objects in binary form
Writing Data
✤ There are similar functions for writing data to files!
✤ write.table!
✤ writeLines!
✤ dump!
✤ dput!
✤ save!
✤ serialize
read.table
✤ The read.table function is one of the most commonly used functions for reading data!
✤ It has a few important arguments:!
✤ file, the name of the file, or a connection!
✤ header, logical indicating if the file has a header line!
✤ sep, a string indicating how the columns are separated!
✤ colClasses, a character vector indicating the class of each column in the dataset!
✤ nrows, the number of rows in the dataset!
✤ comment.char, a character string indicating the comment character!
✤ skip, the number of lines to skip from the beginning!
✤ stringsAsFactors, should character variables be coded as factors?
read.table
✤ For small to moderately sized datasets, you can usually call read.table without specifying any other arguments
✤ R will automatically!
✤ skip lines that begin with #!
✤ figure out how many rows there are (and how much memory needs to be allocated)!
✤ figure what time of variable is in each column of the table (telling R all these things directly makes R run much faster and more efficiently)!
✤ read.csv is identical to read.table except that the default separator is a comma
Larger Datasets
✤ With much larger datasets, doing the following things will make your life easier and will prevent R from choking!
✤ Read the help page for read.table, which contains many hints!
✤ Make a rough calculation of the memory required to store your dataset!
✤ If the dataset is larger than the amount of RAM on your computer, you can probably stop right here!
✤ Set comment.char = “” if there are no commented lines in your file
Larger Datasets
✤ Use the colClasses argument!
✤ Specifying this option instead of using the default can make read.table run much faster, often twice as fast!
✤ In order to use this option, you have to know the class of each column in your data farm!
✤ If all of the columns are numeric, for example, then you can just set colClasses = “numeric”!
✤ Set nrows!
✤ This doesn’t make R run faster but it helps with memory usage
Know Thy System
✤ In general, when using R with larger datasets, it’s useful to know a few things about your system!
✤ How much memory is available?!
✤ What other applications are in use?!
✤ Are there other users logged into the same system?!
✤ What is the Operating System?!
✤ Is the OS 32 or 64 bit?
Calculating Memory Requirements
✤ I have a data frame with 1,500,000 rows and 120 columns, all of which are numeric data. Roughly, how much memory is required to store this data frame? Note: each alphanumeric character requires 8 bits (1 byte) of memory to store.!
✤ 1,500,000 rows * 120 columns = 180,000,000 alphanumeric characters!
✤ 180,000,000 * 8 bits = 1,440,000,000 bytes!
✤ 1,440,000,000 bytes / 1024 = 1,406,250 kb!
✤ 1,406,250 kb / 1024 = 1,373.29 MB!
✤ 1,373.29 MB / 1024 = 1.34 GB!
✤ If you only have 2 GB of RAM, you may want to think twice before doing it. If you've got 4, 6, or 8+ GB of RAM, you should be fine.
Textual Formats
✤ dumping and dputing are useful because the resulting textual format is editable, and in the case of corruption, potentially recoverable!
✤ Unlike writing out a table or csv file, dump and dput preserve the metadata (sacrificing some readability), so that another user doesn't have to specify it all over again!
✤ Can work much better with version control programs like SVN or Git which can only track changes meaningfully in text files!
✤ Can be longer-lived; if there is corruption somewhere in the file, it can be easier to fix the problem!
✤ Downside: Format is not very space-efficient
dput-ting R Objects
✤ Another way to pass data around is by deparsing the R object with dput and reading it back in using dget
Dumping R Objects
✤ Multiple objects can be deparsed using the dump function and read back in using source
Interfaces to the Outside World
✤ Data are read in using connection interfaces!
✤ Connections can be made to files (most common) or to other more exotic things!
✤ file, opens a connection to a file!
✤ gzfile, opens a connection to a file compressed with gzip!
✤ bzfile, opens a connection to a file compressed with bzip2!
✤ url, opens a connection to a webpage
File Connections
✤ description is the name of the file!
✤ open is a code indicating!
✤ “r” read only!
✤ “w” writing (and initializing a new file)!
✤ “a” appending!
✤ “rb”, “wb”, “ab” reading, writing, or appending in binary mode (Windows)
Connections
✤ In general, connections are powerful tools that let you navigate files or other external objects!
✤ In practice, we often don’t need to deal with the connection interface directly
Reading Lines of a Text File
✤ readLines can be useful for reading in lines of webpages
Connecting and Listing Databases
Connecting and Listing Databases
R Packages
✤ When you download R from the Comprehensive Archive Network (CRAN), you get the “base” R system!
✤ The base R system comes with basic functionality and implements the R language!
✤ One reason R is so useful is the large collection of packages that extend the basic functionality of R!
✤ R packages are developed and published by the larger R community
Obtaining R Packages
✤ The primary location for obtaining R packages is CRAN!
✤ For biological applications, many packages are available from the Bioconductor Project!
✤ You can obtain information about the available packages on CRAN with the available.packages() function
Obtaining R Packages
✤ There are over 6,000 packages on CRAN covering a wide range of topics!
✤ A list of some topics is available through the Task Views link (http://cran.r-project.org/web/views/), which groups together many R packages related to a given topic
Installing an R Package
✤ Packages can be installed with the install.packages() function in R!
✤ To install a single package, pass the name of the package to the install.packages() function as the first argument!
✤ The following code installs the zipcode package from CRAN
✤ This command downloads the zipcode package from CRAN and installs it on your computer!
✤ Any packages on which this package depends will also be downloaded and installed
Installing Multiple R Packages
✤ You can install multiple R packages at once with a single call to install.packages()!
✤ Place the names of the R packages in a character vector
Installing Packages via RStudio
Installing Packages via RStudio
Loading R Packages
✤ Installing a package does not make it immediately available to your in R; you must load the package!
✤ The library() function is used to load packages into R!
✤ The following code is used to load the ggplot2 package into R
Loading R Packages
✤ Any packages that need to be loaded as dependencies will be loaded first, before the named package is loaded!
✤ Note: Do not put the package name in quotes!!
✤ Some packages produce messages when they are loaded (but some don’t)
Loading R Packages
✤ After loading a package, the functions exported by that packages will be attached to the top of the search() list (after the workspace)
example()
R Packages Summary
✤ R packages provide a powerful mechanism for extending the functionality of R!
✤ R packages can be obtained from CRAN or other repositories!
✤ The install.packages() can be used to install packages at the R console!
✤ The library() function loads packages that have been installed so that you may access the functionality in the packages
Learning Resources
✤ The R Manualshttp://cran.r-project.org/manuals.html!
✤ QuickR http://www.statmethods.net !
✤ R Mailing Listshttps://stat.ethz.ch/mailman/listinfo!
✤ SwiRl http://swirlstats.com!
✤ Introduction to Data Science https://www.udemy.com/introduction-to-data-science/!
✤ Johns Hopkins Data University Data Science Specialization https://www.coursera.org/specialization/jhudatascience/1
More Learning Resources
✤ R By Examplehttp://www.mayin.org/ajayshah/KB/R/index.html!
✤ R Data Import/Export Manualhttp://cran.r-project.org/doc/manuals/r-release/R-data.html!
✤ Data Analysis with R: Visually Analyze and Summarize Data Setshttps://www.udacity.com/course/data-analysis-with-r--ud651!
✤ Intro to Descriptive Statistics: Mathematics for Understanding Datahttps://www.udacity.com/course/intro-to-descriptive-statistics--ud827!
✤ Intro to Inferential Statistics: Making Predictions from Datahttps://www.udacity.com/course/intro-to-inferential-statistics--ud201
Good Stuff
✤ FiveThirtyEighthttp://fivethirtyeight.com !
✤ Kagglehttps://www.kaggle.com!
✤ Figsharehttp://figshare.com!
✤ OpenCPUhttps://www.openspu.org!
✤ RPyhttp://rpy.sourceforge.net!
✤ rChartshttp://rcharts.io!
✤ Facebook Data Science Research Service https://www.facebook.com/data!
✤ Google’s R Style Guide http://google-styleguide.googlecode.com/svn/trunk/Rguide.xml
Books
✤ Chambers. Software for Data Analysis. Springer, 2008.!
✤ Matloff. The Art of R Programming: A Tour of Statistical Software Design. No Starch Press, 2011.!
✤ Kabacoff. R in Action: Data analysis and graphics with R. Second Edition. Manning Publications, 2015.!
✤ Cielen & Meysman. Introducing Data Science. Manning Publications, 2015.!
✤ Tuckey. Exploratory Data Analysis. Pearson, 1977.
Books
✤ Manning Publications Discount Code
ctwdevob!✤ Packt Discount Code
DEV.Objective
01