9/7/2017

1

BIG DATA SCIENTIFIC AND COMMERCIAL

APPLICATIONS (ITNPBD4)

LECTURE 1: INTRODUCTION

Gabriela Ochoa

http://www.cs.stir.ac.uk/~goc/

OUTLINE

Preliminaries

Module overview and schedule

Resources

Evaluation

What is Big Data?

Big Data

Why now?

Data Science

What is a data scientist?

Summary & What’s next?

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

2

9/7/2017

2

MODULE OVERVIEW

With guest lectures from science and industry,

this course presents a set of case studies of Big

Data in action.

Aim of module is to enable students to:

See what Big Data techniques are used for

Apply the techniques in a coherent and effective way.

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

3

Gabriela Ochoa

Nadarajen Veerapen

Invited Speakers

Student Talks

Schedule

4

9/7/2017

3

RESOURCES

Book

Doing Data Science (2014) by Rachel

Schutt and Cathy O’Neil, O’Relly

Links and websites

IBM Big Data Analytics

The rise of big data

Drew Conway keynote (2014)

Online courses

Udacity

edX

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

5

EVALUATION

50% assignment

Written report

Presentation talk: Weeks 11, 12. Nov 20th – 30th

Deadline report & slides: Friday, 1st December

50% exam

Mostly based on the material by the course lectures:

G. Ochoa & N. Veerapen (i.e. not including the

invited talks)

Lectures & Tutorials

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

6

9/7/2017

4

WHAT IS BIG DATA?

Large complex sets of data

Traditional data processing software is inadequate

Produced by all digital process and social media

exchanges (every click, like, chat ..)

Transmitted by sensors and mobile devices

Much of this data comes in an unstructured form

(i.e. not structured tables in row & columns)

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

Big Data - Massive amounts of complex data that

require special techniques for acquisition, storage,

distribution and analysis. 7

BIG DATA

Relational databases

Spread sheets

Searchable

Text and multimedia

content

e-mail, videos, audio,

written documents

Structured

Unstructured

8

9/7/2017

5

WHY NOW?

Massive amounts data, many aspects of our lives:

Shopping, communicating, reading news, music,

searches, expressing opinions – all is tracked!

Abundance of inexpensive computing power

Growing relevance of data in many sectors and

industries: finance, medicine, bioinformatics, social

welfare, government, retail

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

✚

9

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

There are structured, unstructured and hybrid forms of data.

All are relevant for Big Data 10

9/7/2017

6

UNITS OF DATA STORAGE G

ab

riela

Och

oa

, goc@

cs.stir.ac.u

k

11

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

BRONTOBYTES …

https://www.theregister.co.uk/2012/12/04/hp_discover_autonomy_vertica_big_data/

12

9/7/2017

7

• The incredible scale

of e-commerce,

social media, email,

and other content

creation that

happens on the web.

• Created each year by

Lori Lewis and Chadd

Callahan of Cumulus

Media

• http://www.visualcapita

list.com/happens-

internet-minute-2017/

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

13

http://www.ibmbigdatahub.com/infographic/four-vs-big-data

14

9/7/2017

8

EXAMPLES OF SCIENTIFIC DATA G

ab

riela

Och

oa

, goc@

cs.stir.ac.u

k

Genome Sequencing:

people diagnosed with

cancer 4 EB

Protein Data Bank:

repository of 3D structures

of large biological

molecules

CERN's Large Hadron

Collider: 15 PB per year

Hubble Space Telescope:

10 TB per year

The Laser Interferometer

Gravitational-Wave

Observatory (LIGO) detect

cosmic gravitational waves

Biology Physics & Astronomy

15

WHAT IS DATA SCIENCE?

Getting value out of data

Turns data into insights and actions

Basis for empirical research, hypothesis testing

Used by industry and science to understand a

phenomenon

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

Big Data

Insight Action

16

9/7/2017

9

WHAT IS INSIGHT? G

ab

riela

Och

oa

, goc@

cs.stir.ac.u

k

Insight Data Product

Data Analysis Question Insight

• Extracted from data through a combination of exploratory data

analysis and modelling.

• QUESTIONS:

• Sometimes can be very specific

• Sometimes it requires looking at the data and patterns to come

up with specific questions

The data product

of data science

17

DATA SCIENCE: ORIGINS & HISTORY

1960: The term data science was used as a substitute of computer science (data processing methods). Publication in 1974, by P. Naur

1997: C.F. Jeff Wu gave inaugural lecture entitled "Statistics = Data Science?"

2001: W. Cleveland. Data science as a new discipline. "Data Science: An action plan to expand the field of statistics.”

2002, 2003: Creations of Journals - Data Science Journal, The Journal of Data Science

2008: DJ Patil and J.Hammerbacher—then at LinkedIn and Facebook, respectively—coined the term “data scientist”.

2012: Creation of Wikipedia entry

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

18

9/7/2017

10

EXAMPLE: A BOOK RETAILER

LIKE AMAZON.COM

Customer Data

•Demographic

•Previous Purchases

•Book reviews

Insight

•What kind of books does this customer like?

Action

•Book recommendations

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

• Using past and current information, Data Science

Generate Actions!

• Actionable information for the future: predictions

• Another example of prediction: weather forecast • Action: decide what to on the day

19

DR

EW

CO

NW

AY’S

VE

NN

DIA

GR

AM

OF D

AT

A S

CIE

NC

E

Hacking Skills

• Manipulate text files

• Understand vectorised

operations,

• Think algorithmically

Math & Statistics

• Extract insight from data

• Linear regression, etc.

• Machine learning

Substantive Expertise

• Motivating questions

about the world

• Hypotheses that can be

brought to data and tested

with statistical methods

• Questions first, then data

https://s3.amazonaws.com/aws.drewconway.com/viz/venn

_diagram/data_science.html

20

9/7/2017

11

A D

AT

A S

CIE

NC

E T

EA

M’S

PR

OF

ILE

Skills

• Data visualisation

• Machine learning

• Mathematics

• Statistics

• Computer science

• Communication

• Domain expertise

Data scientist profiles;

alignment between the

data science team profile

and the profile of the

data problems they try

to solve

A data science team

works best when

different skills (profiles)

are represented across

different people, because

no‐ body is good at

everything

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

21

IN

FO

GR

AP

HIC

BY

RZ

YS

ZT

OF

ZA

WA

DZ

KI

How Data Science

translates into real skills.

Data Scientists:

• Have a passion for data

• Relates problems to

analytics

• Cares about

engineering solutions

• Exhibits curiosity

• Communicates with

teammates

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

22

9/7/2017

12

CL

US

TE

RIN

G A

LG

OR

ITH

M T

O D

EF

INE

DA

TA

SC

IEN

CE

Harlan Harris from

Data Community DC

Web-based survey of

Data Scientists, with

the goal of better

understanding the

varieties of people,

skills, and experiences

that fall under this

rather broad buzzword.

http://datacommunitydc.org/blog/2012/08/data-scientists-

survey-results-teaser/

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

23

SUMMARY

Big Data: Massive amounts of complex data that

require special techniques for acquisition,

storage, distribution and analysis

Data Science: Big Data Insight Action

It is difficult (if not impossible) to excel in all the

Data Scientist skills!

Interdisciplinary discipline

Often require teams

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

24

9/7/2017

13

WHAT’S NEXT?

Next Lecture (in 10 minutes): The Data

Science Process

Tutorials

There is no tutorial this week

Big Data Students need to register (via Canvas)

in one of the 3 groups on Mondays 1. 10 AM, Room: 2B38

2. 2 PM, Room: 2B38

3. 4 PM, Room: 3V2

Management Students Thursdays 3 PM, Room: 2X6

Next week: Thursday 21st 1 PM, Room 2B76.

Ga

brie

la O

choa

, goc@

cs.stir.ac.u

k

25