apache pig presentation_siddharth_mathur

1

CSC 5800:

Intelligent Systems: Algorithms and Tools

Pig Latin: A Not-So-Foreign Language for Data Processing

By Siddharth Mathur

2

What we will be covering

Introduction

MapReduce Overview

Pig Overview

Pig Features

Pig Latin

Pig Debugger

Demo

3

Introduction

Enormous data Innovation critically depends upon analyzing terabytes of

data collected everyday SQL can resolve the structure data problems Parallel Database processing

– Data is enormous can’t be analyzed serially.

– Has to be analyzed in parallel.

– Shared nothing clusters are the way to go.

4

Parallel DB Products

Teradata, Oracle RAC, Netezza

Expensive at web scale

Programmers have to write complex SQL queries because of this declarative programming is not preferred

5

Procedural programming

Map-Reduce programming model

It can easily perform a group by aggregation in parallel over a cluster of machines

The programmer provides map functions which is used as a filter or transforming method

The reduce function performs the aggregation

Appealing to the programmer because there are only 2 high level declarative functions to enable parallel processing

6

MapReduce Overview

Programming Model

– To cater large data analytics

– Works over Hadoop

– Java based

– Splits data into independent chunks and process them in-parallel

Program structure

– Mapper

– Reducer

– Driver Program

7

MapReduce Driver Program

Works as ‘Main’ function for MR job

Takes care of

– Number of arguments

– Input Data Location

– Input Data Types

– Output Data Location

– Output Data types

– Number of Mappers

– Number of Reducers

8

Mapper and Reducer Class

Mapper Class

– Main task is to perform any function logic

– Computes tasks like:

• Filtering

• Splitting

• Tokenizing

• Transforming

Reducer Class

– Works as an aggregator

– Aggregates the intermediate results gathered from Mapper

9

Word Count Execution

the quickbrown fox

the fox atethe mouse

how nowbrown cow

Map

Map

Map

Reduce

Reduce

brown, 2fox, 2how, 1now, 1the, 3

ate, 1cow, 1

mouse, 1quick, 1

the, 1brown, 1

fox, 1

quick, 1

the, 1fox, 1the, 1

how, 1now, 1

brown, 1ate, 1

mouse, 1

cow, 1

Input Map Shuffle & Sort Reduce Output

10

MapReduce Word Count Program

public static class Map extends Mapper<LongWritable, Text, Text, IntWritable>

{

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

String line = value.toString();

StringTokenizer tokenizer = new StringTokenizer(line);

while (tokenizer.hasMoreTokens()) {

word.set(tokenizer.nextToken());

context.write(word, one);

}

}

}

public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterable<IntWritable> values, Context context)

throws IOException, InterruptedException {

int sum = 0;

for (IntWritable val : values) {

sum += val.get();

}

context.write(key, new IntWritable(sum));

}

}

11

Map Reduce Limitations

1 input – 2 stage data flow is extremely rigid.

– To perform a task like join or sum iteration task, workaround has to be devised.

– Custom code for common task like filtering or transforming or projection

– The code is difficult to reuse and maintain

Moreover, because of its own data types, workflow and the fact that people have to learn java, makes it’s a tough choice to take.

12

Pig

An Apache open source project.

Provides an engine for executing data flows in parallel on Hadoop.

Includes a language called ‘Pig Latin’ for expressing these data flows.

High level declarative data workflow language.

It has best of both worlds:

– High Level declarative querying like SQL

– Low Level procedural like Map Reduce

13

Hadoop Stack

Data Processing Layer

Resource Management Layer

Storage Layer

…

Hadoop MR

Hive PigHBase

Hadoop Yarn

HDFS

14

Why Choose Pig

Written like SQL, compiled into MapReduce

Fully nested data model

Extensive support for UDFs

Can answer multiple questions in one single workflow.A = load './input.txt';

B = foreach A generate flatten(TOKENIZE((chararray)$0)) as word;

C = group B by word;

D = foreach C generate COUNT(B), group;

store D into './output';

15

Features and Motivation

Design goal of pig is to provide programmers with appealing experience for performing ad-hoc analysis of extremely large data sets.

– DataFlow Language

– QuickStart and Interoperability

– Nested Data Model

– UDF’s

– Debugging Environment

16

Data Flow Language

Each step specifies a single high level data transformation

Different from SQL where all these results are a single output.

The system has given opportunity to provide optimization function.

– Example:A= Load ‘input.txt’;

B= Filter A by UDF (Column1);

C= Filter B by Column1 > 0.8;

17

Quick start and Interoperability

Data Load

– Capability of Ad-Hoc analysis

– Can run queries directly on Data from dump of search engines

– Just have to provide a function that tells Pig how to parse the content of file into tuple.

– Similarly for output

• Any output format.

• These function can be reused.

• Used for visualization or dumped to excel directly

18

Pig as part of workflow

Pig easily becomes a part of workflow eco-system

– Can take most of the input types

– Can output in many of the forms

– Doesn’t take over the data, i.e., it does not lock the data that is being processed.

– Read only data analysis

19

Optional data schemas

Schema can be provided by the user :

– In the beginning

– On the fly

– Example: • A= LOAD ‘input.txt’ as (Column1;Column2);

• B= Filter A by Column1>5;

If the schema is not provided then the columns can be referred by ‘$0’, ‘$1’, ‘$2’…. for the 1st, 2nd, 3rd column etc. Example:

A= LOAD ‘input.txt’;

B= Filter A by $0>5;

20

Nested Data Model

Suppose, for a document, we want to extract the term and its position.

Format of output : Map<document Set<position>>

SQL data model:

Or keep in normalized form, i.e.,

– term_info(termid, String)

– position_info(termid, position, document)

Term Document ID Position

Hi 1 2

Hi 1 5

21

Problem resolved using Pig

In pig we have complex data types like map, tuple or bag to occur as a field of a table itself.

Example:

This approach is good because its more closer to what a programmer thinks.

Data is stored on disk in a nested fashion only

It gives user an ease in writing UDFs.

Term Document ID Position

Hi 1 (2,5,8..)

22

UDFs

Significant part of data analysis is custom processing

For example, user might want to process natural language stemming

Or checking if the page is spam or not, or many other tasks

To work on this, Pig Latin has extensive support for UDFs, most of the tasks can be resolved using the UDFs

It can take non-atomic input and can provide a non-atomic output also

Currently the UDFs can be written in java or python

23

Debugging Environment

In any language, getting a data processing program work correctly usually takes many iterations

First few iterations mostly produce errors

With a large scale data this would result in serious time and resource wastage

Debuggers can help

Pig has a novel debugging environment

Generates concise examples from input data

Data samples are carefully chosen to resemble real data as far as possible

Sample data is carved specially

24

Pig Latin

Language in which data workflow statements are written

It runs on the shell called ‘Grunt’

It has a shared repository name Piggybank

We can create our custom UDFs and add them to Piggybank

25

Data Model

Rich, yet simple data models

Atoms

– Simple atomic values like string or number

Tuple

– A collection of fields each of which can be of any data type

– Analogous to rows in SQL

Bag

– Collection of tuples or both tuples and atoms

– Can also be heterogeneous

26

Data Model (cont.)

Example of a relation

Atom Tuple Bag

T= ‘alice’, (labours,1), {(‘ipod’, 2),‘james’}

Tuple is represented with round braces

Bag is represented with curly braces

27

Specifying Input Data : LOAD

Its the first step in Pig Latin program

Specifying what the input files are

How are its contents to be deserialized, i.e., converted to pig data model.

LOAD command

– Examplequeries= LOAD ‘query_log.csv’

USING PigStorage(‘,’)

AS (userId,queryString,timestamp);

28

LOAD (cont.)

Both the ‘USING’ clause and the ‘AS’ clause are optional

We can work without them as shown earlier ($0 for first field)

Pig Storage is a pre-defined function

Can use custom function instead of Pig Storage

29

Per Tuple Processing : FOREACH

Similar to FOR statements

Its used for applying special processing to each tuple of the dataset

Example– Expanded_query = FOREACH queries GENERATE

UserId, Expand(queryString), timeStamp;

Its not a FILTERING command

‘Expand’ can take atomic input and can generate a bag of outputs

30

Per Tuple Processing : FOREACH(cont.)

The semantics of FOREACH is such that there is no dependency between different tuples of input, therefore permitting efficient parallel implementation

31

Discarding Unwanted Data : FILTER

Used as a where clause

Can provide anything in the expression

– Query = FILTER queries By user_id neq ‘bot’;

We can provide a UDF also, like

– Query = FILTER queries by Isbot(user_id);

32

COGROUP

Similar to Join

Groups bags of different inputs together

Ease of use for UDF’s– Grouped_data = COGROUP results by querystring, revenue by

querystring;

33

JOIN

Not all users want to use COGROUP

Simple equi-join is all that is required

– Example

Join_result = JOIN results by querystring,

revenue by querystring;

Other types of join are also supported:

– Left outer

– Right outer

– Full outer

34

Other Commands

Relational Operators

– UNION

– CROSS

– ORDER

– DISTINCT

– LIMIT

Eval Functions

– Concat

– Count

– Diff

35

PARALLEL clause

It is used to increase the parallelization of the job

We can specify the number of reduce tasks of the MR jobs created by Pig

It only effects the reduce task

No control over map

The system also can figure out number of reducers

Mostly one reduce task is required

36

PARALLEL clause (cont.)

Can be applied to only those commands which come under reduce phase

– COGROUP

– CROSS

– DISTINCT

– GROUP

– JOINS

– ORDERA = LOAD ‘ File1’;

B = LOAD ‘ File2’;

C = CROSS A, B PARALLEL 10;

37

Split Clause

We can split the input record into many by providing condition

A = LOAD ‘data’ AS (F1:int, F2:int, F3:int)

(1,2;3)

(2,3;7)

SPLIT A INTO B IF F1>7, C IF F2==5;

B (1,2,3) C (2,5,7)

(2,5,7)

Any expression can be written

UDFs can be used

It is not partitioning

38

Output

There are two ways to display

– STORE• If you want to store the output in any location

STORE output_1 INTO ‘hadoopuser/output’

– DUMP

• Basically used to display the result in the GRUNT shell itself

• Dumping doesn’t store the output anywhereDUMP query_result;

39

Building a Logical Plan

Pig interpreter first parses all the commands which the client issues

Verifies that the input files, bags or columns referred by the command are valid

Builds a logical plan for every bag the user defines

No processing is carried out

Processing triggers where a user invokes STORE/DUMP command

Called as a Lazy execution approach

Helps in FILTER reordering

40

Debugging Environment

This is used to avoid running the complete code on the entire dataset

User can create a sample data

Difficult to tailor these datasets and end up in self cooked data

Pig Pen is Pig’s debugging environment

Creates side dataset automatically, called as sandbox dataset

Pig Pen has its own user interface

41

Pig Pen

Outputs can be easily analyzed

Errors can be rectified earlier

42

Future Work

User Interface

– Drag-Drop style would help

– Logical plan diagram create made easy

UDF support for other languages

Unified Environment

– Currently, lacks in control structures like loops

– Has to embedded for all iterative tasks

43

Summary

Not So Foreign Language

Aims a sweet spot between SQL and MapReduce

Reusable and easy to use

Novel Debugging Environment: Pig Pen

Pig has an active and growing user base in Yahoo!

Pigs

– Eats anything

– Live anywhere

– Are domestic

44

45

Based on “Pig Latin: A Not-So-Foreign Language for Data Processing”

SIGMOD’08,June 9–12, 2008, Vancouver,BC,Canada

Christopher Olston Benjamin ReedYahoo! Research Yahoo! Research

Utkarsh Srivastava Ravi KumarYahoo! Research Yahoo! Research

Andrew TomkinsYahoo! Research

46

References

http://infolab.stanford.edu/~usriv/papers/pig-latin.pdf

http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html

Book: Programming pig

http://www.brentozar.com/archive/2011/11/good-pig/

http://hortonworks.com/hadoop/pig/