analyzing yellowstone’s network with a raspberry pi cluster

Analyzing Yellowstone’s Network with a Raspberry Pi Cluster

Lauren Patterson

Using a low cost Raspberry Pi cluster to find the interconnect path between two nodes on Yellowstone in order to analyze the performance of jobs.

Objective of the Project

Assembling the Raspberry Pi cluster

Yellowstone Interconnect

Credit: Siddhartha Ghosh

Files Used

• job1_nodes.txt– Gives the job ID and nodes used

• ibnetdiscover.log (Discover File)– Lists connections between switches

• LFTS.txt– Routing table for each switch

What is Hadoop?• HDFS • MapReduce

HDFSName Node

Job TrackerTask TrackerMap/Reduce

Data Node

Task TrackerMap/Reduce

Data NodeTask TrackerMap/Reduce

Data NodeTask TrackerMap/Reduce

InputData

Map Phase

Shuffle phase

Reduce phase

OutputData

MapReduce

Pig• Apache Pig • Pig Latin • Grunt

Pig Latin Script• Created Pig Latin Script to find the path

between two nodes in Yellowstone

http://www.edureka.in/blog/pig-programming-create-your-first-apache-pig-script/

JOIN Operations in PIG Default, Inner Join returns intersection of A and B

Set BSet A A BU

Full, Right and Left Outer Joins return A and B with different parts nulled out (white)

Full

Right

Left

Join

Path Finder Code Flow

Six Hops (seconds) Four Hops (seconds) Two Hops (seconds)

Raspberry Pi Cluster

3216 2514 1584

Virtual Box 450 351 214

250

750

1250

1750

2250

2750

3250

Pig (Single Path)

Seco

nds

Results

±3

±82 ±19 ±15

±3±4

Python• Single Path Python• Parallel Python– Mpi4py 1.3.1

Six Hop (seconds) Four Hop (seconds) Two Hop (seconds)

Raspberry Pi 6.39 4.136 2.226

Virtual Box 0.273 0.174 0.099

0.5

1.5

2.5

3.5

4.5

5.5

6.5

Serial Python (Single Path)Se

cond

s

±0.02

±0.07

±0.006

±0.11

±0.004

±0.11

16*15 Paths (sec-onds)

12*11 Paths (sec-onds)

8*7 Paths (seconds)

One Proces-sor

933 503 197

Two Proces-sors

493 275 108

Four Proces-sors

258 134 57

50250450650850

Parallel Python on Raspberry PiSe

cond

s

±18

±4

±20

±2±7 ±4

±1

±2 ±0.5

What Do All Of These Have In Common?

• Raspberry Pi• Hadoop • Pig • Python

Acknowledgments

Richard LoftKarina Hauser Stephanie BarrBruce Chittenden Amogh SimhaRaghu Raj Prasanna Kumar

Questions?