pattern programming barry wilkinson university of north carolina charlotte cci friday seminar series...
TRANSCRIPT
![Page 1: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/1.jpg)
Pattern Programming
Barry WilkinsonUniversity of North Carolina Charlotte
CCI Friday Seminar Series April 13th, 2012
![Page 2: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/2.jpg)
Acknowledgment
This work was initiated by Jeremy Villalobos and described in his PhD thesis:
“RUNNING PARALLEL APPLICATIONS ON A HETEROGENEOUS ENVIRONMENT WITH ACCESSIBLE
DEVELOPMENT PRACTICES AND AUTOMATIC SCALABILITY,” UNC-Charlotte, 2011.
2
![Page 3: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/3.jpg)
Problem Addressed• To make parallel programming more useable and scalable.
• Parallel programming is writing programs for solving problems using multiple computers, processors and cores, including with physically distributed computers.
3
• A very long history but still a challenge.
• Traditional approaches involve explicitly specifying message-passing with low-level tools such as MPI and thread parallelism with OpenMP and OpenCL.
• Still not mainstream as it should be with the introduction of multicore processors
![Page 4: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/4.jpg)
Pattern Programming Concept
Programmer constructs his application using established computational or algorithmic “patterns” that provide a structure.
Patterns are widely applicable.
4
What patterns are we talking about?• Low-level algorithmic patterns that might be embedded into a
program such as fork-join, broadcast/scatter/gather.
• Higher level algorithm patterns for forming a complete program such as workpool, pipeline, stencil, map-reduce.
We concentrate upon higher-level “computational/algorithm ” level patterns rather than lower level patterns.
![Page 5: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/5.jpg)
Some Patterns
5
Workers
Workpool
Master Two-way connection
Compute node
Source/sink
Derived from Jeremy Villalobos’s PhD thesis defense
![Page 6: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/6.jpg)
6
Workers
Pipeline
Master
Two-way connection
Compute node
Source/sink
One-way connection
Stage 1 Stage 3Stage 2
![Page 7: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/7.jpg)
7
Divide and Conquer
DivideTwo-way connection
Compute node
Source/sink
Merge
![Page 8: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/8.jpg)
8
Stencil
Two-way connection
Compute node
Source/sink
Synchronous
![Page 9: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/9.jpg)
9
All-to-All
Two-way connection
Compute node
Source/sink
![Page 10: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/10.jpg)
Sometimes term “skeleton” used to describe “patterns”, especially directed acyclic graphs with a source, a computation, and a sink.
We do not make that distinction and use the term “pattern” whether directed or undirected and whether acyclic or cyclic.
This is done elsewhere.
10
Note on Terminology“Skeletons”
![Page 11: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/11.jpg)
Skeletons/Patterns• Advantages
• Implicit parallelization• Avoid deadlocks • Avoid race conditions
• Reduction in source code size (lines of code)• Abstracts the Grid/Cloud environment
• Disadvantages• Takes away some of the freedom from the user
programmer• New approach to learn• Performance reduced (5% on top of MPI)
Derived from Jeremy Villalobos’s PhD thesis defense 11
![Page 12: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/12.jpg)
• “Design patterns” have been part of software engineering for many years ... .– Reusable solutions to commonly occurring problems *– Patterns provide guide to “best practices”, not a final implementation– Provides good scalable design structure to parallel programs– Can reason more easier about programs
• Hierarchical designs with patterns embedded into patterns, and pattern operators to combine patterns.
• Leads to an automated conversion into parallel programs without need to write with low level message-passing routines such as MPI.
12
More Advantages/Notes
* http://en.wikipedia.org/wiki/Design_pattern_(computer_science)
![Page 13: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/13.jpg)
Previous/Existing Work• Patterns/skeletons explored in several projects.
• Universities:– University of Illinois at Urbana-Champaign and University
of California, Berkeley– University of Torino/Università di Pisa Italy– ...
• Industrial efforts– Intel– Microsoft– … 13
![Page 14: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/14.jpg)
University of Illinois at Urbana-Champaign and University of California, Berkeley with Microsoft and Intel in 2008 (with combined funding of at least $35 million).
– Co-developed OPL (Our Pattern Language), a pattern language for parallel programming
– Promoted patterns in Workshop on Parallel Programming Patterns, ParaPLoP 2009, 2010, and 2011, and Pattern Languages for Programs conference PloP’2009
14
Universal Parallel Computing Research Centers (UPCRC)
![Page 15: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/15.jpg)
Group of twelve computational patterns identified:
•Finite State Machines •Circuits •Graph Algorithms•Structured Grid•Dense Matrix•Sparse Matrix
in seven general application areas
15
• Spectral (FFT)• Dynamic Programming• Particle Methods• Backtrack• Graphical Models• Unstructured Grid
UPCRC Patterns
![Page 16: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/16.jpg)
Closest to our work
http://calvados.di.unipi.it/dokuwiki/doku.p
hp?id=ffnamespa
ce:about
University of Torino,
Italy /Università di
Pisa
16
![Page 17: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/17.jpg)
IntelFocused on very low level patterns such as fork-join, and provides constructs for them in:
•Intel Threading Building Blocks (TBB)– Template library for C++ to support parallelism
•Intel Cilk plus– Compiler extensions for C/C++ to support parallelism
•Intel Array Building Blocks (ArBB)– Pure C++ library-based solution for vector parallelism
Above are somewhat competing tools obtained through takeovers of small companies. Each implemented differently.
17
![Page 18: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/18.jpg)
New book due out 2012 from Intel authors
“Structured Parallel Programming: Patterns for Efficient Computation,” Michael McCool, James Reinders, Arch Robison, Morgan Kaufmann, 2012
Focuses on Intel tools
15.18B. Wilkinson was a reviewer for this book.
![Page 19: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/19.jpg)
19
Using patterns with Microsoft C#
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=19222
Again very low-level with patterns such as parallel for loops.
![Page 20: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/20.jpg)
Our approach(Jeremy Villalobos’ UNC-C PhD thesis)
Focuses on a few patterns of wide applicability (workpool, pipelined, stencil, and dense matrix patterns) but Jeremy took it much further than UPCRC and Intel.
He developed a higher-level framework called “Seeds”
Uses pattern approach to automatically distribute code across geographical sites and execute the parallel code.
20
![Page 21: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/21.jpg)
“Seeds” Parallel Grid Application Framework
21http://coit-grid01.uncc.edu/seeds/
Some Key Features
•Pattern-programming
(Java) user interface
•Self-deploys on computers, clusters, and geographically distributed computers•Load balances
•Three levels of user interface
![Page 22: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/22.jpg)
Seeds Development Layers Basic
Intended for programmers that have basic parallel computing background
Based on skeletons and patterns Advanced: Used to add or extend functionality such as:
Create new patterns Optimize existing patterns or Adapt existing pattern to non-functional requirements
specific to the application Expert: Used to provide basic services:
Deployment Security Communication/Connectivity Changes in the environment
22Derived from Jeremy Villalobos’s PhD thesis defense
![Page 23: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/23.jpg)
Deployment
• Deployment with Globus
– Globus GSIFTP to transfer “seeds” folder
– GRAM to submit job to run seed nodes
• Deployment with SSH
– now preferred– Globus will be
depreciated in Seeds23
![Page 24: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/24.jpg)
Basic User Programmer Interface
24
To create and execute parallel programs, programmer selects a pattern and implements three principal Java methods:
•Diffuse method – to distribute pieces of data.
•Compute method – the actual computation
•Gather method – used to gather the results
Programmer also has to fill in details in a “bootstrap” class to deploy and start the framework.
Diffuse
Compute
Gather
Bootstrap class
The framework self-deploys on a geographically distributed platform and executes pattern.
![Page 25: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/25.jpg)
25
• Basis on Monte Carlo calculations is use of random selections
• In this case, circle formed with a square
• Points within square chosen randomly
• Fraction of points within circle = /4
• Only one quadrant used in code
Monte Carlo calculation
Example: Deploy a workpool pattern to compute using Monte Carlo method
![Page 26: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/26.jpg)
package edu.uncc.grid.example.workpool;
import java.util.Random;
import java.util.logging.Level;
import edu.uncc.grid.pgaf.datamodules.Data;
import edu.uncc.grid.pgaf.datamodules.DataMap;
import edu.uncc.grid.pgaf.interfaces.basic.Workpool;
import edu.uncc.grid.pgaf.p2p.Node;
public class MonteCarloPiModule extends Workpool {
private static final long serialVersionUID = 1L;
private static final int DoubleDataSize = 1000;
double total;
int random_samples;
Random R;
public MonteCarloPiModule() {
R = new Random();
}
@Override
public void initializeModule(String[] args) {
total = 0;
Node.getLog().setLevel(Level.WARNING); // reduce verbosity for logging
random_samples = 3000; // set number of random samples
}
26
public Data Compute (Data data) {
// input gets the data produced by DiffuseData()
DataMap<String, Object> input = (DataMap<String,Object>)data;
// output will emit the partial answers done by this method
DataMap<String, Object> output = new DataMap<String, Object>();
Long seed = (Long) input.get("seed"); // get random seed
Random r = new Random();
r.setSeed(seed);
Long inside = 0L;
for (int i = 0; i < DoubleDataSize ; i++) {
double x = r.nextDouble();
double y = r.nextDouble();
double dist = x * x + y * y;
if (dist <= 1.0) {
++inside;
}
}
output.put("inside", inside);// store partial answer to return to GatherData()
return output;
}
public Data DiffuseData (int segment) {
DataMap<String, Object> d =new DataMap<String, Object>();
d.put("seed", R.nextLong());
return d; // returns a random seed for each job unit
}
public void GatherData (int segment, Data dat) {
DataMap<String,Object> out = (DataMap<String,Object>) dat;
Long inside = (Long) out.get("inside");
total += inside; // aggregate answer from all the worker nodes.
}
public double getPi() {
// returns value of pi based on the job done by all the workers
double pi = (total / (random_samples * DoubleDataSize)) * 4;
return pi;
}
public int getDataCount() {
return random_samples;
}
}
Complete code for computation
Note: No message passing (MPI etc)
![Page 27: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/27.jpg)
Bootstrap classpackage edu.uncc.grid.example.workpool;
import java.io.IOException;
import net.jxta.pipe.PipeID;
import edu.uncc.grid.pgaf.Anchor;
import edu.uncc.grid.pgaf.Operand;
import edu.uncc.grid.pgaf.Seeds;
import edu.uncc.grid.pgaf.p2p.Types;
public class RunMonteCarloPiModule {
public static void main(String[] args) {
try {
MonteCarloPiModule pi = new MonteCarloPiModule();
Seeds.start( "/path/to/seeds/seed/folder" , false);
PipeID id = Seeds.startPattern(new Operand( (String[])null, new Anchor( "hostname"
, Types.DataFlowRoll.SINK_SOURCE), pi ) );
System.out.println(id.toString() );
Seeds.waitOnPattern(id);
System.out.println( "The result is: " + pi.getPi() ) ;
Seeds.stop();
} catch (SecurityException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
}27
This code deploys framework and starts execution of pattern
Different patterns have similar code
![Page 28: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/28.jpg)
Compiling/executing
• Can be done on the command line (ant script provided) or through an IDE (Eclipse)
28
![Page 29: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/29.jpg)
Another exampleBubble sort with the pipeline pattern
29
Static test - Program executed using a fixed number of cores as given.Dynamic test - Number of cores dynamically changed during execution to improve performance. Platform: Dell 900 server with four quad-core processors and 64GB shared memory.
From Jeremy Villalobos’ PhD thesis. See thesis for more details and results
![Page 30: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/30.jpg)
Pattern operators
Example use: Heat distribution simulation (Laplace’s eq.)• Multiple cells on a stencil pattern work in a loop parallel
fashion, computing and synchronizing on each iteration. • However, every x iterations, they must implement an all-to-all
communication pattern to run an algorithm to detect termination.
30
Can combine patterns.
Example: Adding Stencil and All-to-All synchronous pattern
Directly from Jeremy Villalobos’s PhD thesis
![Page 31: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/31.jpg)
15.31
Tutorial page
![Page 32: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/32.jpg)
15.32
![Page 33: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/33.jpg)
33
Download page
![Page 34: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/34.jpg)
34
Open Source Apache License in progress
http://code.google.com/p/seeds-parallel-pattern-framework/
![Page 35: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/35.jpg)
Work in progress• Tutorial on using all-to-all pattern to solve the n-body
problem, and other documentation
• Documentation on advanced layer to enable programmers to develop their own patterns
• Exploring various ways to enhance and expand the work, using for example Aparapi
35
http://code.google.com/p/aparapi/
![Page 36: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/36.jpg)
ITCS 4145/5145 Parallel Programming• Pattern programming to be introduced into ITCS 4145/5145
Parallel Programming in Fall 2012.
36
To be taught on NCREN in same way as ITCS 4/5146 Grid Computing with instructors at UNC-Charlotte and UNC-Wilmington but in Fall 2012 just two sites.
Subsequently will be offered across NC. Regional/national workshops planned
External funding for this work pending.
![Page 37: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/37.jpg)
Pattern Programming Research Group• 2011
– Jeremy Villalobos (PhD awarded, continuing involvement)– Saurav Bhattara (MS thesis, graduated)
• Spring 2012– Yawo Adibolo (ITCS 6880 Individual Study)– Ayay Ramesh (ITCS 6880 Individual Study)
• Fall 2012– Haoqi Zhao (MS thesis)
Loosely related Spring 2012: Tim Lukacik and Phil Chung (UG senior projects on Eclipse PTP) 37
Openings!
![Page 38: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/38.jpg)
Some publications• Jeremy F. Villalobos and Barry Wilkinson, “Skeleton/Pattern
Programming with an Adder Operator for Grid and Cloud Platforms,” The 2010 International Conference on Grid Computing and Applications (GCA’10), July 12-15, 2010, Las Vegas, Nevada, USA.
• Jeremy F. Villalobos and Barry Wilkinson, “Using Hierarchical Dependency Data Flows to Enable Dynamic Scalability on Parallel Patterns,” High-Performance Grid and Cloud Computing Workshop, 25th IEEE International Parallel & Distributed Processing Symposium, Anchorage (Alaska) USA, May 16-20, 2011.
Also presented by B. Wilkinson as Session 4 in “Short Course on Grid Computing” Jornadas Chilenas de Computación, INFONOR-CHILE 2010, Nov. 18th - 19th, 2010, Antofagasta, Chile. http://coitweb.uncc.edu/~abw/Infornor-Chile2010/GridWorkshop/index.html 38
I regard this conference as a top conference in the field
![Page 39: Pattern Programming Barry Wilkinson University of North Carolina Charlotte CCI Friday Seminar Series April 13 th, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062518/56649e2c5503460f94b1ad8b/html5/thumbnails/39.jpg)
Questions
39