1

History and New Developments in State-wide Grid Computing Course

NC A & T UniversitySaturday Oct 18, 2008

Dr. Barry WilkinsonUniversity of North Carolina

Charlotte

Dr. Clayton FernerUniversity of North Carolina

Wilmington

2

Outline

Grid Computing course 2004, 2005 New direction 2007 Grid Computing course 2008 Grid Computing course

3

Grid Computing Course

Undergraduate/graduate Hands-on with distributed grid infrastructure Teleconferencing facilities - students and faculty at

many institutions participating Expert guest speakers near end of course Probably first such course for undergraduate students

and distributed sites using large-scale teleconferencing facilities and a truly distributed grid infrastructure.

Funding from NSF and UNC- OP

4

Grid Computing Course 2004-2005

Western Carolina University

UNC Greensboro

Appalachian State University

UNC AshevilleWinston-Salem State

University

UNC Chapel Hill

NC State University

NC Central University

Lenoir Rhyne College

UNC Wilmington

Elon University

UNC Pembroke

UNC Charlotte

Wake Tech. Comm. College

© World Sites Atlas (sitesatlas.com)

SOUTH CAROLINA

VIRGINIA

TENNESSEE

GEORGIA

NORTH CAROLINA

5

Problem with previous offerings

In 2004 and 2005, course taught from a bottom-up perspective

We started with Web services then moved on the Grid services

Everything was command-line Only toward the end of the semester did we

introduce a workflow editor and schedulers (such as Condor or Sun Grid Engine)

6

2007 Grid Computing Course

Goal: More top-down perspective Started with a Grid portal interface and then did

command-line tasks afterwards so that students could see the “big” picture

Moved some assignments to be done on student’s own computer to eliminate problems with using a centralized server. Required students to install Globus core, and other

packages themselves Course taught as an experimental course in Fall 2007

with UNC-C, UNC-W and UNC-A only.

Assignment 1 Using Grid computing portal

Portal provides single sign-on to all grid resources.

Registration Portlet

9

Registration

Students filled out request for account Certificate Request generated and sent to

Certificate Authority automatically

Certificate signed , installed in MyProxy server, email confirmation to student

Student logged onto Portal, acquires a proxy, creates a java program locally, transfers it using portal, and submits job.

10

Registration Activities

New User

PURSe on-line registration form

CA/SystemAdministrator

Create accounts, set access control, sign certificate, …

Fill in formProvide password and other information

Email• Request Confirmation• Acknowledgement

11

Assignment 2 (Using command-line)

Students performed the same tasks as in Assignment 1 but used a command-line

Install ssh client (e.g. putty) if necessary Set up your credentials (grid-cert-request) Email Certificate Authority to sign certificate Install certificate and create proxy Submit various jobs with globusrun-ws Some jobs submitted with xml job description files

12

Assignment 3 (Using Condor)

Create proxy Check the status of the Condor pool Create a test submit description text file Submit jobs and check status Using different Condor universes Still a command-line interface (In 2005, we used Sun Grid Engine, which has

a GUI interface)

13

Assignment 4 (Creating and deploying a Grid Service)

Install GT 4 core and associated software JDK 1.4.2+ Ant 1.5.1+ Python 2.4+ Globus 4.0 core

Testing installation Start container

Create, deploy, and test simple GT4 Grid Services Deplore prewritten

service and test with client

Add Functionality to Service

Need to handle WSDL (XML) and other files

14

Assignment 5 (Using a GridNexus)

Install GridNexus workflow editor

Create Web service workflow

Create Grid service workflow

Submitting a job to GRAM

15

Assignment 6 (Implementing portlets with Gridsphere)

Install Gridsphere and associated software Java 6 SDK ant tomcat 5.5.20

Install Gridsphere First start Tomcat

Create portlets Prewritten odd-even

portlet Installation involves

handling deployment descriptor files etc.

Portlet to add, subtract, multiply and divide two numbers.

16

Assignment 6 (Implementing portlets with OGCSE2/Gridsphere)

17

Assignment 7 (MPI Program)

Write simple MPI program (Matrix Multiplication)

We couldn't do much more than work with “embarrassingly parallel” applications since we only had a few weeks to deal with MPI

The students only executed their programs on one cluster (not really using the Grid).

18

Mini Projects

Teams of 3 members Objective was to create a new Grid assignment Assignment had to involve created a Grid

application with a GUI (such as a workflow or portal)

Assignment had to be written up as though the student would give it to their classmates

Focus on dissemination May create future assignments

19

Mini Projects

Teams had to provide a written report (with solutions)

Teams had to give a presentation

20

Keeping to Assignment Schedules

Each assignment allocated 1-2 weeks to complete. Posted three dates:

1. Date assignment was set

2. Date that students had to report any system problems preventing them from proceeding

3. Date due Fall 2004 many system problems (Globus 3.2) Fall 2005 much fewer problems (Globus 4.0) Spring 2007, no system problems reported

21

Keeping to Assignment Schedules

In 2004 and 2005, students created and deployed their Grid services on the same machine

An error in a service could make the Globus container inoperable for the other students

In 2007, student install the Globus 4 core on an individual PC (lab machine or their own) and used their own container

This turned out to be a much better solution!

22

Lessons Learned Students responded positively to using their own

computers that were under their direct control.

Some students did have problems with their installations Nothing that could not resolved quickly such as not

setting paths or using the wrong version of the software.

Some Grid computing assignments still have to be done on a Grid platform but using personal computers where possible avoided many problems

23

Avoiding Problems

It requires immense work to prepare for a hands-on Grid computing course.

Critical that all assignments are fully tested prior to the start of class and that all computer systems are reliable and the software maintained.

Assignments went much smoother by requiring students to use personal computers when possible.

2008 Grid computing course

New NSF funding 2008-2010

Goal:

To demonstrate that GridNexus workflow editor will provide a better user-interface in the classroom

Participating sites: UNC-C, UNC-W, NC A & T and UNC-A

25

http://www.csc.uncw.edu/~cferner/ITCS4146F08/

Password protected

Arranged so that both instructors can modify page – very important

Course Home Page

Organizational details

We attempt to have both instructors give part of each class period to give variety within the class period.

Needs coordination so we have a standing telephone conversation before each class.

New assignments

Assigments added/modified to show benefits of GridNexus:

Assignment 1 User registration and using Portal

Assignment 2 Command line version of assignment 1

Assignment 3 GridNexus job workflow

Assignment 4 Using Condor, now includes ClassAd matchmaking and DAGMan workflow.

Assignment 5 Creating Globus 4.0 services

Assignment 6 GridNexus service workflow

Assignment 7 Portlet design in GridSphere .

28

Assignment 1

Students write a java program that computes the area of a flower bed (piece-wise function) using the trapezoid method.

Easy program

They use the portal to upload the class file to a UNCC server and run it through the portal

29

Assignment 2

Students write a java program that given the area of a flower bed computes the volume and cost of mulch to cover the bed.

Even easier program

They use the command line to upload the class file to a UNCW server and run it use globusrun-ws

30

Assignment 3

Students create a workflow that

executes the program to compute the area of the flower bed on UNCC server

Transfers the results to the UNCW server

Executes the program to compute the volume and cost of the mulch

Transfer the results back to the UNCC server

Graduate students create a workflow to do the same thing but using an RSL including file staging

31

Assignment 3

Students create a workflow that

executes the program to compute the area of the flower bed on UNCC server

Transfers the results to the UNCW server

Executes the program to compute the volume and cost of the mulch

Transfer the results back to the UNCC server

Graduate students create a workflow to do the same thing but using an RSL including file staging

32

Survey 1

Student were asked to fill out an online survey asking them various questions about the first 3 assignments

They were asked to compare how they would create a workflow moving data between servers using a portal and command line.

33

Plan for Assignment 6

Students will be asked to create a workflow to use Web Services within grid environment using GridNexus.

A simple workflow to work with a service where the resource is within the web service (they share the results)

A workflow where they use a factory to create a new resource

A workflow to perform the mulch problem using multiple servers and moving data

Evaluation plan

• sites (UNC-W and UNC-C)

• site (NC A & T)

• Professional evaluator (Barbara Heath)

UNC-C module development

Release to site

Jan 2008

August 2008

June 2008

UNC-W module development

NCAT site testing

Remote testing

site development and testing

Jan 2009

Second year material development:

August 2009

June 2009

Dec. 2008

Dec. 2009

Feedback Remote

evaluation

Exchange

Develop materials

Develop materials

Test collectively on NCREN

(joint course)

Feedback

Test locally Test locally

Remote evaluation

Exchange

Develop materials

Develop materials

Other institutions

Test locally

Feedback

Feedback

Demos