triana dr matthew shields, school of computer science, cardiff university a problem solving...

Triana

Dr Matthew Shields, School of Computer Science, Cardiff University

A Problem Solving Environment and Grid Workflow Tool

Course Outline

IntroductionHistory and Overview

Tools and Toolboxes, Components and Groups

How to compose a workflow

Triana as a workflow editor

Course Outline cont.

Service-oriented ComponentsWeb services as components

Web service workflow

Peer 2 Peer services as components

Distributed service workflow

Grid-oriented ComponentsGrid file and job primitives as components

Complex Grid workflow

Install Triana

Unpack the triana.tar.gz to a location of your choiceSet env $TRIANA (Unix) or %TRIANA% (Windows) to the unpacked directoryRun $TRIANA/bin/triana (Unix) or %TRIANA/bin/triana.bat (Windows)Course website http://www.trianacode.org/coregrid/

About Triana

Distributed Problem Solving EnvironmentWorkflow Based

Multiple Component DomainsSignal Processing (original domain)

Image Processing

Text Processing

Audio

Composite Components

Our Research Motivation

Based on research within the Triana project:Triana - a vehicle for investigating research into distributed systemsa workflow-based Problem Solving Environment

Goals:To make the access of distributed services as seamless as possibleTo connect heterogeneous GridsTo abstract the core capabilities needed for service-based computing (in P2P, Web services or Grid Computing)

Research Scope

Triana is currently used in many projects

GridOneD - Radio Astronomy

DART - Distributed Audio Retrieval with Triana

WHiP - Workflows Hosted in Portals

Data-Mining Grid

BioDiversity WorldSee http://www.trianacode.org/collaborations

Components and Definitions

Component is unit of executionComponents are defined in XML files:

Naming informationInput and output portsParameter information

Why Components?To simplify the application design process and to speed up application development

Triana Workflow

Triana is inherently flow basedData flow - data arriving at component triggers execution

Control flow - control commands trigger execution

Decentralised executionData or Control messages sent along communication “pipes” from sender to receiver causes receiver to execute

Synchronous or Asynchronous messaging (Implementation dependant)

Multiple inputs can block or trigger immediately (Component designer defined)

Taskgraph

Internal object based workflow graph representation

Taskgraph - DAGTasksConnections

External XML representationSimple XML syntaxList of participating Task definitionsParent/Child connectionHierarchical (Compound components)

Simple Workflow Example

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Try It! $TRIANA/bin/triana

Example Continued

Image is converted to Grey scale and then thresholded (set image less than 136 to black and everything higher to white).difference is then taken between this image and a image where the white lines are thinned out (i.e. detail is taken away).The resulting image is basically the detail of the white parts which were pruned by the ShrinkWhite unit which gives an outline of an image.Group the unit and save it as a new compound component

Coalescing Binary Search

GEO 600 Coalescing

Binary Search Algorithm

implemented as a Triana workflow

Triana, the GAT and the GAP

P2PS JXTAWeb

Services

GAP Interface

UDDISOAP

P2PSDiscovery

P2PSPipes

JXTADiscovery

JXTAPipes

GAT Interface

Condor

Globus RLS

Unicore

PBS GridLab

GRMS

SGESSH

WSRF

LDR

.NET

Other..

GridFTP

Grid Computing:

Job Submission,

File servicesA Graphical

Grid Computing

Environment or Portal

Service Based Computing:

Deployment, discovery and communication with distributed services e.g. P2P and (GSI) Web services

Grid services

GAP Interface

Motivation by GAT (GridLab.org)A Simple Service based API, for

Service Deployment,Service DiscoveryPipe Based Communication

Static application interface with multiple middleware bindings

P2PS (name…?)JXTAWeb services

P2PS JXTAWeb

Services

GAP Interface

UDDISOAP

P2PSDiscovery

P2PSPipes

JXTADiscovery

JXTAPipes

Service Oriented Comms

network

babelfish.altavista.

com

BabelFish

en_fr

hello

bonjourService Discovery

Dynamic?Decentralized?

CommunicationMessage Format

SOAP?

Transport ProtocolTCP?UDP?

GAP

GAP Interface

Simple Java APIPeer-to-Peer Discovery

Pipe-Based Communication

Multiple BindingsJXTA

P2PS

Web Services (WSPeer)

P2P GAP Bindings

Two Main BindingsJxta

legacy implementation, due to upgrade soon…

P2PS Simple (lightweight) P2P System

Uses in Triana, NRL (sensor nets) and Simulations (within NS-2)

Useful in highly dynamic environments

Based on some Jxta principles i.e. pipes, rendezvous nodes, dynamic discovery, etc

Capable of building small-world networks (centralised-decentralised)

P2PS Overview

Discovery ServiceBroadcast/Locate Adverts + Queries in Discovery Subnet

Pipe ServiceConnect Pipes using Endpoint Resolvers

Rendezvous ServiceSend/Receive Adverts + Queries from known Rendezvous Peers

ResolversPluggable Network InterfacesFor Ns2 simulation environment

TCP Resolver ??? Resolver

DiscoveryService

RendezvousService

P2PS Peer

PipeService

UDP Resolver

NetworkUDP TCP ???

Service Creation, Discovery, and invocation

Drag a Audio.Output.Play unit to a new scratchpadRight click on the Play unit and select create service

Select Local Service [P2PS] and press CreateLeave the default settings and click ok

Use menu File -> New to create a new scratchpadSelect menu Services -> Discover Services

Type Play in the search name panel and click ok

You should see a number of remote Play unitsDrag one of the remote Play units to the new scratch padDrag a Audio.Input.LoadSound unit and connect to the playDownload a sound file from the tutorial site http://www.trianacode.org/coregrid/Double click the LoadSound unit or right click and select properties, choose the sound file, click ok, click the play button.

Service Deployment

Peer-to-peer servicesDistributing Audio !!! Generic example - can try and distribute using other networksUses multicast or rendezvous to discover servicesShows the basic interface to distributionCan publish compound/group componnentsPublish as Web services

Need a running UDDI server for service publication and discovery

WSPeer

High Level Interface to Web ServicesDiscovery InvocationDeploymentHosting

Abstract from usual Web Service Discovery and Communication Mechanisms (i.e. UDDI and HTTP)

P2PS Web Service Discovery?

Uses Apache AXIS as SOAP EngineExtends Capabilities of Apache AXIS

Stubless Invocation (including complex types)Non Standard Transports (i.e. P2PS)

WSPeer

WSPeer – P2PS

Application

WSPeer – HTTP/UDDI

deploy publish locate invoke

UDDI

HTTPServer

deploy

launch server

publish locate

invoke

deploy

publish locate

invoke

Web servicesConnecting Web services and integrating with other Triana units

Xmethods - http://www.xmethods.com/Services -> Import Services

Bible Verses

http://www.stgregorioschurchdc.org/wsdl/Bible.wsdlDrag the readBible service to a new scratch padConnect a Common.String.StringGen and a Common.String.StringViewer componentDouble click StringGen for the input window

Psalm 119:105Matthew 5:3-12,44-48Matthew 5:3-12;John 1:1-5

Press Play, double click StringViewer to see the output

Web Service Choreography

Multiple Services

Xmethods - http://www.xmethods.com/Services -> Import Services

Urdu Translator

http://www.apniurdu.com/SOAP/Urdu2.wsdlDrag the Translate service to the scratch padDrag a Common.Control.Duplicator unit the scratch padConnect readBible to Duplicatorconnect one output from Duplicator to the StringViewerConnect the other output to the Translator serviceDrag a new StringViewer to the scratch pad and connect the output of the Translator to thatPress play and double click the string viewersTry different inputs for StringGen

Web Services Resource Framework

WSRF defines conventions for accessing stateful resources (WS-Resources) via Web service interactions

Resources are refered to using EPR in message headers

In Triana we refer to EPR/WS-addresses as a “context” for extendibility, i.e. WS-Security context

Example using Globus 4 CounterService example

Import http://d220.astro.cf.ac.uk:8080/wsrf/services/CounterService?wsdl

The link between context and WSTypeGen is a “trigger” node to create a control dependency

Right click on WSTypeGen1, select Node Editor

Select “In Params” tab, click add, select “Trigger node only” select ok

Double click the WSTypeGen unit, type a number into the “int” field

To set the context on the Add service, double click or right click and select “Set Context”

Press play

Drag another WSTypeGen unit and Add service onto the scratch pad, set another integer and play again

GridLab GAT

Application-level API (Gridlab.org)Abstract ‘Common’ Capabilities required by Grid Applications

Resource DiscoveryJob SubmissionFile Transfer

Capabilities extracted through a number of Grid Application Scenarios.Makes life as a Grid programmer easier … Motivation for the GGF SAGA research group (Simple API for Grid Applications)

Using the GAT

GATContext context = new GATContext();Preferences prefs = new Preferences();prefs.put("ResourceBroker.adaptor.name", "grms");

URI exe = new URI("file:////bin/hostname");File outFile = GAT.createFile(context, prefs, new URI("file:///out")

SoftwareDescription sd = new SoftwareDescription();sd.setLocation(exe);sd.setStdout(outFile);

ResourceDescription rd = new HardwareResourceDescription();rd.put("hostname", "bouscat.cs.cf.ac.uk");

JobDescription jd = new JobDescription(sd, rd);ResourceBroker broker = GAT.createResourceBroker(context, prefs);

Job job = broker.submitJob(jd);

Visual GAT

GAT Primitives represented as Workflow ObjectsJob Tasks

File Tasks

Grid Data Flow implicit in Workflow ConnectionsFile Task -> File Task = Data Transfer

File Task -> Job Task = Pre-Stage File

File Task -> Local Task = File Read

Grid Operations handled by GAT AdaptorsGrid File -> Grid File = GridFTP

HTTP File -> Local Task = HTTP

Grid File -> GRMS Job = GRMS Pre-Stage

Grid Workflow in Triana

Local JavaUnit

File (Local/Remote)

TemporaryFile

Job (Local/Remote)

File Write Pre-Stage Post-Stage File Read

Visual GAT Alternative

Grid Operations Expressed ExplicitlyGridFTP Tasks

File Write Tasks

File Read Tasks

Not Intuitive

Not Future Proof

File Input (HTTP)

GridLab GAT

File CPI

Grid

FT

PA

dapto

r

HT

TP

Ada

ptor

Data

Movem

ent

Resource Broker CPI

LocalA

dapto

r

GR

MS

Ada

ptor

GR

AM

Ada

ptor

LocalA

dapto

r

www.gridlab.

org

Triana

WaveToy.par

GridLab GAT

File CPI

Grid

FT

PA

dapto

r

HT

TP

Ada

ptor

Data

Movem

ent

Resource Broker CPI

LocalA

dapto

r

GR

MS

Ada

ptor

GR

AM

Ada

ptor

LocalA

dapto

r

www.gridlab.

org

Testbed

Triana

File Staging (GridFTP)

WaveToy.par

Mercury

GridLab GAT

File CPI

Grid

FT

PA

dapto

r

HT

TP

Ada

ptor

Data

Movem

ent

Resource Broker CPI

LocalA

dapto

r

GR

MS

Ada

ptor

GR

AM

Ada

ptor

LocalA

dapto

r

www.gridlab.

org

Testbed

GRMS

DataMvment

Testbed

Triana

Delphoi

GAS

iGrid

Job Submission 1

WaveToy.par

Cactus

GridLab GAT

File CPI

Grid

FT

PA

dapto

r

HT

TP

Ada

ptor

Data

Movem

ent

Resource Broker CPI

LocalA

dapto

r

GR

MS

Ada

ptor

GR

AM

Ada

ptor

LocalA

dapto

r

www.gridlab.

org

Testbed

GRMS

DataMvment

Portal

Triana

GAS

Job Submission 2

Testbed

Cactus

InstantMess

MercuryDelphoiiGrid

GridLab GAT

File CPI

Grid

FT

PA

dapto

r

HT

TP

Ada

ptor

Data

Movem

ent

Resource Broker CPI

LocalA

dapto

r

GR

MS

Ada

ptor

GR

AM

Ada

ptor

LocalA

dapto

r

www.gridlab.

org

Testbed

Portal

Triana

Job Submission (GRAM)

Testbed

Cactus

InstantMess

GRAM

GridMonSteer

User Side Grid Side

WSPeer

User Interface

Web Service

GridMonSteer (gms)

cactus

phi_xy.jpgphi_yz.jpg

submit

Application(e.g. Triana)

run

monitor

invoke using SOAP

ResourceBroker

(e.g. GRMS)

gms http://www.xyx.org:4802/axis/gms cactus

phi_xy.jpgphi_yz.jpgrundirhttp://www.xyz.org/axis/gms

Triana Grid Workflow Summary

GridLab GATApplication-level APIAbstracts Common Capabilities required by Grid Applications such as Triana

Visual GATGAT Primitives represented in Triana as Workflow ObjectsGrid Data Flow implicit in Workflow Connections

GridMonSteerGeneric Architecture for Monitoring and Steering Legacy Applications

Future

Java GAT 2.0 (coming soon Vrije Universiteit)globus-4 web services, a windows shares adaptor (SAMBA/CIFS), an initial set of glite adaptors

Mrs DART - Distributed Audio Retrieval with TrianaDistribute music analysis workflows to P2P network to analyse user digital audio collections and make content based recommendations

AlchemistGeneric version of DART framework for distributed scientific search and analysis

WHiP - Workflows Hosted in PortalsShare workflows in collaborative Web portals for scientific collaboration and knowledge capture

Links

Triana - http://www.trianacode.org/

WSPeer - http://www.wspeer.org/

WHiP - http://www.whipplugin.org/

Mrs DART - http://www.mrsdart.com/

Java GAT - http://www.cs.vu.nl/ibis/javagat.html