What we do inGrids e-Science

CyberInfrastructureand Peer-to-Peer

NetworksLos Alamos

September 23 2003

Geoffrey FoxCommunity Grids Lab

Indiana Universitygcf@indiana.edu

What do we do?• Portals: co-chair of Grid Computing Environment

research group at GGF• Metadata: co-chair at GGF of Semantic Grid RG

– Apply to Earth Science using GML (Geography mark up)• Collaboration: built a Web Service based collaboration

environment sharing applications and audio/video conferencing to desktops and PDA’s

• Web Service model for all applications• Messaging: Open “Grid Messaging System”

NaradaBrokering linking P2P and (Cellular) Grid• Autonomic services using managed messages• Applications

– CrisisGrid – Indiana and openGIS Consortium– SERVOGrid for Earthquake Science– Biocomplexity Grid-based Computational Environment

Collage of Portals

Earthquakes – NASAFusion – DoEComputing Info – DoDPublications -- CGL

NaradaBrokering

Minicomputer

Firewall

ComputerServer

PDA

Modem

Laptop computerWorkstationPeers

Peers

Audio/VideoConferencing Client

Audio/VideoConferencing Client

NaradaBrokering BrokerNetwork

“GridMPI” v. NaradaBrokering In parallel computing, MPI and PVM provided “all the features

one needed’ for inter-node messaging NB aims to play same role for the Grid but the requirements and

constraints are very different• NB is not MPI ported to a Grid/Globus environment

Typically MPI aiming at microsecond latency but for Grid, time scales are different• 100 millisecond quite normal network latency• 30 millisecond typical packet time sensitivity (this is one audio or video

frame) but even here can buffer 10-100 frames on client (conferencing to streaming)

• 1 millisecond is time for a Java server to “think” Jitter in latency (transit time through broker) due to routing,

processing (in NB) or packet loss recovery is important property Grids need and can use software supported message functions and

trade-offs between hardware and software routing different from parallel computing

NaradaBrokering Based on a network of cooperating broker nodes

• Cluster based architecture allows system to scale in size Originally designed to provide uniform software

multicast to support real-time collaboration linked to publish-subscribe for asynchronous systems.

Now has several core functions • Reliable order-preserving “Optimized” Message transport

(based on performance measurement) in heterogeneous multi-link fashion with TCP, UDP, SSL, HTTP, and will add GridFTP

• General publish-subscribe including JMS & JXTA and support for RTP-based audio/video conferencing

• Distributed XML event selection using XPATH metaphor• QoS, Security profiles for sent and received messages• Interface with reliable storage for persistent events

Laudable Features of NaradaBrokering

Is open source http://www.naradabrokering.org Will have client “plug-in” as well as standalone brokers Will have a discovery service to find nearest brokers Does tunnel through most firewalls without requiring

ports to be opened Supports JXTA, JMS (Java Message Service) and more

powerful native mode Transit time < 1 millisecond per broker Will have setup and broker network administration

module

NaradaBrokering Naturally Supports Filtering of events to support different client

requirements (e.g,. PDA versus desktop, slow lines, different A/V codecs)

Virtualization of addressing, routing, interfaces Federation and Mediation of multiple instances of Grid

services as illustrated by • Composition of Gridlets into full Grids (Gridlets are single

computers in P2P case)• JXTA with peer-group forming a Gridlet

Monitoring of messages for Service management and general autonomic functions

Fault tolerant data transport Virtual Private Grid with fine-grain Security model

Grid Messaging Substrate

Consumer Service

SOAP+HTTPGridFTPRTP ….

Messaging Substrate

Consumer Service

Standard client-serverstyle communication.

Substrate mediatedcommunication removestransport protocoldependence.

SOAP+HTTPGridFTPRTP ….

Any Protocol satisfying QoS

Protocols have become overloaded e.g. MUST use UDP for A/V latency requirements but MUSTn’t use UDP as firewall will not support ………

Heterogeneous Routing in NB

Mediation in Cellular Grid using NB as interface agent• Build Virtual Private

Grid

Gridlets

SatelliteUDP

FirewallHTTP

Dial-upFilter

A

B1

Hand-HeldProtocol

FastLink

Software MulticastB2

B3NB Brokers Client Filtering

Grid formed fromMultiple cells

Architecture of Message Layer Need to optimize not only routing of particular messages but classic

publish/subscribe problem of integrating different requests with related topics (subscribe to sports/basketball/lakers and sports)

Related to Akamai, AOL … caching and Server optimization problem

1-> N Grid Clients

Hypercube ofNB Brokers (logicalnot physical)

N≈100 forDistanceEducationScale to billions of grid clients?

Autonomic Services In a Web (Grid) Service architecture, the state of any service is defined

by its initial condition and all the messages (including ordering) that it receives• This how shared event model of collaboration works

This is a “Finite State Change” model analogous to saving file and “undo” command in many editors

NB plus a robust store can “guarantee” to save all these messages for (all) services

This allows one to build both "autonomic data transport" and "autonomic services" since these services can sustain packet losses in transport and can also sustain failures of apps/brokers• archived messages (previous invocations, published events etc) can

be retransmitted to reconstruct state at the service or to correct a transport error.

Further anomalies in message traffic (such as a publisher or subscriber are silent) can be detected by NB and signal problems

We are building examples of both scenarios using GridFTP as our data transport example

We will build a sample autonomic visualization service with detection of failed servers and brokers

Collaborative SVG Web Service SVG is W3C 2D Vector Graphics standard and is interesting for

visualization and as a simple PowerPoint like application• Further SVG is built on W3C DOM and one can generalize results

to all W3C DOM-based applications (“all” in future?) Apache Batik SVG is Java and open source and so it is practical

to modify it to explore• Real Applications as a Web Service

• Collaboration as a Web Service

• MVC model and web services with implications for portlets We use NaradaBrokering and XGSP to control collaboration;

support PDA Cell-phone and desktop clients; are restructuring Batik as MVC Web Service• Good progress in all areas see

• http://www.svgarena.org for SVG Games

• http://grids.ucs.indiana.edu/ptliupages/projects/carousel/ for PDA

Web Service Model for Application Development

W3C DOM User Interface

W3C DOM Raw (UI) Events

Application as a Web serviceW3C DOM Semantic Events

Data

User FacingPorts

Resource Facing Ports

Events as Messages

Rendering as Messages

View

Control

ModelNarada

Brokering

Interrupts in traditional monolithic applications become“real messages” not directly method callsNatural for collaboration and universal access

Natural inMVC Model

Application as a Web serviceApplication as a Web service

Participating Client

RenderingRendering

User Interface

W3C DOM Events

From Master

FromCollaborationAs a WS

Events

Application as a Web serviceApplication as a Web service

Master Client

RenderingRendering

User Interface

W3C DOM Events

To Collaborative Clients

FromCollaborationAs a WS

Events

Control flow for collaborative SVG clients

Figure 3 Control flow for collaborative SVG clients

Collaborative SVG As A Web ServiceCollaborative SVG As A Web Service

NaradaBrokering

Collaborative SVG Chess Collaborative SVG Chess Game in Batik BrowserGame in Batik Browser

Players

Observers

XGSP Web Service MCU Architecture

SIP H323 Access Grid Native XGSPAdmire

Gateways convert to uniform XGSP Messaging

High Performance (RTP)and XML/SOAP and ..

Media ServersFilters

Session ServerXGSP-based Control

NaradaBrokeringAll Messaging

Use Multiple Media servers to scale to many codecs and manyversions of audio/video mixing

NB Scales asdistributed

WebServices

NaradaBrokering

Polycom, Access Grid and RealVideo views of multiple streams using CGL A/V Web Service

integrating SIP and H323

Integration of PDA, Cell phone and Desktop Grid Access

NaradaBrokering Communication Applications interface to NaradaBrokering through UserChannels

which NB constructs as a set of links between NB Brokers acting as “waystations” which may need to be dynamically instantiated

UserChannels have publish/subscribe semantics with XML topics Links implement a single conventional “data” protocol.

• Interface to add new transport protocols within the Framework

• Administrative channel negotiates the best available communication protocol for each link

Different links can have different underlying transport implementations• Implementations in the current release include support for

TCP,UDP, Multicast, SSL, RTP and HTTP.• Supports communication through proxies and firewalls such as

iPlanet, Netscape, Apache, Microsoft ISA and Checkpoint.

Pentium-3, 1GHz, 256 MB RAM100 Mbps LAN

JRE 1.3 Linux

hop-3

0

1

2

3

4

5

6

7

8

9

100 1000

Tra

nsit

Del

ay

(Mill

isec

onds

)

Message Payload Size (Bytes)

Mean transit delay for message samples in NaradaBrokering: Different communication hops

hop-2

hop-5 hop-7

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1000 1500 2000 2500 3000 3500 4000 4500 5000

Sta

nd

ard

De

via

tion

(M

illis

eco

nd

s)

Message Payload Size (Bytes)

Standard Deviation for message samples in NaradaBrokering Different communication hops - Internal Machines

hop-2hop-3hop-5hop-7

0

10

20

30

40

50

60

0 200 400 600 800 1000 1200 1400 1600 1800 2000

De

lay

(Mill

ise

con

ds)

Packet Number

Average delays per packet for 50 video-clients NaradaBrokering Avg=2.23 ms, JMF Avg=3.08 ms

NaradaBrokering-RTP JMF-RTP

0

1

2

3

4

5

6

7

8

0 200 400 600 800 1000 1200 1400 1600 1800 2000

Jitte

r (

Mill

ise

con

ds)

Packet Number

Average jitter (std. dev) for 50 video clients.

NaradaBrokering Avg=0.95 ms, JMF Avg=1.10 ms

NaradaBrokering-RTP JMF-RTP

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 200 400 600 800 1000 1200 1400 1600 1800 2000

Jitt

er

(M

illis

eco

nd

s)

Packet Number

900 audio clientsNaradaBrokering-RTP

JMF-RTP

0

50

100

150

200

250

300

500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500

Tra

nsit

De

lay

(Mill

ise

co

nd

s)

Message Payload Size (Bytes)

Transit delay for message samples in Narada, JXTA and Narada-JXTA Topology - III (8 routers) Internal Machines

NaradaBrPure JXTA

Narada-JXTAR R

R

R

R R

R

R

(a)

R R

R

R

R R

R

R

N

(b)

N N

N

N

N N

N

N

(c)

Collaboration and Web Services Collaboration has

a) Mechanism to set up members (people, devices) of a “collaborative sessions”

b) Shared generic tools such as text chat, white boards, audio-video conferencing

c) Shared applications such as Web Pages, PowerPoint, Visualization, maps, (medical) instruments ….

b) and c) are “just shared objects” where objects could be Web Services but rarely are at moment

• We can port objects to Web Services and build a general approach for making Web services collaborative

a) is a “Service” which is set up in many different ways (H323 SIP JXTA are standards supported by multiple implementations) – we should make it a WS

Shared Event Collaboration All collaboration is about sharing events defining state changes

• Audio/Video conferencing shares events specifying in compressed form audio or video

• Shared display shares events corresponding to change in pixels of a frame buffer

• Instant Messengers share updates to text message streams

• Microsoft events for shared PowerPoint (file replicated between clients) as in Access Grid

Finite State Change NOT Finite State Machine architecture Using Web services allows one to expose updates of all kinds as

messages• “Event service” for collaboration is similar to Grid notification service and we

effectively define SDE’s (service data elements) in OGSI

Group (Session) communication service is needed for the delivery of the update events• Using Event Messaging middleware makes messaging universal

Global-MMCS 2.0 (1) XGSP MCU We are building an open source protocol independent Web

Service “MCU” which will scale to an arbitrary number of users and provide integrated thousands of simultaneous users collaboration services.

We will deploy it globally and hope to test with later this year. The function of A/V media server will be distributed using

NaradaBrokering architecture.• Media Servers mix and convert A/V streams

Open XGSP MCU based on the following open source projects• openh323 is basis of H323 Gateway

• NIST SIP stack is basis of SIP Gateway

• NaradaBrokering is open source messaging from Indiana

• Java Media Framework basis of Media Servers

WSDisplay

WSViewer

WS Display

WS Viewer

Event(Message)

Service

Master

WSDisplay

WS Viewer

Web Service MessageInterceptor

Collaboration as a WSSet up Session with XGSP

Application orContent source

WSDL

Web Service

F

I

U

O

F

I

R

O

Shared Output Port Collaboration

OtherParticipants

Text ChatWhiteboardMultiplemasters

WSDisplay

WSViewer

WS Display

WS ViewerEvent

(Message)Service

Master

WSDisplay

WS Viewer

Collaboration as a WSSet up Session with XGSP

WebServic

e

F

I

U

O

F

I

R

O

Shared Input Port (Replicated WS) Collaboration

OtherParticipants

WebServic

e

F

I

U

O

F

I

R

O

WebServic

e

F

I

U

O

F

I

R

O

XGSP Conference Control Framework Components

User session management • User session management supports user sign-in, user

create/terminate/join/leave/invite-into XGSP sessions. Application Session Management

• XGSP application session management provides the services to A/V and data application endpoints and communities, controlling multipoint A/V RTP and data channels.

Floor Control • Floor control manages the access to shared collaboration

resources.

vic and RealVideo views of multiple streams

Polycom view of multiple video streams

Performance Test : GlobalMMCS1.0 We conducted extensive performance tests on audio

and video servers. Video:

• The test shows that our video server is capable of supporting 300 clients if there is only one video sender.

• Video Server Machine : 1.2GHz Intel Pentium III dual CPU, 1GB MEM, RedHat Linux 7.3

Audio: • Our tests show that audio server can support 5 concurrent

sessions (250 participants in total) without any packet droppings. • Audio Server Machine: 2.5GHz Pentium 4 CPU, 512MB memory,

Windows XP machine Scale with logarithmic Broker network

0

50

100

150

200

250

300

350

400

450

0 200 400 600 800 1000 1200 1400 1600 1800 2000

De

lay (

Mill

ise

co

nd

s)

Packet Number

Average delays/packet for 12 (of the 400 total) video-clients. NaradaBrokering Avg=80.76 ms, JMF Avg=229.23 ms

NaradaBrokering-RTP JMF-RTP

Comparison between the performance of NaradaBrokering and JMF

0

5

10

15

20

25

0 200 400 600 800 1000 1200 1400 1600 1800 2000

Jitte

r (

Mill

ise

co

nd

s)

Packet Number

Average jitter/packet for 12 (of the 400 total) video clients. NaradaBrokering Avg=13.38 ms, JMF Avg=15.55 ms

NaradaBrokering-RTP JMF-RTP

Comparison between the performance of NaradaBrokering and JMF

Global-MMCS 2.0 (2) Portlets Collaboration clients will be built into portlets by

creating Java Applet or ActiveX controls for the non-HTML clients and adding them into HTML pages.

A collaboration portlet opens local services for XGSP application session management and floor control. • Node Manager portlet invoke the service to control local portlets

Apache Jetspeed seems good open source technology supporting this model

Portlets such as Access Grid portlet (really a VIC portlet) can be reused by Grid Portal Developers

Unicast AG Portlet

Multicast Multi-stream AG Portlet Java applet supports

multicast AG with multiple streams

In Jetspeed, easiest to have fixed size but this doesn’t fit well natural range of 1-20 separate streams

Workflow GCEs and Problem Solving Environments (PSEs)

• There is some confusion between fields of workflow (Grid Computing Environments GCE) and PSEs

• To extent PSEs “just” allow manipulation of “nuggets”, they are indistinguishable from a domain specific GCE

• They are distinct if they support intra nugget operations such as– Integration of mesh and simulation

– Closely coupled code linkage

– Generation of code from high level interface like Mathematica

• Even in latter case, a new generation of PSEs should be built with Grid architecture – e.g. message based – and using Grid services like metadata and notification

Web Services as a Portlet• Each Web Service naturally has a

user interface specified as “just another port” – Customizable for universal access

• This gives each Web Service a Portlet view specified (in XML as always) by WSRP (Web services for Remote Portals)

• So component model for resources “automatically” gives a component model for user interfaces– When you build your

application, you define portletat same time

Application orContent source

WSDL

Web Service

S

R

W

P

Application as a WSGeneral Application PortsInterface with other WebServices

User Face ofWeb ServiceWSRP Ports define WS as a Portlet

Web Services have other ports (Grid Service) to be OGSI compliant

Online Knowledge Center built from Portlets

• Web Services provide a component model for the middleware (see large “common component architecture” effort in Dept. of Energy)

• Should match each WSDL component with a corresponding user interface component

• Thus one “must use” a component model for the portal with again an XML specification (portalML) of portal component

A set of UIComponents

PortletPortlet PortletPortlet PortletPortlet PortletPortlet

XMLRSS, OCS, or otherLocal or remote

HTMLLocal files

JSP or VMLocal templates

WebPageRemote HTML

PortletPortlet

PortletsUser implementedusing Portal API

Portlets

Data

PortletController PortletController

Screen Manager

HTML

PSML

PortletControl

ECS

JSP template

ECS ECS ECS ECS

ECS ECS ECS

ECS Root to HTML

ECS

Turbine ServletJetspeedArchitecture

Portlets and Portal Stacks

• User interfaces to Portal services (Code Submission, Job Monitoring, File Management for Host X) are all managed as portlets.

• Users, administrators can customize their portal interfaces to just precisely the services they want.

Core Grid Services

User facing Web Service Ports

Application Grid Web Services

Aggregation Portals(Jetspeed)

Messa

ge S

ecu

rity, In

form

atio

n

Serv

ices

IU and OGCE Portal ArchitectureC

lient

s (P

ure

HT

ML,

Jav

a A

pple

t ..

)

Agg

rega

tion

and

Ren

derin

g

JetspeedInternalServices

Portlet Class:IFramePortlet

Portlet Class:VelocityPortlet

Portlet Class:JspPortlet

Portlet Class:WebForm

Gateway(IU)

Web/Gridservice

Web/Gridservice

Web/Gridservice

Computing

Data Stores

Instruments

GridPortTexas

(Java)COG Kit

Clients Portal Portlets Libraries Services Resources

LocalPortlets

Remoteor ProxyPortlets

EmphasisLargely taken

from other projects

(Jetspeed)

Hierarchical arrangement

Jetspeed Computing Portal: Choose Portlets

4 available portletslinking to Web ServicesI choose two

Choose Portlet Layout

Choose 1-column Layout

Original 2-column Layout

Lists user files on selected host, noahsark.File operations include

Upload, download, Copy, rename, crossload

Tabs indicate availableportlet interfaces.

File management

Sample page with several portlets:

proxy credential manager,submission, monitoring

Provide information about application

andhost parameters

Select applicationto edit

Administer Grid Portal