interactivity on the grid marcus hardt scc (the insitute formerly known as iwr ) @fzk
DESCRIPTION
Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK. Grid Computing – the dream. Ideal: Computer power Electrical power From Electrical power grid => computational grid Across organisationsal domains / countries Transparent access to Computing - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/1.jpg)
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 GridKa School 2008
Interactivity on the Grid
Marcus HardtSCC
(The insitute formerly known as IWR)
@FZK
![Page 2: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/2.jpg)
GridKa School 2008
Ideal: Computer power <=> Electrical power
From Electrical power grid => computational grid
• Across organisationsal domains / countries• Transparent access to
● Computing● Data● Network
• Large scale installations
Grid Computing – the dream
![Page 3: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/3.jpg)
GridKa School 2008
Before int.eu.grid started Middleware
:= Layer between application and operating system
gLite: one grid middleware●Development driven by CERN●Tools for data+computing of new accelerator●10 TB/year * 20 years, random access
Paradigm: Send job to where the data is
Job: Self contained application
![Page 4: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/4.jpg)
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 GridKa School 2008
g gLite installations in
Europe
![Page 5: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/5.jpg)
GridKa School 2008
Using a lightbulb in the glite world
Voltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
=> Powerplant automaticallychosen for you
=> Send lightbulb to powerplant=> Wait for electricity=> Lightbulb glows
Results come back ● About 20% of the bulbs broken
Describe the lightbulb
![Page 6: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/6.jpg)
GridKa School 2008
The interactive channel
Our idea for a solution
![Page 7: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/7.jpg)
GridKa School 2008
The team
![Page 8: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/8.jpg)
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 GridKa School 2008
Interactive European Grid Project
http://www.interactive-grid.eu
24 Months2,5 ME35 People
![Page 9: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/9.jpg)
9GridKa School 2008
Key achievements Established interest from research communities
Fusion, medicine, environment, HEP, astrophysics
MPI Open MPI (incl. Infiniband support) & PACX-MPI
Collaboration with EGEE
Interactivity: GVid (& steering through Glogin) CrossBroker Integration in Migrating Desktop (user and developer friendly!!!)
![Page 10: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/10.jpg)
10GridKa School 2008
From the Applications point of view
– Analyze requirements of reference applicationsEnsure that middleware copes the reference applications demands
– Application Porting Support– Promote collaborative
environments like AccessGrid
From the Infrastructure point of view
– Operate a production level infrastructure 24/7
– Support Virtual Organizations at all levels
Running the VO (user support)
The challenges of int.eu.grid
From the middleware point of view– Parallel Computing (MPI)
Support intracluster Jobs with OpenMPISupport intercluster Jobs with PACX-MPI
– Advanced visualization tools allowing simulation steering
• GVid, glogin– A Job scheduler that supports it
all– User friendly interface to the
grid supporting all this featuresIntegrating in the Migrating Desktop all the features
![Page 11: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/11.jpg)
11GridKa School 2008
Middleware Requirements
Provide computing resources
MPI support
The job should be started inmediately on the user desktop
Glide-in mechanismor… use an Interactive Session
The graphical interface should be forwarded to the user desktop
– Graphical interface to the grid Migrating Desktop– Supporting Visualization GVid
The user should be able to steer the simulation
– Real Time steering glogin
![Page 12: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/12.jpg)
12GridKa School 2008
MPI types supportedOpenMPI
Joint Effort / OSSBest of FT-MPI, LA-MPI, LAM-MPI, PACX-MPI
PACX-MPIInter-Cluster / SiteSupports Vendor-MPI
MPI-StartShell Scripts
– Abstraction LayerWMS / Schedulers and MPI-ImplementationsFlexibility (relocateable Shellscripts)
PACX-MPI
LAM/MPI
LA-MPI
FT-MPI
MPIImplementations Scheduler
WMS (CrossBroker)
MPI-STARTPACX-MPI
Vendor MPI
Application
Intra Cluster Communication
Inter Cluster Communication
PACX-MPI MPI-START
![Page 13: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/13.jpg)
13GridKa School 2008
MPI Support on the Grid
Our solution, an intermediate layer:
mpi-start
RESOURCE BROKER
MPIImplement.
MPI-START
Scheduler
To use: Add this to your JDL fileJobType = "parallel";SubJobType = "openmpi"; or = "pacxmpi";
![Page 14: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/14.jpg)
14GridKa School 2008
Middleware for Visualization & Steering
• Glogin– Lightweight tool for support
of interactivity on the gridGrid authenticated shell access “glogin host”
– No dedicated daemon needed such as sshd
– TCP Port Forwarding enables access to grid worker nodes with private IPs.
– X11 Forwarding
• GVidGrid Video ServiceVisualization can be executed remotely on a grid resource Transmits the visualization output to the user desktopCommunication of the interaction events back to the remote rendering machine
– Uses Glogin as bi-directional communication channel
![Page 15: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/15.jpg)
15GridKa School 2008
(i2)glogin
Workernode
Workernode
Workernode
Workernode
Workernode
Workernode
Workernode
Workernode
Workernode
WorkernodeGatekeeperGatekeeper
on the Grid
ClientClient
gloginPoint ofContact
job submit
glogin’
fork
socket
glogin’’
fork
inherited socket
return
return
Problem: GASS cache
interactivebidirectional connection
trafficforwarding
To use: Start glogin on your UI:
glogin -p 20800 Add this to your JDL file
InputSandbox = {"/opt/i2g/bin/i2glogin"};
InteractiveAgent = "i2glogin";
InteractiveAgentArguments = "-r
-p 20800:<UI.hostnamme> -t -c";
Interactive = True;
![Page 16: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/16.jpg)
16GridKa School 2008
GVidvt
k A
pp
MP4 Encoder
Event Decoder
TCPTransp.
SocketTransport
FileTransport
PipeTransport
rem
ote
Glu
tA
pp
licat
ion
GV
id
SD
LO
utp
utc
lie
ntMP4
Decoder
Event Encoder
TCPTransp.
SocketTransport
FileTransport
PipeTransport
Grid UI
glogin gloginGSS secured $ grid-proxy-init
$ glogin -x –C altix1`
![Page 17: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/17.jpg)
17GridKa School 2008
Concurrency: glide-in
Glide-in is a different way to submit job Uses condor glide-in
Resource Broker keeps a handle to job Change priority of jobs Submit ”high-priority-jobs” VO policies respected
![Page 18: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/18.jpg)
18GridKa School 2008
SchedulingAgent
ResourceSearcher
ApplicationLauncher
Condor-G DAGMan
CE
WN WN
LCG
CE
WN WN
gLite
MigratingDesktop
InformationIndex
ReplicaManager
CrossBroker
Interactive Job Broker– Automatic Job
Managementfor Parallel Applications
Resource SearchingJob ConditioningLaunching, Monitoring, RetryResult Retrieval
– Workflows, Interactive & Batch Jobs, MPI Support
JDL ExtensionsCompatible
– Best Effort Approachfor Failures / ProblemsImproved Job Startup Time
To use: Use the CrossBroker as RB Use an interactive CE
![Page 19: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/19.jpg)
19GridKa School 2008
User-FriendlyPlatform Independent
JavaRunning Sequential & MPI JobsRoaming Access Server interfaces with the UIData ManagementEasily Extendable
Plugin Mechnism for new Applications
Extension for Interactive Remote Visualization
GUI – Migrating Desktop + Backend
![Page 20: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/20.jpg)
20GridKa School 2008
GUI – Migrating Desktop + Backend
![Page 21: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/21.jpg)
21GridKa School 2008
Example: Fully integrated Application
Fusion Application – It integrates: MPI Support Crossbroker Visualization Videostreaming GUI Support Interactivity
![Page 22: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/22.jpg)
Fusion Reactor “TJ-II” in Zaragoza
![Page 23: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/23.jpg)
Individual trajectories computed on the grid
![Page 24: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/24.jpg)
Interactive Visualizator for ISDEP
![Page 25: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/25.jpg)
Particles hit the vacuume vessel
![Page 26: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/26.jpg)
GridKa School 2008
Another interactive cable: GridSolve
![Page 27: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/27.jpg)
GridKa School 2008
Another interactive cable: GridSolve
![Page 28: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/28.jpg)
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 GridKa School 2008
y=problem(x) <=> y=gs_call('problem', x)
Interface ●API for Java, C, Fortran, Matlab, Octave, ...
Easy to use:
●Transport input parameters to remote side●Execute “problem”●Transport result back
=> Reduce complexity of the grid to one function call
GridSolve interface
![Page 29: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/29.jpg)
GridKa School 2008
Movie of the life demonstration:●http://marcus.hardt-it.de/grid4matlab
Life demo on int.eu.grid
Life-Demo
![Page 30: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/30.jpg)
GridKa School 2008
Source code
![Page 31: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/31.jpg)
GridKa School 2008
![Page 32: Interactivity on the Grid Marcus Hardt SCC (The insitute formerly known as IWR ) @FZK](https://reader035.vdocuments.us/reader035/viewer/2022062409/56814736550346895db47574/html5/thumbnails/32.jpg)
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 GridKa School 2008
Interactive Job Support
Type = "Job";VirtualOrganisation = "imain";JobType = "Parallel";SubJobType = “openmpi";NodeNumber = 11;Interactive = TRUE;InteractiveAgent = “glogin“;InteractiveAgentArguments = “-r –p 195.168.105.65:23433“;Executable = "test-app";InputSandbox = {"test-app", "inputfile"};OutputSanbox = {"std.out", "std.err"};StdErr = "std.err“;StdOutput = "std.out";Rank = other.GlueHostBenchmarkSI00 ;Requirements = other.GlueCEStateStatus == "Production";