virtual ez grid aaa/switch project nabil abdennadher hes-so, hepia (geneva)
DESCRIPTION
Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva). Virtual EZ Grid at a glance. AAA/Switch funding Partners: UniGE, USI, UniNE, HES-SO Start/End dates : February 2009 - July 2010 Objectives : - PowerPoint PPT PresentationTRANSCRIPT
1
Virtual EZ GridAAA/Switch project
Nabil Abdennadher
HES-SO, hepia (Geneva)
2Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid at a glance
• AAA/Switch funding
• Partners: UniGE, USI, UniNE, HES-SO
• Start/End dates : February 2009 - July 2010
• Objectives :
• Build a sustainable desktop Grid platform based on a Volunteer Computing (VC) middleware
• Evaluate it in a real world setting with two medical applications
3Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid objectives• Infrastructure: build a grid platform with more than 1000 PCs
• Non dedicated nodes
• Belong to several institutions.
• Reliability: manage the volatility of the “worker” nodes
• Check pointing, migration, restarting, etc.
• Security: guarantee the security of resources providers
• Negotiation Model: provide a resource-credit system
• Applications: two medical packages
• NeuroWeb: build neural maps extended from brain measurements
• MedGift: medical image analysis application
4Swiss Grid Day, Bern, 26th Nov. 2009
PLAN
• Virtual EZ Grid ingredients
• JOpera
• XtremWeb-CH
• EZ Grid
• Applications
5Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid ingredients
JOpera (developed at USI)
•A grid workflow management system: http://www.jopera.org
XtremWeb-CH (developed at HES-SO)
•A volunteer computing platform : www.xtremwebch.net
EZ grid (developed at UniNE and UniGE)
•Based on virtualization
•Job check-pointing
•Migration and restarting
ARC
6Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid architecture
Infrastructure
EZ-Grid
XtremWeb-CH (XWCH)
JOpera
ARC
7Swiss Grid Day, Bern, 26th Nov. 2009
JOpera
“Provide the scientist with a platform that takes care of all data handling and record keeping chores so that the user can concentrate on the science and not computer science”
Make it easy to build Grid and VC applications composed of multiple jobs
Based on slides by Cesare Pautasso
8Swiss Grid Day, Bern, 26th Nov. 2009
Drag, Drop and Connect
Based on slides by Cesare Pautasso
9Swiss Grid Day, Bern, 26th Nov. 2009
Run, Monitor, Steer and Debug
Based on slides by Cesare Pautasso
10Swiss Grid Day, Bern, 26th Nov. 2009
XtremWeb-CHcoordinator
Service Request
result
PC
Work Request
Work Result
Consumer(Users’ applications)
Providers(Workers)
PC
PC
PC
PC
PCPC
PC
PC
XtremWeb-CH at a glance
Job0
Job2
Job1Job3
Application
11Swiss Grid Day, Bern, 26th Nov. 2009
XtremWeb-CHcoordinator
Communication protocol
Work Request
Worker
Work Result
Work Alive
12Swiss Grid Day, Bern, 26th Nov. 2009
Use
r’ se
rvic
es
Java
AP
IC
/C+
+ A
PI
Pyt
hon
AP
I
Scheduler
XWCHDB
Work request
Work Result
Warehouse
Worker
Work Alive
Wor
ker
and
War
ehou
se s
ervi
ces
Use
r ap
plic
atio
nXtremWeb-CHCoordinator
Admin. Service
13Swiss Grid Day, Bern, 26th Nov. 2009
User application (jobs generator)int main ()
{
int Job0, Job1, Job2, Job3
…
Job0 = addjob (…);
while (getJobStatus (Job0,…)) ≠ COMPLETE);
Job0
Job1 = addjob (…);Job2 = addjob (…);while ((getJobStatus (Job1,…)) ≠ COMPLETE) && (getJobStatus (Job2,…)) ≠ COMPLETE));
Job2
Job1
Job3 = addjob (…); while (getJobStatus (Job3,…)) ≠ COMPLETE);
getJobFileOut (Job3,…);}
Job3
14Swiss Grid Day, Bern, 26th Nov. 2009
User application: waves generator
15Swiss Grid Day, Bern, 26th Nov. 2009
EZ Grid
Isolates XWCH jobs from “local” jobs
Why?: guarantee the privacy of the providers
How?: use virtualization technology
Supports check-point, and migration
Why?: manage the volatility of nodes
How?: remotely monitor the XWCH virtual machine
16Swiss Grid Day, Bern, 26th Nov. 2009
Work request
Work Result
Warehouse
Worker
Work Alive
XWCHcoordinator
OS (Windows)
VM Manager
OS (Linux)
XWCH
Application
EZ
Grid
mo
dul
e
OS (Linux)
XWCH
ApplicationL
oca
l app
lica
tion
XWCH worker
XWCH VM1 XWCH VM2
EZ Gridcoordinator
18Swiss Grid Day, Bern, 26th Nov. 2009
PLAN
• Virtual EZ Grid ingredients
• JOpera
• XtremWeb-CH
• EZ Grid
• Applications
19Swiss Grid Day, Bern, 26th Nov. 2009
Applications
NeuroWeb : build neuronal maps extracted from brain measurements
MedGift : medical image analysis and retrieval application
20Swiss Grid Day, Bern, 26th Nov. 2009
NeuroWebObjective
• Reconstruction of the electromagnetic brain map
• Which neuron is responsible of what?— Epileptic crisis (*)
— Parkinson
— Alzheimer
— Etc.
Why?
• Avoid invasive surgeries
21Swiss Grid Day, Bern, 26th Nov. 2009
Neuronal activity
MEG Scanner
# Sensors = 256dt = 1 millisecond
MEG scannerMagneto-EncephaloGraphy (MEG) scanner:provides temporal information (functional data)
Based on slides by Cédric Bilat
22Swiss Grid Day, Bern, 26th Nov. 2009
?NeuroWeb
# of sensors: 256
A large scale optimization problem
# of neurons: 60’000
Based on slides by Cédric Bilat
23Swiss Grid Day, Bern, 26th Nov. 2009
Mapping of the MEG signals
on voxels (IRM):
IRM
MEG
How does the system work?
Based on slides by Cédric Bilat
24Swiss Grid Day, Bern, 26th Nov. 2009
Start with a random matrix A0
In each step, calculate Ai .
• Ai = F (Ai-1, functional data, anatomic data)
Stop when Ai = Ai-1
~16 Mb
~360 Mb
The Algorithm
25Swiss Grid Day, Bern, 26th Nov. 2009
.
.
.
.
.
.
.
.
.
…
A0A1 Ai-1
Gridification
Ai
26Swiss Grid Day, Bern, 26th Nov. 2009
HD
XWCH
Worker
Worker
node
Main Memory
Data persistence
Persistent
server
XWCH
Job
Why ?• Avoid loading/storing data
from/to the HD
How ?• Data remain in main memory
even after the end of the task
27Swiss Grid Day, Bern, 26th Nov. 2009
Virtual EZ Grid today• Infrastructure: Around 500 workers
HES-SO (Geneva + Yverdon): ~250
UNiGe: ~ 200
UniNe: ~ 20
Univ. Franche Comté (France): ~30
• Reliability: The implementation of EZ Grid is in progress (50%)
• Security: Fully operational. based on SWITCHaai: Shibboleth-based AAI
• Negotiation Model: In progress (30%)
• Applications• NeuroWeb: a prototype (proof-of-concept) is already deployed
• MedGift: gridification will start soon (January 2010)
28Swiss Grid Day, Bern, 26th Nov. 2009
Swiss Grid Portal
Virtual EZ-GridSwiss Multi-Science
Computing Grid
Infrastructure
Virtual EZ Grid : Links with other projects
29Swiss Grid Day, Bern, 26th Nov. 2009
Model 1
• Prices are fixed by a “central” agency.
• All the “institutions” receives initial credit which:
• depends on the quantity of resources provided by the institution
• can be used against the usage.
• The client (from a given institution) chooses the priority of jobs. The goal is to:
• Minimize the cost of execution
• Execute the application ASAP
• The model determines the “best” prices which optimize the use of the platform
30Swiss Grid Day, Bern, 26th Nov. 2009
Model 2
XWCHcoordinator
Worker (1) Proposed price
(2) feedback(3) Price updating