fujitsu laboratories of europe © 2003 unicore technology dr. david snelling grid school july 17,...
Post on 19-Dec-2015
216 views
TRANSCRIPT
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Unicore Unicore TechnologyTechnology
Dr. David SnellingDr. David Snelling
Grid SchoolGrid School
July 17, 2003July 17, 2003
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
HistoryHistory
HistoryHistoryConceived in 1997UNICORE and UNICORE Plus BMBF ProjectsEuroGrid and GRIP EC ProjectsRealityGrid UK e-Science ProjectSupported Service from Pallas (UnicorePro)
Focus on Seamless ComputingFocus on Seamless ComputingOpen Source and Online Demo SiteOpen Source and Online Demo Site
www.unicore.org
OGSA Compliant DemonstratorOGSA Compliant DemonstratorJune 7th 2002: First release.September 4th 2002: Application Steering Deployment
Fast Track project in RealtyGrid (UK e-Sciecnce).www.sve.man.ac.uk/Research/AtoZ/RealityGrid/AHM2002/
Official OGSI based development under way.
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Unicore ImplementationUnicore Implementation
Unicore ArchitectureUnicore ArchitectureOpen Protocol, Abstract Job ObjectReference Implementation (one server, two clients)
Fujitsu Developed ComponentsFujitsu Developed ComponentsSecurity GatewayNetwork Job Superisor (NJS)
Authorization translation, AJO Incarnation, workflow scheduling, file management, ...
Target System Interfaces (TSI)Fujitsu, IBM, NEC, Hitachi, SGI, Sun, Mac, Sony (PS2), ...
Client Tool Kit
Pallas Developed ComponentPallas Developed ComponentProduction Ready GUI Based Client
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Important Unicore FunctionsImportant Unicore Functions
SecuritySecurityCertificate/PKI Based Security ModelInteroperable with Globus CertificatesSupport for stronger trust model than Globus
or Globus Trust Model using proxies.
WorkflowWorkflowLocal, remote, and nested task graphsFile management, transfer, and streamingIf and loop constructsFlow control based on task status, time events, and file-state events
GRID ManagementGRID ManagementHard/soft fail recovery.System administrator’s control interface.Extensive logging supportBuilt in certificate management in the GUI client.
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Functions ContinuedFunctions Continued
Resource BrokerResource BrokerMulti-site resource check prior to submit.
Estimate of time until execution.
Ticket generation and checking.
Dynamic brokering at execution time.
Interactive Access ExtensionInteractive Access ExtensionAllows standard terminal style interaction.
Unicore Single Sign-on Complete multi-site authentication & authorization.
Includes “Interactive Batch”.
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
ArchitectureArchitecture
Batch Subsystem
AJO/UPL
User Certificate
Job preparation/control Plugins
Unsafe Internet (SSL)
User authentication
UNICORESite List
UNICOREPro Client
Target System Interface (TSI)
Incarnated job
Commands
User mapping,job incarnation,job scheduling
TSI TSI
Any clustermanagement system
UNICORE SiteFZJ
...
Preparation andControl of jobs
Network Job Supervisor(NJS)
Safe Intranet (TCP)
IDB
Jobs and data transfer to other UNICORE sites
Status request
SV1 Blade files
UUDB IDBIDB
NJS
UNICORE Gateway
optional firewall
optional firewall
AJO/UPL
Runtime Interface
Arcon Client Toolkit User Certificate
UNICORESite List
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
SGI
Compile
Link
Transfer
Build executable
Distributed JobsDistributed Jobs
Run model
VPPExecute
Export
Get input data
IBMImport
Transfer
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
AJO: Unicore ProgrammingAJO: Unicore Programming
Abstract Job ObjectAbstract Job ObjectCollection of classes representing Grid functionsEncoded as Java objects (XML encoding possible)
Where to build AJOsWhere to build AJOsPallas client GUI - The user’s viewClient plugins - Grid deployer (You)Arcon client tool kit - Hard core (Some of you)
What can’t the AJO doWhat can’t the AJO doApplication level Meta-computing???
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Unicore Can DosUnicore Can Dos
Run & control remoteRun & control remoteShell scriptsApplications (without modifications)
FilesFilesTransfer client <-> serversTransfer servers <-> serversRemote management
Simple VO’sSimple VO’sResourceResource
DiscoveryBrokeringReservation
Provide site autonomyProvide site autonomyData archive accessData archive accessIntegration with local admin.Integration with local admin.
WorkflowWorkflowNested remote jobsNested local jobsLoops &ConditionalsTime based eventsFile state eventsRerun on failure
Seamless ComputingSeamless ComputingFiler streamingFiler streaming
Application to applicationApplication to Client
CORBA IntegrationCORBA IntegrationBuilt in cert. managementBuilt in cert. managementGrid administrationGrid administration
ToolsFailsafe restartDynamic hosting
AJOTour
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
InterfacesInterfaces
Client PluginsClient PluginsApplication specific support (CPMD, Gausian, NASTRAN, ...)
Unicore Protocol LayerUnicore Protocol LayerResource discovery, job submission, job management
Incarnation Data Base and TSI InterfaceIncarnation Data Base and TSI InterfaceIncarnation support
Unicore User Data BaseUnicore User Data BaseUser mapping and authorization
Broker/Scheduler InterfaceBroker/Scheduler InterfaceInterface defined and implemented in NJS.
Independent File Transfer InterfaceIndependent File Transfer InterfaceInterface defined
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
The Unicore ClientThe Unicore Client
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
WorkFlowWorkFlow
Do Construct
If Construct
ComplexDependencies
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
PluginsPlugins
CPMDPlugin & Wizard
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Plugin DevelopmentPlugin Development
Very Seamful: ScriptsVery Seamful: ScriptsDifferent for every system
Seamful: Command TaskSeamful: Command TaskPath to command
Environment variables
Seamless: Appl. ResourceSeamless: Appl. ResourceFully abstracted
No system/site differences
Application metadata
Ideal: PluginIdeal: PluginUser’s view of application
Not a computation,but a
solution
Client DevelopmentClient DevelopmentA few Java classes
GUIPlugin start/stopData container
Use Client FunctionsJob managementResource editor
Examples available
System DevelopmentSystem DevelopmentInstall applicationEdit server data base (IDB)
Application locationEnvironment variablesLibraries and data files
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Protocol LayeringProtocol Layering
Abstract Job ObjectAbstract Job Object
Java Zip Stream (File Transfer Only)Java Zip Stream (File Transfer Only)
Unicore Protocol LayerUnicore Protocol Layer
SSLSSL
IPIP
SSL
Unicore Protocol Layer
Java Zip Stream Structure, Integrity, & Compression
Authorization, Packetizing, & Control
Authentication, Integrity, & Encryption
Underlined functions can be turned off.IP
AJO
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
IDB and TSI InterfacesIDB and TSI Interfaces
Incarnation Data BaseIncarnation Data BaseMaps abstract representation to concrete
Includes resource description
Prototype auto-generation from MDS
Target System InterfaceTarget System InterfacePerl interface to host platform
Very small system specific module for easy porting
Current: NQS (several versions), PBS, Loadleveler, UNICOS, Linux, Solaris, MacOSX, PlayStation-2
Condor: Under development .
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
UUDBUUDB
Maps user certificate to local loginMaps user certificate to local loginSupport for limited roles.
Support for multiple logins per userSupport for multiple logins per userUser name or project based selection
Simple VO supportSimple VO supportMultiple users to a single account
Selection on user, local account, or projectAll users under a CA
ExtensibleExtensibleExternal interfaceTwo example UUDB implementations available.Can be integrated into existing user management
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Resource BrokerResource Broker
Support for simple QoS bidsSupport for simple QoS bidsTurnaround time and price
Ticket basedTicket basedSites retain control of ticket validity.Ticket checking at execution timeAdvanced reservation possible (if site supported)
Planned for 2003Planned for 2003Dynamic site selection supportInteroperable across Unicore and Globus resources
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
ComputeResourceComputeResource
BrokerBroker
NJSNJSIDBIDB UUDBUUDB
ExpertBrokerExpertBroker
DWDLMExpertDWDLMExpert OtherOther
LocalResourceCheckerLocalResourceChecker
UnicoreRCUnicoreRC GlobusRCGlobusRC
TranslatorTranslator
OntologicalTranslatorOntologicalTranslator
OntologyOntology
SimpleTranslatorSimpleTranslator
MDSGRAM
UoM Broker Architecture
TSI
ICMExpertICMExpert
Look up staticresources
Look upconfiguration
Verify delegatedidentities
Delegate to application-domain expert codeDelegate to Grid architecture-specificengine for local resource check
Pass untranslatable resources to Unicore resource checker
Look up resourcesLook updynamicresources
Delegate resource domain translation
Look up translations appropriateto target Globus resource schema
Broker hosted in NJS
To outside world
Get back set ofresource filters and set ofuntranslatable resources
TicketManagerTicketManager
UNICORE Components
EUROGRID Broker
Globus Components
GRIP Broker
Key:
Inheritance relation
Get signed ticket (contract)
Look up signing identity
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Alternate File TransferAlternate File Transfer
Uses independent FTP mechanismsUses independent FTP mechanisms“rcp” or “scp” within an Intranet
GridFTP for sites with relaxed firewall rules
Point to point leased line support
Alternative FTP invisible to usersAlternative FTP invisible to users
Interface extendable to other mechanismsInterface extendable to other mechanisms
FTP Authentication OptionsFTP Authentication OptionsVia Unicore Gateway
Completely independent
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
AFT ArchitectureAFT Architecture
Vsite A Vsite B
NJS A1
NJS A1
NJS B1
AFTM A AFTM B
UPL
UPL (optional)
ATFM Proprietary
GW B
AB
B
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Key TechnologiesKey Technologies
Seamless Computing ModelSeamless Computing ModelJob abstractionIncarnationFile staging, transfer, and streaming support
Security ModelSecurity ModelX509 certificate based security modelConsigner plus endorser modelSeveral CA policies usedFully flexible firewall support
Generic ClientsGeneric ClientsNo changes to application neededClient Tool KitPlugin Interface in GUI Client
Portable ServerPortable ServerJava + Perl
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
Application Steering ChallengeApplication Steering Challenge
Initial State Gss = 2.0 Gss = 2.0 Gss = 0.5
Gss = 0.5 Gss = 1.8 Gss = 1.8 Gss = 1.8
Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003
QuickTime™ and aGIF decompressor
are needed to see this picture.
OGSA RoadmapOGSA Roadmap
UnicoreGateway
Unicore ClientNetwork
JobSupervisor
ResourceDatabase
UserDatabase
Condor
NQS
GT3
ResourceBroker
Multiple firewalllayouts possible
OGSI Style Interfaces (Blue)OGSI Style Interfaces (Blue)Evolve these to standards: “Push and Pull” in the GGF
Key OGSA ServicesKey OGSA ServicesService creation and management
Resource discovery
User authorization
Service orchestration and workflow
Resource brokering
Alternative Client
Ext. AuthService