2 nd december 2002james magowan - surrey e-science day1 james magowan it specialist dynamic...
TRANSCRIPT
2nd December 2002 James Magowan - Surrey e-Science Day 1
James MagowanIT SpecialistDynamic e-business, IBM UK, Hursley Lab
2nd December 2002 James Magowan - Surrey e-Science Day 2
IBM Hursley - Background
3000 Employees
Largest Software Development Lab (of any organisation) outside of the USA
IBM Software Group – Middleware
• CICS• MQ Series• Java Technology Centre• Advanced Technology & eSolutions
2nd December 2002 James Magowan - Surrey e-Science Day 3
Computing:The Grid
Distributed
Networking:TCP/IP
Information:World Wide Web
Communications:e-mail
Internet Evolution
2nd December 2002 James Magowan - Surrey e-Science Day 4
Virtual, collaborative organizations sharing applications and data in an open heterogeneous environment
A vast aggregation of geographically dispersed computing resources
Virtual Servers, Storage and Instruments
Grid Middleware
Distributed Physical Servers and Storage
Grid ComputingDistributed Computing Over the Internet Using Open Standards
2nd December 2002 James Magowan - Surrey e-Science Day 5
StorageData
Applications
Processing I/O Operating System
Microcosm – Pre-Internet “System”Grid Computing
2nd December 2002 6
Grid Computing
....a single unified image
StorageData
Applications
Processing I/O Operating System
Macrocosm – Distributed Resources and Applications
2nd December 2002 James Magowan - Surrey e-Science Day 7
Motivations for Grid Computing
Increase Capacity
• Exploit distributed resources to provide capacity for high-demand applications
– Existing applications that cannot be run effectively on a single processor
– New large scale application that provide strategic business advantages
2nd December 2002 James Magowan - Surrey e-Science Day 8
Motivations for Grid Computing
Increase Capacity
• Exploit distributed resources to provide capacity for high-demand applications
Improve Efficiency / Reduce Costs
• Reduce infrastructure cost associated with over-provisioned resources
• Reduce the cost of manpower to manage and configure resources
2nd December 2002 James Magowan - Surrey e-Science Day 9
Motivations for Grid Computing
Reduce “Time to Results”
• Exploit opportunities for parallel computing to allow business critical computation to be completed in a timely fashion
• Gain competitive advantage by allowing computation to be executed morefrequently and on customer demand
112
2
3
4567
8
9
1011
March
29March
28March
27
Serial Execution
Parallel Execution
2nd December 2002 James Magowan - Surrey e-Science Day 10
Motivations for Grid Computing
Reduce “Time to Results”• Exploit opportunities for parallel computing to
allow business critical computation to be completed in a timely fashion
• Gain competitive advantage by allowing computation to be executed morefrequently and on customer demand
Enable Collaborations• Enable collaboration across
applications to integrate results • Support large multi-disciplinary
collaborations• Both within a single organization
and between partners
Simulation
PricingDesign
Design
Design Analytics
2nd December 2002 James Magowan - Surrey e-Science Day 11
IBMIBMIBM
Provide Reliability / Availability
• Use distributed resources • Monitor work progress• Restart failed jobs
Motivations for Grid Computing
112234
567891011
JobScheduler
TIMEOUT !
JOB 1JOB 1 JOB 2JOB 2 JOB 3JOB 3JOB 1JOB 1Recovery / Restart
2nd December 2002 James Magowan - Surrey e-Science Day 12
Provide Reliability / Availability
• Use distributed resources • Monitor work progress• Restart failed jobs
Support Heterogeneous systems
• Different hardware, system platforms, and available middleware
• Specialized equipment
Motivations for Grid Computing
Linux / Z-OS
IBM
IBM
AIX / Linux
IBM
IBM
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
serverpSeries
IBM
H C R U6
2nd December 2002 James Magowan - Surrey e-Science Day 13
Uses of Grid Technology4 Models, Unique Value Propositions
Increased
Processing
“Aggregate processing power from a distributed collection of heterogeneous systems”
Customer Values:
ProductivityFlexibilityResource useReliability/ Availability
ComplexityTotal cost of ownership
Decreased
2nd December 2002 James Magowan - Surrey e-Science Day 14
Increased
Processing
“Aggregate processing power from a distributed collection of heterogeneous systems”
Data
“Secure access and sharing of distributed data & information ina collaborative fashion”
Customer Values:
ProductivityFlexibilityResource useReliability/ Availability
ComplexityTotal cost of ownership
Decreased
Uses of Grid Technology4 Models, Unique Value Propositions
2nd December 2002 James Magowan - Surrey e-Science Day 15
Increased
Processing
“Aggregate processing power from a distributed collection of heterogeneous systems”
Resiliency“Improve the quality of service of distributed systems, despite unplanned events”
Data
“Secure access and sharing of distributed data & information ina collaborative fashion”
Customer Values:
ProductivityFlexibilityResource useReliability/ Availability
ComplexityTotal cost of ownership
Decreased
Uses of Grid Technology4 Models, Unique Value Propositions
2nd December 2002 James Magowan - Surrey e-Science Day 16
Increased
Processing
“Aggregate processing power from a distributed collection of heterogeneous systems”
Resiliency“Improve the quality of service of distributed systems, despite unplanned events”
Data
“Secure access and sharing of distributed data & information ina collaborative fashion”
On Demand“Access data & processing capabilities in a utility-like fashion…….. Make vs. Buy”
Customer Values:
ComplexityTotal cost of ownership
Decreased
Uses of Grid Technology4 Models, Unique Value Propositions
ProductivityFlexibilityResource useReliability/ Availability
2nd December 2002 James Magowan - Surrey e-Science Day 17
1. Intra-Grids
Grid
NAS/SAN
Grid
NAS/SAN
Grid Deployment OptionsA Function of Business Need, Technology and Organizational Flexibility
2nd December 2002 James Magowan - Surrey e-Science Day 18
1. Intra-Grids
2. Extra-Grids
GridGrid
NAS/SANNAS/SAN
Grid
NAS/SAN
VPN
Grid Deployment OptionsA Function of Business Need, Technology and Organizational Flexibility
2nd December 2002 James Magowan - Surrey e-Science Day 19
1. Intra-Grids
2. Extra-Grids
3. Inter-Grids
GridGrid
NAS/SANNAS/SAN
Grid
NAS/SAN
VPN
Cactus
NTG
(SF) Express Project
MFG
Fin.Services
Grid Deployment OptionsA Function of Business Need, Technology and Organizational Flexibility
2nd December 2002 James Magowan - Surrey e-Science Day 20
Grid & Autonomic Computing
Self-healing Discover, diagnose, and react to disruptions
Self-healing Discover, diagnose, and react to disruptions
Self-optimizingMonitor and tune resources automatically
Self-optimizingMonitor and tune resources automatically
Self-protecting Anticipate, detect, identify, and protect against attacks from anywhere
Self-protecting Anticipate, detect, identify, and protect against attacks from anywhere
Self-configuringAdapt automatically to the dynamically changing
environments
Self-configuringAdapt automatically to the dynamically changing
environmentsSelf-
ConfiguringSelf-
Configuring
Self-Healing
Self-Healing
Self-Optimizing
Self-Optimizing
Self-Protecting
Self-Protecting
2nd December 2002 James Magowan - Surrey e-Science Day 21
IBM and Grid
GGF - Global Grid Forum
• Working Groups• Approves GRID Open Standards
OGSA - Open Grid Services Architecture
• Standard proposed by Globus and IBM• Reference Implementation (Globus-3 Toolkit)
Open Source Software
2nd December 2002 James Magowan - Surrey e-Science Day 22
Applications
Middleware
Systems Management and Automation
Workload / Performance Management
Security
Availability / Service Management
Logical Resource Management
Clustering Services
Connectivity Management
Physical Resource Management
OSOSOS + + +
Open Grid Services Architecture
2nd December 2002 James Magowan - Surrey e-Science Day 23
Pro
fessio
nal S
erv
ices
Network
OGSA Enabled
Storage
OGSA Enabled
Servers
OGSA Enabled
Messaging
OGSA Enabled
Directory
OGSA EnabledFile
Systems
OGSA Enabled
Database
OGSA EnabledWorkflo
w
OGSA Enabled
Security
OGSA Enabled
General Middleware
Web Services
OGSI – Open Grid Services Infrastructure
Grid ServicesSystem Management Sevices
Open Grid Services Architecture (OGSA)
Applications
Au
ton
om
ic C
ap
ab
ilit
ies
Architecture FrameworkOGSA Structure
2nd December 2002 James Magowan - Surrey e-Science Day 24
OGSA Structure – OGSI
Architecture Framework
Web Services
HandleMapNotificationFactoryManagementRegistryLifecycleDiscovery
OGSI – Open Grid Services Infrastructure
Grid ServicesSystem Management Services
Exploits existing web services properties
• Interface abstraction (WSDL)• Protocol, language, hosting platform independence
Enhancement to web services
• State Management• Event Notification• Referenceable Handles• Lifecycle Management• Service Data Extension
2nd December 2002 James Magowan - Surrey e-Science Day 25
Architecture Framework
Web Services
HandleMapNotificationFactoryManagementRegistryLifecycleDiscovery
OGSI – Open Grid Services Infrastructure
Grid ServicesSystem Management Services
the globus project tm
www.globus.org
• Open source• Reference
implementation
Hosting platform
(Java)
Other Possible Hosting
Platforms(environments)
Microsoft (C#)Globus (C/C++)
(Python)
OGSA Structure – Open Hosting
2nd December 2002 James Magowan - Surrey e-Science Day 26
Web ServicesWebsphere Application Server
Architecture Framework
HandleMapNotificationFactoryManagementRegistryLifecycleDiscovery
OGSI – Open Grid Services Infrastructure
Websphere
OGSI
Service Collections
JobScheduling
FileTransfer
DataReplicationProvisioningLogging
ProblemDetermination
ResourceManagement
ClusterManagement
Policy
Grid ServicesSystem Management Services
• Value-added Grid Middleware and management applications
the globus project tm
www.globus.orgTivoli
OGSA Structure – Software Evolution
• Open architecture for “Grid Services”• Some “open source” reference
implementations• Vendor provider “value added”
implementationsthe globus project tm
www.globus.org
WebSphere
2nd December 2002 James Magowan - Surrey e-Science Day 27
European DataGrid
9 TB/day
Large Hadron Collider
2nd December 2002 James Magowan - Surrey e-Science Day 28
European DataGrid Geographically dispersed people and resources Petabytes of data per year (100 PB by 2010) Challenge:
• Coordinated Use of Distributed computing resources • Remote software development and physics analysis• Communication and collaboration at a distance
1800 Physicists150 Institutes32 Countries
2nd December 2002 James Magowan - Surrey e-Science Day 29
OGSA-DAI
Open Grid Services Architecture Database Access and Integration
Provide access to large scientific databases via the Grid.
Open Source and Standards
2nd December 2002 James Magowan - Surrey e-Science Day 30
OGSA-DAI
Data Access and Integration Services
• XML and Relational Database Management Systems• Design for any JDBC-enabled RDBMs• Test with Oracle, DB2, MySQL
Languages
• Java Handle Large Datasets –Optimised Performance
• BLOB’s and/or rows• Multiple Protocols for Data Transport (SOAP/HTTP and
GridFTP) Globus
• Globus-2 (GSI & GridFTP) and Globus-3
2nd December 2002 James Magowan - Surrey e-Science Day 31
e-Diamond - UK Digital Mammography Archive
Support for Breast Cancer Diagnosis, Treatment, Teaching and Epidemiological Studies
National eScience Centre Initiative
2nd December 2002 James Magowan - Surrey e-Science Day 32
e-Diamond - UK Digital Mammography Archive
“One of the pilot e-Science projects is to develop a digital mammography archive, together with an intelligent medical decision support system for breast cancer diagnosis and treatment. An individual hospital will not have supercomputing facilities, but through the Grid it could buy the time it needs. So the surgeon in the operating theatre will be able to pull up a high-resolution mammogram to identify exactly where the tumour can be found”
Tony Blair (speech to the Royal Society – 23 May 2002)
2nd December 2002 James Magowan - Surrey e-Science Day 33
James MagowanIT SpecialistDynamic e-business, IBM UK, Hursley Lab