![Page 1: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/1.jpg)
24 Sept 2007
ADASS XVII : London, UK 1
Cloudspace: virtual environments in the VO
Matthew J. Graham (Caltech)Roy Williams (Caltech)
THE US NATIONAL VIRTUAL OBSERVATORY
![Page 2: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/2.jpg)
24 Sept 2007
ADASS XVII : London, UK 2
Scenario
• For the good of the community, I expose my super data mining algorithm as a service
• It runs on the local cluster in the basement
Time
Performanceefficiency
![Page 3: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/3.jpg)
24 Sept 2007
ADASS XVII : London, UK 3
The problem: utility computing
• Dynamic allocation of computing resource to meet demand: – more users – more data– more jobs
• Traditionally throwing more hardware at a problem requires prior software porting: – complexity of applications– consistency of the environment
• Cannot enforce Quality of Service
![Page 4: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/4.jpg)
24 Sept 2007
ADASS XVII : London, UK 4
The solution: virtualization
• Virtual workspaces– An abstraction of an execution environment that
can be made dynamically available to authorized clients by using well-defined protocols
– resource quota (e.g. CPU, memory share) – software configuration (e.g. O/S, provided
services).
• Implement on Virtual Machines (VMs)– abstraction of a physical host machine– hypervisor intercepts and emulates instructions
from VMs and allows management of VMs– VMWare, Xen, etc.
![Page 5: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/5.jpg)
24 Sept 2007
ADASS XVII : London, UK 5
Existing Technologies
• Amazon EC2 - “computing in the cloud”– Use Xen VMs– Images, Instances and Security
• FlexiScale from XCalibre/Virtual Iron - “cooler than EC2”
• Google/Microsoft?• Problem becomes availability of the right
type of hypervisor
![Page 6: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/6.jpg)
24 Sept 2007
ADASS XVII : London, UK 6
Globus Workspace
• WSRF• Back-end implementation on UChicago
Teraport cluster and EC2
Metadata instance
Deployment request
WorkspaceFactory Service
Workspace Service
Workspace Resource InstanceDeployment RP
Inspect & manage
Notify
Create() Authorize & instantiate
Contains GLOBUS
![Page 7: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/7.jpg)
24 Sept 2007
ADASS XVII : London, UK 7
Cloudspace
• Based on VOSpace– IVOA standard interface for distributed data storage– Lightweight abstraction layer (Façade pattern)– Sits on top of proven solutions such as http and
SRB
• Resource oriented• Data and services are both first-class entities • Universal Worker Service interface to manage
resources
![Page 8: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/8.jpg)
24 Sept 2007
ADASS XVII : London, UK 8
Resource-oriented computing
• A resource is an abstract set of information• Each resource may be identified by one or more logical
identifiers• A logical identifier may be resolved within an
information-context to a physical resource representation• Computation is the reification of a resource to a physical
resource representation• Resource representations are immutable• Transreption is the isomorphic lossless transformation of
one resource representation to another• Computational results are resources and are identified
within an address space
![Page 9: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/9.jpg)
24 Sept 2007
ADASS XVII : London, UK 9
Logical identifiers
• URI: – [scheme-name]://[scheme-specific-address]– Cloudspace scheme is csp - why not ivo://?– URI mapping (links)– Different URI schemes return different representations (http,
ftp)cf. HTTP MIME type and headers
• Data objects– use VOSpace syntax:
vos://nvo.caltech!vospace/myTable1
• Services– use regular IVOA identifiers to address image:
ivo://nvo.caltech/service/paramSweeper– csp to address instantiations:
csp://nvo.caltech/service/paramSweeper/1234
![Page 10: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/10.jpg)
24 Sept 2007
ADASS XVII : London, UK 10
Resource representations
• Data objects use VOSpace <node>– Arbitrary metadata (properties, tags)– Transreption formats (views)
• Service image uses VOResource mechanism– Extension schemata to handle virtualization
metadata (Globus equivalent)
• Service instantiation uses UWS representation
![Page 11: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/11.jpg)
24 Sept 2007
ADASS XVII : London, UK 11
Universal Worker Service
• URIs:– /(jobs): Job List– /(jobs)/(job-id) : Phase, Termination Time,
Quote, Results List– /(jobs)/(job-id)/phase: Phase– /(jobs)/(job-id)/termination: Termination
Time– /(job)/(job-id)/quote: Quote– /(job)/(job-id)/results: Results List
![Page 12: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/12.jpg)
24 Sept 2007
ADASS XVII : London, UK 12
Computing in the cloud
• Can use standard endpoint of service– http://some.service.endpoint
• Define computation in terms of result URI– comp:// scheme refers to computations:
comp://nvo.caltech/service/paramSweeper/1234+data@vos://nvo!caltech/vospace/myTable1+params@vos://nvo!caltech/vospace/myParam1
– Map to vos:// data object:vos://nvo!caltech/vospace/myResult1
• Memoization: data caching a function call
![Page 13: 24 Sept 2007 ADASS XVII : London, UK1 Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) T HE US N ATIONAL V](https://reader037.vdocuments.us/reader037/viewer/2022103022/56649d3a5503460f94a140e2/html5/thumbnails/13.jpg)
24 Sept 2007
ADASS XVII : London, UK 13
Not just blue-sky thinking
• Proof-of-concept system– NetKernel– Ruby script to interface with EC2 as
backend
• Other packaged software– War files