cloud computing with nimbus - open grid forum · 3/2/09 the nimbus toolkit:...

Cloud Computing with Nimbus March 2009, OGF25

Thilo Kielmann (slides by Kate Keahey [email protected])

3/2/09 The Nimbus Toolkit: http//workspace.globus.org

Nimbus

Nimbus goals Allow providers to build clouds

Private clouds (privacy, expense considerations) E.g., Workspace Service: open source EC2 implementation

Allow users to use cloud computing Do whatever it takes to enable scientists to use IaaS E.g.,Context Broker: creates turnkey virtual clusters

Allow developers to experiment with Nimbus For research or usability/performance improvements Community extensions and contributions, e.g Ian Gable

and his team (UVIC) contributed a monitoring component

First released in September 2005


Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

VWS Service

The Workspace Service


The Workspace Service

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

Pool node

The workspace service publishes information about each workspace

Users can find out information about their workspace (e.g. what IP

the workspace was bound to)

Users can interact directly with their

workspaces the same way the would with a

physical machine.

VWS Service


Workspace Service: Interfaces and Clients

Two kinds of interfaces: Web Service Resource Framework (WSRF)

Web Services + state management (WS-Notification) Cloud client (similar commads to the EC2 client)

Elastic Computing Cloud (EC2) Supported: ec2-describe-images, ec2-run-instances, ec2-describe-

instances, ec2-terminate-instances, ec2-reboot-instances, ec2-add-keypair, ec2-delete-keypair

Unsupported: availability zones, security groups, elastic IP assignment, REST

Works with EC2 clients

You can use one, or the other, or both


Workspace Service: Security

Authenticating and authorizing requests WSRF interfaces: GSI

Support for proxies, VOMS, Shibboleth (via GridShib), custom PDPs

EC2 interfaces Vanilla PKI X509 support

Secure access to VMs EC2 key generation or accessed from .ssh

Towards protecting data from the provider Extensions from Vienna University of Technology: Paper: Descher et al., Retaining Data Control in

Infrastructure Clouds, ARES (the International Dependability Conference), 2009.


Workspace Service: Networking

Network configuration External: public IPs or private IPs (via VPN) Internal: private network via a local cluster

network

Each VM can specify multiple NICs mixing private and public networks (WSRF only) E.g., cluster worker nodes on a private

network, headnode on both public and private network


Workspace Components

workspace control

workspace resource manager

workspace pilot

workspace client

workspace service

EC2

WSR

F

OpenNebula Project


Turnkey Virtual Clusters

Turnkey, tightly-coupled cluster Shared trust/security context Shared configuration/context information

Context Broker goals Every appliance Every cloud provider Multiple distributed cloud providers

Used to contextualize 100s of virtual nodes for EC2 HEP STAR runs, Hadoop nodes, HEP Alice nodes…

Working with rPath on developing appliances, standardization


Science Clouds

Goals: Enable scientific projects to experiment with IaaS clouds Evolve software in response to the needs of scientific projects A laboratory for exploration of cloud interoperability issues

Participants University of Chicago (since 03/08, 16 nodes), University of

Florida (05/08, 16-32 nodes, access via VPN), Masaryk University, Brno, Czech Republic (08/08), Wispy @ Purdue (09/08)

In progress: IU, Grid5K, Vrije, others Using EC2 for large runs

http://workspace.globus.org/clouds


Who Runs on Nimbus?

Hadoop

AliEn

GT-scalability

STAR

Montage workflows

GridFTP testing

workspace-team

Testing

OSG

geofest

bioinformatics

Other

Project diversity: Science, CS, education, build&test…


STAR STAR: a high-energy physics experiment Need resources with the right configuration

Complex environments Consistent environments

A virtual OSG STAR cluster OSG cluster: OSG CE (headnode), gridmapfiles, host

certificates, NSF, Torque, worker nodes: SL4 + STAR Requirements

One-click virtual cluster deployment Moving virtual clusters: Science Clouds -> EC2

From proof-of-concept to productions runs Work by Jerome Lauret, Doug Olson, Leve Hajdu, Lidia

Didenko at BNL Signinficant productions runs in progress now: results to be

published at Quark Matter conference and CHEP


Alice HEP Experiment at CERN

Collaboration with CERNVM project


Making a Hadoop Cloud

Papers: “Sky Computing”, by K. Keahey, A. Matsunaga, M. Tsugawa, J.

Fortes. Submitted to IEEE Internet Computing. “CloudBLAST: Combining MapReduce and Virtualization on

Distributed Resources for Bioinformatics Applications” by A. Matsunaga, M. Tsugawa and J. Fortes. eScience 2008.

U of Florida U of Chicago

Purdue

Hadoop cloud


Thoughts…

Science-driven cloud computing How does IaaS fit into our world?

Combine with what we have (grid computing) Explore new potential

Interoperability Academic vs commercial resources Standards: “rough consensus & working code”

Importance of open source Drive requirements into the infrastructure, customize Drive the development of standards

cloud computing with nimbus - open grid forum · 3/2/09 the nimbus toolkit:...

Documents