Cloud Storage Simplified
Adrian Cole / Cloudsoft
Tuesday, November 1, 11
Adrian Cole (@jclouds)founded jclouds march 2009chief evangelist at Cloudsoft
Tuesday, November 1, 11
Agenda
• intro• concepts• jclouds• code• awkward silence (or Q/A)
Tuesday, November 1, 11
Blob Storage
4
Set<String> containers = namespacesInMyAccount;
Map<String, InputStream> keyValues = contentsOfContainer;
Tuesday, November 1, 11
adrian@googlestorage
Love Letters
TronMovies
Goonies The Blob
ShrekThe One
putBlob
3d = trueurl = http://disney.go.com/tron
Blob Storage
5
Tuesday, November 1, 11
6
Can I use BlobStores portably?
Tuesday, November 1, 11
are they close enough?
7
Tuesday, November 1, 11
who builds and runs your service?
8
Tuesday, November 1, 11
9
Tuesday, November 1, 11
tenancy
10
Tuesday, November 1, 11
size and resumability
11
Tuesday, November 1, 11
How many copies?
12
Tuesday, November 1, 11
Consistency model?
13
Tuesday, November 1, 11
Are you experienced? Is your product?
14
Tuesday, November 1, 11
15
Tuesday, November 1, 11
PUT /sushi.avi HTTP/1.1 Host: adriansmovies.s3.amazonaws.comContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: signature x-amz-meta-Chef: Kawasaki
PUT /<api version>/<account>/adriansmovies/sushi.avi HTTP/1.1 Host: storage.clouddrive.com Transfer-Encoding: chunkedX-Auth-Token: session-token X-Object-Meta-Chef: Kawasaki
PUT /adriansmovies/sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.netContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: SharedKey <app>:signaturex-ms-meta-Chef: Kawasaki
POST /namespace/adriansmovies/sushi.avi HTTP/1.1 Content-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTx-emc-uid: <uid> x-emc-signature: signature x-emc-meta: Chef=Kawasaki
CODE AND SIGN THE HTTP REQUEST
16
Tuesday, November 1, 11
GET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes= 734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1
POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1Host: from_above Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...
PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1
CODE AND SIGN THE HTTP REQUEST
17
Tuesday, November 1, 11
POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...
PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1 Content-Length: 8Content-Type: text/plainAuthorization: Basic GpjbG9=Kawasaki
CODE AND SIGN THE HTTP REQUEST
18
Tuesday, November 1, 11
19
GET https://api.gogrid.com/api/grid/server/add?v=1.5&name=serverName&server.ram=memory&image=img55&ip=127.0.0.1&isSandbox=true&description=fooy&sig=3f446f171455fbb5574aecff4997b273&api_key=foo HTTP/1.1
POST http://serverManagementUrl/servers?format=json HTTP/1.1Accept: application/jsonContent-Type: application/json{"server":{"name":"ralphie","imageId":2,"flavorId":1,"sharedIpGroupId":2,"addresses":{"public":["127.0.0.1"]}}}
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1Content-Type: application/x-www-form-urlencodedVersion=2010-06-15&Action=RunInstances&ImageId=ami-voo&MinCount=1&MaxCount=5&KernelId=kernelId&Monitoring.Enabled=true&SecurityGroup.1=group1&SecurityGroup.2=group2&Placement.AvailabilityZone=eu-west-1a
POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1
Accept: application/vnd.vmware.vcloud.vApp+xmlContent-Type: application/vnd.vmware.vcloud.instantiateVAppTemplateParams+xml
<InstantiateVAppTemplateParams xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
name="my-vapp" xsi:schemaLocation="http://www.vmware.com/vcloud/v0.8 http://vcloud.safesecureweb.com/ns/vcloud.xsd">
<VAppTemplate href="https://vcloud.safesecureweb.com/api/v0.8/vAppTemplate/3" />
<InstantiationParams> <VirtualHardwareSection xmlns:q1="http://www.vmware.com/vcloud/v0.8">
<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</InstanceID> <ResourceType
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">3</ResourceType> <VirtualQuantity
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</VirtualQuantity> </Item>
<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">2</InstanceID> <ResourceType
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">4</ResourceType> <VirtualQuantity
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">512</VirtualQuantity> </Item>
<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">9</InstanceID> <ResourceType
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">17</ResourceType> <VirtualQuantity
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1024</VirtualQuantity> </Item>
</VirtualHardwareSection> <NetworkConfigSection>
<NetworkConfig name="my-vapp"> <Features>
<FenceMode>allowInOut</FenceMode> </Features>
<NetworkAssociation href="https://vcloud.safesecureweb.com/network/1990" /> </NetworkConfig>
</NetworkConfigSection> </InstantiationParams>
</InstantiateVAppTemplateParams>
POST https://www-180.ibm.com/cloud/enterprise/beta/api/rest/20090403/instances HTTP/1.1Accept: Accept: application/jsonContent-Type: application/x-www-form-urlencodedlocation=location&imageID=22&name=name&instanceType=instanceType&ip=1&publicKey=MOO&volumeID=2&oss.storage.id.0.mnt-%2Fmnt&insight_admin_password=myPassword1&db2_admin_password=myPassword2&report_user_password=myPassword3
POST https://rimuhosting.com/r/orders/new-vps HTTP/1.1Accept: Accept: application/jsonContent-Type: application/json{"request":{"instantiation_options":{"distro":"lenny","domain_name":"test.ivan.api.com"},"pricing_plan_code":"MIRO1B","meta_data":[]}}
Deal with ErrorsDeal with ConcurrencyDeal with Complexity
do you want to
Tuesday, November 1, 11
open source
feels like java (and clojure)
portability between clouds
deal with web complexity
unit testability
thread-safe and scalable
20
Tuesday, November 1, 11
BlobStore LoadBalancer
Compute Table
Portable APIs
Embeddable
Provider-Specific Hooks
Over 30 Tested Providers!
21
Tuesday, November 1, 11
jclouds locations
Location helps normalize placement across resource types All top-level resources have a locationLocation metadata is extensible
listAssignableLocations
IE
US
SG
22
Tuesday, November 1, 11
jclouds modularity
APIs are software focused Providers are offering focusedAPI + location + defaults = Provider
jclouds-blobstore
s3
aws-s3
walrus
eucaluptus-partnercloud-s3
googlestorage
scality-ring
scaleup-storage
23
Tuesday, November 1, 11
Alternatives to jclouds
Roll-your-own• RESTEasy, Jersey
Cloud-specific SDKs• typica, jets3t, Amazon SDK, vCloud SDK
Dasein Cloud API
24
Tuesday, November 1, 11
BlobStore
25
Tuesday, November 1, 11
java overview github jclouds/jclouds
// initcontext = new BlobStoreContextFactory().createContext("s3", accesskeyid, secret);blobStore = context.getBlobStore();
// create containerblobStore.createContainerInLocation(null, “adriansmovies”);
// add blobblob = blobStore.blobBuilder("sushi.avi").payload(file).build();blobStore.putBlob(“adriansmovies”, blob);
26
Tuesday, November 1, 11
clojure overview github jclouds/jclouds
(use 'org.jclouds.blobstore2)
(def *blobstore* (blobstore “azureblob” account key))
(create-container *blobstore* “movies”)(put-blob *blobstore* “movies” (blob “tron.mp4“ :payload tron-file))
27
Tuesday, November 1, 11
Who’s coding BlobStore?
28
Tuesday, November 1, 11
Filesystem github jclouds/jclouds
props.setProperty(FilesystemConstants.PROPERTY_BASEDIR, “basedir”);
blobStore = new BlobStoreContextFactory(). createContext("filesystem", props);
(blobStore "filesystem" “” “” FilesystemConstants/PROPERTY_BASEDIR “basedir”);
29
Tuesday, November 1, 11
BlobRequestSigner github jclouds/jclouds
HttpRequest request = context.getSigner(). signGetBlob(“adriansmovies”, "sushi.avi");
(let [request (sign-blob-request "adriansmovies" "sushi.avi" {:method :get})])
30
Tuesday, November 1, 11
What’s next?
• ACL support• OutputStream returns• Public/Private pathing
31
Tuesday, November 1, 11
32
github jclouds-examples
Questions?
Tuesday, November 1, 11