beolink - roma tre universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · amazon s3 beolink.org...
TRANSCRIPT
![Page 1: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/1.jpg)
Beolink.org Fabrizio Manfredi Furuholmen
Claudio Bisegni
S3->openAFS
![Page 2: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/2.jpg)
Beolink.org Amazon S3
“The new virtual storage service”
Huge hashtable Service works over http and supports storage of objects
up 5 gb in size You have to have an Amazon Web Services account Minimalistic API, available in both SOAP and REST
10/2/09
![Page 3: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/3.jpg)
Beolink.org Bucket
![Page 4: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/4.jpg)
Beolink.org openAFS
S3 openAFS Bucket Volume key directory value /dir/value meta /dir/meta acl Acl
![Page 5: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/5.jpg)
Beolink.org S3AFS implementation
CREATE BUCKET curl --request PUT http://localhost:8080/S3AFS/myBucket
STORE OBJECT curl --data "@foo.txt" --request PUT --header "Content-Type: text/html" http://localhost:8080/S3AFS/myBucket/foo.txt
READ OBJECT http://localhost:8080/S3AFS/myBucket/foo.txt
DELETE OBJECT curl --request DELETE http://localhost:8080/S3AFS/myBucket/foo.txt
DELETE BUCKET curl --request DELETE "http://localhost:8080/S3AFS/myBucket"
![Page 6: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/6.jpg)
Beolink.org S3 Client
From Kenneth Berland Script sub getAWSAuthorization { my ($HTTPVerb,$bucket)=@_; my $template="%a, %d %b %Y %T %z"; my $Date = time2str($template, time); my $aws_access_key_id = "your id here"; my $aws_secret_access_key = "your key here"; my $host = "s3.amazonaws.com"; my $ContentMD5=""; my $CanonicalizedResource; $CanonicalizedResource="/"; $CanonicalizedResource.=$bucket if $opt{b}; $CanonicalizedResource.=$key if $opt{k}; my $stringToSign; $stringToSign =$HTTPVerb."\n".$ContentMD5."\n"; $stringToSign.=$ContentType if $ContentType; $stringToSign.="\n"; $stringToSign.=$Date."\n"; $stringToSign.=$CanonicalizedAmzHeaders if $CanonicalizedAmzHeaders; $stringToSign.=$CanonicalizedResource;
print "StringToSign:".$stringToSign."\n" if $opt{d};
my $hmac = Digest::HMAC_SHA1->new($aws_secret_access_key); $hmac->add($stringToSign); my $signature = $hmac->b64digest; my $authorization = "AWS"." ".$aws_access_key_id.":".$signature."="; ($authorization,$Date); }
sub doAWSRequest { my ($authorization,$Date,$HTTPVerb,$bucket)=@_; my $ua=LWP::UserAgent->new; my $url="http://s3.amazonaws.com/"; $url.=$bucket if $bucket; $url.=$key if $key; print STDERR "URL: $url\n" if $opt{d}; my $req = HTTP::Request->new($HTTPVerb=>$url); $req->header( "Date" => $Date ); $req->header( "Authorization" => $authorization); $req->header( "Content-Type" => $ContentType ); $req->header( "x-amz-acl" => "public-read") if $opt{p}; if ($HTTPVerb eq "PUT" && $opt{f} && -e $opt{f}){ my $data=read_file($opt{f}); $req->content($data); }
print STDERR "request->as_string: ".$req->as_string if ($opt{d});
my $res=$ua->request($req); if ($res->is_success) { print STDERR localtime()." SUCCESS:".$res->status_line."\n" if ($opt{d}); }else{ warn localtime()." FAILED:".$res->status_line."\n"; print STDERR $res->content."\n"; } $res;
}
![Page 7: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/7.jpg)
Beolink.org openAFS
litteS3/ ?
JAFS/? (Alberto Mancini Patch)
openAFS
client REST Open points
Which technology ?
Security Authentication Cache
Mount Point
Directly from FS ?
![Page 8: Beolink - Roma Tre Universityafscon09/docs/s3-afs.pdf · 2009. 10. 2. · Amazon S3 Beolink.org “The new virtual storage service” Huge hashtable Service works over http and supports](https://reader033.vdocuments.us/reader033/viewer/2022052808/6073c29260ce083e66719184/html5/thumbnails/8.jpg)
Beolink.org Why S3->openAFS ?
Stateless
Many clients available
Work Over HTTP (Proxy)
Scalable
Interoperability with S3
Simple usage