Windows Azure Storage

NameTitleMicrosoft Corporation

Windows Azure Storage

Storage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet

Various storage abstractionsTables, Blobs, Queues, Drives

Windows Azure Storage Account

User specified globally unique account name

Can choose geo-location to host storage accountUS – “North Central” and “South Central”Europe – “North” and “West”Asia – “East” and “Southeast”

Can CDN Enable AccountBlobs delivered via 18 global CDN nodes

Can co-locate storage account with compute account

Explicitly or using affinity groups

Accounts have two independent 512 bit shared secret keys100TB per account

Storage in the Development Fabric

Provides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or above

There are some differences between Cloud and Dev Storage.http://msdn.microsoft.com/dd320275

A good approach for developers:To test pre-deployment, push storage to the cloud first

Use Dev Fabric for compute connect to cloud hosted storage.Finally, move compute to the cloud.

The Storage Client API

In this presentation we’ll cover the underlying RESTful API

Can call these from any HTTP cliente.g. Flash, Silverlight, etc…

Client API from SDK Microsoft.WindowsAzure.StorageClient

Provides a strongly typed wrapper around REST services

Storage Security

Windows Azure Storage provides simple security for calls to storage service

HTTPS endpointDigitally sign requests for privileged operations

Two 512bit symmetric keys per storage account

Can be regenerated independentlyMore granular security via Shared Access Signatures

Windows Azure Storage Abstractions

Blobs – Simple named files along with metadata for the fileDrives – Durable NTFS volumes for Windows Azure applications to use. Based on Blobs.Tables – Structured storage. A Table is a set of entities; an entity is a set of propertiesQueues – Reliable storage and delivery of messages for an application

Blob Storage

Blob Storage Concepts

BlobContainerAccount

contoso

images

PIC01.JPG

videos VID1.AVI

http://<account>.blob.core.windows.net/<container>/<blobname>

Pages/Blocks

Block/Page

Block/Page

PIC02.JPG

Blob Details

Main Web Service OperationsPutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob

Associate Metadata with BlobStandard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)Metadata is <name, value> pairs, up to 8KB per blobEither as part of PutBlob or independently

Blob always accessed by nameCan include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg

Blob Containers

Multiple Containers per AccountSpecial $root container

Blob ContainerA container holds a set of blobsSet access policies at the container level Associate Metadata with ContainerList the blobs in a container

Including Blob Metadata and MD5 NO search/query. i.e. no WHERE MetadataValue = ?

Blobs ThroughputEffectively in Partition of 1Target of 60MB/s per Blob

GET http://.../products?comp=list&delimiter=/

<BlobPrefix>Bikes</BlobPrefix><BlobPrefix>Canoes</BlobPrefix><BlobPrefix>Tents</BlobPrefix>

Enumerating Blobs

GET Blob operation takes parametersPrefixDelimiterInclude= (snapshots, metadata etc…)

GET http://.../products?comp=list&prefix=Tents&delimiter=/

<Blob>Tents/PalaceTent.wmv</Blob><Blob>Tents/ShedTent.wmv</Blob>

http://adventureworks.blob.core.windows.net/Products/Bikes/SuperDuperCycle.jpgProducts/Bikes/FastBike.jpgProducts/Canoes/Whitewater.jpgProducts/Canoes/Flatwater.jpgProducts/Canoes/Hybrid.jpgProducts/Tents/PalaceTent.jpgProducts/Tents/ShedTent.jpg

Pagination

Large lists of Blobs can be paginatedEither set maxresults or;Exceed default value for maxresults (5000)

http://.../products?comp=list&prefix=Canoes&maxresults=2&marker=MarkerValue

<Blob>Canoes/Hybrid.jpg</Blob>

http://.../products?comp=list&prefix=Canoes&maxresults=2

<Blob>Canoes/Whitewater.jpg</Blob><Blob>Canoes/Flatwater.jpg</Blob><NextMarker>MarkerValue</NextMarker>

Tour of the Blob ServiceNameTitleGroup

demo

Two Types of Blobs Under the Hood

Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks

Each block is identified by a Block ID

Size limit 200GB per blobOptimistic Concurrency via ETags

Page BlobTargeted at random read/write workloadsEach blob consists of an array of pages

Each page is identified by its offset from the start of the blob

Size limit 1TB per blobOptimistic or Pessimistic (locking) concurrency via Leases

Uploading a Block Blob

Uploading a large blob

10 GB Movie

Windows Azure Storage

Blo

ck I

d 1

Blo

ck I

d 2

Blo

ck I

d 3

Blo

ck I

d N

blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,

blockId1,…,blockIdN);

TheBlob.wmv

TheBlob.wmv

BenefitEfficient continuation and retryParallel and out of order upload of blocks

Page Blob – Random Read/Write Create MyBlob

Specify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them

Fixed Page Size = 512 bytes

Random Access OperationsPutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)

GetPageRange[0, 4096) returns valid data ranges:

[0,512) , [1536,2560)

GetBlob[1000, 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)

0

10 GB

10 G

B A

dd

ress S

pace

512

1024

1536

2048

2560

Shared Access Signatures

Fine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocation

Use short time periods and re-issueUse container level policy that can be deleted

Two broad approachesAd-hocPolicy based

Ad Hoc Signatures

Create Short Dated Shared Access Signature

Signedresource Blob or ContainerAccessPolicy Start, Expiry and PermissionsSignature HMAC-SHA256 of above fields

Use caseSingle use URLsE.g. Provide URL to Silverlight client to upload to container

http://...blob.../pics/image.jpg?sr=c&st=2009-02-09T08:20Z&se=2009-02-10T08:30Z&sp=w

&sig= dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d

Policy Based Signatures

Create Container Level Policy Specify StartTime, ExpiryTime, Permissions

Create Shared Access Signature URLSignedresource Blob or ContainerSignedidentifier Optional pointer to container policySignature HMAC-SHA256 of above fields

Use caseProviding revocable permissions to certain users/groupsTo revoke: Delete or update container policy

http://...blob.../pics/image.jpg?sr=c&si=MyUploadPolicyForUserID12345

&sig=dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d

Content Delivery Network (CDN)

ScenarioFrequently accessed blobsAccessed from around the world

Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery

20 locations globally (US, Europe, Asia, Australia and South America), and growingSame experience for users no matter how far they are from the geo-location where the storage account is hosted

Blob service URL vs CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/Windows Azure CDN URL: http://<id>.vo.msecnd.net/ Custom Domain Name for CDN: http://cdn.contoso.com/

CostUS located CDN nodes 15c/GB + 1c/10,000 txnRest of World 20c/GB + 1c/10,000 txnTraffic from Storage node to edge node at standard rates

Windows Azure Content Delivery Network

To Enable CDN: Register for CDN via Dev Portal Set container images to public

Windows Azure Blob Service

pic1.jpg

Content Delivery Network

Edge Location

Edge Location Edge Location

GEThttp://guid01.vo.msecnd.net/images/pic.1jpg

http://sally.blob.core.windows.net/images/pic1.jpg

http://sally.blob.core.windows.net/

http://guid01.vo.msecnd.net/

pic1.jpg

404

pic1.jpg

TTL

Drives

Windows Azure Drives

Durable NTFS volume for Windows Azure Instances

Use existing NTFS APIs to access a network attached durable driveUse System.IO from .NET

BenefitsMove existing apps using NTFS more easily to the cloudDurability and survival of data on instance recycle

A Windows Azure Drive is a NTFS VHD Page Blob

Mounts Page Blob over the network as an NTFS driveLocal cache on instance for read operationsAll flushed and unbuffered writes to drive are made durable to the Page Blob

Windows Azure Drive Capabilities

A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)

Drives can be up to 1TB

A Page Blob can be mounted:On one instance at a time for read/writeUsing read-only snapshots to multiple instances at once

An instance can dynamically mount up to 16 drivesRemote Access via standard BlobUI

Can’t remotely mount drive. Can upload the VHD to a Page Blob using the blob interface, and then mount it as a DriveCan download the VHD to a local file and mount locally

Drive Details

Operations performed via Drive API not REST CallsOperations on Drives

CreateDriveCreates a new NTFS formatted VHD in Blob storage

MountDrive/UnmountDriveMounts a drive into Instance at new drive letterUnmounts a drive freeing drive letter

Get Mounted DrivesList mounted drives; underlying blob and drive letter

Snapshot DriveCreate snapshot copy of the drive

VM

How Windows Azure Drives Works

Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushed/unbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)

Windows Azure Blob Service

DemoBlob

Local Cache

OS

Application

Lease

Drive X:

Cloud Drive Client Library Sample

CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");

//Initialize the local cache for drives mounted by this role instanceCloudDrive.InitializeCache(localCacheDir, cacheSizeInMB);

//Create a cloud drive (PageBlob)CloudDrive drive = account.CreateCloudDrive(pageBlobUri);drive.Create(1000 /* sizeInMB */);

//Mount the network attached drive on the local file systemstring pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None);

//Use NTFS APIs to Read/Write files to drive…

//Snapshot drive while mounted to create backupsUri snapshotUri = drive.Snapshot();

//Unmount the drivedrive.Unmount();

Failover with Drives

Must issue NTFS Flush command to persist data

Use System.IO.Stream.Flush()Read/Write Drives protected with leases

1 Minute lease expiryMaintained by Windows Azure OS DriverUnmount on RoleEntryPoint.OnStop

On failureLease will timeout after 1 minuteRe-mount drive on new instance

Queues

Queue Storage Concepts

MessageQueueAccoun

t

order processing

customer ID order ID http://…

customer ID order ID http://…

adventureworks

Loosely Coupled Workflow with Queues

Enables workflow between rolesLoad work in a queue

Producer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps)

Use multiple queuesRead messages in batchesMultiple work items per message

Queue

Input Queue (Work Items)

Web Role

Web Role

Web Role

Worker Role

Worker Role

Worker Role

Worker Role

Queue Details

Simple asynchronous dispatch queueNo limit to queue length subject to storage limit8kb per messageListQueues - List queues in account

Queue operations CreateQueue DeleteQueueGet/Set MetadataClear Messages

Message operationsPutMessage– Reads message and hides for time periodGetMessages – Reads one or more messages and hides themPeekMessages – Reads one or more messages w/o hiding themDeleteMessage – Permanently deletes messages from queue

Queue’s Reliable Delivery

Guarantee delivery/processing of messages (two-step consumption)

Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time”Worker Deletes message when finished processingIf Worker role crashes, message becomes visible for another Worker to process

More on this pattern in the Async Workloads session

Queue

Input Queue (Work Items)

Web Role

Web Role

Web Role

Worker Role

Worker Role

Worker Role

Worker Role

Tables

Table Storage Concepts

EntityTableAccount

contoso

customers

Name =…Email = …

Name =…EMailAdd= …

photos

Photo ID =…Date =…

Photo ID =…Date =…

Table Details

Not an RDBMS! More on table modeling in Storage Strategies sessionTable

Create, Query, DeleteTables can have metadata

EntitiesInsertUpdate

Merge – Partial updateReplace – Update entire entity

DeleteQueryEntity Group Transactions

Multiple CUD Operations in a single atomic transaction

Entity Properties

Entity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency. Exposed as an HTTP ETag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

First Last Birthdate

Kim Akers 2/2/1981

Nancy Anderson

3/15/1965

Mark Hassall May 1, 1976

Fav Sport

Canoeing

No Fixed Schema

First Last Birthdate

Kim Akers 2/2/1981

Nancy Anderson

3/15/1965

Mark Hassall May 1, 1976

Querying

?$filter=Last eq ‘Akers’

Purpose of the PartitionKey

Entity LocalityEntities in the same partition will be stored together

Efficient querying and cache localityEndeavour to include partition key in all queries

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/account

Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions

Each partition can be served by a different storage nodeScale to meet the traffic needs of your table

PartitionKey(Category)

RowKey(Title)

Timestamp ModelYear

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

PartitionKey(Category)

RowKey(Title)

Timestamp

ModelYear

Rafts 14ft Super Tourer … 1999

… … … …

Skis Fabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

PartitionKey(Category)

RowKey(Title)

Timestamp ModelYear

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

Rafts 14ft Super Tourer … 1999

… … … …

Skis Fabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

Partitions and Partition Ranges

Server BTable = Products[Canoes - MaxKey)

Server ATable = Products[MinKey - Canoes)

Server ATable = Products

Windows Azure Storage Summary

Fundamental data abstractions to build your applications

Blobs – Files and large objectsDrives – NTFS APIs for migrating applicationsTables – Massively scalable structured storageQueues – Reliable delivery of messages

Easy to use via the Storage Client LibraryHands on Labs

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.