Windows Azure Storage

NameTitleMicrosoft Corporation

Agenda

Windows Azure Storage

Blob Storage

Drives

Tables

Queues

Windows Azure Storage

Storage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet

Windows Azure Storage AccountUser specified globally unique account name

North Central US

South Central US

Northern Europe

Western Europe East Asia

South East Asia

US Europe Asia

Can choose geo-location to host storage account:

Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 24 global CDN nodes

Can co-locate storage account with compute accountExplicitly or using affinity groups

Accounts have two independent 512 bit shared secret keys

100 TBs per account

New Features

Geo-ReplicationStorage AnalyticsLogs: Provide trace of executed requests for your storage accountsMetrics: Provide summary of key capacity and request statistics for Blobs, Tables, and Queues

Improved HTTP headers for Blobs

Storage in the Development FabricProvides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or above

http://msdn.microsoft.com/en-us/gg433135

The Storage Client API

In this presentation we’ll cover the underlying

RESTful APICan call these from any HTTP cliente.g. Flash, Silverlight, etc…

Client API from SDK Microsoft.WindowsAzure.StorageClientProvides a strongly typed wrapper around REST services

Storage Libraries in Many Languages

Storage Security

Windows Azure Storage provides simple security for calls to storage serviceHTTPS endpointDigitally sign requests for privileged operations

Two 512bit symmetric keys per storage accountCan be regenerated independently

More granular security via Shared Access Signatures

Windows Azure Storage Abstractions

TablesStructured storage. A table is a set of entities; an entity is

a set of properties.

QueuesReliable storage and delivery of messages for an application.

BlobsSimple named files along with metadata for the file.

DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.

Blob Storage

Blob Storage Concepts

BlobContainerAccount

http://<account>.blob.core.windows.net/<container>/<blobname>

Pages/ Blocks

contoso

PIC01.JPG

Block/Page

Block/Page

PIC02.JPG

images

VID1.AVIvideos

Blob Details

Main Web Service

Operations

PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob

Blob Details

Associate Metadata with Blob

Standard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)

Metadata is <name, value> pairs, up to 8KB per blob

Either as part of PutBlob or independently

Blob Details

Blob always accessed by

name

Can include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg

Blob ContainersMultiple Containers per AccountSpecial $root container

Blob ContainerA container holds a set of blobsSet access policies at the container level Associate Metadata with ContainerList the blobs in a containerIncluding Blob Metadata and MD5 NO search/query. i.e. no WHERE MetadataValue = ?

Blobs ThroughputEffectively in Partition of 1Target of 60MB/s per Blob

GET http://.../products?comp=list&prefix=Tents&delimiter=/

<Blob>Tents/PalaceTent.wmv</Blob><Blob>Tents/ShedTent.wmv</Blob>

Enumerating Blobs

GET Blob operation takes parametersPrefixDelimiterInclude= (snapshots, metadata etc…)

http://adventureworks.blob.core.windows.net/ Products/Bikes/SuperDuperCycle.jpg Products/Bikes/FastBike.jpg Products/Canoes/Whitewater.jpg Products/Canoes/Flatwater.jpg Products/Canoes/Hybrid.jpg Products/Tents/PalaceTent.jpg Products/Tents/ShedTent.jpg

http://.../products?comp=list&prefix=Canoes&maxresults=2

&marker=MarkerValue

<Blob>Canoes/Hybrid.jpg</Blob>

Pagination

Large lists of Blobs can

be paginatedEither set maxresults or;Exceed default value for maxresults (5000)

http://.../products?comp=list&prefix=Canoes&maxresults=2

<Blob>Canoes/Whitewater.jpg</Blob><Blob>Canoes/Flatwater.jpg</Blob><NextMarker>MarkerValue</NextMarker>

Tour of the Blob Service

demo

Two Types of Blobs Under the Hood

Block BlobTargeted at streaming workloads

Each blob consists of a sequence of blocksEach block is identified by a Block ID

Size limit 200GB per blob

Optimistic Concurrency via Etags

Page BlobTargeted at random read/write workloads

Each blob consists of an array of pages Each page is identified by its offset from the start of the blob

Size limit 1TB per blob

Optimistic or Pessimistic (locking) concurrency via leases

TheBlob.wmv

Uploading a Block Blob

Uploading a large blob

10 GB Movie

Blo

ck I

d 1

Blo

ck I

d 2

Blo

ck I

d 3

Blo

ck I

d N

blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,

blockId1,…,blockIdN);

TheBlob.wmv

BenefitEfficient continuation and retryParallel and out of order upload of blocks

THE BLOB

Windows AzureStorage

Page Blob – Random Read/Write

Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them

Fixed Page Size = 512 bytesRandom Access Operations

PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)

GetPageRange[0, 4096) returns valid data ranges:

[0,512) , [1536,2560)GetBlob[1000, 2048) returns

All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)

0

10 GB

512

1024

1536

2048

2560

10 G

B A

dd

ress S

pace

Shared Access Signatures

Fine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocationUse short time periods and re-issueUse container level policy that can be deleted

Two broad approachesAd-hocPolicy based

Ad Hoc SignaturesCreate Short Dated Shared Access SignatureSignedresource Blob or ContainerAccessPolicy Start, Expiry and PermissionsSignature HMAC-SHA256 of above fields

Use caseSingle use URLsE.g. Provide URL to Silverlight client to upload to container

http://...blob.../pics/image.jpg?sr=c&st=2009-02-09T08:20Z&se=2009-02-10T08:30Z&sp=w

&sig= dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d

Create Container Level Policy Specify StartTime, ExpiryTime, Permissions

Create Shared Access Signature URLSignedresource Blob or ContainerSignedidentifier Optional pointer to container policySignature HMAC-SHA256 of above fields

Use caseProviding revocable permissions to certain users/groupsTo revoke: Delete or update container policy

http://...blob.../pics/image.jpg?sr=c&si=MyUploadPolicyForUserID12345

&sig=dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d

Policy Based Signatures

Content Delivery Network (CDN)

High-bandwidth global blob content delivery24 locations globally (US, Europe, Asia, Australia and South America), and growing

Same experience for users no matter how far they are from the geo-location where the storage account is hosted

Blob service URL vs. CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/

Windows Azure CDN URL: http://<id>.vo.msecnd.net/

Custom Domain Name for CDN: http://cdn.contoso.com/

pic1.jpg

Windows Azure CDN

To Enable CDN:Register for CDN via Dev PortalSet container images to public

pic1.jpg

GEThttp://guid01.vo.msecnd.net/images/pic.1jpg

http://sally.blob.core.windows.net/images/pic1.jpg

http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/

pic1.jpg

404

TTL Content Delivery Network

Windows Azure Blob Service

EdgeLocation

EdgeLocation

EdgeLocation

Drives

Windows Azure Drives

Durable NTFS volume for Windows Azure InstancesUse existing NTFS APIs to access a network attached durable driveUse System.IO from .NET

BenefitsMove existing apps using NTFS more easily to the cloudDurability and survival of data on instance recycle

A Windows Azure Drive is an NTFS VHD Page BlobMounts Page Blob over the network as an NTFS driveLocal cache on instance for read operationsAll flushed and unbuffered writes to drive are made durable to the Page Blob

Windows Azure Drive Capabilities

A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)Drives can be up to 1TB

A Page Blob can be mounted:On one instance at a time for read/writeUsing read-only snapshots to multiple instances at once

Windows Azure Drive Capabilities

An instance can dynamically mount up

to 16 drivesRemote Access via standard BlobUICan’t remotely mount driveCan upload the VHD to a Page Blob using the blob interface, and then mount it as a DriveCan download the VHD to a local file and mount locally

Drive Details

Operations performed via Drive API not REST CallsOperations on DrivesCreateDriveCreates a new NTFS formatted VHD in Blob storage

MountDrive/UnmountDriveMounts a drive into Instance at new drive letter

Unmounts a drive freeing drive letter

Get Mounted DrivesList mounted drives; underlying blob and drive letter

Snapshot DriveCreate snapshot copy of the drive

VM

How Windows Azure Drives Works Drive is a formatted page blob stored in blob

service

Mount obtains a blob lease

Mount specifies amount of local storage for cache

NTFS flushed/unbuffered writes commit to blob store before returning to app

NTFS reads can be served from local cache or from blob store (cache miss)

DemoBlob

OS

Application

Drive X:

Windows Azure Blob Service

Local Cache

Cloud Drive Client Library Sample

CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");

//Initialize the local cache for drives mounted by this role instanceCloudDrive.InitializeCache(localCacheDir, cacheSizeInMB);

//Create a cloud drive (PageBlob)CloudDrive drive = account.CreateCloudDrive(pageBlobUri);drive.Create(1000 /* sizeInMB */);

//Mount the network attached drive on the local file systemstring pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None);

//Use NTFS APIs to Read/Write files to drive

…

//Snapshot drive while mounted to create backupsUri snapshotUri = drive.Snapshot();

//Unmount the drivedrive.Unmount();

Failover with Drives

Must issue NTFS Flush command

to persist dataUse System.IO.Stream.Flush()

Read/Write Drives protected with leases1 Minute lease expiryMaintained by Windows Azure OS DriverUnmount on RoleEntryPoint.OnStop

On failureLease will timeout after 1 minuteRe-mount drive on new instance

Tables

Table Storage Concepts

EntityTableAccount

contoso

Name =…Email = …

Name =…EMailAdd=

customers

Photo ID =…Date =…

photos

Photo ID =…Date =…

Table Details

InsertUpdate Merge – Partial update

Replace – Update entire entity

UpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction

Create, Query, DeleteTables can have metadata

Not an RDBMS! Table

Entities

Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order

Timestamp Optimistic ConcurrencyExposed as an HTTP Etag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

No Fixed Schema

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

FAV SPORT

Canoeing

Querying

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

?$filter=Last eq ‘Wegner’

Purpose of the PartitionKeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

BikesQuick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Rafts 14ft Super Tourer … 1999

… … … …

SkisFabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

BikesQuick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

Rafts 14ft Super Tourer … 1999

… … … …

SkisFabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

Partitions and Partition Ranges

Server ATable = Products

Server BTable = Products

[Canoes - MaxKey)

Server ATable = Products

[MinKey - Canoes)

Queues

Queue Storage Concepts

MessageTableAccount

customer ID order ID http://…

adventureworks

order processing

customer ID order ID http://…

Loosely Coupled Workflow with QueuesEnables workflow between rolesLoad work in a queueProducer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps) Use multiple queuesRead messages in batchesMultiple work items per message

Queue

Input Queue (Work Items)

Web Role

Web Role

Web Role

Worker Role

Worker Role

Worker Role

Worker Role

Queue Details

Simple asynchronous dispatch queueNo limit to queue length subject to storage limit64kb per messageListQueues - List queues in account

Queue operations CreateQueue DeleteQueueGet/Set MetadataClear Messages

Queue Details

Message operationsPutMessage– Reads message and hides for time periodGetMessages – Reads one or more messages and hides themPeekMessages – Reads one or more messages w/o hiding themDeleteMessage – Permanently deletes messages from queueUpdateMessage – Clients renew the lease and contents

Queue’s Reliable Delivery

Guarantee delivery/processing of messages (two-step consumption)Worker queues message and it is marked as Invisible for a specified “Invisibility Time”Worker Deletes message when finished processingIf Worker role crashes, message becomes visible for another Worker to process

Windows Azure Storage SummaryFundamental data abstractions to build your applicationsBlobs: Files and large objectsDrives: NTFS APIs for migrating applicationsTables: Massively scalable structured storageQueues: Reliable delivery of messages

Easy to use via the Storage Client Library

Hands on Labs

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.