data overview - wikiazure · queues durable messaging. provides asynchronous communication between...

Data Overview

Dave Rendón@DaveRndn

Agenda

Blobs

Files

Queues

Tables

StorSimple

Azure Storage Architecture

Azure Storage Concepts

(Blobs) Container

Table

File shares

Queues

Blobs

Entities

Directories/Files

Messages

Account

West Europe

East US 2West US

Southeast Asia

Japan East

East China

Australia East

Japan West

Premium storage availability

Azure Storage Accounts

An Azure Subscription can have up to 100 storage accounts.

Soft limit – contact support if you need more

Store up to 500TB per storage account

A storage account is uniquely addressable.

https://<account name>.<service name>.core.windows.net

Available from anywhere using REST API’s

Open source client libraries available for .NET, Native C++, Java, Android, Node.js, PHP, Python, Ruby, PowerShell, iOS

Azure Storage Replication

Locally Redundant Storage (LRS)

Replicates your data within the region in which you created your storage account

Zone-redundant storage (ZRS)

Only for block blobs

Replicates your data across two to three facilities, either within a single region or across two regions

Geo-redundant storage (GRS) Replicates your data to a secondary

region that is hundreds of miles away from the primary region

Read-Access geo-redundant storage (RA-GRS) Maximizes availability for your storage

account Provides read-only access to the data

in the secondary location, in addition to the replication across two regions provided by GRS

High Performance Storage for VM Workloads

Ideal for I/O intensive workloads running on DS-Series or GS-Series Virtual Machines

DS-Series VM can support up to 32 data disks and deliver up to 50K IOPs

GS-Series VM can support up to 64 data disks and deliver up to 80K IOPs

MICROSOFT CONFIDENTIAL

File SharesSupports SMB 3.0 protocol.

Can be accessed like a traditional file share.

Share files between multiple Virtual Machines.

A single file share can be up to 5TB.

TablesNoSQL storage of structured data (entities).

Key/value storage.

A single entity can have up to 255 properties and be up to 1MB.

QueuesDurable messaging.

Provides asynchronous communication between application tiers and components.

A single message can be up to 64KB.

BlobsBlock: Text or binary data (.log, .exe, .jpg, etc.).

Up to 200GB.

Page: Optimized for disks (.vhd). Supports random read-write. Up to 1TB.

Append Blob: Writes to end of the blob (4MB max) up to 50k times (~195GB)

Engineer to Engineer

Azure Storage Architecture

Azure Storage Architecture (3 tiers)

FE 2

Partition 3

(F-J)

Stream 2

Azure Storage Partitions A partition is served by a partition server

The scale of a partition is limited to what the partition server can handle

Data in same partition can be updated in an atomic manner. For example, Update a list of blocks in a blob and commit together. Add/Update/Delete entities in a table as part of a transaction. Write a batch of messages to a queue.

Data in separate partitions can be load balanced across partition servers.

http|https://storageaccountname.blob.core.windows.net/

Blob Storage Container Concepts

Container 1

(/VHDs)

VHDs

• /Win2k12-

datacenter.vhd

Container 2

(/Logs)

/Log1.log

Container 3

(/Public)

/Photos

• /Pic1.jpg

/Videos

• /MyVideo.mp4

http|https://storageaccountname.queue.core.windows.net/QUEUENAME/

Queue Storage Concepts

Queue (Order Processing)

Order1 Order2 OrderN…

http|https://storageaccountname.files.core.windows.net/

Files Storage Concepts

Share name 1 (archive)

/Folder

• /File1.docx

• /foldera

• /file2.pptx

/folder

• /folder

• Files

Share name 1 (S Drive)

/Folder

• /File1.docx

• /foldera

• /file2.pptx

/folder

• /folder

• files

http|https://storageaccountname.table.core.windows.net/TABLENAME/

lastname2

Table Storage Concepts

Entity1

• Partition Key=LASTNAME1

• Key=userid

• Att1 = value

Entity2

• Partition Key=LASTNAME1

• Key=userid

• Att1 = value

• Att 2 = value

• Att 3 =value

Entity3

• Partition Key=LASTNAME2

• Key=userid

• Att1 = value

• Att 3 = value

• Att 6 = value

Blobs

Microsoft Azure

Storage Blob

Block Blob Page Blob

Two Types of Blobs Under the Hood

Block Blob

Targeted at streaming workloads

Each blob consists of a sequence of blocks

Each block is identified by a Block ID

Size limit 200GB per blob

Optimistic Concurrency via Etags

Page Blob

Targeted at random read/write workloads

Each blob consists of an array of pages

Each page is identified by its offset from the start of the blob

Size limit 1TB per blob

Optimistic or Pessimistic (locking) concurrency via leases

Blob Storage Concepts

Blob Details – Containers

A container holds a set of blobs

Set access policies at the container level

Associate Metadata with Container

List the blobs in a container

Including Blob Metadata and MD5 no search on metadata WHERE MetadataValue = ?

Can include ‘/‘ or other delimiter in name

e.g. /<container>/myblobs/smurf.png

Blob Details –

blob name

Blob Details

GET Blob operation takes parameters

Prefix

Delimiter

Include = (snapshots, metadata etc…)

Blob sample listing

Uploading a Block Blob

Uploading

Shared Access Signatures

Fine grain access rights to blobs and containers

Sign URL with storage key – permit elevated rights

Ad-hoc: Policy based:

Stored Access Policy Shared Access Signature

Shared Access Signatures – Two broad approaches

Shared Access Signatures – Revocation

Use short time periods and re-issue

Use container level policy that can be deleted

Store Access Policy – Policy Based Signatures

Create Shared Access Signature URL

Signed resource Blob or Container

Signed identifier Optional pointer to container policy

Signature HMAC-SHA256 of above fields

Store Access Policy – Policy Based Signatures

Use case

Providing revocable permissions to certain users/groups

To revoke: Delete or update container policy

http://...blob.../pics/image.jpg?

sr=c&si=MyUploadPolicyForUserID12345

&sig=dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d

Store Access PolicyPolicy Based Signatures

Files

Microsoft Azure

Storage Files

Azure Files – Customer Quotes

“I wish I could go to storage and provision a cloud drive, giving it a namespace, and that drive would then be UNC-addressable by the OSes.”

“I need two VM's running with a shared drive. One will write to the drive, the other will read [it].”

“Hi, I have two VM's in Microsoft Azure. All I want to do is set up a file share between them. Is this possible?”

“Is it possible to share a secondary disk between different VM instances?”

Sharing Files –The old way

Setup an IaaS VM to host a File Share backed by an IaaS Disk

Write code to find the IaaS File Share from the rest of the VMs in

your service.

Write some code to provide high availability

Handle host upgrades, node failures

You can only access the File Share from other VMs

Azure Files

Shared Network File Storage for Azure

Availability, durability, scalability are managed automatically

Supports two interfaces: SMB and REST

Azure Files – Usage

Share data across VMs and applications

Share settings throughout services

Dev/Test/Debug

Queues

Microsoft Azure

Storage Queue

Why use a Queue?

Queue length reflects how well the backend processing nodes are

doing.

Decouples the application.

Flexibility of efficient resource usage within an application.

Absorb traffic bursts and reduce the impact of individual

component failures.

Queue Components

Storage Account: All access to Azure Storage is done through a storage account.

Queue: A queue contains a set of messages.

Message: A message, in any format, of up to 64KB.

Queue URL format

http://{storage-account}.queue.core.windows.net/{queue}

Queues are addressable using the following URL format:

Queue URL format

http://myaccount.queue.core.windows.net/imagesToDownload

Example:

Demo: Faster Web Applications with queues using

asynchronous workloads

Producers Consumers

Queue C1

C2

P1

P2

4 3 12

Queue-based Load Levelling Pattern

Tables

Microsoft Azure

Storage Table

Table Storage Concepts

Table Storage Details

Not an RDBMS Table!

The mental picture is ‘Entities’

Table Storage Details

Entity can have up to 255 properties

Up to 1MB per entity

Table Storage Details

Entity Properties

Table Storage Details

Supports full manipulation (CRUD)

Including Upsert and Entity Group Transactions

Tables can have metadata

StorSimple

Use separate storage accounts for high performance workloads

Use separate storage accounts for logging

Use queues to improve application scale