using azure table storage in your startup

Jose Parrajparra@geeksco.com

@josedavidparra

Using Azure Table Storage in your Startup

A candidate online service must• Runs “almost” by itself• Could be operated from

anywhere• Moving bits instead of atoms• Cheap to run• Clear revenue model• Be able to build a quick MVP

Inspiration

Architecture

Web

UI

XXXXX

XXXXXXX

XXX

XXX XX

XXXXX

XXXXXXX

XXX

XXX XX

XXXXX

XXXXXXX

XXX

XXX XX

Admin

Editors

Biz

Log

ic

Dat

a A

cces

sA

dapt

ers

End Users

Azure Web Role

Portals

Azure Table Storage

Azure BLOBs

Documents

Static Content

Jobs

Logs

Accounts

Payment Gateway(PayPal)

SMTP Gateway

(SendGrid)

Customer Support(UsersVoice)

Our Manifesto• No external funding!• No need to quit our day jobs• No rush• Commitment/discipline to finish it, BUT• No fear of failure• No over-engineer it – not solving issues we do not have yet

• Every tech decision tries to implement a functionality/solve a problem we currently have

• Being open to learn new things (tech/non-tech)!

Technologies/Tools Involved• MVC 4 – Razor• The 3 portals in 1 Single Azure Web Role• Azure Tables and Blobs for Storage• Bootstrap for layout and Responsive UI• Source Code/Task Lists/Build on Azure TFS• GeoIP using MaxMind• AutoMapper (Model <-> DTOs)• SendGrid for SMTP• UserVoice for HelpDesk/Feedback• Glimpse and MiniProfiler for profiling• PayPal as payment gateway• Google Recaptcha

Why Azure?

• CHEAPER!• Reliable-enough• Integrated to VS and .NET• High performance• New datacenters nearby (AUS)• Catching up quickly!

BTW, What is the cloud?A pay-as-you-go approach to computing where you control the provisioning of computing resources

Applicationbuilding blocks

StorageBig data

Caching

CDN

Database

Identity

Media

Messaging

Networking

Traffic

Cloud service

s

Web sites

Virtual machine

sIaaS

PaaS

PaaS

Execution Models

TablesStructured storage. A table is a set of entities; an entity is

a set of properties.

QueuesReliable storage and delivery of messages for an application.

BlobsSimple named files along with metadata for the file.

DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.

Azure SQLRelational SQL Server engine. Clustered, fully managed. SQL Reporting Support

DB

How do you get charged?• Per Endpoint (PaaS)• Per the size of the instance• Per running instance• Per VM and its size (IaaS)• Per MB of storage• Per storage transaction• Per MB of transfer outbound (inboud:

free)• Pricing calculator:

http://www.windowsazure.com/en-us/pricing/calculator/

Why Azure Tables for Storage?• Cost!

• US$1.75/mo for 25GB vs US$76 with SQL Azure

• $0.01 for 100K transactions (N/A for SQL Azure)

• +80% of the data will be unstructured data (logs)!

• The option of massive-scalability• Performance

Windows Azure Storage AccountUser specified globally unique account name

Can choose geo-location to host storage account:

Azure Tables• Provides Structured Storage

• Massively Scalable Tables• Billions of entities (rows) and TBs of data• Can use thousands of servers as traffic

grows

• Highly Available & Durable

• Data is replicated several times (default)• 2x Same region (GRS)• 3x Same sub-region (LRS)

Scalability Targets for Storage Accounts• Capacity – Up to 200 TBs• Transactions – Up to 20,000

entities/messages/blobs per second• Bandwidth for a Geo

Redundant storage account• Ingress – up to 5 gigabits per second• Egress – up to 10 gigabits per second

• Bandwidth for a Locally Redundant storage account

• Ingress – up to 10 gigabits per second• Egress – up to 15 gigabits per second

Table Storage Concepts

EntityTableStorageAccount

contoso

Name =…Email = …

Name =…EMailAdd=

customers

Photo ID =…Date =…

photos

Photo ID =…Date =…

Entity Properties• Entity can have up to 255 properties

• Up to 1MB per entity• Mandatory Properties for every entity

• PartitionKey & RowKey (only indexed properties)• Uniquely identifies an entity• Defines the sort order

• RowKeys must be unique within the same partition

• Timestamp (Auto)• Optimistic Concurrency

• No fixed schema for other properties• Each property is stored as a <name, typed value> pair• No schema stored for a table

Purpose of the PartitionKey•Entity Locality

• Entities in the same partition will be stored together (Efficient querying and cache locality)

•Entity Group Transactions• Atomic multiple Insert/Update/Delete in

same partition in a single transaction•Table Scalability

• Target throughput – 500 tps/partition, several thousand tps/account

• Automatically load balance partitions• Each partition can be served by a different storage node• Scale to meet the traffic needs of your table

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Rafts 14ft Super Tourer … 1999

… … … …

Skis Fabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

Partitions and Partition Ranges

Server ATable = Products

Server BTable = Products

[Canoes - MaxKey)

Server ATable = Products

[MinKey - Canoes)

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

Rafts 14ft Super Tourer … 1999

… … … …

Skis Fabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

Data Types

No Fixed Schema

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

FAV SPORT

Canoeing

Demo Time!

The Storage Client API

• OData (Atom) protocol with WCF Data Services

• Microsoft.WindowsAzure.StorageClient

• REST• LINQ• PowerShell!

InsertUpdate Merge – Partial update

Replace – Update entire entity

UpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction

Create, Query, DeleteUnnormalized dataTable URL: http://<storage account>.table.core.windows.net/<table>

Not an RDBMS! Table

Entities

Single Entity Query

Server A

PartitionKey(Category)

RowKey (Title)

Action Fast & Furious

… 10000 more Action movies

Action The Bourne Ultimatum

… 100000 more Action & Animation movies

Animation Open Season 2

… 100000 more Animation movies

Animation The Ant Bully

Comedy Office Space

… 1000000 more Comedy & SciFi movies

SciFi Star Trek

… 100000 more SciFi & War movies

… 100000 more War movies

War Defiance

Client

Server B

> Where PartitionKey=‘SciFi’ and RowKey = ‘Star Trek’

> Efficient processing> No continuation tokens

28

Request

Result

Client

Server A

Server B

Table Scan Query

Request

PartitionKey(Category)

RowKey(Title)

Rating

Action Fast & Furious 5

… 999 more movies rated > 4

…

… Action and Anim. movies here with rating < 4

…

Animation A Bug’s life 2

… 100 more movies < 4 here

…

Animation The Ant Bully 3

Comedy Are we there yet? 2

… More movies here …

Comedy Office Space 5

… 800000 more movies here

…

Drama A Beautiful Mind 5

… 1200000 more movies here

…

War Defiance 4

Cont.

> Select * from Movies where Rating > 4> Returns Continuation token

> 1000 movies in result set> Partition range boundary

> Serial Processing: Wait for continuation token before proceeding

Request Cont.

Cont.

Request Cont.

Cont.

29

Returns 1000 movies

Partition range boundary hit

Return continuation

Client

Server A

Server B

Make Scans Faster

Request

PartitionKey(Category)

RowKey(Title)

Rating

Action Fast & Furious 5

… More movies here …

Comedy Office Space 5

… More movies here …

Documentary

Planet Earth 4

… More movies here

Drama Seven Pounds 4

Horror Saw 5 3

… More movies here …

Music 8 Mile 2

… More movies here …

SciFi Star Trek 5

… More movies here …

Cont.

> Split “Select * from Movies where Rating > 4” into> Where PartitionKey >= “A” and PartitionKey < “D” and Rating > 4> Where PartitionKey >= “D” and PartitionKey < “I” and Rating > 4> Etc.

> Execute in parallel> Each query handles continuation

Cont.

30

Request

Request

Cont.

Some Patterns..> Ex: Store Account Information and Rental details in

same table..> Maintain same PartitionKey to enforce transactions

> Account ID as PartitionKey> Update total count and Insert new rentals using Entity

Group Transaction> Prefix RowKey with “Kind” code: A = Account, R = Rental

> Row key for account info: [Kind Code]_[AccountId]> Row Key for rental info: [Kind Code]_[Title]

> Rental Properties not set for Account row and vice versaPartitionKey(AccountID)

RowKey(Kind_*)

Kind TotalRentals

Name Address CheckOutOn Title DueOn

… … … … … … … … …

Sally A_Sally Account

8 Sally Field

Ann Arbor, MI

Sally R_Jaws Rental 2009/11/16 Jaws 2009/11/20

Sally R_Taxi Rental 2009/11/16 Taxi 2009/11/20

… … … … … … … … …31

How to Connect to an Azure Table• Get the latest SDK (2.2)• Install the

“WindowsAzure.Storage” nuget• Get Azure Storage Account,

container and keys• Inherit class from TableEntity• Connect to table• Query and display

Demo Time!

Watch Out!

• A reduced set of supported Data types

• Initializing DateTime properties (set them as nullable)

• Retrieving error details• Issuing complex queries• Only supported operators:

• From, Where, Take, First, FirstOrDefault, Select

Demo Time!

Blobs

BlobContainer

Account

http://<account>.blob.core.windows.net/<container>/<blobname>

Pages/ Blocks

contoso

PIC01.JPG

Block/Page

Block/Page

PIC02.JPG

images

VID1.AVIvideos

Blob Storage Concepts

Demo Time!

Thanks!

Jose Parrajparra@geeksco.com

@josedavidparra

Links

• Manage.azure.com• http://www.windowsazure.com/

en-us/pricing/calculator/• http://azurestorageexplorer.cod

eplex.com/