deep thoughts from the real world of azure

Deep Thoughtsfrom the

Real World of Azure

Michele Leroux Bustamante

CIO, SollianceCofounder, Snapboard

michelebusta@solliance.net

Windows Azure FeaturesCompute

VirtualNetwork

TrafficManager

CDN

Storage

Queues

Storage

Tables

Storage

Blobs

SQL Database

SQL Data Sync

SQL Reporting

HDInsight (Hadoop)

Caching Recovery Services

Windows Azure Active Directory

Media Services

Service Bus

Store

Web Site Cloud Service

Virtual Machine

ApplicationServices

Marketplace

DataServices

Networking

MobileServices

BizTalkServices

Azure FeaturesCompute

Web Site Cloud Service

Virtual Machine

Your Datacenter

Virtualization

O/S

Hardware

Network

Data

Applications

Firewall

Web Sites

Applications

Data

Cloud Services

Applications

Firewall Rules

Data

Virtual Network

Virtual Machines

Virtual Network

Data

Applications

Firewall Rules

O/S

Focus on the Application

Microsoft Azure

IAASVIRTUAL MACHINES

Virtual Machine ProvisioningChoose a VM

Blob Storage

Generate

Provision

Select your VHD

Blob Storage

Upload

Provision

PAASCLOUD SERVICES & WEB SITES

WEB SITES

Deep Thoughts on Web Sites

• New portal • Publishing process• Setting overrides• Log streaming• SWAP (but…)

CLOUD SERVICES

Cloud Services Are…

Web Role Worker Role

Upload to Windows Azure

Windows Azure Compute Controller

LB

ServiceService

Packaging & DeploymentServiceArtifacts

Model

Config*.cscfg

*.cspkgMicrosoft

Visual StudioWeb Role

Worker Role

ROLE

VIP Swap

VM1 VM2

VM3 VM4

VM1 VM2

VM3 VM4

Deep Thoughts on Cloud Services

• Packaging settings outside of web.config

• Proper swap, separate VM• Application insights agent• Mostly for worker roles, reading

queues, async processing

Deep Thoughts on Worker Roles

• Set up configuration settings to shut off processing

• Implement graceful shutdown

Azure Features

Storage

Tables

SQL Database

SQL Data Sync

SQL Reporting

HDInsight (Hadoop)

DataServices

SQL DB and SQL Server

• SQL DB (formerly SQL Azure)– 150GB limit– No backup logs / replay– IO performance – Other SQL features

• SQL DB Premium– 500GB limit– 30 days of backups– Restore on demand

• SQL Server on Azure VM– Popular alternative– 1TB limit (blob storage)

What do YOU want in a DB?

• No limits manage yourself– SQL Server VM

• Managed and affordable– SQL DB

• Managed and $$$–MySQL / ClearDB cluster– SQL DB Premium– Other similar options for MySQL/SQL

Server outside of Azure

Deep Thoughts on Relational Data

• All platforms require retry mechanism

• Mind your connection pool settings• Practice backup and restore• Always use data masking to dev, test• PII, encryption• Review queries, profile, tune• Use transactions for Pete’s sake

noSQL

• It’s either obvious…– Very large quantities of data– Easily compartmentalized data, key value lookup or document-

centric– Logs, audit trails, notifications, profile data from social networks,

associative information about users / “things” in the relational store

• Or, it requires planning…– Could be large quantities– Could require noSQL scale, parallel access performance

• Many people return to relational• Can start relational and figure your model out first• Or, combine the two

Understanding noSQL

• http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

Big Data/Map Reduce from 10K Feet

Company B Company CCompany A

AggregateData

(all Companies)

HDInsight

Company A

Server

Company C

Company B

HDInsight(local)

HDInsight(local)

HDInsight(local)

Don’t Drink the KoolaidnoSQL

Hadoop

Big Data

Don’t Drink the Koolaid

• noSQL requires planning• BigData is not for everyone

• Get an expert, like: Lynn Langit– www.lynnlangit.com – Youtube.com/socaldevgal Shout Out!

Azure Features

Service Bus

ApplicationServices

Storage

Queues

DataServices

Service Bus as DMZ

Service Service Service

Web API Web API

Browser BrowserRich

ClientsMVC /JQuery AJAX

Service

MobileClients

DMZ

Corporate Domain

MVCSite

Web Forms

Site

Service Bus to Data On Premise / Migration

Service

Client

Web Application

Service Bus

Corporate Domain

Windows Azure

Queues

Queue

Queue

Topics

TopicSubSubSub

Queue ComparisonService Bus

QueuesStorage Queues

Unlimited message lifetime

7 days expiry

Max 64K message size Max 256K message size

Max 5GB total storage Max 100TB total storageDuplicate detection

Order guarantees

Dead letter queue

Storage metrics

Purge capability

Long polling Manual back-off polling

Queue ComparisonService Bus

QueuesStorage Queues

Unlimited message lifetime

7 days expiry

Max 64K message size Max 256K message size

Max 5GB total storage Max 100TB total storageDuplicate detection

Order guarantees

Dead letter queue

Storage metrics

Purge capability

Long polling Manual back-off polling

Queue ComparisonService Bus

QueuesStorage Queues

EASYCOMPLEX

Deep Thoughts on Async Work

• Queues and worker roles• Use error queues• Monitor queues for expiring items• Use service bus for parallel

processing• Use web jobs to kick off queue

processing

Azure Features

CDN

Storage

Blobs

Media Services

ApplicationServices

DataServices

Networking

Common Blob Storage Usage

• Web site content, javascript, css, images• Logs (setting)• App content

– Can separate by user for key isolation via Azure Explorer

• Media– Can use with media services, encoding services

• Replace file storage– Retain folder structure– Request by path

• Very large files– Set up chunking process for upload– Use hash to check integrity

Blob Storage Integrity

Windows Azure Storage

Blob Container

Service

validatesignature

WindowsAzure

MD5Hash +

MD5Hash +

Shared Access Signatures

Service

Public Blob Access

createupdatedelete

read

Service

Public Container

Access

createupdatedeleteread

list

Service

Private Container

createupdatedeleteread

list

list

Service

Private Container

readaccessfor limited timewithsharedaccesskey

sharedaccesspolicy

Browser Client

Shared AccessSignature (SAS)

>1 hour requiresauthenticationheader in request(no browser)

Deep Thoughts on Blobs

• Doh, protect your content!• Separate containers for management

roles and delegation• Write some reusable tools for saving,

chunking, shared access, reading, writing to streams, etc.

• Backup, backup, backup (AZCopy)• Snapshots, maybe

Azure Features

Azure Active Directory

Access Control

ApplicationServices

Single Sign On

Access Control (THE OLD WAY)

Your App

AccessControl

Google

Yahoo! WindowsLive

FaceBook

Browser

1 25

3

4

User Provisioning

Apps

AzureActive

Directory

Browser

AppsApps

AppsApps

Apps

DirSync with AAD

Apps

AzureActive

Directory

Browser

On PremiseAD

AppsApps

DirSync

AppsApps

Apps

Single Sign-On (sort of)

46

Deep Thoughts on IAM

• Don’t write your own identity server• Use one that can support social and

corporate identity• Provide a smooth sign up/

registration• Choose social login options carefully• You are more secure with a service• Have a back door for login (yep)

Deep Thoughts on Security Audits

• Have separate security logs• Log failed access • Retain logs 1 yr min• No production keys in open (encrypt)• Encrypt PII in database• Mask data to dev, test• Explore every web threat• Protect your content, APIs

WORST PASSWORDS TOP 3 FOR 2013

123456password12345678

Deep Thoughts on Deployment

• Wouldn’t automated deployment / upgrade be nice?

• Be very very careful– Know the location of all parts prod,

staging, dev, and test– Review configurations, again, and again– Beware running jobs (workers)

• Pray• Just kidding

53

Deep Thoughts on Logs

• Is built in better?• Know where your logs are (yep)–WAD, table storage events, security

audits, IIS logs

• Table storage or blobs (csv)?• Real time info• Historical queries• Logging can (should) evolve

Deep Thoughts on Troubleshooting

• Log everything possible• Use an agent if possible for

monitoring• Configure error alerts• Use google analytics, really• Be ready with real time queries

58

59

60

Backup and Recovery

• Back up your data base, test recovery

• Back up blob storage nightly, keep a rolling archive

• Know how to spin up a complete architecture, in emergency – State of code– Data, storage–Moving parts

Azure FeaturesCompute

VirtualNetwork

TrafficManager

CDN

Storage

Queues

Storage

Tables

Storage

Blobs

SQL Database

SQL Data Sync

SQL Reporting

HDInsight (Hadoop)

Caching Recovery Services

Windows Azure Active Directory

Access Control

Media Services

Service Bus

Store

Web Site Cloud Service

Virtual Machine

ApplicationServices

Marketplace

DataServices

Networking

MobileServices

He taught Chuck Norris martial arts

THANK YOU!