Feature Decision Making with Hybrid IT Solutions

Mick Badran

MVP, VTSPb-mickbaninjamic

Scott Scovell

VTSPb-scoscoscothestig

What by saying its name destroys it?

The Cloud

"640K ought to be enough for anybody." - Bill Gates, 1981

Session TopicsOverview of the SolutionHybrid Technology PointsAzure Storage – Discussion & DemoAzure Service Bus – Discussion & DemoSQL Azure & SQL Sync – Discussion & DemoAzure Connect – Discussion & DemoVarious improvementsSummary & Wrap up

Overview of our Solution

Service Bus

On Premise

Azure Storage

TablesBlobs

Queues….

Cloud Compute

ComponentsRolesClients…

Partner Systems.

Data Feeds…

SQL Azur

e

ACS

Azure Storage

TablesBlobs

Queues….

Overview of our SolutionThe Client – An Online Gaming Corp.

The Facts

Global Geo Distributed

High volume near real time

BTS/SQL On-Prem

Tasks

High Volume, low latency public interface(s)

Work with live on-premise high volume feeds

Handle money, prices and betting

Technologies

Mixture of on-Prem, BTS and lots

of Cloud.

Monitoring

Connectivity between worlds –

HOW?

Hybrid Technology

Storage

Data

Integration

Security

Compute

Networking

SQL Server

ADFS2

BizTalk

.NET WCF

Connect

SQL Azure Data Sync

AppFabric Access Control

AppFabric Service

Bus

.NET WCF

Blob Storage

Table Storage

Queue Storage

AppFabric Integration

Azure Storage

Geographically distributed across 3 regionsAnywhere are Anytime access to data>200 Petabytes of raw storage (Dec 2011)

Azure Storage

AbstractionsBlobs – File system in the cloudTables – Massively scalable structured storageQueues – Reliable storage and delivery of messagesDrives – Durable NTFS volumes for Windows Azure applications

Easy client accessEasy to use REST APIs and Client LibrariesExisting NTFS APIs for Windows Azure Drives

Azure Storage Concepts

Account

Container Blobs

Table Entities

Queue Messages

https://<account>.blob.core.windows.net/<container>

https://<account>.table.core.windows.net/<table>

https://<account>.queue.core.windows.net/<queue>

10

Some…Azure Storage ConstraintsQuota Value Description

Storage Account

Storage accounts per subscription 5 By default. Additional accounts can be added via customer support.

Maximum storage for a single account 100 TB Total size across all blobs, tables and queues

Blob

Maximum Blob size

Block blobs 200 GB Note that the storage emulator only supports blob sizes up to 2 GB.

Page blobs 1 TB

Maximum Blob metadata size 8 KB <name,value> pairs up to 8 KB in total size

Table

Maximum number of properties per Entity 255 Combined size of all properties must be < 1 MB

Maximum size of single property value 64 KB

(many more…)

Queue

Maximum number of messages on a queueunlimite

dAs long as max. storage size for account remains < 100 TB

Maximum size for a queued message 64 KB v1.6 release (prev 8 KB)

Azure Drive

Maximum VHD size 1 TB Stored as page blob

Maximum number of drives per role instance 16 Only available to 1 VM at at time

Scalability & Performance ConstraintsSingle storage account:

Capacity – Up to 100 TBsTransactions – Up to 5,000 per secondBandwidth – Up to 3 gigabits per second

Single Queue:Up to 500 messages per second – ‘theoretical’

Single Table Partition:Up to 500 entities per secondNote, this is for a single partition, and not a single table. See why good partitioning is a must!

Single Blob:Up to 60 MB/sec

Our Design challenges

How do you expose dynamic, pre-processed data out to public consumers?How do you balance work across scale-out workers?How do you transfer messages reliably?How do you queue messages that can be greater than 8 KB?

Feature Design

Service Bus

On Premise

Azure Storag

e

TablesBlobs

Queues….

Cloud Compute Componen

tsRoles

Clients…

Partner Systems.

Data Feeds…

SQL Azure

ACS

Azure Storag

e

TablesBlobs

Queues….

On Premise

Azure Storag

e

TablesBlobs

Queues….

Azure Storag

e

TablesBlobs

Queues….

DEMO

Large Message Queue Support

Let’s open up the discussion

How many of you have faced similar challenges?How did you approach the solution?Any other compelling constraints you have hurdled over?…real world goodness shared

Azure Service Bus

Interconnect services across networksNaming and DiscoveryConnectivity w/ NAT and Firewall TraversalAccess Control Integrated Authorization

Cloud-Based Message-Oriented Middleware

Next Generation Message Queue TechnologyRich Publish/Subscribe with FilteringTransaction Integration

Service Bus

Receiver

Frontend

Nodes

Relay Messaging

Using Outbound Connections OnlyNo open inbound firewall/NAT portsOutbound connections default to TCPFall back to HTTP when neededGuarded by Datacenter FirewallSecured with Access Control

Queues

QueueP C

Reliable, durable storage – up to 1GB per Queue*Queues stay in the system once created, no TTL limitMax message size 256KB, Sessions allow groupingMessaging API, WCF, and HTTP/REST Interfaces

Topics

TopicP S

S

S

C

C

C

All the Features of Queues + Publish/SubscribeUp to 2000 Subscriptions on a TopicEach Subscription is a virtual queue getting message copiesSubscriptions can have filters and actions

Service Bus Constraints (1/3)Quota Name Scope Type Behavior when exceeded Value

Queue/Topic size Entity Defined upon

creation of

the

queue/topic.

Incoming messages will be rejected and an

exception will be received by the calling code.

1,2,3,4 or 5 Gigabytes.

Number of concurrent connections on a queue/topic/subscription entity

Entity Static Subsequent requests for additional connections

will be rejected and an exception will be received

by the calling code. REST operations do not count

towards concurrent TCP connections.

100

Number of concurrent listeners on a relay

Entity Static Subsequent requests for additional connections

will be rejected and an exception will be received

by the calling code.

25

Number of concurrent relay listeners

System-wide Static Subsequent requests for additional connections

will be rejected and an exception will be received

by the calling code.

2000

Number of topics/queues per service namespace

System-wide Static Subsequent requests for creation of a new topic or

queue on the service namespace will be rejected.

As a result, if configured through the management

portal, an error message will be generated. If

called from the management API, an exception will

be received by the calling code.

10,000

Service Bus Constraints (2/3)Quota Name Scope Type Behavior when exceeded Value

Message size for a

queue/topic/subscription

entity

System-wide Static Incoming messages that exceed these

quotas will be rejected and an

exception will be received by the

calling code.

Maximum message size: 256KB

Maximum header size: 64KB

Maximum number of header properties in

property bag: MaxValue Maximum size of

property in property bag: No explicit limit.

Limited by maximum header size.

Message size for Message

Buffer

System-wide Static Incoming messages that exceed these

quotas will be rejected and an

exception will be received by the

calling code.

64KB

Message size for

NetOnewayRelayBinding 

and NetEventRelayBinding

 relays

System-wide Static Incoming messages that exceed these

quotas will be rejected and an

exception will be received by the

calling code.

64KB

Message size for

HttpRelayTransportBindingElement

and NetTcpRelayBinding

 relays

System-wide Static Unlimited

Message property size for a

queue/topic/subscription

entity

System-wide Static A SerializationException exception

is generated.

Maximum message property size for each

property is 32K. Cumulative size of all

properties cannot exceed 64K. This applies

to the entire header of the BrokeredMessage

, which has both user properties as well as

system properties (such as 

SequenceNumber, Label, MessageId, and so

on).

Service Bus Constraints (3/3)Quota Name Scope Type Behavior when exceeded Value

Number of

subscriptions per

topic

System-wide Static Subsequent requests for creating additional

subscriptions for the topic will be rejected. As a

result, if configured through the management

portal, an error message will be shown. If called

from the management API an exception will be

received by the calling code.

2,000

Number of SQL

filters per topic

System-wide Static Subsequent requests for creation of additional

filters on the topic will be rejected and an

exception will be received by the calling code.

2,000

Number of

correlation filters

per topic

System-wide Static Subsequent requests for creation of additional

filters on the topic will be rejected and an

exception will be received by the calling code.

100,000

Size of SQL

filters/actions

System-wide Static Subsequent requests for creation of additional

filters will be rejected and an exception will be

received by the calling code.

Maximum length of filter

condition string: 4KMaximum

length of rule action string:

4KMaximum number of

expressions per rule action: 64

Our Design Challenges

How do you achieve load-decoupling between subsystems?How do you balance work across scale-out workers?How do you distribute events to dynamic subscribers? How do you transfer messages reliably and transactionally?How do you reach into on-premise systems across firewalls?How do you organize and discover service endpoints?

Feature Design

Exposing on-premise services via Azure Service Bus relay endpointsLoad balancing and high availability supportLow latency, high volume capability

Service Bus

On Premise

Azure Storag

e

TablesBlobs

Queues….

Cloud Compute Componen

tsRoles

Clients…

Partner Systems.

Data Feeds…

SQL

Azure

ACS

Azure Storag

e

TablesBlobs

Queues….

On Premise

Azure Storag

e

TablesBlobs

Queues….

Azure Storag

e

TablesBlobs

Queues….

Demo

Load Balancing and High Availability

demo

Let’s open up the discussion

How many of you have faced similar challenges?How did you approach the solution?Any other compelling constraints you have hurdled over?…real world goodness shared

SQL Azure & SQL Sync – Discussion & Demo

Data Sync

Database Reporting

Feature Design

Service Bus

On Premise

Azure Storag

e

TablesBlobs

Queues….

Cloud Compute Componen

tsRoles

Clients…

Partner Systems.

Data Feeds…

SQL Azure

ACS

Azure Storag

e

TablesBlobs

Queues….

On Premise

Azure Storag

e

TablesBlobs

Queues….

Azure Storag

e

TablesBlobs

Queues….

Familiar SQL Server relational database model delivered as a service

Support for existing APIs & toolsBuilt for the cloud with high availability & fault toleranceEasily provision and manage databases across multiple datacenters

SQL Azure provides logical serverGateway server that understands TDS protocolLooks like SQL Server to TDS ClientActual data stored on multiple backend data nodes

Logical optimisations supportedIndexes, Query plans etc..

Physical optimizations not supportedFile Groups, Partitions etc…

Transparently manages physical storage

SQL Azure Database

SQL Server Database Migration Considerations

Database sizeI/O requirementsSQL Azure Compatibility

SecurityMany databases (cross database joins)OLE-DB – Not supportedLinked Servers

See MSDN SQL Azure Databasehttp://msdn.microsoft.com/en-us/library/ee336279.aspx

SQL Azure Constraints

Currently SupportedTables, indexes and viewsStored ProceduresFunctionsCursorsTriggersConstraintsTable variables, session temp tables (#t)Spatial typesHierarchyIdSparse ColumnsThrow statement

Not SupportedChange Data CaptureCross Database JoinsData AuditingData CompressionDatabase file placementDatabase mirroringExternal Key Management / Extensible Key ManagementFILESTREAM DataFull-text Search / IndexesGlobal temporary tablesPerformance Data Collection (Data Collector)Policy-Based ManagementResource GovernorSQL-CLRSQL Server ReplicationTransparent Data EncryptionTable PartitionsTables require clustered indexes

Geo-replication of SQL Azure data and spanning on-premises with cloud

geo-replicationadds sync between SQL Server and SQL Azure

Builds on Sync FrameworkSync data between SQL Azure, SQL Compact and SQL ServerNext version enables sync for offline clients

On-Premises

Cloud

Offline Clients

SQL Azure Data Sync

Demo

SQL, SQL Azure + SQL Sync

demo

Let’s open up the discussion

How many of you have faced similar challenges?How did you approach the solution?Any other compelling constraints you have hurdled over?…real world goodness shared

Azure Connect – Discussion & Demo

Secure network connectivity between on-premises and cloud

Supports standard IP protocols

Enables hybrid apps access to on-premises serversAllows remote administration of Windows Azure apps

Simple setup and management

Integrated with WA Service ModelWeb, Worker and VM Roles supported

Enterprise

Windows Azure

Windows Azure Connect

Network policy managed through Windows Azure portal

Granular control of connectivity between WA roles and external machines

Automatic setup of IPsecTunnel firewalls/NAT’s through hosted SSL-based relay Network policies enforced & traffic secured via end-to-end certificate-based IPSecDNS name resolution based on endpoint machine names

Enterprise

Windows Azure

Databases

Dev machines

Relay

Role BRole A

Role C(multiple VM’s)

Windows Azure Connect

Azure Connect - Today

demo

Feature Decision Making with Hybrid IT

Solutions Wrap up

We covered a range of technologiesKnow what the hybrid components areKnow what the limitations are

Thank you!!!!

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions,

it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.