Feature Decision Making with Hybrid IT Solutions
Mick Badran
MVP, VTSPb-mickbaninjamic
Scott Scovell
VTSPb-scoscoscothestig
What by saying its name destroys it?
The Cloud
"640K ought to be enough for anybody." - Bill Gates, 1981
Session TopicsOverview of the SolutionHybrid Technology PointsAzure Storage – Discussion & DemoAzure Service Bus – Discussion & DemoSQL Azure & SQL Sync – Discussion & DemoAzure Connect – Discussion & DemoVarious improvementsSummary & Wrap up
Overview of our Solution
Service Bus
On Premise
Azure Storage
TablesBlobs
Queues….
Cloud Compute
ComponentsRolesClients…
Partner Systems.
Data Feeds…
SQL Azur
e
ACS
Azure Storage
TablesBlobs
Queues….
Overview of our SolutionThe Client – An Online Gaming Corp.
The Facts
Global Geo Distributed
High volume near real time
BTS/SQL On-Prem
Tasks
High Volume, low latency public interface(s)
Work with live on-premise high volume feeds
Handle money, prices and betting
Technologies
Mixture of on-Prem, BTS and lots
of Cloud.
Monitoring
Connectivity between worlds –
HOW?
Hybrid Technology
Storage
Data
Integration
Security
Compute
Networking
SQL Server
ADFS2
BizTalk
.NET WCF
Connect
SQL Azure Data Sync
AppFabric Access Control
AppFabric Service
Bus
.NET WCF
Blob Storage
Table Storage
Queue Storage
AppFabric Integration
Azure Storage
Geographically distributed across 3 regionsAnywhere are Anytime access to data>200 Petabytes of raw storage (Dec 2011)
Azure Storage
AbstractionsBlobs – File system in the cloudTables – Massively scalable structured storageQueues – Reliable storage and delivery of messagesDrives – Durable NTFS volumes for Windows Azure applications
Easy client accessEasy to use REST APIs and Client LibrariesExisting NTFS APIs for Windows Azure Drives
Azure Storage Concepts
Account
Container Blobs
Table Entities
Queue Messages
https://<account>.blob.core.windows.net/<container>
https://<account>.table.core.windows.net/<table>
https://<account>.queue.core.windows.net/<queue>
10
Some…Azure Storage ConstraintsQuota Value Description
Storage Account
Storage accounts per subscription 5 By default. Additional accounts can be added via customer support.
Maximum storage for a single account 100 TB Total size across all blobs, tables and queues
Blob
Maximum Blob size
Block blobs 200 GB Note that the storage emulator only supports blob sizes up to 2 GB.
Page blobs 1 TB
Maximum Blob metadata size 8 KB <name,value> pairs up to 8 KB in total size
Table
Maximum number of properties per Entity 255 Combined size of all properties must be < 1 MB
Maximum size of single property value 64 KB
(many more…)
Queue
Maximum number of messages on a queueunlimite
dAs long as max. storage size for account remains < 100 TB
Maximum size for a queued message 64 KB v1.6 release (prev 8 KB)
Azure Drive
Maximum VHD size 1 TB Stored as page blob
Maximum number of drives per role instance 16 Only available to 1 VM at at time
Scalability & Performance ConstraintsSingle storage account:
Capacity – Up to 100 TBsTransactions – Up to 5,000 per secondBandwidth – Up to 3 gigabits per second
Single Queue:Up to 500 messages per second – ‘theoretical’
Single Table Partition:Up to 500 entities per secondNote, this is for a single partition, and not a single table. See why good partitioning is a must!
Single Blob:Up to 60 MB/sec
Our Design challenges
How do you expose dynamic, pre-processed data out to public consumers?How do you balance work across scale-out workers?How do you transfer messages reliably?How do you queue messages that can be greater than 8 KB?
Feature Design
Service Bus
On Premise
Azure Storag
e
TablesBlobs
Queues….
Cloud Compute Componen
tsRoles
Clients…
Partner Systems.
Data Feeds…
SQL Azure
ACS
Azure Storag
e
TablesBlobs
Queues….
On Premise
Azure Storag
e
TablesBlobs
Queues….
Azure Storag
e
TablesBlobs
Queues….
DEMO
Large Message Queue Support
Let’s open up the discussion
How many of you have faced similar challenges?How did you approach the solution?Any other compelling constraints you have hurdled over?…real world goodness shared
Azure Service Bus
Interconnect services across networksNaming and DiscoveryConnectivity w/ NAT and Firewall TraversalAccess Control Integrated Authorization
Cloud-Based Message-Oriented Middleware
Next Generation Message Queue TechnologyRich Publish/Subscribe with FilteringTransaction Integration
Service Bus
Receiver
Frontend
Nodes
Relay Messaging
Using Outbound Connections OnlyNo open inbound firewall/NAT portsOutbound connections default to TCPFall back to HTTP when neededGuarded by Datacenter FirewallSecured with Access Control
Queues
QueueP C
Reliable, durable storage – up to 1GB per Queue*Queues stay in the system once created, no TTL limitMax message size 256KB, Sessions allow groupingMessaging API, WCF, and HTTP/REST Interfaces
Topics
TopicP S
S
S
C
C
C
All the Features of Queues + Publish/SubscribeUp to 2000 Subscriptions on a TopicEach Subscription is a virtual queue getting message copiesSubscriptions can have filters and actions
Service Bus Constraints (1/3)Quota Name Scope Type Behavior when exceeded Value
Queue/Topic size Entity Defined upon
creation of
the
queue/topic.
Incoming messages will be rejected and an
exception will be received by the calling code.
1,2,3,4 or 5 Gigabytes.
Number of concurrent connections on a queue/topic/subscription entity
Entity Static Subsequent requests for additional connections
will be rejected and an exception will be received
by the calling code. REST operations do not count
towards concurrent TCP connections.
100
Number of concurrent listeners on a relay
Entity Static Subsequent requests for additional connections
will be rejected and an exception will be received
by the calling code.
25
Number of concurrent relay listeners
System-wide Static Subsequent requests for additional connections
will be rejected and an exception will be received
by the calling code.
2000
Number of topics/queues per service namespace
System-wide Static Subsequent requests for creation of a new topic or
queue on the service namespace will be rejected.
As a result, if configured through the management
portal, an error message will be generated. If
called from the management API, an exception will
be received by the calling code.
10,000
Service Bus Constraints (2/3)Quota Name Scope Type Behavior when exceeded Value
Message size for a
queue/topic/subscription
entity
System-wide Static Incoming messages that exceed these
quotas will be rejected and an
exception will be received by the
calling code.
Maximum message size: 256KB
Maximum header size: 64KB
Maximum number of header properties in
property bag: MaxValue Maximum size of
property in property bag: No explicit limit.
Limited by maximum header size.
Message size for Message
Buffer
System-wide Static Incoming messages that exceed these
quotas will be rejected and an
exception will be received by the
calling code.
64KB
Message size for
NetOnewayRelayBinding
and NetEventRelayBinding
relays
System-wide Static Incoming messages that exceed these
quotas will be rejected and an
exception will be received by the
calling code.
64KB
Message size for
HttpRelayTransportBindingElement
and NetTcpRelayBinding
relays
System-wide Static Unlimited
Message property size for a
queue/topic/subscription
entity
System-wide Static A SerializationException exception
is generated.
Maximum message property size for each
property is 32K. Cumulative size of all
properties cannot exceed 64K. This applies
to the entire header of the BrokeredMessage
, which has both user properties as well as
system properties (such as
SequenceNumber, Label, MessageId, and so
on).
Service Bus Constraints (3/3)Quota Name Scope Type Behavior when exceeded Value
Number of
subscriptions per
topic
System-wide Static Subsequent requests for creating additional
subscriptions for the topic will be rejected. As a
result, if configured through the management
portal, an error message will be shown. If called
from the management API an exception will be
received by the calling code.
2,000
Number of SQL
filters per topic
System-wide Static Subsequent requests for creation of additional
filters on the topic will be rejected and an
exception will be received by the calling code.
2,000
Number of
correlation filters
per topic
System-wide Static Subsequent requests for creation of additional
filters on the topic will be rejected and an
exception will be received by the calling code.
100,000
Size of SQL
filters/actions
System-wide Static Subsequent requests for creation of additional
filters will be rejected and an exception will be
received by the calling code.
Maximum length of filter
condition string: 4KMaximum
length of rule action string:
4KMaximum number of
expressions per rule action: 64
Our Design Challenges
How do you achieve load-decoupling between subsystems?How do you balance work across scale-out workers?How do you distribute events to dynamic subscribers? How do you transfer messages reliably and transactionally?How do you reach into on-premise systems across firewalls?How do you organize and discover service endpoints?
Feature Design
Exposing on-premise services via Azure Service Bus relay endpointsLoad balancing and high availability supportLow latency, high volume capability
Service Bus
On Premise
Azure Storag
e
TablesBlobs
Queues….
Cloud Compute Componen
tsRoles
Clients…
Partner Systems.
Data Feeds…
SQL
Azure
ACS
Azure Storag
e
TablesBlobs
Queues….
On Premise
Azure Storag
e
TablesBlobs
Queues….
Azure Storag
e
TablesBlobs
Queues….
Demo
Load Balancing and High Availability
demo
Let’s open up the discussion
How many of you have faced similar challenges?How did you approach the solution?Any other compelling constraints you have hurdled over?…real world goodness shared
SQL Azure & SQL Sync – Discussion & Demo
Data Sync
Database Reporting
Feature Design
Service Bus
On Premise
Azure Storag
e
TablesBlobs
Queues….
Cloud Compute Componen
tsRoles
Clients…
Partner Systems.
Data Feeds…
SQL Azure
ACS
Azure Storag
e
TablesBlobs
Queues….
On Premise
Azure Storag
e
TablesBlobs
Queues….
Azure Storag
e
TablesBlobs
Queues….
Familiar SQL Server relational database model delivered as a service
Support for existing APIs & toolsBuilt for the cloud with high availability & fault toleranceEasily provision and manage databases across multiple datacenters
SQL Azure provides logical serverGateway server that understands TDS protocolLooks like SQL Server to TDS ClientActual data stored on multiple backend data nodes
Logical optimisations supportedIndexes, Query plans etc..
Physical optimizations not supportedFile Groups, Partitions etc…
Transparently manages physical storage
SQL Azure Database
SQL Server Database Migration Considerations
Database sizeI/O requirementsSQL Azure Compatibility
SecurityMany databases (cross database joins)OLE-DB – Not supportedLinked Servers
See MSDN SQL Azure Databasehttp://msdn.microsoft.com/en-us/library/ee336279.aspx
SQL Azure Constraints
Currently SupportedTables, indexes and viewsStored ProceduresFunctionsCursorsTriggersConstraintsTable variables, session temp tables (#t)Spatial typesHierarchyIdSparse ColumnsThrow statement
Not SupportedChange Data CaptureCross Database JoinsData AuditingData CompressionDatabase file placementDatabase mirroringExternal Key Management / Extensible Key ManagementFILESTREAM DataFull-text Search / IndexesGlobal temporary tablesPerformance Data Collection (Data Collector)Policy-Based ManagementResource GovernorSQL-CLRSQL Server ReplicationTransparent Data EncryptionTable PartitionsTables require clustered indexes
Geo-replication of SQL Azure data and spanning on-premises with cloud
geo-replicationadds sync between SQL Server and SQL Azure
Builds on Sync FrameworkSync data between SQL Azure, SQL Compact and SQL ServerNext version enables sync for offline clients
On-Premises
Cloud
Offline Clients
SQL Azure Data Sync
Demo
SQL, SQL Azure + SQL Sync
demo
Let’s open up the discussion
How many of you have faced similar challenges?How did you approach the solution?Any other compelling constraints you have hurdled over?…real world goodness shared
Azure Connect – Discussion & Demo
Secure network connectivity between on-premises and cloud
Supports standard IP protocols
Enables hybrid apps access to on-premises serversAllows remote administration of Windows Azure apps
Simple setup and management
Integrated with WA Service ModelWeb, Worker and VM Roles supported
Enterprise
Windows Azure
Windows Azure Connect
Network policy managed through Windows Azure portal
Granular control of connectivity between WA roles and external machines
Automatic setup of IPsecTunnel firewalls/NAT’s through hosted SSL-based relay Network policies enforced & traffic secured via end-to-end certificate-based IPSecDNS name resolution based on endpoint machine names
Enterprise
Windows Azure
Databases
Dev machines
Relay
Role BRole A
Role C(multiple VM’s)
Windows Azure Connect
Azure Connect - Today
demo
Feature Decision Making with Hybrid IT
Solutions Wrap up
We covered a range of technologiesKnow what the hybrid components areKnow what the limitations are
Thank you!!!!
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions,
it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.