scaling big while sleeping well
DESCRIPTION
TRANSCRIPT
Agenda
• Testing the Waters● What is cloud computing?● What are the advantages of cloud computing?● How’s Azure fit in?
• Diving Deep● Digging into PHP/Ruby on Azure● Leveraging FastCGI● Leveraging Azure Storage● Leveraging SQL Azure
We are all excited about the cloud
Cloud-Addressable Challenges1
Leveraging past investments to provide future value
2
Maintaining security while increasing access and transparency—within and outside the organization
5
6 Security, access, and transparency across the value chain: suppliers, partners, etc.
7Lack of a common platform
1 Infrastructure costs are fixed and ongoing, and distract from the mission of reducing business process friction
4Matching capacity to demand
3 Many data centers at limit—real estate, capacity, cooling and power
In House or Hosted Servers
TIME
IT C
APA
CIT
Y
Actual Load
Allocated IT-
capacities
“Waste“ of capacities
“Under-supply“ of capacities
Fixed cost of IT-capacities
Load Forecast
Barrier forinnovations
Cloud Computing
Actual Load
Allocated IT capacities
Reduction of initial
investments
Reduction of “over-
supply“
No “under-supply“
Possible reduction of IT-capacities
in case of reduced load
Time
IT C
APA
CIT
YLoad
Forecast
Private(On-Premise)
Types of Clouds
Storage
Server HW
Networking
Servers
Databases
Virtualization
Runtimes
Applications
Security & Integration
You
man
age
Platform(as a
Service)
Storage
Server HW
Networking
Servers
Databases
Virtualization
Runtimes
Applications
Security & Integration M
anaged by vendor
You
man
age
Runtimes
Infrastructure
(as a Service)
Storage
Server HW
Networking
Servers
Databases
Virtualization
Applications
Security & Integration
Managed by vendor
You
man
age
The Microsoft Cloud~100 Globally Distributed Data Centers
Quincy, WA Chicago, IL San Antonio, TX Dublin, Ireland Generation 4 DCs
Platform Services
The Microsoft CloudCategories of Services
Software Services
Application Services
Infrastructure Services
The trick is knowing what your systems need,
and how the cloud can help.
Not everything has to or can live in the cloud
Only move to the cloud that which can benefit
The core strategy is ‘hybrid’
Take an À la carteapproach
We call this Software + Services
An Example…
“Packaged”Application
“Home Built”Application
Big Pharmaceutical Example
Hosted “Home Built”
Hosted “Packaged”
“Home Built”using cloud
“Packaged”using cloud
“Software as a Service”
“Platform as a Service”
Clinical Trial
Molecule Research
HR System
CRM
ERP
“Too costly to run this myself, but I’ve made too many customizations”
Bui
ld v
s. B
uyB
uild
Buy
On Premise Hoster Cloud Vendor
“Packaged”Application
“Home Built”Application
Big Pharmaceutical Example
Hosted “Home Built”
Hosted “Packaged”
“Home Built”using cloud
“Packaged”using cloud
“Software as a Service”
“Platform as a Service”
Clinical Trial
Molecule Research
HR System
CRM
ERP
“CRM and Email are commodity services – They have no customizations, and it’s cheaper for someone else to run these”
Bui
ld v
s. B
uyB
uild
Buy
On Premise Hoster Cloud Vendor
“Packaged”Application
“Home Built”Application
Big Pharmaceutical Example
Hosted “Home Built”
Hosted “Packaged”
“Home Built”using cloud
“Packaged”using cloud
“Software as a Service”
“Platform as a Service”
Clinical Trial
Molecule Research
HR System
CRM
ERP
“I can’t afford to maintain this old HR application written in VB4 – it’s driving me mad!”
“…but due to regulatory issues, I cannot store my HR data off-premise”
Bui
ld v
s. B
uyB
uild
Buy
On Premise Hoster Cloud Vendor
“Packaged”Application
“Home Built”Application
Big Pharmaceutical Example
Hosted “Home Built”
Hosted “Packaged”
“Home Built”using cloud
“Packaged”using cloud
“Software as a Service”
“Platform as a Service”
Clinical Trial
Molecule Research
CRM
ERP
HR System
“I wish I had access to cheaper compute and storage when I need it”
Bui
ld v
s. B
uyB
uild
Buy
On Premise Hoster Cloud Vendor
“Packaged”Application
“Home Built”Application
Big Pharmaceutical Example
Hosted “Home Built”
Hosted “Packaged”
“Home Built”using cloud
“Packaged”using cloud
“Software as a Service”
“Platform as a Service”
Clinical Trial
Molecule Research
CRM
ERP
HR System
Bui
ld v
s. B
uyB
uild
Buy
On Premise Hoster Cloud Vendor
“THIS is where I want to spend my IT resources – I’m going to double down on this application!”
Where the Cloud is Compelling
App Characteristic ExampleMassive scale Outback promo on Facebook
High reliability eCommerce apps (Coke, Dominos)
Variable load Online tax preparation (CCH)
Unpredictable lifetime Hurricane relief site
Massive parallel processing Genome sequencing
Needing massive or unpredictable storage
Online video archive
No desire to manage infrastructure SaaS apps (Quark’s Promote app)
Etc…
Only 2 reasons to use the cloud
Improve Bottom
Line
Improve Strategy
Windows Azure VMsSmall
$0.12 Per service hour
Medium
$0.24 Per service hour
Large
$0.48 Per service hour
X Large
$0.96 Per service hour
1 x 1.6Ghz (moderate IO)
1.75 GB memory
2 x 1.6Ghz (high IO)
3.5 GB memory
4 x 1.6Ghz (high IO)
7.0 GB memory
8 x 1.6Ghz(high IO)
14 GB memory
Windows Azure, In One Picture
Desktop
VS Tools
WA SDK
Sto
rage
Clu
ster
MSFT Datacenters
Business Portal
Developer Portal
Com
pute
Clu
ster
……
Service Management Service
REST
REST…
Cloud VM
Runtime API
VM Setup
User Code
Determine an ROI
Consider:- Current hosting costs- Current staff costs- Cooling/power/space- Migration costs- Licensing
AzureROI.CloudApp.net
ZENDFRAMEWORK
Digging into Azure with PHP/Ruby
Windows Azure, In One Picture
Desktop
VS Tools
WA SDK
Sto
rage
Clu
ster
MSFT Datacenters
Business Portal
Developer Portal
Com
pute
Clu
ster
……
Service Management Service
REST
REST…
Cloud VM
Runtime API
VM Setup
User Code
Windows Azure Roles
WebRole• External Facing • HTTP/HTTPS
Endpoint• Short lived calls
• Think Web Server
WorkerRole• Internal Facing • TCP/HTTP
Endpoint• Long running
processes
• Think Service or Deamon
Running PHP in Windows Azure• How to Do It
● Host in Web role (like .NET)● Supply PHP runtime● Point to runtime via FastCGI
configuration in• Web.config• Web.roleConfig
PHP Web RoleInstance 2
VIP
Load
Bal
ance
r
PHP Web RoleInstance 1
Windows Azure 4 Eclipse
• Does a lot of the heavy lifting for you• Creates the Web.config, Web.roleConfig,
PHP implementation and debugs in the dev fabric
• http://windowsazure4e.org
Running EXEs in the Worker Process
• Start with a Worker Role Process to ● Copy the exes to the worker role’s app
directory● Execute the process● Monitor the health of the process● Other auxiliary tasks…
Running Ruby in the Worker Process
• http://code.msdn.microsoft.com/railsonazure • Download and extract the solution • Copy the Ruby bin and lib folders to the
Ruby folder in the RR project • Copy your rails application to the RailsApp
folder in the RR project • Open the solution in Visual Studio • Run in the Dev Fabric• Navigate to http://127.0.0.1:5100
Data Storage
Storage options
• Windows Azure Data• Sql Azure
Windows Azure Data Storage
Account
Queue
Blob
Tables
Drives
Azure Platform Data Storage Options
• Windows Azure Data Storage• Blobs
• Unstructured data storage• Tables
• Semi-structured or tabular data storage• Queues
• Buffered delivery data storage• Drives
• Durable NTFS volumes that Windows Azure applications can use. See: http://microsoftpdc.com/Sessions/SVC14
• SQL Azure• Relational data storage
Admin Windows Azure Data Storage
Windows Azure Data Storage
PHP with Windows Azure Storage• Windows Azure SDK for PHP @
http://phpazure.codeplex.com
• PHP programming model for Windows Azure Storage
• Features ● PHP classes for Blobs, Tables & Queues● Store PHP sessions in Table Storage
Windows Azure Data Storage - Blobs
• Unstructured data storage
• Partitioned by container
• Unlimited containers
Blobs Sample$blobStorage = new Microsoft_WindowsAzure_Storage_Blob();// Createif (!$blobStorage->containerExists($containerName)){ $blobStorage->createContainer($containerName); $blobStorage->setContainerAcl($containerName, Microsoft_WindowsAzure_Storage_Blob::ACL_PUBLIC);}
// Store$blob = $blobStorage->putBlob($containerName, $blobName, $localFilename, $metadata);/* @var $blob Microsoft_WindowsAzure_Storage_BlobInstance */
Blobs Sample Cont…// Copy$result = $blobStorage->copyBlob( $containerName, $blobName, $containerName, $blob2Name);
// Retrieve$tempStore = azure_getlocalresourcepath('tempstore');$tempPath = $tempStore . '\\' . $imageId;$blobStorage->getBlob($containerName, $blobName, $tempPath);
// Delete$result = $blobStorage->deleteBlob($containerName, $blobName);$result = $blobStorage->deleteContainer($containerName);
Blob Stream Wrapper$blobStorage = new Microsoft_WindowsAzure_Storage_Blob();// Register:$blobStorage->registerStreamWrapper(); // registers azure://// or$blobStorage->registerStreamWrapper('blob://'); // use blob://
// Use$fp = fopen('azure://mycontainer/myfile.txt', 'r');// ...fclose($fp);
Storage: XDrive
● NTFS drive in the cloud: X:\● Fixed size between 16MB and 1TB● Max 8 drives on your VM● Implemented as a page blob● Single instance write● Multiple instance read
Queue Workflow Concepts
• Windows Azure Queue Provides● Guarantee delivery (two-step consumption)
1. Worker Dequeues Message and mark it as Invisible
2. Worker Deletes Message when finished processing it
If Worker role crashes, message becomes visible for another Worker to process
● Doesn’t guarantee “only once” delivery● Doesn’t guarantee ordering
• Best effort FIFO
Azure QueueInput Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
Azure Queues
Queue
Msg 1
Msg 2
Msg 3
Msg 4
Worker Role
Worker Role
PutMessage
Web Role
GetMessage (Timeout)
RemoveMessage
Msg 2Msg 1
Worker Role
Msg 2
Loosely Coupled Work with Queues
• Worker-Queue Model● Load work in a queue● Many workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
Queues$queueClient = new Microsoft_WindowsAzure_Storage_Queue();
// Create$result = $queueClient->createQueue('imageQueue');
// Delete$queueClient->deleteQueue('imageQueue');
// Add message$queueClient->putMessage('imageQueue', $message, $ttl);
// Retrieve Messages$messages = $queueClient->getMessages('imageQueue', 10);foreach ($messages as $message) { // Do work here... $queueClient->deleteMessage('imageQueue', $message);}
Windows Azure Data Storage - Tables• Semi-Structured data• Tables contain entities• Entities contain properties• May be partitioned across
thousands of servers.• Support ACID transactions
over single entities• Queries over entire table• .NET and REST interfaces
Windows Azure Data Storage – Tables (Terms Part 1)• Table
● Contains a set of entities. • Entity (Row)
● Basic data items stored in a table. • Property (Column)
● Single value in an entity. • RowKey
● Unique ID of the entity within a partition• Timestamp
● Time it was created
Windows Azure Data Storage – Tables (Terms Part 2)• Partition
● Entities in a table with the same partition key• PartitionKey
● Segments entities in to partitions to automatically distribute the table’s entities over many storage nodes.
• Sort Order● There is a single index provided for the CTP,
where all entities in a table are sorted by PartitionKey and then RowKey
Key Example – Blog Posts
• Getting all of dunnry’s blog posts is fast● Single partition
• Getting all posts after 2008-03-27 is slow● Traverse all partitions
Partition KeyAuthor
Row KeyPermalink
Property 3ChangedO
nProperty 4
Title
smarx i-love-tables 2009-07-04 I Love Tables!
smarx tables-are-awesome 2009-07-12 Tables are
Awesome!
dunnry cheetos 2008-03-27 I Love Cheetos
dunnry blogging-again 2009-07-15 Finally Blogging
Again
dunnry phluffyfotos 2008-04-09 PhluffyFotos!
Partition 1
Partition 2
Table Sample$tableStorage = new Microsoft_WindowsAzure_Storage_Table( 'table.core.windows.net', 'myaccount', 'myauthkey');// Create$result = $tableStorage->createTable($tableName);// List $result = $tableStorage->listTables();foreach ($result as $table) { echo 'Table name is: ' . $table->Name . "\n";}// Delete$tableStorage->deleteTable($tableName);
Tables with Entities// Structured entityclass ImageEntity extends Microsoft_WindowsAzure_Storage_TableEntity{ /** * @azure filename */ public $filename; /** * @azure size Edm.Int64 */ public $size;}
// Unstructured entity// Microsoft_WindowsAzure_Storage_DynamicTableEntity
Tables with Entities Cont…// Insert$image = new ImageEntity($partitionKey, $rowKey);$image->filename = $_FILES['imageUpload']['name'];$image->size = $_FILES['imageUpload']['size'];$image = $tableStorageClient->insertEntity($tableName, $image);
// Retrieve$image = $tableStorage->retrieveEntityById($tableName, $partitionKey, $rowKey, 'ImageEntity');
// Update$image->filename = 'newname.jpg';$result = $tableStorage->updateEntity($tableName, $image);
// Delete$result = $tableStorage->deleteEntity($tableName, $image);
Table Queries// Filter condition$select = "filesize gt 1024 and PartitionKey eq '$partitionKey'";
// Or fluent interface$select = $tableStorage->select()->from($tableName) ->where('filesize gt 1024') ->andWhere('PartitionKey eq ?', $partitionKey);
// Run query$images = $tableStorage->storageClient->retrieveEntities( 'testtable', $select, 'ImageEntity');foreach ($images as $image) { echo 'Name: ' . $image->filename . "\n";}
Batching Operations// Start batch$batch = $tableStorage->startBatch();
// Insert multiple$images = generateEntities();foreach ($images as $image) { $tableStorage->insertEntity($tableName, $image);} // Commit$batch->commit();
Table session handler$sessionHandler = new Microsoft_WindowsAzure_SessionHandler($tableStorage);$sessionHandler->register();
session_start();
if (!isset($_SESSION['start'])) { $_SESSION['start'] = time();}
SQL Azure and Windows Azure Table Comparison
SQL Azure Tables• Fully structured• Strongly typed• Relational (RDMS)• Highly scalable
Windows Azure Tables• Semi-structured• Loosely typed• Non-Relational (Not RDMS)• Massively scalable
MySQL: Simple Configuration
Web
Rol
e
VIP
Load Balancer
Wor
ker
Rol
e
MySQL
MySQL in a Windows Azure Application• Running MySQL in a worker role
● Copy MySQL to the worker role sub-directory● Copy to read-write local storage● Configure MySQL to listen on the right port● Monitor MySQL health
• Consuming MySQL● Discover IP address and port● Normal access from then on● Handle topology changes
Simple Configuration
VIP
Load Balancer
MySQL
Replication
VIP
Load Balancer
MySQL MySQLMySQL
MS S
Windows Azure Drive with Hot Spare
VIP
Load Balancer
MySQL MySQL
Windows Azure Drive with Hot Spare
VIP
Load Balancer
MySQL MySQL
SQL Azure
SQL Azure Features
• Supported● Tables, Indexes, Views● Stored Procedures● Triggers● Constraints● Table Variables● Temp Tables (#Name)
• Not Supported● Physical Server Access ● Catalog DDL● Common Language
Runtime● Service Broker● Reporting Services● Analysis Services● Distributed
Transactions and Queries
SQL AzureDeployment
Web Portal(API)
SQL AzureTDS
DB Script
SQL AzureAccessing databases
Web Portal(API)
SQL AzureTDS
Your App
Change Connection String
Database Replicas
Replica 1
Replica 2
Replica 3
DB
Single Database Multiple Replicas
Single Primary
SQL AzureDatabase Monitoring & Recovery
Web Portal(API)
SQL AzureTDS
Your App !
SQL Azure Server Creation
SQL Azure Firewall Maintenance• Simple rules• Easy one-screen portal maintenance
SQL Azure Database Connection String
• An administrative user is created with the server
• User has system administrator permissions like sa”
• Server=tcp:itte80vcfq.database.windows.net; Database=FabrikamAzureDB;User ID=SaPaulM;Password=myPassword;Trusted_Connection=False;Encrypt=True;
Database Migrations
• Scripts● Transfer Schema
• BCP● Transfer Data
• SSIS (SQL Server Integration Service)● Transfer Schema and Data
Database Migrations (Cont.)
• The SQL Azure Migration Wizard helps you migrate your local SQL Server 2005 / 2008 databases into SQL Azure. The wizard walks you through the selection of your SQL objects, creates SQL scripts suitable for SQL Azure, and allows you to migrate your data.”
http://sqlazuremw.codeplex.com/
Database Management using SQL Management Studio
PHP with SQL Azure
• SQL Server Driver for PHP @ http://sqlsrvphp.codeplex.com/
• Supports PHP access to SQL Azure
• Features● Choose between SQL Server and SQL Azure by
changing connection string● Use from on-premises or in Windows Azure
More resources
• Microsoft Windows Azure Interop• http://www.microsoft.com/windowsazure/interop/
• Interop Bridges• http://www.interoperabilitybridges.com/
More resources
• PHP• http://www.windowsazure4e.org • http://phpazure.codeplex.com/
• MySQL● Windows Azure MySQL PHP Solution
Accelerator• http://code.msdn.microsoft.com/winazuremysqlphp
More resources
• Python● Sriram Krishnan
• http://www.sriramkrishnan.com/blog/2008/11/python-wrapper-for-windows-azure.html
• Ruby● Simon Davies
• http://blogs.msdn.com/simondavies• http://code.msdn.microsoft.com/railsonazure
● Johnny Halife• http://github.com/johnnyhalife/waz-storage • http://waz-storage.heroku.com/rdoc/index.html
