windows azure and a little sql data services
DESCRIPTION
User group talk on storing data in the cloud. Main focus is on Table storage in Windows Azure but with reference to SQL Data Services.TRANSCRIPT
Eric NelsonDeveloper & Platform GroupMicrosoft [email protected] http://geekswithblogs.net/IUpdateable http://twitter.com/ericnel
Cloud101Microsoft and CloudAzure Services Platform
Storage in the CloudWindows Azure Storage + SQL Data Services
Windows AzureOverviewFocus on storage
SQL Data ServicesFutures
It is not that hard
SaaS = Software as a ServiceAka “On Demand” vs “On Premise” software
Many customers hate “On Premise” hassle
Independent Software Vendors (ISVs) exploredSalesForce.com championed this model – “No Software”
Single tenant vs Multi tenant etcS+S = Software + Services
Microsoft marketing department documenting realityPowerful client software working with powerful cloud based servicesThink Xbox Live, Itunes, Skype, MessengerSalesForce.com really do this...
Cloud ComputingRun/Store stuff in the cloud -
Somebody else has the data center
Amazon.com key role in Cloud ComputingAmazon S3 (Simple Storage Service) – objectsAmazon EC2 (Elastic Compute Cloud) – virtual machines
WebServices and RESTful WebServices
users
eric
bill
sarah
tim
HTTP RequestURL
VERB
Payload
HTTP Response
Status
GETPOSTPUT
DELETEXML JSON
Payload
XML JSON
listAllUsers() vs http://mysite.com/users/ ?addUser() vs POST http://mysite.com/users/ deleteUser() vs DELETE http://mysite.com/users/eric updateUser() vs PUT http://mysite.com/users/eric listUserComputers() vs http http://mysite.com/users/eric/computers/
SQL Server Data Services (SSDS) – announced at MIX 08 (March 2008)
“SQL Server in the cloud”
ADO.NET Data Services - part of .NET Framework 3.5 SP1(July 2008)
Not just about Cloud but all about RESTUsed by Windows AzureBeing explored by SQL Data Services
Azure Services Platform – announced at PDC 08 (October 2008)
Windows Azure “O.S. for the Cloud”SQL Services+ more
Azure Services Platform – marketing name Brings together many products from many teams
Windows Azure – Operating SystemComputeStorage
SQL ServicesSQL Data Services (SDS)
Formally SSDSSQL Labs – more on that later
.NET Services – Enterprise focusAccess ControlService BusWorkflow Service
Live Services – Consumer focusLive Mesh,...
Azure Services Platform – marketing name Brings together many products from many teams
Windows Azure – Operating SystemComputeStorage
SQL ServicesSQL Data Services (SDS)
Formally SSDSSQL Labs – more on that later
.NET Services – Enterprise focusAccess ControlService BusWorkflow Service
Live Services – Consumer focusLive Mesh,...
2233
Azure Storage SDS
Vision
Access
Relational? (today)
Relational? (tomorrow)
Analogy
Azure Storage SDS
Vision Highly scalable, highly available store in the Cloud
Access Uses ADO.NET Data Services - REST
Relational? (today)
No
Relational? (tomorrow)
No
Analogy
Azure Storage SDS
Vision Highly scalable, highly available store in the Cloud
Highly scalable, highly available relational store in the Cloud
Access Uses ADO.NET Data Services - REST
Uses custom WCF – REST or SOAP
Relational? (today)
No Yes – but with many limitations
Relational? (tomorrow)
No Yes – with less limitations
Analogy
Azure Storage SDS
Vision Highly scalable, highly available store in the Cloud
Highly scalable, highly available relational store in the Cloud
Access Uses ADO.NET Data Services - REST
Uses custom WCF – REST or SOAP
Relational? (today)
No Yes – but with many limitations
Relational? (tomorrow)
No Yes – with less limitations
Analogy File System RDBMS
MIX 09 in March is our next major conference
Expect announcements Windows Azure will go live in 2009?
Will continue to have two storage solutions
Similar to why we have file system and databases
Pre-requisitesVista or Server 2008 Visual Studio 2008 SP1 or VS Web Express Version SQL Express 2005 or 2008 (if you already have a full version of SQL Server running, you must install Express as a new instance) .NET 3.5 SP1 IIS 7 with ASP.NET and WCF HTTP activation enabled
http://www.programmerfish.com/how-to-create-and-deploy-a-simple-hello-world-application-on-windows-azure
Install the SDK SamplesDevelopment Fabric
Install the Visual Studio 2008 plug-inProject Templates
Run as AdminOptional
Get an account www.azure.com Enroll, wait, get invite(token), start deploying
Many is better than oneLoose couplingSimple stores scale
Big, reliable, expensiv
e machine
Big, reliable, expensiv
e machine
AA AA AA BB BB CC
FF GG GG GG GG GG
MM NN OO OO OO PP
DD EE EE EE
HH II JJ KK
QQ RR SS TT
EE
LL
UU
QQ QQ QQ
AA BB
CCDD
LB
public partial class _Default : System.Web.UI.Page { protected void Button1_Click(object sender,EventArgs e) { var order = txtOrder.Text; ProcessOrder(order); }
protected void ProcessOrder(string order) { //Make some coffee!
... }
}
Windows Azure Queues
LB
public partial class _Default : System.Web.UI.Page { protected void Button1_Click(object sender,EventArgs e) { var order = txtOrder.Text;
QueueStorage qStore = QueueStorage.Create(_account);
MessageQueue orderQ = qStore.GetQueue("OrderQueue");
orderQ.PutMessage(new Message(order)); }
}
public class WorkerRole : RoleEntryPoint { public override void Start() {
QueueStorage qStore = QueueStorage.Create(_account); MessageQueue orderQ = qStore.GetQueue("OrderQueue"); while (true) { Message msg = orderQ.GetMessage();
if( msg != null) ProcessOrder(msg.ContentAsString()); } } protected void ProcessOrder(string order) {
//Make some coffee! ...
}
Azure Storage (blob, table, queue)Azure Storage (blob, table, queue)
Web RoleWeb RoleLBLB
n
Worker RoleWorker Role
m
SQ
L D
ata
Serv
ices
SQ
L D
ata
Serv
ices
Storage that isDurable, Scalable, Highly Available, Secure, Performant
Rich Data AbstractionsService communication: queues, locks, …Large user data items: blobs, blocks, …Service state: tables, caches, …
Simple and Familiar Programming Interfaces
REST Accessible and ADO.NET
AccountAccount
Container
Container BlobsBlobs
TableTable EntitiesEntities
QueueQueue MessagesMessages
http://<account>.blob.core.windows.net/<container>http://<account>.blob.core.windows.net/<container>
http://<account>.table.core.windows.net/<table>http://<account>.table.core.windows.net/<table>
http://<account>.queue.core.windows.net/<queue>http://<account>.queue.core.windows.net/<queue>
Massively Scalable TablesBillions of entities (rows) and TBs of dataAutomatically scales to thousands of servers as traffic grows
Highly AvailableCan always access your data
DurableData is replicated at least 3 times
TableA Storage Account can create many tablesTable name is scoped by Account
Data is stored in TablesA Table is a set of Entities (rows)An Entity is a set of Properties (columns)
EntityTwo “key” properties that together are the unique ID of the entity in the Table
PartitionKey – enables scalabilityRowKey – uniquely identifies the entity within the partition
Partition KeyDocument Name
Row KeyVersion
Property 3Modification Time
…..
Property NDescription
Examples Doc V1.0 8/2/2007 ….. Committed version
Examples Doc V2.0.1 9/28/2007 Alice’s working version
FAQ Doc V1.0 5/2/2007 Committed version
FAQ Doc V1.0.1 7/6/2007 Alice’s working version
FAQ Doc V1.0.2 8/1/2007 Sally’s working version
Partition Partition 11
Partition Partition 22
ADO.NET Data ServicesClient
REST Interface
.NET Framework 3.5 SP1
Use any HTTP stack
Data represented as .NET objects
Data represented in Atom (XML)
DataServiceContext methods for updates
HTTP verbs for updates
LINQ to define queries URLs to define queries
Example using ADO.NET Data Services Table Entities are represented as Class Objects[DataServiceKey("PartitionKey", "RowKey")]
public class Customer{ // Partition key – Customer Last name public string PartitionKey { get; set; }
// Row Key – Customer First name public string RowKey { get; set; }
// User defined properties here public DateTime CustomerSince { get; set; }
public double Rating { get; set; }
public string Occupation { get; set; }}
Every Account has a master table called “Tables”
It is used to keep track of the tables in your accountTo use a table it has to be inserted into “Tables”[DataServiceKey("TableName")]
public class TableStorageTable{ public string TableName { get; set; }}
TableStorageTable table = new TableStorageTable("Customers");
context.AddObject("Tables", table);DataServiceResponse response = context.SaveChanges();
// serviceUri is “http://<Account>.table.core.windows.net/”DataServiceContext context = new DataServiceContext(serviceUri);
Create a new Customer and Insert into TableCustomer cust = new Customer(
“Lee”, // Partition Key = Last Name “Geddy”, // Row Key = First Name DateTime.UtcNow, // Customer Since 2.0, // Rating “Engineer” // Occupation);
context.AddObject(“Customers”, cust);DataServiceResponse response = context.SaveChanges();
// Service Uri is “http://<Account>.table.core.windows.net/”DataServiceContext context = new DataServiceContext(serviceUri);
LINQ// Service Uri is “http://<Account>.table.core.windows.net/”DataServiceContext context = new DataServiceContext(serviceUri);
var customers = from o in context.CreateQuery<Customer>(“Customers”)
where o.PartitionKey == “Lee”select o;
foreach (Customer customer in customers) { }
GET http://<Account>.table.core.windows.net/Customers? $filter= PartitionKey eq ‘Lee’
context.DeleteObject(cust); DataServiceResponse response = context.SaveChanges();
cust.Occupation = “Musician”;context.UpdateObject(cust);DataServiceResponse response = context.SaveChanges();
Customer cust = ( from c in context.CreateQuery<Customer> (“Customers”) where c.PartitionKey == “Lee” // Partition Key = Last Name && c.RowKey == “Geddy” // Row Key = First Name select c) .FirstOrDefault();
BlockBlobContainerAccount
AccountAccount
picturespictures
IMG001.JPG
IMG001.JPG
IMG002.JPG
IMG002.JPG
moviesmovies MOV1.AVIMOV1.AVI
Block 1Block 1
Block 2Block 2
Block 3Block 3
Many is better than oneLet Microsoft worry about this
Loose couplingWeb Role and Worker Role
Simple stores scaleTable, Blob, Queue
Property Type Value
Metadata
ID EntityId VWGOLF-01
Kind EntityKind
Car
FlexProps
Description
String Reliable, one owner, …
Price Numeric 12000.00
ListingDate
Datetime 01-01-2008
LocationZip
String 98052Property Type Value
Metadata
ID EntityId MINICOOPER-264
Kind EntityKind
FunCar
FlexProps
Description
String Reliable, one owner, …
Price Numeric 12000.00
ListingDate
String 1st January, 2008
LocationZip
String 98052
EngineSize Numeric 1600
Language patterned after LINQ syntaxfrom e in container
where e.Kind == “FunCar” && e[“Zip”] == 98053 && e[“Model”] == “Mini Cooper” select e
Plenty of good Feedback on the CTPSimple to useLiked the REST interface to dataNew app developers happy with “semi-relational”
BUT alsoAzure Storage and SDS too similarAzure Storage and SDS too differentWhat about existing apps that rely on relational?I want TSQL pleaseI want full relational please
http://www.microsoft.com/azure/sqllabs.mspxADO.NET Data Services access
To SQL Data Services
Data Mining in the cloud – try todayhttp://www.sqlserverdatamining.com/cloud/ Works against SDS
Reporting in the cloudWorks against SDS
Sync Enabled Cloud HubSync to Access, SQL Express, SQL Compact, ...
Just a CTP for feedback – we got plenty My recommendation
Do not learn SQL Data Services – just yet!Learn ADO.NET Data ServicesStick with Windows Azure Storage for the moment
Watch for MIX 09 Announcements
http://geekswithblogs.net/IUpdateableOrhttp://iupdateable.com
© 2008 Microsoft Ltd. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the
date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
<?xml version="1.0"?><ServiceConfiguration serviceName=“DemoService”>
<Role name="WebRole">
<Instances count="1"/> <ConfigurationSettings>
<Setting name ="LogLevel" value ="Verbose"/>
</ConfigurationSettings> </Role></ServiceConfiguration>
...
if (RoleManager.GetConfigurationSetting("LogLevel") == "Verbose") RoleManager.WriteToLog("Information", "Some log message");