us westus east us northus south us centralus east 2 europe northeurope west asia eastasia south east...
TRANSCRIPT
Microsoft Azure StorageJai HaridasAndrew Edwards
DCIM-B384
IntroductionWhat’s New What’s ComingPatterns for Scalable & Resilient Apps
Agenda
Introduction
Microsoft Azure Storage• Cloud Storage - Anywhere and anytime
access• Blobs, Disks, Tables and Queues
• Highly Durable, Available and Massively Scalable • Easily build “internet scale” applications• More than 25 trillion stored objects• 2.5+ Million requests/sec on average
• Pay for what you use• Exposed via easy and open REST APIs,
Client Libraries and Tools
16 regions worldwide in 2014
Abstractions – Blobs and DisksBlobs – Massively scalable object store in the cloud
Simple REST interface (Put, Get, Delete)Data sharing – share documents, pictures, video, music, etc.Big Data – store raw data/logs and compute/map reduce over dataBackups – data and device backups
Disks – Network mounted durable disks for VMs in Azure
Move on-premises applications to cloudMounted disks are VHDs stored in Azure Blobs
Abstractions – Tables and QueuesTables – Massively scalable NoSQL cloud store
Key/Attribute(s) store at scaleAuto load balance partitions to meet traffic needsStore user, device or any type of metadata for your serviceOData protocol (AtomPub or JSON)
Queues – Reliable messaging system Reliable, low latency, high throughput messaging systemDecouple components/roles
Web role to worker role communicationAllows roles to scale independently
Implement scheduling of asynchronous tasksBuilding process/work flows
Additional Services, Tools and LibrariesAzure Import/Export • Move TBs of data into and out of Azure Blobs by shipping disks• Submit and monitor jobs via REST and Portal• All disks encrypted with BitLocker
Tools and Libraries• Client libraries• .NET, Java, C++, Node.js• Windows Phone & Windows Runtime• PowerShell commands• CLI tools• AzCopy – copy blobs and disks (tables later this year)• For backups, copying between accounts, and between on premise and
cloud
Geo Redundant Storage (GRS)Data geo-replicated across regions hundreds of miles apart
Provide data durability in face of potential major regional disastersProvided for Blob, Tables and Queues
User chooses primary region during account creationEach primary region has a predefined secondary region
Asynchronous geo-replicationOff critical path of live requestsUS West US East
US North US South
US Central US East 2
Europe North Europe West
Asia East Asia South East
China North China South
Japan East Japan West
South Brazil US South
Read-Only Access to GRS (RA-GRS) – Scenarios• Read-only access to secondary data even if
primary is unavailable• Access to an eventually consistent copy of the data in the other region
• For these, the application semantics need to allow for eventually consistent reads
• Recently GA’d
RA-GRS – How it Works? (1 of 2)• Customers using GRS can opt to have read-only
access to the eventually consistent copy of data on Secondary• Customer selects primary region, and the secondary region is fixed
• Get two endpoints for accessing your storage account• Primary endpoint• accountname.<service>.core.windows.net
• Secondary endpoint• accountname-secondary.<service>.core.windows.net
• Applications control which location they read data from• Use one of the above two endpoints• Our client libraries provides features to select location • Retry options: PrimaryOnly, SecondaryOnly, PrimaryThenSecondary, etc.
RA-GRS – How it Works? (2 of 2)• Same storage keys work for both
endpoints• Consistency
• All Writes go to the Primary• Reads to Primary are Strongly Consistent • Reads to Secondary are Eventually Consistent
• Applications can query the current max geo-replication delay for each service (blob, table, queue) in their storage account• Separate storage analytics metrics for
primary and secondary locations
RA-GRS – How it Works? (2 of 2)• Same storage keys work for both
endpoints• Consistency
• All Writes go to the Primary• Reads to Primary are Strongly Consistent • Reads to Secondary are Eventually Consistent
• Applications can query the current max geo-replication delay for each service (blob, table, queue) in their storage account• Separate storage analytics metrics for
primary and secondary locations
What’s new?
• Increased Scale Targets for Storage Accounts• Each storage account can hold up to 500TBs for all regions• Increased BW for US regions per storage account• 10Gbps Ingress and 20Gbps Egress
• Improved Versioning for Shared Access Signatures
• Client Libraries & Tools• .NET Library Desktop, Phone and Runtime with support for Files and Rest Version
2014-02-14• Java 1.0 RTM• Android 0.1 CTP • C++ Library CTP• AzCopy for Files CTP• PowerShell for Files CTP
• Azure Files Preview
What is New?
• Provide control to clients consuming SAS tokens (when sv=2014-02-14) to choose REST version using api-version query param
• Previously:• sv controlled the authentication, authorization and protocol version• sv is controlled by service distributing the SAS token• Clients may be a different version and not understand the protocol for version in sv
• Now:• Token Providers: Use 2014-02-14 version to create SAS tokens• Clients using storage client library: Just works• Clients building own REST protocol: Set api-version query param to intended version• Version for Authentication and authorization is still controlled by sv
Shared Access Signatures (SAS)
Use Storage Client Library 4.0 or later for generating SAS tokens
Clients using these tokens and storage client library will just work!Clients using these tokens to access storage resources but custom REST protocol implementation will need to add api-version query parameter
Use HTTPS when transmitting SAS tokensWhen resources like documents/html are accessed using SAS tokens via browsers, ensure content does not have clickable links
HTTP referrer header will expose SAS tokenUse javascript to block clicks or route it via custom application that remove SAS tokens
SAS Best Practices
Azure Files (Preview)
“I wish I could go to storage and provision a cloud drive, giving it a namespace, and that drive would then be UNC-addressable by the OSes.”
Azure Files – Customer Quotes
“I need two VM's running with a shared drive. One will write to the drive, the other will read [it].”
“Hi, I have two VM's in Microsoft Azure. All I want to do is set up a file share between them. Is this possible?”
“Is it possible to share a secondary disk between different VM instances?”
• Setup an IaaS VM to host a File Share backed by an IaaS Disk• Write code to find the IaaS File Share from the rest of the VMs in
your service.• Write some code to provide high availability • Handle host upgrades, node failures
• You can only access the File Share from other VMs
Sharing Files – The old way
IaaS VM
IaaS VM
IaaS VM
PaaS VM
IaaS VM(Sharing IaaS Disk)
Backup IaaS VMs (Mount/Share after failover)
Azure Files• Shared Network File Storage for Azure• Availability, durability, scalability are managed automatically• Supports two interfaces: SMB and REST
IaaS VM
IaaS VM
IaaS VM
PaaS VM
Azure File Share(PaaS)
• Share data across VMs and applications• Multiple writers, multiple readers using standard file system semantics.
• Share settings throughout services• VMs can read settings and files from a common, shared location.
These can be updated externally via REST.
• Dev/Test/Debug• Very useful to have a shared location for installing applications, setting
up VMs, running tools, and keeping notes while developing, testing, and debugging cloud services.
Azure Files - Scenarios
Azure Files - SMB 2.1 Protocol• Enables moving on-premises applications that
rely on shared file storage to Azure • Azure VMs can “net use” to a share
• Natively supported by OS APIs, libraries, and tools• Windows (CreateFile, ReadFile, WriteFile, …)• CRTs (fopen, fread, fwrite, …)• .Net (FileStream.Read, FileStream.Write, …)• Many more
• Supports standard file system semantics• Move and rename files and directories• Read-only, write through, overlapped• Change notifications
• Allows internet access to the same shared file system• Build hybrid applications (on premises +
cloud)• Supports a variety of common APIs:• Create/Delete Files and Directories• Write/Read Files• Get File and Directory properties• List Files
Azure Files - File REST APIs
Demo
Azure Files Part 1
Azure Files
Azure Files
Azure Files
Azure Storage Architecture
Massive Scale Out & Auto Load Balancing Index Layer
Distributed Replication Layer
Blob/DiskHead
QueueHead
TableHead
REST REST REST REST
File ShareHead
SMB
“Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011
Azure Files vs BlobsDescription Azure Blobs Azure Files
Durability Options
LRS, ZRS, GRS (and RA-GRS for higher availability)
LRS, GRS
Accessibility REST APIsSMB 2.1 (standard file system APIs)REST APIs
Connectivity REST – WorldwideSMB 2.1 - Within regionREST – Worldwide
Endpoints http://myaccount.blob.core.windows.net/mycontainer/myblob
\\myaccount.file.core.windows.net\myshare\myfile.txt
http://myaccount.file.core.windows.net/myshare/myfile.txt
DirectoriesFlat namespace however prefix listing can simulate virtual directories
True directory objects
Case Sensitivity of Names
Case sensitive Case insensitive, but case preserving
Capacity Up to 500TB containers 5TB file shares
Throughput Up to 60 MB/s per blob Up to 60 MB/s per share
Object size Up to 1 TB/blob Up to 1 TB/file
Billed capacity Based on bytes written Based on file size
Azure Files vs DisksDescription Disk Azure Files
Relationship with Azure VMs
Required for booting (OS Disk)
Scope Exclusive/Isolated to a single VM Shared access across multiple VMs
Snapshots and Copy Yes No
ConfigurationConfigured via portal/Management APIs and available at boot time
Connect after boot (via net use on windows)
Built-in authentication Built-in authentication Set up authentication on net use
Cleanup Resources can be cleaned up with VM if neededManually via standard file APIs or REST APIs
Access via RESTCan only access as fixed formatted VHD (single blob) via REST. Files stored in VHD cannot be accessed via REST.
Individual files stored in share are accessible via REST
Max Size 1TB Disk5TB File Share
1TB file within share
Max 8KB IOps 500 IOps 1000 IOps
Throughput Up to 60 MB/s per Disk Up to 60 MB/s per File Share
• Windows Supported:• Windows Server 2008 R2• Windows Server 2012• Windows Server 2012 R2
• Investigating Linux Support:• Ubuntu 13.10• Ubuntu 14.04 LTS
Azure Files – Client OS Support
• Request a token• Tokens will start to be granted in batches by end of May 2014
• Redeem token• Create new storage account• Create share (using powershell)• Put files into share (azcopy)• Connect to share from VM
Azure Files: Getting Started
Demo
Azure Files Part 2
Website Served From Azure File Share
Load Balancer
Azure VM
Azure VM
Azure File Share
…
REST APIs
SMB 2.1
Azure Files
Azure Files
What’s coming?
• Client Libraries• Node.js Library support for 2014-02-14 CTP• iOS Library CTP (By end of CY ‘14)• JavaScript Library CTP (By end of CY 14)
• Additional Durability Offering - Zone Redundant Storage for Block Blobs
What is Coming?
• LRS• Stores 3 replicas of the data within a single zone (facility) in a single region• Provides data durability for disk, node and rack failures
• ZRS• Available only for block blobs• Stores 3 replicas of the data across multiple zones (facilities). Designed to keep all 3
replicas across zones within a single region, but may span across two regions. • Provides additional durability to protect data against zone failures (e.g., fire in a
facility)
• GRS• Stores 6 replicas of the data across two regions (3 in each region)• Provides additional durability to protect data against major regional disasters (e.g.,
tornado, hurricane, earthquake, etc.)
3 Types of Durability offered for Azure Storage
Microsoft Azure Storage Pricing
Patterns for Scalable & Resilient Applications
• Scenario: Encode images that are being uploaded• Add a message to the queue with blob Uri
and maximum timeout to wait for blob upload• Upload the blob• Worker role processes messages• If blob not present, wait until the conservative max timeout• If blob exists, • Encode image and store encoded image• Delete the original image• Delete the message
Pattern for pre-processing resources
• Scenario: OneDrive needs to expand beyond a single account
• Create only as many accounts as needed today• Keep a map of user name+ bucket id to a storage
account name• When a bucket id fills up or account reaches limits
(capacity/throughput), create a new bucket id and pick a storage account from pool for storing data
Pattern – Scale beyond single storage account
User Name
Bucket Id
Storage Account
Storage Resource User Resource
brunopitman
1 jaidemo jaidemo/container/brunopitman/photo1.jpg
brunopitman/1/photo1.jpg
selmaramsey
1 jaidemo1 jaidemo1/container/selmaramsey/video1.wmv
selmaramsey/1/video1.wmv
brunopitman
2 jaidemo1 jaidemo1/container/brunopitman/backup.dat
brunopitman/2/backup.dat
selmaramsey
2 jaidemo22
jaidemo22/container/selmaramsey/resume.doc
selmaramsey/1/resume.doc
User Name
Bucket Id
Storage Account
Storage Resource User Resource
brunopitman
1 jaidemo jaidemo/container/brunopitman/photo1.jpg
brunopitman/1/photo1.jpg
selmaramsey
1 jaidemo1 jaidemo1/container/selmaramsey/video1.wmv
selmaramsey/1/video1.wmv
• Scenario: Table Queries for Tracking Game States• Web service maintains cache for game
state • On cache miss, retrieves state from Azure
Table• On failure, it fails fast and expects clients to
retry but asynchronously in the background retries • Retry on secondary and if secondary lag is
within X minutes use secondary data
Pattern for higher read availability & consistent low latency web requests
Demo
Read From Secondary
• Azure Storage• Durable, Scalable and highly Available Cloud Storage• Auto load balances to meet scale needs
• Azure Files – File shares in the cloud• Storage Durability Options – LRS, ZRS, and GRS• RA-GRS • Provides Higher Availability as applications can read from secondary when primary is
not available. • Client Library retries provides this capability out of the box
• Details on Internals can be found in the SOSP paper:• “Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consis
tency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011
Summary
Come Visit Us in the Microsoft Solutions Experience!
Look for Datacenter and Infrastructure ManagementTechExpo Level 1 Hall CD
For More InformationWindows Server 2012 R2http://technet.microsoft.com/en-US/evalcenter/dn205286
Windows Server
Microsoft Azure
Microsoft Azurehttp://azure.microsoft.com/en-us/
System Center
System Center 2012 R2http://technet.microsoft.com/en-US/evalcenter/dn205295
Azure PackAzure Packhttp://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack
Resources
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
msdn
Resources for Developers
http://microsoft.com/msdn
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Meet Storage Engineers
Storage engineering team will be available in the Azure booth area for the duration of Tech Expo (look for us under “Developer Platform & Tools”)You can also meet us at the “Ask The Experts/Meet and Geek” on Tuesday from 6:30-8:30pmWe look forward to meeting you!
Discuss your experience using Azure Storage
For customers already using Azure Storage in production workloads, we’d love to get your feedback at this Storage session:Date: Tuesday, May 13, 2014Time: 4p-5:30pLocation: Hilton Americas – Houston (connected to the convention center)Room # 3271600 Lamar Street, Houston
Complete an evaluation and enter to win!
Evaluate this session
Scan this QR code to evaluate this session.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.