azure data storage
TRANSCRIPT
![Page 1: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/1.jpg)
Azure Data StorageKen Cenerelli@KenCenerelli
Microsoft Azure
![Page 2: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/2.jpg)
Agenda
Blobs Files Queues Tables
![Page 3: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/3.jpg)
Agenda
SQL
Database
SQL Server
VMDocumentDB Search HDInsight
![Page 4: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/4.jpg)
Microsoft Azure
Storage Blob
![Page 5: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/5.jpg)
Two Types of Blobs Under the Hood
![Page 6: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/6.jpg)
Two Types of Blobs Under the Hood
Block Blob:
![Page 7: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/7.jpg)
Two Types of Blobs Under the Hood
Page Blob:
![Page 8: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/8.jpg)
Blob Storage Concepts
![Page 9: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/9.jpg)
Blob Details –
![Page 10: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/10.jpg)
Shared Access Signatures
![Page 11: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/11.jpg)
Policy based:
Stored Access PolicyUse for longer dated revocable permission sets
Points to a Container level policy
Can only revoke by deleting the blob or waiting for expiry
Use case: Providing revocable permissions to certain users/groups
Shared Access Signatures – Two broad approaches
Ad-hoc:
Shared Access SignatureUse for very short dated single use scenarios
Can only revoke by deleting the blob or waiting for expiry
Use case: Single use URLs
E.g. Provide URL to mobile client to upload to container
Always endeavour to use least permission set possible
![Page 12: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/12.jpg)
Microsoft Azure
Storage Files
![Page 13: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/13.jpg)
Azure Files
Shared Network File Storage for Azure
Availability, durability, scalability are managed automatically
Supports two interfaces: SMB and REST
![Page 14: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/14.jpg)
Azure Files – Usage
Share data across VMs and applications
Share settings throughout services
Share tools amongst developers in a group
Dev/Test/Debug
![Page 15: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/15.jpg)
Azure Files vs Blobs
Description Azure Blobs Azure Files
Durability
Options
LRS, ZRS, GRS (and RA-GRS for higher
availability)LRS, GRS
Accessibility REST APIsSMB 2.1 (standard file system APIs)
REST APIs
Connectivity REST – WorldwideSMB 2.1 - Within region
REST – Worldwide
Endpointshttp://myaccount.blob.core.windows.net/
mycontainer/myblob
\\myaccount.file.core.windows.net\myshare\myfile.txt
http://myaccount.file.core.windows.net/myshare/myfile.txt
DirectoriesFlat namespace however prefix listing
can simulate virtual directoriesTrue directory objects
Case Sensitivity of Names Case sensitive Case insensitive, but case preserving
Capacity Up to 500TB containers 5TB file shares
Throughput Up to 60 MB/s per blob Up to 60 MB/s per share
Object size Up to 1 TB/blob Up to 1 TB/file
Billed capacity Based on bytes written Based on file size
![Page 16: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/16.jpg)
Microsoft Azure
Storage Queue
![Page 17: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/17.jpg)
Queue length reflects how well the backend processing nodes are doing.
Decouples the application and makes scaling easier.
Flexibility of efficient resource usage within an application.
Absorb traffic bursts and reduce the impact of individual component failures.
Why use a Queue?
![Page 18: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/18.jpg)
Storage Account: All access to Azure Storage is done through
a storage account. It can store millions of messages until the
storage account’s maximum capacity of 500 TB is reached.
Queue: A queue contains a set of messages.
Message: A message, in any format, of up to 64KB.
Queue Components
![Page 19: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/19.jpg)
Queues are addressable using the following URL format:
http://{storage-account}.queue.core.windows.net/{queue}
Queue URL format
![Page 20: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/20.jpg)
Faster Web Applications with queues
using asynchronous workloads
Demo
![Page 21: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/21.jpg)
Messages are ordered but not guaranteed FIFO.
Message will be processed at least once.
Message may be processed more than once.
.DequeueCount increases every time.
Messages are stored up to 7 days.
Queue Considerations
![Page 22: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/22.jpg)
Microsoft Azure
Storage Table
![Page 23: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/23.jpg)
Table Storage Concepts
![Page 24: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/24.jpg)
Table Storage Details
![Page 25: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/25.jpg)
Table Storage Details
![Page 26: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/26.jpg)
Table Storage Details – Entity Properties
![Page 27: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/27.jpg)
Table Storage Details
![Page 28: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/28.jpg)
Table Storage Details – Purpose of the PartitionKey
![Page 29: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/29.jpg)
Microsoft Azure SQL Database
![Page 30: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/30.jpg)
From private to public Cloud
Microsoft Relational Storage Options
A Continuous offering
![Page 31: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/31.jpg)
A Server is not a machine
![Page 32: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/32.jpg)
SQL Server database technology “as a Service”
Fully Managed
Enterprise-ready with automatic support for HA, DR, Backups, replication and more
SQL Database – The Basics
![Page 33: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/33.jpg)
Scale out with ElasticScale
Built-in regional database replicas for additional protection
Uptime SLA of 99.99%
SQL Database – The Basics
![Page 34: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/34.jpg)
The Service head contains databases
Connect via automatically generated FQDN: {name}.database.windows.net
Initially contains only a master database
SQL Database – Server Definition
![Page 35: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/35.jpg)
Auto backups, transactional logs every 5 min
Backups in Azure Storage and geo-replicated
Creates a side-by-side copy, non-disruptive
Backups retention policy: 7, 14 or 35 days
Automated export of logical backups for long-term backup protection
Point-in-time restore - “oops recovery”
![Page 36: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/36.jpg)
Available in all tiers: Basic, Standard and Premium
Built on geo-redundant Azure Storage
Recover to any Azure region
Geo-restore – Emergency data recovery when you need it most
![Page 37: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/37.jpg)
Focus on logical vs. physical administration
Database and log files automatically placed
Three high-availability replicas maintained for every database
Tables require a clustered index
Maximum database size is 500 GB
Azure SQL Database vs. SQL Server
![Page 38: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/38.jpg)
Transact-SQL
Use Familiar Technologies
(obviously)
![Page 39: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/39.jpg)
.NET Framework (C#, Visual Basic, F#): ADO.NET
C / C++: ODBC
Java: Microsoft JDBC provider
PHP: Microsoft PHP provider
Use Familiar Technologies - Languages
![Page 40: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/40.jpg)
OData
Entity Framework
WCF Data Services
NHibernate (etc.)
Use Familiar Technologies - Frameworks
![Page 41: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/41.jpg)
SQL Server Management Studio (>=2008 R2)
SQL Server command-line utilities (SQLCMD, BCP)
Visual Studio IDE for database development
Use Familiar Technologies - Tools
![Page 42: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/42.jpg)
Use command, distributed transactions, distributed views
Service Broker
Common Language Runtime (CLR)
SQL Agent
SQL Profiler
Native Encryption
Unsupported SQL Server Features
![Page 43: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/43.jpg)
Elastic Scale
![Page 44: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/44.jpg)
Classic 3-tier enterprise architecture:
Scale out the front ends to multiple instances is easy
Scale the data-tier is more challenging
Web
Role
L
SQL
Worker
Role
Canonical cloud app architecture
![Page 45: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/45.jpg)
SQL Database Considerations and Best Practices
Elastic Scale across thousands of databases via custom sharding
Scale out via .NET Client libraries consumed by customer applications to support sharded database pattern
Enables developer and manageability functions
Supports split, merge, and move operations on data
![Page 46: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/46.jpg)
Vertical: Scale-up or scale-down
Horizontal: Scale-out or scale-in
Basic
Standard
Premium
Basic Basic Basic Basic Basic Basic
Premium
Standard
Scale out/in
Sca
le u
p/d
ow
n
Scalability options in Azure SQL DB
![Page 47: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/47.jpg)
Web
Role
J
SQL
Worker
Role
J
J
Elastic Scale architecture
![Page 48: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/48.jpg)
SQL Server VM
![Page 49: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/49.jpg)
Run SQL on VM
Run any SQL product on cloud VM
Support for SQL Server, Oracle, MySQL
Ready to go VM images available in the Gallery
Persistent storage using attached disk in blob storage
Has all the benefits and powers of VMs combined with the full features of a SQL Engine
![Page 50: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/50.jpg)
SQL Database SQL IaaS
Why (at least) two offerings of SQL in Azure?
![Page 51: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/51.jpg)
SQL Database SQL IaaS
![Page 52: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/52.jpg)
Other features SQL IaaS supports that SQL Database doesn’t (yet)
![Page 53: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/53.jpg)
Microsoft Azure DocumentDB
![Page 54: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/54.jpg)
Fully managed, scalable JSON document database service
![Page 55: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/55.jpg)
Microsoft Azure Data Services
fully managed, scalable, queryable, schemafree JSON
document database service for modern applications
transactional processing
rich query
managed as a service
elastic scale
internet accessible http/rest
schema-free data model
arbitrary data formats
![Page 56: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/56.jpg)
DocumentDB – Reasons to use
![Page 57: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/57.jpg)
Ways to work with data
RESTful access methodsFor Create/Read/Update/Delete (CRUD) operations
DocumentDB SQLA query language with SQL-derived syntax
Executing logic in the database
Stored procedures
Triggers
User-defined functions (UDFs)
- Allow extending DocumentDB SQL
All written in JavaScript
![Page 58: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/58.jpg)
Azure Search
![Page 59: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/59.jpg)
Azure Search
Embed a sophisticated search experience into web and mobile applications without having to worry about the complexities of full-text search and without having to deploy, maintain or manage any infrastructure.
![Page 60: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/60.jpg)
Azure Search
Perfect for enterprise cloud developers, cloud software vendors, cloud architects who need a fully-managed search solution.
![Page 61: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/61.jpg)
Simple HTTP/JSON API for creating indexes, pushing documents, searching
Keyword search with user-friendly operators (+, -, *, “”, etc.)
Hit highlighting
Faceting (histograms over ranges, typically used in catalog browsing)
Search Functionality
![Page 62: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/62.jpg)
Suggestions (auto-complete)
Rich structured queries (filter, select, sort) that combines with search
Scoring profiles to model search result relevance
Geo-spatial support integrated in filtering, sorting and ranking (such as finding all restaurants within 5 KM of your current location)
Search Functionality
![Page 63: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/63.jpg)
What are customers doing with Azure Search?
high
high heelshigh topshigh arch
Azure Search doesn’t
provide any UI components
Users expect suggestions, so
Azure Search supports this
![Page 64: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/64.jpg)
HDInsight
![Page 65: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/65.jpg)
Data size
Access
Updates
Structure
Integrity
Scaling
Relational DB vs. Hadoop
![Page 66: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/66.jpg)
![Page 67: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/67.jpg)
Distributed Storage
(HDFS)
Query
(Hive)
Distributed Processing
(MapReduce)
OD
BC
Legend■ Core Hadoop
■ Data processing
■ Data Movement
■ Packages
Hadoop ecosystem
![Page 68: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/68.jpg)
Hadoop Core +
Hive, Pig, HBase
C#, F#, .NET
Azure Storage (WASB)
Office 365 Power BI
(Excel, PowerQuery,
PowerView,
BI Sites)
World's Data (Azure Data
Marketplace)
HDInsight and Hadoop
ODBCSqoop for SQL
ServerPowerShell
![Page 69: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/69.jpg)
Microsoft Azure
The Big Picture
![Page 70: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/70.jpg)
The Big Picture
The traditional world:
![Page 71: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/71.jpg)
The Big Picture
A modern view:
![Page 72: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/72.jpg)
The Big Picture
Microsoft Azure technologies:
![Page 73: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/73.jpg)
The Big Picture
Why Use a NoSQL Technology on Azure?
![Page 74: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/74.jpg)
The Big Picture
Choosing a Data Technology
![Page 75: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/75.jpg)
Resources
![Page 76: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/76.jpg)
Resources
Microsoft Azure Essentials: Fundamentals of Azure
Michael S. Collier and Robin E. Shahan
February 2015
246 pages
http://bit.ly/1zRa1UW
![Page 77: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/77.jpg)
Resources
• Azure Data Services series by Adarsha Datta
• http://bit.ly/1aTDSBA
• Get Ready for Global Azure Bootcamp
• http://bit.ly/1BkD0Lf
• Azure Documentation Center
• http://azure.microsoft.com/en-us/documentation/
• http://www.hadooponazure.com/
![Page 78: Azure Data Storage](https://reader030.vdocuments.us/reader030/viewer/2022032422/55a9e4a21a28abfb728b46fa/html5/thumbnails/78.jpg)