aws storage services for hybrid cloud - files.meetup.comfiles.meetup.com/19647895/aws storage...
TRANSCRIPT
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alex Tomic, Solutions Architect
March 23, 2016
AWS Storage Services
for Hybrid Cloud
Why AWS storage?
Introduction: Why choose AWS for storage
Compelling
Economics Easy to Use Reduce risk
Speed,
Agility, Scale
Pay as you go
No upfront investment
No commitment
No risky capacity
planning
No need to provision
for redundancy
or overhead
Self service
administration
SDKs for simple
integration
Durable and Secure
Avoid risks of physical
media handling
Reduce time to market
Focus on your
business, not your
infrastructure
AWS Global Infrastructure
12 Regions
33 Availability Zones
54 Edge locations
Control your geographic locality
for performance and compliance
Storage Choices in AWS
Amazon S3
Durable object storage
Amazon EBS
Block storage for use
with Amazon EC2
Amazon Glacier
Archival storage
Amazon EFS
File storage for use with
Amazon EC2
Amazon S3-IA
Infrequently accessed
data store
Block
Object
Amazon S3 and S3-Infrequent Access
Highly durable object storage for all types of data
Internet-scale storage
Grow without limits
Benefit from AWS’s
massive security
investments
Built-in redundancy
Designed for
99.999999999%
durability
Low price per GB
per month
No commitment
No up-front cost
Amazon Glacier
Archival storage for infrequently accessed data
Amazon Glacier
is optimized for
infrequent retrieval
Stop managing
physical media
Even lower cost than
Amazon S3;
Same high durability
Selecting the right Object Storage
S3
S3-IA
Glacier
L
i
f
e
c
y
c
l
e
Available S3: 99.99%
S3-IA: 99.9%
Performant Low Latency
High Throughput ≥ 30 Days ≥ 128K
≥ 90 Days
Durable 99.999999999%
Scalable Elastic capacity
No preset limits
> 0K $0.007/GB per month
$0.0125/GB per month
“Hot” Data Active and/or
Temporary Data
“Warm” Data Infrequently
Accessed Data
“Cold” Data Archive and
Compliance Data
≥ 0 Days > 0K $0.03/GB per month
3 – 5 Hrs
$0.01/GB retrieval
$0.01/GB retrieval < 5%
Why hybrid approach?
Why Hybrid IT
Leverage existing investments
Increase agility
Flatten learning curve
A
$
The Good News is that Cloud isn’t an ‘All or Nothing’
Choice
Corporate
Data Centers
On-Premises
Resources
Cloud
Resources Integration
Hybrid infrastructure services
AWS
Import/Export
Amazon Virtual
Private Cloud AWS Direct
Connect
Virtual Private
Network
Directory
Services
Identity & Access
Management CloudTrail Key Management
Service
Amazon
Route 53
VPC
peering
Hybrid architecture:
how do we connect to AWS?
VPC subnet
Availability Zone
Security group
VPC subnet
Availability Zone
Security group
Virtual Gateway
AWS Virtual Private Network (IPSec VPN)
o IPSec hardware VPN connection
Supported VPN appliances
o Encryption and Validation
o Private RFC 1918 Addressing
o Uses Border Gateway Protocol
(BGP) for routing and fail-over
o VPN Service provides managed
redundant end-points
Corporate data center
Users
Data center router
Servers
Internet
IPSec VPN
AWS Direct Connect
o Requires Layer 2 single mode fiber
1000BASE-LX or 10GBASE-LR
o Requires 802.1Q VLANs across
connection.
Tagging of IP traffic
o Routing uses BGP A/A or A/P
multipath.
o Each DX is mapped to a single
AWS Region
Corporate data center
Users
VPC subnet
Availability Zone
Security group
VPC subnet
Availability Zone
Security group
Data center router
Customer router
Servers
AWS Direct Connect location
AWS Direct Connect routers
Virtual Gateway
Hybrid architecture:
how to integrate authentication?
AWS Directory Service
Features
AWS Directory Service for Microsoft
Active Directory (Enterprise Edition)
Avoid complexity and cost of hosting
SAML-based federation infrastructure
Establish trust between on-premises AD
and AWS Directory Service
Requires IPSec VPN or Direct Connect connectivity
AWS Directory Service Connect
Corporate data center
Users
AD.Domain
Servers
Domain controller
VPC subnet
Availability Zone
Security group
Virtual Gateway
VPC subnet
Availability Zone
Security group
Active Directory DCs in your VPC
o Reduced back-reach Traffic
o Reduced Latency for Authentication
o Additional Resiliency
o Enablement of both:
Multi-Master Read/Write Domain Controllers
Read-only Domain Controllers (RODCs)
o Requires IPSec VPN or Direct Connect connectivity
Active Directory Replication
Corporate data center
Users
AD.Domain
Servers
Domain controller
Domain controller
VPC subnet
Availability Zone
Security group
Virtual Gateway
Domain controller
VPC subnet
Availability Zone
Security group Type Port Number
TCP 54, 88, 135, 137, 139, 389, 445, 464, 636, 3268, 3269, 5722, 49152-65535
UDP 53,67,123, 138, 389, 445, 464, 2535, 5355, 49152-65535
Replication
Enterprise Federation
Integrate identity management with AWS • Secure access to AWS resources using your IDM
• Provide SSO to AWS Management Console or API’s
• Build your own SSO federation using AWS STS service, or
• Federate with on-premise directories like Active Directory,
TFIM, OAM or another SAML 2.0 compliant IdP
Hybrid architecture: how to move data to AWS?
v
Import/Export Snowball
E-ink shipping label
Ruggedized
case
“8.5G Impact”
All data encrypted
end-to-end
Rain & dust
resistant
Tamper-resistant
case & electronics
50 TB
10Gb network
What about complex enterprise scenarios?
v
Amazon Storage Partner Ecosystem
v
Backup and archiving o Backup gateways integrated with
Amazon S3 o De-duplication
o Compression
o WAN Acceleration
o Leverage Amazon S3 archival to Amazon Glacier
Corporate data center
Amazon S3
Amazon S3-IA
Application server
Virtual server
File server
Database server
Backup system
Cloud Gateway
NetApp AltaVault cloud-integrated
storage appliance
AWS Marketplace Partners
Amazon Glacier
v
Storage Expansion
o Virtual volumes presented to local network iSCSI,
NFS and CIFS volumes
o Local disk cache to provide fast on-premise access
o Gateway side encryption for security
Corporate data center
Application server
Virtual server
File server
Storage appliance
Cloud Gateway
NetApp AltaVault cloud-
integrated storage appliance
AWS Marketplace Partners
Amazon S3
Amazon S3-IA
IAS
Amazon Glacier
v
Storage Replication
o Synchronize data with AWS EBS storage
o Perform byte-level replication by volume
o Virtual appliance in VPC for added networking
security and connectivity
Corporate data center
EC2 instance with EBS volumes
Application server
Virtual server
File server
Storage appliance
NetApp Cloud ONTAP
clustered storage operating system
AWS Marketplace Partners
EC2 Appl-iance
How do you use data in AWS?
ENTERPRISE APPS
DEVELOPMENT & OPERATIONS MOBILE SERVICES APP SERVICES ANALYTICS
Data Warehousing
Hadoop/Spark
Streaming Data Collection
Machine Learning
Elastic Search
Virtual Desktops
Sharing & Collaboration
Corporate Email
Backup
Queuing & Notifications
Workflow
Search
Transcoding
One-click App Deployment
Identity
Sync
Single Integrated Console
Push Notifications
DevOps Resource Management
Application Lifecycle Management
Containers
Triggers
Resource Templates
TECHNICAL & BUSINESS SUPPORT
Account Management
Support
Professional Services
Training & Certification
Security & Pricing Reports
Partner Ecosystem
Solutions Architects
MARKETPLACE
Business Apps
Business Intelligence
Databases DevOps Tools
Networking Security Storage
Regions Availability Zones
Points of Presence
INFRASTRUCTURE
CORE SERVICES
Compute VMs, Auto-scaling, & Load Balancing
Storage Object, Blocks, Archival, Import/Export
Databases Relational, NoSQL, Caching, Migration
Networking VPC, DX, DNS
CDN
Access Control Identity Management
Key Management & Storage
Monitoring & Logs
Assessment and reporting
Resource & Usage Auditing
SECURITY & COMPLIANCE
Configuration Compliance
Web application firewall
HYBRID ARCHITECTURE
Data Backups
Integrated App Deployments
Direct Connect
Identity Federation
Integrated Resource Management
Integrated Networking
API Gateway
IoT
Rules Engine
Device Shadows
Device SDKs
Registry
Device Gateway
Streaming Data Analysis
Business Intelligence
Mobile Analytics
Case Study: Big Data Analytics
What Nasdaq needed • Replacement of on-premises legacy warehouse • Reduction of cost and increase in data capacity
Why they chose AWS (specifically Amazon Redshift) • Fulfillment of security and regulatory requirements • Cost efficiencies without sacrificing functionalities
Benefits realized • System that moves an average of 5.5 billion rows into
Amazon Redshift every day (with 14 billion on a peak day in Oct of 2014)
• Ability to increase accessibility of historic data to a growing number of internal groups
“The Nasdaq Group has been a user of Amazon Redshift since it was released and we are extremely happy with it…. Currently, our system is moving an average of 5.5 billion rows into Amazon Redshift every day.”
- Nate Simmons, Principal Architect
“As a new business within Broadridge, it’s important that we are able to reduce upfront costs while ensuring peak scalability. AWS’s ability to scale infinitely and provide a robust suite of services and capabilities meets our needs perfectly.”
- Robert Krugman, VP of Digital Strategy
What Broadridge needed • Technology platform to deliver investment industry content
to millions of consumers
Why they chose AWS • Ability to meet scale – documents from hundreds of
providers to millions of consumers • Fulfilment of security requirements to protect sensitive
information like monthly statements, trade confirmations, tax documents, and regulatory disclosures
Benefits realized • Inlet, the resulting platform, is delivering content through
Broadridge’s relationships with Financial Services companies
Case Study: Digital Innovation
“Using AWS helps us reduce a 10-day process to 10 minutes. That’s trans-formative: it broadens our ability to discover.”
- Peter Phillips, Managing Director
What Aon needed • Perform actuarial calculations with greater computing power • Information delivery within shorter time frames and less cost
Why they chose AWS • Ability to spin up large numbers of Graphical Processing Units
(or GPUs) quickly and inexpensively • Quick delivery of an entire environment and functionality
Benefits realized • By processing on AWS, recalculating policies takes minutes
rather than hours or days • Ability to deliver client solutions more quickly, with richer risk
assessments • Cost savings that are passed to the customer
Case Study: High Performance Computing (HPC)
“We are growing rapidly, and our capacity, availability, and resiliency requirements are constantly changing. We needed to be able to scale our infrastructure, but that was becoming difficult due to physical limitations and slow response times from our datacenters.”
- Andy Montgomery, Head of Division for IT Operations
and Solution Design
What St. James’s Place needed • Better infrastructure that could keep up with growth • Overcome siloes due to new and legacy systems Why they chose AWS • New functionalities including data warehousing and
electronic business processing system • Ability to move legacy systems and upgrade in the cloud
Benefits realized • Better management of growth, including ~50% increase in
client and associate traffic annually • Scalability to manage demand fluctuations • Faster recovery model
Case Study: Re-architecting for Growth