Download - Hybrid architectures for database
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hybrid architectures for database backups & file migrations
S T G 3 1 3
Jeff Bartley
Solutions Architect, Hybrid Cloud Storage and Transfer
Amazon Web Services
Peter Levett
Storage Specialist Solutions Architect
Amazon Web Services
Agenda
AWS Storage Gateway overview
Scenario 1: SQL Server backup to AWS
Scenario 2: NFS migration with AWS DataSync and AWS Storage Gateway
Start working
Related sessions
STG201 - Leadership session: Storage state of the union
STG225 - Getting started with hybrid file storage using File Gateway
STG226 - Hands-on with hybrid block storage using a volume gateway
STG217 - Shift your tape backups to AWS to save time and money
STG305 - Build hybrid storage architectures with AWS Storage Gateway
STG336 - Using hybrid cloud storage to close a data center and migrate
STG208 - Backup-and-restore and disaster-recovery solutions with AWS
STG204 - Get data to AWS: How to choose and use data migration services
STG224 - Getting started with AWS DataSync
STG220 - How to move 700 TB over the wire to AWS with AWS DataSync
STG316 - Get hands-on & learn best practices for AWS data migrations
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway
Provides on-premises access to virtually unlimited cloud storage . . .
. . . regardless of cloud adoption stage
Move on-premises backups
to the cloud
Provide low-latency access
for on-premises applications
to cloud data
Shift on-premises storage to
cloud-backed file shares
AWS CloudCustomer premise
Files
(NFS/SMB)
Volumes
(iSCSI)
Tapes
(iSCSI VTL)
AWS Storage Gateway
Integrated with AWS Identity and Access Management
(IAM), AWS Key Management Service (AWS KMS),
AWS CloudTrail, Amazon CloudWatch services
Amazon S3
Glacier
Amazon Simple
Storage Service
(Amazon S3)
Amazon Elastic
Block Store
(Amazon EBS)
AWS Storage Gateway
Configuration: VMware ESXi, Microsoft Hyper-V,
Amazon Elastic Compute Cloud (Amazon EC2),
hardware appliance
AWS Backup
Amazon S3
Glacier Deep
Archive
Storage Gateway serviceStorage Gateway
HTTPS
Move on-premises backups to the cloud
iSCSI VTL
AWS Cloud
File
gateway
Volume
gateway
Tape
gateway
AWS
Storage
Gateway
managed
service
Database/
application
server
Application
server
Backup
server
iSCSI
NFS/SMB
Tape library
(Amazon S3)Tape archive
(S3 Glacier/GDA)
Amazon S3 Amazon EBSAWS Backup
HTTPS
HTTPS
HTTPS
On premises
Any S3 storage class
Lifecycle
Amazon S3
Eject
Maintain your backup workflows while reducing your backup infrastructure on-premises
File Gateway for on-premises backupMove database and file backups into the cloud and free up on-premises storage capacity
Features
NFS/SMB protocol support; mount shares directly
on database and application servers
Files stored durably in Amazon S3, lifecycle to
any S3 storage class
Local cache for accessing recent backups
Windows ACL support to control access to
backup files
Support for S3 object lock
Bandwidth optimized; only changes are
transferred
Reduce on-premises storage for backups
Easily integrates with SAP, SQL Server,
Oracle, HDFS, and other applications
Restore backups on-premises or in the
cloud on Amazon EC2 or Amazon RDS
Benefits
AWS Cloud
HTTPSFile
gateway
NFS/SMBDatabase/
application
server
Any S3 storage class
On premises
Amazon S3
Lifecycle
Shift on-premises storage to cloud-backed file sharesAccess virtually unlimited, highly durable cloud storage using common file protocols
Features Benefits
Supports NFS and SMB protocols; no application
changes required
Files stored durably in Amazon S3, lifecycle to
any S3 storage class
SMB shares integrate with Microsoft Active
Directory
Amazon CloudWatch events for automated
workflows
Reduce costs by moving storage to Amazon
S3 while still accessing from on premises
Virtually unlimited cloud storage; no more
running out of capacity
Eliminate expensive hardware refresh cycles
Files stored as native S3 objects for further
processing in AWS
AWS Cloud
HTTPSFile
gateway
NFS/SMBApplication
Any S3 storage class
On premises
Amazon S3
Lifecycle
NAS storage
Low-latency access for on-premises applications to cloud data Access files quickly from distributed locations and scale capacity as needed
Features Benefits
Generate data in the cloud or ingest from on
premises using AWS DataSync or AWS Snowball
Up to 16 TB local cache per gateway
Fully managed gateway cache provides low-
latency access to data
Refresh cache at the bucket or prefix level
Access cloud storage from any on-premises
location
Process data in the cloud and refresh
gateway cache for up-to-date results
Data stored cost-effectively and centrally in
the cloud
AWS Cloud
Application
NFS/SMB
Cache refresh
HTTPS
Cache refresh
HTTPS
Application
NFS/SMB
On-premises
File gateway
On-premises
File gatewayIn-cloud processing
AWS
DataSync
AWS
Snowball
2019 feature releases
YTD 2019 Just launched
Protocols &
services
• Tape gateway supports archiving existing and
new tapes to Amazon S3 Glacier deep archive
• AWS PrivateLink support for all gateway types
• FGW adds SMB ACLs and additional encryption
options for enhanced security
• S3 object lock support to enable WORM
workloads
Management
& automation
features
• Hardware appliance available in the EU
• Selective cache refresh to enable content
distribution
• TGW support for Amazon S3 Glacier deep archive
• Tag-on-create, tag-based access control
• High availability for VMware deployments across
all gateway types
• Improved Amazon CloudWatch logging and
metrics for all gateway types
• Gateway software update scheduling for all
gateway types
Limits• 5× read perf. increase for file gateway (to 4.8
Gbps)
• 3× perf. increase for tape gateway (to 2.7 Gbps)
Regions • ARN, BAH, HKG, GovCloud (PDT), GRU (TGW)
Learn more
aws.amazon.com/storagegateway/
• Whitepapers
• Webinars
• Demos & videos
• Developer resources
• Recorded AWS re:Invent
Sessions
• Product documentation
• Pricing
• FAQs
• Customer case studies
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workshop scenario 1: MS SQL backup to Amazon S3
AWS Cloud
Region (eu-west-1)
Availability Zone
SQL instance
Domain controller
instance
AWS Storage Gateway
instance
CacheShare
(smb)
AWS Cloud
Region (eu-central-1)
Amazon CloudWatchAWS Storage
Gateway
Amazon S3 Glacier
Amazon S3: Infrequent access
Backup
bucket
Expire/delete backup after 7 years
Lifecycle after 30 days
Amazon SQS
Bastion Amazon RDS
instance
Workshop scenario 1: MS SQL backup to Amazon S3
AWS Cloud
Region (eu-west-1)
Availability Zone
SQL instance
Domain controller
instance
AWS Storage Gateway
instance
AWS Cloud
Region (eu-central-1)
1. Deploy VPC, domain controller,
SQL Server, and Storage
Gateway (simulate on-premises
in eu-west-1)
2. Active gateway in eu-central-1
AWS Storage
Gateway
Workshop scenario 1: MS SQL backup to Amazon S3
AWS Cloud
Region (eu-west-1)
Availability Zone
SQL instance
Domain controller
instance
AWS Storage Gateway
instance
Share
(smb)
AWS Cloud
Region (eu-central-1)
Backup
bucket
1. Create S3 bucket
2. Attach gateway to AD
domain
3. Create SMB share attached
to new bucket
4. Test access to bucket from
SQL Server
Workshop scenario 1: MS SQL backup to Amazon S3
AWS Cloud
Region (eu-west-1)
Availability Zone
SQL instance
Domain controller
instance
AWS Storage Gateway
instance
CacheShare
(smb)
AWS Cloud
Region (eu-central-1)
Amazon CloudWatchAWS Storage
Gateway
Backup
bucket
Amazon SQS
1. Create SQS queue
2. Create Amazon CloudWatch rule that triggers Amazon SQS
Workshop scenario 1: MS SQL backup to Amazon S3
AWS Cloud
Region (eu-west-1)
Availability Zone
SQL instance
Domain controller
instance
AWS Storage Gateway
instance
CacheShare
(smb)
AWS Cloud
Region (eu-central-1)
Amazon CloudWatchAWS Storage
Gateway
Backup
bucket
Amazon SQS
1. Run backup process
Workshop scenario 1: MS SQL backup to Amazon S3
AWS Cloud
Region (eu-west-1)
Availability Zone
SQL instance
Domain controller
instance
AWS Storage Gateway
instance
CacheShare
(smb)
AWS Cloud
Region (eu-central-1)
Backup
bucket
Domain controller
instance
Amazon RDS
instance
1. Provision Bastion host & RDS instance
2. On-premises backup to RDS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrate on-premises NFS server data to Amazon S3, validate
access to Amazon S3 from File Gateway, and shut down the on-
premises NFS server
Workshop goal
Workshop scenario 2
Application
server
NFS
server
NFS
In-cloud Region (eu-central-1)On-premises Region (eu-west-1)
DataSync
agent
File gateway
NFS
S3 bucket
DataSync
managed service
HTTPS
TLSNFS
• Resources deployed using CloudFormation templates
• Connect to the application server, mount NFS, and verify files
Deploy resources in 2 regions and verify data on the NFS server
Workshop scenario 2: Module 1
Application
server
NFS
server
NFS
In-cloud Region (eu-central-1)On-premises Region (eu-west-1)
DataSync
agent
File gateway
S3 bucket
Activate the DataSync agent, create source and destination locations, trigger the task to copy files to S3
Copy file data to Amazon S3 using AWS DataSync
Workshop scenario 2: Module 2
Application
server
NFS
server
NFS
In-cloud Region (eu-central-1)On-premises Region (eu-west-1)
DataSync
agent
File gateway
S3 bucket
DataSync
managed service
TLSNFS
Source
location
Destination
location
Activate the file gateway, create an NFS share, mount the share
on the application server
Access the S3 bucket on premises using File Gateway
Workshop scenario 2: Module 3
Application
server
NFS
server
NFS
In-cloud Region (eu-central-1)On-premises Region (eu-west-1)
DataSync
agent
File gateway
NFS
S3 bucket
DataSync
managed service
HTTPS
TLSNFS
Rerun the DataSync task to copy remaining files from the NFS
server to the S3 bucket
Perform an incremental transfer to get remaining data
Workshop scenario 2: Module 4
Application
server
NFS
server
NFS
In-cloud Region (eu-central-1)On-premises Region (eu-west-1)
DataSync
agent
File gateway
NFS
S3 bucket
DataSync
managed service
HTTPS
TLSNFS
Unmount the NFS server on the application server, all traffic
now goes to File Gateway
Cutover to File Gateway and shut down the NFS server
Workshop scenario 2: Module 5
Application
server
NFS
server
In-cloud Region (eu-central-1)On-premises Region (eu-west-1)
DataSync
agent
File gateway
NFS
S3 bucketHTTPS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Go to https://dashboard.eventengine.run
Click on the AWS Consolebutton
Launch the AWS Console
(opens a new tab)
Click on the Readme for next steps
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Visit aws.amazon.com/training/path-storage/
Classroom offerings, like Architecting on AWS, feature AWS expert instructors and hands-on activities
45+ free digital courses cover topics related to cloud storage, including:
Learn storage with AWS Training and Certification
• Amazon S3
• AWS Storage Gateway
• Amazon S3 Glacier
• Amazon Elastic File System
(Amazon EFS)
• Amazon Elastic Block Store
(Amazon EBS)
Resources created by the experts at AWS to help you build cloud storage skills
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.