persistent storage for containers using amazon efs · 2020. 8. 21. · •intro: why persistent...
Post on 30-Dec-2020
3 Views
Preview:
TRANSCRIPT
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Will Ochandarena (ochanw@)Principal Product Manager, AWS
Persistent Storage for Containers using Amazon EFS
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Agenda
• Intro: Why Persistent Storage for Containers?• General Concepts: Container, Task, & Pod Identity• Using EFS From ECS (including Fargate!)• Using EFS from EKS (using the CSI Driver)• Best Practices: Performance, Cost, & Ingest
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Intro:Why Persistent Storage for Containers?
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Many containerized applications need persistent storage
Long-running Stateful Applications
Shared Data Sets
Developer Tools
Web & Content Management
Machine Learning
Data Science Tools
WordPressDrupalnginx
JenkinsJiraGit
MXNetTensorFlow
Jupyter(hub)Airflow
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Traditional storage is not designed for modern applications
Lack of scalability
Administrative overhead
Lack of Agility
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
Highly reliable
Amazon EFSServerless File Storage
Amazon Elastic File System (Amazon EFS)
Cost optimizedCloud native
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Simplify Persistent Storage for Modern Applications
with Amazon EFS Elastic
Amazon ECS, Amazon EKS, AWS Fargate, and Amazon EFS are elastic, scale up and down rapidly based on
demand. Customers pay only for what they use.
Available and DurableAmazon ECS, Amazon EKS, AWS
Fargate, and Amazon EFS are regional services. Customers can
build applications that span multiple availability zones, with
automatic failover.
SimpleAmazon EFS configuration is inside Amazon ECS/EKS task definition, so developers can focus on their applications, not infrastructure.
SecureAccess to Amazon EFS can be
restricted based on the IAM role of the Amazon ECS task.
Amazon EFS Access Points can enforce file system permissions when multiple apps share a file
system.
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
EFS support for container services
ManagementDeployment, Scheduling, Scaling & Management of containerized applications
HostingWhere the containers run
Amazon Elastic Container Service
Amazon Elastic Container Service for Kubernetes
Amazon EC2 AWS Fargate
EFS Support Coming Soon
EFS Currently Supported
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. ”“
”“
Caltech Uses Amazon EFS to Automate File ManagementAs more and more internal customers requested new websites and other workloads, Caltech’s IT team struggled to quickly fulfill requests due to the limitations of its on-premises systems.
Caltech uses Amazon EFS and Amazon ECS to store files and run containerized applications on AWS, supporting HPC environments used by faculty and administrators.
• Centralizes file storage to support the needs of 300 internal customers
• Sets up new environments in 2 hours instead of 2 days
• Reduces the number of systems from 500 to 150
SolutionChallenge Benefits
With Amazon EFS and Amazon ECS, we’re aggregating containers across compute instances, providing the ability to quickly deploy and scale applications. That removes the capacity and scalability problems we had before, and we no longer have any limits on what we can do.
– Dan Caballero, Senior Systems Administrator, California Institute of Technology
Company: California Institute of Technology
Industry: Education
Country: United States
Employees: 300
Website: caltech.edu
About the California Institute of TechnologyBased in Pasadena, the California Institute of Technology (Caltech) is a private research university often ranked as one of the top-10 universities in the world. Founded in 1891, Caltech is one of a small group of US technology institutes primarily devoted to instruction in pure and applied sciences.
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. ”“
”“
Modernized applications to employ microservices. Deployed containers via Kubernetes and Mesos with EFS providing persistent storage and ability to dynamically scale application without storage management overhead
T-Mobile scales modern application deployments with Amazon EFS
Customer facing application with large spikes in usage based on time of day and month of year. Existing infrastructure was not able to support the scalability required without overprovision of infrastructure to support peak usage.
• 16,000 containers under management
• Reduced cost of NFS storage by 70% compared to DIY while reducing storage management overhead
• Improved cycle time for deploying application services
SolutionChallenge Benefits
We are a large organization that has lots of applications with varying requirements for availability and performance. EFS provides us with a common storage platform that meets these requirements across the board.
Amreth Chandrasehar, Principal Architect, T-Mobile
Company: T-Mobile
Industry: Mobile Communications
Country: Global
Employees: 52,000
Website: www.t-mobile.com
About T-MobileAs America's Un-carrier, T-Mobile US, Inc. is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide network delivers outstanding wireless experiences to 79.7 million customers who are unwilling to compromise on quality and value.
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Journey to (and in) the cloud
• Moved containerized data science environment to AWS for agility and cost benefits
• Enabled self-service provisioning of containerized analytics applications and compute resources
• Migrated to a managed service for better stability, application scaling and ease of operations, reducing storage management time by 90%
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. ”“
The company began using Amazon EFS for centralized file storage for its containerized data science platform. Faculty also takes advantage of AWS CloudFormation scripts to provision code.
Faculty Uses Amazon EFS to Scale Innovative Machine-Learning Platform
As Faculty’s customer base grew, the company needed a more scalable shared-file storage system to support machine-learning projects requiring up to 10 TB of storage.
• Scales to support 10 TB of storage
• Deploys platform days faster
• Gives developers more time to build innovative features
• Increases collaboration
SolutionChallenge Benefits
Company: Faculty
Industry: Software & Internet
Country: United Kingdom
Employees: 100
Website: https://faculty.ai/
About Faculty
”“ Headquartered in the United Kingdom,
Faculty is a provider of data science, machine-learning, and artificial intelligence solutions. The company’s data science platform gives data scientists the ability to use code to build machine-learning models and gain access to large data sets.
.
The sign of a great technology is that you forget it’s there. Amazon EFS just works. It requires zero maintenance on our end.
Scott Stevenson, Data Engineer, Faculty
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
General Concepts: Container, Task, & Pod Identity
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Goals for Security & Identity
1. File systems should only be mountable by the applications that need them
2. Apps that mount file systems should only have access to the data they need
Amazon EFS File System
$ cat /my_app/data
### SUCCESS THIS IS MY FILE ###
$ cat /someone_elses_app/data
cat: /someone_elses_app/data : Permission denied
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
Using IAM for File System Access
{
“Statement” : {
“Effect” : “allow”,
“Action” : “elasticfilesystem:Client*”,
“Principal” : { “AWS”: “FargateRole” }
}
}
Amazon Elastic Container Service
AWS Fargate
Task RoleTask Definition Amazon EFS
File System
AWS IAM
{
“Statement” : {
“Effect” : “allow”,
“Action” : “elasticfilesystem:Client*”,
"Resource": ”fs-deadbeef"
}
}
ECSEKS
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
Handling EFS Authorization Using IAM
Anonymous Task
Task “semitrust”
Task “fulltrust”
“Effect” : “allow”,
“Action” : “elasticfilesystem:ClientMount”,
“Principal” : “*”
“Effect” : “allow”,
“Action” : [“elasticfilesystem:ClientMount”,
“elasticfilesystem:ClientWrite”],
“Principal” : { “AWS”: “semitrust” }
“Effect” : “allow”,
“Action” : [“elasticfilesystem:ClientMount”,
“elasticfilesystem:ClientWrite”,
“elasticfilesystem:ClientRootAccess],
“Principal” : { “AWS”: “fulltrust” }
ECSEKS
Squashed to 65535
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Understanding Container Identity
ECS Task
Task Identity (IAM Role)
AWS IAM
Container Image
App IdentityUser: RootGroup: Root
$ ls –l /efs/home
drwx------ bob . BobHome
drwx------ sally . SallyHome
drwxrwx--- . biusers BI_Shared
By default, POSIX identity comes from the container image, not the task/pod runtime.
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
Application-specific Access with EFS Access Points
{
“Name”: “MyApp”,
"FileSystemId": ”fs-deadbeef",
“PosixUser”: {
“Uid”: 123
“Gid”: 123,
“SecondaryGids”: [100, 200, 300]
},
“RootDirectory”: {
“Path”: “/apps/myapp”,
“CreationInfo”: {
“OwnerUid”: 123,
“OwnerGid”: 123,
“Permissions”: “0700”
}
}
}
Creates App-specific Directory & PermissionsNo EC2 instance required!Apps only see data they need
Enforces File System IdentityRoot containers can’t escalate accessArbitrary users aren’t locked out
ECSEKS
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
{
“Name”: “MyApp”,
“PosixUser”: {
“Uid”: 123
“Gid”: 123,
“SecondaryGids”: [100, 200, 300]
},
“RootDirectory”: {
“Path”: “/apps/myapp”,
“CreationInfo”: {
“OwnerUid”: 123,
“OwnerGid”: 123,
“Permissions”: “0700”
}
}
}
How EFS Access Points Work
File System with POSIX Permissions
“Effect” : “allow”,
“Action” : “elasticfilesystem:Client*”,
“Principal” : { “AWS”: “approle” },
“Condition”* : {“accessPointArn” : “fsap-1234”
ECSEKS
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Best Practices for IAM and Access Points, & Security
• Use EFS Access Points, even if single app per file system!• Simplifies directory permission setup• Consistent experience regardless of user/group setup in container• Future-proof for adding apps to share data
•Use IAM Authorization• Use Resource Policies to restrict IAM roles to Access Points• Use Identity Policies to give single role “admin” access to file systems
•Enable Encryption @ Rest and Encryption in Motion• Simple setup, no performance penalty
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Using EFS From ECS (including Fargate!)
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
New: Amazon ECS and AWS FargateSupport for Amazon EFS
Simple: All EFS configuration is inside the ECS task definition, and connectivity is handled behind the scenes.
Serverless: AWS Fargate tasks can now leverage shared persistent storage.
Secure: Access to file systems can be authorized by IAM, and access to data controlled by EFS Access Points.
Amazon Elastic Container Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
How it works
Task
Container 1
Container 2
Amazon ECS
Amazon EC2 AWS Fargate
Amazon EFS
File system
EFSVolumeConfiguration
Amazon Elastic Container Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
Amazon Elastic Container Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
Amazon Elastic Container Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
"containerDefinitions": [{
..."mountPoints": [
{"readOnly": null,"containerPath": "/data",
"sourceVolume": "FargateDemoEFS"}
],...
"name": "FileBrowser"
}],
"taskRoleArn": "arn:aws:iam::..:role/FargateRole",...
"volumes": [
{"efsVolumeConfiguration": {
"transitEncryptionPort": null,"fileSystemId": "fs-41c7f3c1","authorizationConfig": {
"iam": "ENABLED","accessPointId": "fsap-0f7741bf379626fc2"
},"transitEncryption": "ENABLED","rootDirectory": "/"
},"name": "FargateDemoEFS",
Amazon Elastic Container Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Using EFS with EKS (using the CSI Driver)
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
EFS & EKS: Concepts
• Container Storage Interface (CSI)
• Industry standard interface for connecting storage providers (block or file) to a container.
• EFS CSI Driver
• Implementation of CSI for connecting EFS file systems to containers.
• Storage Class (SC)
• Administrator-defined class of storage that Persistent Volumes can be created from.
• Persistent Volume (PV)
• Administrator-created unit of storage that can be attached to a container. Has its own lifecycle.
• Persistent Volume Claim (PVC)
• Request to allocate an available PV from a SC to a container.
Amazon Elastic Kubernetes Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
EKS Storage – Process Flow
Storage Class (name: GeneralPurposeEFS)
Persistent Volume
Name: PV1 FS:fs-deadbeef Path: /pv1/
Persistent Volume
Name: PV2 FS:fs-deadbeef Path: /pv2/
Persistent Volume
Name: PV3 FS:fs-deadbeef Path: /pv3/
Persistent Volume
Name: PV4 FS:fs-deadbeef Path: /pv4/
Persistent Volume
Name: PV5 FS:fs-deadbeef Path: /pv5/
1. Admin Creates SC & PVs2. Dev Claims PVs from SC
Persistent Volume Claim
Name: MyAppClaim
SC: GeneralPurposeEFS
Pod
Name: MyApp
PVC: MyAppClaim
3. Dev Launches Pod Referencing PV Claim
Amazon Elastic Kubernetes Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Attaching an EFS file system to a Pod (Admin)Create Storage Class Create Persistent Volume
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
mountOptions:
- tls
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
mountOptions:
- tls
csi:
driver: efs.csi.aws.com
volumeHandle: fs-deadbeef
Amazon Elastic Kubernetes Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Attaching an EFS file system to a Pod (User)Create Persistent Volume Claim Launch Pod
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: efs-sc
resources:
requests:
storage: 5Gi
apiVersion: v1
kind: Pod
metadata:
name: efs-app
spec:
containers:
- name: web-container
image: httpd
ports:
- containerPort: 80
name: “http-server”
volumeMounts:
- name: persistent-storage
mountPath: /mnt-efs
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: efs-claim
Amazon Elastic Kubernetes Service
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Best Practices: Performance, Cost, & Ingest
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Best Practices - Performance
• Use General Purpose for most apps
• GP lower latency, now supports up to 35K read IOPS
• MaxIO for scale-out analytics/ML that need 100k+ IOPS
• Configure provisioned throughput for initial need
• As your file system grows you’ll eventually be given higher throughput
• Set up Amazon CloudWatch, monitor throughput, IOPS, and burst credits*
* https://github.com/aws-samples/amazon-efs-tutorial/tree/master/monitoring
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
When should I use EFS vs EBS?
• I need to share data between containers
• I’d like to run across instances or AZs
• I’d like to take advantage of spot pricing
• I need low latency (e.g. MySQL)
• I need point in time snapshots
Amazon Elastic Block Store
Amazon Elastic File
System
Note: Amazon FSx for Lustre can be used for containers that require ultra-high throughput and very low latency file sharing
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark
Optimize cost with Amazon EFS Infrequent Access Amazon EFS IA storage class for infrequently accessed files for $0.025/GB/mo*
* Pricing in the US East (N. Virginia) region
Automated lifecycle
management
Costsavings up
to 92%
No changes to existing applications using
Amazon EFS
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Backup for Amazon EFS
• EFS file systems can be backed up and restored usingAWS Backup
• AWS Backup provides automated backup scheduling and retention per user defined policy
• AWS Backup offers two classes of service backup storage with the ability to lifecycle to cold storage
• Restore individual files and directoriesCold storage
AWS Backup
Warm storage
Amazon EFS
Backup encryption
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Migrating NFS workloads to EFS
On-Premise
NAS FilerLinu
x Ap
plic
atio
n Se
rver
s NFS
EFS File System
AWS RegionLinux EC2 Instances
NFS
AWS DataSync
DataSyncagent
AWS Direct Connect
Internet
VPN
Virtual machine
NFS
AWS DataSync: Online transfer service that simplifies, automates, andaccelerates moving data between on-premises storage and AWS
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Where to learn more
Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate
(Parts 1-3)
By Massimo Re Ferre’
Amazon EFS: Secure data persistence with Amazon ECS and AWS Fargate
(YouTube demo)
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Thank you!
top related