persistent storage for containers using amazon efs · 2020. 8. 21. · •intro: why persistent...

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Will Ochandarena (ochanw@)Principal Product Manager, AWS

Persistent Storage for Containers using Amazon EFS

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.

Agenda

• Intro: Why Persistent Storage for Containers?• General Concepts: Container, Task, & Pod Identity• Using EFS From ECS (including Fargate!)• Using EFS from EKS (using the CSI Driver)• Best Practices: Performance, Cost, & Ingest


Intro:Why Persistent Storage for Containers?


Many containerized applications need persistent storage

Long-running Stateful Applications

Shared Data Sets

Developer Tools

Web & Content Management

Machine Learning

Data Science Tools

WordPressDrupalnginx

JenkinsJiraGit

MXNetTensorFlow

Jupyter(hub)Airflow


Traditional storage is not designed for modern applications

Lack of scalability

Administrative overhead

Lack of Agility

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.©2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or Trademark

Highly reliable

Amazon EFSServerless File Storage

Amazon Elastic File System (Amazon EFS)

Cost optimizedCloud native


Simplify Persistent Storage for Modern Applications

with Amazon EFS Elastic

Amazon ECS, Amazon EKS, AWS Fargate, and Amazon EFS are elastic, scale up and down rapidly based on

demand. Customers pay only for what they use.

Available and DurableAmazon ECS, Amazon EKS, AWS

Fargate, and Amazon EFS are regional services. Customers can

build applications that span multiple availability zones, with

automatic failover.

SimpleAmazon EFS configuration is inside Amazon ECS/EKS task definition, so developers can focus on their applications, not infrastructure.

SecureAccess to Amazon EFS can be

restricted based on the IAM role of the Amazon ECS task.

Amazon EFS Access Points can enforce file system permissions when multiple apps share a file

system.


EFS support for container services

ManagementDeployment, Scheduling, Scaling & Management of containerized applications

HostingWhere the containers run

Amazon Elastic Container Service

Amazon Elastic Container Service for Kubernetes

Amazon EC2 AWS Fargate

EFS Support Coming Soon

EFS Currently Supported

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. ”“

”“

Caltech Uses Amazon EFS to Automate File ManagementAs more and more internal customers requested new websites and other workloads, Caltech’s IT team struggled to quickly fulfill requests due to the limitations of its on-premises systems.

Caltech uses Amazon EFS and Amazon ECS to store files and run containerized applications on AWS, supporting HPC environments used by faculty and administrators.

• Centralizes file storage to support the needs of 300 internal customers

• Sets up new environments in 2 hours instead of 2 days

• Reduces the number of systems from 500 to 150

SolutionChallenge Benefits

With Amazon EFS and Amazon ECS, we’re aggregating containers across compute instances, providing the ability to quickly deploy and scale applications. That removes the capacity and scalability problems we had before, and we no longer have any limits on what we can do.

– Dan Caballero, Senior Systems Administrator, California Institute of Technology

Company: California Institute of Technology

Industry: Education

Country: United States

Employees: 300

Website: caltech.edu

About the California Institute of TechnologyBased in Pasadena, the California Institute of Technology (Caltech) is a private research university often ranked as one of the top-10 universities in the world. Founded in 1891, Caltech is one of a small group of US technology institutes primarily devoted to instruction in pure and applied sciences.

https://www.caltech.edu/


”“

Modernized applications to employ microservices. Deployed containers via Kubernetes and Mesos with EFS providing persistent storage and ability to dynamically scale application without storage management overhead

T-Mobile scales modern application deployments with Amazon EFS

Customer facing application with large spikes in usage based on time of day and month of year. Existing infrastructure was not able to support the scalability required without overprovision of infrastructure to support peak usage.

• 16,000 containers under management

• Reduced cost of NFS storage by 70% compared to DIY while reducing storage management overhead

• Improved cycle time for deploying application services


We are a large organization that has lots of applications with varying requirements for availability and performance. EFS provides us with a common storage platform that meets these requirements across the board.

Amreth Chandrasehar, Principal Architect, T-Mobile

Company: T-Mobile

Industry: Mobile Communications

Country: Global

Employees: 52,000

Website: www.t-mobile.com

About T-MobileAs America's Un-carrier, T-Mobile US, Inc. is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide network delivers outstanding wireless experiences to 79.7 million customers who are unwilling to compromise on quality and value.


Journey to (and in) the cloud

• Moved containerized data science environment to AWS for agility and cost benefits

• Enabled self-service provisioning of containerized analytics applications and compute resources

• Migrated to a managed service for better stability, application scaling and ease of operations, reducing storage management time by 90%


The company began using Amazon EFS for centralized file storage for its containerized data science platform. Faculty also takes advantage of AWS CloudFormation scripts to provision code.

Faculty Uses Amazon EFS to Scale Innovative Machine-Learning Platform

As Faculty’s customer base grew, the company needed a more scalable shared-file storage system to support machine-learning projects requiring up to 10 TB of storage.

• Scales to support 10 TB of storage

• Deploys platform days faster

• Gives developers more time to build innovative features

• Increases collaboration


Company: Faculty

Industry: Software & Internet

Country: United Kingdom

Employees: 100

Website: https://faculty.ai/

About Faculty

”“ Headquartered in the United Kingdom,

Faculty is a provider of data science, machine-learning, and artificial intelligence solutions. The company’s data science platform gives data scientists the ability to use code to build machine-learning models and gain access to large data sets.

.

The sign of a great technology is that you forget it’s there. Amazon EFS just works. It requires zero maintenance on our end.

Scott Stevenson, Data Engineer, Faculty

https://faculty.ai/


General Concepts: Container, Task, & Pod Identity


Goals for Security & Identity

1. File systems should only be mountable by the applications that need them

2. Apps that mount file systems should only have access to the data they need

Amazon EFS File System

$ cat /my_app/data

### SUCCESS THIS IS MY FILE ###

$ cat /someone_elses_app/data

cat: /someone_elses_app/data : Permission denied


Using IAM for File System Access

{

“Statement” : {

“Effect” : “allow”,

“Action” : “elasticfilesystem:Client*”,

“Principal” : { “AWS”: “FargateRole” }

}

}


AWS Fargate

Task RoleTask Definition Amazon EFS

File System

AWS IAM

{

“Statement” : {



"Resource": ”fs-deadbeef"

}

}

ECSEKS


Handling EFS Authorization Using IAM

Anonymous Task

Task “semitrust”

Task “fulltrust”


“Action” : “elasticfilesystem:ClientMount”,

“Principal” : “*”


“Action” : [“elasticfilesystem:ClientMount”,

“elasticfilesystem:ClientWrite”],

“Principal” : { “AWS”: “semitrust” }


“Action” : [“elasticfilesystem:ClientMount”,

“elasticfilesystem:ClientWrite”,

“elasticfilesystem:ClientRootAccess],

“Principal” : { “AWS”: “fulltrust” }

ECSEKS

Squashed to 65535


Understanding Container Identity

ECS Task

Task Identity (IAM Role)

AWS IAM

Container Image

App IdentityUser: RootGroup: Root

$ ls –l /efs/home

drwx------ bob . BobHome

drwx------ sally . SallyHome

drwxrwx--- . biusers BI_Shared

By default, POSIX identity comes from the container image, not the task/pod runtime.


Application-specific Access with EFS Access Points

{

“Name”: “MyApp”,

"FileSystemId": ”fs-deadbeef",

“PosixUser”: {

“Uid”: 123

“Gid”: 123,

“SecondaryGids”: [100, 200, 300]

},

“RootDirectory”: {

“Path”: “/apps/myapp”,

“CreationInfo”: {

“OwnerUid”: 123,

“OwnerGid”: 123,

“Permissions”: “0700”

}

}

}

Creates App-specific Directory & PermissionsNo EC2 instance required!Apps only see data they need

Enforces File System IdentityRoot containers can’t escalate accessArbitrary users aren’t locked out

ECSEKS


{

“Name”: “MyApp”,

“PosixUser”: {

“Uid”: 123

“Gid”: 123,

“SecondaryGids”: [100, 200, 300]

},

“RootDirectory”: {

“Path”: “/apps/myapp”,

“CreationInfo”: {

“OwnerUid”: 123,

“OwnerGid”: 123,

“Permissions”: “0700”

}

}

}

How EFS Access Points Work

File System with POSIX Permissions



“Principal” : { “AWS”: “approle” },

“Condition”* : {“accessPointArn” : “fsap-1234”

ECSEKS


Best Practices for IAM and Access Points, & Security

• Use EFS Access Points, even if single app per file system!• Simplifies directory permission setup• Consistent experience regardless of user/group setup in container• Future-proof for adding apps to share data

•Use IAM Authorization• Use Resource Policies to restrict IAM roles to Access Points• Use Identity Policies to give single role “admin” access to file systems

•Enable Encryption @ Rest and Encryption in Motion• Simple setup, no performance penalty


Using EFS From ECS (including Fargate!)


New: Amazon ECS and AWS FargateSupport for Amazon EFS

Simple: All EFS configuration is inside the ECS task definition, and connectivity is handled behind the scenes.

Serverless: AWS Fargate tasks can now leverage shared persistent storage.

Secure: Access to file systems can be authorized by IAM, and access to data controlled by EFS Access Points.



How it works

Task

Container 1

Container 2

Amazon ECS

Amazon EC2 AWS Fargate

Amazon EFS

File system

EFSVolumeConfiguration



"containerDefinitions": [{

..."mountPoints": [

{"readOnly": null,"containerPath": "/data",

"sourceVolume": "FargateDemoEFS"}

],...

"name": "FileBrowser"

}],

"taskRoleArn": "arn:aws:iam::..:role/FargateRole",...

"volumes": [

{"efsVolumeConfiguration": {

"transitEncryptionPort": null,"fileSystemId": "fs-41c7f3c1","authorizationConfig": {

"iam": "ENABLED","accessPointId": "fsap-0f7741bf379626fc2"

},"transitEncryption": "ENABLED","rootDirectory": "/"

},"name": "FargateDemoEFS",



Using EFS with EKS (using the CSI Driver)


EFS & EKS: Concepts

• Container Storage Interface (CSI)

• Industry standard interface for connecting storage providers (block or file) to a container.

• EFS CSI Driver

• Implementation of CSI for connecting EFS file systems to containers.

• Storage Class (SC)

• Administrator-defined class of storage that Persistent Volumes can be created from.

• Persistent Volume (PV)

• Administrator-created unit of storage that can be attached to a container. Has its own lifecycle.

• Persistent Volume Claim (PVC)

• Request to allocate an available PV from a SC to a container.

Amazon Elastic Kubernetes Service


EKS Storage – Process Flow

Storage Class (name: GeneralPurposeEFS)

Persistent Volume

Name: PV1 FS:fs-deadbeef Path: /pv1/

Persistent Volume


Persistent Volume


Persistent Volume


Persistent Volume


1. Admin Creates SC & PVs2. Dev Claims PVs from SC

Persistent Volume Claim

Name: MyAppClaim

SC: GeneralPurposeEFS

Pod

Name: MyApp

PVC: MyAppClaim

3. Dev Launches Pod Referencing PV Claim



Attaching an EFS file system to a Pod (Admin)Create Storage Class Create Persistent Volume

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

name: efs-sc

provisioner: efs.csi.aws.com

mountOptions:

- tls

apiVersion: v1

kind: PersistentVolume

metadata:

name: efs-pv

spec:

capacity:

storage: 5Gi

volumeMode: Filesystem

accessModes:

- ReadWriteOnce

persistentVolumeReclaimPolicy: Retain

storageClassName: efs-sc

mountOptions:

- tls

csi:

driver: efs.csi.aws.com

volumeHandle: fs-deadbeef



Attaching an EFS file system to a Pod (User)Create Persistent Volume Claim Launch Pod

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: efs-claim

spec:

accessModes:

- ReadWriteOnce

storageClassName: efs-sc

resources:

requests:

storage: 5Gi

apiVersion: v1

kind: Pod

metadata:

name: efs-app

spec:

containers:

- name: web-container

image: httpd

ports:

- containerPort: 80

name: “http-server”

volumeMounts:

- name: persistent-storage

mountPath: /mnt-efs

volumes:

- name: persistent-storage

persistentVolumeClaim:

claimName: efs-claim



Best Practices: Performance, Cost, & Ingest


Best Practices - Performance

• Use General Purpose for most apps

• GP lower latency, now supports up to 35K read IOPS

• MaxIO for scale-out analytics/ML that need 100k+ IOPS

• Configure provisioned throughput for initial need

• As your file system grows you’ll eventually be given higher throughput

• Set up Amazon CloudWatch, monitor throughput, IOPS, and burst credits*

* https://github.com/aws-samples/amazon-efs-tutorial/tree/master/monitoring


When should I use EFS vs EBS?

• I need to share data between containers

• I’d like to run across instances or AZs

• I’d like to take advantage of spot pricing

• I need low latency (e.g. MySQL)

• I need point in time snapshots

Amazon Elastic Block Store

Amazon Elastic File

System

Note: Amazon FSx for Lustre can be used for containers that require ultra-high throughput and very low latency file sharing


Optimize cost with Amazon EFS Infrequent Access Amazon EFS IA storage class for infrequently accessed files for $0.025/GB/mo*

* Pricing in the US East (N. Virginia) region

Automated lifecycle

management

Costsavings up

to 92%

No changes to existing applications using

Amazon EFS


Backup for Amazon EFS

• EFS file systems can be backed up and restored usingAWS Backup

• AWS Backup provides automated backup scheduling and retention per user defined policy

• AWS Backup offers two classes of service backup storage with the ability to lifecycle to cold storage

• Restore individual files and directoriesCold storage

AWS Backup

Warm storage

Amazon EFS

Backup encryption


Migrating NFS workloads to EFS

On-Premise

NAS FilerLinu

x Ap

plic

atio

n Se

rver

s NFS

EFS File System

AWS RegionLinux EC2 Instances

NFS

AWS DataSync

DataSyncagent

AWS Direct Connect

Internet

VPN

Virtual machine

NFS

AWS DataSync: Online transfer service that simplifies, automates, andaccelerates moving data between on-premises storage and AWS


Where to learn more

Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate

(Parts 1-3)

By Massimo Re Ferre’

Amazon EFS: Secure data persistence with Amazon ECS and AWS Fargate

(YouTube demo)

https://aws.amazon.com/blogs/containers/developers-guide-to-using-amazon-efs-with-amazon-ecs-and-aws-fargate-part-1/

https://youtu.be/FJlHXBcDJiw


Thank you!

persistent storage for containers using amazon efs · 2020. 8. 21. · •intro: why persistent...

Documents