containerizing gpu applications with docker for scaling to the …€¦ · containerizing gpu...
TRANSCRIPT
![Page 1: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/1.jpg)
Containerizing GPU Applications with Docker for Scaling to the Cloud
FUTURE OF PACKAGING APPLICATIONS SUBBU RAMA
![Page 2: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/2.jpg)
CPU
CPU
CPU
GPU GPU
GPU
GPU Mem
Mem Mem Mem
CPU CPU CPU CPU
GPU GPU GPU GPU
GPU GPU GPU GPU
Mem Mem Mem Mem
Mem Mem Mem Mem
Data Center Virtual
Supercomputer
GPU
GPU
Mem
GPU GPU
CPU Mem Mem
Mem
Turns Discrete Computing Resources into a Virtual Supercomputer
![Page 3: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/3.jpg)
What problems are we trying to solve? Software is Stuck:
◦ proper installation can take days
◦ operating system requirements
◦ library dependencies
◦ drivers
◦ interoperability between tools
Hardware is Stuck:
proper setup and optimization can take days
code portability
performance portability
resource provisioning
![Page 4: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/4.jpg)
Goal Given:
Applications from different vendors Systems of different capabilities Heterogeneous hardware
Compose a workflow that: Works: individual components work, thus workflow works Is Portable: workload can be migrated across infrastructure Is Performant: has the ability to take advantage of GPU hardware Is Secure: individual components can be easily audited
![Page 5: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/5.jpg)
Current Solutions Current solutions revolve around a common denominator:
Operating system that works for all tools in chain Compute nodes which satisfy the most memory hungry application Need GPUs? Must deploy on top of GPU only nodes Cost sensitive? Must deploy on low-end CPU only nodes
Common denominator shortcoming: Inefficiencies Poor utilization / over provisioning Non-performant
![Page 6: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/6.jpg)
Solution Containerize all applications
◦ Create GPU/CPU versions
Assemble containers into workflow templates ◦ To represent particular use cases and pipelines
Use workflow templates to create virtual clusters ◦ Optimize performance / budget via virtual clusters
![Page 7: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/7.jpg)
What are Containers? Containers are nothing new
◦ Part of Linux for last 10 years
◦ LXC, FreeBSD Jails, Solaris Containers, etc.
What is new are APIs ◦ Docker
◦ Rocket
◦ etc.
Specifically ◦ A complete runtime environment: OS, application, libraries, dependencies, binaries, and configuration files
◦ Can be quickly deployed on a set of container hosts when needed
![Page 8: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/8.jpg)
Containers vs. VMs (Stack Comparison)
![Page 9: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/9.jpg)
Why Containers? Easy Deployment
◦ Avoid hours of environment / application setup
◦ Fast environment spin-up / tear-down
Flexibility ◦ Applications use preferred version of OS, libs language versions, etc.
◦ Move data to application, or move Application to data
Reproducibility / Reliability / Scaling ◦ Workflow steps start with clean and immutable images
◦ Reliability through easy migration and checkpointing
![Page 10: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/10.jpg)
GPU Containers the NVIDIA Way Much easier that it used to be
◦ One no longer has to fully reinstall the NVIDIA driver within the container
◦ No more container vs. host system driver matching conflicts - container works with host OS driver - there is still a drive and toolkit dependency
◦ https://github.com/NVIDIA/nvidia-docker
Requirements ◦ Host has NVIDIA Drivers
◦ Host has Docker installed
![Page 11: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/11.jpg)
GPU Container Getting Started (CAFFE) Create a Dockerfile
◦ Very small, easy to re-build/update container if needed
◦ Reproducible builds
◦ Specify Operating System
◦ Install Operating System basics
◦ Install Application Dependencies
◦ Install Application
Once Dockerfile is done:
Build Container, Test Container, Store Container in Repo
Quickly spin up and container where and when
needed
Enables “fire-and-forget” GPU applications
What about data?
Long answer: we’ll get to that in a bit
Short: Put it somewhere else, keep containers small
![Page 12: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/12.jpg)
Dockerfile Code Demo
![Page 13: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/13.jpg)
Demo 1: Deploy GPU Container Across Clouds Interactive shell demo which will show:
Launch container on Cloud #1 and execute application (example: execute container on AWS and the K520s) Take exact same container and launch on Cloud #2 and execute application (example : execute container on Softlayer on the
K80s) Highlight the following:
◦ container runs on different clouds
◦ container uses different types of GPUs and drivers
◦ and everything works transparently
◦ “fire and forget” GPU applications on GPU hardware you need wherever it may be
![Page 14: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/14.jpg)
Container Performance People are sceptical about container performance vs bare-metal
◦ There are special cases where performance can be an issue, but in general performance is on par, and better than VMs
◦ Docker versus Bare Metal is within 10% performance W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An Updated Performance Comparison of Virtual Machines and Linux
Containers.Technology, 28:32, 2014. (IBM)
![Page 15: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/15.jpg)
So what about Data? In general, avoid storing data in containers.
◦ Container ought to be immutable ◦ bring it up, perform a task, return the result, shut it down
◦ Containers ought to be small ◦ size of containers impacts startup times
◦ size of containers impacts time it takes to pull container from repository
Discuss best practices for data storage/sharing ◦ give example of data sharing between containers using Data Volume Containers
![Page 16: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/16.jpg)
Application Flow Pipelines & Scheduling Sophisticated tool flows rarely consist of a single application
◦ Some steps may only run on CPUs
◦ Some steps may execute on a CPU or a GPU
Challenge is how to schedule these flow efficiently to either obtain faster turnaround times or better overall throughput, while maintaining reproducible results
![Page 17: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/17.jpg)
Example Workflow: Semiconductor Circuit Design
![Page 18: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/18.jpg)
Example Workflow: Semiconductor Circuit Design
CPU/GPU App
![Page 19: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/19.jpg)
Cluster Scheduling Constraints In general several assumptions can be made about today’s clusters
◦ # CPU nodes >> # GPU nodes
◦ GPU nodes have a fixed #GPUs in them
◦ Best machine for an application is usually determined by
◦ amount of memory
◦ amount and type of CPUs
◦ amount and type of GPUs
How can containers help with scheduling give this constraint vs. regular schedulers
![Page 20: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/20.jpg)
Example Scheduler: Regular vs. Container based
Kubernetes Mesosphere
![Page 21: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/21.jpg)
What if we can break Physical Machine Limitations?
Most cloud service provider and data centers are limited by physics ◦ Example: Largest machines has 2 GPUs (Softlayer), 4 GPUs (AWS)
◦ Rack can only have max amount of GPUs due to power constraints
What if we could create virtual machines and clusters and present them to applications as a single virtual machines?
How would this change the clusters and schedulers? * Elastic Containers or Elastic Machines via Containers (grow or shrink)
![Page 22: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/22.jpg)
Introduce Bitfusion Boost Containers We can:
◦ Combine Bitfusion Boost and Containers -> create magic!
What things can we build? ◦ Create a machine which has 16 or more virtual GPUs!
◦ Run an application across these GPUs without having to setup MPI, SPARK, HADOOP!
◦ Run GPU applications on non-GPU machines by automatically offloading to GPU machines in the cluster
◦ All of the above can be done WITHOUT CODE CHANGES for GPU enabled applications!
![Page 23: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/23.jpg)
Boost Container Building Blocks Boost Server Container
◦ Boost provisioned container with Boost Server
◦ Runs on any GPU provisioned host
◦ Can act as a client at the same time
Boost Client Container ◦ Boost provisioned container with Boost Client and End User Application
◦ Runs on any-type of instance including CPU only instances
![Page 24: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/24.jpg)
Boost Container Architecture
![Page 25: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/25.jpg)
Demo 2: Build Virtual GPU Instances in the Cloud
Demo will show the following using containers:
How in minutes we can create virtual GPU cluster configurations How we can provision GPU machines which don’t exist in the physical world How we can run GPU applications on non-GPU machines How we can execute applications across these configurations without changing a single line of code!
![Page 26: Containerizing GPU Applications with Docker for Scaling to the …€¦ · Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU](https://reader034.vdocuments.us/reader034/viewer/2022042711/5f6e43bc63beff18c47b4857/html5/thumbnails/26.jpg)
Thank You Visit us at our Booth located at:
◦ To learn more about GPU Containers
◦ To learn more about Bitfusion Boost
◦ And, to see more physics defying demos!
◦ AWS monster machine and CloudX
@Bitfusionio | @subburama | [email protected]