accelerating machine learning devops with kubeflow · accelerating machine learning devops with...
TRANSCRIPT
![Page 1: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/1.jpg)
Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank
![Page 2: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/2.jpg)
Who am I?• Head of Engineering, JP Morgan Chase Commercial Bank • [email protected] • Blog at http://derekmferguson.wixsite.com/ml4nonmath • Previously a tech evangelist in the Microsoft space
• Editor-in-Chief of the .NET Developer’s Journal • Author of Mobile.NET and Broadband Internet Access for Dummies
• Life-long Chicago native • Graduated DePaul & started commercial DSL with InterAccess
![Page 3: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/3.jpg)
City Scholars• Partnership between Chase, UIUC and City of Chicago • Students work half-time in the Spring and full-time in Summer • Our remits this year:
• Build some great machine learning models for the Business • Improve the state of machine learning DevOps for the Industry
![Page 4: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/4.jpg)
The Problem
![Page 5: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/5.jpg)
The Problem
![Page 6: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/6.jpg)
The Problem
![Page 7: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/7.jpg)
The Problem
![Page 8: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/8.jpg)
The Problem
![Page 9: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/9.jpg)
The Problem
![Page 10: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/10.jpg)
The Problem
![Page 11: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/11.jpg)
The Problem
![Page 12: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/12.jpg)
The Problem
![Page 13: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/13.jpg)
The Problem
![Page 14: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/14.jpg)
Sneak Peek: the Solution
Kubernetes Machine Learning Kubeflow
![Page 15: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/15.jpg)
Agenda• Brief overview of the Fashion MNIST sample data • Overview of Kubernetes
• Starting with a brief overview of Docker • All about Kubeflow • Q&A
![Page 16: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/16.jpg)
Fashion MNIST• Sample data set with 10 different kinds of clothing • 60k training images and labels • 10k test images and labels • Images start as 784-position arrays containing numbers 0 to 255
![Page 17: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/17.jpg)
The Old Days (i.e. 5 years ago)
![Page 18: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/18.jpg)
Today, IT Ops has a lot to do…
![Page 19: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/19.jpg)
VMs take a long time to setup…
![Page 20: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/20.jpg)
Containers cheaper and faster - but still slow
![Page 21: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/21.jpg)
Behold, Kubernetes!
![Page 22: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/22.jpg)
Challenge 1: Network Topology• Framework needs to know location of Master • Master needs to know locations of workers • Workers need to know locations of parameter servers
![Page 23: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/23.jpg)
Challenge 2: It doesn’t turn off
![Page 24: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/24.jpg)
Challenge 3: Everything starts open• Jupiter notebooks open by default • Model storage open by default • Network calls unencrypted by default • Network addresses and ports unauthenticated by default
![Page 25: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/25.jpg)
What is Kubeflow?• Kubeflow is an end-to-end lifecycle orchestration tool for machine
learning • Vision would be to let data scientists get models from initial training
into Production with minimal human intervention • Enabling technology is Kubernetes
• There is *no* mandatory tie to Tensorflow
![Page 26: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/26.jpg)
Secure Jupyter Notebooks
![Page 27: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/27.jpg)
TF Job
![Page 28: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/28.jpg)
Fairing Attributes
![Page 29: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/29.jpg)
MiniKF• A fast and easy way to deploy Kubeflow onto your desktop • Full-fledged, production-ready Kubeflow instance that deploys in
minutes • Few clicks and you’re up for experimentation, including running full
Kubeflow pipelines • To train at scale, move to a public cloud deployment with one click and
no recoding • http://kubeflow.org/docs/started/getting-started-minikf • Discussion on #minikf Slack channel. Ask questions, request features
and get Support. • Contributed by Arrikto - core contributors to Kubeflow
![Page 30: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/30.jpg)
The Demo• Download: pull 70k images and labels from Internet • Pre-process: turn arrays into 28x28 matrixes with values 0 to 1 • Train: pass 60k images and labels through a neural network • Evaluate: test our model against 10k images and their labels • Serve: push model up to Tensorflow Serving
![Page 31: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/31.jpg)
Demo Architecture
![Page 32: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/32.jpg)
Demo: Pipelines
![Page 33: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/33.jpg)
Additional Points of Note• Securing the prediction layer (Tensorflow Serving, for example) • Securing S3 • Common K8S On Premises Issues
• Container lifespan limitations • K8S action limitations (e.g. creating and configuring namespaces) • Procurement challenges
• Live installation directly from the Internet • Insecure Docker images in distribution
![Page 34: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/34.jpg)
In Conclusion• All source is at https://github.com/JavaDerek/FashionMnistKF • Please contact me with any comments, questions or concerns…
![Page 35: Accelerating Machine Learning DevOps with Kubeflow · Accelerating Machine Learning DevOps with Kubeflow Derek Ferguson Head of Engineering, JP Morgan Chase Commercial Bank. ... •](https://reader036.vdocuments.us/reader036/viewer/2022062919/5ee2ba38ad6a402d666d077a/html5/thumbnails/35.jpg)
Questions?