Upload File

distributed and automated model inference on big data ... · distributed and automated model...

  • Home
  • Documents
  • Distributed and Automated Model Inference on Big Data ... · Distributed and Automated Model Inference on Big Data Streaming Frameworks Authors: Jiaming Song, Dongjie Shi, Qiyuan
26
Cluster Serving: Distributed and Automated Model Inference on Big Data Streaming Frameworks Authors: Jiaming Song, Dongjie Shi, Qiyuan Gong, Lei Xia, Jason Dai

Upload: others

Post on 28-Jan-2021

3 views

Category:

Documents


0 download

Report

TRANSCRIPT

  • Cluster Serving: Distributed and Automated Model Inference on Big Data Streaming Frameworks

    Authors: Jiaming Song, Dongjie Shi, Qiyuan Gong, Lei Xia, Jason Dai

  • Outline

    Challenges AI productions facing

    Integrated Big Data and AI pipeline

    Scalable online serving

    Cross-industry end-to-end use cases

  • “Machine Learning Yearning”, Andrew Ng, 2016

    Big Data & Model Performance

    *Other names and brands may be claimed as the property of others.

  • Real-World ML/DL Applications Are Complex Data Analytics Pipelines

    “Hidden Technical Debt in Machine Learning Systems”,Sculley et al., Google, NIPS 2015 Paper

    *Other names and brands may be claimed as the property of others.

  • Outline

    Challenges AI productions facing

    Integrated Big Data and AI pipeline

    Scalable online serving

    Cross-industry end-to-end use cases

  • Integrated Big Data Analytics and AI

    Production Data pipeline

    Prototype on laptopusing sample data

    Experiment on clusterswith history data

    Production deployment w/ distributed data pipeline

    • Easily prototype end-to-end pipelines that apply AI models to big data

    • “Zero” code change from laptop to distributed cluster

    • Seamlessly deployed on production Hadoop/K8s clusters

    • Automate the process of applying machine learning to big data

    Seamless Scaling from Laptop to Distributed Big Data

    *Other names and brands may be claimed as the property of others.

  • Seamless Scaling from Laptop to Distributed Big Data

    Distributed, High-Performance

    Deep Learning Framework for Apache Spark*

    https://github.com/intel-analytics/bigdl

    Unified Analytics + AI Platform for TensorFlow*, PyTorch*, Keras*, BigDL,

    OpenVINO, Ray* and Apache Spark*

    https://github.com/intel-analytics/analytics-zoo

    AI on Big Data

    *Other names and brands may be claimed as the property of others.

    software.intel.com/bigdlhttps://github.com/intel-analytics/analytics-zoo

  • Analytics Zoo

    Recommendation

    Distributed TensorFlow* & PyTorch* on Spark*

    Spark* Dataframes & ML Pipelines for DL

    RayOnSpark

    InferenceModel

    Models & Algorithms

    End-to-end Pipelines

    Time Series Computer Vision NLP

    Unified Data Analytics and AI Platform

    https://github.com/intel-analytics/analytics-zoo

    ML Workflow AutoML Automatic Cluster Serving

    Compute Environment

    K8S* Cluster Cloud

    Python Libraries (Numpy/Pandas/sklearn/…)

    DL Frameworks (TF*/PyTorch*/OpenVINO*/…)

    Distributed Analytics (Spark*/Flink*/Ray*/…)

    Laptop Hadoop* Cluster

    Powered by oneAPI

    *Other names and brands may be claimed as the property of others.

    https://github.com/intel-analytics/analytics-zoo

  • Outline

    Challenges AI productions facing

    Integrated Big Data and AI pipeline

    Scalable online serving

    Cross-industry end-to-end use cases

  • What’s Serving

    Input DataResult

    model

    Preprocessing Inference Postprocessing

    *Other names and brands may be claimed as the property of others.

  • Example of Classical Web Serving

    *Other names and brands may be claimed as the property of others.

  • Node

    Node

    Node

    Node

    Distributed Model Serving

    DataSource

    Analytics Zoo Model

    Analytics Zoo Model

    Analytics Zoo Model

    Distributed model serving in Flink*, Spark*, Kafka*, Storm*, etc

    Node

    Node

    Node

    *Other names and brands may be claimed as the property of others.

  • Architecture of Main Version of Cluster Serving

    driver(job manager)

    executor(task manager)

    executor(task manager)

    input(data)

    output(result)

    Redis*

    Apache Flink*

    Version based on Spark* Streaming is also supported.

    data source

    *Other names and brands may be claimed as the property of others.

    InferenceModel

    data sink

    data source

    InferenceModel

    data sink

  • Advantages of Analytics Zoo Cluster Serving

    Ease of DeploymentOne container with all dependencies & leverage existed YARN/K8S cluster

    Wide Range Deep Learning model supportTensorflow*, Caffe*, OpenVINO*, Pytorch*, BigDL*

    Low LatencyContinuous Streaming pipeline is supported by Apache Flink* and Spark*

    High Throughput & ScalabilityOptimization of multithread control, and could easily scale out to clusters

    *Other names and brands may be claimed as the property of others.

  • Data pipeline User Perspective

    P5P4

    P3P2

    P1

    R4R3

    R2R1R5

    http request

    Output Queue (or files/DB tables) for prediction results

    Local node or Docker container

    Hadoop*/YARN* (or K8S*) cluster

    Network connection

    Model

    Simple Python script

    HTTP Server

    Input Queue for requests

    http response

    *Other names and brands may be claimed as the property of others.

  • Cluster Serving Workflow Overview

    1. Install and prepare Cluster Serving environment on a local node2. Launch the Cluster Serving service3. Distributed, real-time (streaming) inference

    *Other names and brands may be claimed as the property of others.

  • Very Quick Start

    Start docker container#docker run -itd --name cluster-serving --net=host intelanalytics/zoo-cluster-serving:0.7.0

    Log into container#docker exec -it cluster-serving bash

    Start Serving#cluster-serving-start

    https://github.com/intel-analytics/analytics-zoo/blob/master/docs/docs/ClusterServingGuide/ProgrammingGuide.md

    *Other names and brands may be claimed as the property of others.

    https://github.com/intel-analytics/analytics-zoo/blob/master/docs/docs/ClusterServingGuide/ProgrammingGuide.md

  • API Introductions

    http sync APIdata are represented by json formatcall http post method to enqueue your data into pipelinehttp API is compatible with TFServing*

    pub-sub python async APIdata are represented by ndarraycall python method to enqueue your data into pipeline

    *Other names and brands may be claimed as the property of others.

  • API Introductions - HTTP

    http APIdata are represented by json formatSupport

    scalarstensorssparse tensorsimage encodings

    *Other names and brands may be claimed as the property of others.

  • Python APIdata are represented by python objectsSupport

    scalarstensorssparse tensorsimage encodings

    API Introductions - Pub-sub

    *Other names and brands may be claimed as the property of others.

  • Outline

    Challenges AI productions facing

    Integrated Big Data and AI pipeline

    Scalable online serving

    Cross-industry end-to-end use cases

  • Garbage classification on Tianchi Competition

    *Other names and brands may be claimed as the property of others.

  • Medical Imaging Analysis

    Bottleneck: Preprocessing, inference, up to 1-2 hours per large piece

    https://en.wikipedia.org/wiki/X-ray

    *Other names and brands may be claimed as the property of others.

    https://en.wikipedia.org/wiki/X-ray

  • Unified Analytics + AI Platform

    Distributed TensorFlow*, Keras*, PyTorch* & BigDL on Apache Spark*

    https://github.com/intel-analytics/analytics-zoo

    End-to-End Big Data and AI PipelinesSeamless Scaling from Laptop to Production

    *Other names and brands may be claimed as the property of others.

    https://github.com/intel-analytics/analytics-zoo

  • Legal Disclaimers• Intel technologies’ features and benefits depend on system configuration and may require enabled

    hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.

    • No computer system can be absolutely secure.

    • Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

    Intel, the Intel logo, Xeon, Xeon phi, Lake Crest, etc. are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.© 2019 Intel Corporation

    http://www.intel.com/performance

Automated versus do-it-yourself methods for causal ... · Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition Vincent Doriey,

Relatively Complete Verification of Higher- Order Programs (via Automated Refinement Type Inference) Tachio Terauchi Nagoya University TexPoint fonts used

Automated Scalable Bayesian Inference via Data ...people.csail.mit.edu/tbroderick/files/broderick_2018...Automated Scalable Bayesian Inference via Data Summarization ITT Career Development

Computing a Second Opinion: Automated Reasoning and Statistical ...aiellom/tesi/emerencia.pdf · Automated Reasoning and Statistical Inference applied to Medical Data Ando Emerencia

sum of squaresweb.stanford.edu/class/ee364b/lectures/sos_slides.pdf · 23 Sum of Squares S. Lall, Stanford 2011.04.18.01 Automated Inference and Algebra Automated inference is a well-known

Alembic: Automated Model Inference for Stateful Network … · Modern production networks include a large numberof propri-etary network functions (NFs), such as firewalls (FWs),

Variational Inference and Mean Field · Variational Inference “Variational inference”: Formulate inference problem as constrained optimization. Approximate the function or constraintsto

SketchLearn: Relieving User Burdens in Approximate ...pclee/www/pubs/tech_sketch...SketchLearn: Relieving User Burdens in Approximate Measurement with Automated Statistical Inference

Low precision Inference on GPU - Nvidia · 3 INFERENCE • Inference: using a trained model to make predictions • Much of inference is fwd pass in training • Inference engines

INDUCTION (probable inference) : inference moving from specific facts to general conclusions. DEDUCTION (necessary inference): inference moving from general

Bayesian and frequentist inference for ecological ... · Key Words and Phrases: ecological inference, Bayesian inference, frequentist inference, voting patterns. 1 Introduction to

Lecture 5 Fuzzy expert systems: Fuzzy inference Mamdani fuzzy inference Mamdani fuzzy inference Sugeno fuzzy inference Sugeno fuzzy inference Case study

Name Definitions: Evidence Inference Period Inference vs

Portfolio of MA Dongjie

Automated Likelihood Based Inference for Stochastic ...Automated Likelihood Based Inference for Stochastic Volatility Models⁄ Hans J. Skaugy, Jun Yu z October 7, 2008 Abstract: In

Parametric Inference Maximum Likelihood Inference …bioucas/IP/files/statistical_inference.pdf · Statistical Inference Parametric Inference Maximum Likelihood Inference Exponential

Mind Reading Machines: Automated Inference of Cognitive ...pr10/publications/smc04.pdf · paper describes a system for the automated inference of cognitive mental states from observed

INFaaS: Automated Model-less Inference Serving

PhD Dissertation Proposal - Northwestern Universityagupta/Files/PhD/Inference... · Web viewZeus*: Automated Runtime Inference for Virtual Machine application’s resource demands

Automated Sensitivity Analysis for Bayesian …Automated Sensitivity Analysis for Bayesian Inference via Markov Chain Monte Carlo: Applications to Gibbs Sampling LIANA JACOBI Department

Direct Inference and Inverse Inference Teddy … Inference and Inverse Inference Teddy Seidenfeld The Journal of ... section 11 I consider Ian Hacking's novel ... Direct Inference

STATS 200: Introduction to Statistical Inference 200: Introduction to Statistical Inference ... Statistical inference Statistical inference = Probability 1 ... STATS 200: Introduction

Automated Documentation Inference to Explain Failed Tests Sai Zhang University of Washington Joint work with: Cheng Zhang, Michael D. Ernst

Second order inference in NL semantics - UZHattempto.ifi.uzh.ch/site/cnl2012/slides/pulman_secondorder.pdf · Logical forms sent to automated theorem prover (resolution, tableau...)

Automated Inference with Adaptive Batchesproceedings.mlr.press/v54/de17a/de17a.pdf · Automated Inference with Adaptive Batches Soham De, Abhay Yadav, David Jacobs and Tom Goldstein

ALEMBIC: AUTOMATED MODEL INFERENCE FOR ......ALEMBIC: AUTOMATED MODEL INFERENCE FOR STATEFUL NETWORK FUNCTIONS. Soo-Jin Moon. Jeffrey Helt, Yifei Yuan, Yves Bieri, Sujata Banerjee,

Understanding and Detecting Evolution-Induced ...dongjieh/papers/ase2018.pdf · Understanding and Detecting Evolution-Induced Compatibility Issues in Android Apps Dongjie He State

Automated Inference of Access Control Policies for Web Applications

Automated, predictive, and interpretable inference of ... · Automated, predictive, and interpretable inference of Caenorhabditis elegans escape dynamics Bryan C. Danielsa, William

Automated Computational Inference Engine for Bayesian ... · Automated computational inference engine 1585 to be rigorously assessed. The Bayesian inferential methodology for source

M ETHODS OF INFERENCE Hasan Zafari. M ETHODS OF INFERENCE What is reasoning? Inferences with rules trees The inference tree Inference by Inheritance Inference

Automated Inference of Atomic Sets for Safe …Automated Inference of Atomic Sets for Safe Concurrent Execution Gul Agha University of Illinois at Urbana-Champaign Joint work with

Automated Documentation Inference to Explain Failed Tests

Automated Facial Trait Judgment and Election Outcome ...€¦ · tion and computer vision. Furthermore, research in social psychology suggests that people also make inference about

Tutorial on Approximate Bayesian Computation · Inference for simulator-based models Exact inference Approximate inference Rejection ABC algorithm Approximations to make inference