su yuxin jan 20, 2014 petuum: an iterative-convergent distributed machine learning framework

SU YUXINJAN 20, 2014

Petuum: An Iterative-Convergent Distributed

Machine Learning Framework

Outline

Introduction

Implementation

Questions

Demo

Introduction to Petuum

Bulk Synchronous Parallel

Asynchronous

Parameters read / update at any time

Stale Synchronous Parallel

Convergence

Programming

read(table, row, col)

inc(table, row, col, value)

iteration()

Implementation

Overview in Logic

Overview in the Real

Main Components

Table

ConsistencyController::DoGet()

ConsistencyController::iterate()

Server::GetRow()

Least-Recently-Used(LRU) Strategy

Questions

Is Lock-Free Possible ?

Data exchange in real-time ? next …

Is Auto-Rescheduling Possible ?

sub-centralized server

reduce communication cost

Is Auto-Partition Possible ?

Run ML algorithms like that in a single thread

A Solution for all ML algorithms

In-Memory or In-Storage ?

Data capacity is greater than memory size.

Memory should be a cache for disk storage.

Solution for disk storage: Hadoop Spark ….

New Schema to Reduce the Upper Bound?

STRADS Scheduler

Variable Correlations Auto-Parallelization

Dynamic Prioritization Monitor the contribution of variables to objective

function

Load-Balancing in Task

Demo

Switch to my laptop …