torch internals - collobertronan.collobert.com/torch/internals.pdftorch internals ronan collobert...
TRANSCRIPT
![Page 2: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/2.jpg)
Introduction
![Page 3: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/3.jpg)
Machine Learning Library
• Around since 2000
• 5 versions, implemented in C, C++, Objective C…
• Implements main machine learning algorithms:
• SVMs, Neural Networks, HMMs, GMMs…
• Goal: research on large-scale machine learning
![Page 4: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/4.jpg)
It is for researchers…• It has to be modular!
organized as packages
• It must be easy to extend!
• It must be simple to understand!
a high-level language is essential
• It must be fast (large-scale machine learning)
JIT-compile or
efficient C interface for core routines
![Page 5: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/5.jpg)
Overview
![Page 6: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/6.jpg)
Overview
TH (C)
torch (lua)
optim
nn
svm
sndfile
image
nngraph
![Page 7: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/7.jpg)
Overview
TH (C)
torch (lua)
optim
nn
svm
sndfile
image
nngraph
core
![Page 8: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/8.jpg)
Overview
TH (C)
torch (lua)
optim
nn
svm
sndfile
image
nngraph
core
packages
![Page 9: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/9.jpg)
Overview
TH (C)
torch (lua)
optim
nn
svm
sndfile
image
nngraph
core
packages
![Page 10: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/10.jpg)
Overview
TH (C)
torch (lua)
optim
nn
svm
sndfile
image
nngraph
core
packages
![Page 11: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/11.jpg)
Overview
TH (C)
torch (lua)
optim
nn
svm
sndfile
image
nngraph
core
packages
![Page 12: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/12.jpg)
Overview
TH (C)
torch (lua)
optim
nn
svm
sndfile
image
nngraph
core
packages
luaT (C)
cwrap
glue
![Page 13: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/13.jpg)
Which Programming Language?
• LISP? (Lush http://lush.sourceforge.net)
• A subset could be compiled
• Could inline C code
• JIT-compiled Languages? (e.g. LuaJIT)
• Only a subset is actually compiled
• Python?
• err… Lua?
• Looks like pseudo-code (yet very powerful)
• Simple C API
![Page 14: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/14.jpg)
Which Programming Language?
• C++? (Torch3, PLearn, EBLearn…)
• Objective C? (Torch4)
• Avoid too many abstractions
• Avoid complicated syntax
• We chose C for Torch7
![Page 15: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/15.jpg)
The CoreTH and torch
![Page 16: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/16.jpg)
The core library (TH+torch)
• ML algorithms manipulate all kind of data
• Represent data as a n-dimensional Tensor
• 1D: a bunch of features
• 2D: a gray image, some audio features
• 3D: RGB images
• 4D: Videos…
• The core lib provides many tensor operations
• Available from C (TH lib) and Lua (torch package)
![Page 17: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/17.jpg)
The Core Library (TH+torch)• Avoid memory copies
• A Tensor is a view of a Storage (memory chunk)
• Storages might have different views (Tensors)
• Storages can be in-memory, or on-disk (mmap!)
![Page 18: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/18.jpg)
The Core Library (TH+torch)• Avoid memory copies
• A Tensor is a view of a Storage (memory chunk)
• Storages might have different views (Tensors)
• Storages can be in-memory, or on-disk (mmap!)
• Easy to get different views from another tensor x:
• x:narrow(dim, idx, size)
• x:select(dim, idx)
• x[idx]
• x:unfold(dim, kw, dw)
• x:transpose(dim1, dim2)
![Page 19: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/19.jpg)
The Core Library (TH+torch)• Avoid memory copies
• A Tensor is a view of a Storage (memory chunk)
• Storages might have different views (Tensors)
• Storages can be in-memory, or on-disk (mmap!)
• Easy to get different views from another tensor x:
• x:narrow(dim, idx, size)
• x:select(dim, idx)
• x[idx]
• x:unfold(dim, kw, dw)
• x:transpose(dim1, dim2)
new tensors sharesame storage
![Page 20: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/20.jpg)
The Core Library (TH+torch)
• Avoid memory copies
• All C functions are of the form
• THTensor_foobar(THTensor *dst, THTensor *src1, THTensor *src2)
• The Lua interface supports the destination as an optional argument:
torch.foobar([dst], src1, src2)
or
dst:foobar(src1, src2)
![Page 21: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/21.jpg)
The GlueluaT and cwrap
![Page 22: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/22.jpg)
C/Lua Glue• Lua is a stack-based language
• In Lua: a = f("how", t.x, 14)
• In C:
• How to extend it? (luaT or FFI)
• Lua provides the “userdata” type (a GCed C pointer)
• Need to do proper type checking both in C and Lua
• How to limit the pain? (cwrap or FFI)
![Page 23: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/23.jpg)
C/Lua Glue (luaT)
![Page 24: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/24.jpg)
C/Lua Glue (luaT)
check argumentson the stack
![Page 25: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/25.jpg)
C/Lua Glue (luaT)
check argumentson the stack
push the result
![Page 26: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/26.jpg)
C/Lua Glue (luaT)
check argumentson the stack
push the resultcall TH
![Page 27: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/27.jpg)
C/Lua Glue (cwrap)
![Page 28: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/28.jpg)
C/Lua Glue (cwrap)
for all tensor types
![Page 29: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/29.jpg)
C/Lua Glue (cwrap)
for all tensor typestorch.cmul(dst, src)
ordst:cmul(src)
![Page 30: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/30.jpg)
C/Lua Glue (cwrap)
![Page 31: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/31.jpg)
C/Lua Glue (cwrap)
another default value
![Page 32: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/32.jpg)
C/Lua Glue (cwrap)
two different C functions
another default value
![Page 33: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/33.jpg)
![Page 34: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/34.jpg)
C/Lua Glue (FFI)
• Cut-and-paste the C header
• Caveats:
• Need to do proper argument checking (sub-classes…)
• Need to overload methods/functions properly
• Less robust across different systems (#define…)
• Need also to handle different types (templates)
![Page 35: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/35.jpg)
Tensor Types
![Page 36: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/36.jpg)
Dynamic Typing (lua)• Write generic code with torch.Tensor()
• Specify alias with (e.g.) torch.setdefaulttensortype(‘torch.FloatTensor’)
![Page 37: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/37.jpg)
Dynamic Typing (lua)• Write generic code with torch.Tensor()
• Specify alias with (e.g.) torch.setdefaulttensortype(‘torch.FloatTensor’)
return atorch.Tensor()
![Page 38: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/38.jpg)
CUDA
• cutorch package defines CudaTensor
• Most CPU Tensor methods are available
• Relies on few custom-made iterators and thrust
• x:cuda() or x:float() for GPU <-> CPU copies
• cunn package provides a GPU backend to nn
• Most popular nn layers have a GPU backend
![Page 39: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/39.jpg)
Deep Learningnn
![Page 40: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/40.jpg)
nn package
![Page 41: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/41.jpg)
nn package
![Page 42: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/42.jpg)
nn package
• Two main classes: Module and Criterion
• Three main methods:
• updateOutput(input)
• updateGradInput(input, gradOutput)
• accGradParameters(input, gradOutput)
![Page 43: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/43.jpg)
nn package
![Page 44: Torch Internals - Collobertronan.collobert.com/torch/internals.pdfTorch Internals Ronan Collobert locronan@fb.com torch.ch Introduction Machine Learning Library • Around since 2000](https://reader034.vdocuments.us/reader034/viewer/2022051604/5ffb86be53b3800fd3637bcf/html5/thumbnails/44.jpg)
Tutorial• Torch7
• https://github.com/torch/torch7
• nn package
• https://github.com/torch/nn
• itorch
• https://github.com/facebook/iTorch
• Installation instructions
• http://torch.ch/docs/getting-started.html
• Get the tutorial
• http://ronan.collobert.com/torch