![Page 1: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/1.jpg)
deep learning with c++an introduction to tiny-dnn
by Taiga Nomiembedded software engineer, Osaka, Japan
![Page 2: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/2.jpg)
deep learning
Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
Facial recognition
Image understanding
Finance
Game playing
Translation
Robotics
Drug discovery
Text recognition
Video processing
Text generation
![Page 3: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/3.jpg)
Deep learning- Learning a complicated function from many data
- Composed of trainable, simple mathematical functions
Input OutputTrainable Building Blocks
- text
- audio
- image
- video
- ...
- text
- audio
- image
- video
- ...
![Page 4: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/4.jpg)
deep learning framework
![Page 5: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/5.jpg)
![Page 6: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/6.jpg)
A modern deep learning framework for C++ programmers
![Page 7: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/7.jpg)
1400 stars 500 forks 35 contributors 100 clones/day
![Page 8: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/8.jpg)
“A Modern Deep Learning module” by Edgar Riba
“Deep Learning with Quantization for Semantic Saliency Detection” by Yida Wang
https://summerofcode.withgoogle.com/archive/
![Page 9: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/9.jpg)
1.Easy to introduce
2.Simple syntax
3.Extensible backends
![Page 10: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/10.jpg)
1.Easy to introduce- Just put the following line into your cpp
tiny-dnn is header only - No installation
tiny-dnn is dependency-free - No prerequisites
#include <tiny_dnn/tiny_dnn.h>
![Page 11: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/11.jpg)
1.Easy to introduce- You can bring Deep Learning to any target you have a C++ compiler
- Officially supported (by CI builds):
- Windows (msvc2013 32/64bit, msvc2015 32/64bit)
- Linux (gcc4.9, clang3.5)
- OSX(LLVM 7.3)
- tiny-dnn might run on other compiler that support C++11
![Page 12: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/12.jpg)
1.Easy to introduce- Caffe model converter is also available
- TensorFlow converter - coming soon!
- Close the gap between researcher and engineer
![Page 13: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/13.jpg)
1.Easy to introduce
2.Simple syntax
3.Extensible backends
![Page 14: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/14.jpg)
2.Simple syntaxExample: Multi layer perceptron
![Page 15: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/15.jpg)
Caffe prototxt
input: "data"input_shape { dim: 1 dim: 1 dim: 1 dim: 20}layer { name: "ip1" type: "InnerProduct" inner_product_param { num_output: 100 } bottom: "ip1" top: "ip2"}layer { name: "a1" type: "TanH" bottom: "ip1" top: "ip1"}layer { name: "ip2" type: "InnerProduct" inner_product_param { num_output: 10 } bottom: "ip1" top: "out"}layer { name: "a1" type: "TanH" bottom: "out" top: "out"}
![Page 16: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/16.jpg)
Tensorflow
w1 = tf.Variable(tf.random_normal([10, 100]))w2 = tf.Variable(tf.random_normal([100, 20]))b1 = tf.Variable(tf.random_normal([100]))b2 = tf.Variable(tf.random_normal([20]))
layer1 = tf.add(tf.matmul(x, w1), b1)layer1 = tf.nn.relu(layer1)layer2 = tf.add(tf.matmul(x, w2), b2)layer2 = tf.nn.relu(layer2)
![Page 17: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/17.jpg)
Keras
model = Sequential([ Dense(100, input_dim=10), Activation('relu'), Dense(20), Activation('relu'),])
![Page 18: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/18.jpg)
tiny-dnn
network<sequential> net;net << dense<relu>(10, 100) << dense<relu>(100, 20);
![Page 19: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/19.jpg)
tiny-dnn, another solution
auto net = make_mlp<relu>({10, 100, 20});
- modern C++ enable us to keep code simple- type inference, initializer list
![Page 20: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/20.jpg)
2.Simple syntaxExample: Convolutional Neural Networks
![Page 21: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/21.jpg)
Caffe prototxt
name: "LeNet"layer { name: "data" type: "Input" top: "data" input_param { shape: { dim: 64 dim: 1 dim: 28 dim: 28 } }}layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" param { lr_mult: 1 } param { lr_mult: 2 } convolution_param { num_output: 20 kernel_size: 5 stride: 1 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" pooling_param { pool: MAX kernel_size: 2 stride: 2 }}layer { name: "conv2" type: "Convolution" bottom: "pool1" top: "conv2" param { lr_mult: 1 } param { lr_mult: 2 } convolution_param { num_output: 50 kernel_size: 5 stride: 1 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "pool2" type: "Pooling" bottom: "conv2" top: "pool2" pooling_param { pool: MAX kernel_size: 2 stride: 2 }}layer { name: "ip1" type: "InnerProduct" bottom: "pool2" top: "ip1" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 500 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "relu1" type: "ReLU" bottom: "ip1" top: "ip1"}layer { name: "ip2" type: "InnerProduct" bottom: "ip1" top: "ip2" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 10 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "prob" type: "Softmax" bottom: "ip2" top: "prob"}
}}layer { name: "conv2" type: "Convolution" bottom: "pool1" top: "conv2" param { lr_mult: 1 } param { lr_mult: 2 } convolution_param { num_output: 50 kernel_size: 5 stride: 1 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "pool2" type: "Pooling" bottom: "conv2" top: "pool2" pooling_param { pool: MAX kernel_size: 2 stride: 2 }}layer { name: "ip1" type: "InnerProduct" bottom: "pool2" top: "ip1" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 500 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "relu1" type: "ReLU" bottom: "ip1" top: "ip1"}layer { name: "ip2" type: "InnerProduct" bottom: "ip1" top: "ip2" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 10 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "prob" type: "Softmax" bottom: "ip2" top: "prob"}
param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 500 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "relu1" type: "ReLU" bottom: "ip1" top: "ip1"}layer { name: "ip2" type: "InnerProduct" bottom: "ip1" top: "ip2" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 10 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "prob" type: "Softmax" bottom: "ip2" top: "prob"}
}}layer { name: "relu1" type: "ReLU" bottom: "ip1" top: "ip1"}layer { name: "ip2" type: "InnerProduct" bottom: "ip1" top: "ip2" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 10 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "prob" type: "Softmax" bottom: "ip2" top: "prob"}
![Page 22: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/22.jpg)
Tensorflow
x = tf.Variable(tf.random_normal([-1, 28, 28, 1]))wc1 = tf.Variable(tf.random_normal([5, 5, 1, 32]))wc2 = tf.Variable(tf.random_normal([5, 5, 32, 64]))wd1 = tf.Variable(tf.random_normal([7*7*64, 1024]))wout = tf.Variable(tf.random_normal([1024, n_classes]))bc1 = tf.Variable(tf.random_normal([32]))bc2 = tf.Variable(tf.random_normal([64]))bd1 = tf.Variable(tf.random_normal([1024]))bout = tf.Variable(tf.random_normal([n_classes]))
conv1 = conv2d(x, wc1, bc1)conv1 = maxpool2d(conv1, k=2)conv1 = tf.nn.relu(conv1)conv2 = conv2d(conv1, wc2, bc2)conv2 = maxpool2d(conv2, k=2)conv2 = tf.nn.relu(conv2)fc1 = tf.reshape(conv2, [-1, wd1.get_shape().as_list()[0]])fc1 = tf.add(tf.matmul(fc1, wd1), bd1)fc1 = tf.nn.relu(fc1)fc1 = tf.nn.dropout(fc1, dropout)out = tf.add(tf.matmul(fc1, wout), bout)
![Page 23: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/23.jpg)
Keras
model = Sequential([ Convolution2D(32, 5, 5, input_shape=[28,28,5]), MaxPooling2D(pool_size=2), Activation('relu'), Convolution2D(64, 5, 5), MaxPooling2D(pool_size=2), Activation('relu'), Dense(1024), Dropout(0.5), Dense(10),])
![Page 24: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/24.jpg)
tiny-dnn
network<sequential> net;net << conv<>(28, 28, 5, 1, 32) << max_pool<relu>(24, 24, 2) << conv<>(12, 12, 5, 32, 64) << max_pool<relu>(8, 8, 64, 2) << fc<relu>(4*4*64, 1024) << dropout(1024, 0.5f) << fc<>(1024, 10);
![Page 25: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/25.jpg)
1.Easy to introduce
2.Simple syntax
3.Extensible backends
![Page 26: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/26.jpg)
3.Extensible backendsCommon scenario1:
“We have a good GPU machine to train networks, but we need to deploy trained model into mobile device”
Common scenario2:
“We need to write platform-specific code to get production-level performance... but it’s painful to understand whole framework”
![Page 27: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/27.jpg)
3.Extensible backendsSome performance critical layers have backend engine
Layer API
backend::internal
pure-c++ code
backend::avx
avx-optimized code …backend::nnpack
x86/ARM
backend::opencl
GPU
Optional
![Page 28: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/28.jpg)
3.Extensible backends
// select an engine explicitlynet << conv<>(28, 28, 5, 1, 32, backend::avx) << ...;
// switch them seamlesslynet[0]->set_backend_type(backend::opencl);
![Page 29: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/29.jpg)
Model serialization (binary/json)
Regression training
Basic image processing
Layer freezing
Graph visualization
Multi-thread execution
Double precision support
Basic functionality
![Page 30: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/30.jpg)
Caffe importer (requires protobuf)
OpenMP support
Intel TBB support
NNPACK backend (same to caffe2)
libdnn backend (same to caffe-opencl)Extra modules
(requires 3rd-party libraries)
![Page 31: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/31.jpg)
Future plans
![Page 32: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/32.jpg)
- GPU integration- GPU backend is still experimental- cudnn backend
- More mobile-oriented- iOS/Android examples- Quantized operation for less RAM
- TensorFlow Importer- Performance profiling tools- OpenVX support
We need your help!
![Page 33: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/33.jpg)
User chat for QA:
https://gitter.im/tiny-dnn
Official documents:
http://tiny-dnn.readthedocs.io/en/latest/
For users
![Page 34: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/34.jpg)
Join our developer chat:
https://gitter.im/tiny-dnn/developers
or
Check out docs, and our issues marked as “contributions welcome”:
https://github.com/tiny-dnn/tiny-dnn/blob/master/docs/developer_guides/How-to-contribute.mdhttps://github.com/tiny-dnn/tiny-dnn/labels/contributions%20welcome
For developers
![Page 35: Deep learning with C++ - an introduction to tiny-dnn](https://reader030.vdocuments.us/reader030/viewer/2022013117/58f9ad8e760da3da068b997f/html5/thumbnails/35.jpg)
code: github.com/tiny-dnn/tiny-dnnslide: https://goo.gl/Se2rzu