iotg/vmc/vision products/icv novak alexander...change order • reshape • permutation • concat +...
TRANSCRIPT
![Page 1: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/1.jpg)
IOTG/VMC/Vision Products/ICV
Novak Alexander
![Page 2: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/2.jpg)
Internet of Things Group 2
Agenda• What is network optimization ?
• Tradition execution path
• Graph representations
• General optimizations
• Network compilers
• Results
![Page 3: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/3.jpg)
![Page 4: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/4.jpg)
Internet of Things Group 4
Problem statement
Neural Network model
General limitations
• Performance
• Accuracy
• Size of network
• Number and type of layers
HW limitations
• Memory
• Computing power
• Available layers
• HW accelerators and special units
+ + =Optimized
Neural Network model
![Page 5: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/5.jpg)
![Page 6: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/6.jpg)
Internet of Things Group 6
Tradition execution path: • Graph representation
• One of the popular CNN framework :TensorFlow, Keras, PyTorch, Caffe, MXNet, Caffe2, OpenVINO, etc
• Optimized backend
• HW specific optimized librariesCuBlas, MKL-DNN, CLDNN
• Full graph: transformations implemented in framework.Layout transform elimination, layer fusion, memory management
• New platform enablement -> Integration of layer library and framework tuning
![Page 7: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/7.jpg)
![Page 8: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/8.jpg)
Internet of Things Group 8
Graph representations: General
• Include different nodes and edges
• We can build some additional
graphs (interference graph, tiles graph etc)
• Optimizations on graph
![Page 9: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/9.jpg)
![Page 10: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/10.jpg)
Internet of Things Group 10
General optimizations on graph: Merge layers
• Convolution + Scale
• Convolution + BatchNormalization
• Convolution + ReLu
• Normalization
BatchNormalization
weights
biases
Scale OR
weights`
biases`
Convolution
Convolution`
![Page 11: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/11.jpg)
Internet of Things Group 11
General optimizations on graph: Replace and Change order
• Reshape
• Permutation
• Concat + Smth
• Pool and Conv
Concat
Pool
Pool Pool Pool
Concat
![Page 12: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/12.jpg)
![Page 13: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/13.jpg)
Internet of Things Group 13
HW specific optimizations: General
• Some operations works faster than others
• Convolution
• Merged operations and try express smth to HW op
Deconvolution = Upsampling Convolution+
For example:
![Page 14: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/14.jpg)
Internet of Things Group 14
HW specific optimizations: Memory optimizations
• Devices has different types of memory (cache, fast memory etc)Memory is very limited resource
• Optimal allocation is hard task
• Tiling
• Special allocators
• Interference graph
• Inplace operations
![Page 15: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/15.jpg)
![Page 16: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/16.jpg)
Internet of Things Group 16
Networks performance (FPS)Network Without
optimizationsGraph
optimizationsHW
optimizationsHW + graph
optimizations
GoogleNet V1 12.01 14.48 84.11 89.08
GoogleNet V2 10.1 11.5 79.40 84.65
Yolo Tiny V1 15.34 17.73 51.01 68.35
Yolo Tiny V2 9.29 10.42 41.40 49.83
SqueezeNet 1.1 47.32 51.7 248 272.10
VGG 16 2.01 2.26 7.9 10.2
ResNet 18 13.5 16.3 93.09 93.99
ResNet 50 6.5 8.6 30.76 31.93
MobileNet SSD 20.3 20.29 50.07 56.99
![Page 17: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/17.jpg)
![Page 18: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/18.jpg)
Internet of Things Group 18
Future Network compilers: Old way problems
• Graph representation is too high level
• It knows nothing about the backend
• optimizations are limited
• Topologies enablement and layer fusion is limited by vendor specific library
• Problems with custom layers integration
![Page 19: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/19.jpg)
Internet of Things Group 19
Future Network compilers
• Compiler based approach removes meaning of the layer
• Compiler defines tensor as a native data type
• Compiler knows HW backend specifics
• optimization choices are based on the cost model
• Assumption: convolution defines speed of the topology– the rest is memory bound
![Page 20: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/20.jpg)
![Page 21: IOTG/VMC/Vision Products/ICV Novak Alexander...Change order • Reshape • Permutation • Concat + Smth • Pool and Conv Concat Pool Pool Pool Pool Concat. Internet of Things Group](https://reader034.vdocuments.us/reader034/viewer/2022042911/5f41799a02e99a14dc7fb218/html5/thumbnails/21.jpg)
Internet of Things Group 21
Links and Materials
• TVM (https://github.com/dmlc/tvm)
• PlaidML (https://github.com/plaidml/plaidml)
• DL coursera (https://www.coursera.org/learn/machine-learning)
• Yandex (https://www.coursera.org/learn/vvedenie-mashinnoe-obuchenie)