to accelerate customer software algorithm · sdsoc effects: it's possible to make an...

© Copyright 2016 Co., Ltd. 1

A success story of leveraging SDSoCTM

to accelerate customer software algorithm

Agenda:

Corporate Profile

Story Summary

Story Detail

MPSoC DEMO

Wataru TakahashiManager


Corporate name

Office address

Capital

Stockholder

Founded

CEO

Employees

Business description

OKI IDS Co., Ltd. (OIDS)

3-1 Futaba-cho, Takasaki, Gunma-ken, Japan

50 million yen

OKI IDS Co.,Ltd. (100% owned by Oki)

March 10, 2014

Noriaki Anada

78Development, design, sales and consulting of firmware/hardwarefor multimedia (video, audio, communication)Product development for Medical instruments, Industrial instruments

Kusatsu Hot Spring Takasaki DarumaTomioka Silk Mill

The largest natural water flow ofall hot springs in Japan

Takasaki city is the top producer ofDaruma Doll in Japan

Registered asUnesco World Heritage in 2014


OKI IDS strongly supports customer�s product design with highly-versatilesolution products based the technologies of its specialty

TargetApplication

Solution

Technology


� Goal of the project was to achieve moving object detection system on ZC706.

� The system was designed to processes input video from camera in real time,and output specific signal when a moving object is detected.

� The customer already had entire algorithm developed on PC, and requestedOKI to implement it onto ZC706.

� In so doing, part of the processing needed to be hardware accelerated.

� SDSoC worked efficiently for hardware acceleration of the system.

� SDSoC enabled hardware acceleration of key processing of the system andgreatly contributed to customer satisfaction.

© Copyright 2016 Co., Ltd.

Zynq PL

DDR

5

The processing time for 1 frame at the this point was:

400 ms for 1000 feature points -> 2�3 fps

logiCVC

HDMIcamera

HDMIFMCboard

Zynq PS

logiWIN

Xylon's IP

ZC706

Moving Object Detection processing� Video intake� Moving Object Detection� Graphical overlay on the detected

object

New DMAC

OKI IDS DesignVIVADO


Corner Harris

Open CV version Corner Harris Software

void cornerHarris(const Mat& src, Mat& dst, int blockSize, int apertureSize, double k, intborderType=BORDER_DEFAULT)

Vivado HLS version Corner Harris

template<int blockSize,int Ksize,typename KT,int SRC_T,int DST_T,int ROWS,int COLS>void CornerHarris(hls::Mat<ROWS, COLS, SRC_T> &_src, hls::Mat<ROWS, COLS, DST_T> &_dst, KT k );

Auviz Corner Harris

template<int ROWS, int COLS, int DEPTH, int NPC, int IN_WW, int OUT_WW, int FILTERSIZE, int MAXPNTS,int PIPELINEFLAG>void auCornerHarris( hls::stream < AU_TNAME(IN_WW) > & _src_mat, hls::stream < AU_TNAME(OUT_WW) > & _dstlist, uint16_timg_height, uint16_t img_width,uint16_t _filter_width, uint16_t _block_width, uint16_t _nms_radius,uint16_t _threshold, uint16_t val, uint32_t*nCorners)


OpenCV / VivadoHLS / Auviz

OpenCV VivadoHLS Auviz

Software Hardware Hardware

Input format Mat Format Mat Format Mat Format

Output format Mat Format Mat Format List Format(xy coordinate list)

NonMaxSuppressionprocessing,pick up feat.point

Not Supported Not Supported Supported


The basic principles of Corner Harris

S( u, v ) = � w( x, y )[ I( x+u, y+v ) I(x, y)]2


S( u, v ) = � w( x, y )[ I( x+u, y+v ) I(x, y)]��2

Take taylor expansion about I( x+u, y+v ).Partial differentials of I are Ix , Iy then

S( u, v ) � � w( x, y )[ Ix (x, y)u Iy (x, y)v]��2

S( u, v ) �(u v) A ��

Put it as matrix expression;

A = � w(x, y)��Ix2 IxIyIxIy Iy2

Ix = ��

�Iy= ��

�



It can be determined by reviewing the eiganvalues, �1 �2.However the computational cost is high in this way.Instead, the following formula can be used to determine corner strength:

M det( A ) k trace A )2

x

y

Mfeature point



SDSoCTM


Making bitstream and interface library


Making application software

DMA andother interface

lib


Source example

void feature_detector( unsigned short *image_in,int *keypointlist,int rows,int cols,int offset,float kpara,float threshold_f,float threshold_f2,int *keypointnum )

{

myGoodFeature_sds( image_in,keypointlist,rows,cols,offset,kpara,threshold_f,threshold_f2,keypointnum );

}

#pragma SDS data mem_attribute( image_in:PHYSICAL_CONTIGUOUS )#pragma SDS data mem_attribute( keypointlist:PHYSICAL_CONTIGUOUS )#pragma SDS data mem_attribute( keypointnum:PHYSICAL_CONTIGUOUS )#pragma SDS data access_pattern( image_in:SEQUENTIAL )#pragma SDS data access_pattern( keypointlist:SEQUENTIAL )#pragma SDS data access_pattern( keypointnum:SEQUENTIAL )#pragma SDS data data_mover( image_in:AXIDMA_SG )#pragma SDS data data_mover( keypointlist:AXIFIFO )#pragma SDS data data_mover( keypointnum:AXIFIFO )#pragma SDS data copy( image_in[ 0:WIDTH*HEIGHT ] )

void myGoodFeature_sds( unsigned short image_in[ WIDTH * HEIGHT ],int *keypointlist,int rows,int cols,int offset,float kpara,float threshold_f,float threshold_f2,int *keypointnum )

{

//

[ HLS source code ]

//

//

HW InterfaceDescription

(DMA, FIFO, etc)Call point fromapp software


SDSoC effects:

It's possible to make an environment that enables to compare the variousresult easier.

Open CV Corner Harris Softwarevoid cornerHarris(const Mat& src, Mat& dst, int blockSize, int apertureSize, double k, intborderType=BORDER_DEFAULT)

Vivado HLS Corner Harristemplate<int blockSize,int Ksize,typename KT,int SRC_T,int DST_T,int ROWS,int COLS>void CornerHarris(hls::Mat<ROWS, COLS, SRC_T> &_src, hls::Mat<ROWS, COLS, DST_T> &_dst, KT k );

Auviz Corner Harristemplate<int ROWS, int COLS, int DEPTH, int NPC, int IN_WW, int OUT_WW, int FILTERSIZE, int MAXPNTS,int PIPELINEFLAG>void auCornerHarris( hls::stream < AU_TNAME(IN_WW) > & _src_mat, hls::stream < AU_TNAME(OUT_WW) > & _dstlist, uint16_timg_height, uint16_t img_width,uint16_t _filter_width, uint16_t _block_width, uint16_t _nms_radius,uint16_t _threshold, uint16_t val, uint32_t*nCorners)

SDSoC can correspond to thedifference in API easily.


Installed the Corner Harris Detector to zcu102's PL

Base design is Xilinx's TRD

16


V4L2 API

Feature DetectThread

View Thread

libdrm, etc Qt API,OpenGL API

Duplicator

Corner Harris

SelectFeature-point

CapturePipeline

DPDMA DisplayPipeline

Main Thread

x-y list

x-y list

DisplayPort

APU

PL

17


Thank you for yourattention!

18

to accelerate customer software algorithm · sdsoc effects: it's possible to make an...

Documents