fpl15 talk: deep convolutional neural network on fpga

A Deep ConvolutionalNeural Network Based on

Nested Residue Number System

Hiroki Nakahara1 Tsutomu Sasao2

1Ehime University, Japan2Meiji University, Japan

1

Outline

• Background

• Deep convolutional neural network (DCNN)

• Residue number system (RNS)

• DCNN using nested RNS (NRNS)

• Experimental results

• Conclusion

2

Background

• Deep Neural Network

– Multi-layer neuron model

– Used for embedded vision system

• FPGA realization is suitable for real-time systems

– faster than the CPU

– Lower power consumption than the GPU

– Fixed point representation is sufficient

• High-performance per area is desired

3

Deep Convolutional Neural Network (DCNN)

4

Artificial Neuron

+

x0=1x0=1

x1

x2

xN

...

w0 (Bias)

w1

w2

wN

f(u)u y

xi: Input signalwi: Weightu: Internal statef(u): Activation function (Sigmoid, ReLU, etc.)y: Output signal

5

y f (u)

u wixii0

N

Deep Convolutional Neural Network(DCNN)for ImageNet

• 2D convolutional layer, pooling layer, and fully connection layer

6

2D Convolutional Layer• Consumes more than 90% of the computation time

– Multiply-accumulation (MAC) operation is performed

7

zij yij xim, jnwmnn0

K1

m0

K1

xij: Input signalyij : Biaswmn: WeightK: Kernel sizezij: Output signal

K

K

FPGA

Realization of 2D Convolutional Layer• Requires more than billion MACs!• Our realization

– Time multiplexing– Nested Residue Number System(NRNS)

8

Off‐chip Memory

** **

++

** **

++

++

** **

++

** **

++

++

BRAMBRAM BRAMBRAM

FPGA

Off‐chip Memory

** **

BRAMBRAM

****

** ****

** **

BRAMBRAM

****

** ****

** **

BRAMBRAM

****

** ****

** **

BRAMBRAM

****

** ****

➔ConverterConverter ConverterConverter ConverterConverter ConverterConverter

ConverterConverter ConverterConverter ConverterConverter ConverterConverter

Residue Number System(RNS)

9

Residue Number System (RNS)

Defined by a set of L mutually prime integer constants 〈m1,m2,...,mL〉

No pair modulus have a common factor with any other

Typically, prime number is used as moduli set

An arbitrary integer X can be uniquely represented by a tuple of L integers (X1,X2,…,XL), where

Dynamic range

10

)(mod ii mXX

M mii1

L

Parallel Multiplication

Multiplication on RNS

Moduli set〈3,4,5〉, X=8, Y=2

Z=X×Y=16=(1,0,1)

X=(2,0,3), Y=(2,2,2)

Z=(4 mod 3,0 mod 4,6 mod 5)

=(1,0,1)=16

11

Binary2RNS Conversion

RNS2Binary Conversion

➔ ➔

Binary2RNS Converter

12

X mod 2 mod 3 mod40 0 0 01 1 1 12 0 2 23 1 0 34 0 1 05 1 2 1

➔

13

00 01 10 11

00011011

0111

1100

0111

1100

X1=(x1, x2)

X2=(x3, x4)

h(X1) ０ ０１ １

x1 0 0 1 1x2 0 1 0 1

h(X1) 0 1 0 1

0 100 0 101 1 110 1 011 1 0

x3,x4

h(X1)

Functional Decomposition

24x1=16 [bit] 22x1+23x1=12 [bit]

Column multiplicity=2

Bound variables

Freevariables

Decomposition Chart for X mod 3

14

000 001 010 011

00011011

0120

1201

2012

0120

X2=(x3, x4, x5)

X1=(

x1, x

2)

100 101 110 111

1201

2012

0120

1201

0 mod 3 = 01 mod 3 = 12 mod 3 = 23 mod 3 = 04 mod 3 = 15 mod 3 = 26 mod 3 = 07 mod 3 = 18 mod 3 = 29 mod 3 = 0

10 mod 3 = 1

…

Freevariables

Bound variables

Decomposition Chart for X mod 3

15

0 1 2

00011011

0120

1201

2012

X2=(

x3, x

4, x5

)X 1

=(x1

, x2)

0 mod 3 = 01 mod 3 = 12 mod 3 = 23 mod 3 = 04 mod 3 = 15 mod 3 = 26 mod 3 = 07 mod 3 = 18 mod 3 = 29 mod 3 = 0

10 mod 3 = 1

…

Fre

eBound

x3 0 0 0 0 1 1 1 1x4 0 0 1 1 0 0 1 1x5 0 1 0 1 0 1 0 1

h(X2) 0 1 2 0 1 2 0 1

Binary2RNS Converter

16

LUT cascade for X mod m1

LUT cascade for X mod m2

BRAM

BRAM

BRAM

RNS2Binary Converter (m=30)

17

x1 y10 01 15

x2 y20 01 102 20

x3 y30 01 62 123 184 24

Mod mAdder

Mod mAdder

carry

carry

Problem• Moduli set of RNS consists of mutually prime numbers

– sizes of circuits are all different

• Example: <7,11,13>

18

6‐inputLUT

8‐inputLUT

8‐inputLUT

34

4

443

3

4

4

Binary2RNSConverter

byBRAMs

RNS2BinaryConverter

byDSP blocksand BRAMs

➔ ➔

DCNN using Nested RNS

19

Nested RNS

• (Z1,Z2,…,Zi,…, ZL) (Z1,Z2,…,(Zi1,Zi2,…,Zij),…, ZL)

• Ex: <7,11,13>×<7,11,13>

<7,<5,6,7>11,<5,6,7>13>×<7,<5,6,7>11,<5,6,7>13>

20

1. Reuse the same moduli set

2. Decompose a large modulo into smaller ones

Original modulus

➔

Example of Nested RNS• 19x22(=418) on <7,<5,6,7>11,<5,6,7>13>

19×22

=<5,8,6>×<1,0,9>

=<5,<3,2,1>11,<1,0,6>13>×<1,<0,0,0>11,<4,3,2>13>

=<5,<0,0,0>11,<4,0,5>13>

=<5,0,2>

=418

21

Modulo Multiplication

Bin2RNS on NRNS

RNS2Bin

Binary2NRNS Conversion

Realization of Nested RNS

22

<5,6,7>2Bin

Bin2<7,11,13>

3

<7,11,13>2Bin

<5,6,7>2Bin

Bin2<5,6,7>

Bin2<5,6,7>

6‐inputLUT

6‐inputLUT

6‐inputLUT

6‐inputLUT

6‐inputLUT

6‐inputLUT

6‐inputLUT

Bin2<7,11,13> Bin2

<5,6,7>

Bin2<5,6,7>

44

3

44

3333

3

3

Binary2NRNS

NRNS2Binary

Realized by BRAMs                      LUTs      BRAMs and DSP blocks   

Moduli Set for NRNS

• Conventional RNS (uses 23 moduli)

<3,4,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,71,73,79,83>

• Applied the NRNS to moduli that are greater than 15

<3,4,5,7,11,13,

<3,4,5,7,11,13>17,

<3,4,5,7,11,13>19,

<3,4,5,7,11,13,<3,4,5,7,11,13>17>23,

<3,4,5,7,11,13,<3,4,5,7,11,13>17>29,

…, <3,4,5,7,11,13,<3,4,5,7,11,13>17>83>

23All the 48-bit MAC operations are decomposed into 4-bit ones

DCNN Architecture using the NRNS

24

...

16 parallel modulo mi2D convolutional units

...

...

. . .

BRAM BRAM BRAM...

BRAM BRAM BRAM...

BRAM BRAM BRAM...

. . .

Parallel Bin2NRNSConverters

Tree‐based NRNS2BinConverters

Sequencer             

External DDR3SODIMM

DDR3 Ctrl.DDR3 Ctrl.On‐chipMemory

RNS2Bin

RNS2Bin

RNS2Bin

RNS2Bin

RNS2Bin

......

...

...

Experimental Results

25

Implementation Setup

• FPGA board: Xilinx VC707

– FPGA: Virtex7 VC485T

– 1GB DDR3SODIMM

(Bus@800MHz, 64 bit width)

• Realized the pre-trained

ImageNet by Convnet2

– 48-bit fixed precision

• Synthesis tool: Xilinx Vivado2014.1

– Timing constrain: 400MHz

26

Comparison with Other Implementations

27

Precision Max.Freq.[MHz]

FPGA Performance[GOPS]

Performanceper area[GOPS/

Slice x 10‐4]ASAP2009 16bit fixed 115 Viretex5 LX330T 6.7 1.3

PACT2010 ‐‐‐ fixed 125 Viretex5 SX240T 7.0 1.9

FPL2009 48bit fixed 125 Spartax3A DSP3400 5.3 2.2

ISCA2010 48bit fixed 200 Virtex5 SX240T 16.0 4.3

ICCD2013 ‐‐‐ fixed 150 Virtex6 LVX240T 17.0 4.5

FPGA2015 32bit float 100 Virtex7 VX485T 61.6 8.1

Proposed 48bit fixed 400 Virtex7 VX485T 132.2 25.2

Conclusion

• Realized the DCNN on the FPGA

– Time multiplexing– Nested RNS

• MAC operation is realized by small LUTs

• Functional decomposition are used as follows:

– Bin2NRNS converter is realized by BRAMs

– NRNS2Bin converter is realized by DSP blocks and BRAMs

• Performance per area (GOPS/Slice)– 5.86 times higher than ISCA10’s

28