energy efficient and programmable architecture for...

Energy Efficient and Programmable Architecture

for Wireless Vision Sensor Node

PhD Thesis presentation

MUHAMMAD IMRAN

Faculty of Science, Technology and Media,

Mid Sweden University, Sweden.

PHD THESIS ERRATA On page 6, the correct sentence is FLASH based

FPGA offers lower operating frequency.

On page 52, Figure 4-8 needs to be replaced with Fig.10

of paper IV.

On page 74, StrategyA3 output data is 421 bytes and for

StrategyA4 the output data is 411 bytes.

On page 91, the timing is 235 msec instead of 215 msec.

On page 91-92, the supply voltage is 3.6 instead of 3.5.

On page 102, paper VII and VIII paper numbering should

be interchanged.

2013-11-11 M. Imran, PhD thesis errata 2

2013-11-11 M. Imran, PhD presentation 3

OUTLINE Introduction

Challenges

Architectures analysis

Data reduction techniques

VSN with SRAM based FPGA

Conclusion


INTRODUCTION

2013-11-11 M. Imran, PhD presentation

Surveillance Machine vision

Camera theory research in 5th century.

Camera technology started in 20th century.

Use of camera for surveillance started in 1942.

5

TOWARDS AUTOMATION


Surveillance

Automated Surveillance Automated Machine inspection

Machine Inspection

6

CAMERA FOR INTERNET OF THINGS

Remote meter reading


SMART CAMERA The technological development have enabled camera

to integrate different components.


Image sensor and Lighting

Processing unit and Memory

Controlling unit and transciever

Power supply

8

COMMERCIAL SMART

CAMERAS


Manufacturer Product Processor Power FPS

Texus

Instruments NI 1722

400 MHz Freescale

PowerPC 11 W 60

Datalogic, A20 series 1,334 MIPS 12 W 60

Matrox imaging GT300 1.6 GHz processor 10 W 110

NI1722 A20 GT300

9

SMART CAMERAS BY

RESEARCH GROUPS


VSN Processing platforms Power FPS Applications

Firefly

Mosaic

ARM7TDMI (32-bit), TMEL

Atmega1281, 572.3 mW 5.2 Assisted living

MeshEye AT91SAM7S (32-bit) 290 mW 10 Surveillance

MircreEye ATMEL FPSLIC SoC (FPGA

plus microcontroller) 500 mW 15 People detection

Binary

Sensor

Flash-based FPGA, IGLOO

M1-AGL600 231 mW 5 People counting

Firefly MeshEye MireEye

10

EXISTING APPROACHES Generally researchers employ two approaches.

– Capture and process locally and transmit final results.

– Capture and transmit to server for processing.


Capturing data

Transmit data

All required vision tasks

ResultsReceive

data

VSN Server

Capturing data

Transmit results

All required vision tasks

Receive results

VSN Server

11

Partitioning processing load

among

– Software

– Hardware

– Server.

VSN ARCHITECTURES

ANALYSIS


TEST CASES Particle detection

Bird detection

People counting

Remote meter reading


ALGORITHM FLOW


MorphologySegment

Compress

Capture

Remove Bubble

Radio

Pre Processing

LabellingFeature extract

Remove Bubble

14

STRATEGY16 Data sent is approx. 114 Bytes


MorphologySegment

Compress

Capture

Remove Bubble

Radio

Pre Processing


Remove Bubble

SegmentCapturePre

Processing

Analysis

FPGA

SERVER

µ-controller

Transmit

VSN

MorphologyRemove Bubble

LabellingExtract

Features

15

STRATEGY36 Data sent is approx. 250KB


MorphologySegment

Compress

Capture

Remove Bubble

Radio

Pre Processing


Remove Bubble

Capture


LabellingExtract

Features

FPGA

SERVER

µ-controller

Transmit

VSN

Segment

Buffer

External memory

Pre-processing

16

STRATEGY13 Data sent is approx. 500 Bytes


MorphologySegment

Compress

Capture

Remove Bubble

Radio

Pre Processing


Remove Bubble

SegmentCapturePre

Processing

Remove Bubble

LabellingExtract

Features

FPGA

SERVER

Morphology

µ-controller

Transmit

VSN

Binary Compression

17

LIFETIME WITH SIMULATED POWER

FLASH BASED VSN

IMPLEMENTATION


SERVERWIRELESS VISION SENSOR NODE

RadioTransceiver

Processning

SENTIO32

Camera

FLASH based FPGA

RS

232

RadioTransceiver

Flash Memory

Light

AVR32Micro-

controller

Capture

Flash ControllerSPI

Pre-processing

Buffer

Vision processing

19

LIFETIME WITH MEASURED

POWER


0

1

2

3

4

5

6

7

Life

tim

e [Y

ears

]

Time between frames [minutes]

Strategy1

Strategy2

Strategy3

Strategy4

Strategy5

Strategy6

Strategy7

Strategy8

Strategy9

Strategy10

Strategy11

Strategy12

Strategy13

Strategy16

20

Strategy13

Strategy16

SegmentCapturePre

Processing

Analysis

FPGA

SERVER

µ-controller

Transmit

VSN


LabellingExtract

Features

SegmentCapturePre

Processing

Remove Bubble

LabellingExtract

Features

FPGA

SERVER

Morphology

µ-controller

Transmit

VSN

Binary Compression

Strategy16

Strategy13

FLASH BASED VSN

ARCHITECTURE



RadioTransceiver

Processning

SENTIO32

Camera

FLASH based FPGA

RS

232

RadioTransceiver

Flash Memory

Light

AVR32Micro-

controller

Capture

Flash ControllerSPI

Pre-processing

Binary compression

Morphology

Segmenta-tion

21

ENERGY CONTRIBUTION VSN


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.03

0.05

0.07

0.17

0.50

1.50

5.00

15.00

45.00

130.00

480.00

720.00

1440.00

Re

lati

ve e

ne

rgy

con

sum

pti

on

Sample period[m]

E_HARDWARE_PROC

E_COMM

E_SLEEP_PLATFORM

All required vision tasks on software

(Strategy34) All required vision tasks on hardware

(Strategy16)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.03

0.05

0.07

0.17

0.50

1.50

5.00

15.00

45.00

130.00

480.00

720.00

1440.00

Rel

ativ

e en

ergy

co

nsu

mpt

ion

Sample period [m]

E_HARDWARE_PROC

E_COMM

E_SLEEP_PLATFORM

0%10%20%30%40%50%60%70%80%90%100%

0.03

0.05

0.07

0.17

0.50

1.50

5.00

15.00

45.00

130.00

480.00

720.00

1440.00R

ela

tive

en

erg

y co

nsu

mp

tio

n

Sample Period [m]

E_SOFTWARE_PROC

E_COMM

E_SLEEP_SOFTWARE

Front end tasks and binary

compression on hardware (Strategy13)

22

Bi-level image coding

Bi-level video coding

DATA REDUCTION



RadioTransceiver

Processning

SENTIO32

Camera

FLASH based FPGA

RS

232

RadioTransceiver

Flash Memory

Light

AVR32Micro-

controller

Capture

Flash ControllerSPI

Pre-processing

Binary compression

Morphology

Segmenta-tion

BI-LEVEL IMAGE CODING

Investigated compression schemes

– G4

– G3

– JBIG2

– Rectangular

– GZIP

– GZIP_pack

– and JPEG_LS


0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

Ene

rgy

con

sum

pti

on

[m

J]E_HARDWARE_PROC

E_COMM

NEED FOR BI-LEVEL VIDEO

CODING Data reduction beyond limit of simple image coding.

Bi-level video codec is required but not available.

Coding complexity needs to be reduced.


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.03

0.05

0.07

0.17

0.50

1.50

5.00

15.00

45.00

130.00

480.00

720.00

1440.00

Rel

ativ

e en

ergy

co

nsu

mpt

ion

Sample period [m]

E_HARDWARE_PROC

E_COMM

E_SLEEP_PLATFORM

Strategy13

BI-LEVEL VIDEO CODEC Bi-Level video codec functionality

– Image coding

– Change coding

– ROI coding

– Change-ROI coding


1 2 3

1 2

Change

coded image

Change

coded image

3

ROI coded

image

5

4

ROI coded

image

Change-ROI

coded image

4

Change-ROI

coded image

5

(a)

(b)

Change coding

ROI coding G4

compression

Huffman and Run

length codes Mixer

ROI row runs

ROI imageHuffman

codes

8

Bi-level data

26

CHANGE CODING Only changes are coded.


Calculate black and

white runs

Previous frame

runs access

Produce Black/

white value upto

count value

FIFO

Reset

Clk

Data

Frame_sync

Line_sync

B_runs

W_runs

Synchronization

XOR

Data_Frame

Data_Frame-1

Full/Empty

Line_sync

Frame_sync

Data

Runs

Change coding

ROI coding G4

compression

Huffman and Run

length codes Mixer

ROI row runs

ROI imageHuffman

codes

8

Bi-level data

27

ROI CODING Area of image with objects is coded.


200

300

580

730

1244

1744

2400

0

1600

24

00

Change coding

ROI coding G4

compression

Huffman and Run

length codes Mixer

ROI row runs

ROI imageHuffman

codes

8

Bi-level data

28

ROI CODING


Runs Symbols

colour

Run length

coding (8 bits)

Run length

coding (9 bits)

200 Black 200,0* 200,0*

100 White 100 100

280 Black 0,25 280

656 Black 0,0,146 0,145

200

300

580

730

1244

1744

2400

0

1600

24

00

29

G4 COMPRESSION G4 used 2-Dimensional line by line coding.

In this, the position of changing picture elements rather

than alternating black and white runs are calculated.


Reference line

Coding line

a0 a1

b1 b2

a2

Change coding

ROI coding G4

compression

Huffman and Run

length codes Mixer

ROI row runs

ROI imageHuffman

codes

8

Bi-level data

30

1716 Bytes


250 Kbytes

500 Bytes

223 Bytes

125 Bytes

33 Bytes

Input Images

Image coding

Change coding

ROI coding

Change-ROI coding

Particle detection Meter reading Bird detection People counting

250 Kbytes

585 Bytes

203 Bytes

150 Bytes

65 Bytes

768 Kbytes

411 Bytes

249 Bytes

667 Bytes

517 Bytes

200 Kbytes

2060 Bytes

2552 Bytes

2605 Bytes

31

VSN WITH BI-LEVEL VIDEO

CODEC


SegmentPre-

processingCapture

Background image

Morphology

SENTIO32

Camera

FPGASPI

Buffer

RS

232

AVR32

Bi-level Video coding

LED ring

Radio

Battery

Memory controller

FLASH memory

32

COMPARISON OF SYSTEMS

Reduced energy consumption of 1.5 to 376 times.

Reduced output data by a factor of approximately 3 to 246.


Applications Systems E_Proc

(mJ)

E_Comm

(mJ)

Avg.

Output data

(bytes)

Bits/

pixel

Max.

Freq.

(FPS)

Particle

detection Published [8] 0.63 3.52 500 0.016 60

Proposed 2.04 0.38 33 0.001 100

Meter

reading Published [2] 430.3 207.9 6912 0.500 0.04

Published_scaled [2] 515.3 478.3 16000 0.500 0.03

Proposed 2.04 0.60 65 0.002 100

Bird

detection Conventional 6.03 4.9 703 0.007 33

Proposed 5.76 1.83 249 0.003 33

People

counting Published [3] 0.004 4.29 2048 2.000 15

Published_scaled [3] 0.08 27.06 51200 2.000 5

Proposed 1.57 11.6 1715 0.067 18

33

SRAM based FPGA for duty

cycling

Low complexity background

subtraction

FPGA BASED VSN: SENTIOF-CAM


LOW COMPLEXITY

BACKGROUND


Image sensorScaled down

Internal Memroy

Scaled up

Microcontroller/FLASH Memory

CAM/FLASH

-

First time or when needed

FPGA

Splitter

CAM/FLASHCurrent Pixels

Background Pixels

35

VSN ARCHITECTURE


Camera

FPGA

SP

I

AVR32

LED ring

Radio

Battery

Scaled down

Scaled up

Memory buffer

Flash memory

ROI

CC Mixer

Compress

Bi-level video codecBackground storage model

Flash controller

SegmentPre-

processingCapture Morphology

Front end tasks

36

SETUP: SENTIOF-CAM


A B C D

0 10 20 30 40 50 60 700

200

400

600

800

1000

1200

X: 64.5

Y: 0.09596

Time (ms)

Insta

nta

neous C

urr

ent

(mA

)

One cycle instantaneous current:meter reading

transmission

wakeup statesleep

state

sleep-to-wakeup state

FPGA

power-on

FPGA configurationprocessing

sleep

state

SENTIO32 SENTIOF-CAM GUI and received data FOV GUI

Measurement setup Instantaneous current for a duty cycle

37

CURRENT MEASUREMENT

FOR DUTY CYCLE


0 10 20 30 40 50 60 700

200

400

600

800

1000

1200

X: 64.5

Y: 0.09596

Time (ms)

Insta

nta

neous C

urr

ent

(mA

)

One cycle instantaneous current:meter reading

transmission

wakeup statesleep

state

sleep-to-wakeup state

FPGA

power-on

FPGA configurationprocessing

sleep

state

38

POWER AND TIME OF

DIFFERENT PROCESSES


Applications Config.

time

(ms)

Config.

power

(W)

Proc.

time

(ms)

Proc.

power

(W)

Comm.

Time

(ms)

Comm.

power

(W)

Total

energy

(mJ)

Particle detection 23.6

0.12

21

0.67 3.2 0.13 17.2

Meter reading 23.6

0.12

21

0.67 4.3 0.13 17.4

39

LIFETIME


0

2

4

6

8

10

12

14

Life

tim

e [y

ear

s]

Sample period [m]

PUBLISHED_PARTICLE

MEASURED_PARTICLE

PUBLISHED_METER

MEASURED_METER

40

An energy efficient and programmable VSN is developed and

implemented.

– Low complexity

– Generic architecture

To reduce communication energy on hardware implemented

VSN, a bi-level video coding is developed and implemented.

A VSN with proposed Bi-level video coding reduced energy

consumption by a factor of 1.5 to 376.

A VSN with SRAM based FPGA has been evaluated for duty

cycle applications.

– Depending on application a lifetime of 3.2 years with 37 kJ energy (AA).

CONCLUSION


FUTURE WORK This study can be extended for applications which require

greyscale or colour data for classification.

Bi-level video coding which handle movement of objects.

A taxonomy which has been proposed can be improved by

making it more exhaustive.

Wireless smart camera technology can be integrated for Internet

of things.


–Prof. Mattias O’Nils

–Najeem Lawal

–Prof. Bernhard Rinner

–Committee members

–All audience

THANKS!


http://www.orissadiary.com

http://lifeboat.com/ex/security .preserver

http://www.epicsysinc.com/bl og/machine_vision_history

http://edenprairieweblogs.org/scottneal/page/45/

http://www.marketingsavant.com/2011/11/getting-started-in-a-

social-media-career-part-i/

REFERENCES

2013-11-11 44 M. Imran, PhD presentation

energy efficient and programmable architecture for...

Documents