real-time hdr video - innovatefpga.com · team em076 real-time hdr video independent sp-team...

Team EM076

Real-time HDR video Independent SP-Team

Vladislav Sharshin

Andrey Papushin

Yelena Kirichenko

Team EM076


Vladislav Sharshin

Andrey Papushin

Yelena Kirichenko

EM076

Our project: Real-time HDR video

TEAM EM076

Vladislav Sharshin Andrey Papushin Yelena Kirichenko

EM076

Problem definition

Ordinary camera Human eye

The human eye considerably exceeds modern cameras so it’s impossible to transmit the image using technical devices

just the way a person being would see it

EM076Dynamic range of the human eye and ordinary camera

Problem definition

Dynamic range

EM076

Problem definition

Bright details lost

Bright details lost

Dark details lost

Dark details lostDynamic range

Dynamic range

EM076

Problem definition

Screenshot of a football match online broadcast between Costa Rica and Serbia, taken on a sunny day

Details in bright area

are lost in the frame

EM076

Problem definition

As we can see,

that football player

wearing a white

uniform can hardly

be seen

Screenshot of a football match online broadcast between Costa Rica and Serbia, taken on a sunny day

EM076

High dynamic range (HDR)

technique

+ =

High exposure

Low exposure

HDR

EM076

HDR video. Popular algorithm

making the first image

rearranging exposure

combining the two images =

HDR frame

making the second image

EM076

Disadvantages of the popular

HDR algorithm

• Reduction of the FPS by at

least 2 times

• Doubling of the image

Example of unsuccessful

use of HDR

while 1 camera is used

EM076

We decided to use two or more

cameras to produce HDR algorithm

High-level diagram of the device

EM076

We decided to use two or more

cameras to produce HDR algorithm

Device prototype

EM076

Implementation in the Intel SoC

fpga2sdram bridgehps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2

av

l_str

eam

pa

ralla

x elim

inatio

n

Raw2rgb

bilinear

interp

HDR

Tone

mapping

Raw2rgb

bilinear

interp

HDMI

TX

sdram

write

sdram

read

MU

X 1

conv

filter

screen

2x20 pin

GPIOS

M M

M S S

FPGA part

HPS space

raw_0

raw_1

rgb_0

rgb_1

mem_gamma

rgb_hdr

rgb_tm

32

hps_switch

64 32

div

_co

ef

shift_

coe

f

coef_

con

v

4

8

data addrsel

cam

rg

b_

ifc_

fb

rg

b_

ifc_

hd

mi

885x9

lwhps2fpga bridge

Gam

m co

rrectio

n

SCCB camera

config

hist calc

M

M

Y_comp

rgb_filter

MU

X 2

S

32

S

M

S

pa

ra

llax

_co

rr

R

G

B

Y

line_

req

uest

8

4

hist_switch

mem_avl_ifc

da

ta_

freq

_h

ist

EM076


2x20 pin

GPIO

FPGA part

HPS space

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

2x20 pin

GPIOS

M

FPGA part

HPS space

data addrsel

cam

SCCB camera

config

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

2x20 pin

GPIOS

M

FPGA part

HPS space

data addrsel

cam

SCCB camera

config

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

2x20 pin

GPIOS

M

FPGA part

HPS space

mem_gamma

32

data addrsel

cam

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

M

S

M

S

mem_avl_ifc

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

pa

rallax elimin

ation

2x20 pin

GPIOS

M

FPGA part

HPS space

mem_gamma

32

8

data addrsel

cam

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

M

S

M

S

pa

ralla

x_

corr

mem_avl_ifc

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

pa

rallax elimin

ation

Raw2rgb

bilinear

interp

Raw2rgb

bilinear

interp

2x20 pin

GPIOS

M

FPGA part

HPS space

raw_0

raw_1

mem_gamma

32

8

data addrsel

cam

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

M

S

M

S

pa

ralla

x_

corr

mem_avl_ifc

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

pa

rallax elimin

ation

Raw2rgb

bilinear

interp

HDR

Raw2rgb

bilinear

interp

2x20 pin

GPIOS

M

FPGA part

HPS space

raw_0

raw_1

mem_gamma

32

8

data addrsel

cam

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

M

S

M

S

pa

ralla

x_

corr

mem_avl_ifc

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

pa

rallax elimin

ation

Raw2rgb

bilinear

interp

HDR

Tone

mapping

Raw2rgb

bilinear

interp

2x20 pin

GPIOS

M

FPGA part

HPS space

raw_0

raw_1

mem_gamma

32

8

data addrsel

cam

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

M

S

M

S

pa

ralla

x_

corr

mem_avl_ifc

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

pa

rallax elimin

ation

Raw2rgb

bilinear

interp

HDR

Tone

mapping

Raw2rgb

bilinear

interp

MU

X 1

2x20 pin

GPIOS

M

FPGA part

HPS space

raw_0

raw_1

rgb_0

rgb_1

mem_gamma

rgb_hdr

rgb_tm

32

hps_switch

4

8

data addrsel

cam

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

M

S

M

S

pa

ralla

x_

corr

mem_avl_ifc

EM076


hps2fpga bridge

Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

pa

rallax elimin

ation

Raw2rgb

bilinear

interp

HDR

Tone

mapping

Raw2rgb

bilinear

interp

MU

X 1

conv

filter

2x20 pin

GPIOS

M

FPGA part

HPS space

raw_0

raw_1

rgb_0

rgb_1

mem_gamma

rgb_hdr

rgb_tm

32

hps_switch

div

_co

ef

shift_

coe

f

coef_

con

v

4

8

data addrsel

cam

885x9

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

M

S

M

S

pa

ralla

x_

corr

mem_avl_ifc

EM076



Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2av

l_stream

pa

rallax elimin

ation

Raw2rgb

bilinear

interp

HDR

Tone

mapping

Raw2rgb

bilinear

interp

sdram

write

MU

X 1

conv

filter

2x20 pin

GPIOS

M

M S

FPGA part

HPS space

raw_0

raw_1

rgb_0

rgb_1

mem_gamma

rgb_hdr

rgb_tm

32

hps_switch

64 32

div

_co

ef

shift_

coe

f

coef_

con

v

4

8

data addrsel

cam

rgb

_ifc_

fb

885x9

lwhps2fpga bridge

Gam

m correction

SCCB camera

config

hist calc

M

M

Y_comp

rgb_filter

MU

X 2

S

S

M

S

pa

ralla

x_

corr

R

G

B

Y

8

4

hist_switch

mem_avl_ifc

da

ta_freq

_h

ist

EM076



Cam 0

Cam 1

hps_register

SCCB_cam1

SCCB_cam0

con

vert2

av

l_strea

m

pa

rallax elimin

ation

Raw2rgb

bilinear

interp

HDR

Tone

mapping

Raw2rgb

bilinear

interp

HDMI

TX

sdram

write

sdram

read

MU

X 1

conv

filter

screen

2x20 pin

GPIOS

M M

M S S

FPGA part

HPS space

raw_0

raw_1

rgb_0

rgb_1

mem_gamma

rgb_hdr

rgb_tm

32

hps_switch

64 32

div

_co

ef

shift_

coe

f

coef_

con

v

4

8

data addrsel

cam

rgb

_ifc_

fb

rgb

_ifc_

hd

mi

885x9

lwhps2fpga bridge

Gam

m co

rrection

SCCB camera

config

hist calc

M

M

Y_comp

rgb_filter

MU

X 2

S

32

S

M

S

pa

ralla

x_

corr

R

G

B

Y

line_

requ

est

8

4

hist_switch

mem_avl_ifc

da

ta_freq

_h

ist

EM076

Graphic User Interface

FEATURES

•Cameras choosing

•Exposure changing

•Parallax fixing

•Gamma correction

•HDR mode

activating

• Filters choosing

•Histograming mode

activating

• Tone mapping

EM076

Additional features

Real image SHARPEN EDGE DETECTOR BLUR EMBOSS

Filters implementation

Our

realization

EM076

Additional features

HISTOGRAM PLOTTING

EM076

Advantages of FPGA

Why is FPGA the best option for this project?

1. FPGA allows us to make the necessary calculations parallel, which isimpossible to do on CPU. If required, we can connect 3, 4, 10 cameras,while RAW streams from all the cameras will be simultaneously convertedto RGB

2. FPGA allows us to use pipeline method for data flow processing, which isimpossible to do on CPU. Module calculations occur one by one with nofeedback. The data stream gets into the input and is transferred to theoutput (upon a certain latency) with no breakups.

3. FPGA design reconfiguration. We can replace the cameras with others;change output interface from HDMI to VGA, DVI, etc.; to reassign input andoutput pins and voltage standards, and so on.

EM076

Advantages of FPGA

Parallel data processing Pipelining

EM076

Opportunities to apply

Our work on the HDR real-time HD video

implementation can be successfully used for

• security systems,

• dashcams,

• traffic control cameras, or in cases of streaming

objects, moving with a different speed

• located against the light or in different shadows,

• live broadcasting from sport events by

professional cameras

EM076

RESULTS

The main aim of the project is to realize

HDR real-time HD video via Terasic

DE10-Nano KIT with Intel SoC (FPGA

Cyclone V + ARM A9), that is succeed.

We have realized some features

• Additional filters as well as real-time

video;

• Tone-mapping;

• PC remote control by “Ethernet“;

• Gamma-correction;

• Real-time color histogram for each color

channel, brightness histogram for each

camera’s data-stream and output stream.

EM076

Thank you for your attention

Team EM076


Vladislav Sharshin

Andrey Papushin

Yelena Kirichenko

real-time hdr video - innovatefpga.com · team em076 real-time hdr video independent sp-team...

Documents