visual processing sparks a new class of processors · videantis con!dential | copyright videantis...

Copyright videantis GmbH videantis confidential |

Visual processing sparks a new class of processors

[email protected]

Copyright videantis GmbH videantis confidential | Copyright videantis GmbH videantis confidential |

People like to capture visual information 19 July 2015 10


People like to consume visual information 19 July 2015 11

• Global Internet Video Traffic   64% of consumer traffic in 2014   62% in 2013   57% in 2012

• Global Mobile Video Traffic   55% of mobile traffic in 2014   52% in 2013   50% in 2012

Mary Meeker’s Internet trends: http://www.kpcb.com/internet-trends


Display 19 July 2015 12

1995 2005 2015 6x 8x


Capture 19 July 2015 13

1995 2005 2015 6x 8x

Apple Quicktake VGA

Nokia N90 2MPixel

Samsung S6 16Mpixel


Processing 19 July 2015 14

Intel Pentium 0.35um

Intel Pentium D 65nm

Intel Core i5 14nm

~25x ~25x

More and more processing available per pixel


Altering pictures 19 July 2015 15

bokeh tilt-shift HDR


Computational photography – 360 cameras 19 July 2015 16


Computational imaging – camera arrays 20 July 2015 17


Computational imaging – lensless camera 20 July 2015 18


3D cameras 20 July 2015 19

• Time of Flight • Structured light • Stereo cameras • Structure from Motion


cameras being used more often for machine vision

19 July 2015 20


Driverless vehicles

Life saver, time saver, cost saver

Today >2020

19 July 2015 21


Driverless vehicles

WHO: 1.2M deaths per year world-wide

Today >2020


19 July 2015 22


Driverless vehicles

USA: ~5 years of your life in a car

Today >2020


19 July 2015 23


Driverless vehicles

USA insurance: ~$0.10/mile

Today >2020


19 July 2015 24


Driverless vehicles

90% autonomous: doubles road capacity

Today >2020


19 July 2015 25


Only autonomous vehicle today… 19 July 2015 26


Hype cycle

"Hype-Cycle-General" by NeedCokeNow

20 July 2015 27


Big leap from current cars to driverless

Legislation, R&D, maps, V2V/V2I, … Two key components: image sensors and image processing architectures

… and change will be gradual

19 July 2015 28


Automotive camera markets 19 July 2015 29

•  1 wide angle camera •  Viewing: rear view, dewarp •  Analytics: object detection, distance, etc

Rear view camera

•  2-4 cameras •  Viewing: surround / top view, dewarp, stitch •  Analytics: automatic parking assist, object detect

Surround and mirror cameras

•  1 or 2 cameras •  Viewing: not needed •  Analytics: monitoring driver status

Driver monitoring

•  1-3 cameras •  Analytics: lane detection, pedestrian detection,

vehicle detection, traffic sign recognition, headlight control

Front camera


Capture: 3D camera (e.g. Google Tango)

measure environment

gesture interfaces

augmented reality

20 July 2015 30


Microsoft Kinect 20 July 2015 31


Searching and indexing 19 July 2015 32


Apple’s fingerprint recognition 20 July 2015 33


Capture – always on

Dashcam Google Glass

20 July 2015 34


Capture – rugged

Camera pil GoPro

20 July 2015 35


Capture: fly and follow 20 July 2015 36


Failures 20 July 2015 37

• Android’s face unlock • Samsung Galaxy’s gesture interfaces • Amazon Fire’s eye tracking • Amazon’s camera-based search app • Google Goggles • Google Glass


http://image-sensors-world.blogspot.nl/2015/01/yole-updates-cmos-sensor-market-reports.html

Image sensors outgrow semi industry by 2x (8-10% CAGR)

20 July 2015 38


What does visual processing look like?

19 July 2015 39


Feature Detection

1.  Sobel in x 2.  Sobel in y 3. Derivative calc 4. Box Filter 5. Harris calc

6. Max location 7. Threshold 8. Dilate 9.  Select

3 3 2

3 3 2

2 2

M = w(x, y)Ix2 Ix Iy

Ix Iy Iy2

!

"

##

$

%

&&

x,y∑

(λ0,λ1) = eigenvalues(M )

λ1

λ0

λ0 ≈ λ1big

λ1 >> λ0

λ0 >> λ1λ0 ≈ λ1small

“edge”

“corner”

“edge”

“flat”

Find edges in horizontal and

vertical directions

Find edges in horizontal and

vertical directions

1

Two strong gradient

directions? ! found corner

2

Select corners R > threshold K (local maxima)

3

20 July 2015 40


HOG/SVM Pedestrian Detection 19 July 2015 41

Grayscale image

Multiscale image pyramid

Gamma normalization

Gradient calculation (angle and magnitude)

Histogram per block

SVM per window position

Non-max suppression

128

64

16 16

8 8 8

8

4 histograms of 9 cells

Gradient direction & magnitude

Dalal & Triggs, “Histograms of Oriented Gradients for Human Detection”, INRIA (France), 2005

Feature vector: •  105 (16x16 positions) •  x4 (8x8 blocks) •  X9 (histogram bins) ! 3780 elements Multiply SVM vector ! pedestrian yes/no

1

2

3

4

5

6

7


•  Binocular

•  Stereopsis •  Convergence

•  Monocular

•  Motion parallax •  Depth from motion •  Kinetic depth effect •  Aerial & curvilinear

perspective, size, accommodation, occlusion, texture gradient, lighting and shading, defocus blur, elevation

Perceiving 3D

Structure from

motion

Humans primarily use monocular vision

to sense depth

20 July 2015 42


19 July 2015 43

Structure from motion algorithm

+ camera origin and direction

+ calibrated camera

Structure from Motion

Feature detect and track in 2D image domain

  Calculates camera pose and 3D point cloud of the scene using a single standard 2D camera

  Target applications: automated parking, obstacle detection, object segmentation, etc

3D point cloud


Deep learning / convolutional neural nets 20 July 2015 44


Machines are beating humans at identifying objects

20 July 2015 45

http://www.sbs.com.au/news/article/2015/06/22/comment-dreams-googles-ai-are-equal-parts-amazing-and-disturbing


•  50% of the brain used for vision •  Body uses 100W •  Brain consumes 20W •  ! about 10W for vision analysis

Half our brain is used for visual processing

•  Challenge: beat the human •  Build machines that are

faster, safer, cheaper, last longer, more accurate, etc.

20 July 2015 46


Half of mobile SOCs for visual processing 19 July 2015 47

Samsung Exynos 5 die shot



Samsung Exynos 5 die shot

50M+ lines of

code (Android

is 30M)

<1M LOC

10K

~100K


mobile

Computer vision

Video coding

home automotive

Target markets 19 July 2015 49

Processor IP for visual processing


What does videantis do? 19 July 2015 50

Low-power, high-performance video/vision processor IP


10 years of videantis – company highlights

  Customers in Europe, US, Asia   Close ties to automotive algorithm ISVs,

semis, Tier 1s, and OEMs Fourth generation scalable unified video/vision processing IP Lowest power, highest performance video/image processing platform

2004 2006 2008 2010 2012 2014

10 years In business

1st Video IP Licensee

1st Vision

Licensee

1st Automotive

Volume

Deloitte Fast 50

3rd Gen Chip

1st Gen Chip

2005 2007 2009 2011 2013

1st Automotive

Licensee

1997 ..

R&D at Leibniz University

2nd Gen Chip

20 July 2015 51


• Heterogenous v-MP4280HDX arch:   4th generation (v-MP4000HDX series)   2 v-SP cores •  Enables bitstream processing high

quality video encode and decode   8 v-MP cores •  High-performance VLIW/SIMD

computer vision and video codec signal processing cores

• Software programmable video/vision:   H.264 encode or decode (and 15

other video codecs) •  1080p60, high-quality, low delay

  High-performance vision processing •  >384GOPS •  >64GMAC/s

videantis unified video/vision processor 19 July 2015 52

v-SP and v-MP for H.264 video codec

8x v-MP processor for vision processing

Highly-optimized, unified, programmable architecture for vision and video processing


Architecture for computer vision processing CPU GPU Imaging DSP videantis

ILP Superscalar

power hungry undisclosed Needs CPU

>4-issue NOPs and code size increase

2-issue VLIW right trade off

SIMD 128-bit

requires second pipeline, RF, etc.

wide array control flow not efficient

>256-bit SIMD control flow not efficient

64/128-bit right trade off

Multicore

1-4 cores cache coherency

introduces overhead

Many cores with many restrictions

1 core not enough

performance

1-8+ cores scales: low to high end apps

Processor frequency

2GHz+ long pipelines

cause overhead

~1GHz medium/long

pipelines

500MHz-1GHz medium pipeline

500MHz-1GHz medium pipeline

Cache & DMA

multi-level caches

multi-level caches

cache single DMA

No I/D caches DMA each core

19 July 2015 53


Videantis: more performance than 192-core GPU 19 July 2015 54

NVIDIA K1 Algorithm: HOG/SVM Pedestrian detect

Videantis v-MP4280HDX Algorithm: HOG/SVM Pedestrian detect

Performance: VGA at 3.4fps

Performance: VGA at >40fps

Power: 4W+ 1W/frame

Power at 40fps: 40mW ~1mW/frame

• Same algorithm

• >10x performance

• <1000x lower power

http://youtu.be/nCoMaLD56wQ http://youtu.be/grwzV2Umx-U



videantis 10-core video DSP •  Computer vision •  4K video coding •  Computational photography Samsung Exynos 5 die shot


Conclusion 19 July 2015 56

• Visual computing field is growing quickly   Computer vision drives many new applications

• Visual computing processor is a fragmented field today   CPU, GPUs  Hard-wired  DSPs, vision processors

• Time is right for new class of processors that are specifically designed for visual processing

Thank you !

visual processing sparks a new class of processors · videantis con!dential | copyright videantis...

Documents