real-world vision systems design: challenges and techniques

Copyright © 2016 Itseez 1

Real-World Vision Systems Design

Challenges and TechniquesYury Gorbachev

May 3, 2016


• Computer vision experts, 10 years on market

• Maintainers of OpenCV library

• Own products

• ADAS algorithms suite

• Facence (Face detection, recognition and analysis)

• 3D Scanning (Itseez3D)

• Accelerated CV (Optimized Computer Vision functions)

• Consulting in CV area

• Lot of Proof of Concept work per customer requests

Itseez at Glance


• Specific handling of requirements for vision projects

• Underestimated role of data

• Compute platform selection fallacies

• Hardware assisted vision algorithms

• Planning for customer feedback

• Functionality for easier product maintenance

Topics to Cover


• Requirements for vision products require special handling

• Not always possible to predict data variation upfront

• Specific cases will appear during design or test

• Missed scenarios

• Lighting conditions, blur and glare effects, etc.

• Impact can be pretty substantial

• More scenarios to handle — increase in algorithm complexity

• Additional data could be required

Requirements for Vision Products


• Initial requirement — detect speed limits in Europe

• Standards describe signs pretty well (size, color, etc.)

• Pretty easy to put into requirements

• Some things are not so straightforward to predict though

Example — Speed Limit Sign Detection


• Computer vision heavily depends on data for training and test

• Innovative products require specific data

• Usually not available in public datasets

• Data collection stage is needed prior to algorithm design

• Collect videos/images, preferably with variation and target HW

• Annotate collected data. At first stages done manually

• This is not always well understood by non-vision related customers

• Not always correctly planned within a project

Good Dataset is a Major Part of Solution


• Statistics from road sign dataset used at Itseez ADAS

• Not considering different sign types, weather and lighting conditions

• Annotation will take even more

Example — Speed Limit Signs Dataset


• Plenty of options for compute platform selection

• GPU, DSP and SIMD-enabled SoCs

• Frequently hardware selection is separated from software design

• Could be even pre-selected already before project start

• FACT: Not all compute platforms fit all algorithms

• Separated selection usually results in suboptimal systems

• Higher power consumption & unnecessary complicated algorithms

Premature Compute Platform Selection


• Vectorization is commonly seen as a solution for most of the problems

• Some SoCs sport few vector units with low frequency and few cores

• Obvious choice, perfect fit for vision!

• Not really… Following algorithm will have almost no benefit from SIMD

Example — Suboptimal Hardware Selection

Keypoint

search

External

sensor

Detailed

search

(Cascade)ROIs Candidates

Candidate

analysis


• Optics and sensors should be selected for a given task

• Color vs. grayscale imagers, FOV vs. distance

• Some tasks are significantly simpler when depth information is available

• E.g., stairway detection for robotics

• Vision algorithm can benefit from other sensor systems

• IMU to consider motion and position (e.g., visual odometry)

• LIDAR like cheap sensors in ADAS to limit search range

• Consider cloud for heavy offline processing if no latency requirements

Missed Opportunities for HW Assisted Vision


• Algorithm working on RGBD is much easier comparing to RGB

• Consider following inputs

Example — Staircase Detection

RGBD image courtesy of: Reza Farid. Region growing planar segmentation for robot action planning


• Prototype algorithms prior to major work and HW selection

• Understand bottlenecks and challenges update requirements

• Perform HW selection considering algorithm specifics

• Plan for sync between HW and SW during development

Possible Vision Product Design Model

Algorithm

PoC design Prototype

results

Hardware

approach selection

Hardware

Arch.

Software

drop

Algorithm

development

Hardware

drop

HW

Design

Final

productOriginal reqs

Updated reqs


• Vision in products is still pretty new to consumers

• Designers of the product create it with some use cases in mind

• Consumers usually use them differently

• This gap need attention prior to product release

• Target group testing, people outside of CV area

• Leave some time for corrective actions

• Algorithm enhancements and changes

• Documentation updates, tutorial videos, etc.

Customer Acceptance Need Some Work


• Itseez3D application

• Freely available on iPad, needs sensor (Occipital)

• Just walk around a person and scan to get 3D model

• Consumer use cases needed attention

• Tracking from sensor SDK did not support scan interruption

• Own enhanced tracking was implemented

• UI enhancements and video tutorials were done

• Currently about 300 scans per day

Example — Full Body Scanning


• Most vision products require calibration

• Re-calibration might be needed after some time

• Surveillance cameras change position due to wind/shaking

• Focus/visibility loss in optics due to harsh conditions

• Product tampering

• E.g., cover optics intentionally for some time periods

• Always good to be able to detect those things in software

• Blur & sharp edges detection, circular buffer recording

Maintenance Issues in Vision Products


One Video Tells More Than Few Slides

• Two problems here

• Optics needs maintenance (algorithm works though)

• Low sales figures...


• Do not consider requirements as final

• Periodically update with findings, exceptions

• For complicated algorithms perform PoC and iterate planning

• Perform analysis of existing datasets plan for dataset collection

• Investigate/prototype algorithm prior to hardware selection if possible

• Consider additional sensors as help to vision algorithms

• Ask for customer feedback prior product release

• Place frame source checks in software to detect maintenance issues

Conclusion: Useful Hints


Q&A


• Itseez Web — http://www.itseez.com

• Itseez3D Scanning app — http://www.itseez3d.com

• Contact me: [email protected]

Resources and Contacts

http://www.itseez.com

http://www.itseez3d.com

mailto:[email protected]