ROBOREALM INTRO FOR DPRG

FINDINGS OF A NOVICE

DABBLING WITH MACHINE VISION

CARL OTT 11 June 2016

ROBOREALM INTRO FOR DPRG

• What? / Why?

• Pricing

• vs. OpenCV vs. CMUCAM

• Architectural options

• Examples

• Framework / IDE

• APIs

• Inbound Interface Options

• Outbound Interface Options

• Resources

• Generic Recommendation for DPRG

• RoboRealm Tips & Tricks

CARL OTT 11 June 2016 22016 JUNE 11 – DPRG – RoboRealm Preso

WHAT? / WHY?

• Machine Vision development and runtime framework

• Lets you easily

• Create, Optimize and Use image processing scripts

• Process live & stored video & images

• Connect to a variety of

• Video, image & sensor sources

• Actuator, robot & API platforms

• Visualize & Measure algorithm execution & performance

• Process commands from stored scripts or external API calls

• Output data, files, video streams, commands or external API responses

• Pre-compiled processing modules match a large subset of OpenCV

CARL OTT 11 June 2016 32016 JUNE 11 – DPRG – RoboRealm Preso

PRICING

CARL OTT 11 June 2016 4

Get the latesthttp://www.roborealm.com/registration/purchase.php

VS. ALTERNATIVES

CARL OTT 11 June 2016 5

Pro Con It Depends

CMUCAM Plug-n-play

Easily embedable

Runs fast out of the box

Forced to use built-in camera Limited algorithm options & controls

RoboRealm Much more capable than CMUCAM

Self-Contained

Rapid prototyping

Pre-compiled image processing

GUI, Tutorials, Docs, Support, etc.

Forced to use Windows Much more DIY than CMUCAM

Framework drives design patterns

OpenCV More capable than RoboRealm

Wide OS & Language support

More embedable than RoboRealm

Large open source community

Bigger learning curve Even more DIY than RoboRealm

Granular options, controls, patterns, etc.

2016 JUNE 11 – DPRG – RoboRealm Preso

ARCHITECTURAL OPTIONS

1. Self contained RoboRealm scripts

• do everything you need within RoboRealm

• great way to get started

2. Core logic in RoboRealm scripts call external resources

• could be handy to have RoboRealm directly call an Arduino, or Lidar

3. Core logic in external programs call RoboRealm as needed

• great way to scale

• and hopefully focus on abstractions and coding styles which minimize

dependencies on proprietary aspects of RoboRealm

a) Process individual frames one at a time

b) Process video streams

• accept live or stored input

• rebroadcast processed video to a new stream

CARL OTT 11 June 2016 62016 JUNE 11 – DPRG – RoboRealm Preso

• Dice Counting http://www.roborealm.com/tutorial/Counting_Dice/index.php

• Hand Gestures http://www.roborealm.com/tutorial/Hand_Gestures/index.php

• Color Object Tracking http://www.roborealm.com/tutorial/color_object_tracking_2/slide010.php

• Object Segmentation http://www.roborealm.com/tutorial/Object_Segmentation/index.php

• Obstacle Avoidance http://www.roborealm.com/tutorial/Obstacle_Avoidance/slide010.php

• Path Planning http://www.roborealm.com/tutorial/Path_Planning/slide010.php

• And more at http://www.roborealm.com/tutorials.php

• Sidewalk following https://www.youtube.com/watch?v=Rbhd50XlnEk

• Motion Detection https://youtu.be/rbCNk61vXpI

• DPRG Line Following – Challenge Course

EXAMPLES

CARL OTT 11 June 2016 72016 JUNE 11 – DPRG – RoboRealm Preso

FRAMEWORK / IDE

• Some example modules http://www.roborealm.com/screenshots/index.php

CARL OTT 11 June 2016 82016 JUNE 11 – DPRG – RoboRealm Preso

• API http://www.roborealm.com/help/API.php (all licenses)

• DLL http://www.roborealm.com/help/DLL.php (commercial licenses only)

APIs: INBOUND INTERFACE OPTIONS

CARL OTT 11 June 2016 92016 JUNE 11 – DPRG – RoboRealm Preso

External Programs can call RoboRealm scripts

APIs: OUTBOUND INTERFACE OPTIONS

CARL OTT 11 June 2016 102016 JUNE 11 – DPRG – RoboRealm Preso

RoboRealm Scripts can call External Stuff

GENERIC RECOMMENDATIONSFOR PURPOSES OF DPRG ROBOT COMPETITIONS

• RoboRealm gives a great path to get started with Machine Vision

• to quickly explore different image processing strategies

• to explore algorithms beyond the stock capabilities of CMUCAM

• RoboRealm could be a great implementation option

• if you can include a Windows machine in the loop

• if you want to avoid some of the grubby details

which (I presume) come with using raw OpenCV

• Consider placing more complex main robot logic outside of RoboRealm

• treat RoboRealm as a specialized compute resource

• use one of the officially supported API/COM object interfaces e.g. via C or Python

(unless you have some reason or desire to dabble with raw sockets and XML)

• stay tuned for more on my ongoing exercise calling RoboRealm from node.js

• let's hope that somebody eventually writes a node.js library linking RoboRealm to Cylon.js, or a library

linking RoboRealm to ROS

CARL OTT 11 June 2016 112016 JUNE 11 – DPRG – RoboRealm Preso

ROBOREALM TIPS & TRICKS• Set maximum processing and preview frame rates in the options menu

• Learn about the processing pipeline model & other intro tips http://www.roborealm.com/help/Getting%20Started.php

• Learn how to use different tabs (i.e. different functions, each with a sequence of processing modules)

• Learn to group processing modules into functions, and how to pass control from one function to the next

Eg check out http://www.roborealm.com/help/Call.php

• Note that function tabs can be set to run always or run once

• Note that the pipeline appears event driven, and only executes when input data changes, eg when the keyboard

read module regularly checks for key presses or when the camera provides a new frame

• Be careful when reading the keyboard from within RoboRealm, because keypresses can get detected unexpectedly

parsed by both your code and the IDE, and thereby trigger unexpected actions…

• Restart the GUI clean every now and then

• or certainly when the tab icon is not shown

• => Hold cntrl when click the icon at startup

• Edit the raw .robo file to reorder function tabs or copy functions between files (it's straightforward XML)

• Use helper script extension modules within your RoboRealm project

• for math operations on variables

• or for more complex logic than the native operators

• Analyze performance with per module execution timing in the GUI

• Take your questions to the RoboRealm forum

• somebody has probably asked already, or if you can't find the answer, note that:

• STeven is really, really good about helping with well formed questions

CARL OTT 11 June 2016 122016 JUNE 11 – DPRG – RoboRealm Preso

RESOURCES

• RoboRealm

• Tutorials http://www.roborealm.com/tutorials.php

• Documentation http://www.roborealm.com/help/index.php

• Forum http://www.roborealm.com/forum.php

• Resource list http://www.roborealm.com/links.php

• Other Machine Vision references

• Computer Vision: Algorithms and Applications http://szeliski.org/Book/

• CS 143 Intro to Computer Vision - Brown http://cs.brown.edu/courses/cs143/

• 15-463 Computational Photography – CMU http://graphics.cs.cmu.edu/courses/15-463/2015_fall/

• 6.869 Advances in Computer Vision – MIT http://people.csail.mit.edu/torralba/courses/6.869/6.869.computervision.htm

• CSE 576 Computer Vision – U Washington https://courses.cs.washington.edu/courses/cse576/08sp/

• Aforge, .net framework to facilitate OpenCV http://www.aforgenet.com/

CARL OTT 11 June 2016 132016 JUNE 11 – DPRG – RoboRealm Preso