vision recognition for mixed reality...
TRANSCRIPT
![Page 1: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/1.jpg)
Vision Recognitionfor Mixed Reality Applications
Richard Fung
October 2010
1
With annotations
![Page 2: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/2.jpg)
Vision Recognition System
1. Webcam Capture
2. Vision Recognition
3. Application
2
![Page 3: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/3.jpg)
Webcam
Some work better than others…
• Tried and tested– Design Interfaces Lab
(Dr. Igarashi)
– MIT Media Lab
• Characteristics– Wide angle lens
– Auto focus
– 800 x 600 @ 30 fps
– 1600 x 1200 @ 10 fpsLogitech C905QuickCam Pro for Notebooks
3
![Page 4: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/4.jpg)
Webcam
4
VirtualDub:
File > Capture AVI Device > DirectShowVideo > Capture Pin
![Page 5: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/5.jpg)
Markers
• Characteristics
– Square
– Continuous border
– Not rotationally symmetric
– Black border:
ARToolkit and variants
5
![Page 6: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/6.jpg)
Markers
• Pattern design
– From webcam
• ARToolkit/bin/mk_patt.exe
– Use existing patterns
• ARToolkit/patterns– Hiro
– Kanji
• Search the web
6
![Page 7: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/7.jpg)
Markers
• Issues– Size of marker
• further away = larger
– Complexity of pattern• False positives
– Reflection of light• Use felt instead of laser print
• Print at low toner option
7
![Page 8: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/8.jpg)
Vision Recognition System
1. Webcam Capture
2. Vision Recognition
3. Application
8
![Page 9: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/9.jpg)
Various Packages
• ARToolkit C++ open/proprietary
• ARToolkitPlus C++ open
• ARTag C++ proprietary
• NyARToolkit Java, C# open
• Goblin XNA C# open
• Igarashi Java, C# proprietary
9
![Page 10: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/10.jpg)
NyARToolkit
• Setup
– Calibration
– Create marker definition files
• Issues
– Incomplete
– Documentation in Japanese
10
![Page 11: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/11.jpg)
NyARToolkit
• Source code
– Distribution 2.5.2.1
– Added a new method: NyARDetectMarker.getCorners()
– Download from iLab Cookbook
11
![Page 12: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/12.jpg)
Dependencies
• Visual Studio 2008 or later
• Microsoft DirectX SDK 2002 to August 2007
– Requires Managed Direct X 1.1
• Webcam
– Next slide…
12
![Page 13: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/13.jpg)
Webcam Capture
• Windows: Microsoft DirectShow
– API provided by toolkit
• Issues
– Auto focus blur
– Image: Contrast, lighting, flickering
– Colour image vs. B&WW
13
![Page 14: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/14.jpg)
14
![Page 15: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/15.jpg)
15
![Page 16: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/16.jpg)
Vision Recognition System
1. Webcam Capture
2. Vision Recognition
3. Application
16
![Page 17: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/17.jpg)
Vision Recognition
• Runtime
– Initialization
• Select a webcam next slide…
• Load webcam parameters to NyARToolkit
• Load pattern(s) to detect
17
![Page 18: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/18.jpg)
18
using NyARToolkitCSUtils.Capture;using NyARToolkitCSUtils.NyAR;
public partial class MainWindow : Window, CaptureListener{
private CaptureDevice m_cap;
public MainWindow(){
[…]CaptureDeviceList cl = new CaptureDeviceList();CaptureDevice cap = cl[0];m_cap.SetCaptureListener(this);m_cap.PrepareCapture(800, 600, 30);[…]
}
void CaptureListener.OnBuffer(CaptureDevice i_sender, double i_sample_time,IntPtr i_buffer, int i_buffer_len)
{}
Width, height, frame rate
Enumerate webcams
Callback method
Callback interface
![Page 19: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/19.jpg)
19
using jp.nyatla.nyartoolkit.cs;using jp.nyatla.nyartoolkit.cs.core;using jp.nyatla.nyartoolkit.cs.detector;
public partial class MainWindow : Window, CaptureListener{
private NyARDetectMarker m_ar;
public MainWindow(){
[…]//AR用カメラパラメタファイルをロード// AR camera parameters file to readNyARParam ap = new NyARParam();ap.loadARParamFromFile("../../data/camera_para.dat");ap.changeScreenSize(800, 600);
//AR用のパターンコードを読み出し//AR's pattern to detect from the webcamNyARCode code = new NyARCode(16, 16);code.loadARPattFromFile("../../data/patt.kanji");this.m_ar = new NyARDetectMarker(ap,
new NyARCode[]{code}, new double[]{80.0}, 1, NyARBufferType.BYTE1D_B8G8R8_24);
this.m_ar.setContinueMode(false);
Detection object
Width, height again
Webcam Calibration
16 blocks inside black border
80 mm marker size
Recognize on a 24 bpp bitmap
![Page 20: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/20.jpg)
Vision Recognition
• Runtime
– New frame event
• B & W threshold
• Marker detection next slide…
• Application logic
– Better design
• Marker detection + logic in a timer loop
20
![Page 21: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/21.jpg)
21
void CaptureListener.OnBuffer(CaptureDevice i_sender, double i_sample_time, IntPtr i_buffer, int i_buffer_len)
{// calculate size of the frame bitmapint w = i_sender.video_width;int h = i_sender.video_height;int s = w * (i_sender.video_bit_count / 8); // stride
// thresholding […]
// detectMarkerLite requires BGR image from DirectX// m_threshold 0-255int detectedMkrs = this.m_ar.detectMarkerLite(ra, m_threshold);
// save the result of the detectionNyARSquare square = null;if (detectedMkrs > 0){
// vertices of the square are returnedNyARTransMatResult transMat = new NyARTransMatResult();NyARDoublePoint2d[] points = m_ar.getCorners(0); // RichFsquare = new NyARSquare();square.sqvertex = points;
}
square application logic for recognized marker
![Page 22: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/22.jpg)
22
void CaptureListener.OnBuffer(CaptureDevice i_sender, double i_sample_time, IntPtr i_buffer, int i_buffer_len) {
[…]AForge.Imaging.Filters.FiltersSequence seq =
new AForge.Imaging.Filters.FiltersSequence();
// order here is important, pipe and filters design patternseq.Add(new AForge.Imaging.Filters.Grayscale(0.2125, 0.7154, 0.0721));seq.Add(new AForge.Imaging.Filters.Threshold(127));seq.Add(new AForge.Imaging.Filters.GrayscaleToRGB()); // 24 bit image returned
// run the threshold algorithmAForge.Imaging.UnmanagedImage srcImg =
new AForge.Imaging.UnmanagedImage(i_buffer, w, h, s, System.Drawing.Imaging.PixelFormat.Format32bppRgb);
AForge.Imaging.UnmanagedImage outputImg = seq.Apply(srcImg);
Grayscale+ B/W threshold
Original cameraImage from DirectShow is 32 bpp
![Page 23: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/23.jpg)
23
[…] thresholding
// load a RGB buffer into the NyAR format, which we copy from AForge.ImagingNyARRgbRaster_RGB ra = new NyARRgbRaster_RGB(w, h, false);byte[] destArr = new byte[outputImg.Stride * outputImg.Height];System.Runtime.InteropServices.Marshal.Copy(outputImg.ImageData, destArr,
0, outputImg.Stride * outputImg.Height);ra.wrapBuffer(destArr);
// detectMarkerLite requires BGR image from DirectX// m_threshold 0-255int detectedMkrs = this.m_ar.detectMarkerLite(ra, m_threshold);
[…] application logic
Move Aforge.imaging bitmap NyARToolkit bitmap
![Page 24: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/24.jpg)
Examples
• WPF:
– My example
• Packaged examples:– NyARToolkitCS-2.5.2.1\forFW2.0\sample
24
![Page 25: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/25.jpg)
Examples
• RawTest– bitmap data + recognizer
• CaptureTest– webcam + recognizer + System.Drawing.Bitmap
• SimpleLiteDirect3d
• SingleNyIdMarkerDirect3d– webcam + recognizer + Direct3D
25
1 pattern
2 patterns
![Page 26: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/26.jpg)
Where to go?
• ARToolkitPlus Fudicials with built-in ID
• Goblin XNA Extends ARTag.NET
• Igarashi Detection Orthogonal viewingSmaller marker sizes
26
![Page 27: Vision Recognition for Mixed Reality Applicationsgrouplab.cpsc.ucalgary.ca/cookbook/uploads/Technologies/pres-vision-recog.pdf•WPF: –My example •Packaged examples: –NyARToolkitCS-2.5.2.1\forFW2.0\sample](https://reader033.vdocuments.us/reader033/viewer/2022060600/605404f8ebe1ad11973d570b/html5/thumbnails/27.jpg)
Thanks
• Miaosen Wang ARToolkit example
• Paul Lapides webcam info & ARToolkitPlus
27