multi-touch technologies: software, tracking and functions
DESCRIPTION
Lecture series. Topic covers: Introduction to software programming, Gesture interaction, MT engines and applicationsTRANSCRIPT
![Page 1: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/1.jpg)
Lecture 3 : Multi-touch technologies: Software, tracking and functions.
By Jazmi Jamal
![Page 2: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/2.jpg)
Have you twit today?
Twitter : toaBmd
![Page 3: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/3.jpg)
Outline
Introduction to software programming for multi-touch
Gesture interaction
MT engines or applications
![Page 4: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/4.jpg)
Re-cap: Multi-touch hardware
![Page 5: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/5.jpg)
Re-cap: Multi-touch hardware
![Page 6: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/6.jpg)
Software Programming MT
programming language Eg:
AS3
Python
C
C++
Java
Same programming with other language.
Certain protocols, methods and standards in the MT programming.
Read & translate raw blob as input, relay this info through pre-defined protocols and interact with an application.
![Page 7: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/7.jpg)
Tracking
Tracking re-identify object in series of video frames.
Background-subtraction technique.
Foreground identify as centroids (points). Tracking algorithm predicts the centroids movement.
http://vimeo.com/13282333
![Page 8: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/8.jpg)
Computer Vision
• Touch pattern detection
• Pattern detection for tangible objects with orientation and position
• Calibration between camera and projector
• Background and noise subtraction
![Page 9: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/9.jpg)
CV framework outline
![Page 10: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/10.jpg)
…
Gesture Interaction
![Page 11: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/11.jpg)
Gesture Interaction
“Communication through gestures has been one of the oldest form of interaction in human civilization owing to various psychological reasons.”
“Thus a gesture based interface is a perfect candidate for social and collaborative tasks as well for applications involving an artistic touch. The interface is physical, more visible and with direct manipulation.”
![Page 12: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/12.jpg)
Gesture Interaction
Goal of gesture interaction: higher immersion with easy and familiar movement.
Eg: scaling, rotating and translating images with 2 fingers. (Natural gestures)
![Page 13: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/13.jpg)
Gesture Interaction (Apple MT)
![Page 14: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/14.jpg)
Gesture Interaction (Activity)
My MT gesture
Establish a group of 3, create and discuss 2 inventive MT gestures. Some criterions must be applied
Criterions
What is it?
How does it function?
Where does this apply ? Why?
Why you prefer that gesture ?
![Page 15: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/15.jpg)
Save (example)
What is it?
Shortcut button to save file. (In PC Ctrl+S, MAC Cmd+S)
![Page 16: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/16.jpg)
Save (example)
How does it function?
Narrow finger 1,2,3,4. (like grasping object movement)
(1: thumb, 2: index finger, 3: middle finger, 4: ring finger, and 5: pinky)
![Page 17: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/17.jpg)
Save (example)
Where does this apply ?
Applicable only on table-top multi-touch.
Why?
The gesture activity take space. Not suitable for small devices with space constraint.
![Page 18: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/18.jpg)
Save (example)
Why you prefer that gesture ?
The movement of grasp shows to own or hold something.
The movement suitable for save function.
![Page 19: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/19.jpg)
35 minutes ~
MT Application
![Page 20: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/20.jpg)
MT apps & engines
Ready-made multi-touch engines / software to control the camera, process, blob detection and communication with other client applications.
![Page 21: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/21.jpg)
MT apps & engines
Several open sources MT engines.
Criterions
Touchlib
CCV (Community Core Vision)
ReacTIVision
OpenTouch
Touche and etc…
![Page 22: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/22.jpg)
CCV
CCV or tbeta is an open source/cross-platform solution for computer vision and machine sensing.
Suitable for optical techniques approach.
Developed by NUI Group Community.
Platform: Mac, Windows, Linux
http://ccv.nuigroup.com/
http://vimeo.com/2034557
![Page 23: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/23.jpg)
CCV
http://wiki.nuigroup.com/Getting_Started_with_tbeta
![Page 24: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/24.jpg)
CCV (configuration)
• You may Download the tbeta. & launch it.
• Click Next/Previous camera to choose desirable input.
• Then, press to background remove background to eliminate bg. Dynamic substract when the lighting is often changes
![Page 25: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/25.jpg)
CCV (configuration)
For DI users only:
• Turn off the smooth and amplify filters
• Turn on Highpass filter
• Adjust highpass blur and noise sliders until see fingers are clear
• If blobs are weak, turn on the amplify filter to brighten it.
![Page 26: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/26.jpg)
CCV (configuration)
• Adjust threshold until only fingers are tracked and no false blobs
• Press Save settings
• Continue to Calibration guide to perform calibration
![Page 27: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/27.jpg)
CCV (calibration)
Calibrating allows touch points to line up with elements on screen.
http://www.youtube.com/watch?v=Clp1cKg0xks&feature=player_embedded
![Page 28: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/28.jpg)
CCV (running demos)
• Download FLASH demos specifically for the tbeta
• In order to run demo swf files, security settings first need to be set. Go to Flash Global Security Settings. Click on Edit Locations... and then Add Location.
• Browse the demo folder and click confirm
• Launch the flash demo (.swf)
http://wiki.nuigroup.com/Getting_Started_with_tbeta
![Page 29: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/29.jpg)
CCV (overview)
•Configure blob tracking
•Calibrate with screen
•Send signal to XML Flash
•Allow global security settings in Flash
•Run Flash demo.
![Page 30: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/30.jpg)
Try this out (MiniMT)
http://nuigroup.com/forums/viewthread/1731/
![Page 31: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/31.jpg)
Need a break ?
Group activity
![Page 32: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/32.jpg)
Group activity
Please twit and share 2 links regarding setup or process between CCV and Flash communication.
Don’t forget to add this guy: toaBmd
Each group require to present and discuss the link that have been twitted.
![Page 33: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/33.jpg)
35 minutes
![Page 34: Multi-Touch technologies: Software, tracking and functions](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b47ffb4a795928128b456d/html5/thumbnails/34.jpg)
Finish! Q&A session