mobile motion tracking using onboard camera
DESCRIPTION
Mobile Motion Tracking using Onboard Camera. Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit Wong Yuk Man. Agenda. Motivation & Objective Previous Work Improvement & New components Sample Applications Conclusion Q&A. Motivation. - PowerPoint PPT PresentationTRANSCRIPT
Mobile Motion Tracking using Onboard Camera
Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit
Wong Yuk Man
Agenda
Motivation & ObjectivePrevious Work Improvement & New components Sample Applications
Conclusion
Q&A
Motivation
Rapid increase in the use of camera-phone
Camera-phone can perform image processing tasks on the device itself
Enhance human-computer interaction by mobile phone
Objective
Implement real-time motion tracking on Symbian phone, without requiring additional hardware
Translational motion trackingRotational motion tracking
To develop a tracking engine for other mobile devices developers to use
Previous Work
A translational motion tracking engineBlocking matching• SEA (Successive Elimination Algorithm)• PPNM• PDE (Partial Distortion Elimination)• Adaptive Window Search• Spiral Scan
Feature selection
A translational motion tracking engine
Block Matching Algorithm
Block Matching Algorithm
Evaluate the "goodness" of a match by Sum of Absolute Difference
Select the candidate block with the lowest error
N
i
N
j
jiYjiXyxSAD1 1
|),(),(|),(
Feature SelectionA good feature block:
High variance -> complex blockNot in a repeated pattern
If a good feature block is found, the performance of the motion tracking is improved
We have made improvements to our existing algorithm
squares
Old Feature Selection
Divide the current frame into squares
Apply Feature Selection Algorithm to each squaresHowever, the best feature block may appear between two squares
New Feature Selection
More blocks are sampledSample 26x26 variances of blocks with size 15x15 in a 54x54 window
Search in spiral way and stop searching when selection criteria are matched
Prefer to find a feature block in the centerPrevent out of bound problem (block appear out of screen)
New Feature SelectionIf a feature block is found on the edge, the following will happen:The tracking algorithm will fail to find the exact matchSolution:
Apply additional constraint
New Feature selectionIn order to find a feature block that is not on the edge, we apply a checking algorithmImportant difference in all directions => interest point
New Feature Selection
Now our newest feature selection algorithm becomes:
Find a block with a large variance -> which indicate the complexity of the blockCheck if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors
• If either one of the SAD is small -> the block is on edge -> reject
• Else the block is not on edge
SSD is used instead of SAD
SSD = Sum of Squared DifferenceLast term, SAD is used as matching criteriaSAD is commonly used because of its lower complexity (faster)But we chose to use SSD finally
SSD is used instead of SAD
Experiment shows that SSD has better performance in accuracy andAlgorithm with SSD run as fast as that with SADReason (of why as fast as SAD):
Elimination effects from SEA, PPNM & PDE become large when SSD is used Compensate the higher complexity of SSD
SSD is used instead of SADSEA and PPNM lower bound is adjusted using the inequality (12) proposed in the following paper
J.J. Francis and G. de Jager. A Sum Square Error based Successive Elimination Algorithm for Block Motion Estimation (2002)
Experimental Result on Symbian phone
Frame rateMaximum frame rate supported by Nokia 6600 is about 14 frames/secRunning our algorithm (1/0.007 frames/sec) once only for each frame do not slow down the displaying frame rate, in other word, the frame rate can still be 14 frames/sec, our algorithm doesn’t lag the display
Rotation Tracking Engine
As we rotate the phone, the object can still be tracked correctly (center of object roughly equals center of green box)
Observation and Motivation
Rotation Tracking Engine
Our approachIf two blocks are tracked at the same time, angle of line connecting the two blocks reflect the tiling angle of the phone
Two-block approach
Rotation Tracking Engine
Another approachTrack one block by simple motion tracking engine, then find which rotation gives the best matchFail if tracking object is the same for different angle
One-block approach
Rotation Tracking Engine
Modification of motion tracking algorithm to suit rotation tracking
Linear Adaptive method used in translation motion tracking is not used here because phone’s motion can be both linear motion and circular motion
Rotation Tracking EngineWhat to predict?
Given the coordinates of the tracking blocks in the last 2 previous framesPredict the coordinates of the blocks in the next frame
θθ
A simplified mathematic problem (the following assumptions are approximately correct)
All three lines pass through the circle’s center
End points of the three lines lie on the circle
Line L2
Line L1
Line L3
Rotation Tracking Engine
SolutionAngle between L2 and L3 = θ
• θ = Tan-1(slope(L2)) – Tan-1(slope(L3))
Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix
2
2
1
1
cossin
sincos
L
L
L
L
y
x
y
x
Rotation Tracking EngineSolution
The prediction of the position of the next tracking block should also take the translational movement into accountHorizontal displacement = Tx
• Tx = ( xL21 + xL22
– xL31 - xL32
)/2
Vertical displacement = Ty • Ty = ( yL21
+ yL22– yL31
- yL32
)/2
Coordinates of the next tracking block (xL1, yL1) is calculated by
1111
cossin
sincos
12
2
1
1
L
L
L
L
y
x
Ty
Tx
y
x
Rotation Tracking Engine
A simple drawing to show the detected rotation angle of the phone
Rotation Tracking Engine
Reducing error by using levelApply threshold on output, increase/decrease one level only when change is largeTo give less sensitive but more desirable output for game e.g. skiing game: skier face only to 7 directionsReduce difficulty, increase reliabilityA small rotation or a small detection error will not increase/decrease one level
Conditions to be a good background for both engines
ObjectiveMax. performance measurementIncrease usability
Condition:Feature selection can always find a good feature pointWithin certain distance
• No repeat pattern• Very distinct pattern
Pattern is nearly the same when rotated. E.g. Circle which is good for rotation detection
Virtual Mouse
An application that make full use of the translational motion detection engineRemote control the mouse of PC by Symbian phone using motion trackingAdvantages:
It allows input in all directions It provides high levels of control Joystick can still be used as rough moving while camera input can be used as fine translation
Virtual Mouse
ServerIn server side (Windows), the Bluetooth is configured to provide RFComm Service and server regards the Bluetooth transmission port as Comm. Port Server receives message from that Comm. PortCall function in MouseAction.h to make the mouse move and trigger mouse click event
Virtual Mouse
ClientSearch for Bluetooth device nearbyConnect to the selected Bluetooth deviceEvery time motion tracking algorithm finishes, results are sent to server (12 times/sec)Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)
Car Racing GameA “Car Racing Game” is developed using the motion tracking engine.Game lags. It is because:
The engine takes time to find the motion vectorCPU speed of the Symbian phones are lowThe game engine thread uses most of the CPU power while the thread for updating the screen cannot proceed
Solutions:Double BufferingDirect Screen AccessSingle Thread
Double Buffered AreaDouble-buffering – two complete color buffers
One is displayed while the other is calculating next scene to be drawn,When the drawing of the next frame is completed, the content of the back buffer is copied to the front screen
Computing Frame 2
Show Frame 2
Buffer 1 Buffer 2copy
Frame 1
Computing……
Frame 2
Front Screen
Frame 2 ready
Direct Screen AccessIn Symbian, traditional drawing requires the help of Window Server
Overhead in context switchingLower speed
Access the screen directly in 3 ways:Creating and using CfbsScreenDeviceAccessing screen memory directly.Using CdirectScreenAccess.
Using DSA can bypass the Window ServerGet rid of context switchingFaster
Traditional Graphic Programming
Update Display
Device Screen and Keypad
Window Server
RWsSession RWsSessionRWsSession
Application 1 Application 1 Application 1
Key Presses
Direct Screen Access
2. If successful, a region for drawing to is returned to the application.
Window Server
1. Request to Window Server to work in Direct Screen Access mode
Application
Single Thread
Prevent context switchingNo need to do synchronization
Car Racing Game
With the use of efficient graphic programming algorithms, the tracking engine will not affect the performance of the game
It shows that we can use the engine to develop other applications without performance degradation
Skiing Game
Sample program from the book “Developing Series 60 Applications: A Guide for Symbian Os C++ Developers”Based on this game, we plug in our rotation tracking engine into it
The game becomes more interactiveRotate the phone to control the angle of the skier
Skiing Game
Skiing Game
The process of using the engine is very simple
Create an instance of the tracking engineCall the function of the engine to find the motion vectorUse the motion vector for the game logic
Skiing Game
Demo
Experimental Result on Symbian phone
Testing EnvironmentPlatform
• Symbian Phone Nokia 6600 Algorithm
• Our final (hybrid-type) Algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method)
Algorithm parameter• Block size = 17 x 17 (pixels)• Search window size = 16 x 16 (pixels)
Number of block to track in each run of algorithm = 1 block ONLY
Experimental Result on Symbian phone
Testing Result (avg time to run)Final algorithm • 7ms
Final algorithm without adaptive window method and SEA, PPNM method • 22ms
Full Exhaustive Search algorithm (the most simplest one) • 55ms
Q & A