![Page 1: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/1.jpg)
1
A Hierarchical Design Scheme for Application of Augmented Reality
in a Telerobotic Stereo-Vision System
BySyed Mohammed Shamsul Islam
Lecturer B and PhD studentDepartment of Computer Engineering
KFUPM
Co-authors: Prof. Mayez Al-Mouhamed Dr. S. M. Buhari Dr. Talal Al Kharobi
![Page 2: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/2.jpg)
2
Outline
Introduction Design Issues Implementation Performance Evaluation Conclusions & Recommendations
![Page 3: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/3.jpg)
3
Telerobotics:
A scheme that allows humans to extend their manipulative skills over a network.
Robot
LAN
Robot Contoller
Digital Cameras
`
`
`
Force feedback
Visual feedback
Visual feedback
Force feedback
![Page 4: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/4.jpg)
4
Telerobotic Applications Hazardous situations
e.g. bomb disposal
Scaled-down and scaled-up situations e.g. micro-surgery
Situations affected by human presence e.g. closed-chest heart bypass
Teaching, training, maintenance, entertainment.
![Page 5: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/5.jpg)
5
Problem of Conventional Tele-robotics
Time delay: Processing, copying video data, network delays Move-n-Wait strategy
Operator has to follow trial and error method to perform a task, which might be dangerous for example in telesurgery.
![Page 6: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/6.jpg)
6
Improving Performance by Using AR
By overlaying virtual image with the real image operator e.g. surgeon can make a plan before going for actual teleoperation safety.
Rehearsal and correction can be made in the simulation plan.
Saving bandwidth by sending (less frequently) only the planned data.
![Page 7: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/7.jpg)
7
Challenges of AR Graphical robot arm must mach
with real robot arm in the video.
Real video image
Graphics Subsystem
Graphic Co-ordinate
Space
RemoteCo-ordinate
Space(Video)
Bi-directional one-to-one mapping between graphic and real co-ordinate spaces
![Page 8: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/8.jpg)
8
Challenges of AR
Camera Calibration: Finding accurate
camera calibration is challenging.
Viewpoint Registration
![Page 9: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/9.jpg)
9
Design Issues
![Page 10: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/10.jpg)
10
Telerobotic Stereo-Vision System to be Augmented
LAN
Vision Client
Decision Server Interface
Client User Interface
MasterArm Component
Master
Arm
Stereo Display
Eye-Shuttering Glasses
Robot
VisionServer
Internet
DecisionServer
Server UI
PUMAComponent
UnimationController
Force Component
DigitalCameras Horizontal disparity = 6 cm
N
![Page 11: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/11.jpg)
11
Design MethodologyPerformance evaluation of the approach
Superimposition of the virtual object on to the stereo-video and augmented tele-manipulation
Interfacing to the telerobotic stereo-vision system
Moving capability of the virtual objects
Displaying the virtual objects
Drawing the virtual objects
Deriving the robot model equations
Evaluation
Augmentation ofStereo-vision
Interface
Animation
Virtual object rendering
Virtual object drawing
Mathematical models
![Page 12: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/12.jpg)
12
Mathematical Model of the Robot Manipulator
PUMA 560 Slave Arm: 6 DOF 6 links are interconnected serially to each other except first and last. All the joints are rotational.
A rotational joint
![Page 13: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/13.jpg)
13
Mathematical Model of the Robot Manipulator
Each link of the arm is attached with a frame of reference Ri (xi,yi,zi) to specify its position and orientation.
Link Li+1 rotates w.r.t Li when frame Ri+1 rotates w.r.t either axes of xi, yi, or zi
Link vector OiOi+1,i can be expressed as
1,11
,1
iiiiiiii OOMOO
],,[ ,1,1,11
iiiiiiii zyxM Where,
Now position and orientation of Oi+1 w.r.t Oi-1 can be expressed as
1,1111,11,11 iiiiiiiiiii OOMOOOO
11
11
i
iii
ii MMM
![Page 14: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/14.jpg)
14
Mathematical Model of the Robot Arm
This model will give us a skeleton of the graphical arm.
Where,nn
n MMMMM 132
21
100 ......
nnnn
nn OOMOOOO ,100,100,0
)}(),({)( 00,0 nn MOOG
Position and orientation of the end effector is expressed relative to the frame of the base. So the geometric model of robot arm can be expressed as:
![Page 15: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/15.jpg)
15
Building Body Shapes Around the Skeleton of Graphical Arm
Alternatives: Taking as is: link 3 and 4 are
trapezoidal, others are cylindrical.
Considering all as cylindrical
![Page 16: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/16.jpg)
16
Building Body Shapes Around the Skeleton of Graphical Arm
Cylinder is drawn with finite element representation.
Increasing number of segments will improve quality of view but increase computational complexity.
Alternatives of primitive for drawing a cylinder:
Triangles: Triangle stripe, Triangle fan, Triangle list
Lines Line stripe, Line list v 1
v 2
v 3
v 4
v 5
v 6
v 7
(a ) Tr ia n g le s tr ip
![Page 17: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/17.jpg)
17
Data Structure Design Types of data and nature of manipulation:
A link is represented by its start and end point co-ordinates, radius, axis of rotation, joint angle, orientation matrix etc.
Each link is also associated with the vertices of its body shape.
Links are serially connected and position and orientation of a link depends on position and orientation of its previous link.
There is frequent movement of links in teleoperation and each movement involves matrix calculation for new position and orientation of the links to be changed.
Objectives: To reduce processing time by reducing computation. To provide flexibility and generality
![Page 18: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/18.jpg)
18
Data Structure Design
Our strategy: Each link is defined as an object
geometric data of each link is now apart from the other link,.
Configuration data (number of links, number of segments in cylinder etc) kept in separate data file.
World and view transformation matrices are combined to reduce matrix multiplication.
Vertex data of body shape of each link is stored in separate vertex buffer.
![Page 19: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/19.jpg)
19
Displaying the Graphical Arm
Given the robot structure in its model space Rendering on the display screen:
Scene layout setup: Transformation matirces, Camera position, target position,
viewing angle, lighting. Viewing Models
Solid Model or Wire-frame Model Psychological studies and one performance studies in
telerobotic system of shows no significant diff. Wire-frame models have some adv.-reduce occlusion,
lesser rendering time. Rendering:
Scanline rendering, Z-buffering algorithm for HSR.
![Page 20: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/20.jpg)
20
Sample Display of Graphical Arm
Solid viewWire-frame view
![Page 21: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/21.jpg)
21
Movement of Graphical Arm in Joint Space
C al c ul ate ne w val ue s o f the jo i nt ang l e ,
Star t
C o m pute the P o s i t i o n ve c to r s
C o m pute the O r i e ntat i o n m atr i c e s
C o m pute the ve r t i c e s o f the bo dy shape s
R e nde r the w ho l e r o bo t w i th update d ve r t i c e s ( thus m o vi ng to a ne w
l o c at i o n)
N o
Ye s
R e c e i ve the i nc r e m e nt o f jo i nt ang l e ,
Ang l e w i thi n bo und?
).( 11
00ii
ii MMM
).( ,100,100,0 iiii
ii OOMOOOO
![Page 22: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/22.jpg)
22
Movement of Graphical Arm in Cartesian Space
Start
Receive the incremental position (X) and orientation (M) matrices
Calculate new position, Xnew(t) and orientation, Mnew(t) matrices
based on current frame of reference
Calculate corresponding angular values Using Inverse Geometric Transformation
(new=G-1(Xnew, Mnew)
Angle within bound?
B
![Page 23: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/23.jpg)
23
Movement of Graphical Arm in Cartesian Space…
Compute the vertices of the body shapes
Render the w hole robot w ith updated vertices
Compute the position vector of a ll the links
Compute the orienta tion matrices).( 1
100
ii
ii MMM
).( ,100,100,0 iiii
ii OOMOOOO
End
B
![Page 24: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/24.jpg)
24
Acquisition of Real Video Image
The MDTF Approach is used for stereovision. It is multi-threaded, distributed and provide better performance in
video transfer. Real video image of the slave robot at the server side is
captured simultaneously by two video cameras. Then a reliable client-server connection is established and upon
a request from the client a stereo frame comprising of two pictures is sent over LAN through window sockets.
On the client side after detecting and making connection with the server pictures are received and displayed on the screen and on the HMD when connected.
A double buffer, concurrent transfer approach is used to maximize overlapped transfer activities between cameras, processor and the network.
![Page 25: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/25.jpg)
25
3D Visualization Tools and Techniques
Technique Alternatives: Sync-Doubling, Page-flipping Chosen: Page-flipping
Provides higher resolution and avoid ”flashing” problem of 3D imaging.
Display device Alternatives: HMD, monitor, eye-shuttering glass Chosen: HMD, monitor
No flickering, easy to work
![Page 26: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/26.jpg)
26
3D Visualization Example
A snap shot of a stereo-image.
![Page 27: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/27.jpg)
27
Camera Calibration
Heikkila’s calibration method which provides very accurate results. It shows accuracy up to 1/50 of the pixel size.
Pinhole camera model with perspective projection and least-square error optimization.
The intrinsic camera parameters: Focal length, fc Principal point, cc Skew co-efficient, alpha_c Lens distortion co-efficient, kc
![Page 28: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/28.jpg)
28
Camera Calibration…The aspect ratio= fc(2) / fc(1)
The field of view angle can be calculated from,
)2/cot(2
)2(
)2/cot(2
)1(
y
x
fovW
fc
fovH
fc
![Page 29: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/29.jpg)
29
Camera Calibration…
Extrinsic ParametersIf XX=[X,Y,Z] and XXc are co-ordinate of P in grid and camera ref, then
1_*1_ TcXXRcXX C
Where, translation vector Tc_1 is the co-ordinate vector of O in camera ref frame and the rotation matrix Rc_1 is the surface normal vector of the grid plane in the camera ref frame.
![Page 30: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/30.jpg)
30
3D Visualization Algorithm
Step 1: Acquire video image and copy this memory stream to a surface( say FrontSurf).
Step 2: Copy the FrontSurf surface to another temporary surface (say, AugSurf).
Step 3: Draw graphical objects/change in graphical objects on AugSurf.
Step 4: Copy AugSurf to the surface that will be used to display (say, backSurf).
Step 5: Display backSurf with left camera image to left view port/monitor.
Step 6: Display backSurf with right image to right view port/monitor.
Step 7: Observe 3D with HMD.
![Page 31: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/31.jpg)
31
Graphical Tele-manipulation
How Graphic Manipulation Corresponds to the Real If the base link of the real robot remains fixed relative to the
video cameras, the base link of the graphical arm will also remain fixed relative to the graphical cameras.
The end-effector of the graphical arm can be manipulated in the graphical coordinate space, relative to objects in the task space (keeping base link in same location of real robot base).
![Page 32: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/32.jpg)
32
GUI Design
Client Side Input/Output Specification Connecting to the server PC, master arm and HMD Receiving and displaying the stereo video. Taking user's input for movement of the real and graphical
robot for simulation
Means of User Interaction: Master arm Joystick Keypad Mouse
![Page 33: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/33.jpg)
33
Implementation
![Page 34: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/34.jpg)
34
Tools Used
Client-server communication platform MS .NET with .NET Remoting
Programming Language Visual C#.NET
Graphics Tool Microsoft DirectX
3D Graphics API Alternatives:
Windows GDI Java3D by Sun MicroSystems Open GL (Open Graphics Language) by SGI Silicon Graphics Direct3D
![Page 35: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/35.jpg)
35
Motivation for Using Direct3D It increases performance by using hardware acceleration.
It allows applications to run full-screen instead of embedded in a window. As we are using HMD this feature is very helpful in our case to have the 3D effect.
Direct3D perfectly match with Microsoft's various Windows operating systems and with Microsoft .NET framework which are used in our telerobotic client-server framework
![Page 36: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/36.jpg)
36
Direct3D Architecture
W indows Application
Direct3D API
G DI
Hardware Abstraction Layer(HAL)
Device Driver In terface(DDI)
G raphics Device
![Page 37: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/37.jpg)
37
Graphics Implementation
Virtual Object Modeling Module Defines the structure of the virtual objects (robot, cubes
etc.) Handles functions like movement of virtual objects.
Display Module: Synchronization of real and virtual data Projection on video surface Augmentation of real video Page Flipping for HMD stereo visualization
![Page 38: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/38.jpg)
38
Interfacing to Telerobotic Stereovision System Client Modules:
LAN
VisionClientComponent
DXInterface Component
Display Surface(HMD, Monitor
etc)
Stereo Video
Camera Calibration Component
Left, Right Images
DecisionServer
MasterArmComponent
),( MX
Robot Model
Augmented Video
),( newnewMX
),( MX
Robot Client
Virtual object
Modeling
User Interface
![Page 39: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/39.jpg)
39
GUI Implementation Main Form
![Page 40: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/40.jpg)
40
GUI Implementation Stereo Form
![Page 41: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/41.jpg)
41
Performance Evaluation
![Page 42: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/42.jpg)
42
Evaluation of Graphical System: Refresh Rate:
Environment Graphics Complexities Avg. Refresh Rate
(frame/sec)
Graphics without overlaying real video
image
Without any drawing 273.36
With an object and the graphical arm in the scene 243.74
Only graphical arm with 8 segments in each cylinder 253.59
Only graphical arm with 50 segments in each cylinder (solid view)
239.78
Graphics without overlaying real video
image
Without any drawing 11.498
With an object and the graphical arm in the scene 11.384
Only graphical arm with 8 segments in each cylinder 11.347
Only graphical arm with 50 segments in each cylinder (solid view)
11.325
[Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]
![Page 43: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/43.jpg)
43
Time Required for Rendering Graphical Arm
Environment Graphics Complexities Rendering Time (ms)
Wire-frame
Solid
Graphics without
overlaying real video image
Only graphical arm with 8 segments in each cylinder
64.088 64.283
Only graphical arm with 50segments in each cylinder
65.034 65.151
Graphics overlaying real
video image
Only graphical arm with 8 segments in each cylinder
105.02 108.037
Only graphical arm with 50segments in each cylinder
105.37 109.362
[Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]
![Page 44: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/44.jpg)
44
Accuracy Re-projection error with the calibration method: [Pixel Error (0.11689,0.11500)]
Pix
el A
xis
in t
he y
dire
ctio
n
Pixel Axis in the x direction
![Page 45: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/45.jpg)
45
Comparison to Other Apporaches Iqbal, A. [1] augmented with only a small red ball at the
position of gripper in comparison to our whole graphical arm. Iqbal, A. [1] used Faugeras [4] calibration with Kuno[5]’s affine
frame of reference which led him to noticeable mismatch with real error in the matching shown in his figure. Whereas our computer vision-based calibration reduces error upto 1/50 of pixel size.
J. Vallino reports in his PhD thesis refresh rate of 10fps to be required for AR, when we gets above 11-17fps after overlaying graphical arm with live stereo video.
Graphics rendering of our system is faster than Iqbal, A. [1],[2][3] for our use of Direct3D.
Our system is comparatively cheaper due to the use of commodity hardware (PC) and software.
![Page 46: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/46.jpg)
46
Conclusion
![Page 47: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/47.jpg)
47
Summary of the Work and Contributions Using hardware accelerated graphics rendition that
provides us with excellent refresh rate of the output screen.
Improvement in accuracy of execution by using better calibration method and graphic aids (accuracy up to 1/50 of pixel size).
User-friendly graphical user interface for simple manipulation in the telerobotic AR system.
![Page 48: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/48.jpg)
48
Summary of the Work and Contributions…
Flexible and generalized data structure suitable for telerobotic visualization.
Identifying design strategy for intelligent switching in VR and AR mode for ensuring QoS.
Use of cheap and commercially available hardware and software
Can be used as a cheap and flexible visual tool for showing robot manipulation in the classrooms.
![Page 49: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/49.jpg)
49
Future Research Directions Providing an intelligent system to switch
between VR and AR modes of operation based on network delays to ensure QoS.
Using multi-processor system for processing video and graphics data more efficiently.
Using commercial software available to extract exact 3D model of the workspace objects which will facilitate more accurate task manipulation.
![Page 50: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/50.jpg)
50
References
1. Iqbal, A. Multistream realtime control of a distributed telerobotic system. M.Sc. Thesis, King Fahd University of Petroeum and Minerals,June 2003.
2. A Rastogi, P. Milgram, and D. Drascic. Telerobotic control with stereoscopic augmented reality. SPIE, Vol.2653: Stereoscopic Displays and Virtual Reality Systems III:135{146, Feb. 1996.
3. R. Marin, P.J. Sanz, and J.S. Sanchez. A very high level interface to teleoperate a robot via web including augmented reality. Proc. IEEEInternational Conference on Robotics and Automation, 2002 ICRA '02, Vol.3:2725 - 2730, May 2002.
4. O.D. Faugeras and G. Toscani. The calibration problem for stereo. Proceedings of Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, Vol. 5, No. 3, June, 15-20 1986.
![Page 51: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/51.jpg)
51
References
[5]Y. Kuno, K. Hayashi, K.H. Jo, and Y. Shirai. Human-robot interface using uncalibrated stereo vision. International Conference on Intelligent Robots and Systems 95, 1:525{530, 1995.
[6]J. Abdullah and K. Martinez. Camera self-calibration for the ar-toolkit. The First IEEE International Workshop on Augmented Reality Toolkit, page 5, Sept. 2002.
[7]J.-Y. Herve, C. Duchesne, and V. Pradines. Dynamic registration for augmented reality in telerobotics applications. IEEE International Conference on Systems, Man, and Cyberneticsn, Vol.2:1348-1353, October 2000.
![Page 52: 1 A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d775503460f94a59614/html5/thumbnails/52.jpg)
52