Robot Homing Workshop

1.The Camera Prospective Projection Model

Figure 1.1The camera pinhole model

y'

r

r’

z

y

x

x'(x’,y’)

z

f

Image Planeinverted

Object point(x,y,z)

z

yfy

z

xfx ' '

Figure 1.2Image plane: shifted and rotated

Z

X

Y

x

y r

x’

y’

(x,y,z)

(x’,y’)

Y’

X’

r’

2.Affine Invariant Shape Correspondence Model

),(),...,,(),( 221,1 NN yxyxyxC

Let Shapes C and C’ be described by points lying on their boundaries:

),(),...,,(),( 221,1 NN vuvuvuC

Supposed the following relation exits:

22221

11211

byaxav

byaxau

iii

iii

1100122221

11211

i

i

i

i

y

x

baa

baa

v

u

C’ is an affine transformation version of C (Or vice versa)

For any three points )(),,(),,( , kkjjii yxyxyx from shape C and their

corresponding points )(),,(),,( , kkjjii vuvuvu from shape C’ there exists

the relation:

11110011122221

11211

kji

kji

kji

kji

yyy

xxx

baa

baa

vvv

uuu

111

10011122221

11211

kji

kji

kji

kji

yyy

xxx

baa

baa

vvv

uuu

( Determinant notation )

kjiaffine

kji areaaaaaarea ,,21122211,, )(

1112

1 kji

kji

yyy

xxx

The triangle area created by )(),,(),,( , kkjjii yxyxyx is

)2.1(

2.1 Shape Feature Vector

Let’s look at ),(),...,,(),( 221,1 NN yxyxyxC

For each boundary point Cyx ii )( , we define the feature vector iF

Let ],...,,[ 21T

iMii fffFi Where

111

2

1jiiji

jiiji

yyy

xxx

fji

2

1

N

M

+/-)cyclic around N(

is the triangle area created byfji )(),,(),y ,( ,j-i jijiiiji yxyxx

iF can be groups into 3 parts:

• local features

• global features

• and again local features

)1( )1(

T

......... features features

MiilliikkiliF ffffff

featureslocalglobalocall

i

is normalized to ˆ

1 1

N

i

M

j

i

fji

FiF iF

Now for shape C we can define the set: ˆ ˆ{ , 1, 2,... }C iF F i N

For ),(),...,,(),( 221,1 NN vuvuvuC in a similar way we have:

ˆ ˆ' { ' , 1, 2,... }C iF F i N

According to 2.1

fjiaaaayyy

xxx

aabavvv

uuu

jif jiiji

jiiji

jiiji

jiiji

)(

111

2

1)(

111

2

1' 2121111121211111

, , 11 22 12 21 , , ( ( ) )affinei j k i j karea a a a a area we get

)aa-a(' 21122211 ii FaF

11 22 12 21

11 22 12 211 1 1 1 1 1

' ( )ˆ ˆ ' ' ( )

i i iN M N M N M

i j i j i j

F a a a a F FF i Fi

f ji a a a a fji fji

For each point

,( )i ix y and its affine-transformed corresponding

point ( , )i iu v there exits iFiF 'ˆˆ

2.2 Global Shift Index

Let Shape C’ be an affine-transformed version of C.

),(),...,,(),( 221,1 NN yxyxyxC ),(),...,,(),( 221,1 NN vuvuvuC

If the points from the shape’s boundary have been taken ‘smoothly’ then:

( , ) ( , ) 0,1,..., 1i i i s i sx y u v s N ( + is cyclic around N )

S is the global shift Index.

Estimating s

Estimation of s means finding the most suitable number ˆ ˆ , 0,1,..., 1s s N

to be the value of s.

)ˆˆ()ˆˆ( )()(, isiT

isisi FFGFFE

N

isis EE

1,

G is MxM diagonal matrix1

( )M 2 2

2

2

[ ]

1

2

mm M M

m

mm

G g

g e

)1,...,2,1,0(

minargˆNssEsThe chosen value for s is

2.3 Similarity Matching of ShapesShapes C and C’ are considered similar if they are the same or if one shape is an affine-transformed version of the other.

Let’s define )1,...,2,1,0(

min

NssEE

It is assumed that the feature vector set CF is a unique characteristic of

of a particular shape (or its affine-transformed version)

for similar shapes E is quite small (ideally 0)

for different shapes E should be larger

Local Shift Index

If shapes C and C’ are similar, for each point

)( , ii yx on C

the suitable point )( , iiii ss vu on C’ can be found locally:

)ˆˆ()ˆˆ( )()(, isiT

isisi FFGFFE

)1,...,2,1,0(

,minargNssii Es

Let n be the number of times when

ˆ 0is s n N

For similar shapes n should be close to N.

Similarity Matching Function

For shapes C and C’ ),(),...,,(),( 221,1 NN yxyxyxC

),(),...,,(),( 221,1 NN vuvuvuC

UpxUp

Upxxxh

)(

If

If

ThETotal

ThETotal

C and C’ are similar

C and C’ are different

EEnNh

Total

2

3.Robot Homing Workshop

3.1 Workshop Hardware and Software

Matlab: Getting Image

Matlab: Image Processing

Lejos

vfm

3.2 Workshop Hypothesis• Camera Pinhole Model

Image received by the camera is twisted according to:

z

yfy

z

xfx

• The camera (the robot) is ‘far enough’ from the arena walls

The ratio z

fis approximately constant

The received shape image is an affine-transformed version of the shape printout.

The similarity matching function can be applied to identify the home shape.

3.3 The Matlab Program Aspects

• Traveling around the arena and grabbing the wall images

• Analyzing the received images and retrieving the shape figures

• Determine whether the retrieved shape is the home shape

• Traveling towards the home shape when it has been detected

4 .Analyzing The Camera Images

4.1 Image Segmentation• Converting the RGB Image Into a gray level one.

•Applying Canny Edge Detector with threshold on the converted image.

RGB Image Gray Level Image Applying Edge Detection

• Finding 8-connectivity Image Elements.

1 2 3

8 4

7 56

Pixel’s 8 neighbours

An Image containing two 8-connected elements

8-Connectivity Element Finding Algorithm

  1.  Scan the image left to right, top to bottom.

2. If pixel value is not 0, then

 

(a) If only one of its upper and left neighbours (neighbours 1,2,3,8 the previous figure )

has a label, than copy it.

(b) If all neighbours have the same label, then copy the label.

(c) If they have different labels, then copy one of them and enter the labels in

the equivalence table as equivalent labels.

(d) Otherwise assign a new label to this pixel and enter this label in the

equivalence table.

  3. If there are more pixels to consider, then go to step 2.

  4. Find the lowest label for each equivalent set in the equivalence table.

5. Scan the picture. Replace each label by the lowest label in its equivalent set.

4.2 Analyze of Image Elements

Image Element can be one of the following groups:

1. A whole shape Boundary

Shape Image Suitable Image Element

2 .Part of A shape Boundary

3 .Arena Corners

Image of partial shape & arena corners

Suitable Image Elements

Image of the arena wall

4 .Arena wall

Image Elements due to blank wall image

4.2.1 Wall Detection

• There are “too many” Elements in the image.

4.2.2 Close Line Detection • A whole shape boundary is a smooth close line.(Hamiltonian circle)

We need to find out weather the image element is a closed line .

• The Image Elements due to the arena corners are open lines.

We should find the element boundary.

Image Pixel

Image Interior Pixel

Image Boundary Pixel

Boundary Following Algorithm

.cs

Ss1 .Find the starting pixel for the region using a systematic scan,

say from left to right and from top to bottom of the image .

2 .Let the current pixel in the boundary tracking be denoted by c .

Set cs and let the 4-neighbour to the left of s be .Sb

3 .Let the 8-neighbours of c starting with b in the clockwise order be

821 ,...,, nnn .Find in ,for the first i that is in S.

4 .Set inc and .1 inb

5 .Repeat Steps 3 and 4 until

1 2 3

8 4

7 56

Pixel’s 8 neighbours

• Applying the algorithm to a whole shape boundary (“smooth” closed line)

returns its boundary pixels ordered clockwise.

• If the image element is a smooth closed line the first boundary pixel to be

come across again is only the starting one c.

• This Algorithm doesn’t enable to distinguish between open lines due to

the arena corners and open lines due to partial boundary of a shape

(which may be of interest).

Image ElementsImage

4.2.3 Detection of the Home Shape

• The boundary of the shape considered home is a closed line.

• The (synthetic) home figure is stored in the robot data base.(simply a file)

and its boundary is of course closed. We can use the the matching algorithm described above.

• Sampling in “smooth manner” N points from the home boundary as

in the data base.

• Sampling in “smooth manner” N point from the received shape

boundary (suitable element pixels)

•Applying the matching algorithm

The original home shape

as in the data base

The shape as received by

the robot camera

Problems:

• The received shape is twisted and only approximation of an affine transformation. •The sample points from the received shape can not be promised to be

the corresponding affine transformation version of the sampled points from the

original home shape.

• How to determine NTh , and

Partial Solutions:

• Intuitively the above problems mainly affect the values of the local features and

and have less influence on global features.The matrix reduces the weight of

the local features.

• The values of NTh , and should be found experimentally.

4.2.4 Calculation Performed on a Received Shape.

•Weight of its boundary ( number of pixels)

•The area of the shape bounding box

•The shape’s center of mass

4.2.5 A shape at the image margins• It results in an open line element at the image margins

Elements at the margins should be treated.

Problem: Elements can also be due to the arena corners.

Image

Element in Image

Margins?

Edge Detection

Segmentation

Element Element

Wall?

Closed Shape?

Calculate:Boundary Pixel Number

Bounding Box AreaCenter of Mass

Home?

yes

yes

Image Analysis

5 .Scanning For the Home Shape

• For Traveling inside the arena we need to deduce the 3D orientation out of 2D images.

• Implementing spatial orientation is not an easy task and beyond the scope…

The Scanning process is very limited and primitive.

Detecting the home shape is not promised.

5.1 The Scanning Cycle• The search process is divided into finite number of so called scanning cycles.

• The search is terminated after all the scanning cycles have been exhausted,

regardless the home finding.

StartScan cycle

ClosedShapeFound

HomeFound

WallDetected

Should find ashape OR

Something intop margin

Get an ImageAnalyze it

Move towardshome

Check if the shapeis the smallest

Refine Location

Check if should findshape in next image

Scan cycle ends

Turn towards theSmallest image

Rotate the camerain small angle

Scan cycles exhausted

Move forwards

FinishScanning

yes

no

yes

no

no

no

no

no

yes

yes

Home Scanning Procedure

yes

yes

5.2 The Refine Location Procedure.

The procedure handles two case:

• There is an element at the top margin of the image so it might be a part of a shape

to which the robot camera is to close.

• The current image analysis yields no close shape whereas according to the previous

image a shape should have been found.(element at the left/right margin)

Problem: Because of the insufficient segmentation the elements can be due to

the arena walls,texture and etc.

A finite number of trials should be taken in order to find a shape .

IterNum 1Home Found 0

IterNum<=2and

Element at top margin

Begin

Go backwards

Get an imageAnalyze it

Closed

Shape

Found

Home

Found

IterNumIterNum +1

HomeFound1

yes

yes

yes

no

no

no

Move towards home

End

Refine Location Case 1

HomeFound

ClosedShapeFound

HomeFound1

i = 1

Turn back in small angle

Get an imageAnalyze it

ii +1

Return to theinitial orientation

i<=2

no

no

no

yes

yes

yes

Move towardshome

End

Refine Location Case 2

5.3 Moving towards Home Procedure

• Once the home shape has been detected,the robot moved towards it.

Problem: The home shape doesn’t have to be in the image center so while moving towards it

the shape might be out of the sight field of the camera.

• Calculating the home shape center of mass.

• The robot moves forward in small “steps” checking after each step if the home shape

is still in camera field of sight .

• If not, The robot moves backwards to its previous location and turns a “little” towards

the home shape according to the location of the shape’s center of mass.

Problem: The robot shouldn’t approach the home shape too much,otherwise the

affine transformation approximation won’t be correct to be used and

the matching algorithm won’t be consistent.

• Calculation the area of the home shape bounding box.

• If the area is “too large” the robot stops and doesn’t try to get closer.

End

Begin

Close enough to

home

Move forwards

Get an imageAnalyze it

HomeFound

Move back

Turn back toinitial orientation

End

Turn towardshome

HomeFound

Get an imageAnalyze it

yes

no

yes

no

yes

no

no

yes

Moving towards home

1j

4j

1 jj