hirsh1

School of ComputingFACULTY OF ENGINEERING

3D Noughts & Crosses with Baxter, a Humanoid Robot

Benjamin Leonardo Hirsh

Submitted in accordance with the requirements for the degree ofBSc Computer Science (Industry)

2014 - 2015

i

The candidate confirms that the following have been submitted.

Items Format Recipient(s) and DateReport (Hard Copy) Report SSO (03/06/15)Report (Digital) Signed forms in envelop SSO (03/06/15)Software Code Repository URL Supervisor, Assessor

(02/06/15)Demonstration Video URL Supervisor, Assessor

(02/06/15)

Type of project: Exploratory Software (ESw)

The candidate confirms that the work submitted is their own and the appropriate credithas been given where reference has been made to the work of others.

I understand that failure to attribute material which is obtained from another sourcemay be considered as plagiarism.

(Signature of Student)

c○ 2014 - 2015 The University of Leeds and Benjamin Leonardo Hirsh

ii

Summary

This report documents the development of a software system run on the University’sBaxter Research Robot (named Lucas), enabling him to participate in playing 3D Noughts& Crosses with another human player. The aim was to create a piece of software to berun in ROS (Robot Operating System) in order to allow Baxter to do this. The finalproduct had Baxter successfully playing legal games against human opponents.

iii

Acknowledgements

I would like to thank my supervisors, Professor Tony Cohn and Doctor Eris Chinellato,for assisting me throughout the project with both the development and the writing of thisreport, making sure it complies with standards. I would like to thank the other membersof the STRANDS team, Doctor Ioannis Gatsoulis, Paul Duckworth, Jawad Tayyub andMuhannad Omari, along with Aryana Tavanai, all of whom gave me advice on solutionsand challenged me on my methods to ensure they were the most suitable. The level atwhich I now understand of robotics and computer vision could certainly not have beenreached without them. I would also like to thank my flatmate, Sam Brown, for all hishelp and support.

Contents

1 Introduction 31.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Report Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Background Research 52.1 Board Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 3D Noughts & Crosses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2.1 Physical Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2.2 Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2.3 Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2.4 Computer Representation . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 Adversarial Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3.1 Game Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3.2 Minimax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.4 Robotic Paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.4.1 Hierarchical Paradigm . . . . . . . . . . . . . . . . . . . . . . . . . 122.4.2 Reactive Paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.5 Robot Operating System (ROS) . . . . . . . . . . . . . . . . . . . . . . . . 132.5.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.5.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.5.3 Topics & Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.6 Baxter Research Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.6.1 Baxter Software Development Kit (SDK) . . . . . . . . . . . . . . . 15

2.7 Existing Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.7.1 Worked Example Visual Servoing . . . . . . . . . . . . . . . . . . . 162.7.2 Connect Four Demo . . . . . . . . . . . . . . . . . . . . . . . . . . 162.7.3 Baxter Solves Rubiks Cube . . . . . . . . . . . . . . . . . . . . . . 16

2.8 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.8.1 Programming Languages . . . . . . . . . . . . . . . . . . . . . . . . 172.8.2 Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.8.3 Version Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

iv

CONTENTS v

3 Development & Analysis 183.1 Hardware Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1.1 Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.1.2 Grippers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.3 Custom Board Game . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.2 Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2.1 Blob Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2.2 Contour Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.2.3 Background Subtraction . . . . . . . . . . . . . . . . . . . . . . . . 243.2.4 Hough Circle Detection . . . . . . . . . . . . . . . . . . . . . . . . . 253.2.5 World Coordinate System Transformations . . . . . . . . . . . . . . 273.2.6 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.2.7 Pixel Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3 Game Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.3.1 2D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.3.2 3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.4 Robot Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.4.1 Basic Movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.4.2 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.4.3 Inverse Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.4.4 Head Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.5 Playing Noughts & Crosses . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.5.1 How to run programs . . . . . . . . . . . . . . . . . . . . . . . . . . 393.5.2 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.5.3 Robot Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.5.4 Board Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.5.5 Space Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4 Evaluation 444.1 Board Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.1.1 Pixel Point Accuracy of Hough Circle Detection . . . . . . . . . . . 444.1.2 Pixel Point Convergence . . . . . . . . . . . . . . . . . . . . . . . . 45

4.2 Object Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.2.1 Pixel Point Accuracy of Contours . . . . . . . . . . . . . . . . . . . 47

4.3 Game Play . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.3.1 Vision Classifications . . . . . . . . . . . . . . . . . . . . . . . . . . 494.3.2 Game Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.4 Code Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

CONTENTS 1

5 Conclusion 525.0.1 Time Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.1 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525.1.1 Depth Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.1.2 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.1.3 Game Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.1.4 Artificial Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . 545.1.5 Audio & Voice Commands . . . . . . . . . . . . . . . . . . . . . . . 545.1.6 Head Display Nod . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.1.7 Other Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

References 56

Appendices 58

A Personal Reflection 59

B Ethical Issues Addressed 61

C Video 62

D Code 63D.1 Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63D.2 New3DtictacBaxter.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63D.3 BoardClass3D.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

E Board Detection Times Under Varying Light Conditions 91

CONTENTS 2

Chapter 1

IntroductionThe aim of this project is to develop a software system which runs on a humanoid robot,allowing it to play a game of 3D Noughts & Crosses with a human partner. The robotshould be able to observe and interact with the playing area.

1.1 Methodology

This project has been approached with a rapid prototyping process in mind, reducing theobjectives into small, manageable tasks. The nature of the software development workswith this approach as each iteration adds features and correct flaws from the last. In doingthis, if something were to go wrong, the entire project did not need to be restarted fromscratch or changed drastically. Developing software to enable the robot to perform simplemovements in the Robot Operating System (ROS) environment was the initial iteration,and further, more complex manoeuvres were developed on top of this, according to theobjectives of Section 1.2.

1.2 Objectives

The following are a set of increasingly ambitious and complex objectives drawn out beforethe project began:

1. The robot can pick up a playing piece and place it on an arbitrary space on a 3x3grid. This is a good starting point to ensure that the robot can actually pick upand place pieces in specific positions.

2. The robot is able to stack up to three playing pieces on top of one another. Due tothe nature of the game, being able to place one playing piece on top of another insuch a way that the stack will not fall over is essential.

3. The robot is able to recognise the current state of the game by detecting piecesas they are played and remembering what position they are placed into. This is acomputer vision challenge. Using the cameras on the robot, the state of the gamecan be monitored, with a trigger to the robot (visual change to the board) to lethim know that the human player’s turn is over.

4. The robot understands the game rules and can play the game against a human ina software environment, using the best possible strategy. This is a software task,implementing the game logic into the robot’s functionality.

3

CHAPTER 1. INTRODUCTION 4

5. The robot develops strategy for playing the game over time. This is an AI problem,where a learning function will be implemented into the robot’s code. The bestpossible strategy preprogrammed will have to be removed at this point and used asa reference to measure the robot’s success.

All but objective 5 have been achieved, and the robot is able to play an entire legalgame.

1.3 Report Structure

The sequence of the chapters in this report follows my development process.Chapter 2 covers research carried out, prior to development. This included documen-

tation on the hardware and software I would be using as well as aspects surrounding theNoughts & Crosses game and related work.

Chapter 3 documents the bulk of the development process. An analysis of each of thesystem’s components is performed alongside its development.

Chapter 4 evaluates the completed system, testing each individual component’s per-formance.

Chapter 5 is a conclusion and final summary of the project. It also discusses possibleextensions and improvements.

1.4 Schedule

Figure 1.1 shows a Gantt chart of the estimated time it would have taken to completeeach of the objectives. In week 7 (9th March 2015) there was a student progress meetingin which I prepared a small demo for my supervisor and assessor. The ’demo’ will bereferred to throughout this report when mentioning this student progress meeting.

Figure 1.1: Time plan for each objective.

Chapter 2

Background ResearchThis chapter documents the various aspects of prerequisite research before developmentbegan; Firstly, the nature of the board game including physical properties, rules, beststrategies and any existing computer representations. Next, potential algorithmic ap-proaches to playing the game via a set of rules, which is followed by robotic paradigmsand their relevance. Finally, research into the robot itself and how one can operate itprogrammatically via ROS, including any previously done projects where the robot hasbeen programmed to pick up and place objects. Further research had to be done duringdevelopment in order to create alternative solutions to methods that did not work asplanned. This has also been documented in this chapter.

2.1 Board Games

The board game I have chosen for the robot to play is 3D Noughts & Crosses. I chosethis over a more complex game with a complex set of rules, like chess, as I wanted thefocus of the project to be robotics, rather than AI. Incidentally, the robotics aspect wouldbe relatively easy to do with board games that feature playing pieces across a 2D plane.Selecting the three-dimensional version of Noughts & Crosses (or Tic-Tac-Toe) was doneso that more work could be put into developing the robot’s motor functions and visionmethods.

2.2 3D Noughts & Crosses

Just like regular Noughts & Crosses, the objective is to align 3 of your own playing piecesin a row, whilst preventing your opponent from doing the same. The exception is that,rather being played in a 3x3 grid, the game is played in a 3x3x3 cube, where the heightof a piece being played, is determined by previously played pieces in that same position(similar to ‘Connect 4’). Figure 2.1 shows a retail copy of the game. Looking at theimage demonstrates that it is not possible to play a piece in a higher position withoutother pieces existing below it.

2.2.1 Physical Properties

The retail version comes with 27 playing pieces. Each piece is either a 3D cross or a sphere(nought). Every piece features a hole running down the interior length. The ‘board’ isessentially a small platform with 9 poles, evenly spaced to form a 3x3 grid (when looked

5

CHAPTER 2. BACKGROUND RESEARCH 6

Figure 2.1: 3D Noughts & Crosses [1].

at from above). The diameter of the poles are no more than a millimetre smaller thanthe diameter of the holes in the playing pieces. This ensures that when impaling a pieceon the poles, it falls to the lowest possible position without any friction. This requiresvery finite and precise movement, which Baxter was not be able to perform, resulting inthe need for a custom version of the board game (Section 3.1.3).

2.2.2 Rules

The game is for 2 players only. One player is in possession of the noughts playing pieces,and the other player possesses the crosses. A starting player is randomly determined andplay begins. On a player’s turn, that player takes one of their playing pieces and places iton one of the 9 poles. Once the piece has fallen into position, the turn ends and the nextplayer takes their turn. This action of placing a piece onto the board I will now refer toas ‘taking a position’ and the player using crosses as the first player.

The end condition of the game is when either all 27 pieces have filled the playing area,or one player has taken three positions with his pieces in a straight line. This line can bediagonal, vertical or horizontal across all axis.

There are a total of 49 winning lines in a 3x3x3 cube (see Figure 2.2). On his blog[8], Owen Elton talks about how tic-tac-toe could be played in a 3x3x3x3 hypercube.Calculating the winning lines on a 2D grid can be done by assigning ‘addresses’ to eachspace in the grid, containing the space’s row and column (such as in Figure 2.3).

address = (row, column)

It is possible to identify whether a set of three distinct addresses form a winning lineby looking at the differences in each element of the addresses. Take the set ‘(a,a,A3)’,


Figure 2.2: Possible winning lines [8].

Figure 2.3: Addresses of grid spaces [8].

notice how the first element of each address is identical and is therefore a winning line.For horizontal and vertical winning lines, the alphabetic or numeric properties of the cells’addresses are all the same. For the diagonals, the properties of the cells’ addresses mustbe all different, however, a caveat must be added that if both alphabetic and numericproperties are all different then the set must contain the central cell (b) to qualify as awinning line [8].

This article helped me when applying these rules to find the 49 winning lines of a 3-dimensional cube, in which each address comprised of 3 elements instead of 2. Section 3.5.4describes in more detail how I applied this information to generate the winning linesprogrammatically.


2.2.3 Strategy

The original game of Noughts & Crosses, has a very simple strategy in which if bothplayers play optimally, the game will result in a draw. Allen Newell and Herbert Simoncreated a Tic-Tac-Toe program in 1972 [7] which followed the following set of rules:

1. Win: If the player has two pieces in a row, they can place a third piece to get threein a row.

2. Block: If the opponent has two pieces in a row, the player must play the third piecethemselves to block the opponent.

3. Fork: Create an opportunity where the player has two possible lines to win.

4. Option 1: The player should create a ‘Block’ situation with two pieces in a rowto force the opponent into defending, as long as it doesn’t result in them creating a‘Fork’ situation.

5. Option 2: If there is a configuration where the opponent can create a ‘Fork’ situ-ation, then the player should block that fork.

6. Center: A player marks the centre.

7. Opposite corner: If the opponent plays a piece in the corner, the player plays apiece in the opposite corner.

8. Empty corner: The player plays in a corner space.

9. Empty side: The player plays in a side space.

This rule set was very useful when developing the 2D version of the game to play. InSection 3.3 I use a similar, but simplified version of these rules for the robot to follow.

For the opening move, there are 9 places the first player can play into. However, thiscan be logically simplified to 3 places, as all the corner spaces are strategically equivalentand all the side spaces are strategically equivalent. This is only for the first turn. Of thesethree moves, the one with the most options to win for the first player is the corner [9].This is because, out of the 8 remaining spaces for the second player, they can only avoid aloss by taking the centre space. If the first player had taken the centre, the second playeravoids a loss by taking a corner, of which there are 4. This information was not usedhowever, for the implementation of the game logic in the robot always had the humanplayer acting first. This decision was made so that certain states of the board could beforced for testing purposes.

The 3D variant of the game has a different strategy. When played on a 3x3x3 cube,the game is easily winnable once a player has a piece in the central position [24]. Withthe retail version of the game, this can occur when one player takes the centre position on


the lowest level of the board, but also having another piece next to it, forcing the otherplayer to block by taking a position on the lowest level. Figure 2.4 shows how the secondplayer is unable to take the central position without losing to the first player on the lowestlevel. It is even possible to force a win from a much earlier point in the game by havingthe first player take the central space on the lowest level, and then taking the space aboveit on their next turn.

Figure 2.4: Two instances where the second player is forced to block.

The result of this is the game devolves into the winner being the player who takes thefirst turn. This discovery made the incorporation of the game logic into the robot veryeasy. It shows that the game is ‘solvable’, which means there is always an optimum moveto take. As the focus of the project is not to develop a challenging AI but rather havethe robot interact in physical space with accuracy, the game did not need to be changedin light of this.

2.2.4 Computer Representation

There are many online games of 3D Noughts & Crosses. However, due to the flawednature of the 3x3x3 variant, the majority of them are played on a 4x4x4 board, which ismuch less predictable. I also discovered that it is possible to play the game without thegravity factor of the retail version. Unfortunately this allows the first player to take thecentral position at the start of the game, further simplifying the game’s strategy.

Figure 2.5: Sequence of moves until game end in 3D [2].

Using screenshots from one of these games, Figure 2.5 shows the next logical movesfrom the example in Figure 2.4, once the first player (orange) takes the central position.


The second image shows the second player (blue) is forced to block by taking the topcentre position, allowing the first player to take a move leading them to win the game.Whatever move the second player makes in this state of the game, they cannot preventthe first from winning.

Switching to the 4x4x4 variant of the game is a possible extension I discuss in Sec-tion 5.1.3. Developing the AI for this will be more challenging though, and like the 3x3x3variant the first player can force a win [12].

2.3 Adversarial Search

There are a number of algorithms that can be used to solve the best strategy for particulargames. This section explores the algorithms which were appropriate for the implementa-tion of the robot’s game logic.

Games usually fall under adversarial search problems. Noughts & Crosses is a commonexample used in game theory, it can be classed as a deterministic, turn-taking, two-player,zero-sum game of perfect information [19]. The game is deterministic because there is noelements of chance, such as dice or decks of cards. Such games are much easier to solvethan games of chance. Perfect information refers to all elements of the game being visibleat all times, whereas imperfect information in games is featured as hidden details, such asan opponent’s hand of cards in poker. Each move made by a player (with the aim to winthe game) adversely affects the opponent’s chances of winning. This is known as a zero-sum game, and as the players are each trying to ensure they win whilst simultaneouslyensuring their opponent loses, it makes the situation adversarial.

2.3.1 Game Tree

A game tree is a directed graph which shows all the possible states of a game. A full treeconsists of the initial state of the game as the root node, and branches down to all thepossible variations of terminal states at the leaf nodes. Each level of the tree representsall the moves which can be made by a player for each turn in the game, and this alternatesbetween the players every level. In the case of a 3x3 grid, the first player can make oneof nine moves from the initial state therefore nine states branch out from the root node.The next level contains all the possible states of the game after the second player makesa move in response to the first, and so on. Figure 2.6 shows a more streamlined gametree for Noughts & Crosses, showing the 3 possible opening moves (mentioned in theprevious section). There are fewer than 9! = 362,880 terminal nodes [19] in the full treeof a standard game of Noughts & Crosses. This relatively small number means that it isentirely plausible to generate a game tree for the 3D variant of Noughts & Crosses, andimplement an algorithmic approach to the robot’s artificial intelligence when playing itagainst a human opponent.


Figure 2.6: Game tree, showing states of playing in a corner, side or in the centre [23].

2.3.2 Minimax

The minimax algorithm can be used to search a game tree to calculate the best possiblesolution at each given state. As mentioned in Section 2.3.1, each level of the Noughts &Crosses tree alternates between the players. Minimax goes through these levels maximis-ing the chances of winning for one player and minimising for the other. The algorithm canbe described by naming the players ‘Min’ and ‘Max’. Max will go first and both playerswill keep playing until the end of the game. This creates a game tree similar to Figure 2.6.When the terminal states are reached, scores (known as utility values) are placed at theleaf nodes, a high value for a terminal state where Max won and a low value for whereMin won. These utility values are calculated at all the child nodes in the tree. Figure 2.7shows an example of how this is done. Starting at the parents of the leaf nodes, Maxwill choose the highest value from each of the child nodes (In the left-most branch, Maxchooses a 3 because it is higher than the 2). The chosen value is labelled at the parentnode, and Min now must choose the lowest value from these nodes to label their parentnodes (in the left-most branch, Min has also picked the 3 because it is lower than the 9).Going up through the tree, the initial node is reached, which is the 3 in this example.This path is how the current player decides which move to make. The move being chosenwill follow the path that benefits them (Max) the most whilst hindering the other player(Min) the most.

The algorithm performs a complete depth-first exploration of the tree, using a simplerecursive computation of the minimax values for each successor state [19]. At each node(or state) of the tree, depending on whether the depth of that node is either Max’s orMin’s turn, the algorithm will choose the highest or lowest utility value of its child nodes,respectively. However, in order to work out the utility value of the child nodes, the mini-max computation must be run on their respective child nodes. This recursion continuesto the leaf nodes (or terminal states) of the tree, and then the minimax values are passedup through the tree where they were required, resulting in a tree like Figure 2.7. Once


Figure 2.7: Minimax Example [3].

all the child nodes have utility values assigned to them, the decision aspect mentioned inthe previous paragraph can be carried out.

2.4 Robotic Paradigms

A paradigm describes the approach to a problem whilst implying a possible structure forits potential solution [11]. A robot paradigm makes use of three primitives which can beused to categorize functions within the robot.

• Sense - Handle raw information from sensing/detecting aspects of the robot’s en-vironment. Examples include camera feeds and range sensors.

• Plan - Take information from either sensory input or innate knowledge, then com-pute appropriate actions to reach a goal state.

• Act - Perform actions which result in the manipulation of the environment. Thisincludes a change in location of the robot itself via movement.

A given paradigm describes the relationship between these three primitives [11].

2.4.1 Hierarchical Paradigm

This paradigm features a top-down approach. The robot first takes sensory data of thesurrounding environment until it has enough data to move onto the next stage. Fromthe sense stage, it takes the data and formulates a best possible action to perform in theplan stage. Finally, this action is physically performed by the robot’s motors in the actstage. This continually loops until a goal state has been reached. The downside of thisparadigm is that a lot of time is lost during the plan stage. In a dynamic environmentthis will come across as an unresponsive robot, since the computations of the plan stagewill not take into account that aspects of the environment can change during this time.


However, due to the environment featured in my project remaining static for the durationof the game, this is the paradigm that I adopted.

2.4.2 Reactive Paradigm

The reactive paradigm comprises of multiple SENSE-ACT couplings, eliminating the needfor a plan stage. This is the solution to unresponsive robots in dynamic environmentsfor each type of sensory input directly translates to a particular action. The downsideof this paradigm is that for complicated sensory inputs, there must be an increase inthe number of SENSE-ACT couplings for each possible environmental reaction. As thenumber of these environmental factors grow (sense), so do the reaction functions (act),and the difficulty arises in managing them.

2.5 Robot Operating System (ROS)

ROS is a free, open-source framework for writing robotics software. Its flexibility comesfrom a wide selection of tools, libraries and conventions. The Baxter research robot usedin this project runs on ROS, so becoming familiar with it was a prerequisite step.

2.5.1 Installation

Each distribution of ROS is targeted at running in specific releases of the Ubuntu distribu-tion of Linux. ROS Indigo is the distribution that the robot is currently running, which isprimarily targeted at the Ubuntu 14.04 LTS release [18]. After installing VirtualBox VMsoftware on my machine, I installed Ubuntu 14.04 with ROS Indigo in order to familiarizemyself with the ROS framework and follow the tutorials found on the wiki.

2.5.2 Architecture

ROS works very much like a file system. Most of the functionality comes in the formof terminal commands; ‘roscd’ to change directory, ‘rosrun’ to run a program. Everyproject created in this file system is stored in the form of a package. The package containsall software and data, as well as header files to import necessary dependencies. Once apackage is ready to be built, the ‘rosmake’ command is entered to compile all the packagesin the file system alongside all the required dependencies.

2.5.3 Topics & Services

A robot is usually a series of connected nodes and sensors. The nodes tend to be motorswhich allow the robot to interact with the physical environment and the sensors are aform of input, containing information about the robot’s surroundings. ROS is able to


interact with these aspects via the medium of Topics and Services. ‘rostopic list’ willlist all the available topics. Topics are the buses in which the nodes in a robot exchangemessages. An example of a topic could be the rotational state of a particular joint in thearm of a robot.

Services are subscribed to in order to utilize information which is feeding into therobot. For example, to output the image feed from a camera on the robot, the softwaresubscribes to that camera’s service. ‘rosservice list’ lists all the available services.

2.6 Baxter Research Robot

Figure 2.8: Baxter Research Robot by Rethink Robotics [4].

Baxter is an industrial robot built by Rethink Robotics and introduced in September2012 [21]. Baxter is a 3-foot tall (without pedestal; 5’10" - 6’3" with pedestal), two-armed robot with a screen that can be used as an animated face. It weighs 165 lbswithout the pedestal and 306 lbs with the pedestal [16]. Throughout the project I workedwith the robot whilst it was mounted on its pedestal. Every joint in each arm features aservo motor, whose angles are adjusted in order to achieve movement. Holding the ‘cuff’


off an arm puts the robot into ‘Zero Force Gravity Compensation mode’, allowing freemanipulation of the limb by physical guidance. The robot has 3 cameras, one in the wristof each arm and one built into the face display. There are also infra-red sensors on thewrists which calculate distance. The robot comes with several interchangeable grippers forthe arms each varying in width and length. There is a selection of holes that the gripperscan be screwed into in order to adjust the overall distance between them (see Figure 2.9).The grippers are opened and closed by a servo motor as well. I will be referring to theBaxter Research Robot as ‘Baxter’ for the rest of this report.

Figure 2.9: Camera, infra-red sensor and gripper adjustment [15].

2.6.1 Baxter Software Development Kit (SDK)

The Baxter SDK provides an interface to allow custom applications to be run on theBaxter. The main interface of the Baxter SDK is via ROS Topics and Services, which iswhy learning ROS was a necessity. The SDK comes with libraries to control Baxter fromprograms. The ‘Hello Baxter!’ tutorial on Rethink Robotics wiki shows how to store theangles of the motors in Baxter’s arm, manipulate them, and then update the arm withthe new angles. The SDK also includes example scripts to run on Baxter, including theimportant ‘tuck_arms.py’ which I used at the start and end of each development sessionto store the robot. There is another example which maps every motor in the robot tothe keyboard. This is a very impractical way of controlling Baxter for the movements arenot precise. However, I found the control of the grippers in this script useful in testingdifferent gripper sizes on varying objects.


2.7 Existing Projects

It was important to find out whether my project had been done before. Finding existingprojects prove useful in seeing where others have succeeded or failed in the research youare planning to do. In searching for projects identical to my own, I discovered that thisproject is entirely unique. However, projects with similar goals and solutions were found.

2.7.1 Worked Example Visual Servoing

This example [14] is targeted at 1st-2nd year undergraduates on courses similar to my own.The aim of the project is to have Baxter pick up golf balls, and place them into slots in anegg tray. The example uses Visual Servoing for workspace relationship and calibration,Visual Object Recognition and target selection. It also makes use of inverse kinematics.This project helped me greatly during the development phase of my project. Section 3.4.3,Section 3.2.5, and Section 3.2.7 detail development work which was influenced by thisexample, specifics can been seen in each section. I will be referring to this project as the‘Golf Example’ from now on.

2.7.2 Connect Four Demo

The Connect Four Demo [17] features Baxter playing the original game of Connect 4,either against a human opponent or itself. There is a requirement for a custom game piecefeeder which allows Baxter to pick up the game pieces with ease. The board detectionaspect has the user click and drag over the image coming from the camera feed, to highlightthe board. An HSV blob detector (like the one I use in Section 3.2.1) is used to identifywhich colour game pieces have been played.

2.7.3 Baxter Solves Rubiks Cube

This project [6] has Baxter solving the Rubik’s cube puzzle. The stages of runtime areseparated into detection, verification, solving and manipulation. Detecting the currentstate of the cube is done with the use of a flatbed scanner. Baxter does not utilise anyof its detection hardware for this project. The location of the scanner is taught by usingBaxter’s ‘Zero Force Gravity Compensation mode’. It then scans each of the six sidesof the cube and verifies that it has scanned them correctly. The Kociemba algorithm isthen run to solve the cube and return a list of the required manipulations. Baxter thenperforms these manipulations, in a manner similar to that of a human. This project didnot feature any aspects that crossed over with my objectives, therefore I did not refer toit further during development.


2.8 Software

2.8.1 Programming Languages

The available programming languages for ROS are Python and C++. I selected to usePython, due to its simplicity and the fact that the Baxter SDK examples were also inPython. The performance advantage gained from using C++ is not necessary for thepurposes of this project.

2.8.2 Libraries

Apart from the Baxter SDK, another library I made use of was OpenCV. As the nameimplies, it is an open source set of programming functions aimed at doing computer visionin both images and video frames. All of the computer vision aspects of the project madeuse of this library.

2.8.3 Version Control

A repository was set up on my Bitbucket account for all the source code (including thisreport) developed during the project. I used git commands to pull and update code fromthe repository when changes were made. This method allowed me to work on the projectfrom multiple places as well as provided a backup in case of any hard disk failures. Thelink to the repository has been included in Appendix D.

Chapter 3

Development & AnalysisThis chapter investigates the aims and objectives set in the first chapter, and the possiblesolutions to go about achieving them.

The structure of this chapter mimics the hierarchical paradigm of Sense, Plan, Act(from Section 2.4) used in the robot’s software. The first section covers the setup ofthe hardware used in the project. This is then followed by the Vision (Sense), GameLogic (Plan) and Robot Control (Act) sections. The final section summarises all the priorsections and how the components developed in each of them fit together. This includes amore in-depth look at the code.

Each iteration of development builds upon the previous, with additional features beingadded. All the code has been stored in separate files for each iteration and are referredto as a ‘.py’ program (e.g ‘examplescript.py’). Details on how to run these programscan be found in Section 3.5.1.

3.1 Hardware Setup

3.1.1 Environment

All development done on the robot had been in a constant environment, with sufficientlighting.

Figure 3.1: Baxter’s physical workspace environment

18

CHAPTER 3. DEVELOPMENT & ANALYSIS 19

Figure 3.1 shows the table in which Baxter is stood in front of, which was kept at thesame height throughout the project.

3.1.2 Grippers

The Baxter research robot comes with a number of grippers of varying size which can beattached to the arm.

Size

Each pair has a minimum width, in Figure 3.2 the two pairs of grippers in the centre werenarrow enough to pick up the custom made game pieces. I settled on the right-most oneas I did not require the longer length. The shorter piece meant that more tension wouldbe applied to the picked up object, lessening the chances of it being dropped by accident.

Figure 3.2: Electric Parallel Gripper Kit [5].

Shape

The kit comes with several pairs of rubber tips which slot onto the ends of the grippers.The ends of these tips are either rounded or square. Initially I did not take into con-sideration the importance of deciding which tip to use. When it came to implementingHough Circle detection (See Section 3.2.4) I noticed that the rounded gripper tips werebeing detected as circles by the program. You can see in the screenshots that the endsof the grippers are always in view. Figure 3.8 shows the rounded tips in the image feed.Changing the tips was a simple matter. The difference the square tips make can be seenin Figure 3.9.

3.1.3 Custom Board Game

The pieces of the retail version of 3D Noughts & Crosses were too small for Baxter’sgrippers. Therefore, part of the development process during the project was to createa custom version of the board game, playable by Baxter. To do this, I cut a 2 metre


long, 62mm x 37mm plank of wood into 62mm x 62mm x 37mm blocks. These were thencoloured green (crosses) and blue (noughts). Figure 3.3 shows the playing pieces.

Figure 3.3: Custom board game pieces (originally red).

Figure 3.4 shows the design of the final board. I created a Python script, ‘genBoard.py’to generate this image using OpenCV. This was to ensure symmetry and positioning ofgrid spaces and the circles, and matching the image resolution to the pixel dimensions ofA3 paper. OpenCV has functions for drawing perfect squares and circles which were allanchored off the image’s central pixel coordinates. The circles were used to calculate thecentre points of the grid positions. More details of this are in Section 3.2.4.

Figure 3.4: New board design, generated in OpenCV.

White circles were drawn onto one side of the human player’s pieces (Figure 3.5). Thiswas due to the limitations of background subtraction (described in Section 3.2.3) whendetecting two pieces of the same colour on top of one another. The implementation ofthis can be found in Section 3.2.4.

3.2 Vision

To achieve the first objective (Section 1.2), I had to implement the ability of enablingBaxter to pick up an object from the table, regardless of the object’s position. In orderto do this, the robot would first have to correctly identify objects. This section is divided


Figure 3.5: Updated custom game pieces.

into the different methods Baxter uses to detect and locate objects in the environment,along with their analyses.

3.2.1 Blob Detection

To simplify this task, I took advantage of two programs developed by members of theActivity Analysis group working with the robot Baxter. Both programs ‘subscribe’ to thelimb camera and work on the output image feed. They are designed to work in conjunctionwith one another and are included with the rest of the software developed for the project.

The first program ‘Object_finder.py’ is used to isolate objects based on their colour.It features RGB sliders for both the upper and lower ends of the colour spectrum (totallingin 6 sliders). Adjusting the sliders alters a black and white filter on the image feed fromBaxter’s limb camera. The purpose of the program is to isolate each object to be tracked.This is done by adjusting the sliders in such a way that the object appears white, withthe rest of the image feed entirely black (an example of this can be seen in the right imageof Figure 3.6). The program generates a text file containing the saved RGB values foreach object. The ‘Object_tracker.py’ program uses the generated text file as an inputconfiguration file.

Blob detection is a mathematical method aimed at detecting regions in a digital imagethat differ in properties, such as brightness or colour, compared to areas surrounding thoseregions [22]. The ‘Object_tracker.py’ program uses a blob detector in OpenCV to locatethe objects in the camera feed. By analysing all the pixels of every frame, a cross canbe drawn at the centre point (or centroid) of the similarly coloured groups of pixels (or‘blobs’), which match the RGB values of the configuration file. This tracking methodwas used to differentiate between the different coloured pieces of the custom board game.Using something like an edge detector would not have been a useful solution as all thepieces are of similar shape. If using different shaped pieces, such as in the retail version,using an edge detector would be a viable option. However, the aspects of physicallypicking up various shapes of different pieces would add to the complexity of the project,


which isn’t the project’s focus.

Colour Calibration

When working with Baxter in the evenings, the change in lighting would greatly affectthe colour levels coming in from the camera feed. The colours of the game objects Iwas working with at the time were red, green and yellow. With the ‘Object_finder.py’program, it was very hard to isolate the original red blocks from the green blocks in lowlight conditions. It was possible, but the small amount of pixels detected would resultin a fluctuating target; Baxter would target one block, and as his arm moved over it,the small amount of light would be blocked out. The robot would then target anotherblock and move towards it, blocking out the light again. This process of moving backand forth would repeat to no end. The program for isolating colours uses an HSV (Hue,Saturation, Value) input from the camera feed. Changing this to RGB and isolating eachchannel separately may have fixed this problem. However, due to time constraints I didnot employ this method. Instead I wrapped each of the red painted pieces in a matt blueinsulation tape. This blue was much easier to isolate from the other colours and viceversa.

Area Threshold

The blob detector uses image moments, a weighted average of the image pixels’ intensities,from the resulting mask of a filtered frame (like the right most image of Figure 3.6) towork out properties such as the area and the centroid of a blob. The target object oftenhas multiple blobs across its surface. This can be seen on the green blocks of the leftimage of Figure 3.6. Here, the green contours drawn around the blobs are showing thedetected areas. A reason for these divisions could be a reflection of light across the surfaceof the object, which will split up the blob. To avoid this problem, I set a minimum areathreshold. Using the moments from the image I can draw contours around the blobs whichmeet this threshold. Working out this threshold was a trial-and-error process. Having thethreshold too high lead to the object not being detected, whilst having it too low resultedin one object being detected as multiple objects.

3.2.2 Contour Detection

In a later iteration of the project, a flaw arose in using solely blob detection. Whenwanting to detect two pieces of the same colour, the centroid of the blob becomes thecentre of the two objects. In order to differentiate between objects, contours had to beimplemented. A contour is the outline or bounding shape of an object. In OpenCV, it ispossible to draw contours around blobs of pixels. By combining the output mask from theblob detector with the contour detection features of OpenCV, I was able to draw crosses


at the centroids of the contours rather than the blobs. Figure 3.6 shows how this resultedin being able to accurately differentiate between objects of the same colour.

Figure 3.6: Screenshots of Contour tracking (left) over blob detection output (right).

Detecting the board and working out where all the spaces are, was first attemptedwith this contour/blob method. This was when a yellow piece of A3 paper was used forthe board. Using this method, working out the centre point of the paper was trivial forit was just a very large blob. I then had to calculate the 8 remaining points to make upthe playing grid. Initially, I wanted to only use the output mask from the blob detection.Testing showed that Baxter converged to the centroid of a blob quicker than the centroidof a contour based on that blob. This was because the contours constantly adjust basedon the blob, so the centroid is less stable. Unfortunately, OpenCV has no functions tocalculate the dimensions of a blob, which is what was required to establish the grid spaceson the board. Switching to contours meant I could draw a bounding rectangle aroundthe contour of the board, which would give me the dimensions I needed. Figure 3.7shows a screenshot of the tracked board with calculated playing spaces. There was a

Figure 3.7: Blob and contour detection of the board with contours (green), boundingrectangle (red) and grid spaces (purple).

problem with this implementation, however. The grid centres’ positions are all based onthe centroid of the contour surrounding the yellow board. If the arm was to move to oneof the outer grid points, part of the board will fall off the edge of the camera’s field ofview. When this occurs, the size of the contour gets smaller, which in turn changes whereBaxter believes the centre point to be, because the calculated centroid will shift position.


In order to remedy this, I hard coded the dimensions of the A3 paper. After finding apose above the centre point of the board (using contours) at runtime, the positions inCartesian coordinates of the other points are calculated using these dimensions. Thismethod changed with the implementation of the new board design. Details of how thepositions above the other grid points are calculated with pixel-to-world transformations,can be seen in Section 3.2.5.

Analysis

Contour detection proved to not be the most accurate way of calculating the positionson the original board design, seen in Figure 3.7. Seeing as the board did not featurean actual grid (only a superimposed grid which was calculated by Baxter) there wasno middle ground for where each player should be placing their pieces. This became aproblem in the 3D implementation of the game. A grid point for the human player wasnot exactly the same as where Baxter was calculating it, which resulted in stacks of blocksnot being stable. This was mainly due to the board being a big yellow square where thehuman had to guess the locations of the grid lines. For these reasons, the new boarddesign (Figure 3.4) was produced.

3.2.3 Background Subtraction

The question arose of how the robot would be able to detect and recognise moves made bythe human player. Using the position stored as the centre point above the board object, Icould compare frames from the camera feed of before and after the human player has madea move. By taking the newer frame and subtracting the old one, the resulting image wouldreveal any new changes to the board. This method is known as background subtraction.The final implementation of this in the project was as follows; after Baxter has calculatedall the board positions from its centre point above the board, it stores a frame from thecamera feed. Then prior to each of Baxter’s turn in the game, the arm returns to thiscentre point, stores a second frame and compares the two. Figure 3.8 shows an exampleof how the software processes this information. After using OpenCV to subtract thebackground, the new objects are revealed. In order to identify the correct object, I usedthe blob detector with contours on the resulting image to calculate the centroid of the newtarget object. The distance between this point and each of the nine grid points (shownin Figure 3.7) are compared, and the shortest distance is used to determine which of thenine grid spaces the human player has played a piece. This iteration can be seen in the‘detectBoardPositions.py’ program.

Analysis

For the 3D implementation of the game, background subtraction had its limits. Themethod worked for when the human player played a piece straight onto the playing grid


Figure 3.8: Left to right, top to bottom:Sequence of background subtraction functionality.

or on top of one of the robot’s pieces. It did not work if the human player wanted toplay a piece of top of his own piece, because the subtracted image would not show a bigenough difference for the blob detector to recognise another green piece being played. Atan attempt to fix this, the infra-red sensor was reintroduced to scan the heights of eachstack but due to the sensors limitations (described in Section 3.4.2), this solution was notviable. The most accurate way to detect same-coloured pieces on top of one another wasdone via the use of Hough Circle detection and drawing white circles onto one side of thecustom playing pieces (Figure 3.5).

3.2.4 Hough Circle Detection

Hough Transform is a technique used to extract features from an image in computervision. The circle Hough Transform is a specialized version of this used to detect circles.OpenCV has a ‘HoughCircles’ function which extracts circles from an image and placesthem into an iterable object.

The new board was designed to utilize Hough Circle detection. ‘detect_object2.py’was the code I used to experiment ideas using this type of detection. The program usesOpenCV to draw circles around the points where circles are detected. The centre pointsof these circles are stored in an array. The point closest to each corner of the screen isidentified by using Pythagoras theorem to calculate the shortest distance. These fourcorners correspond to the four circles of the new board design (see Figure 3.4). The nextstep was to calculate the centres of each grid space from the coordinates of the circles’centres. The result of this can be seen in Figure 3.9.

This method works because the generation of the board was completely symmetrical.The distances between the grid spaces’ centre points were equal, and the distances between


Figure 3.9: Calculating the grid spaces of the new board design, using Hough Circledetection.

the circles’ centre points and the nearest grid space’s centre point to that circle were alsoequal. The top and bottom rows of the grid were simple to calculate. For the top row,the change in x and y pixel coordinates was divided by four. Then adding these dx anddy values to the top-left circle four times and drawing dots each time, results in the toprow of grid centres being drawn. The same calculations were used for the bottom row.The middle row had some added complexity. The point between the left pair of circlesand the point between the right pair of circles had to first be calculated. Then, using thesame calculations for the other rows, the middle set of grid centres could be drawn.

Background subtraction was not an entirely viable method for detecting moves madeby the human player, the case of a piece being played on top of a same coloured piece, inparticular. The solution to this was to implement the Hough Circle method I used for thedetection of the new board design. Figure 3.5 shows the update of painting equal sized,white circles onto one side of all the human player’s game pieces. A new game rule hadto be introduced to utilize this method; when the human player makes the move to playa piece on top of their own, they must place the new playing piece with the circle sideup. If there is already a circle on the piece being played on, the new piece must be playedwith the circle side down. For all other cases such as playing onto an empty grid space oron top of one of the robot’s playing pieces, the new piece must be played with the circleside down.

If the background subtraction method fails to detect any new pieces when Baxterchecks for the human player’s move, another frame is stored. The Hough Circle methodis applied to this frame to extract the centre points of any circles. The function hasparameters to specify a maximum and minimum diameter for which circles are detected.This allowed me to isolate the circles on the game pieces from the circles on the cornersof the new board design. Any circles in this new frame have the their distances toeach grid space’s centre point compared. If the distance is less than a certain threshold


number of pixels, then that circle is determined to be inside that grid space (similar tothe background subtraction method). The program stores which grid spaces have alreadyhave circles detected inside them. This allows Baxter to differentiate between a circle thatwas revealed in a previous turn, a new circle or whether a circle has been occluded byanother piece. The latter of these cases occurs when the human player vertically stacksthree of his pieces in a row.

These Hough Circle methods were implemented into the final iteration of the project,‘New3DtictacBaxter.py’.

Analysis

Using Hough Circle detection did not only provide the solution to detecting same-colourstacked pieces, but also provided accurate piece positioning for Baxter via a more reliableboard detection. The method allowed for the board to be oriented at off-angles. As longas the board was of landscape orientation, then Baxter’s poses for placing pieces wouldbe calculated accurately. This did not work with the contour detection, for the boundingbox used to calculate the grid spaces (via the old method) would align itself with thecamera instead of the board.

An initial advantage to using Hough Circles to detect the board, was that the playingpieces did not obscure the corner circles, allowing the board to be detected mid-play. Thismeant that if the board was knocked or shifted out of place for whatever reason, Baxterwould be able to readjust to the new position and correctly update its poses for each gridspace. Having circles painted on the backs of the playing pieces lead me to believe thatthis feature would be not be possible due to disturbance of multiple circles in the playarea, affecting detection. In hindsight, I believe the implementation of this feature wasentirely possible due to the parameters on the ‘HoughCircles’ OpenCV function filteringthe sizes of radii detected, preventing disturbance.

3.2.5 World Coordinate System Transformations

In order to improve the accuracy of the stored poses above each of the grid space centres,a world coordinate system transformation was implemented.

World Coordinate System (WCS) transformations is a way of converting pixel coordi-nates to real world coordinates. In the project, the WCS transformation performed wasconverting the pixel coordinate differences of the camera feed centre point to each of thegrid space centre points, into world coordinates. These would then be converted withinverse kinematics into joint angles, allowing Baxter to position its limb above the correctgrid space. This equation [14] shows the conversion from the Golf Example [14]:

B = (Pp − Cp)× Cc ×Dt +Bp +Go


where:

• B = Baxter coordinates

• Pp = target pixel coordinates

• Cp = centre pixel coordinates

• Bp = Baxter pose

• Go = gripper offset

• Cc = camera calibration factor

• Dt = distance from table

When implementing this into the project, a simplified version was used:

B = (Pp − Cp)× Cc ×Dt +Bp

This was because the code for picking up an object already took into account the gripperoffset. The camera calibration factor had to be calculated by performing a calibration teston the camera. This test is documented in Section 3.2.6. The method was implementedinto the final iteration, ‘New3DtictacBaxter.py’. After the central grid space is aligned(using the pixel alignment method in Section 3.2.7) with the limb’s camera, this equationis used to calculate the eight surrounding points of the grid. Originally, this was done bycalculating the offsets to the central pose using hard-coded dimensions of the game board.

3.2.6 Camera Calibration

The World Coordinate System transformations in Section 3.2.5, required a constant value(Cc) to correctly convert pixel distances into world distances. This constant is referred toas the camera calibration factor and this section documents the test performed in orderto find it.

Using a method similar to generating the board design, ‘gen_test.py’ is a script whichgenerates a symmetrical image as shown in Figure 3.10. All the shapes were drawn usingOpenCV functions. The purple cross in the centre was drawn using the image’s pixeldimensions in order to get the exact positioning. The centre points of the circles are alsoperfectly represented by smaller purple crosses.

Using the following equation based on the WCS transformation equation from Sec-tion 3.2.5:

Cc =(DC

Dt)

Pd

where:


Figure 3.10: Camera calibration test image.


• Dc = world distance between the image centre and circle centre

• Dt = distance from table

• Pd = pixel distance between the image centre and circle centre

The ‘pixel_calib.py’ program calculates the camera calibration factor over thecourse of runtime. Using the Hough Circle technique from Section 3.2.4, the distancesbetween all the circles and the centre point is calculated. The cross at the centre of theprinted image must be aligned with the cross drawn on the image feed by the program,which represents the centre pixel point of the camera. The Dc value of the equation isthe distance measured with a ruler between the image centre point and the circles of theprinted image. The results of this after 1000 iterations can be seen in Figure 3.11. Thecamera calibration factor for each circle is calculated and then a ‘final c’ average is takenfrom these. It was this value that was used for any WCS transformations in the project.

As an experiment, I wanted to see if the camera calibration factor stayed the samefor all heights. This would mean that the lens in the camera was curved to allow WCStransformations to work at any height with the same constant. Dropping the height by10mm yielded a result that differed by 0.0001, as shown in Figure 3.12. This meant thatthe lens did not compensate for different heights and that the calibration factor was afunction of the distance to the table.

3.2.7 Pixel Alignment

There was an issue with picking up objects directly from a pose using World CoordinateSystem transformations. In the 3D implementation of the game, a stack of playing pieces


Figure 3.11: Camera calibration test results.

can obstruct the line of motion from above the board, to a particular grid space. To solvethis issue, a crane-like motion was adopted to ensure that the limb would only move downto the plane of the board when it was directly above the desired location. The advantageof this came in the simplicity of the limb only needing to move through the X and Y axis,keeping Z axis constant. Once the limb reached a position above the object, the arm couldthen descend along the Z axis, using the functionality of the ‘pickupDropwithRange.py’program, and pick up the object. This technique was influenced by the Golf Example[14].

‘pickupColour.py’ is the program that was developed for the project demo. It cor-rectly aligned the limb with the object, and then descended to pick it up once it wasaccurately positioned above it. It even followed the object around if it was moved beforethe descent. In order to correctly align the arm with the object, the centre pixel of theimage feed had to be aligned with the centre pixel of the object.

The Golf Example [14] had a relatively simple way of doing this. By calculating the Xand Y offsets of the object from the camera’s centre, the overall position of the arm couldbe updated (using inverse kinematics, Section 3.4.3) along the X and Y axis, keeping thez position the same. This was done by the following equation for each axis:

O = −Dp × Cc ×Do

where:

• O = offset


Figure 3.12: Camera calibration test results at a lower height.

• Dp = displacement of target pixel from the image’s centre


• Do = distance to the top of the object

Notice how similar this equation is to the equation used for WCS transformations inSection 3.2.5.

An error value is also calculated and the whole process is repeated until the errorbecomes small enough to meet a set threshold. At this point, the arm’s pose will beupdated, but this time the X and Y values were kept constant and the Z value waschanged to the distance to the top of the object.

I chose not to make use of the IR sensor for the descent of the arm for the purposes ofthe demo. The reason for was that iterating through the heights until meeting a thresholdwas very slow, especially in combination with the iterations for finding the correct Xand Y positions. The Golf Example comes with a setup script, ‘golf_setup.py’, whichcalculates the distance from the limb’s start position to the table, using the IR sensor.It then outputs this to a file which I then use for my program. It proved much quickerto calculate all the heights prior to the robot’s runtime, the only height that had to behard-coded was the height of the object, any further heights being a multiple of this.

This pixel alignment method was used to track the centre pixel of the board. Sec-tion 3.2.4 describes how this centre point is calculated. Just like when targeting an object,an error value is updated until a predefined threshold is met, resulting in a pose located


directly above the centre of the board. This is when the poses above the other grid spacesare calculated.

Offsets

After Baxter targets an object, it pick it up by updating the Z value of the arm’s endpointuntil the distance to the object has been reached. As you can see in Figure 2.9, the camerais a fair distance away from the gripping point of the grippers. In order to ensure thegrippers enclose around the object, and offset has to be added to the pose when pickingit up.

The implementation of this offset featured a trial-and-error approach in finding thebest values. First, I physically measured the height and width offset from the centre ofthe camera lens to the middle of the grippers. These values were added (or subtracted)to any pose that went into the inverse kinematics function when Baxter lowered its armto pick up an object. By stopping the arm right before the grippers would close, I couldmeasure how far off the grippers were from the centre of the object, and increment ordecrement the offsets accordingly. The height at which the gripper picked up the objectwas another offset that had to be set. During the robot’s setup, it works out the distancefrom its arm to the table using the IR sensor. The offset to be set is the distance betweenthe centre of the IR sensor and the end point of the grippers. The current calculation topick up an object of predefined height is a change in the Z axis. The table distance, minusthe object’s height, plus the offset gives this value. It was important to ensure that thegripper did not close around the object at too low a height. This could lead in collisionswith the table and damage to Baxter. If the grippers closed around too near the top ofthe object, there is a chance that on occasion the grippers could fail to pick up the objectentirely.

These offsets are also applied when Baxter places pieces onto the game board.

Stacking

As an add-on to the project demo, I implemented a stacking feature and a class-basedstructure for the current code. The new structure is described in more detail in Sec-tion 3.5.2 and is viewable in the program ‘pickupStack.py’. The stacking feature re-members the pose for the location above the first object, locates and picks up the secondobject, placing it on the first. The height to release the second object from is calculatedusing the vertical offsets. By doubling the stored height of the object and using thatvalue in the pose for placing down the object, the new height is equal to the height of twoblocks. Due to the new class based structure, this implementation was trivial. It can beeasily modified to stack a higher number of objects (tripling the stored object height tostack of three blocks, for example).

This add-on was used as the prime functionality of playing the board game. Baxter’s


turn to play a piece into a space on the board, uses the stacking feature to stack theobject onto the space. The level at which the object was stacked on was dependent onhow many objects were currently in that space (if there were zero, then the object wasplaced directly onto the board).

3.3 Game Logic

3.3.1 2D

Baxter needed to know how to play the game. According to the objectives, this wouldfirst have to be achieved in a software environment before applying Baxter’s physicalcapabilities.

The first step was to essentially create a simple tic-tac-toe game in Python. The‘tictac.py’ program is a game of the 2D version of Noughts & Crosses, playable inthe Terminal window. Implementing the 2D version of the game is a prerequisite stepbefore the full 3D version. I set up the code so that it could be easily adapted to higherdimensions.

The program features play between a human and the computer. On the turn of thecomputer, the AI has been set up to follow this sequence of possible moves in order:

1. A winning move

2. A move to stop the opponent winning

3. A free random corner space, if there is one

4. The centre space, if it’s free

5. A free random side space

The method to check whether a winning move can be made or whether the playerneeds to be blocked to prevent them from winning, is done by making a copy of the board(function of the board class in Section 3.5.4) for every free space and filling each spacewith either playing piece. Then each copy of the board is checked against the possiblewinning lines (‘checkWinner()’ function, also from the board class) to see if that potentialmove will result in a win for either side.

This is not an unbeatable AI, it is possible to win by taking two opposite corner spaces.Now that the AI will try to block the player, a third corner can be taken on the player’snext turn, putting them in a position of winning the game. This strategy is stoppable bytaking an opposing side space, as shown in Figure 3.13 and mentioned in Martin Gardner’sbook [9]. For the sake of testing, I have not programmed in this defence. This is so thatwhen Baxter comes to playing the game, I now have strategies to force a Win, Loss andTie.


Figure 3.13: The second player taking the opposing side space of the taken corner.

3.3.2 3D

The 3D implementation of the game took the game’s current data model, which was a 2Dmatrix represented by an array of arrays, and stretched it into another dimension. The3D matrix of the new implementation featured three of the 2D matrices from the previousiteration, one for each level of the cube.

Along with a new set of winning lines, which Section 3.5.4 describes the method ofgeneration, the implementation required a new set of conditions for the AI to play thegame. I kept the code for making a move into the grid on a 2D interface the same; the(row, col) format of where the move was made now places a piece in the lowest possiblelevel of that position. This meant the rules followed in Section 3.3.1 did not need to bealtered much. The new sequence of possible moves the AI follows are:

1. A winning move (unchanged)

2. A move to stop the opponent winning (unchanged)

3. The centre space on the middle level, if there is a robot piece below it

4. The centre space on the bottom level, if it’s free (unchanged)

5. A free random corner space on the bottom level, if there is one

6. The centre space on the middle level, if its free

7. A free random corner space, if there is one (unchanged)

8. A free random side space (unchanged)

This is by no means a perfect AI, but still presents a decent enough challenge whilstallowing me to force certain board states for testing purposes.


Generating Winning Lines

The 2D version of the game had all the possible winning lines hard coded into the script.This was plausible as there were only 8 possible lines. The 3D version, however, has atotal of 49 winning lines, detailed in Section 2.2.2. With the help of Owen Elton’s blog[8], I devised a way to generate these lines at runtime. The ‘BoardClass3D.py’ class doesthis in its constructor. The generator function generateSolutions(), iterates throughevery group of three distinct spaces in a 3x3x3 cube. Certain conditions are then checkedto determine which of these groups form a winning line. Each space in the cube has anaddress of three values ranging from 0 to 2. The address is read:

space = (level, row, column)

For example, a bottom corner would be (0,2,2) and the central piece (1,1,1).Every group of three addresses has two conditions checked on each element of all the

addresses, totalling in six ‘checks’ (two conditions on all the levels, two conditions onall the rows, etc.). The first condition is whether an element of the address is the sameacross all the addresses. If one element across all the addresses is the same, then we havea winning line in the 2D game [8]. If two elements across all the three addresses are equal,then we have a winning line in the 3D game. This works for all the non-diagonal lines inthe cube.

The second condition is whether an element of all the addresses is distinct but alsoin an ascending or descending order (all the lines are sorted before these conditions arechecked). Golomb and Hales study the analogue of tic-tac-toe played on a k-dimensionalhypercube of side n [10]. The idea for the second condition was inspired by the paper. Nowif all the elements across the three addresses were either in an ascending or descendingorder, then the group would represent one of the four diagonal lines in the cube, movingthrough the central space (e.g. [(0,0,0), (1,1,1), (2,2,2)] and [(0,2,2) ,(1,1,1), (2,0,0)]).

Finding the rest of the winning lines of the cube was done using a combination of thetwo conditions on each group of three addresses. If two elements across the addresses werein either an ascending or descending order (as per condition two) and if the third elementacross the addresses are equal (condition one), then this group represents a winning line.

3.4 Robot Control

The first successes of controlling Baxter was via the SDK example programs and the‘Hello Baxter!’ online tutorial. The first program I developed was based on this tutorialand was an important stepping stone in understanding how Baxter sees the environment.


3.4.1 Basic Movement

The first objective in Section 1.2 required Baxter to pick up and place a playing pieceinto an arbitrary space. The first task I set myself in order to complete this objective wasto program Baxter to pick up and place objects at pre-set locations. Though seeminglysimple and not autonomous, the task became an essential learning experience of howBaxter follows commands.

The position of the arm is determined by an array of angles, one for each of theservo motors in that arm. The function ‘joint_angles()’ returns the array for thecurrent orientation of the motors. The ‘pickupDrop.py’ program prompts the user tomove the limb into several positions, storing the joint angles of each position. It theniterates between them picking up an object from one position and dropping at another.At first, the program only stored two positions; one for the target object and one forthe destination. The problem with this was that there are multiple solutions for themotors to move from one state of joint angles to another. This risks the limb gettinginto unwanted collisions in the environment, such as with the table. The solution was toinclude intermediate positions between the origin and destination to ensure that the pathof the limb was free of any obstacles. The final number of stored positions in this wasfour.

3.4.2 Sensors

Baxter has several methods of sensing the surrounding environment. The purpose of thenext iteration was to utilize the infra-red sensor. This would calculate the distance thelimb is from the table in order to pick up an object of predefined height.

The ‘pickupDropwithRange.py’ program lowers the limb until a threshold height ismet. At this point, the grippers close around the object and the arm picks it up. Thismovement is shown in Figure 3.14.

The output of the infra-red sensor is accessed via a ROS Service. In the Python code,the method I used was to create a Subscriber object to the service ‘/robot/range/right_-hand_range/state’ and a callback function to handle the incoming data from it. Oncethe program correctly output readings from the IR sensor, the next step was to implementa descending crane-like motion to the limb.

Infra-Red Limitations

For reasons described in Section 3.2.3, the method was not sufficient to detect the humanplayer playing a piece on top of his own. Using the IR sensor on Baxter’s arm was thefirst attempt at solving this problem.

If the background detector did not reveal a new piece being played, Baxter wouldmove his arm to the positions (set in Section 3.2.5) over where each of the opponents


Figure 3.14: Left to right, top to bottom:Example functionality of the ‘pickupDropwithRange.py’ program.

pieces were located. Distance readings from the IR sensor would be checked against thepre-programmed height of the playing pieces in order to determine if a stack of one, twoor three pieces were there. In theory, this solution seemed to be the most efficient andshowed good use of Baxter’s capabilities. However, in practice the readings from thesensor were not at all accurate.

I set up a test iteration of the program (now titled ‘broken3DtictacBaxterRange.py’)to take multiple readings from every grid space and repeat this for a number of times.When the entire grid was empty, the distances for each space were not all equal, butbetween repetitions of the test the distance values for each space did not change, regardlessof whether it was correct or not. I then put blocks in each of the spaces, so that the entireplaying grid was one level higher. Upon restarting the test, I noticed that some readingscame back correct but others returned as if there was nothing populating the space. Again,the readings between repetitions remained consistent.

I considered that the poses for Baxter’s arm above each position were not accurateenough, so I modified the test program to take several readings from different locationswithin each grid space. The results for this test were much more accurate, though therewere still a couple of spaces which gave readings as if no object was detected there. Idecided to replace the playing pieces on the board so that they were all of one colour. Atthis point in the development the available playing pieces were blocks either wrapped inblue insulation tape or painted green. Interestingly, using only the blue blocks providedmuch more accurate results over the green blocks. I put this down to the material beingmore absorbing and having a more consistent surface than the naked wood.


Lastly, I ran the test again using only the blue blocks and having stacks of them upto three levels high. The results got much less accurate with this test. The stacks ofone level in height were detected as being two or three levels high, when next to a tallerstack. I deemed this solution not viable, especially due to the possibility that IR sensorwas faulty.

Another solution was to take distance readings as the arm moved the IR sensor acrosseach row of the grid, constructing a curve of heights. This seemed like a viable optionat the time but again, due to the possibility of the IR sensor being faulty, I thoughtit more wise to spend my time finding an alternative solution; using the Hough Circlemethod to detect drawn on circles on the backs of the human player’s playing pieces (seeSection 3.2.4).

3.4.3 Inverse Kinematics

Baxter updates the pose of its limbs by an array of angles in radians. This is very hard towork with when wanting to perform simple manoeuvres that update the position of thelimb’s endpoint. It is possible to output the limb’s endpoint in Cartesian coordinates bysubscribing to another service, ‘/robot/limb/right/endpoint_state’. There is no topicor service, however, that enables updating the endpoint by manipulating its coordinates.

Inverse kinematics refers to the use of the kinematics equations of a robot to determinethe joint parameters that provide a desired position of the end-effector [13]. In otherwords, it transforms the Cartesian coordinates of the endpoint into joint angles for thelimb. The Golf Example has a function (‘baxter_ik_move()’), which I included inthe code to enable manipulation of the limb’s endpoint via Cartesian coordinates. The‘pickupDropwithRange.py’ program contained an initial test done by taking the outputof the endpoint service, decrementing the Z value of the coordinates and then updatingthe limb’s joint angles using the inverse kinematics function. This decrementing of the Zvalue continues inside a loop until the output of the infra-red sensor meets a thresholdequal to the height of the object being picked up. This is how the motion shown inFigure 3.14 occurs.

Inverse kinematics is used for all movements made by Baxter after its initial startposes are taken.

Joint Limitations

All the movements in 3D space made with inverse kinematics were done with the ‘hand’(out-most joint) locked facing downwards. The function ‘baxter_ik_move()’ takes sixparameters. The first three are the Cartesian coordinates (XYZ) of the target point inspace. The latter three are used to determine the orientation of Baxter’s gripper at thispoint. Seeing as I kept this orientation constant throughout the project, it meant thatthere were certain points Baxter could not reach with inverse kinematics, even though they


were within his range. The problem stems from the concept that a particular configurationof joint angles will give only one point in 3D space. However, a point in 3D space can beconverted into many joint angles for Baxter’s arm. If the orientation of the arm is fixed,then the number of possible joint configurations is severely reduced, sometimes to zero.

The work around to this problem, was to incrementally increase the angle of the pitchof the arm (the second of the latter three parameters which make up the orientation ofa pose) until a joint configuration is found for the target point. This worked relativelywell, objects which used to be out of Baxter’s reach were now being picked up. Whenthey were placed, there was a bit of ‘turbulence’ for the change in orientation meant thatthe object was being dropped at an angle which may cause the piece to not be positionedcorrectly. Due to time constraints I was unable to fully test this solution.

3.4.4 Head Display

Baxter’s head display has a related topic ‘/robot/xdisplay’ of which images can bepublished to. To give the user a sense of where in the code Baxter is, I set different facesto appear on Baxter’s head display. There is a default smiling face which will alwaysdisplay when the robot is between functions. This almost works as a confirmation thatsome detection feature has been completed successfully. Whenever Baxter is performingany form of vision, the camera feed is published live (frame by frame) to the head displayto give the human player a better view of how Baxter is perceiving the board. This alsogreatly assisted testing. During the human player’s turn, a ‘Your Turn’ image is displayedwith a countdown, to prompt the player to make their move. Finally, either a ‘happy’ or‘angry’ face is displayed at the end of the game depending on of Baxter won or not.

3.5 Playing Noughts & Crosses

This section concludes the development done on the project. It details how the individualcomponents work together from a code perspective. It also provides an explanation ofhow to run programs I developed.

3.5.1 How to run programs

As mentioned in Section 2.5, Baxter requires a specific software environment to be con-trolled from. Once ROS environment and Baxter SDK has been installed on a linuxmachine, the project folder needs to be added into the ‘src’ folder of the ROS workspace(e.g. ‘ros_ws/src’). Running ‘catkin_make’ inside the workspace folder will make suredependencies for all projects are installed. After this, the terminal must be connected toBaxter by running the shell script (‘./baxter.sh’, as mentioned in the Baxter SDK tu-torial). Each program can then be run as: ‘rosrun <project folder name> <program


name>.py’. All code developed for the project can be accessed from the repository, whichcan be found in Appendix D.

3.5.2 Structure

The class-based structure facilitated the management of the growing amount code acrossiterations. It allowed me to store global variables as class properties, which were usefulwhen needing to adjust values such as offsets. The final iteration of the program can beseen in ‘new3DtictacBaxter.py’. The main() function is kept outside of the class andis used to call the class and its various higher level functions. The game loop created in‘tictac3d.py’ has been used in this function, and the rest of the game logic has beenput into the class function ‘take_turn()’. Figure 3.15 shows an overview of the main()

function.I created separate classes to handle and store the state of the board. ‘BoardClass3D.py’

features two classes; one for the board (Section 3.5.4) and one for each space (Section 3.5.5)in the board. Most of the functions are ‘getters’ and ‘setters’ used in the robot class. Theseare good coding practice to utilize as it prevents any accidental changes to class properties.

3.5.3 Robot Class

The robot class contains all the functions used by Baxter to interact with the gameenvironment. The class properties contain a number of variables; object dimensions, startpositions, camera offsets and speeds. In this constructor, cameras are configured to pixeldimensions and subscribed to. A function to move the unused arm out of the way is calledand the actuators of the robot are also enabled.

The class functions are separated into sections; vision, game logic and movement.

Vision

Before the start of the game, the locate_board() function is called to calculate andstore positions of the board. This is done with a combination of Hough Circle detection(Section 3.2.4) and pixel alignment (Section 3.2.7) to centre the arm over the board.World coordinate system transformations (Section 3.2.5) are then performed to calculatethe poses for each grid space.

The detect_and_draw() function is the main computer vision aspect of the program.The function serves as a callback to the subscribed camera. The message passed to thecallback is an image message which is converted to a Cv2 (OpenCV) image using theCvBridge (an object from the ROS SDK). Functions for detecting the robot’s pieces andthe board extract features from this image using OpenCV, the workings of which aredescribed in Section 3.2.


The function which handles the detection of moves made by the human player, detect_pieces(),contains two inner functions that are based on detect_and_draw(). detect_pieces()

features a while loop that notifies the human player of their turn by countdown. At theend of this countdown the first inner function is called to check if any changes to theboard have been detected. check_circles() detects if a piece with a white circle hasbeen played by the human player, using Hough Circle detection. If no move is detected,the subtract_background() function checks a single frame against a frame taken froman earlier state in the game to detect if any other kind of move has been made, and where(Using the background subtraction method in Section 3.2.3).

Game Logic

The game logic function take_turn() is called from the game loop when Baxter is totake its turn. Section 3.3 describes how the robot decides which move to make based onthe current state of the board. This function mainly uses the class functions of the Boardclass (Section 3.5.4), such as checkWinner() which determines if a winning move can bemade. The same function is used to check for a winner between every turn, as well as afunction used for checking for ties.

Movement

After updating the board model with the desired move, the place_piece() function iscalled to instruct Baxter to show this move in the physical playing area. This functioncomprises other functions, which break down the movements as follows:

1. start_position() - returns the arm to the position between the board and thepieces.

2. toggleDisplayFeed() - sets Baxter’s head display to output feed from the armcamera.

3. find_object() - searches for playing pieces using contour detection (Section 3.2.2)and pixel alignment.

4. pickup_object() - lowers and raises the arm to and from the object with inversekinematics, closing the grippers around the object at the low point.

5. drop_on() - Places the picked up piece in a position on the board. The parametersof this function take a pose stored for the grid space and an integer to determine atwhat height the piece should be placed.

As mentioned previously, all movement is made by the robot with inverse kinematicsinside the baxter_ik_move() function. More details of this function can be found inSection 3.4.3.


3.5.4 Board Class

The functions of this class are focussed on the manipulation and observance of the boardstate during the game. The constructor of the class generates an empty cube of 3x3x3spaces and a set of possible winning lines which can be referred to when checking a win.

checkWinner()

The function takes the current state of the board and a playing piece reference as aparameter. It then returns true if one of the generated winning lines exists in the stateof the board, made up of that playing piece.

isFull()

This function iterates through all the spaces of the top level of the cube and returns trueif none of the spaces are free of content. This function is used to check for ties.

getCopy()

This function uses the ‘copy’ library from Python to create a ‘deep copy’ of the boardstate. This is so that the copy can be manipulated free from affecting the original. Theuse of this function can be seen in the Game Logic section of Section 3.5.3.

3.5.5 Space Class

All the functions of this class are ‘getters’ and ‘setters’ and store information about aparticular space on the board. The properties of this class are the contents of the space,the pose for Baxter’s arm to position above the space and the pixel coordinates of thespace for use in the background subtraction method (Section 3.2.3). The class also storeswhether a circle has been detected in that space, which is used by the Hough Circlemethod (Section 3.2.4).


Figure 3.15: Overview of Baxter’s runtime.

Chapter 4

EvaluationThis chapter evaluates the performance of the robot. The most important aspects offunctionality that required evaluation were the computer vision features. Such featureswill be covered using both qualitative and quantitative measures. I deemed aspects of therobot’s movements, such as the inverse kinematics solver and other Baxter SDK functions,unnecessary to evaluate. This was due to the fact that results from such evaluations wouldonly give indication to the robot’s performance as a unit, rather than of how well the codeI have written to run on it executes.

Light Conditions

All tests were performed with varying light conditions; minimal, artificial and naturallight.

• Minimal light featured all the lights in Baxter’s workspace turned off and the blindson the nearby windows closed.

• Artificial light had the lights turned on, and with the blinds kept closed.

• With natural light, all the lights were turned off so that the only light coming inwas through the open blinds.

4.1 Board Detection

The final product detected the board via the use of Hough Circle detection (Section 3.2.4).The circles on the board were evenly spaced so that the detector could calculate the midpoints of the grid spaces.

4.1.1 Pixel Point Accuracy of Hough Circle Detection

This first test was to determine how accurate the calculated centre point of the boardspace was. The program ‘hough_eval.py’ was created to take the calculated centre pointaverage of the board over the course of its runtime. It also provided a function to printthe pixel coordinates of a mouse click on the image feed. By measuring a central point onthe physical board with a ruler and placing a dot visible enough to click on in the imagefeed, I was able to print the exact pixel coordinates board’s centre point. This value wasused as a ground truth.

44

CHAPTER 4. EVALUATION 45

Lighting Average Pixel (x,y) Ground Truth (x,y) Difference (x,y)Minimal (467,293) (470,244) (3,-49)Artificial (467,293) (470,244) (3,-49)Natural (467,293) (470,244) (3,-49)

Table 4.1: Hough Circle pixel point accuracy test results.

Results

The results of the test (Table 4.1) shows a consistent average pixel across all light con-ditions. These averages were recorded over 1000 frames. Hough Circle detector uses agrayscale image as an input so the consistency in the calculated averages was expected.As long as the circles contrast to the background they are drawn on, brightness has littleeffect on their detection. The ground truth value was recorded once and used for everylight condition (where a new average was calculated each time). In this case, we canassume that the ground truth was slightly off centre and due to human error on my part.A repeat of the test with a better fixed ground truth returned a difference of (2,-1)

across all lighting conditions. Such a small pixel difference does not affect the overallperformance of Baxter.

4.1.2 Pixel Point Convergence

The previous test featured the detection of the board from a stationary arm. This test wasdone to evaluate the speed of which Baxter could centre his arm above the board’s centralpoint from its start position, under varying light conditions. The test was performedby running the final version of the project deliverable and using a stop watch to timefrom when the ‘Locating Board...’ image was displayed until the ‘default.png’ face wasdisplayed, which is an indication to the user that Baxter has successfully located theboard.

Results

Appendix E.1 shows the recorded times of 10 runs of the program under three separatelighting conditions. Table 4.2 shows the mean and standard deviation of the recordedtimes. All the times have been plotted onto bar charts for each lighting condition andfeature a horizontal black line, representing the mean time taken in that condition. Thisdata can be seen together in Figure 4.1 or individually in Appendix E. Several conclusionscan be drawn from the results of this experiment.

According to the mean times (represented by horizontal lines), Baxter located theboard fastest in minimal lighting conditions, however the standard deviation of the recordedtimes in this test was much greater than in other conditions. The high deviation infersthat the consistency of Baxter’s performance suffers in low lighting conditions. It is pos-


Lighting Mean Standard DeviationMinimal 13.201 34.4Artificial 24.616 7.2Natural 15.466 3.4

Table 4.2: Mean and standard deviation of the recorded times.

20

40

60

80

100

Tim

e(Secon

ds)

Minimal Artificial Natural

Figure 4.1: Experiments arranged in ascending order by time taken. Means drawn ashorizontal lines in corresponding colour.


sible to see several recorded times straying far from the mean time line. It is unclearwhether the two longest time results are either anomalies or signs of Baxter’s possibleunreliability in these conditions.

The slowest mean time was recorded in artificial light. From Figure 4.1 and Table 4.2,it is possible to see that this average is a reliable representation of Baxter’s performance.The relatively low standard deviation shows consistency between the times.

Natural lighting conditions showed the most consistent out of all the others. The chartshows the even spread around the mean time. Having performed this experiment, it wasconcluded that natural light was the most suitable condition for Baxter when locatingthe game board. This is due to consistency in the recorded times and the low mean time,which was not far off from the fastest average. However, Baxter still performed adequatelyacross all light conditions, indicating that there is no specific lighting requirement for therobot to run in.

4.2 Object Detection

Baxter is able to differentiate between the coloured pieces by using a blob detector. Theoutput of this detection has contours extracted from it, which is then used to locateindividual objects of the desired colour (Section 3.2.2).

4.2.1 Pixel Point Accuracy of Contours

A test similar to the one carried out in Section 4.1.1 was performed to evaluate the accu-racy of the contour detector. In the main program, the robot aligns itself with the centrepoint of a bounding box drawn around the extracted contour of an object. The program‘eval_cont.py’ takes an average of the calculated centre points of each object from a sta-tionary point, with the initial setup show in Figure 4.2. Like the ‘eval_hough.py’ test,a ground truth value was set for each piece’s centre point, based on the returned pixelcoordinates of mouse clicks. The aim was to find out which lighting conditions provedmost suitable for detecting playing pieces.

Results

The results of the test (Table 4.3) under natural lighting conditions were both accurateand reliable. The offsets from the ground truth values were minimal and the reliability ofthe detection can be seen in Figure 4.3 with a stable set of contours.

The results for the test under minimal and artificial lighting conditions were not re-liable enough to include in this evaluation. The number of detected contours fluctuatedbetween frames, resulting in disruption amongst the averages of the centre points, for theprogram was unable to determine which centre points were tied to each block. From thistest we are unable to make comparisons on the accuracy of the contour detector under


Figure 4.2: Contour accuracy test setup. Blocks 1 to 10 (left to right, top to bottom).

Block Number Average Pixel (x,y) Ground Truth (x,y) Difference (x,y)1 (459,205) (458,203) (1,2)2 (598,190) (602,188) (-4,2)3 (461,314) (463,315) (-2,-1)4 (594,301) (594,302) (0,-1)5 (466,426) (467,426) (-1,0)6 (596,422) (599,424) (-3,-2)7 (468,552) (471,553) (-3,-1)8 (602,534) (606,539) (-4,-5)

Table 4.3: Contour accuracy test results after 1000 frames under natural lighting condi-tions.

varying light conditions, however by comparing the images of the detected contours, itstill gives indication of its reliability. The artificial light condition image in Figure 4.4reveals that not all the blocks were detected, and those that were had imperfect contoursdrawn around them. Though the number of detected contours was relatively stable, itwas less than the number required by the test so the centre points were not recorded. Thenumber of detected contours under minimal light had the greatest instability. Figure 4.5shows a time lapse of the detected blocks and how the number of contours changed. Aswith the artificial light results, the centre points of the contours in this test were also notrecorded to to this instability.

In conclusion to this test, the order at which the contour detector had the greateststability under varying light conditions is as follows:

1. Natural Light

2. Artificial Light

3. Minimal Light


Figure 4.3: Detected contours under natural light.

Figure 4.4: Detected contours under artificial light.

Though natural light is the optimal condition, Baxter will still function correctly inall other tested conditions.

4.3 Game Play

The final iteration of the 3D Noughts & Crosses program was run on Baxter 10 times undereach lighting condition. It was at this point when the timed results of the experiment inSection 4.1.2 were recorded. The purpose of evaluating full length run-throughs was totest how well the individual components synchronised together and to fully test the gameAI.

4.3.1 Vision Classifications

One observation made during the run-throughs was how well the robot classified piecesplayed by its opponent. The aspects of the robot evaluated in Section 4.2 were present


Figure 4.5: Detected contours under minimal light.

during this but not observed. Of these tests Baxter correctly classified all objects in 100%of run-throughs across all three varying light conditions.

4.3.2 Game Logic

The test from Section 4.3.1 involved a lot of observance of the robot’s game logic. Dueto the rule based system, the outcome of each game was predictable. The more I playedthe game, the quicker I was able to beat the robot. The number of turns to win agame converged to five, leaving nine pieces on the board. The results of this evaluationcorresponds to the prior researched carried out and that it does not work as a game. Thisis assuming that a game must present some level challenge and a chance for either side towin. However in Section 2.2.3, it mentions that the game is already ‘broken’ as the firstplayer can always force a win.

The robot is able to play the game to a degree which is challenging enough for newplayers. Baxter beat all of the three other people I had test the system in their firstgame against it. In some cases, Baxter managed to win further games against them. Thehuman players eventually worked out a strategy to win (not necessarily the quickest).It is possible to change the code slightly to allow the robot to play first, decreasing thechances of the human players to win.

4.4 Code Evaluation

The program features a class-based structure (Section 3.5.2) which provides an easy tounderstand set of functions and properties. The functions after the constructor are splitinto sections; utility/setup (such as subscribers and boolean switches), game logic, track-


ing and movement. The data models for the board and individual spaces are stored inseparate classes. The code was written in a style to allow modification of the game.

Chapter 5

ConclusionThe final system meets all but one of the objectives from Section 1.2. This outstandingobjective was to implement a level of intelligence into the robot so that it develops astrategy for playing the game over time. The goal proved to be too ambitious given thescope of the project. Therefore, I spent more time ensuring the other objectives were metand the resulting functionality worked robustly.

• Baxter successfully locates and picks up playing pieces and places them into specificareas of the game board with the ability of stacking them up to three levels.

• Baxter is able to recognise moves made by the human player and store their corre-sponding locations.

• Baxter can respond to the human player’s moves strategically and by following thegame rules. The game has also been implemented into a software-only environment.

5.0.1 Time Plan

Figure 5.1 shows a revised version of the Gantt chart from Section 1.4 with the actualtimes it took to complete each objective. The overall time of the project was extendedby three weeks due to a change in the deadline. This change was at a request from meto the university regarding family related problems at the time, incurring a set back inprogress. The obvious change in the time plan is that the first objective took more timethan expected. This is because the objective had many prerequisite steps, such as locatingthe board and calculating the grid positions. The second objective was easy to implementonce a certain number of these steps were completed. Objective four (implementing gamelogic in a software environment) took a bit longer than planned because it was workedon at separate points; the first few days of the segment a 2D version of the game wasdeveloped, and the last few days the code was upgraded to the 3D version. A similaroccurrence happened with objective three; the background subtraction was implementedin the first five days of the segment, and development on the Hough Circle method startedwhen the 3D version of the game logic was implemented. The time it took to completethis objective still managed to stay true to the estimate.

5.1 Extensions

This section discuses the ways in which the project can be extended for increased func-tionality and different uses. The number of possible extensions is fairly limited by Baxter

52

CHAPTER 5. CONCLUSION 53

Figure 5.1: Actual time plan for each objective.

not being easily modifiable as a unit.

5.1.1 Depth Sensing

One way the robot can have its hardware modified is by the inclusion of a Kinect sensor(by Microsoft, originally built for the Xbox 360). The Kinect features an RGB cameraand a depth sensor [20], allowing it to perceive environments in 3D. Such a devise can beused to upgrade the tracking aspects of the project.

Attaching a Kinect to the front of the robot will negate the need for pixel align-ment (Section 3.2.7) and World Coordinate System transformations (Section 3.2.5), forthe world coordinates of the block centre points can be calculated from two differentperspectives also giving a much greater accuracy.

By attaching the sensor to the ‘cuff’ of the robot (facing down towards the board), theheights of objects played on the board can be detected with the depth sensor. Currentlythe background subtraction (Section 3.2.3) and Hough Circles (Section 3.2.4) methodsrecognise moves made by the player. Having the Kinect sensor implemented would negatethe requirement to have circles drawn on one side of each of the human player’s pieces.

5.1.2 Error Handling

For the purposes of time, I did not put much focus into handling any errors that the usermay cause in running the demo.

If the human player played a piece in between two spaces, Baxter will register it asbeing played in the first space which has the smallest distance from the piece’s centroid.If Baxter were to play a piece in the empty of those two spaces, it would collide with thepiece played by the human player for it covers an unexpected area. A fix for this wouldbe to calculate the offset of the human’s piece from the centre of space it is registered asbeing played in. Baxter could then shift all the playing positions of the other grid spacesby this offset, so as not to collide with any pieces. An alternative fix would be to haveBaxter pick up the human’s piece and place it correctly in the registered space. On theassumption that the human player will always place their pieces in the designated squares


of the board, I did not implement either of these.

5.1.3 Game Variants

There are many variants of Noughts & Crosses. ‘Qubic’ features a 4x4x4 board andpresents a much bigger challenge to both the player and someone developing an AI toplay it (Though also solvable [12], like the 3x3x3 variant). The code in this project hasbeen developed to allow modifications to work with larger game boards. Playing theregular 2D version of the game is also very possible, as it was a prior iteration before thefinal. Other variants such as ‘Connect Four’ are worth a mention but are not as easilyadapted to as official variants of Noughts & Crosses.

5.1.4 Artificial Intelligence

Due to the nature of the board game, there came a small amount of challenge when playingit over time. The focus of its behaviour was based on the 49 generated winning lines andstrategic value of particular spaces (such as the central space). The system can be mademore intelligent by a number of methods. I learnt from Newell and Simon’s Tic-Tac-Toeprogram[7], that Baxter could be a more formidable opponent if it was aware of the ‘fork’moves, which the entire strategy of Noughts & Crosses is based on. A possible way ofdoing this would be to generate an array of all the possible ‘fork’ moves, like the winninglines.

Adaptive strategy based on game state could be implemented with Minimax. Sec-tion 2.3.2 discusses Minimax and how the algorithm works out a best possible move ateach state. Implementing this into the robot’s game logic would provide even more chal-lenging game play.

It is possible to program an unbeatable strategy for the robot if it were to play first.This would not provide a fair game to the human player. Implementing an AI ‘level’(such as easy, medium and hard, where ‘hard’ utilizes the unbeatable) could simulatemore realistic gameplay, with a further add-on randomizing the starting player.

5.1.5 Audio & Voice Commands

The current method of Baxter recognising a move made by a player, is to continuallyloop through detection methods between timers until a change is detected. As this is aturn-based game, this real-time aspect may hinder a user’s strategy. A simple solution tothis is to implement a simple voice command, where the human player can say a phrase(such as ‘turn!’) to indicate to the robot that it should start its detection methods to findwhere the human has made their move.

The acknowledgement method of a move being successfully detected is currently topublish the resulting image to Baxter’s head display. Using a text-to-speech synthesiser,


this method can be replaced with the robot ‘speaking’ to indicate to the human playerthat their turn has been detected.

5.1.6 Head Display Nod

Another way of acknowledging a move made by the human player is to manipulate theposition of the head display to nod. Though equally trivial as the ‘speaking’ method ofthe previous section, it provides further exploration of Baxter’s available functions.

5.1.7 Other Games

Aside from the Noughts & Crosses variants, there are other games this project can beextended to work for. The functions of the ‘robot’ Class in the code (described in Sec-tion 3.5.3), provide movements which are universal to most board games. Along with thedetection functions, most of the code can be applied to play different games, the onlyrequirement being a change in the game logic and data models. One possible game couldbe Draughts (or Checkers).

References[1] http://www.jaqueslondon.co.uk/noughts-crosses-3d.html. [Online; last ac-

cessed 01-June-2015].

[2] http://logicgame.com.ua/game.php?id=tic_tac_toe_3d&l=en1. [Online; lastaccessed 01-June-2015].

[3] http://kylim.tistory.com/91. [Online; last accessed 01-June-2015].

[4] http : / / www . hizook . com / blog / 2012 / 09 / 18 /

baxter-robot-rethink-robotics-finally-unveiled. [Online; last accessed01-June-2015].

[5] http://onexia.com/baxter/Baxter-Electric-Parallel-Gripper.html. [Online;last accessed 01-June-2015].

[6] H. Coles-Abell and V. Pugh. Connect Four Demo. http://sdk.rethinkrobotics.com/wiki/Connect_Four_Demo. [Online; last accessed 30-May-2015].

[7] K. Crowley and R. S. Siegler. Flexible strategy use in young childrens tic-tac-toe.Cognitive Science, 17:531–561, 1993.

[8] O. Elton. 4d noughts and crosses. http://matheminutes.blogspot.co.uk/2012/

07/4d-noughts-and-crosses.html, 2012.

[9] M. Gardner. Hexaflexagons and Other Mathematical Diversions. University ofChicago Press, 1988.

[10] S. W. Golomb and A. W. Hales. Hypercube tic-tac-toe. In More games of no chance(Berkeley, CA, 2000), volume 42 of Math. Sci. Res. Inst. Publ., pages 167–182.Cambridge Univ. Press, Cambridge, 2002.

[11] R. R. Murphy. Introduction to AI Robotics. MIT Press, Cambridge, 2001.

[12] O. Patashnik. Qubic: 4 x 4 x 4 Tic-Tac-Toe. Mathematical Magazine 53, pages202–216, 1980.

[13] R. P. Paul. Robot manipulators: mathematics, programming, and control : the com-puter control of robot manipulators. MIT Press, Cambridge, 1981.

[14] V. Pugh. Worked Example Visual Servoing. http://sdk.rethinkrobotics.com/

wiki/Worked_Example_Visual_Servoing.

56

http://www.jaqueslondon.co.uk/noughts-crosses-3d.html

http://logicgame.com.ua/game.php?id=tic_tac_toe_3d&l=en1

http://kylim.tistory.com/91

http://www.hizook.com/blog/2012/09/18/baxter-robot-rethink-robotics-finally-unveiled

http://www.hizook.com/blog/2012/09/18/baxter-robot-rethink-robotics-finally-unveiled

http://onexia.com/baxter/Baxter-Electric-Parallel-Gripper.html

http://sdk.rethinkrobotics.com/wiki/Connect_Four_Demo


http://matheminutes.blogspot.co.uk/2012/07/4d-noughts-and-crosses.html

http://matheminutes.blogspot.co.uk/2012/07/4d-noughts-and-crosses.html

http://sdk.rethinkrobotics.com/wiki/Worked_Example_Visual_Servoing

http://sdk.rethinkrobotics.com/wiki/Worked_Example_Visual_Servoing

REFERENCES 57

[15] Rethink Robotics. Arms. http://sdk.rethinkrobotics.com/wiki/Arms. [Online;last accessed 01-June-2015].

[16] Rethink Robotics. Baxter Research Robot Tech Specs. http : / / www .

rethinkrobotics.com/baxter-research-robot/tech-specs/.

[17] Rethink Robotics. Connect Four Demo. http://sdk.rethinkrobotics.com/wiki/Connect_Four_Demo. [Online; last accessed 30-May-2015].

[18] ROS.org. Ros Indigo Igloo. http://wiki.ros.org/indigo/, July 2014.

[19] S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. PearsonEducation Limited, 2013.

[20] S. Totilo. Natal Recognizes 31 Body Parts, Uses TenthOf Xbox 360 — Kotaku. http : / / kotaku . com / 5442775 /

natal-recognizes-31-body-parts-uses-tenth-of-xbox-360-computing-resources,January 2010. [Online; accessed 28-May-2015].

[21] Wikipedia. Baxter (robot) — Wikipedia, The Free Encyclopedia. http://en.

wikipedia.org/wiki/Baxter_%28robot%29. [Online; accessed 21-April-2015].

[22] Wikipedia. Blob detection — Wikipedia, The Free Encyclopedia. http://en.

wikipedia.org/wiki/Blob_detection. [Online; last accessed 01-June-2015].

[23] Wikipedia. Game tree — Wikipedia, The Free Encyclopedia. http://en.

wikipedia.org/wiki/Game_tree. [Online; last accessed 01-June-2015].

[24] Wikipedia. Tic-tac-toe — Wikipedia, The Free Encyclopedia. http://en.

wikipedia.org/wiki/Tic-tac-toe. [Online; accessed 20-April-2015].

http://sdk.rethinkrobotics.com/wiki/Arms

http://www.rethinkrobotics.com/baxter-research-robot/tech-specs/

http://www.rethinkrobotics.com/baxter-research-robot/tech-specs/



http://wiki.ros.org/indigo/

http://kotaku.com/5442775/natal-recognizes-31-body-parts-uses-tenth-of-xbox-360-computing-resources

http://kotaku.com/5442775/natal-recognizes-31-body-parts-uses-tenth-of-xbox-360-computing-resources

http://en.wikipedia.org/wiki/Baxter_%28robot%29

http://en.wikipedia.org/wiki/Baxter_%28robot%29

http://en.wikipedia.org/wiki/Blob_detection

http://en.wikipedia.org/wiki/Blob_detection

http://en.wikipedia.org/wiki/Game_tree

http://en.wikipedia.org/wiki/Game_tree

http://en.wikipedia.org/wiki/Tic-tac-toe

http://en.wikipedia.org/wiki/Tic-tac-toe

Appendices

58

Appendix A

Personal ReflectionI found the entire project a very engaging experience. Up until this point, my educationfeatured timetabled lectures, which were assessed via coursework and exams. The movefrom this style to one where I could essentially choose the times I wanted to work was agreat change. Responsibility and discipline play a big part when it comes to working byyour own timetable. I found learning this to be enjoyable and even tempts me to continueresearch as a career option.

Being given access to the robot at all times proved essential, along with my ownworkstation located by it. The near-solidarity of the project was unusual. I was usedto completing coursework in time for deadlines alongside my peers in previous years.This semester I found myself rather alone in the work I had to do, as everyone elsewere off completing their own projects. That being said, I found the PhD and post-doctorate students in the area I had been allocated to work more than helpful and veryaccommodating.

The project started slowly. There were problems setting up workstations to work withBaxter both in the lab and in virtual machines on my laptop. This was time consuming,and the virtual machines were not even used. Another problem which took several dayslay with initializing the OpenCV library which resulted in a full reinstallation of theoperating system. After these initial setbacks the project gained momentum. Workingwith Baxter was very rewarding. Unlike a piece of code that works and outputs datacorrectly, Baxter shows a physical representation of successful code. The choice to usePython was something I thanked myself for. The ease of debugging and the familiarityof the syntax made the entire coding aspect comfortable. On top of this, the majorityof previous projects and online tutorials relating to Baxter use Python code in theirexamples.

A large proportion of the subject matter was new to me. I had never worked withrobots, nor done particularly well in modules involving artificial intelligence. I had noteven taken the computer vision module for the first semester of this year. Nevertheless,with the online tutorials and help from my colleagues, I gained useful, highly interestingknowledge which I was able to apply. Computer vision, in the end, turned out to be themost enjoyable aspect of the project. It shames me to think what I could of learnt if Ihad taken it as a module. Having a better knowledge of detection techniques prior tothe project may have resulted in less time being spent on vision, with more time spentattempting to complete the final objective.

There were two major setbacks over the course of the project. The first was familyrelated, and incurred in a loss of three weeks where I could have been working. Thankfully,

59

APPENDIX A. PERSONAL REFLECTION 60

the university granted me an extension. The second came when attempting to utilize theinfra-red sensor on Baxter’s arm to detect heights of stacked playing pieces. This was byfar the most frustrating point of the project and incurred several consecutive late nightsworking with the robot. On the assumption that the hardware was malfunctioning, a new,more robust solution was used. I wish I had not wasted so much time trying to makethe infra-red method work, however, I was very happy to see that there was a possiblealternative solution and that the robot was eventually able to function as desired.

Overall, the project was a great success. It’s rather entertaining to watch peopleplay against the robot and feel shamed by loosing to it on their first try. There are alsotalks of having the software used on open days for prospective students, where they candemo the robot themselves. The amount of invaluable knowledge I gained in doing theproject has started to show. When looking back over the demo projects which I readover for background reading, I now have a solid understanding of how they work whereaspreviously, I found the content confusing and rather daunting. Continuing a career inrobotics was something I never had considered until now.

My advice to future students attempting a project similar to this is to be generouswith time estimates. Robots do not have the second natures and common sense that ushumans do. A simple action that we can perform, such as picking up an orange, needs tobe broken down to many levels to enable a robot to mimic us. Think about how we see anorange, its colour, its shape. A robot needs to be programmed to know what a ‘circle’ isand what ‘orange’ is. When we pick it up, our sense of touch in our hands give us a largedescription of how much force to apply and from what angle to grasp it. Robots need tobe programmed all this information, and many of them do not have an obvious sense oftouch. Baxter was no exception, everything I had to work with came down to what wasseen through a camera, and making movements based on this. This and the fact thatworking with hardware in general provides many opportunities for failure, including theconfiguration of work environments, is why I suggest leaving a project more time thanoriginally assumed.

Appendix B

Ethical Issues AddressedThere is no way my project, the nature of which revolves around playing a game, has anyethical impact. There is an aspect of robots replacing the jobs of humans (in supermarkets,for example), but my project does not incur such problems.

61

Appendix C

VideoThis YouTube link shows a recorded demonstration of the code running on Baxter:

https://youtu.be/DMy8C9eJ4LE

62

https://youtu.be/DMy8C9eJ4LE

Appendix D

Code

D.1 Repository

The following link is a public repository containing all code developed for this project.The repository contains a readme detailing how to set up the work environment and runthe code. The rest of this appendix contains the code of the final version of the project.

https://bitbucket.org/ben2000677/fyp/

D.2 New3DtictacBaxter.py

#!/usr/bin/env python

import roslib

import sys

import inspect, os

import rospy

import string

from sensor_msgs.msg import Image

from sensor_msgs.msg import Range

from std_msgs.msg import Header

from moveit_commander import conversions

from baxter_core_msgs.srv import SolvePositionIK, SolvePositionIKRequest

import baxter_interface

from cv_bridge import CvBridge

import cv

import cv2

import numpy as np

import math

import random

from BoardClass3D import Board

# init rospy node

rospy.init_node(’pickup_and_stack’)

# file directory

63

https://bitbucket.org/ben2000677/fyp/

APPENDIX D. CODE 64

directory =

os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))

def main():

# get setup parameters

limb, distance = get_distance_data()

# read colour data from file

colours = get_object_data()

# create board class

theBoard = Board()

# create stack class instance

baxter = robot(limb, distance, colours, theBoard)

baxter.send_image(’default’)

print "limb = ", limb

print "distance = ", distance

# open the gripper

baxter.gripper.open()

# move to start position

baxter.start_position()

# send image to baxters head display

baxter.send_image(’locateboard’)

# sleep to wait for camera feedback

rospy.sleep(1)

# displays image feed from camera on head display

baxter.toggleDisplayFeed(1)

# initialises board detection using Hough Circles

baxter.locate_board()


baxter.start_position()

game = True

while game:

theBoard.printBoard() # prints game state to terminal

baxter.detect_pieces() # check for player’s move

APPENDIX D. CODE 65

theBoard.printBoard()

if theBoard.checkWinner(baxter.playerPiece): # check if human has won

baxter.send_image(’anger2’)


print(’Hooray! You have won the game!’)

break

elif theBoard.isFull(): # check if board is full



print(’The game is a tie!’)

break


baxter.take_turn() # initialises Baxter’s game logic

if theBoard.checkWinner(baxter.robotPiece):

baxter.send_image(’happy2’)


print(’The computer has beaten you! You lose.’)

game = False

elif theBoard.isFull():



print(’The game is a tie!’)

break

sys.exit()

rospy.spin()

class robot():

def __init__(self, arm, distance, colours, board):

# piece data

self.playerPiece = ’X’

self.playerPieceIndex = 1 # ( 1 or 2)

self.robotPiece = ’R’

self.robotPieceIndex = 2

# space detection

self.board = board

self.storePointPix = True

self.detectedAllCorners = False

APPENDIX D. CODE 66

self.pixelPoints = []

self.board_tolerance = 0.00375

# board xyz

spaces = self.board.getSpaces()

self.boardZ = len(spaces) # levels

self.boardY = len(spaces[0]) # rows

self.boardX = len(spaces[0][0]) # columns

# Background subtraction

self.bgsubImgmsg = None

self.imgmsg = None

# Circle piece detection

self.numCircleFrames = 1

self.spacePixelDist = 50.0

# arm ("left" or "right")

self.limb = arm

self.limb_interface = baxter_interface.Limb(self.limb)

if arm == "left":

self.other_limb = "right"

else:

self.other_limb = "left"

self.other_limb_interface = baxter_interface.Limb(self.other_limb)

# gripper ("left" or "right")

self.gripper = baxter_interface.Gripper(arm)

# object color dictionary data

self.objects = colours

# zeroed lists for pixel coordinates of objects

self.x = np.zeros(shape=(len(colours)), dtype=int)

self.y = np.zeros(shape=(len(colours)), dtype=int)

# speeds

self.normal = 0.8

self.slow = 0.1

APPENDIX D. CODE 67

# start positions

self.start_pos_x = 0.50 # x = front back

self.start_pos_y = 0.30 # y = left right

self.start_pos_z = 0.15 # z = up down

self.roll = -1.0 * math.pi # roll = horizontal

self.pitch = -0.0 * math.pi # pitch = vertical

self.yaw = 0.0 * math.pi # yaw = rotation

self.pose = [self.start_pos_x, self.start_pos_y, self.start_pos_z, \

self.roll, self.pitch, self.yaw]

# distances

self.distance = distance

self.block_height = 0.04

self.block_pickup_height = self.distance - self.block_height

self.block_grip_height = 0.102

self.block_tolerance = 0.005

# camera parameters (NB. other parameters in open_camera)

self.cam_calib = 0.0022762 # constant for distance = 0.383

self.cam_x_offset = 0.045 # original camera gripper offset

self.cam_y_offset = -0.015

self.width = 960 # Camera resolution

self.height = 600

# Enable the actuators

baxter_interface.RobotEnable().enable()

# set speed as a ratio of maximum speed

self.limb_interface.set_joint_position_speed(0.8)

self.other_limb_interface.set_joint_position_speed(0.8)

# calibrate the gripper

self.gripper.calibrate()

# set camera resolution

self.config_camera(self.limb, self.width, self.height)

# subscribe to required camera

self.subscribe_to_camera(self.limb)

APPENDIX D. CODE 68

# Display camera feed to face display

self.displayCamera = False

self.displayContours = False

self.pub = rospy.Publisher(’/robot/xdisplay’, Image, latch=True,

queue_size=1)

# move other arm out of harms way

if arm == "left":

self.baxter_ik_move("right", (0.25, -0.50, 0.2, math.pi, 0.0, 0.0))

else:

self.baxter_ik_move("left", (0.25, 0.50, 0.2, math.pi, 0.0, 0.0))

def toggleDisplayFeed(self,on,contours=False):

"""

Toggles booleans to display camera feed on head display.

"""

if contours:

self.displayContours = True

else:

self.displayContours = False

if on:

self.displayCamera = True

else:

self.displayCamera = False

def send_image(self,img_name):

"""

Send the image located at the specified path to the head

display on Baxter.

@param path: path to the image file to load and send

"""

self.toggleDisplayFeed(0) # Turn off feed to head display

global directory

path = directory+’/images/’+img_name+’.png’

img = cv2.imread(path)

self.publish_img(img,False)

APPENDIX D. CODE 69

rospy.sleep(1)

# Sleep to allow for image to be published.

def publish_img(self,img,resize=True):

"""

Publisher to head display.

"""

if resize:

img = cv2.resize(img, (1024, 600))

msg = CvBridge().cv2_to_imgmsg(img, encoding="passthrough")

self.pub.publish(msg)

# Game Logic

#--------------------------------------------------------------------------------------#

def take_turn(self):

"""

Baxter’s turn, following game logic.

"""

def random_move(movesList,level):

# Returns a valid move from the passed list on the passed board.

# Returns None if there is no valid move.

possibleMoves = []

for i in movesList:

if not self.board.getPiece(level,i[0],i[1]):

possibleMoves.append(i)

if len(possibleMoves) != 0:

return random.choice(possibleMoves)

else:

return None

# check to see if winning move is possible

for row in range(self.boardY):

for space in range(self.boardX):

copy = self.board.getCopy()

if not self.board.getPiece(self.boardZ-1,row,space): # if space

is free

copy.makeMove(row,space, self.robotPiece)

APPENDIX D. CODE 70

if copy.checkWinner(self.robotPiece):

z = self.board.makeMove(row,space, self.robotPiece)

self.place_piece(row, space, z)

return

# Check if the human player could win on his next move, and block them.


for space in range(self.boardX):

copy = self.board.getCopy()

if not self.board.getPiece(self.boardZ-1,row,space): # if space

is free

copy.makeMove(row,space, self.playerPiece)

if copy.checkWinner(self.playerPiece):

z = self.board.makeMove(row,space, self.robotPiece)

self.place_piece(row, space, z)

return

# Try to take the middle center, if it’s free and has a robot piece

below it.

if not self.board.getPiece(1,1,1) and self.board.getPiece(0,1,1) ==

self.robotPiece:

level = self.board.makeMove(1,1,self.robotPiece)

self.place_piece(1,1,level)

return

# Try to take the bottom center, if it’s free.

if not self.board.getPiece(0,1,1):



return

# Try to take one of the corners of the bottom level, if they are free.

move = random_move([(0,0),(2,0),(0,2),(2,2)], 0)

if move != None:

level = self.board.makeMove(move[0],move[1],self.robotPiece)

self.place_piece(move[0], move[1], level)

return

# Try to take the middle center, if it’s free.

if not self.board.getPiece(1,1,1):


APPENDIX D. CODE 71


return

# Try to take any corner, if it’s free.

move = random_move([(0,0),(2,0),(0,2),(2,2)], 2)

if move != None:



return

# Move on one of the sides.

move = random_move([(0,1), (1,0), (2,1), (1,2)], 2)



# Tracking Stuff

#--------------------------------------------------------------------------------------#

def get_contours(self,i,hsv):

"""

Gets contours from image.

"""

mask = cv2.inRange(hsv, np.array(self.objects[i][0:3]),

np.array(self.objects[i][3:6]))

# filter and fill the mask

kernel =

cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(self.objects[i][6],self.objects[i][6]))

mask2 = cv2.morphologyEx(mask,cv2.MORPH_OPEN,kernel)

ret,thresh = cv2.threshold(mask2,127,255,0)

contours, hierarchy =

cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)

return contours

def detect_and_draw(self,imgmsg):

"""

Main detection function, constantly loops through.

"""

APPENDIX D. CODE 72

# converts image message to cv image

img = CvBridge().imgmsg_to_cv2(imgmsg, desired_encoding=’passthrough’)

# stores img message for background subtraction

self.imgmsg = imgmsg

#-----------------------------------HOUGH_CIRCLES--------------------------------------#

# for detecting the board

# get grayscale version of image

cimg = img[:,:,0:3]

cimg = cv2.cvtColor(cimg,cv2.COLOR_BGR2GRAY)

cimg = cv2.medianBlur(cimg,5)

# extract circles from the image

circles =

cv2.HoughCircles(cimg,cv.CV_HOUGH_GRADIENT,1,20,param1=50,param2=30,minRadius=10,maxRadius=20)

# ensures all 4 corners are visible and that point storing is enabled

if circles != None and self.storePointPix:

circles = np.uint16(np.around(circles))

centers = []

for i in circles[0,:]:

centers.append([i[0],i[1],i[2]])

dis_TR = 100000

dis_TL = 100000

dis_BR = 100000

dis_BL = 100000

# calculate which corners the detected circles are (closest distance)

for i in centers:

dis = np.sqrt((i[0]-1000)**2 + (i[1])**2)

if dis < dis_TR:

dis_TR = dis

TR = i

dis = np.sqrt((i[0])**2 + (i[1])**2)

if dis < dis_TL:

dis_TL = dis

TL = i

APPENDIX D. CODE 73

dis = np.sqrt((i[0]-1000)**2 + (i[1]-1000)**2)

if dis < dis_BR:

dis_BR = dis

BR = i

dis = np.sqrt((i[0])**2 + (i[1]-1000)**2)

if dis < dis_BL:

dis_BL = dis

BL = i

TR = map(int,TR)

TL = map(int,TL)

BR = map(int,BR)

BL = map(int,BL)

# draw circles around detected circles

cv2.circle(img,(TR[0],TR[1]),TR[2],(0,255,0),2)

cv2.circle(img,(TR[0],TR[1]),2,(0,0,255),3)

cv2.circle(img,(TL[0],TL[1]),TL[2],(0,0,255),2)

cv2.circle(img,(TL[0],TL[1]),2,(0,0,255),3)

cv2.circle(img,(BR[0],BR[1]),BR[2],(255,0,0),2)

cv2.circle(img,(BR[0],BR[1]),2,(0,0,255),3)

cv2.circle(img,(BL[0],BL[1]),BL[2],(150,30,200),2)

cv2.circle(img,(BL[0],BL[1]),2,(0,0,255),3)

# distances between the grid space centres

dx = (float(TR[0])-float(TL[0]))/4

dy = (float(TR[1])-float(TL[1]))/4

dx2 = (float(BR[0])-float(BL[0]))/4

dy2 = (float(BR[1])-float(BL[1]))/4

# middle row beginning and end points

MLx = TL[0]+int(np.round((float(BL[0])-float(TL[0]))/2))

MLy = TL[1]+int(np.round((float(BL[1])-float(TL[1]))/2))

MRx = TR[0]+int(np.round((float(BR[0])-float(TR[0]))/2))

MRy = TR[1]+int(np.round((float(BR[1])-float(TR[1]))/2))

# distances between the grid space centres of the middle row

dx3 = (MRx - MLx)/4

dy3 = (MRy - MLy)/4

self.pixelPoints = range(self.boardX **2)

for i in range(0,self.boardX):

APPENDIX D. CODE 74

top =

(TL[0]+int(np.round(dx*(i+1))),TL[1]+int(np.round(dy*(i+1))))

# top row

mid = (MLx+int(np.round(dx3*(i+1))),MLy+int(np.round(dy3*(i+1))))

# middle row

bot =

(BL[0]+int(np.round(dx2*(i+1))),BL[1]+int(np.round(dy2*(i+1))))

# bottom row

self.pixelPoints[i] = top

self.pixelPoints[i+3] = mid

self.pixelPoints[i+6] = bot

cv2.circle(img,top,2,(80*(i+1),0,0),3)

cv2.circle(img,mid,2,(0,80*(i+1),0),3)

cv2.circle(img,bot,2,(0,0,80*(i+1)),3)

# if all the corners are detected

if len(centers) == 4:

self.detectedAllCorners = True

#----------------------------BLOB_DETECTION---------------------------------------#

# for detecting robot blocks

# Blur each frame

simg = cv2.GaussianBlur(img,(5,5),0)

# Convert BGR to HSV

hsv = cv2.cvtColor(simg, cv2.COLOR_BGR2HSV)

i = self.robotPieceIndex

contours = self.get_contours(i, hsv)

my_contours = []

minDist = 10000

for cnt in contours:

if cv2.contourArea(cnt)>1400: # check contours above a certain area

my_contours.append(cnt)

x,y,w,h = cv2.boundingRect(cnt) # get dimensions of the

contour

if self.displayContours:

cv2.rectangle(img,(x,y),(x+w,y+h),(0,0,255),2)

cx = x + (w/2) # centre points

cy = y + (h/2)

APPENDIX D. CODE 75

dist = self.distance_to((cx,cy),(self.width,self.height)) #

target the most-right-bottom blob

if dist < minDist:

minDist = dist

self.x[i-1] = cx

self.y[i-1] = cy

if self.displayContours:

cv2.drawContours(img,my_contours,-1,(0,255,0),3)

if self.displayCamera:

self.publish_img(img)

cv2.imshow(’RGB’,img)

k = cv2.waitKey(5) & 0xFF

#--------------------------------------------------------------------------------------#

# create subscriber to the required camera

def subscribe_to_camera(self, camera):

if camera == "left":

#callback = self.left_camera_callback

camera_str = "/cameras/left_hand_camera/image"

elif camera == "right":

#callback = self.right_camera_callback

camera_str = "/cameras/right_hand_camera/image"

else:

sys.exit("ERROR - subscribe_to_camera - Invalid camera")

image_topic = rospy.resolve_name(camera_str)

# subscribe to detection topic

rospy.Subscriber(image_topic, Image, self.detect_and_draw)

# used to place camera over target object

def block_iterate(self, iteration, block_centre):

# find displacement of target from centre of image

pixel_dx = (self.width / 2) - block_centre[0]

pixel_dy = (self.height / 2) - block_centre[1]

pixel_error = math.sqrt((pixel_dx * pixel_dx) + (pixel_dy * pixel_dy))

APPENDIX D. CODE 76

error = float(pixel_error * self.cam_calib *

self.block_pickup_height)

x_offset = - pixel_dy * self.cam_calib * self.block_pickup_height

y_offset = - pixel_dx * self.cam_calib * self.block_pickup_height

# update pose and find new block location data

self.update_pose(x_offset, y_offset)

# find displacement of target from centre of image

pixel_dx = (self.width / 2) - block_centre[0]

pixel_dy = (self.height / 2) - block_centre[1]

pixel_error = math.sqrt((pixel_dx * pixel_dx) + (pixel_dy * pixel_dy))

error = float(pixel_error * self.cam_calib *

self.block_pickup_height)

return block_centre, error

def config_camera(self, camera, x_res, y_res):

if camera == "left":

cam = baxter_interface.camera.CameraController("left_hand_camera")

elif camera == "right":

cam = baxter_interface.camera.CameraController("right_hand_camera")

else:

sys.exit("ERROR - open_camera - Invalid camera")

# set camera parameters

cam.resolution = (int(x_res), int(y_res))

cam.exposure = -1 # range, 0-100 auto = -1

cam.gain = -1 # range, 0-79 auto = -1

cam.white_balance_blue = -1 # range 0-4095, auto = -1

cam.white_balance_green = -1 # range 0-4095, auto = -1

cam.white_balance_red = -1 # range 0-4095, auto = -1

def detect_pieces(self):

"""

Detect moves made by the opponent.

"""

def subtract_background():

"""

APPENDIX D. CODE 77

Apply backround subtraction.

"""

# take old and new image

oldImg = CvBridge().imgmsg_to_cv2(self.bgsubImgmsg,

desired_encoding=’passthrough’)

img = CvBridge().imgmsg_to_cv2(self.imgmsg,


fgbg = cv2.BackgroundSubtractorMOG()

oldImg = oldImg[:,:,0:3]

fgmask = fgbg.apply(oldImg)

img = img[:,:,0:3]

fgmask = fgbg.apply(img)

# subtract images

img1_bg = cv2.bitwise_and(img,img,mask = fgmask)

# Blur image

simg = cv2.GaussianBlur(img1_bg,(5,5),0)

# Convert BGR to HSV

hsv = cv2.cvtColor(simg, cv2.COLOR_BGR2HSV)

# check for contours of the player’s piece

contours = self.get_contours(self.playerPieceIndex, hsv)

areaMax = 0.0

cx = 0

cy = 0

for cnt in contours:

if cv2.contourArea(cnt)>1400 and cv2.contourArea(cnt)>areaMax:

areaMax = cv2.contourArea(cnt)

x,y,w,h = cv2.boundingRect(cnt)

cv2.rectangle(img1_bg,(x,y),(x+w,y+h),(0,0,255),2)

cx = x + (w/2) # Centre points

cy = y + (h/2)

pos = (cx,cy)

if pos != (0,0): # if a contour was

detected

self.bgsubImgmsg = self.imgmsg # update stored image

self.publish_img(img1_bg) # send to face

APPENDIX D. CODE 78

minDist = float(’inf’)

x = 10

y = 10

# find the closest grid space to the contour


for col in range(self.boardX):

dist = self.distance_to(pos,self.board.getPixel(row,col))

if dist < minDist:

minDist = dist

x = row

y = col

# update the board data model

self.board.makeMove(x,y,self.playerPiece)

return True

else:

return False

def check_circles():

"""

Checks for pieces with circles drawn on them.

"""

centers = []

for frames in range(self.numCircleFrames): # checks a number of

frames

img = CvBridge().imgmsg_to_cv2(self.imgmsg,


# for circle pieces

cimg = img[:,:,0:3]

cimg = cv2.cvtColor(cimg,cv2.COLOR_BGR2GRAY)

cimg = cv2.medianBlur(cimg,5)

# extract circles

circles =

cv2.HoughCircles(cimg,cv.CV_HOUGH_GRADIENT,1,20,param1=50,param2=30,minRadius=31,maxRadius=40)

if circles != None:

circles = np.uint16(np.around(circles))

for i in circles[0,:]:

center = (i[0],i[1])

APPENDIX D. CODE 79

if center not in centers:

centers.append(center)

cv2.circle(img,(i[0],i[1]),i[2],(0,255,255),3)

cv2.circle(img,(i[0],i[1]),2,(0,0,255),3)

# show detected circle on head display

self.publish_img(img)

if frames > 0:

rospy.sleep(1)



# check places with a player piece in them

if self.board.getHighPiece(row,col) == self.playerPiece:

dists = []

# checks if the highest piece of the stack already has a

circle

hasCircle = self.board.getCircle(row,col)

for center in centers:

# finds closest grid space

dist =

self.distance_to(center,self.board.getPixel(row,col))

dists.append(dist)

if not dists:

if hasCircle:

# updates board data model

self.board.makeMove(row,col,self.playerPiece,False)

# update stored image

self.bgsubImgmsg = self.imgmsg

return True

else:

# check to see if a circle piece has been played

if min(dists) < self.spacePixelDist and not hasCircle:


self.board.makeMove(row,col,self.playerPiece,True)



return True

# check to see if a circle piece was removed

elif min(dists) > self.spacePixelDist and hasCircle:


self.board.makeMove(row,col,self.playerPiece,False)

APPENDIX D. CODE 80



return True

return False

# positions arm above board’s central point

self.pose = self.board.getPosition(1,1)

self.baxter_ik_move(self.limb, self.pose)

nomove = True

while nomove:

self.send_image(’yourturn’)

self.send_image(’3’) # countdown prompt

self.send_image(’2’)

self.send_image(’1’)

self.send_image(’go’)

rospy.sleep(2)

if check_circles() or subtract_background():

# then there is a change

rospy.sleep(3)

return

def distance_to(self,i,j):

a = (i[0]-j[0])

b = (i[1]-j[1])

c2 = (a*a) + (b*b)

return math.sqrt(c2)

# Movement Functions

#--------------------------------------------------------------------------------------#

def place_piece(self, x, y, level):

self.start_position()

self.toggleDisplayFeed(1,True)

self.find_object()

self.send_image(’default’)

self.pickup_object()

self.drop_on(self.board.getPosition(x,y),level+1)

def start_position(self):

self.setSpeed(self.normal)

# move back to start position

APPENDIX D. CODE 81

self.pose = [self.start_pos_x, self.start_pos_y, self.start_pos_z, \

self.roll, self.pitch, self.yaw]


def find_object(self,BoardMode=False):

# iterates to find closest pose above object and returns it

if not BoardMode:

tolerance = self.block_tolerance

i = self.robotPieceIndex-1

else:

tolerance = self.board_tolerance

# middle pixel point index of the pixelpoint array

i = int(len(self.pixelPoints)/2)

error = 2 * tolerance

iteration = 1

# iterate until arm over centre of target

while error > tolerance or not self.detectedAllCorners:

if not BoardMode:

block_centre = (self.x[i],self.y[i])

else:

if self.pixelPoints[i] == i: # if a middle pixel point has

not been generated

continue

block_centre = (self.pixelPoints[i][0],self.pixelPoints[i][1])

block_centre, error = self.block_iterate(iteration, \

block_centre)

iteration += 1

return self.pose

def pickup_object(self):

self.setSpeed(self.slow)

# save return pose

origin = self.pose

# move down to pick up object

self.pose = (self.pose[0] + self.cam_x_offset,

self.pose[1] + self.cam_y_offset,

self.pose[2] + (self.block_grip_height -

self.block_pickup_height),

APPENDIX D. CODE 82

self.pose[3],

self.pose[4],

self.pose[5])


self.gripper.close()

# return to origin

self.baxter_ik_move(self.limb, origin)


def drop_on(self,target,level):

# drop object onto target below obj pose param

self.pose = target



self.pose = (self.pose[0] + self.cam_x_offset,

self.pose[1] + self.cam_y_offset,

self.pose[2] + (self.block_grip_height - (self.distance -

(self.block_height * level))),

self.pose[3],

self.pose[4],

self.pose[5])


self.gripper.open()

self.pose = target



def locate_board(self):


centre = self.find_object(True)

# stop storing pixel points at centre position and update Board class

model

self.storePointPix = False

counter = 0

centrePix = self.pixelPoints[4]



self.board.setPixel(row,col,self.pixelPoints[counter])

self.board.setPosition(row,col,(centre[0] +

((self.pixelPoints[counter][1] - centrePix[1]) *

self.cam_calib * self.distance),

APPENDIX D. CODE 83

centre[1] +

((self.pixelPoints[counter][0] -

centrePix[0]) * self.cam_calib *

self.distance),

centre[2],

centre[3],

centre[4],

centre[5]))

counter += 1


# update pose in x and y direction

def update_pose(self,dx, dy):

x = self.pose[0] + dx

y = self.pose[1] + dy

self.pose = [x, y, self.pose[2], self.roll, self.pitch, self.yaw]


def baxter_ik_move(self, limb, rpy_pose):

"""

Performs all inverse kinematics conversions from cartesian coordinates.

Upgraded to change orientation if valid configuration is not found.

"""

quaternion_pose = conversions.list_to_pose_stamped(rpy_pose, "base")

limb_interface = baxter_interface.Limb(limb)

node = "ExternalTools/" + limb + "/PositionKinematicsNode/IKService"

ik_service = rospy.ServiceProxy(node, SolvePositionIK)

ik_request = SolvePositionIKRequest()

hdr = Header(stamp=rospy.Time.now(), frame_id="base")

ik_request.pose_stamp.append(quaternion_pose)

try:

rospy.wait_for_service(node, 5.0)

ik_response = ik_service(ik_request)

except (rospy.ServiceException, rospy.ROSException), error_message:

rospy.logerr("Service request failed: %r" % (error_message,))

sys.exit("ERROR - baxter_ik_move - Failed to append pose")

if ik_response.isValid[0]:

APPENDIX D. CODE 84

# convert response to joint position control dictionary

limb_joints = dict(zip(ik_response.joints[0].name,

ik_response.joints[0].position))

# move limb

if self.limb == limb:

self.limb_interface.move_to_joint_positions(limb_joints)

else:

self.other_limb_interface.move_to_joint_positions(limb_joints)

else:

# display invalid move message on head display

self.send_image(’errorjoint’)

self.pose = [self.pose[0], self.pose[1], self.pose[2], self.pose[3],

self.pose[4]-(0.01 * math.pi), self.pose[5]]

print ’adjusting’

self.baxter_ik_move(limb,self.pose)

self.send_image(’default’)

if self.limb == limb: # if working arm

quaternion_pose = self.limb_interface.endpoint_pose()

position = quaternion_pose[’position’]

# if working arm remember actual (x,y) position achieved

self.pose = [position[0], position[1], \

self.pose[2], self.pose[3], self.pose[4], self.pose[5]]

def setSpeed(self,speed):

self.limb_interface.set_joint_position_speed(speed)

# Setup Functions

#--------------------------------------------------------------------------------------#

def get_distance_data():

# read the setup parameters from setup.dat

file_name = directory + "/setup.dat"

try:

f = open(file_name, "r")

except ValueError:

APPENDIX D. CODE 85

sys.exit("ERROR: golf_setup must be run before golf")

#find limb

s = string.split(f.readline())

if len(s) >= 3:

if s[2] == "left" or s[2] == "right":

limb = s[2]

else:

sys.exit("ERROR: invalid limb in %s" % file_name)

else:

sys.exit("ERROR: missing limb in %s" % file_name)

#find distance to table

s = string.split(f.readline())

if len(s) >= 3:

try:

distance = float(s[2])

except ValueError:

sys.exit("ERROR: invalid distance in %s" % file_name)

else:

sys.exit("ERROR: missing distance in %s" % file_name)

return limb, distance

def get_object_data():

Objects_pointer = open(directory+’/objects.txt’, ’r’)

OBJ = {}

for line in Objects_pointer:

line = line.strip(’,\n’)

if line == ’END’:

break

fields = line.split(’,’)

fields = map(int, fields)

OBJ[fields[0]] = fields[1:len(fields)]

return OBJ

APPENDIX D. CODE 86

if __name__ == "__main__":

main()

D.3 BoardClass3D.py

#!/usr/bin/env python

import copy

class Board():

def __init__(self,x=3,y=3,z=3):

self.cube = []

self.spaces = []

# create a 3D matrix of empty spaces

for i in range(0,z):

level = []

for j in range(0,y):

row = []

for k in range(0,x):

space = Space()

row.append(space)

self.spaces.append((i,j,k))

level.append(row)

self.cube.append(level)

self.solutions = self.generateSolutions()

def generateSolutions(self):

"""

Generates winning lines for 3D Noughts & Crosses

"""

def allEqual(x,y,z):

return (x==y) and (y==z)

def isSorted(x,y,z):

return (x > y and y > z) or (z > y and y > x)

solutions = []

for a in self.spaces:

for b in self.spaces:

APPENDIX D. CODE 87

for c in self.spaces:

# check every combination of 3 distinct spaces

if a != b and b != c and a != c:

line = [a,b,c]

line.sort()

if not line in solutions:

level_eq = allEqual(a[0],b[0],c[0])

row_eq = allEqual(a[1],b[1],c[1])

col_eq = allEqual(a[2],b[2],c[2])

level_sort = isSorted(a[0],b[0],c[0])

row_sort = isSorted(a[1],b[1],c[1])

col_sort = isSorted(a[2],b[2],c[2])

# logic statements to check if line is a winning line

if ((level_eq and row_eq) or (level_eq and col_eq) or

(col_eq and row_eq)) or \

(level_sort and row_sort and col_sort) or \

((level_eq and row_sort and col_sort) or (level_sort

and row_eq and col_sort) or (level_sort and

row_sort and col_eq)):

solutions.append(line)

return solutions

def getCopy(self):

return copy.deepcopy(self)

def checkWinner(self,letter):

board = self.cube

for solution in self.solutions:

if

board[solution[0][0]][solution[0][1]][solution[0][2]].getContent()

== letter \

and


== letter \

and


== letter:

return True

def makeMove(self, row, col, letter):

"""

Updates the data model with a new piece

APPENDIX D. CODE 88

"""

for level in range(len(self.cube)):

if not self.getPiece(level,row,col):

self.setPiece(level,row,col,letter)

return level

def isFull(self):

# Return True if every space on the top of the board has been taken.

Otherwise return False.

for row in self.cube[len(self.cube)-1]:

for space in row:

if not space.getContent(): # if space is free

return False

return True

def setPosition(self, row, col, pose):

self.cube[0][row][col].setPose(pose)

def getPosition(self, row, col):

return self.cube[0][row][col].getPose()

def getSpaces(self):

return self.cube #used to be self.spaces

def setPiece(self, lvl, row, col, piece):

self.cube[lvl][row][col].setContent(piece)

def getPiece(self, lvl, row, col):

return self.cube[lvl][row][col].getContent()

def getHighPiece(self, row, col):

"""

Returns the highest piece in a grid space

"""

cube = self.getSpaces()

if self.getPiece(0,row,col) and not self.getPiece(len(cube)-1,row,col):

#make sure stack is not full or empty

for level in range(len(cube)-2,-1,-1):

piece = self.getPiece(level,row,col)

if piece:

return piece

APPENDIX D. CODE 89

else:

return 0

def setPixel(self, row, col, pixel):

self.cube[0][row][col].setPix(pixel)

def getPixel(self,row, col):

return self.cube[0][row][col].getPix()

def printBoard(self):

"""

Prints the board data model to terminal

"""

for level in range(len(self.cube)-1,-1,-1):

counter = 0

print " 0 1 2"

for row in self.cube[level]:

print " -------"

string = str(counter)

for space in row:

string += "|"

if not space.getContent():

string += " "

else:

string += space.getContent()

print string + "|"

counter += 1

print " -------"

class Space():

def __init__(self):

self.piece = 0

self.pose = (0,0,0,0,0,0)

self.pixel = (0,0)

def setPose(self,pose):

self.pose = pose

def getPose(self):

return self.pose

APPENDIX D. CODE 90

def setContent(self,piece):

self.piece = piece

def getContent(self):

return self.piece

def setPix(self,pixel):

self.pixel = pixel

def getPix(self):

return self.pixel

Appendix E

Board Detection Times Under VaryingLight Conditions

Attempt Minimal Light (secs) Artificial Light (secs) Natural Light (secs)1 18.10 18.10 10.002 13.67 39.78 21.193 13.76 28.16 13.854 13.27 32.32 16.155 29.17 24.62 18.326 09.65 27.10 12.937 81.34 13.40 20.788 13.91 20.32 14.259 95.95 19.05 13.5910 23.19 23.31 13.60

Figure E.1: Times taken to locate the game board under varying light conditions.

91

APPENDIX E. BOARD DETECTION TIMES UNDER VARYING LIGHT CONDITIONS92

20

40

60

80

100

Tim

e(Secon

ds)

Figure E.2: Recorded times in Minimal lighting conditions, arranged in ascending order.

15

20

25

30

35

40

Tim

e(Secon

ds)

Figure E.3: Recorded times in Artificial lighting conditions, arranged in ascending or-der.

10

12

14

16

18

20

22

Tim

e(Secon

ds)

Figure E.4: Recorded times in Natural lighting conditions, arranged in ascending order.

hirsh1

Documents