Documente Academic
Documente Profesional
Documente Cultură
FACULTY OF ENGINEERING
2014 - 2015
i
The candidate confirms that the following have been submitted.
Items
Report (Hard Copy)
Report (Digital)
Software Code
Format
Report
Signed forms in envelop
Repository URL
Demonstration
Video URL
The candidate confirms that the work submitted is their own and the appropriate credit
has been given where reference has been made to the work of others.
I understand that failure to attribute material which is obtained from another source
may be considered as plagiarism.
(Signature of Student)
ii
Summary
This report documents the development of a software system run on the Universitys
Baxter Research Robot (named Lucas), enabling him to participate in playing 3D Noughts
& Crosses with another human player. The aim was to create a piece of software to be
run in ROS (Robot Operating System) in order to allow Baxter to do this. The final
product had Baxter successfully playing legal games against human opponents.
iii
Acknowledgements
I would like to thank my supervisors, Professor Tony Cohn and Doctor Eris Chinellato,
for assisting me throughout the project with both the development and the writing of this
report, making sure it complies with standards. I would like to thank the other members
of the STRANDS team, Doctor Ioannis Gatsoulis, Paul Duckworth, Jawad Tayyub and
Muhannad Omari, along with Aryana Tavanai, all of whom gave me advice on solutions
and challenged me on my methods to ensure they were the most suitable. The level at
which I now understand of robotics and computer vision could certainly not have been
reached without them. I would also like to thank my flatmate, Sam Brown, for all his
help and support.
Contents
1 Introduction
1.1 Methodology . .
1.2 Objectives . . . .
1.3 Report Structure
1.4 Schedule . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2 Background Research
2.1 Board Games . . . . . . . . . . . . . . .
2.2 3D Noughts & Crosses . . . . . . . . . .
2.2.1 Physical Properties . . . . . . . .
2.2.2 Rules . . . . . . . . . . . . . . . .
2.2.3 Strategy . . . . . . . . . . . . . .
2.2.4 Computer Representation . . . .
2.3 Adversarial Search . . . . . . . . . . . .
2.3.1 Game Tree . . . . . . . . . . . . .
2.3.2 Minimax . . . . . . . . . . . . . .
2.4 Robotic Paradigms . . . . . . . . . . . .
2.4.1 Hierarchical Paradigm . . . . . .
2.4.2 Reactive Paradigm . . . . . . . .
2.5 Robot Operating System (ROS) . . . . .
2.5.1 Installation . . . . . . . . . . . .
2.5.2 Architecture . . . . . . . . . . . .
2.5.3 Topics & Services . . . . . . . . .
2.6 Baxter Research Robot . . . . . . . . . .
2.6.1 Baxter Software Development Kit
2.7 Existing Projects . . . . . . . . . . . . .
2.7.1 Worked Example Visual Servoing
2.7.2 Connect Four Demo . . . . . . .
2.7.3 Baxter Solves Rubiks Cube . . .
2.8 Software . . . . . . . . . . . . . . . . . .
2.8.1 Programming Languages . . . . .
2.8.2 Libraries . . . . . . . . . . . . . .
2.8.3 Version Control . . . . . . . . . .
iv
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
(SDK)
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
3
4
4
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
5
5
5
6
8
9
10
10
11
12
12
13
13
13
13
13
14
15
16
16
16
16
17
17
17
17
CONTENTS
3 Development & Analysis
3.1 Hardware Setup . . . . . . . . . . . . . . . . . . .
3.1.1 Environment . . . . . . . . . . . . . . . .
3.1.2 Grippers . . . . . . . . . . . . . . . . . . .
3.1.3 Custom Board Game . . . . . . . . . . . .
3.2 Vision . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Blob Detection . . . . . . . . . . . . . . .
3.2.2 Contour Detection . . . . . . . . . . . . .
3.2.3 Background Subtraction . . . . . . . . . .
3.2.4 Hough Circle Detection . . . . . . . . . . .
3.2.5 World Coordinate System Transformations
3.2.6 Camera Calibration . . . . . . . . . . . . .
3.2.7 Pixel Alignment . . . . . . . . . . . . . . .
3.3 Game Logic . . . . . . . . . . . . . . . . . . . . .
3.3.1 2D . . . . . . . . . . . . . . . . . . . . . .
3.3.2 3D . . . . . . . . . . . . . . . . . . . . . .
3.4 Robot Control . . . . . . . . . . . . . . . . . . . .
3.4.1 Basic Movement . . . . . . . . . . . . . . .
3.4.2 Sensors . . . . . . . . . . . . . . . . . . . .
3.4.3 Inverse Kinematics . . . . . . . . . . . . .
3.4.4 Head Display . . . . . . . . . . . . . . . .
3.5 Playing Noughts & Crosses . . . . . . . . . . . . .
3.5.1 How to run programs . . . . . . . . . . . .
3.5.2 Structure . . . . . . . . . . . . . . . . . .
3.5.3 Robot Class . . . . . . . . . . . . . . . . .
3.5.4 Board Class . . . . . . . . . . . . . . . . .
3.5.5 Space Class . . . . . . . . . . . . . . . . .
4 Evaluation
4.1 Board Detection . . . . . . . . . . . . . . . .
4.1.1 Pixel Point Accuracy of Hough Circle
4.1.2 Pixel Point Convergence . . . . . . .
4.2 Object Detection . . . . . . . . . . . . . . .
4.2.1 Pixel Point Accuracy of Contours . .
4.3 Game Play . . . . . . . . . . . . . . . . . . .
4.3.1 Vision Classifications . . . . . . . . .
4.3.2 Game Logic . . . . . . . . . . . . . .
4.4 Code Evaluation . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . . . . .
Detection
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18
18
18
19
19
20
21
22
24
25
27
28
29
33
33
34
35
36
36
38
39
39
39
40
40
42
42
.
.
.
.
.
.
.
.
.
44
44
44
45
47
47
49
49
50
50
CONTENTS
5 Conclusion
5.0.1 Time Plan . . . . . . . . .
5.1 Extensions . . . . . . . . . . . . .
5.1.1 Depth Sensing . . . . . . .
5.1.2 Error Handling . . . . . .
5.1.3 Game Variants . . . . . .
5.1.4 Artificial Intelligence . . .
5.1.5 Audio & Voice Commands
5.1.6 Head Display Nod . . . .
5.1.7 Other Games . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
52
52
52
53
53
54
54
54
55
55
References
56
Appendices
58
A Personal Reflection
59
61
C Video
62
D Code
63
D.1 Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
D.2 New3DtictacBaxter.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
D.3 BoardClass3D.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
E Board Detection Times Under Varying Light Conditions
91
CONTENTS
Chapter 1
Introduction
The aim of this project is to develop a software system which runs on a humanoid robot,
allowing it to play a game of 3D Noughts & Crosses with a human partner. The robot
should be able to observe and interact with the playing area.
1.1
Methodology
This project has been approached with a rapid prototyping process in mind, reducing the
objectives into small, manageable tasks. The nature of the software development works
with this approach as each iteration adds features and correct flaws from the last. In doing
this, if something were to go wrong, the entire project did not need to be restarted from
scratch or changed drastically. Developing software to enable the robot to perform simple
movements in the Robot Operating System (ROS) environment was the initial iteration,
and further, more complex manoeuvres were developed on top of this, according to the
objectives of Section 1.2.
1.2
Objectives
The following are a set of increasingly ambitious and complex objectives drawn out before
the project began:
1. The robot can pick up a playing piece and place it on an arbitrary space on a 3x3
grid. This is a good starting point to ensure that the robot can actually pick up
and place pieces in specific positions.
2. The robot is able to stack up to three playing pieces on top of one another. Due to
the nature of the game, being able to place one playing piece on top of another in
such a way that the stack will not fall over is essential.
3. The robot is able to recognise the current state of the game by detecting pieces
as they are played and remembering what position they are placed into. This is a
computer vision challenge. Using the cameras on the robot, the state of the game
can be monitored, with a trigger to the robot (visual change to the board) to let
him know that the human players turn is over.
4. The robot understands the game rules and can play the game against a human in
a software environment, using the best possible strategy. This is a software task,
implementing the game logic into the robots functionality.
3
CHAPTER 1. INTRODUCTION
5. The robot develops strategy for playing the game over time. This is an AI problem,
where a learning function will be implemented into the robots code. The best
possible strategy preprogrammed will have to be removed at this point and used as
a reference to measure the robots success.
All but objective 5 have been achieved, and the robot is able to play an entire legal
game.
1.3
Report Structure
1.4
Schedule
Figure 1.1 shows a Gantt chart of the estimated time it would have taken to complete
each of the objectives. In week 7 (9th March 2015) there was a student progress meeting
in which I prepared a small demo for my supervisor and assessor. The demo will be
referred to throughout this report when mentioning this student progress meeting.
Chapter 2
Background Research
This chapter documents the various aspects of prerequisite research before development
began; Firstly, the nature of the board game including physical properties, rules, best
strategies and any existing computer representations. Next, potential algorithmic approaches to playing the game via a set of rules, which is followed by robotic paradigms
and their relevance. Finally, research into the robot itself and how one can operate it
programmatically via ROS, including any previously done projects where the robot has
been programmed to pick up and place objects. Further research had to be done during
development in order to create alternative solutions to methods that did not work as
planned. This has also been documented in this chapter.
2.1
Board Games
The board game I have chosen for the robot to play is 3D Noughts & Crosses. I chose
this over a more complex game with a complex set of rules, like chess, as I wanted the
focus of the project to be robotics, rather than AI. Incidentally, the robotics aspect would
be relatively easy to do with board games that feature playing pieces across a 2D plane.
Selecting the three-dimensional version of Noughts & Crosses (or Tic-Tac-Toe) was done
so that more work could be put into developing the robots motor functions and vision
methods.
2.2
Just like regular Noughts & Crosses, the objective is to align 3 of your own playing pieces
in a row, whilst preventing your opponent from doing the same. The exception is that,
rather being played in a 3x3 grid, the game is played in a 3x3x3 cube, where the height
of a piece being played, is determined by previously played pieces in that same position
(similar to Connect 4). Figure 2.1 shows a retail copy of the game. Looking at the
image demonstrates that it is not possible to play a piece in a higher position without
other pieces existing below it.
2.2.1
Physical Properties
The retail version comes with 27 playing pieces. Each piece is either a 3D cross or a sphere
(nought). Every piece features a hole running down the interior length. The board is
essentially a small platform with 9 poles, evenly spaced to form a 3x3 grid (when looked
5
2.2.2
Rules
The game is for 2 players only. One player is in possession of the noughts playing pieces,
and the other player possesses the crosses. A starting player is randomly determined and
play begins. On a players turn, that player takes one of their playing pieces and places it
on one of the 9 poles. Once the piece has fallen into position, the turn ends and the next
player takes their turn. This action of placing a piece onto the board I will now refer to
as taking a position and the player using crosses as the first player.
The end condition of the game is when either all 27 pieces have filled the playing area,
or one player has taken three positions with his pieces in a straight line. This line can be
diagonal, vertical or horizontal across all axis.
There are a total of 49 winning lines in a 3x3x3 cube (see Figure 2.2). On his blog
[8], Owen Elton talks about how tic-tac-toe could be played in a 3x3x3x3 hypercube.
Calculating the winning lines on a 2D grid can be done by assigning addresses to each
space in the grid, containing the spaces row and column (such as in Figure 2.3).
address = (row, column)
It is possible to identify whether a set of three distinct addresses form a winning line
by looking at the differences in each element of the addresses. Take the set (a,a,A3),
2.2.3
Strategy
The original game of Noughts & Crosses, has a very simple strategy in which if both
players play optimally, the game will result in a draw. Allen Newell and Herbert Simon
created a Tic-Tac-Toe program in 1972 [7] which followed the following set of rules:
1. Win: If the player has two pieces in a row, they can place a third piece to get three
in a row.
2. Block: If the opponent has two pieces in a row, the player must play the third piece
themselves to block the opponent.
3. Fork: Create an opportunity where the player has two possible lines to win.
4. Option 1: The player should create a Block situation with two pieces in a row
to force the opponent into defending, as long as it doesnt result in them creating a
Fork situation.
5. Option 2: If there is a configuration where the opponent can create a Fork situation, then the player should block that fork.
6. Center: A player marks the centre.
7. Opposite corner: If the opponent plays a piece in the corner, the player plays a
piece in the opposite corner.
8. Empty corner: The player plays in a corner space.
9. Empty side: The player plays in a side space.
This rule set was very useful when developing the 2D version of the game to play. In
Section 3.3 I use a similar, but simplified version of these rules for the robot to follow.
For the opening move, there are 9 places the first player can play into. However, this
can be logically simplified to 3 places, as all the corner spaces are strategically equivalent
and all the side spaces are strategically equivalent. This is only for the first turn. Of these
three moves, the one with the most options to win for the first player is the corner [9].
This is because, out of the 8 remaining spaces for the second player, they can only avoid a
loss by taking the centre space. If the first player had taken the centre, the second player
avoids a loss by taking a corner, of which there are 4. This information was not used
however, for the implementation of the game logic in the robot always had the human
player acting first. This decision was made so that certain states of the board could be
forced for testing purposes.
The 3D variant of the game has a different strategy. When played on a 3x3x3 cube,
the game is easily winnable once a player has a piece in the central position [24]. With
the retail version of the game, this can occur when one player takes the centre position on
the lowest level of the board, but also having another piece next to it, forcing the other
player to block by taking a position on the lowest level. Figure 2.4 shows how the second
player is unable to take the central position without losing to the first player on the lowest
level. It is even possible to force a win from a much earlier point in the game by having
the first player take the central space on the lowest level, and then taking the space above
it on their next turn.
Figure 2.4: Two instances where the second player is forced to block.
The result of this is the game devolves into the winner being the player who takes the
first turn. This discovery made the incorporation of the game logic into the robot very
easy. It shows that the game is solvable, which means there is always an optimum move
to take. As the focus of the project is not to develop a challenging AI but rather have
the robot interact in physical space with accuracy, the game did not need to be changed
in light of this.
2.2.4
Computer Representation
There are many online games of 3D Noughts & Crosses. However, due to the flawed
nature of the 3x3x3 variant, the majority of them are played on a 4x4x4 board, which is
much less predictable. I also discovered that it is possible to play the game without the
gravity factor of the retail version. Unfortunately this allows the first player to take the
central position at the start of the game, further simplifying the games strategy.
10
The second image shows the second player (blue) is forced to block by taking the top
centre position, allowing the first player to take a move leading them to win the game.
Whatever move the second player makes in this state of the game, they cannot prevent
the first from winning.
Switching to the 4x4x4 variant of the game is a possible extension I discuss in Section 5.1.3. Developing the AI for this will be more challenging though, and like the 3x3x3
variant the first player can force a win [12].
2.3
Adversarial Search
There are a number of algorithms that can be used to solve the best strategy for particular
games. This section explores the algorithms which were appropriate for the implementation of the robots game logic.
Games usually fall under adversarial search problems. Noughts & Crosses is a common
example used in game theory, it can be classed as a deterministic, turn-taking, two-player,
zero-sum game of perfect information [19]. The game is deterministic because there is no
elements of chance, such as dice or decks of cards. Such games are much easier to solve
than games of chance. Perfect information refers to all elements of the game being visible
at all times, whereas imperfect information in games is featured as hidden details, such as
an opponents hand of cards in poker. Each move made by a player (with the aim to win
the game) adversely affects the opponents chances of winning. This is known as a zerosum game, and as the players are each trying to ensure they win whilst simultaneously
ensuring their opponent loses, it makes the situation adversarial.
2.3.1
Game Tree
A game tree is a directed graph which shows all the possible states of a game. A full tree
consists of the initial state of the game as the root node, and branches down to all the
possible variations of terminal states at the leaf nodes. Each level of the tree represents
all the moves which can be made by a player for each turn in the game, and this alternates
between the players every level. In the case of a 3x3 grid, the first player can make one
of nine moves from the initial state therefore nine states branch out from the root node.
The next level contains all the possible states of the game after the second player makes
a move in response to the first, and so on. Figure 2.6 shows a more streamlined game
tree for Noughts & Crosses, showing the 3 possible opening moves (mentioned in the
previous section). There are fewer than 9! = 362,880 terminal nodes [19] in the full tree
of a standard game of Noughts & Crosses. This relatively small number means that it is
entirely plausible to generate a game tree for the 3D variant of Noughts & Crosses, and
implement an algorithmic approach to the robots artificial intelligence when playing it
against a human opponent.
11
Figure 2.6: Game tree, showing states of playing in a corner, side or in the centre [23].
2.3.2
Minimax
The minimax algorithm can be used to search a game tree to calculate the best possible
solution at each given state. As mentioned in Section 2.3.1, each level of the Noughts &
Crosses tree alternates between the players. Minimax goes through these levels maximising the chances of winning for one player and minimising for the other. The algorithm can
be described by naming the players Min and Max. Max will go first and both players
will keep playing until the end of the game. This creates a game tree similar to Figure 2.6.
When the terminal states are reached, scores (known as utility values) are placed at the
leaf nodes, a high value for a terminal state where Max won and a low value for where
Min won. These utility values are calculated at all the child nodes in the tree. Figure 2.7
shows an example of how this is done. Starting at the parents of the leaf nodes, Max
will choose the highest value from each of the child nodes (In the left-most branch, Max
chooses a 3 because it is higher than the 2). The chosen value is labelled at the parent
node, and Min now must choose the lowest value from these nodes to label their parent
nodes (in the left-most branch, Min has also picked the 3 because it is lower than the 9).
Going up through the tree, the initial node is reached, which is the 3 in this example.
This path is how the current player decides which move to make. The move being chosen
will follow the path that benefits them (Max) the most whilst hindering the other player
(Min) the most.
The algorithm performs a complete depth-first exploration of the tree, using a simple
recursive computation of the minimax values for each successor state [19]. At each node
(or state) of the tree, depending on whether the depth of that node is either Maxs or
Mins turn, the algorithm will choose the highest or lowest utility value of its child nodes,
respectively. However, in order to work out the utility value of the child nodes, the minimax computation must be run on their respective child nodes. This recursion continues
to the leaf nodes (or terminal states) of the tree, and then the minimax values are passed
up through the tree where they were required, resulting in a tree like Figure 2.7. Once
12
2.4
Robotic Paradigms
A paradigm describes the approach to a problem whilst implying a possible structure for
its potential solution [11]. A robot paradigm makes use of three primitives which can be
used to categorize functions within the robot.
Sense - Handle raw information from sensing/detecting aspects of the robots environment. Examples include camera feeds and range sensors.
Plan - Take information from either sensory input or innate knowledge, then compute appropriate actions to reach a goal state.
Act - Perform actions which result in the manipulation of the environment. This
includes a change in location of the robot itself via movement.
A given paradigm describes the relationship between these three primitives [11].
2.4.1
Hierarchical Paradigm
This paradigm features a top-down approach. The robot first takes sensory data of the
surrounding environment until it has enough data to move onto the next stage. From
the sense stage, it takes the data and formulates a best possible action to perform in the
plan stage. Finally, this action is physically performed by the robots motors in the act
stage. This continually loops until a goal state has been reached. The downside of this
paradigm is that a lot of time is lost during the plan stage. In a dynamic environment
this will come across as an unresponsive robot, since the computations of the plan stage
will not take into account that aspects of the environment can change during this time.
13
However, due to the environment featured in my project remaining static for the duration
of the game, this is the paradigm that I adopted.
2.4.2
Reactive Paradigm
The reactive paradigm comprises of multiple SENSE-ACT couplings, eliminating the need
for a plan stage. This is the solution to unresponsive robots in dynamic environments
for each type of sensory input directly translates to a particular action. The downside
of this paradigm is that for complicated sensory inputs, there must be an increase in
the number of SENSE-ACT couplings for each possible environmental reaction. As the
number of these environmental factors grow (sense), so do the reaction functions (act),
and the difficulty arises in managing them.
2.5
ROS is a free, open-source framework for writing robotics software. Its flexibility comes
from a wide selection of tools, libraries and conventions. The Baxter research robot used
in this project runs on ROS, so becoming familiar with it was a prerequisite step.
2.5.1
Installation
Each distribution of ROS is targeted at running in specific releases of the Ubuntu distribution of Linux. ROS Indigo is the distribution that the robot is currently running, which is
primarily targeted at the Ubuntu 14.04 LTS release [18]. After installing VirtualBox VM
software on my machine, I installed Ubuntu 14.04 with ROS Indigo in order to familiarize
myself with the ROS framework and follow the tutorials found on the wiki.
2.5.2
Architecture
ROS works very much like a file system. Most of the functionality comes in the form
of terminal commands; roscd to change directory, rosrun to run a program. Every
project created in this file system is stored in the form of a package. The package contains
all software and data, as well as header files to import necessary dependencies. Once a
package is ready to be built, the rosmake command is entered to compile all the packages
in the file system alongside all the required dependencies.
2.5.3
A robot is usually a series of connected nodes and sensors. The nodes tend to be motors
which allow the robot to interact with the physical environment and the sensors are a
form of input, containing information about the robots surroundings. ROS is able to
14
interact with these aspects via the medium of Topics and Services. rostopic list will
list all the available topics. Topics are the buses in which the nodes in a robot exchange
messages. An example of a topic could be the rotational state of a particular joint in the
arm of a robot.
Services are subscribed to in order to utilize information which is feeding into the
robot. For example, to output the image feed from a camera on the robot, the software
subscribes to that cameras service. rosservice list lists all the available services.
2.6
15
off an arm puts the robot into Zero Force Gravity Compensation mode, allowing free
manipulation of the limb by physical guidance. The robot has 3 cameras, one in the wrist
of each arm and one built into the face display. There are also infra-red sensors on the
wrists which calculate distance. The robot comes with several interchangeable grippers for
the arms each varying in width and length. There is a selection of holes that the grippers
can be screwed into in order to adjust the overall distance between them (see Figure 2.9).
The grippers are opened and closed by a servo motor as well. I will be referring to the
Baxter Research Robot as Baxter for the rest of this report.
2.6.1
The Baxter SDK provides an interface to allow custom applications to be run on the
Baxter. The main interface of the Baxter SDK is via ROS Topics and Services, which is
why learning ROS was a necessity. The SDK comes with libraries to control Baxter from
programs. The Hello Baxter! tutorial on Rethink Robotics wiki shows how to store the
angles of the motors in Baxters arm, manipulate them, and then update the arm with
the new angles. The SDK also includes example scripts to run on Baxter, including the
important tuck_arms.py which I used at the start and end of each development session
to store the robot. There is another example which maps every motor in the robot to
the keyboard. This is a very impractical way of controlling Baxter for the movements are
not precise. However, I found the control of the grippers in this script useful in testing
different gripper sizes on varying objects.
2.7
16
Existing Projects
It was important to find out whether my project had been done before. Finding existing
projects prove useful in seeing where others have succeeded or failed in the research you
are planning to do. In searching for projects identical to my own, I discovered that this
project is entirely unique. However, projects with similar goals and solutions were found.
2.7.1
This example [14] is targeted at 1st-2nd year undergraduates on courses similar to my own.
The aim of the project is to have Baxter pick up golf balls, and place them into slots in an
egg tray. The example uses Visual Servoing for workspace relationship and calibration,
Visual Object Recognition and target selection. It also makes use of inverse kinematics.
This project helped me greatly during the development phase of my project. Section 3.4.3,
Section 3.2.5, and Section 3.2.7 detail development work which was influenced by this
example, specifics can been seen in each section. I will be referring to this project as the
Golf Example from now on.
2.7.2
The Connect Four Demo [17] features Baxter playing the original game of Connect 4,
either against a human opponent or itself. There is a requirement for a custom game piece
feeder which allows Baxter to pick up the game pieces with ease. The board detection
aspect has the user click and drag over the image coming from the camera feed, to highlight
the board. An HSV blob detector (like the one I use in Section 3.2.1) is used to identify
which colour game pieces have been played.
2.7.3
This project [6] has Baxter solving the Rubiks cube puzzle. The stages of runtime are
separated into detection, verification, solving and manipulation. Detecting the current
state of the cube is done with the use of a flatbed scanner. Baxter does not utilise any
of its detection hardware for this project. The location of the scanner is taught by using
Baxters Zero Force Gravity Compensation mode. It then scans each of the six sides
of the cube and verifies that it has scanned them correctly. The Kociemba algorithm is
then run to solve the cube and return a list of the required manipulations. Baxter then
performs these manipulations, in a manner similar to that of a human. This project did
not feature any aspects that crossed over with my objectives, therefore I did not refer to
it further during development.
2.8
2.8.1
17
Software
Programming Languages
The available programming languages for ROS are Python and C++. I selected to use
Python, due to its simplicity and the fact that the Baxter SDK examples were also in
Python. The performance advantage gained from using C++ is not necessary for the
purposes of this project.
2.8.2
Libraries
Apart from the Baxter SDK, another library I made use of was OpenCV. As the name
implies, it is an open source set of programming functions aimed at doing computer vision
in both images and video frames. All of the computer vision aspects of the project made
use of this library.
2.8.3
Version Control
A repository was set up on my Bitbucket account for all the source code (including this
report) developed during the project. I used git commands to pull and update code from
the repository when changes were made. This method allowed me to work on the project
from multiple places as well as provided a backup in case of any hard disk failures. The
link to the repository has been included in Appendix D.
Chapter 3
Development & Analysis
This chapter investigates the aims and objectives set in the first chapter, and the possible
solutions to go about achieving them.
The structure of this chapter mimics the hierarchical paradigm of Sense, Plan, Act
(from Section 2.4) used in the robots software. The first section covers the setup of
the hardware used in the project. This is then followed by the Vision (Sense), Game
Logic (Plan) and Robot Control (Act) sections. The final section summarises all the prior
sections and how the components developed in each of them fit together. This includes a
more in-depth look at the code.
Each iteration of development builds upon the previous, with additional features being
added. All the code has been stored in separate files for each iteration and are referred
to as a .py program (e.g examplescript.py). Details on how to run these programs
can be found in Section 3.5.1.
3.1
3.1.1
Hardware Setup
Environment
All development done on the robot had been in a constant environment, with sufficient
lighting.
18
19
Figure 3.1 shows the table in which Baxter is stood in front of, which was kept at the
same height throughout the project.
3.1.2
Grippers
The Baxter research robot comes with a number of grippers of varying size which can be
attached to the arm.
Size
Each pair has a minimum width, in Figure 3.2 the two pairs of grippers in the centre were
narrow enough to pick up the custom made game pieces. I settled on the right-most one
as I did not require the longer length. The shorter piece meant that more tension would
be applied to the picked up object, lessening the chances of it being dropped by accident.
Shape
The kit comes with several pairs of rubber tips which slot onto the ends of the grippers.
The ends of these tips are either rounded or square. Initially I did not take into consideration the importance of deciding which tip to use. When it came to implementing
Hough Circle detection (See Section 3.2.4) I noticed that the rounded gripper tips were
being detected as circles by the program. You can see in the screenshots that the ends
of the grippers are always in view. Figure 3.8 shows the rounded tips in the image feed.
Changing the tips was a simple matter. The difference the square tips make can be seen
in Figure 3.9.
3.1.3
The pieces of the retail version of 3D Noughts & Crosses were too small for Baxters
grippers. Therefore, part of the development process during the project was to create
a custom version of the board game, playable by Baxter. To do this, I cut a 2 metre
20
long, 62mm x 37mm plank of wood into 62mm x 62mm x 37mm blocks. These were then
coloured green (crosses) and blue (noughts). Figure 3.3 shows the playing pieces.
3.2
Vision
To achieve the first objective (Section 1.2), I had to implement the ability of enabling
Baxter to pick up an object from the table, regardless of the objects position. In order
to do this, the robot would first have to correctly identify objects. This section is divided
21
3.2.1
Blob Detection
To simplify this task, I took advantage of two programs developed by members of the
Activity Analysis group working with the robot Baxter. Both programs subscribe to the
limb camera and work on the output image feed. They are designed to work in conjunction
with one another and are included with the rest of the software developed for the project.
The first program Object_finder.py is used to isolate objects based on their colour.
It features RGB sliders for both the upper and lower ends of the colour spectrum (totalling
in 6 sliders). Adjusting the sliders alters a black and white filter on the image feed from
Baxters limb camera. The purpose of the program is to isolate each object to be tracked.
This is done by adjusting the sliders in such a way that the object appears white, with
the rest of the image feed entirely black (an example of this can be seen in the right image
of Figure 3.6). The program generates a text file containing the saved RGB values for
each object. The Object_tracker.py program uses the generated text file as an input
configuration file.
Blob detection is a mathematical method aimed at detecting regions in a digital image
that differ in properties, such as brightness or colour, compared to areas surrounding those
regions [22]. The Object_tracker.py program uses a blob detector in OpenCV to locate
the objects in the camera feed. By analysing all the pixels of every frame, a cross can
be drawn at the centre point (or centroid) of the similarly coloured groups of pixels (or
blobs), which match the RGB values of the configuration file. This tracking method
was used to differentiate between the different coloured pieces of the custom board game.
Using something like an edge detector would not have been a useful solution as all the
pieces are of similar shape. If using different shaped pieces, such as in the retail version,
using an edge detector would be a viable option. However, the aspects of physically
picking up various shapes of different pieces would add to the complexity of the project,
22
3.2.2
Contour Detection
In a later iteration of the project, a flaw arose in using solely blob detection. When
wanting to detect two pieces of the same colour, the centroid of the blob becomes the
centre of the two objects. In order to differentiate between objects, contours had to be
implemented. A contour is the outline or bounding shape of an object. In OpenCV, it is
possible to draw contours around blobs of pixels. By combining the output mask from the
blob detector with the contour detection features of OpenCV, I was able to draw crosses
23
at the centroids of the contours rather than the blobs. Figure 3.6 shows how this resulted
in being able to accurately differentiate between objects of the same colour.
Figure 3.6: Screenshots of Contour tracking (left) over blob detection output (right).
Detecting the board and working out where all the spaces are, was first attempted
with this contour/blob method. This was when a yellow piece of A3 paper was used for
the board. Using this method, working out the centre point of the paper was trivial for
it was just a very large blob. I then had to calculate the 8 remaining points to make up
the playing grid. Initially, I wanted to only use the output mask from the blob detection.
Testing showed that Baxter converged to the centroid of a blob quicker than the centroid
of a contour based on that blob. This was because the contours constantly adjust based
on the blob, so the centroid is less stable. Unfortunately, OpenCV has no functions to
calculate the dimensions of a blob, which is what was required to establish the grid spaces
on the board. Switching to contours meant I could draw a bounding rectangle around
the contour of the board, which would give me the dimensions I needed. Figure 3.7
shows a screenshot of the tracked board with calculated playing spaces. There was a
Figure 3.7: Blob and contour detection of the board with contours (green), bounding
rectangle (red) and grid spaces (purple).
problem with this implementation, however. The grid centres positions are all based on
the centroid of the contour surrounding the yellow board. If the arm was to move to one
of the outer grid points, part of the board will fall off the edge of the cameras field of
view. When this occurs, the size of the contour gets smaller, which in turn changes where
Baxter believes the centre point to be, because the calculated centroid will shift position.
24
In order to remedy this, I hard coded the dimensions of the A3 paper. After finding a
pose above the centre point of the board (using contours) at runtime, the positions in
Cartesian coordinates of the other points are calculated using these dimensions. This
method changed with the implementation of the new board design. Details of how the
positions above the other grid points are calculated with pixel-to-world transformations,
can be seen in Section 3.2.5.
Analysis
Contour detection proved to not be the most accurate way of calculating the positions
on the original board design, seen in Figure 3.7. Seeing as the board did not feature
an actual grid (only a superimposed grid which was calculated by Baxter) there was
no middle ground for where each player should be placing their pieces. This became a
problem in the 3D implementation of the game. A grid point for the human player was
not exactly the same as where Baxter was calculating it, which resulted in stacks of blocks
not being stable. This was mainly due to the board being a big yellow square where the
human had to guess the locations of the grid lines. For these reasons, the new board
design (Figure 3.4) was produced.
3.2.3
Background Subtraction
The question arose of how the robot would be able to detect and recognise moves made by
the human player. Using the position stored as the centre point above the board object, I
could compare frames from the camera feed of before and after the human player has made
a move. By taking the newer frame and subtracting the old one, the resulting image would
reveal any new changes to the board. This method is known as background subtraction.
The final implementation of this in the project was as follows; after Baxter has calculated
all the board positions from its centre point above the board, it stores a frame from the
camera feed. Then prior to each of Baxters turn in the game, the arm returns to this
centre point, stores a second frame and compares the two. Figure 3.8 shows an example
of how the software processes this information. After using OpenCV to subtract the
background, the new objects are revealed. In order to identify the correct object, I used
the blob detector with contours on the resulting image to calculate the centroid of the new
target object. The distance between this point and each of the nine grid points (shown
in Figure 3.7) are compared, and the shortest distance is used to determine which of the
nine grid spaces the human player has played a piece. This iteration can be seen in the
detectBoardPositions.py program.
Analysis
For the 3D implementation of the game, background subtraction had its limits. The
method worked for when the human player played a piece straight onto the playing grid
25
3.2.4
26
Figure 3.9: Calculating the grid spaces of the new board design, using Hough Circle
detection.
the circles centre points and the nearest grid spaces centre point to that circle were also
equal. The top and bottom rows of the grid were simple to calculate. For the top row,
the change in x and y pixel coordinates was divided by four. Then adding these dx and
dy values to the top-left circle four times and drawing dots each time, results in the top
row of grid centres being drawn. The same calculations were used for the bottom row.
The middle row had some added complexity. The point between the left pair of circles
and the point between the right pair of circles had to first be calculated. Then, using the
same calculations for the other rows, the middle set of grid centres could be drawn.
Background subtraction was not an entirely viable method for detecting moves made
by the human player, the case of a piece being played on top of a same coloured piece, in
particular. The solution to this was to implement the Hough Circle method I used for the
detection of the new board design. Figure 3.5 shows the update of painting equal sized,
white circles onto one side of all the human players game pieces. A new game rule had
to be introduced to utilize this method; when the human player makes the move to play
a piece on top of their own, they must place the new playing piece with the circle side
up. If there is already a circle on the piece being played on, the new piece must be played
with the circle side down. For all other cases such as playing onto an empty grid space or
on top of one of the robots playing pieces, the new piece must be played with the circle
side down.
If the background subtraction method fails to detect any new pieces when Baxter
checks for the human players move, another frame is stored. The Hough Circle method
is applied to this frame to extract the centre points of any circles. The function has
parameters to specify a maximum and minimum diameter for which circles are detected.
This allowed me to isolate the circles on the game pieces from the circles on the corners
of the new board design. Any circles in this new frame have the their distances to
each grid spaces centre point compared. If the distance is less than a certain threshold
27
number of pixels, then that circle is determined to be inside that grid space (similar to
the background subtraction method). The program stores which grid spaces have already
have circles detected inside them. This allows Baxter to differentiate between a circle that
was revealed in a previous turn, a new circle or whether a circle has been occluded by
another piece. The latter of these cases occurs when the human player vertically stacks
three of his pieces in a row.
These Hough Circle methods were implemented into the final iteration of the project,
New3DtictacBaxter.py.
Analysis
Using Hough Circle detection did not only provide the solution to detecting same-colour
stacked pieces, but also provided accurate piece positioning for Baxter via a more reliable
board detection. The method allowed for the board to be oriented at off-angles. As long
as the board was of landscape orientation, then Baxters poses for placing pieces would
be calculated accurately. This did not work with the contour detection, for the bounding
box used to calculate the grid spaces (via the old method) would align itself with the
camera instead of the board.
An initial advantage to using Hough Circles to detect the board, was that the playing
pieces did not obscure the corner circles, allowing the board to be detected mid-play. This
meant that if the board was knocked or shifted out of place for whatever reason, Baxter
would be able to readjust to the new position and correctly update its poses for each grid
space. Having circles painted on the backs of the playing pieces lead me to believe that
this feature would be not be possible due to disturbance of multiple circles in the play
area, affecting detection. In hindsight, I believe the implementation of this feature was
entirely possible due to the parameters on the HoughCircles OpenCV function filtering
the sizes of radii detected, preventing disturbance.
3.2.5
In order to improve the accuracy of the stored poses above each of the grid space centres,
a world coordinate system transformation was implemented.
World Coordinate System (WCS) transformations is a way of converting pixel coordinates to real world coordinates. In the project, the WCS transformation performed was
converting the pixel coordinate differences of the camera feed centre point to each of the
grid space centre points, into world coordinates. These would then be converted with
inverse kinematics into joint angles, allowing Baxter to position its limb above the correct
grid space. This equation [14] shows the conversion from the Golf Example [14]:
B = (Pp Cp ) Cc Dt + Bp + Go
28
where:
B = Baxter coordinates
Pp = target pixel coordinates
Cp = centre pixel coordinates
Bp = Baxter pose
Go = gripper offset
Cc = camera calibration factor
Dt = distance from table
When implementing this into the project, a simplified version was used:
B = (Pp Cp ) Cc Dt + Bp
This was because the code for picking up an object already took into account the gripper
offset. The camera calibration factor had to be calculated by performing a calibration test
on the camera. This test is documented in Section 3.2.6. The method was implemented
into the final iteration, New3DtictacBaxter.py. After the central grid space is aligned
(using the pixel alignment method in Section 3.2.7) with the limbs camera, this equation
is used to calculate the eight surrounding points of the grid. Originally, this was done by
calculating the offsets to the central pose using hard-coded dimensions of the game board.
3.2.6
Camera Calibration
The World Coordinate System transformations in Section 3.2.5, required a constant value
(Cc ) to correctly convert pixel distances into world distances. This constant is referred to
as the camera calibration factor and this section documents the test performed in order
to find it.
Using a method similar to generating the board design, gen_test.py is a script which
generates a symmetrical image as shown in Figure 3.10. All the shapes were drawn using
OpenCV functions. The purple cross in the centre was drawn using the images pixel
dimensions in order to get the exact positioning. The centre points of the circles are also
perfectly represented by smaller purple crosses.
Using the following equation based on the WCS transformation equation from Section 3.2.5:
( DDCt )
Cc =
Pd
where:
29
3.2.7
Pixel Alignment
There was an issue with picking up objects directly from a pose using World Coordinate
System transformations. In the 3D implementation of the game, a stack of playing pieces
30
31
32
directly above the centre of the board. This is when the poses above the other grid spaces
are calculated.
Offsets
After Baxter targets an object, it pick it up by updating the Z value of the arms endpoint
until the distance to the object has been reached. As you can see in Figure 2.9, the camera
is a fair distance away from the gripping point of the grippers. In order to ensure the
grippers enclose around the object, and offset has to be added to the pose when picking
it up.
The implementation of this offset featured a trial-and-error approach in finding the
best values. First, I physically measured the height and width offset from the centre of
the camera lens to the middle of the grippers. These values were added (or subtracted)
to any pose that went into the inverse kinematics function when Baxter lowered its arm
to pick up an object. By stopping the arm right before the grippers would close, I could
measure how far off the grippers were from the centre of the object, and increment or
decrement the offsets accordingly. The height at which the gripper picked up the object
was another offset that had to be set. During the robots setup, it works out the distance
from its arm to the table using the IR sensor. The offset to be set is the distance between
the centre of the IR sensor and the end point of the grippers. The current calculation to
pick up an object of predefined height is a change in the Z axis. The table distance, minus
the objects height, plus the offset gives this value. It was important to ensure that the
gripper did not close around the object at too low a height. This could lead in collisions
with the table and damage to Baxter. If the grippers closed around too near the top of
the object, there is a chance that on occasion the grippers could fail to pick up the object
entirely.
These offsets are also applied when Baxter places pieces onto the game board.
Stacking
As an add-on to the project demo, I implemented a stacking feature and a class-based
structure for the current code. The new structure is described in more detail in Section 3.5.2 and is viewable in the program pickupStack.py. The stacking feature remembers the pose for the location above the first object, locates and picks up the second
object, placing it on the first. The height to release the second object from is calculated
using the vertical offsets. By doubling the stored height of the object and using that
value in the pose for placing down the object, the new height is equal to the height of two
blocks. Due to the new class based structure, this implementation was trivial. It can be
easily modified to stack a higher number of objects (tripling the stored object height to
stack of three blocks, for example).
This add-on was used as the prime functionality of playing the board game. Baxters
33
turn to play a piece into a space on the board, uses the stacking feature to stack the
object onto the space. The level at which the object was stacked on was dependent on
how many objects were currently in that space (if there were zero, then the object was
placed directly onto the board).
3.3
3.3.1
Game Logic
2D
Baxter needed to know how to play the game. According to the objectives, this would
first have to be achieved in a software environment before applying Baxters physical
capabilities.
The first step was to essentially create a simple tic-tac-toe game in Python. The
tictac.py program is a game of the 2D version of Noughts & Crosses, playable in
the Terminal window. Implementing the 2D version of the game is a prerequisite step
before the full 3D version. I set up the code so that it could be easily adapted to higher
dimensions.
The program features play between a human and the computer. On the turn of the
computer, the AI has been set up to follow this sequence of possible moves in order:
1. A winning move
2. A move to stop the opponent winning
3. A free random corner space, if there is one
4. The centre space, if its free
5. A free random side space
The method to check whether a winning move can be made or whether the player
needs to be blocked to prevent them from winning, is done by making a copy of the board
(function of the board class in Section 3.5.4) for every free space and filling each space
with either playing piece. Then each copy of the board is checked against the possible
winning lines (checkWinner() function, also from the board class) to see if that potential
move will result in a win for either side.
This is not an unbeatable AI, it is possible to win by taking two opposite corner spaces.
Now that the AI will try to block the player, a third corner can be taken on the players
next turn, putting them in a position of winning the game. This strategy is stoppable by
taking an opposing side space, as shown in Figure 3.13 and mentioned in Martin Gardners
book [9]. For the sake of testing, I have not programmed in this defence. This is so that
when Baxter comes to playing the game, I now have strategies to force a Win, Loss and
Tie.
34
Figure 3.13: The second player taking the opposing side space of the taken corner.
3.3.2
3D
The 3D implementation of the game took the games current data model, which was a 2D
matrix represented by an array of arrays, and stretched it into another dimension. The
3D matrix of the new implementation featured three of the 2D matrices from the previous
iteration, one for each level of the cube.
Along with a new set of winning lines, which Section 3.5.4 describes the method of
generation, the implementation required a new set of conditions for the AI to play the
game. I kept the code for making a move into the grid on a 2D interface the same; the
(row, col) format of where the move was made now places a piece in the lowest possible
level of that position. This meant the rules followed in Section 3.3.1 did not need to be
altered much. The new sequence of possible moves the AI follows are:
1. A winning move (unchanged)
2. A move to stop the opponent winning (unchanged)
3. The centre space on the middle level, if there is a robot piece below it
4. The centre space on the bottom level, if its free (unchanged)
5. A free random corner space on the bottom level, if there is one
6. The centre space on the middle level, if its free
7. A free random corner space, if there is one (unchanged)
8. A free random side space (unchanged)
This is by no means a perfect AI, but still presents a decent enough challenge whilst
allowing me to force certain board states for testing purposes.
35
3.4
Robot Control
The first successes of controlling Baxter was via the SDK example programs and the
Hello Baxter! online tutorial. The first program I developed was based on this tutorial
and was an important stepping stone in understanding how Baxter sees the environment.
3.4.1
36
Basic Movement
The first objective in Section 1.2 required Baxter to pick up and place a playing piece
into an arbitrary space. The first task I set myself in order to complete this objective was
to program Baxter to pick up and place objects at pre-set locations. Though seemingly
simple and not autonomous, the task became an essential learning experience of how
Baxter follows commands.
The position of the arm is determined by an array of angles, one for each of the
servo motors in that arm. The function joint_angles() returns the array for the
current orientation of the motors. The pickupDrop.py program prompts the user to
move the limb into several positions, storing the joint angles of each position. It then
iterates between them picking up an object from one position and dropping at another.
At first, the program only stored two positions; one for the target object and one for
the destination. The problem with this was that there are multiple solutions for the
motors to move from one state of joint angles to another. This risks the limb getting
into unwanted collisions in the environment, such as with the table. The solution was to
include intermediate positions between the origin and destination to ensure that the path
of the limb was free of any obstacles. The final number of stored positions in this was
four.
3.4.2
Sensors
Baxter has several methods of sensing the surrounding environment. The purpose of the
next iteration was to utilize the infra-red sensor. This would calculate the distance the
limb is from the table in order to pick up an object of predefined height.
The pickupDropwithRange.py program lowers the limb until a threshold height is
met. At this point, the grippers close around the object and the arm picks it up. This
movement is shown in Figure 3.14.
The output of the infra-red sensor is accessed via a ROS Service. In the Python code,
the method I used was to create a Subscriber object to the service /robot/range/right_hand_range/state and a callback function to handle the incoming data from it. Once
the program correctly output readings from the IR sensor, the next step was to implement
a descending crane-like motion to the limb.
Infra-Red Limitations
For reasons described in Section 3.2.3, the method was not sufficient to detect the human
player playing a piece on top of his own. Using the IR sensor on Baxters arm was the
first attempt at solving this problem.
If the background detector did not reveal a new piece being played, Baxter would
move his arm to the positions (set in Section 3.2.5) over where each of the opponents
37
38
Lastly, I ran the test again using only the blue blocks and having stacks of them up
to three levels high. The results got much less accurate with this test. The stacks of
one level in height were detected as being two or three levels high, when next to a taller
stack. I deemed this solution not viable, especially due to the possibility that IR sensor
was faulty.
Another solution was to take distance readings as the arm moved the IR sensor across
each row of the grid, constructing a curve of heights. This seemed like a viable option
at the time but again, due to the possibility of the IR sensor being faulty, I thought
it more wise to spend my time finding an alternative solution; using the Hough Circle
method to detect drawn on circles on the backs of the human players playing pieces (see
Section 3.2.4).
3.4.3
Inverse Kinematics
Baxter updates the pose of its limbs by an array of angles in radians. This is very hard to
work with when wanting to perform simple manoeuvres that update the position of the
limbs endpoint. It is possible to output the limbs endpoint in Cartesian coordinates by
subscribing to another service, /robot/limb/right/endpoint_state. There is no topic
or service, however, that enables updating the endpoint by manipulating its coordinates.
Inverse kinematics refers to the use of the kinematics equations of a robot to determine
the joint parameters that provide a desired position of the end-effector [13]. In other
words, it transforms the Cartesian coordinates of the endpoint into joint angles for the
limb. The Golf Example has a function (baxter_ik_move()), which I included in
the code to enable manipulation of the limbs endpoint via Cartesian coordinates. The
pickupDropwithRange.py program contained an initial test done by taking the output
of the endpoint service, decrementing the Z value of the coordinates and then updating
the limbs joint angles using the inverse kinematics function. This decrementing of the Z
value continues inside a loop until the output of the infra-red sensor meets a threshold
equal to the height of the object being picked up. This is how the motion shown in
Figure 3.14 occurs.
Inverse kinematics is used for all movements made by Baxter after its initial start
poses are taken.
Joint Limitations
All the movements in 3D space made with inverse kinematics were done with the hand
(out-most joint) locked facing downwards. The function baxter_ik_move() takes six
parameters. The first three are the Cartesian coordinates (XYZ) of the target point in
space. The latter three are used to determine the orientation of Baxters gripper at this
point. Seeing as I kept this orientation constant throughout the project, it meant that
there were certain points Baxter could not reach with inverse kinematics, even though they
39
were within his range. The problem stems from the concept that a particular configuration
of joint angles will give only one point in 3D space. However, a point in 3D space can be
converted into many joint angles for Baxters arm. If the orientation of the arm is fixed,
then the number of possible joint configurations is severely reduced, sometimes to zero.
The work around to this problem, was to incrementally increase the angle of the pitch
of the arm (the second of the latter three parameters which make up the orientation of
a pose) until a joint configuration is found for the target point. This worked relatively
well, objects which used to be out of Baxters reach were now being picked up. When
they were placed, there was a bit of turbulence for the change in orientation meant that
the object was being dropped at an angle which may cause the piece to not be positioned
correctly. Due to time constraints I was unable to fully test this solution.
3.4.4
Head Display
Baxters head display has a related topic /robot/xdisplay of which images can be
published to. To give the user a sense of where in the code Baxter is, I set different faces
to appear on Baxters head display. There is a default smiling face which will always
display when the robot is between functions. This almost works as a confirmation that
some detection feature has been completed successfully. Whenever Baxter is performing
any form of vision, the camera feed is published live (frame by frame) to the head display
to give the human player a better view of how Baxter is perceiving the board. This also
greatly assisted testing. During the human players turn, a Your Turn image is displayed
with a countdown, to prompt the player to make their move. Finally, either a happy or
angry face is displayed at the end of the game depending on of Baxter won or not.
3.5
This section concludes the development done on the project. It details how the individual
components work together from a code perspective. It also provides an explanation of
how to run programs I developed.
3.5.1
As mentioned in Section 2.5, Baxter requires a specific software environment to be controlled from. Once ROS environment and Baxter SDK has been installed on a linux
machine, the project folder needs to be added into the src folder of the ROS workspace
(e.g. ros_ws/src). Running catkin_make inside the workspace folder will make sure
dependencies for all projects are installed. After this, the terminal must be connected to
Baxter by running the shell script (./baxter.sh, as mentioned in the Baxter SDK tutorial). Each program can then be run as: rosrun <project folder name> <program
40
name>.py. All code developed for the project can be accessed from the repository, which
can be found in Appendix D.
3.5.2
Structure
The class-based structure facilitated the management of the growing amount code across
iterations. It allowed me to store global variables as class properties, which were useful
when needing to adjust values such as offsets. The final iteration of the program can be
seen in new3DtictacBaxter.py. The main() function is kept outside of the class and
is used to call the class and its various higher level functions. The game loop created in
tictac3d.py has been used in this function, and the rest of the game logic has been
put into the class function take_turn(). Figure 3.15 shows an overview of the main()
function.
I created separate classes to handle and store the state of the board. BoardClass3D.py
features two classes; one for the board (Section 3.5.4) and one for each space (Section 3.5.5)
in the board. Most of the functions are getters and setters used in the robot class. These
are good coding practice to utilize as it prevents any accidental changes to class properties.
3.5.3
Robot Class
The robot class contains all the functions used by Baxter to interact with the game
environment. The class properties contain a number of variables; object dimensions, start
positions, camera offsets and speeds. In this constructor, cameras are configured to pixel
dimensions and subscribed to. A function to move the unused arm out of the way is called
and the actuators of the robot are also enabled.
The class functions are separated into sections; vision, game logic and movement.
Vision
Before the start of the game, the locate_board() function is called to calculate and
store positions of the board. This is done with a combination of Hough Circle detection
(Section 3.2.4) and pixel alignment (Section 3.2.7) to centre the arm over the board.
World coordinate system transformations (Section 3.2.5) are then performed to calculate
the poses for each grid space.
The detect_and_draw() function is the main computer vision aspect of the program.
The function serves as a callback to the subscribed camera. The message passed to the
callback is an image message which is converted to a Cv2 (OpenCV) image using the
CvBridge (an object from the ROS SDK). Functions for detecting the robots pieces and
the board extract features from this image using OpenCV, the workings of which are
described in Section 3.2.
41
The function which handles the detection of moves made by the human player, detect_pieces(),
contains two inner functions that are based on detect_and_draw(). detect_pieces()
features a while loop that notifies the human player of their turn by countdown. At the
end of this countdown the first inner function is called to check if any changes to the
board have been detected. check_circles() detects if a piece with a white circle has
been played by the human player, using Hough Circle detection. If no move is detected,
the subtract_background() function checks a single frame against a frame taken from
an earlier state in the game to detect if any other kind of move has been made, and where
(Using the background subtraction method in Section 3.2.3).
Game Logic
The game logic function take_turn() is called from the game loop when Baxter is to
take its turn. Section 3.3 describes how the robot decides which move to make based on
the current state of the board. This function mainly uses the class functions of the Board
class (Section 3.5.4), such as checkWinner() which determines if a winning move can be
made. The same function is used to check for a winner between every turn, as well as a
function used for checking for ties.
Movement
After updating the board model with the desired move, the place_piece() function is
called to instruct Baxter to show this move in the physical playing area. This function
comprises other functions, which break down the movements as follows:
1. start_position() - returns the arm to the position between the board and the
pieces.
2. toggleDisplayFeed() - sets Baxters head display to output feed from the arm
camera.
3. find_object() - searches for playing pieces using contour detection (Section 3.2.2)
and pixel alignment.
4. pickup_object() - lowers and raises the arm to and from the object with inverse
kinematics, closing the grippers around the object at the low point.
5. drop_on() - Places the picked up piece in a position on the board. The parameters
of this function take a pose stored for the grid space and an integer to determine at
what height the piece should be placed.
As mentioned previously, all movement is made by the robot with inverse kinematics
inside the baxter_ik_move() function. More details of this function can be found in
Section 3.4.3.
3.5.4
42
Board Class
The functions of this class are focussed on the manipulation and observance of the board
state during the game. The constructor of the class generates an empty cube of 3x3x3
spaces and a set of possible winning lines which can be referred to when checking a win.
checkWinner()
The function takes the current state of the board and a playing piece reference as a
parameter. It then returns true if one of the generated winning lines exists in the state
of the board, made up of that playing piece.
isFull()
This function iterates through all the spaces of the top level of the cube and returns true
if none of the spaces are free of content. This function is used to check for ties.
getCopy()
This function uses the copy library from Python to create a deep copy of the board
state. This is so that the copy can be manipulated free from affecting the original. The
use of this function can be seen in the Game Logic section of Section 3.5.3.
3.5.5
Space Class
All the functions of this class are getters and setters and store information about a
particular space on the board. The properties of this class are the contents of the space,
the pose for Baxters arm to position above the space and the pixel coordinates of the
space for use in the background subtraction method (Section 3.2.3). The class also stores
whether a circle has been detected in that space, which is used by the Hough Circle
method (Section 3.2.4).
43
Chapter 4
Evaluation
This chapter evaluates the performance of the robot. The most important aspects of
functionality that required evaluation were the computer vision features. Such features
will be covered using both qualitative and quantitative measures. I deemed aspects of the
robots movements, such as the inverse kinematics solver and other Baxter SDK functions,
unnecessary to evaluate. This was due to the fact that results from such evaluations would
only give indication to the robots performance as a unit, rather than of how well the code
I have written to run on it executes.
Light Conditions
All tests were performed with varying light conditions; minimal, artificial and natural
light.
Minimal light featured all the lights in Baxters workspace turned off and the blinds
on the nearby windows closed.
Artificial light had the lights turned on, and with the blinds kept closed.
With natural light, all the lights were turned off so that the only light coming in
was through the open blinds.
4.1
Board Detection
The final product detected the board via the use of Hough Circle detection (Section 3.2.4).
The circles on the board were evenly spaced so that the detector could calculate the mid
points of the grid spaces.
4.1.1
This first test was to determine how accurate the calculated centre point of the board
space was. The program hough_eval.py was created to take the calculated centre point
average of the board over the course of its runtime. It also provided a function to print
the pixel coordinates of a mouse click on the image feed. By measuring a central point on
the physical board with a ruler and placing a dot visible enough to click on in the image
feed, I was able to print the exact pixel coordinates boards centre point. This value was
used as a ground truth.
44
CHAPTER 4. EVALUATION
Lighting
Minimal
Artificial
Natural
45
Results
The results of the test (Table 4.1) shows a consistent average pixel across all light conditions. These averages were recorded over 1000 frames. Hough Circle detector uses a
grayscale image as an input so the consistency in the calculated averages was expected.
As long as the circles contrast to the background they are drawn on, brightness has little
effect on their detection. The ground truth value was recorded once and used for every
light condition (where a new average was calculated each time). In this case, we can
assume that the ground truth was slightly off centre and due to human error on my part.
A repeat of the test with a better fixed ground truth returned a difference of (2,-1)
across all lighting conditions. Such a small pixel difference does not affect the overall
performance of Baxter.
4.1.2
The previous test featured the detection of the board from a stationary arm. This test was
done to evaluate the speed of which Baxter could centre his arm above the boards central
point from its start position, under varying light conditions. The test was performed
by running the final version of the project deliverable and using a stop watch to time
from when the Locating Board... image was displayed until the default.png face was
displayed, which is an indication to the user that Baxter has successfully located the
board.
Results
Appendix E.1 shows the recorded times of 10 runs of the program under three separate
lighting conditions. Table 4.2 shows the mean and standard deviation of the recorded
times. All the times have been plotted onto bar charts for each lighting condition and
feature a horizontal black line, representing the mean time taken in that condition. This
data can be seen together in Figure 4.1 or individually in Appendix E. Several conclusions
can be drawn from the results of this experiment.
According to the mean times (represented by horizontal lines), Baxter located the
board fastest in minimal lighting conditions, however the standard deviation of the recorded
times in this test was much greater than in other conditions. The high deviation infers
that the consistency of Baxters performance suffers in low lighting conditions. It is pos-
46
CHAPTER 4. EVALUATION
Lighting
Minimal
Artificial
Natural
100
Time (Seconds)
80
60
40
20
Minimal
Artificial
Natural
Figure 4.1: Experiments arranged in ascending order by time taken. Means drawn as
horizontal lines in corresponding colour.
CHAPTER 4. EVALUATION
47
sible to see several recorded times straying far from the mean time line. It is unclear
whether the two longest time results are either anomalies or signs of Baxters possible
unreliability in these conditions.
The slowest mean time was recorded in artificial light. From Figure 4.1 and Table 4.2,
it is possible to see that this average is a reliable representation of Baxters performance.
The relatively low standard deviation shows consistency between the times.
Natural lighting conditions showed the most consistent out of all the others. The chart
shows the even spread around the mean time. Having performed this experiment, it was
concluded that natural light was the most suitable condition for Baxter when locating
the game board. This is due to consistency in the recorded times and the low mean time,
which was not far off from the fastest average. However, Baxter still performed adequately
across all light conditions, indicating that there is no specific lighting requirement for the
robot to run in.
4.2
Object Detection
Baxter is able to differentiate between the coloured pieces by using a blob detector. The
output of this detection has contours extracted from it, which is then used to locate
individual objects of the desired colour (Section 3.2.2).
4.2.1
A test similar to the one carried out in Section 4.1.1 was performed to evaluate the accuracy of the contour detector. In the main program, the robot aligns itself with the centre
point of a bounding box drawn around the extracted contour of an object. The program
eval_cont.py takes an average of the calculated centre points of each object from a stationary point, with the initial setup show in Figure 4.2. Like the eval_hough.py test,
a ground truth value was set for each pieces centre point, based on the returned pixel
coordinates of mouse clicks. The aim was to find out which lighting conditions proved
most suitable for detecting playing pieces.
Results
The results of the test (Table 4.3) under natural lighting conditions were both accurate
and reliable. The offsets from the ground truth values were minimal and the reliability of
the detection can be seen in Figure 4.3 with a stable set of contours.
The results for the test under minimal and artificial lighting conditions were not reliable enough to include in this evaluation. The number of detected contours fluctuated
between frames, resulting in disruption amongst the averages of the centre points, for the
program was unable to determine which centre points were tied to each block. From this
test we are unable to make comparisons on the accuracy of the contour detector under
CHAPTER 4. EVALUATION
48
Figure 4.2: Contour accuracy test setup. Blocks 1 to 10 (left to right, top to bottom).
Block Number Average Pixel (x,y) Ground Truth (x,y) Difference (x,y)
1
(459,205)
(458,203)
(1,2)
2
(598,190)
(602,188)
(-4,2)
3
(461,314)
(463,315)
(-2,-1)
4
(594,301)
(594,302)
(0,-1)
5
(466,426)
(467,426)
(-1,0)
6
(596,422)
(599,424)
(-3,-2)
7
(468,552)
(471,553)
(-3,-1)
8
(602,534)
(606,539)
(-4,-5)
Table 4.3: Contour accuracy test results after 1000 frames under natural lighting conditions.
varying light conditions, however by comparing the images of the detected contours, it
still gives indication of its reliability. The artificial light condition image in Figure 4.4
reveals that not all the blocks were detected, and those that were had imperfect contours
drawn around them. Though the number of detected contours was relatively stable, it
was less than the number required by the test so the centre points were not recorded. The
number of detected contours under minimal light had the greatest instability. Figure 4.5
shows a time lapse of the detected blocks and how the number of contours changed. As
with the artificial light results, the centre points of the contours in this test were also not
recorded to to this instability.
In conclusion to this test, the order at which the contour detector had the greatest
stability under varying light conditions is as follows:
1. Natural Light
2. Artificial Light
3. Minimal Light
CHAPTER 4. EVALUATION
49
4.3
Game Play
The final iteration of the 3D Noughts & Crosses program was run on Baxter 10 times under
each lighting condition. It was at this point when the timed results of the experiment in
Section 4.1.2 were recorded. The purpose of evaluating full length run-throughs was to
test how well the individual components synchronised together and to fully test the game
AI.
4.3.1
Vision Classifications
One observation made during the run-throughs was how well the robot classified pieces
played by its opponent. The aspects of the robot evaluated in Section 4.2 were present
CHAPTER 4. EVALUATION
50
4.3.2
Game Logic
The test from Section 4.3.1 involved a lot of observance of the robots game logic. Due
to the rule based system, the outcome of each game was predictable. The more I played
the game, the quicker I was able to beat the robot. The number of turns to win a
game converged to five, leaving nine pieces on the board. The results of this evaluation
corresponds to the prior researched carried out and that it does not work as a game. This
is assuming that a game must present some level challenge and a chance for either side to
win. However in Section 2.2.3, it mentions that the game is already broken as the first
player can always force a win.
The robot is able to play the game to a degree which is challenging enough for new
players. Baxter beat all of the three other people I had test the system in their first
game against it. In some cases, Baxter managed to win further games against them. The
human players eventually worked out a strategy to win (not necessarily the quickest).
It is possible to change the code slightly to allow the robot to play first, decreasing the
chances of the human players to win.
4.4
Code Evaluation
The program features a class-based structure (Section 3.5.2) which provides an easy to
understand set of functions and properties. The functions after the constructor are split
into sections; utility/setup (such as subscribers and boolean switches), game logic, track-
CHAPTER 4. EVALUATION
51
ing and movement. The data models for the board and individual spaces are stored in
separate classes. The code was written in a style to allow modification of the game.
Chapter 5
Conclusion
The final system meets all but one of the objectives from Section 1.2. This outstanding
objective was to implement a level of intelligence into the robot so that it develops a
strategy for playing the game over time. The goal proved to be too ambitious given the
scope of the project. Therefore, I spent more time ensuring the other objectives were met
and the resulting functionality worked robustly.
Baxter successfully locates and picks up playing pieces and places them into specific
areas of the game board with the ability of stacking them up to three levels.
Baxter is able to recognise moves made by the human player and store their corresponding locations.
Baxter can respond to the human players moves strategically and by following the
game rules. The game has also been implemented into a software-only environment.
5.0.1
Time Plan
Figure 5.1 shows a revised version of the Gantt chart from Section 1.4 with the actual
times it took to complete each objective. The overall time of the project was extended
by three weeks due to a change in the deadline. This change was at a request from me
to the university regarding family related problems at the time, incurring a set back in
progress. The obvious change in the time plan is that the first objective took more time
than expected. This is because the objective had many prerequisite steps, such as locating
the board and calculating the grid positions. The second objective was easy to implement
once a certain number of these steps were completed. Objective four (implementing game
logic in a software environment) took a bit longer than planned because it was worked
on at separate points; the first few days of the segment a 2D version of the game was
developed, and the last few days the code was upgraded to the 3D version. A similar
occurrence happened with objective three; the background subtraction was implemented
in the first five days of the segment, and development on the Hough Circle method started
when the 3D version of the game logic was implemented. The time it took to complete
this objective still managed to stay true to the estimate.
5.1
Extensions
This section discuses the ways in which the project can be extended for increased functionality and different uses. The number of possible extensions is fairly limited by Baxter
52
CHAPTER 5. CONCLUSION
53
5.1.1
Depth Sensing
One way the robot can have its hardware modified is by the inclusion of a Kinect sensor
(by Microsoft, originally built for the Xbox 360). The Kinect features an RGB camera
and a depth sensor [20], allowing it to perceive environments in 3D. Such a devise can be
used to upgrade the tracking aspects of the project.
Attaching a Kinect to the front of the robot will negate the need for pixel alignment (Section 3.2.7) and World Coordinate System transformations (Section 3.2.5), for
the world coordinates of the block centre points can be calculated from two different
perspectives also giving a much greater accuracy.
By attaching the sensor to the cuff of the robot (facing down towards the board), the
heights of objects played on the board can be detected with the depth sensor. Currently
the background subtraction (Section 3.2.3) and Hough Circles (Section 3.2.4) methods
recognise moves made by the player. Having the Kinect sensor implemented would negate
the requirement to have circles drawn on one side of each of the human players pieces.
5.1.2
Error Handling
For the purposes of time, I did not put much focus into handling any errors that the user
may cause in running the demo.
If the human player played a piece in between two spaces, Baxter will register it as
being played in the first space which has the smallest distance from the pieces centroid.
If Baxter were to play a piece in the empty of those two spaces, it would collide with the
piece played by the human player for it covers an unexpected area. A fix for this would
be to calculate the offset of the humans piece from the centre of space it is registered as
being played in. Baxter could then shift all the playing positions of the other grid spaces
by this offset, so as not to collide with any pieces. An alternative fix would be to have
Baxter pick up the humans piece and place it correctly in the registered space. On the
assumption that the human player will always place their pieces in the designated squares
CHAPTER 5. CONCLUSION
54
5.1.3
Game Variants
There are many variants of Noughts & Crosses. Qubic features a 4x4x4 board and
presents a much bigger challenge to both the player and someone developing an AI to
play it (Though also solvable [12], like the 3x3x3 variant). The code in this project has
been developed to allow modifications to work with larger game boards. Playing the
regular 2D version of the game is also very possible, as it was a prior iteration before the
final. Other variants such as Connect Four are worth a mention but are not as easily
adapted to as official variants of Noughts & Crosses.
5.1.4
Artificial Intelligence
Due to the nature of the board game, there came a small amount of challenge when playing
it over time. The focus of its behaviour was based on the 49 generated winning lines and
strategic value of particular spaces (such as the central space). The system can be made
more intelligent by a number of methods. I learnt from Newell and Simons Tic-Tac-Toe
program[7], that Baxter could be a more formidable opponent if it was aware of the fork
moves, which the entire strategy of Noughts & Crosses is based on. A possible way of
doing this would be to generate an array of all the possible fork moves, like the winning
lines.
Adaptive strategy based on game state could be implemented with Minimax. Section 2.3.2 discusses Minimax and how the algorithm works out a best possible move at
each state. Implementing this into the robots game logic would provide even more challenging game play.
It is possible to program an unbeatable strategy for the robot if it were to play first.
This would not provide a fair game to the human player. Implementing an AI level
(such as easy, medium and hard, where hard utilizes the unbeatable) could simulate
more realistic gameplay, with a further add-on randomizing the starting player.
5.1.5
CHAPTER 5. CONCLUSION
55
this method can be replaced with the robot speaking to indicate to the human player
that their turn has been detected.
5.1.6
Another way of acknowledging a move made by the human player is to manipulate the
position of the head display to nod. Though equally trivial as the speaking method of
the previous section, it provides further exploration of Baxters available functions.
5.1.7
Other Games
Aside from the Noughts & Crosses variants, there are other games this project can be
extended to work for. The functions of the robot Class in the code (described in Section 3.5.3), provide movements which are universal to most board games. Along with the
detection functions, most of the code can be applied to play different games, the only
requirement being a change in the game logic and data models. One possible game could
be Draughts (or Checkers).
References
[1] http://www.jaqueslondon.co.uk/noughts-crosses-3d.html. [Online; last accessed 01-June-2015].
[2] http://logicgame.com.ua/game.php?id=tic_tac_toe_3d&l=en1. [Online; last
accessed 01-June-2015].
[3] http://kylim.tistory.com/91. [Online; last accessed 01-June-2015].
[4] http : / / www . hizook . com / blog / 2012 / 09 / 18 /
baxter-robot-rethink-robotics-finally-unveiled.
[Online; last accessed
01-June-2015].
[5] http://onexia.com/baxter/Baxter-Electric-Parallel-Gripper.html. [Online;
last accessed 01-June-2015].
[6] H. Coles-Abell and V. Pugh. Connect Four Demo. http://sdk.rethinkrobotics.
com/wiki/Connect_Four_Demo. [Online; last accessed 30-May-2015].
[7] K. Crowley and R. S. Siegler. Flexible strategy use in young childrens tic-tac-toe.
Cognitive Science, 17:531561, 1993.
[8] O. Elton. 4d noughts and crosses. http://matheminutes.blogspot.co.uk/2012/
07/4d-noughts-and-crosses.html, 2012.
[9] M. Gardner. Hexaflexagons and Other Mathematical Diversions.
Chicago Press, 1988.
University of
56
57
REFERENCES
http : / / www .
Appendices
58
Appendix A
Personal Reflection
I found the entire project a very engaging experience. Up until this point, my education
featured timetabled lectures, which were assessed via coursework and exams. The move
from this style to one where I could essentially choose the times I wanted to work was a
great change. Responsibility and discipline play a big part when it comes to working by
your own timetable. I found learning this to be enjoyable and even tempts me to continue
research as a career option.
Being given access to the robot at all times proved essential, along with my own
workstation located by it. The near-solidarity of the project was unusual. I was used
to completing coursework in time for deadlines alongside my peers in previous years.
This semester I found myself rather alone in the work I had to do, as everyone else
were off completing their own projects. That being said, I found the PhD and postdoctorate students in the area I had been allocated to work more than helpful and very
accommodating.
The project started slowly. There were problems setting up workstations to work with
Baxter both in the lab and in virtual machines on my laptop. This was time consuming,
and the virtual machines were not even used. Another problem which took several days
lay with initializing the OpenCV library which resulted in a full reinstallation of the
operating system. After these initial setbacks the project gained momentum. Working
with Baxter was very rewarding. Unlike a piece of code that works and outputs data
correctly, Baxter shows a physical representation of successful code. The choice to use
Python was something I thanked myself for. The ease of debugging and the familiarity
of the syntax made the entire coding aspect comfortable. On top of this, the majority
of previous projects and online tutorials relating to Baxter use Python code in their
examples.
A large proportion of the subject matter was new to me. I had never worked with
robots, nor done particularly well in modules involving artificial intelligence. I had not
even taken the computer vision module for the first semester of this year. Nevertheless,
with the online tutorials and help from my colleagues, I gained useful, highly interesting
knowledge which I was able to apply. Computer vision, in the end, turned out to be the
most enjoyable aspect of the project. It shames me to think what I could of learnt if I
had taken it as a module. Having a better knowledge of detection techniques prior to
the project may have resulted in less time being spent on vision, with more time spent
attempting to complete the final objective.
There were two major setbacks over the course of the project. The first was family
related, and incurred in a loss of three weeks where I could have been working. Thankfully,
59
60
the university granted me an extension. The second came when attempting to utilize the
infra-red sensor on Baxters arm to detect heights of stacked playing pieces. This was by
far the most frustrating point of the project and incurred several consecutive late nights
working with the robot. On the assumption that the hardware was malfunctioning, a new,
more robust solution was used. I wish I had not wasted so much time trying to make
the infra-red method work, however, I was very happy to see that there was a possible
alternative solution and that the robot was eventually able to function as desired.
Overall, the project was a great success. Its rather entertaining to watch people
play against the robot and feel shamed by loosing to it on their first try. There are also
talks of having the software used on open days for prospective students, where they can
demo the robot themselves. The amount of invaluable knowledge I gained in doing the
project has started to show. When looking back over the demo projects which I read
over for background reading, I now have a solid understanding of how they work whereas
previously, I found the content confusing and rather daunting. Continuing a career in
robotics was something I never had considered until now.
My advice to future students attempting a project similar to this is to be generous
with time estimates. Robots do not have the second natures and common sense that us
humans do. A simple action that we can perform, such as picking up an orange, needs to
be broken down to many levels to enable a robot to mimic us. Think about how we see an
orange, its colour, its shape. A robot needs to be programmed to know what a circle is
and what orange is. When we pick it up, our sense of touch in our hands give us a large
description of how much force to apply and from what angle to grasp it. Robots need to
be programmed all this information, and many of them do not have an obvious sense of
touch. Baxter was no exception, everything I had to work with came down to what was
seen through a camera, and making movements based on this. This and the fact that
working with hardware in general provides many opportunities for failure, including the
configuration of work environments, is why I suggest leaving a project more time than
originally assumed.
Appendix B
Ethical Issues Addressed
There is no way my project, the nature of which revolves around playing a game, has any
ethical impact. There is an aspect of robots replacing the jobs of humans (in supermarkets,
for example), but my project does not incur such problems.
61
Appendix C
Video
This YouTube link shows a recorded demonstration of the code running on Baxter:
https://youtu.be/DMy8C9eJ4LE
62
Appendix D
Code
D.1
Repository
The following link is a public repository containing all code developed for this project.
The repository contains a readme detailing how to set up the work environment and run
the code. The rest of this appendix contains the code of the final version of the project.
https://bitbucket.org/ben2000677/fyp/
D.2
New3DtictacBaxter.py
#!/usr/bin/env python
import roslib
import sys
import inspect, os
import rospy
import string
from sensor_msgs.msg import Image
from sensor_msgs.msg import Range
from std_msgs.msg import Header
from moveit_commander import conversions
from baxter_core_msgs.srv import SolvePositionIK, SolvePositionIKRequest
import baxter_interface
from cv_bridge import CvBridge
import cv
import cv2
import numpy as np
import math
import random
from BoardClass3D import Board
63
64
APPENDIX D. CODE
directory =
os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
def main():
# get setup parameters
limb, distance = get_distance_data()
# read colour data from file
colours = get_object_data()
# create board class
theBoard = Board()
# create stack class instance
baxter = robot(limb, distance, colours, theBoard)
baxter.send_image(default)
print "limb
= ", limb
65
APPENDIX D. CODE
theBoard.printBoard()
if theBoard.checkWinner(baxter.playerPiece): # check if human has won
baxter.send_image(anger2)
theBoard.printBoard()
print(Hooray! You have won the game!)
break
elif theBoard.isFull(): # check if board is full
baxter.send_image(default)
theBoard.printBoard()
print(The game is a tie!)
break
baxter.send_image(default)
baxter.take_turn()
if theBoard.checkWinner(baxter.robotPiece):
baxter.send_image(happy2)
theBoard.printBoard()
print(The computer has beaten you! You lose.)
game = False
elif theBoard.isFull():
baxter.send_image(default)
theBoard.printBoard()
print(The game is a tie!)
break
sys.exit()
rospy.spin()
class robot():
def __init__(self, arm, distance, colours, board):
# piece data
self.playerPiece = X
self.playerPieceIndex = 1 # ( 1 or 2)
self.robotPiece = R
self.robotPieceIndex = 2
# space detection
self.board = board
self.storePointPix = True
self.detectedAllCorners = False
66
APPENDIX D. CODE
self.pixelPoints = []
self.board_tolerance = 0.00375
# board xyz
spaces = self.board.getSpaces()
self.boardZ = len(spaces)
# levels
= arm
self.limb_interface = baxter_interface.Limb(self.limb)
if arm == "left":
self.other_limb = "right"
else:
self.other_limb = "left"
self.other_limb_interface = baxter_interface.Limb(self.other_limb)
# gripper ("left" or "right")
self.gripper = baxter_interface.Gripper(arm)
# object color dictionary data
self.objects = colours
# zeroed lists for pixel coordinates of objects
self.x = np.zeros(shape=(len(colours)), dtype=int)
self.y = np.zeros(shape=(len(colours)), dtype=int)
# speeds
self.normal = 0.8
self.slow = 0.1
67
APPENDIX D. CODE
# start positions
self.start_pos_x = 0.50
# x
= front back
self.start_pos_y = 0.30
# y
= left right
self.start_pos_z = 0.15
# z
= up down
self.roll
= -1.0 * math.pi
# roll = horizontal
self.pitch
= -0.0 * math.pi
# pitch = vertical
self.yaw
= 0.0 * math.pi
# yaw
= rotation
self.cam_y_offset = -0.015
self.width
= 960
self.height
= 600
# Camera resolution
APPENDIX D. CODE
path = directory+/images/+img_name+.png
img = cv2.imread(path)
self.publish_img(img,False)
68
69
APPENDIX D. CODE
rospy.sleep(1)
# Sleep to allow for image to be published.
def publish_img(self,img,resize=True):
"""
Publisher to head display.
"""
if resize:
img = cv2.resize(img, (1024, 600))
msg = CvBridge().cv2_to_imgmsg(img, encoding="passthrough")
self.pub.publish(msg)
# Game Logic
#--------------------------------------------------------------------------------------#
def take_turn(self):
"""
Baxters turn, following game logic.
"""
def random_move(movesList,level):
# Returns a valid move from the passed list on the passed board.
# Returns None if there is no valid move.
possibleMoves = []
for i in movesList:
if not self.board.getPiece(level,i[0],i[1]):
possibleMoves.append(i)
if len(possibleMoves) != 0:
return random.choice(possibleMoves)
else:
return None
70
APPENDIX D. CODE
if copy.checkWinner(self.robotPiece):
z = self.board.makeMove(row,space, self.robotPiece)
self.place_piece(row, space, z)
return
# Check if the human player could win on his next move, and block them.
for row in range(self.boardY):
for space in range(self.boardX):
copy = self.board.getCopy()
if not self.board.getPiece(self.boardZ-1,row,space): # if space
is free
copy.makeMove(row,space, self.playerPiece)
if copy.checkWinner(self.playerPiece):
z = self.board.makeMove(row,space, self.robotPiece)
self.place_piece(row, space, z)
return
# Try to take the middle center, if its free and has a robot piece
below it.
if not self.board.getPiece(1,1,1) and self.board.getPiece(0,1,1) ==
self.robotPiece:
level = self.board.makeMove(1,1,self.robotPiece)
self.place_piece(1,1,level)
return
# Try to take the bottom center, if its free.
if not self.board.getPiece(0,1,1):
level = self.board.makeMove(1,1,self.robotPiece)
self.place_piece(1,1,level)
return
# Try to take one of the corners of the bottom level, if they are free.
move = random_move([(0,0),(2,0),(0,2),(2,2)], 0)
if move != None:
level = self.board.makeMove(move[0],move[1],self.robotPiece)
self.place_piece(move[0], move[1], level)
return
71
APPENDIX D. CODE
self.place_piece(1,1,level)
return
# Tracking Stuff
#--------------------------------------------------------------------------------------#
def get_contours(self,i,hsv):
"""
Gets contours from image.
"""
mask = cv2.inRange(hsv, np.array(self.objects[i][0:3]),
np.array(self.objects[i][3:6]))
# filter and fill the mask
kernel =
cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(self.objects[i][6],self.objects[i][6])
mask2 = cv2.morphologyEx(mask,cv2.MORPH_OPEN,kernel)
ret,thresh = cv2.threshold(mask2,127,255,0)
contours, hierarchy =
cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
return contours
def detect_and_draw(self,imgmsg):
"""
Main detection function, constantly loops through.
"""
APPENDIX D. CODE
72
#-----------------------------------HOUGH_CIRCLES--------------------------------------#
# for detecting the board
# get grayscale version of image
cimg = img[:,:,0:3]
cimg = cv2.cvtColor(cimg,cv2.COLOR_BGR2GRAY)
cimg = cv2.medianBlur(cimg,5)
# extract circles from the image
circles =
cv2.HoughCircles(cimg,cv.CV_HOUGH_GRADIENT,1,20,param1=50,param2=30,minRadius=10,ma
# ensures all 4 corners are visible and that point storing is enabled
if circles != None and self.storePointPix:
circles = np.uint16(np.around(circles))
centers = []
for i in circles[0,:]:
centers.append([i[0],i[1],i[2]])
dis_TR = 100000
dis_TL = 100000
dis_BR = 100000
dis_BL = 100000
# calculate which corners the detected circles are (closest distance)
for i in centers:
dis = np.sqrt((i[0]-1000)**2 + (i[1])**2)
if dis < dis_TR:
dis_TR = dis
TR = i
dis = np.sqrt((i[0])**2 + (i[1])**2)
if dis < dis_TL:
dis_TL = dis
TL = i
APPENDIX D. CODE
dis = np.sqrt((i[0]-1000)**2 + (i[1]-1000)**2)
if dis < dis_BR:
dis_BR = dis
BR = i
dis = np.sqrt((i[0])**2 + (i[1]-1000)**2)
if dis < dis_BL:
dis_BL = dis
BL = i
TR = map(int,TR)
TL = map(int,TL)
BR = map(int,BR)
BL = map(int,BL)
# draw circles around detected circles
cv2.circle(img,(TR[0],TR[1]),TR[2],(0,255,0),2)
cv2.circle(img,(TR[0],TR[1]),2,(0,0,255),3)
cv2.circle(img,(TL[0],TL[1]),TL[2],(0,0,255),2)
cv2.circle(img,(TL[0],TL[1]),2,(0,0,255),3)
cv2.circle(img,(BR[0],BR[1]),BR[2],(255,0,0),2)
cv2.circle(img,(BR[0],BR[1]),2,(0,0,255),3)
cv2.circle(img,(BL[0],BL[1]),BL[2],(150,30,200),2)
cv2.circle(img,(BL[0],BL[1]),2,(0,0,255),3)
# distances between the grid space centres
dx = (float(TR[0])-float(TL[0]))/4
dy = (float(TR[1])-float(TL[1]))/4
dx2 = (float(BR[0])-float(BL[0]))/4
dy2 = (float(BR[1])-float(BL[1]))/4
# middle row beginning and end points
MLx = TL[0]+int(np.round((float(BL[0])-float(TL[0]))/2))
MLy = TL[1]+int(np.round((float(BL[1])-float(TL[1]))/2))
MRx = TR[0]+int(np.round((float(BR[0])-float(TR[0]))/2))
MRy = TR[1]+int(np.round((float(BR[1])-float(TR[1]))/2))
# distances between the grid space centres of the middle row
dx3 = (MRx - MLx)/4
dy3 = (MRy - MLy)/4
self.pixelPoints = range(self.boardX **2)
for i in range(0,self.boardX):
73
74
APPENDIX D. CODE
top =
(TL[0]+int(np.round(dx*(i+1))),TL[1]+int(np.round(dy*(i+1))))
# top row
mid = (MLx+int(np.round(dx3*(i+1))),MLy+int(np.round(dy3*(i+1))))
# middle row
bot =
(BL[0]+int(np.round(dx2*(i+1))),BL[1]+int(np.round(dy2*(i+1))))
# bottom row
self.pixelPoints[i] = top
self.pixelPoints[i+3] = mid
self.pixelPoints[i+6] = bot
cv2.circle(img,top,2,(80*(i+1),0,0),3)
cv2.circle(img,mid,2,(0,80*(i+1),0),3)
cv2.circle(img,bot,2,(0,0,80*(i+1)),3)
# if all the corners are detected
if len(centers) == 4:
self.detectedAllCorners = True
#----------------------------BLOB_DETECTION---------------------------------------#
# for detecting robot blocks
# Blur each frame
simg = cv2.GaussianBlur(img,(5,5),0)
# Convert BGR to HSV
hsv = cv2.cvtColor(simg, cv2.COLOR_BGR2HSV)
i = self.robotPieceIndex
contours = self.get_contours(i, hsv)
my_contours = []
minDist = 10000
for cnt in contours:
if cv2.contourArea(cnt)>1400:
my_contours.append(cnt)
x,y,w,h = cv2.boundingRect(cnt)
contour
if self.displayContours:
cv2.rectangle(img,(x,y),(x+w,y+h),(0,0,255),2)
cx = x + (w/2)
cy = y + (h/2)
# centre points
75
APPENDIX D. CODE
dist = self.distance_to((cx,cy),(self.width,self.height)) #
target the most-right-bottom blob
if dist < minDist:
minDist = dist
self.x[i-1] = cx
self.y[i-1] = cy
if self.displayContours:
cv2.drawContours(img,my_contours,-1,(0,255,0),3)
if self.displayCamera:
self.publish_img(img)
cv2.imshow(RGB,img)
k = cv2.waitKey(5) & 0xFF
#--------------------------------------------------------------------------------------#
# create subscriber to the required camera
def subscribe_to_camera(self, camera):
if camera == "left":
#callback = self.left_camera_callback
camera_str = "/cameras/left_hand_camera/image"
elif camera == "right":
#callback = self.right_camera_callback
camera_str = "/cameras/right_hand_camera/image"
else:
sys.exit("ERROR - subscribe_to_camera - Invalid camera")
image_topic = rospy.resolve_name(camera_str)
# subscribe to detection topic
rospy.Subscriber(image_topic, Image, self.detect_and_draw)
= (self.width / 2) - block_centre[0]
pixel_dy
= (self.height / 2) - block_centre[1]
76
APPENDIX D. CODE
error
= float(pixel_error * self.cam_calib *
self.block_pickup_height)
x_offset = - pixel_dy * self.cam_calib * self.block_pickup_height
y_offset = - pixel_dx * self.cam_calib * self.block_pickup_height
# update pose and find new block location data
self.update_pose(x_offset, y_offset)
# find displacement of target from centre of image
pixel_dx
= (self.width / 2) - block_centre[0]
pixel_dy
= (self.height / 2) - block_centre[1]
= float(pixel_error * self.cam_calib *
self.block_pickup_height)
return block_centre, error
def config_camera(self, camera, x_res, y_res):
if camera == "left":
cam = baxter_interface.camera.CameraController("left_hand_camera")
elif camera == "right":
cam = baxter_interface.camera.CameraController("right_hand_camera")
else:
sys.exit("ERROR - open_camera - Invalid camera")
# set camera parameters
cam.resolution
= (int(x_res), int(y_res))
cam.exposure
= -1
cam.gain
= -1
cam.white_balance_blue = -1
cam.white_balance_green = -1
cam.white_balance_red = -1
def detect_pieces(self):
"""
Detect moves made by the opponent.
"""
def subtract_background():
"""
77
APPENDIX D. CODE
Apply backround subtraction.
"""
# take old and new image
oldImg = CvBridge().imgmsg_to_cv2(self.bgsubImgmsg,
desired_encoding=passthrough)
img = CvBridge().imgmsg_to_cv2(self.imgmsg,
desired_encoding=passthrough)
fgbg = cv2.BackgroundSubtractorMOG()
oldImg = oldImg[:,:,0:3]
fgmask = fgbg.apply(oldImg)
img = img[:,:,0:3]
fgmask = fgbg.apply(img)
# subtract images
img1_bg = cv2.bitwise_and(img,img,mask = fgmask)
# Blur image
simg = cv2.GaussianBlur(img1_bg,(5,5),0)
# Convert BGR to HSV
hsv = cv2.cvtColor(simg, cv2.COLOR_BGR2HSV)
# check for contours of the players piece
contours = self.get_contours(self.playerPieceIndex, hsv)
areaMax = 0.0
cx = 0
cy = 0
for cnt in contours:
if cv2.contourArea(cnt)>1400 and cv2.contourArea(cnt)>areaMax:
areaMax = cv2.contourArea(cnt)
x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img1_bg,(x,y),(x+w,y+h),(0,0,255),2)
cx = x + (w/2) # Centre points
cy = y + (h/2)
pos = (cx,cy)
if pos != (0,0):
# if a contour was
detected
self.bgsubImgmsg = self.imgmsg
self.publish_img(img1_bg)
78
APPENDIX D. CODE
minDist = float(inf)
x = 10
y = 10
# find the closest grid space to the contour
for row in range(self.boardY):
for col in range(self.boardX):
dist = self.distance_to(pos,self.board.getPixel(row,col))
if dist < minDist:
minDist = dist
x = row
y = col
# update the board data model
self.board.makeMove(x,y,self.playerPiece)
return True
else:
return False
def check_circles():
"""
Checks for pieces with circles drawn on them.
"""
centers = []
for frames in range(self.numCircleFrames): # checks a number of
frames
img = CvBridge().imgmsg_to_cv2(self.imgmsg,
desired_encoding=passthrough)
# for circle pieces
cimg = img[:,:,0:3]
cimg = cv2.cvtColor(cimg,cv2.COLOR_BGR2GRAY)
cimg = cv2.medianBlur(cimg,5)
# extract circles
circles =
cv2.HoughCircles(cimg,cv.CV_HOUGH_GRADIENT,1,20,param1=50,param2=30,minRadiu
if circles != None:
circles = np.uint16(np.around(circles))
for i in circles[0,:]:
center = (i[0],i[1])
79
APPENDIX D. CODE
if center not in centers:
centers.append(center)
cv2.circle(img,(i[0],i[1]),i[2],(0,255,255),3)
cv2.circle(img,(i[0],i[1]),2,(0,0,255),3)
# show detected circle on head display
self.publish_img(img)
if frames > 0:
rospy.sleep(1)
80
APPENDIX D. CODE
# update stored image
self.bgsubImgmsg = self.imgmsg
return True
return False
# countdown prompt
self.send_image(2)
self.send_image(1)
self.send_image(go)
rospy.sleep(2)
if check_circles() or subtract_background():
# then there is a change
rospy.sleep(3)
return
def distance_to(self,i,j):
a = (i[0]-j[0])
b = (i[1]-j[1])
c2 = (a*a) + (b*b)
return math.sqrt(c2)
# Movement Functions
#--------------------------------------------------------------------------------------#
def place_piece(self, x, y, level):
self.start_position()
self.toggleDisplayFeed(1,True)
self.find_object()
self.send_image(default)
self.pickup_object()
self.drop_on(self.board.getPosition(x,y),level+1)
def start_position(self):
self.setSpeed(self.normal)
# move back to start position
81
APPENDIX D. CODE
self.pose = [self.start_pos_x, self.start_pos_y, self.start_pos_z, \
self.roll, self.pitch, self.yaw]
self.baxter_ik_move(self.limb, self.pose)
def find_object(self,BoardMode=False):
# iterates to find closest pose above object and returns it
if not BoardMode:
tolerance = self.block_tolerance
i = self.robotPieceIndex-1
else:
tolerance = self.board_tolerance
# middle pixel point index of the pixelpoint array
i = int(len(self.pixelPoints)/2)
error
= 2 * tolerance
iteration = 1
# iterate until arm over centre of target
while error > tolerance or not self.detectedAllCorners:
if not BoardMode:
block_centre = (self.x[i],self.y[i])
else:
if self.pixelPoints[i] == i:
+= 1
return self.pose
def pickup_object(self):
self.setSpeed(self.slow)
# save return pose
origin = self.pose
# move down to pick up object
self.pose = (self.pose[0] + self.cam_x_offset,
self.pose[1] + self.cam_y_offset,
self.pose[2] + (self.block_grip_height self.block_pickup_height),
APPENDIX D. CODE
self.pose[3],
self.pose[4],
self.pose[5])
self.baxter_ik_move(self.limb, self.pose)
self.gripper.close()
# return to origin
self.baxter_ik_move(self.limb, origin)
self.setSpeed(self.normal)
def drop_on(self,target,level):
# drop object onto target below obj pose param
self.pose = target
self.baxter_ik_move(self.limb, self.pose)
self.setSpeed(self.slow)
self.pose = (self.pose[0] + self.cam_x_offset,
self.pose[1] + self.cam_y_offset,
self.pose[2] + (self.block_grip_height - (self.distance (self.block_height * level))),
self.pose[3],
self.pose[4],
self.pose[5])
self.baxter_ik_move(self.limb, self.pose)
self.gripper.open()
self.pose = target
self.baxter_ik_move(self.limb, self.pose)
self.setSpeed(self.normal)
def locate_board(self):
self.setSpeed(self.slow)
centre = self.find_object(True)
# stop storing pixel points at centre position and update Board class
model
self.storePointPix = False
counter = 0
centrePix = self.pixelPoints[4]
for row in range(self.boardY):
for col in range(self.boardX):
self.board.setPixel(row,col,self.pixelPoints[counter])
self.board.setPosition(row,col,(centre[0] +
((self.pixelPoints[counter][1] - centrePix[1]) *
self.cam_calib * self.distance),
82
83
APPENDIX D. CODE
centre[1] +
84
APPENDIX D. CODE
# convert response to joint position control dictionary
limb_joints = dict(zip(ik_response.joints[0].name,
ik_response.joints[0].position))
# move limb
if self.limb == limb:
self.limb_interface.move_to_joint_positions(limb_joints)
else:
self.other_limb_interface.move_to_joint_positions(limb_joints)
else:
# display invalid move message on head display
self.send_image(errorjoint)
# if working arm
quaternion_pose = self.limb_interface.endpoint_pose()
position
= quaternion_pose[position]
# Setup Functions
#--------------------------------------------------------------------------------------#
def get_distance_data():
# read the setup parameters from setup.dat
file_name = directory + "/setup.dat"
try:
f = open(file_name, "r")
except ValueError:
APPENDIX D. CODE
sys.exit("ERROR: golf_setup must be run before golf")
#find limb
s = string.split(f.readline())
if len(s) >= 3:
if s[2] == "left" or s[2] == "right":
limb = s[2]
else:
sys.exit("ERROR: invalid limb in %s" % file_name)
else:
sys.exit("ERROR: missing limb in %s" % file_name)
#find distance to table
s = string.split(f.readline())
if len(s) >= 3:
try:
distance = float(s[2])
except ValueError:
sys.exit("ERROR: invalid distance in %s" % file_name)
else:
sys.exit("ERROR: missing distance in %s" % file_name)
return limb, distance
def get_object_data():
Objects_pointer = open(directory+/objects.txt, r)
OBJ = {}
for line in Objects_pointer:
line = line.strip(,\n)
if line == END:
break
fields = line.split(,)
fields = map(int, fields)
OBJ[fields[0]] = fields[1:len(fields)]
return OBJ
85
APPENDIX D. CODE
if __name__ == "__main__":
main()
D.3
BoardClass3D.py
#!/usr/bin/env python
import copy
class Board():
def __init__(self,x=3,y=3,z=3):
self.cube = []
self.spaces = []
# create a 3D matrix of empty spaces
for i in range(0,z):
level = []
for j in range(0,y):
row = []
for k in range(0,x):
space = Space()
row.append(space)
self.spaces.append((i,j,k))
level.append(row)
self.cube.append(level)
self.solutions = self.generateSolutions()
def generateSolutions(self):
"""
Generates winning lines for 3D Noughts & Crosses
"""
def allEqual(x,y,z):
return (x==y) and (y==z)
def isSorted(x,y,z):
return (x > y and y > z) or (z > y and y > x)
solutions = []
for a in self.spaces:
for b in self.spaces:
86
87
APPENDIX D. CODE
for c in self.spaces:
# check every combination of 3 distinct spaces
if a != b and b != c and a != c:
line = [a,b,c]
line.sort()
if not line in solutions:
level_eq = allEqual(a[0],b[0],c[0])
row_eq = allEqual(a[1],b[1],c[1])
col_eq = allEqual(a[2],b[2],c[2])
level_sort = isSorted(a[0],b[0],c[0])
row_sort = isSorted(a[1],b[1],c[1])
col_sort = isSorted(a[2],b[2],c[2])
# logic statements to check if line is a winning line
if ((level_eq and row_eq) or (level_eq and col_eq) or
(col_eq and row_eq)) or \
(level_sort and row_sort and col_sort) or \
((level_eq and row_sort and col_sort) or (level_sort
and row_eq and col_sort) or (level_sort and
row_sort and col_eq)):
solutions.append(line)
return solutions
def getCopy(self):
return copy.deepcopy(self)
def checkWinner(self,letter):
board = self.cube
for solution in self.solutions:
if
board[solution[0][0]][solution[0][1]][solution[0][2]].getContent()
== letter \
and
board[solution[1][0]][solution[1][1]][solution[1][2]].getContent()
== letter \
and
board[solution[2][0]][solution[2][1]][solution[2][2]].getContent()
== letter:
return True
def makeMove(self, row, col, letter):
"""
Updates the data model with a new piece
88
APPENDIX D. CODE
"""
for level in range(len(self.cube)):
if not self.getPiece(level,row,col):
self.setPiece(level,row,col,letter)
return level
def isFull(self):
# Return True if every space on the top of the board has been taken.
Otherwise return False.
for row in self.cube[len(self.cube)-1]:
for space in row:
if not space.getContent(): # if space is free
return False
return True
def setPosition(self, row, col, pose):
self.cube[0][row][col].setPose(pose)
def getPosition(self, row, col):
return self.cube[0][row][col].getPose()
def getSpaces(self):
return self.cube
#used to be self.spaces
89
APPENDIX D. CODE
else:
return 0
def setPixel(self, row, col, pixel):
self.cube[0][row][col].setPix(pixel)
def getPixel(self,row, col):
return self.cube[0][row][col].getPix()
def printBoard(self):
"""
Prints the board data model to terminal
"""
for level in range(len(self.cube)-1,-1,-1):
counter = 0
print " 0 1 2"
for row in self.cube[level]:
print " -------"
string = str(counter)
for space in row:
string += "|"
if not space.getContent():
string += " "
else:
string += space.getContent()
print string + "|"
counter += 1
print " -------"
class Space():
def __init__(self):
self.piece = 0
self.pose = (0,0,0,0,0,0)
self.pixel = (0,0)
def setPose(self,pose):
self.pose = pose
def getPose(self):
return self.pose
APPENDIX D. CODE
def setContent(self,piece):
self.piece = piece
def getContent(self):
return self.piece
def setPix(self,pixel):
self.pixel = pixel
def getPix(self):
return self.pixel
90
Appendix E
Board Detection Times Under Varying
Light Conditions
Figure E.1: Times taken to locate the game board under varying light conditions.
91
Time (Seconds)
80
60
40
20
Figure E.2: Recorded times in Minimal lighting conditions, arranged in ascending order.
40
Time (Seconds)
35
30
25
20
15
Figure E.3: Recorded times in Artificial lighting conditions, arranged in ascending order.
22
Time (Seconds)
20
18
16
14
12
10
Figure E.4: Recorded times in Natural lighting conditions, arranged in ascending order.