Table of Contents

Abstract ...................................................................................................................................... ii
Table of Contents ........................................................................................................................ iii
List of Figures ............................................................................................................................. v
List of Tables.............................................................................................................................. vi
Chapter 1 Introduction .......................................................................................................... 1
1.1 Motivation............................................................................................................................... 1
1.2 Problem Statement ................................................................................................................. 3
1.3 Previous work.......................................................................................................................... 5
1.4 Assumptions ............................................................................................................................ 8
1.5 Scope ....................................................................................................................................... 8
1.6 Approach ................................................................................................................................. 9
1.7 Material and Equipment ......................................................................................................... 9
1.8 Structure of Thesis .................................................................................................................. 9
Chapter 2 Color Theory and Challenges ................................................................................ 10
2.1 Visible color spectrum........................................................................................................... 10
2.2 Color Dimension.................................................................................................................... 12
2.3 Color Space and Significance ................................................................................................ 14
2.3.1 Physics and technic based color space ......................................................................... 14
2.3.2 Uniform Color Space ..................................................................................................... 16
2.3.3 Perception-based Color Space ...................................................................................... 18
2.3.4 Color space selection .................................................................................................... 19
2.5 Digital Color Image Formation ................................................................................................... 20
2.4 Effect of Color Filter in Monochrome Camera ...................................................................... 21
2.5 Color-based Recognition Challenges .................................................................................... 22
2.6.1 Color constancy and Illumination Problem .......................................................................... 22
2.6.2 Sensitivity to Shadows and Occlusions ................................................................................ 22
Chapter 3 Marker Choice and Design.................................................................................... 23
3.1 Color Choice .......................................................................................................................... 23
3.1.1 Surgical Environment Color Analysis.................................................................................... 23
3.1.2 RAL Colors ............................................................................................................................ 24

iii
 3.1.1 Color Selection for Markers .............................................................................................. 25
3.2 Shape and Size Selection....................................................................................................... 26
3.2.1 Shape-based Recognition Challenges ........................................................................... 26
3.2.2 Optimum Shape Recognition Factor ................................................................................. 26
3.2.3 Marker Individual Arrangement ....................................................................................... 26
3.2.4 Size Factor for Recognition Ability .......................................................................................... 26
4 Recognition Strategy and Implementation .......................................................................... 26
4.1 Recognition Problems Revisited ........................................................................................... 26
4.2 Recognition Steps.................................................................................................................. 26
4.3 Position and Orientation Computation................................................................................. 26
4.4 Real-time Compatibility ........................................................................................................ 26
5 Experiments and Results ..................................................................................................... 27
Future Work .............................................................................................................................. 27
Conclusion ................................................................................................................................. 27
APPENDIX .................................................................................................................................. 28
Program codes ................................................................................................................................. 28
References................................................................................................................................. 29

iv
List of Figures
Figure 1-1 modiCAS system in operating room ...................................................................................... 1

Figure 1-2 Components of the navigated robotic system for surgical assistance .................................. 2

Figure 2-1 The electromagnetic and sound spectra ............................................................................. 11

Figure 2-2 Color hue.............................................................................................................................. 13

Figure 2-3 Reflection of value through shade, tone and tint................................................................ 13

Figure 2-4 Reflection of purity of colors (chroma)................................................................................ 13

Figure 2-5 RGB Color Space (Andreas Koschan, 2008) ......................................................................... 15

Figure 2-6 RGB chromaticity diagram (Maxwell Color Triangle)........................................................... 15

Figure 2-7 Standard primaries' wavelengths and spectral power ........................................................ 16

Figure 2-8 CIE Chromaticity Diagram .................................................................................................... 17

Figure 2-9 HSI Color Space .................................................................................................................... 18

Figure 2-10 HSV Color Space ................................................................................................................. 18

Figure 3-1 Example of RAL shades in violet category ........................................................................... 24

v
 Other works in this field includes using compressed chromaticity histograms of color-
channel-normalized image (Mark S. Drew, 1998) in which seminal methods of Swain and Ballard is
extended to discount changes in illumination. However, the processing is computationally intensive
for real-time even though the results are excellent. Bilinear models (D'Zmura, 1991) is also
introduced in which a particular combination of photoreceptors and models for reflectance and
illuminants is used. However, since this research uses three photoreceptors that originates from
initially known color filters the illumination procedures cannot be adapted into the current scope of
research. Illumination invariant color detection schemes also proposed using statistical method as in
(Mirgitta Martinkauppi, 2003). Even though it seems to work for small illumination chromaticity
changes, the broader range did not work.

The previous work shape understanding prevails some interesting aspect for this
research. Circular and elliptic object tracking in real time which proves to be the easiest recognition
procedures met is also presented by the team of researchers from Magdeburg, Germany (Christian
Teutsch, 2006). This research focuses more on easier shape recognition even though color images
are used for experiments. Therefore, instead of just relying on the color, the shape descriptors
combine both information for object recognition. Even though the 3D tracking prove great results,
the target almost have no background clutters and dominates the input image regions which is
therefore irrelevant for current research.

Figure 1-7 Contours descriptors result. Circular shape has significantly small values.

An iterative segmentation method for combining both shape and color descriptors (J.M
Chassery, 1984) also proves good result in which color similarity is done first using log-transformed
images and then followed by convex hull and convexity criterion. The experiment is done on cell-
shape images which in contrast to the research scope, target dominates the input images. However,
since the methodology is quite simple and might work in real time, the procedures will also be
tested during experimental procedures for this research.

7
1.4 Assumptions

This research assumed that the original setup of the modiCAS system is retained. The surgical
environment is the main scene with fluorescent light illumination. The spectral power distribution of
the white illumination fluorescent light is assumed to be constant at all times. All markers have the
consistent shape through all video frames; their corresponding colors are constant in visible areas
within the same illumination. The objects of interest (markers) are assumed undistorted in front of
cameras, and color sensitivity is assumed identical for both cameras, in which no specular variation
exist between left and right frames.

1.5 Scope

The scope of this research is to develop color markers that have:

i) High discriminating power, in this case the color combination of 3 different markers
made it possible to distinguish them without any arrangement computation
ii) Robustness against object occlusion and cluttering, in this case the most feasible size
and shape for the markers to be recognized within 2m range
iii) Robustness against changes in direction, intensity and spectral power distribution of the
illumination, which translates to color constancy in all viewing directions and locations.
iv) Minimum computation complexity in which the accuracy is within acceptable range and
most importantly fit for real time computing (at least 24 frames/sec).
v) Color characteristics that has commercial availability and producible.

This research will not span into merging the current result with previous modiCAS system and only
serve for justification purposes whether the color marker is adequate for the recognition in surgical
environment.

8
1.6 Approach

This research has taken an approach of studying the possibility of using colored markers and tested
them against the crucial factors, time and accuracy. Therefore, the recognition algorithm is mainly
tested in MATLAB using still pictures and video streams for the ability to quickly justify the results.
Once the result reaches the appropriate accuracy, the algorithm then programmed into a C++
programming to reduce the time consumption for the recognition. The experiment will justify
whether color markers are suitable to replace the current infrared markers with acceptable accuracy
and processing time.

1.7 Material and Equipment

Markers are chosen to be in spherical shape, made of wood and mounted on ice cream wooden stick
to provide suitable beacons for position and orientation determination.

Stereo cameras are used to provide a video streaming input, and calibration procedures are
executed to provide intrinsic parameters useful for unwrapping target position in real time.

1.8 Structure of Thesis

In the next chapter, Chapter 2, the foundation of colors as well as representation of colors in
different color space is described. Furthermore, the reason behind the choice of certain colors and
color space representation is debated and disclosed. In Chapter 3, the marker development process
is discussed. The choice of marker’s shape and size is explained and how the final choice is
determined. In Chapter 4, recognition strategy as well as color identification with illumination
invariant properties is described. The new algorithm is explained together with flowchart to
understand how the color marker can be recognized. In Chapter 5, the results from experiments are
presented in graphical and tabular form and explanations are provided to explain the success rate
and failures. Lastly, conclusion and summary of general issues are discussed in the final chapter.
Future work is also presented.

9
Chapter 2 Color Theory and Challenges

Color-based recognition is relatively new to the field of real time object recognition. It has not
received much attention in the past mainly because of large processing time and the incapability of
computers to process relatively large amount of data and apply recognition strategies
simultaneously. In order to study the advantage of color object recognition, it is of great importance
to dig the very basic understanding of color and finally how it improves recognition ability. This
chapter will begin with the foundation of colors and subsequently highlight the importance of
choosing the right color space for the image processing; and how color distance provides means to
substantiate the different object’s presence in the acquired image.

2.1 Visible color spectrum

Colors that we can see with our eyes originate from colorless light of the sunlight. The light reflected
from an object is absorbed by the cone cells of human visual system hence leads to color perception
(Tinku Acharya, 2005). The color of the object is the result of the ability of the surface of the object
to absorb the other color in the spectrum and reflect a particular color (i.e. light of certain
wavelength) to give its color identity. An apple for example absorbs all wavelengths of incident light
other than the wavelengths of red spectrum (i.e. approximately 770nm) and then reflects it to
human eyes to give a red color identity of the apple. Considering there are three cone classes in
human vision system to perceive color, the different amount of absorptions by these cone cells
determines the color of the incident light.

Digital color images also adapt the same way human eyes perceive colors in which additive
model is applied. The differing color stimuli are combined into one color through projective
overlapping. In the case of color images, once the light of different wavelengths hit the image
sensor; different arrangement of red, green and blue filters masked on the sensor element will
produce distinctive colors of the scene at hand.

10
 Figure 2-1 The electromagnetic and sound spectra

However, contrary to popular belief, color is a perceived phenomenon. It is non-measurable

and not a physical dimension even though the electromagnetic radiation of the visible wavelength
spectrum that enumerates the color is a measurable quantity. Therefore, color recognition has
undergone a broad research due to this fact. Being perceptual in nature, the same color might be
seen differently by different observers. Therefore, standardization has been employed in color
representation and models to categorize colors according to their dimensions. It is important for this
scope of research since the goal is to produce appropriate marker colors for recognition.

Color models that are usually discussed are in the form of additive and subtractive models.
Human eyes and imaging device adapted to additive models in which primary colors are used
additively to provide other colors. Color printing, color dyes and paints on the other hand use
subtractive primary colors mixing in which cyan, magenta, and yellow as the primary. CYMK model
that is used for printing uses these complementary primary colors with a combination of different

11
amounts of cyan that will control how much red is shown; magenta which will control how much
green will be shown and yellow to show how much blue is produced.

Figure 2-2Additive color mixing

Figure 2-3 Subtractive color mixing

With these two different models explained, the color of painted object produced by the
manufacturers undergoes different procedures than the way color is displayed on a monitor.
Therefore, careful consideration for choosing marker colors is important to reduce probability of
choosing different colors than that of displayed by the computer monitors.

2.2 Color Dimension

Color dimension defines how colors are categorized and therefore limit the generalities of defining
colors. Scientifically, there are three dimensions of color namely hue, value and chroma. Hue
identifies color by name. Every color falls into definite hue as indicated by the spectrum color name
on the color wheel. Value determines the lightness or darkness of a hue achieved by adding white,
gray or black. White added to a hue creates tint, gray addition results tone and black addition
creates shade. Chroma on the other hand refers to the degree of intensity, strength, saturation or
purity of a color. Purity of a color refers to its freedom from white, black or grey.

12
 Figure 2-4 Color hue

Figure 2-5 Reflection of value through shade, tone and tint

Figure 2-6 Reflection of purity of colors (chroma)

Since color is perceptual, it is therefore quite difficult for color naming or categorization to be done
empirically without any standards or collectable research. Hence, many organizations produce their
own color collection for easier reference especially those that involve primarily in printing or paint
production.

13
2.3 Color Space and Significance

In order to transfer the color information for the computer to understand, color images must
have a form of representation to be stored, displayed and processed. Either for the sake of
mathematical demands, processing algorithms, technical conditions of hardware; even though the
fact that these demands cannot be met simultaneously, a color representation is crucial to correctly
process them. In other words, color spaces provide representation of sensor outputs for easier
computation as well as accommodate the way human perceive colors. Therefore, the birth of
different color spaces which can be categorized largely by physics and technic-based color space,
uniform color space and perception-based color space is gratefully welcomed.

Color space is the color coordinate system by which the image values of color image are
represented. The difference between two image values in a color space is called color distance. As
explained earlier, since there are different type of color space is available, color distance
computation also differ from one type of color space to another.

The most important characteristics of the color space that is advantageous for color
recognition is the chromaticity diagram. It represents color information that survives the elimination
of brightness information. In other words, color recognition can be done in two dimensional
relieving the stress on extended computation. The chromaticity diagram differs depending on the
color space and will be explain accordingly later.

2.3.1 Physics and technic based color space

This category is defined as the color spaces that have a direct technical reference. The most
well known color space, RGB belongs to this category. RGB color space is based on additive mixture
of three primary colors, Red, Green and Blue hence the name even though it is actually perceptual
colors that resulted from a standardized long, middle and short wavelengths of visible light.

14
 Figure 2-7 RGB Color Space (Andreas Koschan, 2008)

RGB color space is characterized as a cube in which it is distinctive integer-wise in the range of [0-
255] for every axis (for 24-bit-8 bit/channel). Chromaticity of RGB color space lies on a two
dimensional triangle of R+G+B=1 and the achromatic colors (gray tones) lies on principle diagonal
R=G=B. Distinctive colors are calculated by accounting this space and Euclidean space in which
Euclidean distance is used to measure the color distance.

Figure 2-8 RGB chromaticity diagram (Maxwell Color Triangle)

The significant advantage of using this color space for image processing is the fact that
imaging hardware sends the signal in this color space values, therefore eliminating the needs of any
further conversion for the signal to be processed. It is important whenever the application is speed-
oriented such as real-time application. However, the color distance of RGB space does not reflect the
same perceptual changes of colors human sees. It means, two different colors at a distance, d apart

15
in one part of the color space does not imply the same degree of perceptual difference as two other
colors the same distance apart in another part of the color space. Color features recognition using
color distance is therefore relatively inaccurate using RGB color space.

Other types of color spaces in this category are of less significance to this research because they
belong to printing colors (CYMK), color television systems (YUV, YIQ) and digital videos (YCBCR). As
self-explanatory color spaces, these are application specific and belongs outside the scope of this
research.

2.3.2 Uniform Color Space

This category signifies color spaces that have the same size changes in color coordinates as
the change in visible color tones and saturation. This means that the Euclidean distance between
two color points in any part of the color space corresponds to the perceptual difference between the
two colors by the human vision system. Two color spaces CIELAB and CIELUV, in which both are
commenced by the International Lighting Commission, CIE (Commission Internationale de
l’Eclairage) belong to this category. These color spaces are derived from XYZ standard color system.
CIE has set a standard using long, middle and short wavelength of certain spectral powers for
primary values.

Figure 2-9 Standard primaries' wavelengths and spectral power

XYZ is virtual primary values defined by CIE using the conversion of real spectral value
curves. It is a linear transformation as explained below:

16
CIELAB was developed in accordance to Munsell color order system in which colors are described
using attributes hue, lightness and chroma. It is characterized by (a*,b*) as chromaticity axis and L*
as lightness. L* is scaled approximately equidistant according to sensitivity.

Figure 2-10 CIE Chromaticity Diagram

CIELUV shares the same L* information of CIELAB but differs for the chromaticity (u*,v*). In contrary
with CIELAB, the chromaticity points are mapped the same way as in standard color tables therefore
is advantageous for additive mixture calculation. The main advantage using this type of color space is
the fact that it is uniform in which it can be treated as Euclidean space with much less errors
compared to RGB. Euclidean distance is therefore accurate to signify the distinctive colors. Hence, it
is great for comparing very similar colors.

CIELAB color space is normally used for describing body colors; CIELUV on the other hand is used for
the description of colors of light. The downside of this color space is the computational times. This is
a significant drawback for video real time processing.

17
2.3.3 Perception-based Color Space

Color spaces that belong to this category are based intuitively on human color perception.
HSI and HSV fit in this category. HSI uses hue, saturation and intensity as coordinate axes. Red is
specified as reference color in which hue at 0o and 360o correspond to this color. Saturation is in the
range of [0,1] in which 1 refers to pure color and 0 refers to achromatic colors.

Figure 2-11 HSI Color Space

The intensity corresponds to relative brightness also in range [0,1] where 0 signifies black, and 1
signifies white.

HSV color space uses the same coordinate axes; however it is formed by projecting RGB unit
cube along diagonals of white to black in which a hexacone forms the top of HSV pyramid.

Figure 2-12 HSV Color Space

18
The significant thing for HSV color space is that at the very bottom of the pyramid, hue and
saturation has no significance and on the top; not all colors with the same brightness are visible. The
advantage of this type of color space is that it is highly intuitive especially for selecting colors or
neglecting some colors because it can be perceived the way human perceive colors. Both of HSV and
HSI relies directly to RGB therefore, conversion to and fro is highly accurate. However, both have
discontinuity at 00 making any arithmetic calculations difficult. Furthermore, no hue or saturation is
defined in achromatic colors which can produce inaccurate conversion when camera’s nonlinearity
characteristic is taken into account.

2.3.4 Color space selection

For this research, different color space will be used in different stages of color recognition. Since
CIELAB is the most accurate color space for finding similar colors and color distance is easy to
calculate, it is therefore best fitted for image segmentation procedures. This color space is used for
analyzing the surgical environment in pre-processing stage. Color matching is performed against RAL
colors, and the minimum distance is used to characterize pixel to certain RAL colors. Any RAL color
that does not fit any pixel color is chosen as marker’s color candidate. RGB classifiers are built using
this color candidate for real time computing. Even though CIELAB fits for segmentation procedures
in real-time but since it is computationally intensive, other methods are considered.

In real time, since speed is the most crucial factor; it is of great interest if no color space
conversion is done excessively saving the time for matching only. Therefore, RGB color space is used
with a twist. A feature model presented by Ohta, Kanade, Sakai in which the components are
defined by:

is used. The RGB classifiers are also converted in the same way and matching is done to find the
marker colors. Consideration for color recognition challenges will be discussed later.

19
2.5 Digital Color Image Formation

Understanding the color image formation is inevitable for the correct image processing. Additive
color model that applies to human eyes is also adapted to imaging sensors for image acquisition
process. Color images can be acquired by monochrome cameras using color filters, one chip CCD
color cameras or three-chip CCD color cameras.

One chip color cameras has its photosensitive cells covered with a color filter mask. Red,
green and blue filters are arranged into the form of mosaic filter. Image resolution in both vertical
and horizontal arrangement is halved therefore resulted in half number of pixels that the image
elements.

Figure 2-13 Color mask composition of one-chip CCD camera

This camera does not provide separate RGB signal and the sensitivity of the camera is
increased by the complementary sample. The major disadvantage is the transmission behavior of
filter masks is not always identical resulting in lesser color fidelity. Furthermore, more errors arise at
high frequency signals due to inefficient illumination of CCD elements at macro pixel stage.

Three chip color cameras on the other hand employ light beam splitting with semi
transparent prisms and steer the separate spectral interval toward the three locally separated CCD
chips. This provides a much higher quality of digital images and more homogeneous transmission
behavior. However the underside of this type of camera is the expensive price and it carries larger
dimension and greater weight.

20
 Figure 2-14 Light beam splitting in 3 chip CCD cameras

Therefore, the choice of color cameras is important for the color recognition in the sense of its
quality, and compatibility with the available image format.

2.4 Effect of Color Filter in Monochrome Camera

Multispectral imaging using color filters in front of the lens of a black and white camera generates
color images in which color mapping is controllable. This can be done by means of known spectral
transmission factors of color filters chosen for the application. Color filters either of gelatin or glass,
can be fabricated according to needs.

The interesting characteristic is that it is possible to produce color images with non-overlapping
spectral sensitivities which are impossible using color cameras. Therefore, distinctive colors with
large spectrum distance can easily be recognized in the scene.

However, the biggest drawback using color filters on a black and white camera is the prerequisite of
static scene (Andreas Koschan, 2008). Since the spectral transmissions are provided time-wise
successively, any sequence of movement or dynamic scenes will falsify the results. Therefore,
tracking procedures which is the integral part of this research is out of possibility if this method of
image acquisition is used.

21
2.5 Color-based Recognition Challenges

A sole color-based recognition has a very distinctive feature but suffer from color constancy and
illumination problem as well as sensitive to shadows and occlusions which are often addressed by
many in the research area.

2.6.1 Color constancy and Illumination Problem

Varying illumination propose a great risk for color-based recognition mainly because the target’s
color will appear inconsistent depending on the illumination. This means color information might
vary due to intensity change, chromaticity change or both. The usual solution presented is using
color constancy in which either correcting colors of the image or using illumination invariant color
features or descriptors (Mirgitta Martinkauppi, 2003). Theoretically, illumination problem can be
addressed by using chromaticity diagrams. However in practice, the chromaticity of the illumination
field over an object is non uniform and the difference between calibrated and real time illumination
chromaticity is significant.

To address this problem, Gevers and Smeulders (T.Gevers, 1997) presents color models, l1l2l3
that is invariant to highlight and illumination for both matte and shiny surfaces. This model performs
histogram matching for color recognition.

2.6.2 Sensitivity to Shadows and Occlusions

Color based recognition also suffers from these influences even though better than intensity images.
However, color clusters for the object must be extended to govern the shadows and highlights in
which the usual solution is using chromaticity diagrams solely throwing out the brightness
information. Enlarging the color clusters also introduces the risk of detecting other object of no
interest and therefore additional cleaning operation must be done to get rid of background
occlusion.

22
Chapter 3 Marker Choice and Design

3.1 Color Choice

Choosing the correct color for the marker is the crucial part of this research. Considering both
challenges presented in the previous chapter, the chosen color must survive the change of
illumination intensity and must be distinctive enough for the markers to be significantly different
from the surroundings.

3.1.1 Surgical Environment Color Analysis

Surgical rooms are usually dominated by white/neutral colors (wall, bed sheets, patients cover,
surgical gloves), green (surgical vest), red (bloodline), grey (surgical instruments) and numerous skin
color usually dominated from pearl white to brown spectrum; therefore other colors that are not in
this range is considered. The ambience i.e. the brightness of the surgical room is also considered as a
factor for choosing appropriate marker color. In this sense, color that is too saturated, too pale or
too luminous is avoided to reduce the effect of illumination.

Firstly, set of surgical room pictures are analyzed in MATLAB to get the color distribution. For
the color image segmentation procedures, appropriate color space must be determined for
optimum results. Initially it is done in HSV color space, in which intuitively the idea of distinct hues
can be clearly seen. However, since the categorization of hue ranges, saturation ranges and value
ranges is solely self-determined; the accuracy of this categorization is vague. Hence the search for
reliable, intuitive and perceptually standardized color categorization is done.

The main importance for this color categorization will be the ability to be reproduced and
availability in the mass market. Furthermore, it must be in relatively small set size for the
competitive matching computation. Therefore, the search narrows down to various commercial tints
available in the market. NBS-ISCC color system introduced by the US Inter-Society Color Council with
collaboration of National Bureau Standards for example narrows down 267 color centroids and color
identification is done based on Munsell System. Even though this color system is highly lucrative for
color identification, the availability of these colors in with the same name in Germany is highly
improbable since different countries and regions as well as companies adapt different color naming
strategy.

23
 Therefore, the search continues on different commercial color collections such as Duron
Color Collection, Dutch Paint Standard Design and Millennium Color Collection. However, since these
color collection is not readily available in the nearest stores the decision finally converges to RAL
colors in which the color companies in Germany used the terminology and color naming nationwide.

3.1.2 RAL Colors

RAL is the color matching system that is widely used for paints, varnish and powder coatings in
Germany as well as in Europe. It is developed by German’s Reichsausschuß für Lieferbedingungen
und Gütesicherung in 1927 to standardize the producible colors. It contains 213 different colors and
categorized in yellow, blue, violet, orange, grey, neutral, red and brown hues.

The reason behind choosing RAL colors is mainly because of its availability and small distinct
color size for comparison. Furthermore if the tints are unavailable in the surgical images trained, it
can be obtained without the hassle to find the correct tints in the market.

Figure 3-1 Example of RAL shades in violet category

The RAL color collection also provide the corresponding RGB and CIEL*a*b* values for each tints and
therefore easier to categorize. These colors are recorded manually in CIEL*a*b* for color distance
computation reference. This research utilize this readily available color samples for color matching
procedures during pre-processing in order to correctly identify suitable paint for the marker. Color
matching procedures are done using minimum distance calculation pixel-by-pixel.

24
3.1.1 Color Selection for Markers

This analysis is done prior to real-time recognition procedures. Color images acquired from
color camera are first converted pixel-wise into CIELAB color space. The 213 distinctive RAL colors
are also converted into CIELAB color space with 2o observant and base fluorescent light constants.
Distance measurements using Euclidean distance in surgical rooms images are done against the set
of RAL colors. Any RAL color that does not match with the acquired image pixels is considered as
marker’s color candidate.

In comparison with 23 different surgical room environments, color details of all images are
simultaneously compared with RAL colors. The result of this computation yields 13 distinct RAL
colors that are nowhere to be found in all those images. These colors are RAL1004 Golden Yellow,
RAL 1003 Signal Yellow, RAL 1028 Melon Yellow, RAL 2003 Pastel Orange, RAL 2004 Pure Orange,
RAL 2008 Bright Red Orange, RAL2011 Deep Orange, RAL 3020 Traffic Red, RAL 3026 Luminous
Bright Red and RAL 4005 Blue Lilac. Categorizing these colors according to their hues, they belong to
yellow, orange, red and violet hues. In this research, the goal is identifying at least three different
colors for instant marker identity during operation. Even though there are 13 different RAL colors,
only 4 different hues are present. Taking into account the illumination problem that will definitely
occur during operation, red hues are the least possible candidates because of a probable
interference with different red hues especially bloodline, disclosed area during operation or blood
bag.

Looking at the color description itself, the colors are mostly luminous and fluorescent. To
avoid getting confused with the same hues in different illumination, marker colors are chosen from
all different hues. Since availability of these colors are limited in small containers, RAL1004 Golden
Yellow, RAL2004 Pure Orange and RAL 4005 Blue Lilac are chosen due to their availability in nearest
stores.

These colors are painted on the markers and the color distribution for the distance of
markers from camera in [30, 50, 100, 150, and 200] cm sets are obtained. From these images, new
RGB classifiers are built using Hill Climbing segmentation to get color variation of the markers due to
the effect of distance and illumination. These classifiers are then used for reference in real-time
recognition.

25
3.2 Shape and Size Selection
In determining the markers size and shape, there are a lot of recognition factors that must be taken
into account. The fact that this research is for real time medical application signifies that speed and
accuracy is the most important criteria. Therefore, markers must be design to fit these criteria and
also easily producible or available in current market since they must be available during
experiments.

3.2.1 Shape-based Recognition Challenges

For fast computation, it is important to consider the least complicated geometry for the markers.

3.2.2 Optimum Shape Recognition Factor

3.2.3 Marker Individual Arrangement

3.2.4 Size Factor for Recognition Ability

4 Recognition Strategy and Implementation

4.1 Recognition Problems Revisited

4.2 Recognition Steps

4.3 Position and Orientation Computation

4.4 Real-time Compatibility

26
5 Experiments and Results

Future Work

Conclusion

27
APPENDIX

Program codes

28
References
Andreas Koschan, M. A. (2008). Digital Color Image Processing. John Wiley & Sons, Inc.

Christian Teutsch, D. B. (2006). Real-time detection of elliptic shapes for automated object
recognition and object tracking. IEEE.

Cruces, R. A. (2008). Concept and Design of a Cooperative Robotic Assistant Surgery System. PhD
Thesis, University Of Siegen, ZESS, Siegen.

D. Sykora, D. S. (2008). Real Time Color Ball Tracking for Augmented Reality. EGVE Symposium. The
Biographics Association.

D'Zmura, M. (1991). COlor COnstancy: surface color from changing illuminations. California, USA:
Josa Communications.

Formwalt, B. P. (2000). Real Time Color Vision Processing. Ohio: Air Force Institute of Technology.

J.M Chassery, C. G. (1984, November). AN Iterative Segmentation Method Based on a Contextual

Color and Shape Criterion. IEEE Transactions on Pattern Analysis and Machine Intelligence , pp. 794-
800.

Mark S. Drew, J. W.-N. (1998). Illumination Invariant Color Object Recognition via Compressed
Chromaticity Histograms of Color Channel Normalized Images. Sixth International Conference on
Computer Vision (pp. 533-540). IEEE.

Mirgitta Martinkauppi, M. S. (2003). Detection of Skin Colour under Changing Illumination: A

Comparative Study. 12th International Conference on Image Analysis and Processing (ICIAP). IEEE.

T.Gevers, A. S. (1997). Color Based Object Recognition. In Misc., Lecture Notes in Computer Science
(pp. 319-326). Heidelberg: Springer Berlin / Heidelberg.

Tinku Acharya, A. K. (2005). Image Processing: Principles and Applications. John Wiley & Sons.

Y-I Ohta, T. K. (1980). Color Information for Region Segmentation. Computer Graphics and Image
Processing 13 , 222-241.

29
<frontpage>
Abstract

xxx
Table of Contents

Abstract ...................................................................................................................................... ii
Table of Contents ........................................................................................................................ iii
List of Figures ............................................................................................................................. v
List of Tables.............................................................................................................................. vi
Chapter 1 Introduction .......................................................................................................... 1
1.1 Motivation............................................................................................................................... 1
1.2 Problem Statement ................................................................................................................. 3
1.3 Previous work.......................................................................................................................... 5
1.4 Assumptions ............................................................................................................................ 8
1.5 Scope ....................................................................................................................................... 8
1.6 Approach ................................................................................................................................. 9
1.7 Material and Equipment ......................................................................................................... 9
1.8 Structure of Thesis .................................................................................................................. 9
Chapter 2 Color Theory and Challenges ................................................................................ 10
2.1 Visible color spectrum........................................................................................................... 10
2.2 Color Dimension.................................................................................................................... 12
2.3 Color Space and Significance ................................................................................................ 14
2.3.1 Physics and technic based color space ......................................................................... 14
2.3.2 Uniform Color Space ..................................................................................................... 16
2.3.3 Perception-based Color Space ...................................................................................... 18
2.3.4 Color space selection .................................................................................................... 19
2.5 Digital Color Image Formation ................................................................................................... 20
2.4 Effect of Color Filter in Monochrome Camera ...................................................................... 21
2.5 Color-based Recognition Challenges .................................................................................... 22
2.6.1 Color constancy and Illumination Problem .......................................................................... 22
2.6.2 Sensitivity to Shadows and Occlusions ................................................................................ 22
Chapter 3 Marker Choice and Design.................................................................................... 23
3.1 Color Choice .......................................................................................................................... 23
3.1.1 Surgical Environment Color Analysis.................................................................................... 23
3.1.2 RAL Colors ............................................................................................................................ 24

xxxi
 3.1.1 Color Selection for Markers .............................................................................................. 25
3.2 Shape and Size Selection....................................................................................................... 26
3.2.1 Shape-based Recognition Challenges ........................................................................... 26
3.2.2 Optimum Shape Recognition Factor ................................................................................. 26
3.2.3 Marker Individual Arrangement ....................................................................................... 26
3.2.4 Size Factor for Recognition Ability .......................................................................................... 26
4 Recognition Strategy and Implementation .......................................................................... 26
4.1 Recognition Problems Revisited ........................................................................................... 26
4.2 Recognition Steps.................................................................................................................. 26
4.3 Position and Orientation Computation................................................................................. 26
4.4 Real-time Compatibility ........................................................................................................ 26
5 Experiments and Results ..................................................................................................... 27
Future Work .............................................................................................................................. 27
Conclusion ................................................................................................................................. 27
APPENDIX .................................................................................................................................. 28
Program codes ................................................................................................................................. 28
References................................................................................................................................. 29

xxxii
List of Figures
Figure 1-1 modiCAS system in operating room ...................................................................................... 1

Figure 1-2 Components of the navigated robotic system for surgical assistance .................................. 2

Figure 2-1 The electromagnetic and sound spectra ............................................................................. 11

Figure 2-2 Color hue.............................................................................................................................. 13

Figure 2-3 Reflection of value through shade, tone and tint................................................................ 13

Figure 2-4 Reflection of purity of colors (chroma)................................................................................ 13

Figure 2-5 RGB Color Space (Andreas Koschan, 2008) ......................................................................... 15

Figure 2-6 RGB chromaticity diagram (Maxwell Color Triangle)........................................................... 15

Figure 2-7 Standard primaries' wavelengths and spectral power ........................................................ 16

Figure 2-8 CIE Chromaticity Diagram .................................................................................................... 17

Figure 2-9 HSI Color Space .................................................................................................................... 18

Figure 2-10 HSV Color Space ................................................................................................................. 18

Figure 3-1 Example of RAL shades in violet category ........................................................................... 24

xxxiii
List of Tables
No table of figures entries found.

xxxiv
Chapter 1 Introduction

This chapter will commence the reason behind establishing this research, the related works that
previously done for the marker recognition, furthermore defining the problems, specifying the scope
and approach, and consequently summarize what will be dissected in the following chapters.

1.1 Motivation

Advancement in mechatronics products and applications nowadays are in amazing pace, it’s a
definite loss not to extend the expertise to the medical field for the betterment of humanity.
Considering this fact, a research team in the Institute for Sensor Systems (abbr. ZESS) University of
Siegen has put its focus on developing integral solution for assisting the medical personnel in
enhancing the conventional medical procedures or complicated surgical procedures.

As a result, modiCAS (Cruces, 2008) (modular interactive Computer Assisted Surgery) system has
been successfully developed in which the benefit from this system is experienced firsthand by the
surgeons for the world’s first robotic system to support the implementation of the acetabular cup.

Figure 1-1 modiCAS system in operating room

1
This modular system represents an integral solution for computer and robot assisted surgery, in
which it offers continuous support from preoperative planning to robot-assisted surgery. Since the
overall structure of the system is modular, modiCAS is adaptable for various surgical disciplines given
that additional modules are added to the existing basic system. This system can be viewed as a
smart surgical tool that extends surgeons ability to treat patients by working in cooperative fashion.

The basic concept of the modiCAS project is integrating the navigation system and robotic arm into a
system that appears a single unit in which it combines both the advantages of navigation and
surgical robotics. Patient registration is performed using the navigation system in which a 3D
digitizing device is used to register the patient’s anatomy, while robotic arm is applied as
mechatronics extension of the navigation system for guiding and positioning the surgical
instruments. Therefore modiCAS is comprised of three aspects for it to be functional; preoperative
planning, intra-operative navigation and mechatronics assistance.

Figure 1-2 Components of the navigated robotic system for surgical assistance

Computer assisted preoperative planning is done based on X-ray, CT-scans, MRI-images or other
image formats in which surgeons can produce a workflow for the operations using functions
provided by the software. The intra-operative patient registration and navigation on the other hand
is used in combination with mechatronics assistance. Patient’s anatomy are registered and matched

2
with pre-operative image data, and this extends to visualize the position of the instruments in pre-
operative planning images. The modiCAS mechatronics assistance enables the surgeon to
interactively control the robotic arm in which the surgeons have the full authority to interrupt or
move the arm during the surgery. The mechatronics assistance can also track patient movements in
real time. This means the alignment between surgical instruments and the target remains constant
at all time. This special feature of real-time tracking is made possible using the infrared markers
which are mounted on several places visible by the camera to locate the real-time position and
orientation of the target. While this method has proven accuracy and fit for the surgical operation,
an extended research is done to further enhance the tracking capability in which time and accuracy
are the crucial factors.

1.2 Problem Statement

Research in color based marker recognition originates from the reality check of previously engaged
markers for modiCAS in surgical environment. While the excellence and quality of recognition are
inevitable, the cost of producing such an infrared sensitive marker is skyrocketing. More importantly
these infrared beacons, while provide the position and orientation of the target; relies on separate
arrangement computation to specify their identity. For example, during operation the target
position and orientation can be computed using at least 3 different markers and the identity of these
markers can only be known by the further arrangement computation instead of by first- glance
recognition.

Therefore, this research is headed to produce the low cost, easily recognized and computationally
inexpensive markers which lead to color based recognition. Digital color image produces more than
16 million distinctive colors which is by default the amount that can be produced by CRT color
monitors considering the spectral transmission of visible light. Since color based recognition has so
many possibilities and larger space to accommodate various combination of object recognition
strategies, it is therefore decided to proceed with this method for a greater quality of marker
recognition rather than grey level images that often clouded or restrict the object recognition
quality.

The process of discovering color based markers begins with deciding shape, size and color
appropriate with the surgical environment. Then, the distance between the camera and marker is
also taken into account. Since the markers are considered to be located as far as 2 meters from the

3
stereo cameras, the appropriate size is tested and determined for best recognition ability.
Furthermore, the possible shape with the easiest recognition is considered since it has to be
computationally inexpensive for it to work in real time.

The next step is choosing the appropriate image processing technique for the markers. Since in color
image processing technique can be divided into two classes which are the monochromatic-based
techniques and vector valued techniques; the direction for this decision also arrived at two different
possibilities.

Monochromatic-based techniques treat information from individual color channels then combine
the individual result or in other words disregard any correlation between color channels and treat
them independently. Depending on the color space chosen, the processing will disregard either one
or the other two channels completely in some of the phase of the recognition and finally
recombined to produce better results.

Vector valued techniques on the other hand treat color information as a whole i.e. as color vectors
in vector space provided with vector norm. While being computationally expensive in the past, the
current technology allows greater computational ability and therefore retains the color
interdependence which produces more accurate results.

Therefore, it is considered as possibility to use a monochrome camera (black and white) and place
suitable color filters to enhance marker’s color. The object recognition techniques using this type of
image processing usually borrowed from grey level image processing and therefore faster and
simpler.

The other possibility is to use a color camera for image acquisition and use either one or two color
channels simultaneously to obtain acceptable recognition.

The next step is the position computation. Considering the ability of stereo cameras to be able to
compute accurate position results with lower nuisance and occlusion problem, the advantage is
taken into account to calculate the relative marker position in regard to the stereo cameras.

The final step is the tracking ability of the markers. Since it is the most important criterion for the
marker to be recognized in real time, the recognition strategy is developed so that it would be fast
enough before the next frame acquisition and initial processing happens.

Therefore this research will reveal the innovative technique used for the aforementioned steps to be
realized successfully.

4
1.3 Previous work

The feasibility study begins with three aspects in mind; real time color-based recognition,
illumination invariant methodology and fast feature extraction procedures. Since real time
procedures are inevitable for this research to have a strong ground, any aspect that was not
performed in real-time is disregarded.

Real-time color based recognition is successfully done in real time for augmented reality
markers (D. Sykora, 2008). This research offers robustness to partial occlusions, and illumination
problem. It proves using relatively normal PC workstation with running resolution of 640x480, the
detection of 8 markers of 4 different colors and spherical in shape is 11ms.

Figure 1-3 Real-time recognition sequence using RGB classifier until refinement of circular shape

However, the drawback of this research is the accuracy of 1mm only holds for non-occluded
daylight illumination with diameter of 7 cm at the distance of 1m from camera. This shows that the
target has relatively large diameters in image space; hence the object recognition is much easier.
Even though the shape of the markers is considered easy for recognition procedures, the size is
deemed inappropriate for the scope of this research since medical tracking markers cannot be
irrelevantly large because multiple markers are needed at the scene. However, the pre-processing
procedures of color segmentation for color classifiers are adapted into this research for the tested
accuracy and appropriateness.

Figure 1-4 Mean shift segmentation mapped into chromaticity diagram of CIE L*u*v* color space

5
 Fromwalt in his research for Real Time Color Vision Processing (Formwalt, 2000) proposed a
real-time solution for a stereovision system. Feature clustering is done using histogram zooming
method in which two dimensional histogram are taken of the contrast feature points in each image
bin. Mean geometric centers calculated for each of selected bins. All contrast feature points are used
to zoom in particular object of interest.

Figure 1-5 Histogram zooming results on SUV target

Feature group correspondence is accomplished by color cross optimizing the epipolar

disparity of each histogram window. All frame images are of 160x120, however with 4 frames per
second. The accuracy is within 1 pixel resolution with consistent overhead of 20 ms per frame pair.
However, as can be seen from the experimental setup an approximation of 3 frames per second is
inappropriate for this scope of research even though the histogram zooming method is attractive.
On illumination problem, Gevers and Smeulders (T.Gevers, 1997) presents color models that
are tested invariant to viewing direction, object geometry, shading, highlights and change in
illumination color. They propose a color model that is directly derived from RGB values which are
the set of normalized color difference in which they claimed well suited even for shiny surfaces.

Figure 1-6 The l1l2l3 color model

Since it is not stated in their research whether this will work on real time, the proposed algorithm
will be tested further in this experiment due to the acclaimed good results to counter illumination
problem.

6
 Other works in this field includes using compressed chromaticity histograms of color-
channel-normalized image (Mark S. Drew, 1998) in which seminal methods of Swain and Ballard is
extended to discount changes in illumination. However, the processing is computationally intensive
for real-time even though the results are excellent. Bilinear models (D'Zmura, 1991) is also
introduced in which a particular combination of photoreceptors and models for reflectance and
illuminants is used. However, since this research uses three photoreceptors that originates from
initially known color filters the illumination procedures cannot be adapted into the current scope of
research. Illumination invariant color detection schemes also proposed using statistical method as in
(Mirgitta Martinkauppi, 2003). Even though it seems to work for small illumination chromaticity
changes, the broader range did not work.

The previous work shape understanding prevails some interesting aspect for this
research. Circular and elliptic object tracking in real time which proves to be the easiest recognition
procedures met is also presented by the team of researchers from Magdeburg, Germany (Christian
Teutsch, 2006). This research focuses more on easier shape recognition even though color images
are used for experiments. Therefore, instead of just relying on the color, the shape descriptors
combine both information for object recognition. Even though the 3D tracking prove great results,
the target almost have no background clutters and dominates the input image regions which is
therefore irrelevant for current research.

Figure 1-7 Contours descriptors result. Circular shape has significantly small values.

An iterative segmentation method for combining both shape and color descriptors (J.M
Chassery, 1984) also proves good result in which color similarity is done first using log-transformed
images and then followed by convex hull and convexity criterion. The experiment is done on cell-
shape images which in contrast to the research scope, target dominates the input images. However,
since the methodology is quite simple and might work in real time, the procedures will also be
tested during experimental procedures for this research.

7
1.4 Assumptions

This research assumed that the original setup of the modiCAS system is retained. The surgical
environment is the main scene with fluorescent light illumination. The spectral power distribution of
the white illumination fluorescent light is assumed to be constant at all times. All markers have the
consistent shape through all video frames; their corresponding colors are constant in visible areas
within the same illumination. The objects of interest (markers) are assumed undistorted in front of
cameras, and color sensitivity is assumed identical for both cameras, in which no specular variation
exist between left and right frames.

1.5 Scope

The scope of this research is to develop color markers that have:

i) High discriminating power, in this case the color combination of 3 different markers
made it possible to distinguish them without any arrangement computation
ii) Robustness against object occlusion and cluttering, in this case the most feasible size
and shape for the markers to be recognized within 2m range
iii) Robustness against changes in direction, intensity and spectral power distribution of the
illumination, which translates to color constancy in all viewing directions and locations.
iv) Minimum computation complexity in which the accuracy is within acceptable range and
most importantly fit for real time computing (at least 24 frames/sec).
v) Color characteristics that has commercial availability and producible.

This research will not span into merging the current result with previous modiCAS system and only
serve for justification purposes whether the color marker is adequate for the recognition in surgical
environment.

8
1.6 Approach

This research has taken an approach of studying the possibility of using colored markers and tested
them against the crucial factors, time and accuracy. Therefore, the recognition algorithm is mainly
tested in MATLAB using still pictures and video streams for the ability to quickly justify the results.
Once the result reaches the appropriate accuracy, the algorithm then programmed into a C++
programming to reduce the time consumption for the recognition. The experiment will justify
whether color markers are suitable to replace the current infrared markers with acceptable accuracy
and processing time.

1.7 Material and Equipment

Markers are chosen to be in spherical shape, made of wood and mounted on ice cream wooden stick
to provide suitable beacons for position and orientation determination.

Stereo cameras are used to provide a video streaming input, and calibration procedures are
executed to provide intrinsic parameters useful for unwrapping target position in real time.

1.8 Structure of Thesis

In the next chapter, Chapter 2, the foundation of colors as well as representation of colors in
different color space is described. Furthermore, the reason behind the choice of certain colors and
color space representation is debated and disclosed. In Chapter 3, the marker development process
is discussed. The choice of marker’s shape and size is explained and how the final choice is
determined. In Chapter 4, recognition strategy as well as color identification with illumination
invariant properties is described. The new algorithm is explained together with flowchart to
understand how the color marker can be recognized. In Chapter 5, the results from experiments are
presented in graphical and tabular form and explanations are provided to explain the success rate
and failures. Lastly, conclusion and summary of general issues are discussed in the final chapter.
Future work is also presented.

9
Chapter 2 Color Theory and Challenges

Color-based recognition is relatively new to the field of real time object recognition. It has not
received much attention in the past mainly because of large processing time and the incapability of
computers to process relatively large amount of data and apply recognition strategies
simultaneously. In order to study the advantage of color object recognition, it is of great importance
to dig the very basic understanding of color and finally how it improves recognition ability. This
chapter will begin with the foundation of colors and subsequently highlight the importance of
choosing the right color space for the image processing; and how color distance provides means to
substantiate the different object’s presence in the acquired image.

2.1 Visible color spectrum

Colors that we can see with our eyes originate from colorless light of the sunlight. The light reflected
from an object is absorbed by the cone cells of human visual system hence leads to color perception
(Tinku Acharya, 2005). The color of the object is the result of the ability of the surface of the object
to absorb the other color in the spectrum and reflect a particular color (i.e. light of certain
wavelength) to give its color identity. An apple for example absorbs all wavelengths of incident light
other than the wavelengths of red spectrum (i.e. approximately 770nm) and then reflects it to
human eyes to give a red color identity of the apple. Considering there are three cone classes in
human vision system to perceive color, the different amount of absorptions by these cone cells
determines the color of the incident light.

Digital color images also adapt the same way human eyes perceive colors in which additive
model is applied. The differing color stimuli are combined into one color through projective
overlapping. In the case of color images, once the light of different wavelengths hit the image
sensor; different arrangement of red, green and blue filters masked on the sensor element will
produce distinctive colors of the scene at hand.

10
 Figure 2-1 The electromagnetic and sound spectra

However, contrary to popular belief, color is a perceived phenomenon. It is non-measurable

and not a physical dimension even though the electromagnetic radiation of the visible wavelength
spectrum that enumerates the color is a measurable quantity. Therefore, color recognition has
undergone a broad research due to this fact. Being perceptual in nature, the same color might be
seen differently by different observers. Therefore, standardization has been employed in color
representation and models to categorize colors according to their dimensions. It is important for this
scope of research since the goal is to produce appropriate marker colors for recognition.

Color models that are usually discussed are in the form of additive and subtractive models.
Human eyes and imaging device adapted to additive models in which primary colors are used
additively to provide other colors. Color printing, color dyes and paints on the other hand use
subtractive primary colors mixing in which cyan, magenta, and yellow as the primary. CYMK model
that is used for printing uses these complementary primary colors with a combination of different

11
amounts of cyan that will control how much red is shown; magenta which will control how much
green will be shown and yellow to show how much blue is produced.

Figure 2-2Additive color mixing

Figure 2-3 Subtractive color mixing

With these two different models explained, the color of painted object produced by the
manufacturers undergoes different procedures than the way color is displayed on a monitor.
Therefore, careful consideration for choosing marker colors is important to reduce probability of
choosing different colors than that of displayed by the computer monitors.

2.2 Color Dimension

Color dimension defines how colors are categorized and therefore limit the generalities of defining
colors. Scientifically, there are three dimensions of color namely hue, value and chroma. Hue
identifies color by name. Every color falls into definite hue as indicated by the spectrum color name
on the color wheel. Value determines the lightness or darkness of a hue achieved by adding white,
gray or black. White added to a hue creates tint, gray addition results tone and black addition
creates shade. Chroma on the other hand refers to the degree of intensity, strength, saturation or
purity of a color. Purity of a color refers to its freedom from white, black or grey.

12
 Figure 2-4 Color hue

Figure 2-5 Reflection of value through shade, tone and tint

Figure 2-6 Reflection of purity of colors (chroma)

Since color is perceptual, it is therefore quite difficult for color naming or categorization to be done
empirically without any standards or collectable research. Hence, many organizations produce their
own color collection for easier reference especially those that involve primarily in printing or paint
production.

13
2.3 Color Space and Significance

In order to transfer the color information for the computer to understand, color images must
have a form of representation to be stored, displayed and processed. Either for the sake of
mathematical demands, processing algorithms, technical conditions of hardware; even though the
fact that these demands cannot be met simultaneously, a color representation is crucial to correctly
process them. In other words, color spaces provide representation of sensor outputs for easier
computation as well as accommodate the way human perceive colors. Therefore, the birth of
different color spaces which can be categorized largely by physics and technic-based color space,
uniform color space and perception-based color space is gratefully welcomed.

Color space is the color coordinate system by which the image values of color image are
represented. The difference between two image values in a color space is called color distance. As
explained earlier, since there are different type of color space is available, color distance
computation also differ from one type of color space to another.

The most important characteristics of the color space that is advantageous for color
recognition is the chromaticity diagram. It represents color information that survives the elimination
of brightness information. In other words, color recognition can be done in two dimensional
relieving the stress on extended computation. The chromaticity diagram differs depending on the
color space and will be explain accordingly later.

2.3.1 Physics and technic based color space

This category is defined as the color spaces that have a direct technical reference. The most
well known color space, RGB belongs to this category. RGB color space is based on additive mixture
of three primary colors, Red, Green and Blue hence the name even though it is actually perceptual
colors that resulted from a standardized long, middle and short wavelengths of visible light.

14
 Figure 2-7 RGB Color Space (Andreas Koschan, 2008)

RGB color space is characterized as a cube in which it is distinctive integer-wise in the range of [0-
255] for every axis (for 24-bit-8 bit/channel). Chromaticity of RGB color space lies on a two
dimensional triangle of R+G+B=1 and the achromatic colors (gray tones) lies on principle diagonal
R=G=B. Distinctive colors are calculated by accounting this space and Euclidean space in which
Euclidean distance is used to measure the color distance.

Figure 2-8 RGB chromaticity diagram (Maxwell Color Triangle)

The significant advantage of using this color space for image processing is the fact that
imaging hardware sends the signal in this color space values, therefore eliminating the needs of any
further conversion for the signal to be processed. It is important whenever the application is speed-
oriented such as real-time application. However, the color distance of RGB space does not reflect the
same perceptual changes of colors human sees. It means, two different colors at a distance, d apart

15
in one part of the color space does not imply the same degree of perceptual difference as two other
colors the same distance apart in another part of the color space. Color features recognition using
color distance is therefore relatively inaccurate using RGB color space.

Other types of color spaces in this category are of less significance to this research because they
belong to printing colors (CYMK), color television systems (YUV, YIQ) and digital videos (YCBCR). As
self-explanatory color spaces, these are application specific and belongs outside the scope of this
research.

2.3.2 Uniform Color Space

This category signifies color spaces that have the same size changes in color coordinates as
the change in visible color tones and saturation. This means that the Euclidean distance between
two color points in any part of the color space corresponds to the perceptual difference between the
two colors by the human vision system. Two color spaces CIELAB and CIELUV, in which both are
commenced by the International Lighting Commission, CIE (Commission Internationale de
l’Eclairage) belong to this category. These color spaces are derived from XYZ standard color system.
CIE has set a standard using long, middle and short wavelength of certain spectral powers for
primary values.

Figure 2-9 Standard primaries' wavelengths and spectral power

XYZ is virtual primary values defined by CIE using the conversion of real spectral value
curves. It is a linear transformation as explained below:

16
CIELAB was developed in accordance to Munsell color order system in which colors are described
using attributes hue, lightness and chroma. It is characterized by (a*,b*) as chromaticity axis and L*
as lightness. L* is scaled approximately equidistant according to sensitivity.

Figure 2-10 CIE Chromaticity Diagram

CIELUV shares the same L* information of CIELAB but differs for the chromaticity (u*,v*). In contrary
with CIELAB, the chromaticity points are mapped the same way as in standard color tables therefore
is advantageous for additive mixture calculation. The main advantage using this type of color space is
the fact that it is uniform in which it can be treated as Euclidean space with much less errors
compared to RGB. Euclidean distance is therefore accurate to signify the distinctive colors. Hence, it
is great for comparing very similar colors.

CIELAB color space is normally used for describing body colors; CIELUV on the other hand is used for
the description of colors of light. The downside of this color space is the computational times. This is
a significant drawback for video real time processing.

17
2.3.3 Perception-based Color Space

Color spaces that belong to this category are based intuitively on human color perception.
HSI and HSV fit in this category. HSI uses hue, saturation and intensity as coordinate axes. Red is
specified as reference color in which hue at 0o and 360o correspond to this color. Saturation is in the
range of [0,1] in which 1 refers to pure color and 0 refers to achromatic colors.

Figure 2-11 HSI Color Space

The intensity corresponds to relative brightness also in range [0,1] where 0 signifies black, and 1
signifies white.

HSV color space uses the same coordinate axes; however it is formed by projecting RGB unit
cube along diagonals of white to black in which a hexacone forms the top of HSV pyramid.

Figure 2-12 HSV Color Space

18
The significant thing for HSV color space is that at the very bottom of the pyramid, hue and
saturation has no significance and on the top; not all colors with the same brightness are visible. The
advantage of this type of color space is that it is highly intuitive especially for selecting colors or
neglecting some colors because it can be perceived the way human perceive colors. Both of HSV and
HSI relies directly to RGB therefore, conversion to and fro is highly accurate. However, both have
discontinuity at 00 making any arithmetic calculations difficult. Furthermore, no hue or saturation is
defined in achromatic colors which can produce inaccurate conversion when camera’s nonlinearity
characteristic is taken into account.

2.3.4 Color space selection

For this research, different color space will be used in different stages of color recognition. Since
CIELAB is the most accurate color space for finding similar colors and color distance is easy to
calculate, it is therefore best fitted for image segmentation procedures. This color space is used for
analyzing the surgical environment in pre-processing stage. Color matching is performed against RAL
colors, and the minimum distance is used to characterize pixel to certain RAL colors. Any RAL color
that does not fit any pixel color is chosen as marker’s color candidate. RGB classifiers are built using
this color candidate for real time computing. Even though CIELAB fits for segmentation procedures
in real-time but since it is computationally intensive, other methods are considered.

In real time, since speed is the most crucial factor; it is of great interest if no color space
conversion is done excessively saving the time for matching only. Therefore, RGB color space is used
with a twist. A feature model presented by Ohta, Kanade, Sakai in which the components are
defined by:

is used. The RGB classifiers are also converted in the same way and matching is done to find the
marker colors. Consideration for color recognition challenges will be discussed later.

19
2.5 Digital Color Image Formation

Understanding the color image formation is inevitable for the correct image processing. Additive
color model that applies to human eyes is also adapted to imaging sensors for image acquisition
process. Color images can be acquired by monochrome cameras using color filters, one chip CCD
color cameras or three-chip CCD color cameras.

One chip color cameras has its photosensitive cells covered with a color filter mask. Red,
green and blue filters are arranged into the form of mosaic filter. Image resolution in both vertical
and horizontal arrangement is halved therefore resulted in half number of pixels that the image
elements.

Figure 2-13 Color mask composition of one-chip CCD camera

This camera does not provide separate RGB signal and the sensitivity of the camera is
increased by the complementary sample. The major disadvantage is the transmission behavior of
filter masks is not always identical resulting in lesser color fidelity. Furthermore, more errors arise at
high frequency signals due to inefficient illumination of CCD elements at macro pixel stage.

Three chip color cameras on the other hand employ light beam splitting with semi
transparent prisms and steer the separate spectral interval toward the three locally separated CCD
chips. This provides a much higher quality of digital images and more homogeneous transmission
behavior. However the underside of this type of camera is the expensive price and it carries larger
dimension and greater weight.

20
 Figure 2-14 Light beam splitting in 3 chip CCD cameras

Therefore, the choice of color cameras is important for the color recognition in the sense of its
quality, and compatibility with the available image format.

2.4 Effect of Color Filter in Monochrome Camera

Multispectral imaging using color filters in front of the lens of a black and white camera generates
color images in which color mapping is controllable. This can be done by means of known spectral
transmission factors of color filters chosen for the application. Color filters either of gelatin or glass,
can be fabricated according to needs.

The interesting characteristic is that it is possible to produce color images with non-overlapping
spectral sensitivities which are impossible using color cameras. Therefore, distinctive colors with
large spectrum distance can easily be recognized in the scene.

However, the biggest drawback using color filters on a black and white camera is the prerequisite of
static scene (Andreas Koschan, 2008). Since the spectral transmissions are provided time-wise
successively, any sequence of movement or dynamic scenes will falsify the results. Therefore,
tracking procedures which is the integral part of this research is out of possibility if this method of
image acquisition is used.

21
2.5 Color-based Recognition Challenges

A sole color-based recognition has a very distinctive feature but suffer from color constancy and
illumination problem as well as sensitive to shadows and occlusions which are often addressed by
many in the research area.

2.6.1 Color constancy and Illumination Problem

Varying illumination propose a great risk for color-based recognition mainly because the target’s
color will appear inconsistent depending on the illumination. This means color information might
vary due to intensity change, chromaticity change or both. The usual solution presented is using
color constancy in which either correcting colors of the image or using illumination invariant color
features or descriptors (Mirgitta Martinkauppi, 2003). Theoretically, illumination problem can be
addressed by using chromaticity diagrams. However in practice, the chromaticity of the illumination
field over an object is non uniform and the difference between calibrated and real time illumination
chromaticity is significant.

To address this problem, Gevers and Smeulders (T.Gevers, 1997) presents color models, l1l2l3
that is invariant to highlight and illumination for both matte and shiny surfaces. This model performs
histogram matching for color recognition.

2.6.2 Sensitivity to Shadows and Occlusions

Color based recognition also suffers from these influences even though better than intensity images.
However, color clusters for the object must be extended to govern the shadows and highlights in
which the usual solution is using chromaticity diagrams solely throwing out the brightness
information. Enlarging the color clusters also introduces the risk of detecting other object of no
interest and therefore additional cleaning operation must be done to get rid of background
occlusion.

22
Chapter 3 Marker Choice and Design

3.1 Color Choice

Choosing the correct color for the marker is the crucial part of this research. Considering both
challenges presented in the previous chapter, the chosen color must survive the change of
illumination intensity and must be distinctive enough for the markers to be significantly different
from the surroundings.

3.1.1 Surgical Environment Color Analysis

Surgical rooms are usually dominated by white/neutral colors (wall, bed sheets, patients cover,
surgical gloves), green (surgical vest), red (bloodline), grey (surgical instruments) and numerous skin
color usually dominated from pearl white to brown spectrum; therefore other colors that are not in
this range is considered. The ambience i.e. the brightness of the surgical room is also considered as a
factor for choosing appropriate marker color. In this sense, color that is too saturated, too pale or
too luminous is avoided to reduce the effect of illumination.

Firstly, set of surgical room pictures are analyzed in MATLAB to get the color distribution. For
the color image segmentation procedures, appropriate color space must be determined for
optimum results. Initially it is done in HSV color space, in which intuitively the idea of distinct hues
can be clearly seen. However, since the categorization of hue ranges, saturation ranges and value
ranges is solely self-determined; the accuracy of this categorization is vague. Hence the search for
reliable, intuitive and perceptually standardized color categorization is done.

The main importance for this color categorization will be the ability to be reproduced and
availability in the mass market. Furthermore, it must be in relatively small set size for the
competitive matching computation. Therefore, the search narrows down to various commercial tints
available in the market. NBS-ISCC color system introduced by the US Inter-Society Color Council with
collaboration of National Bureau Standards for example narrows down 267 color centroids and color
identification is done based on Munsell System. Even though this color system is highly lucrative for
color identification, the availability of these colors in with the same name in Germany is highly
improbable since different countries and regions as well as companies adapt different color naming
strategy.

23
 Therefore, the search continues on different commercial color collections such as Duron
Color Collection, Dutch Paint Standard Design and Millennium Color Collection. However, since these
color collection is not readily available in the nearest stores the decision finally converges to RAL
colors in which the color companies in Germany used the terminology and color naming nationwide.

3.1.2 RAL Colors

RAL is the color matching system that is widely used for paints, varnish and powder coatings in
Germany as well as in Europe. It is developed by German’s Reichsausschuß für Lieferbedingungen
und Gütesicherung in 1927 to standardize the producible colors. It contains 213 different colors and
categorized in yellow, blue, violet, orange, grey, neutral, red and brown hues.

The reason behind choosing RAL colors is mainly because of its availability and small distinct
color size for comparison. Furthermore if the tints are unavailable in the surgical images trained, it
can be obtained without the hassle to find the correct tints in the market.

Figure 3-1 Example of RAL shades in violet category

The RAL color collection also provide the corresponding RGB and CIEL*a*b* values for each tints and
therefore easier to categorize. These colors are recorded manually in CIEL*a*b* for color distance
computation reference. This research utilize this readily available color samples for color matching
procedures during pre-processing in order to correctly identify suitable paint for the marker. Color
matching procedures are done using minimum distance calculation pixel-by-pixel.

24
3.1.1 Color Selection for Markers

This analysis is done prior to real-time recognition procedures. Color images acquired from
color camera are first converted pixel-wise into CIELAB color space. The 213 distinctive RAL colors
are also converted into CIELAB color space with 2o observant and base fluorescent light constants.
Distance measurements using Euclidean distance in surgical rooms images are done against the set
of RAL colors. Any RAL color that does not match with the acquired image pixels is considered as
marker’s color candidate.

In comparison with 23 different surgical room environments, color details of all images are
simultaneously compared with RAL colors. The result of this computation yields 13 distinct RAL
colors that are nowhere to be found in all those images. These colors are RAL1004 Golden Yellow,
RAL 1003 Signal Yellow, RAL 1028 Melon Yellow, RAL 2003 Pastel Orange, RAL 2004 Pure Orange,
RAL 2008 Bright Red Orange, RAL2011 Deep Orange, RAL 3020 Traffic Red, RAL 3026 Luminous
Bright Red and RAL 4005 Blue Lilac. Categorizing these colors according to their hues, they belong to
yellow, orange, red and violet hues. In this research, the goal is identifying at least three different
colors for instant marker identity during operation. Even though there are 13 different RAL colors,
only 4 different hues are present. Taking into account the illumination problem that will definitely
occur during operation, red hues are the least possible candidates because of a probable
interference with different red hues especially bloodline, disclosed area during operation or blood
bag.

Looking at the color description itself, the colors are mostly luminous and fluorescent. To
avoid getting confused with the same hues in different illumination, marker colors are chosen from
all different hues. Since availability of these colors are limited in small containers, RAL1004 Golden
Yellow, RAL2004 Pure Orange and RAL 4005 Blue Lilac are chosen due to their availability in nearest
stores.

These colors are painted on the markers and the color distribution for the distance of
markers from camera in [30, 50, 100, 150, and 200] cm sets are obtained. From these images, new
RGB classifiers are built using Hill Climbing segmentation to get color variation of the markers due to
the effect of distance and illumination. These classifiers are then used for reference in real-time
recognition.

25
3.2 Shape and Size Selection
In determining the markers size and shape, there are a lot of recognition factors that must be taken
into account. The fact that this research is for real time medical application signifies that speed and
accuracy is the most important criteria. Therefore, markers must be design to fit these criteria and
also easily producible or available in current market since they must be available during
experiments.

3.2.1 Shape-based Recognition Challenges

For fast computation, it is important to consider the least complicated geometry for the markers.

3.2.2 Optimum Shape Recognition Factor

3.2.3 Marker Individual Arrangement

3.2.4 Size Factor for Recognition Ability

4 Recognition Strategy and Implementation

4.1 Recognition Problems Revisited

4.2 Recognition Steps

4.3 Position and Orientation Computation

4.4 Real-time Compatibility

26
5 Experiments and Results

Future Work

Conclusion

27
APPENDIX

Program codes

28
References
Andreas Koschan, M. A. (2008). Digital Color Image Processing. John Wiley & Sons, Inc.

Christian Teutsch, D. B. (2006). Real-time detection of elliptic shapes for automated object
recognition and object tracking. IEEE.

Cruces, R. A. (2008). Concept and Design of a Cooperative Robotic Assistant Surgery System. PhD
Thesis, University Of Siegen, ZESS, Siegen.

D. Sykora, D. S. (2008). Real Time Color Ball Tracking for Augmented Reality. EGVE Symposium. The
Biographics Association.

D'Zmura, M. (1991). COlor COnstancy: surface color from changing illuminations. California, USA:
Josa Communications.

Formwalt, B. P. (2000). Real Time Color Vision Processing. Ohio: Air Force Institute of Technology.

J.M Chassery, C. G. (1984, November). AN Iterative Segmentation Method Based on a Contextual

Color and Shape Criterion. IEEE Transactions on Pattern Analysis and Machine Intelligence , pp. 794-
800.

Mark S. Drew, J. W.-N. (1998). Illumination Invariant Color Object Recognition via Compressed
Chromaticity Histograms of Color Channel Normalized Images. Sixth International Conference on
Computer Vision (pp. 533-540). IEEE.

Mirgitta Martinkauppi, M. S. (2003). Detection of Skin Colour under Changing Illumination: A

Comparative Study. 12th International Conference on Image Analysis and Processing (ICIAP). IEEE.

T.Gevers, A. S. (1997). Color Based Object Recognition. In Misc., Lecture Notes in Computer Science
(pp. 319-326). Heidelberg: Springer Berlin / Heidelberg.

Tinku Acharya, A. K. (2005). Image Processing: Principles and Applications. John Wiley & Sons.

Y-I Ohta, T. K. (1980). Color Information for Region Segmentation. Computer Graphics and Image
Processing 13 , 222-241.

29