Department of Computer Science and Engineering

University of Moratuwa

CS4202 - Research and Development Project

OPTICAL COLOR SORTING MACHINE FOR TEA
PARTICLE SORTING

Group Members
120026E J.P.B Anushka
120126K A.Y Dissanayake
120162P H.V.L.C Gamage
120496R K.A.D.B Priyadarshana

Supervisors
Mr. Nalin Karunasinghe

Coordinated by
Dr. Charith Chithranjan

THIS REPORT IS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS

FOR THE AWARD OF THE DEGREE OF BACHELOR OF SCIENCE OF
ENGINEERING AT UNIVERSITY OF MORATUWA, SRI LANKA.

April 06, 2017

 DECLARATION

We, the project group Sortex hereby declare that except where specified reference is made to
the work of others, the project Sortex - an optical tea sorter for Sri Lankan tea industry- is our
own work and contains nothing which is the outcome of work done in collaboration with
others, except as specified in the text and Acknowledgement.

Signatures of the candidates:

.......................................................
J.P.B Anushka
[120026E]
.......................................................
A.Y Dissanayake
[120126K]
.......................................................
H.V.L.C Gamage
[120162P]
.......................................................
K.A.D.B Priyadarshana
[120496R]

Supervisor:
.......................................................
(Signature and Date)
Mr. Nalin Karunasinghe

Coordinator:
.......................................................
(Signature and Date)
Dr. Charith Chithranjan

i
 ABSTRACT

Tea products are present in incredible varieties in terms of shape, size, color and as the
market grows more demanding, the tea products are subdivided in various categories and are
destined to different segments. The definition and characterization of different attributes of
tea are very important for the business and for the consumer, making it necessary to establish
norms of classification and standardization, thus making commercial trading more efficient
and allowing for higher awareness on the part of consumers. A computer vision system
(CVS) has proven very successful in the analysis on the basis of color, size, shape, texture etc
in food engineering. Through this project we use these computer vision and image processing
techniques for implementing the optical tea sorting machine. And apart from that through the
research part, the advantages of using a color pixels classification algorithm in a Hybrid
Color Space with derived color components were presented. This new color space depends on
the application. This introduced a new method which is a lightweight solution for
implementation and more accurate than feed forward neural network. To increase the
accuracy of result evaluation the characteristic curve of histogram was used instead of
histogram overlapping calculation method which reduces the dependency on training sample
and the accuracy level.

ii
 ACKNOWLEDGEMENT
The final year research and development project module (CS4202) is great chance for the
students to explore new fields of study in computer science and engineering discipline
expand our knowledge. We would like express our gratitude towards

Department of Computer Science & Engineering for giving us the opportunity to perform
this kind of work by offering the module. We would like to thank Eng. Nalin for the support
in both knowledge-wise and resources-wise to make this project a success. And also we
would like to thank Eng. Yasura Vithana for sharing his expertise knowledge in the field and
spending his valuable time to guide us through the project.

Last but not least, we would like to express our heartiest gratitude towards the evaluation
panel members Dr. Chandana Gamage and Dr. Charith (project co-ordinator) for all the
support & advices given and the effort made in conducting this module and for the resources
we were given carry out the project.

Finally, we would like express our heartiest gratitude towards our friends and family for the
immense support given throughout the project.

iii
 TABLE OF CONTENTS
Chapter 1: Introduction 9
1.1 Background 9
1.2 Problem statement 9
1.3 Motivation 10
1.4 Requirement Specification 10
Chapter 2 : Literature Review 11
2.1 Abstract 11
2.2 Introduction 11
2.3 Computer Vision System 13
2.4 Image Processing System 13
2.5 Color Feature extraction 14
2.6 Morphological feature extraction 16
2.7 Shape feature extraction 17
2.8 Texture feature extraction 18
2.9 Knowledge base Learning 23
2.10 Hardware Used for Sorting 27
Chapter 3: Experimental Procedure 30
3.1 Preprocessing 30
3.1.2 Background removing 58
3.1.3 White Balancing 58
3.1.3 Data extraction 59
3.1.4 Data cleaning 59
3.1.5 Feature Selections 60
3.1.6 Data Transformation 63
3.2 Testing With Different Machine Learning Model & Evaluation of the Results 64
Evaluation Criteria 64
3.2.1 Artificial Neural Network: 66
3.2.2 Perceptron 67
3.2.3 Naive Bayes 67
3.2.4 Support Vector Machine 68
ROC - SVM Linear kernel 69
ROC - SVM- rbf kernel 69
3.2.5 Random Forest 70
3.3 Result Optimization 70

iv
 3.3.1 Feature Selection 70
3.3.2 Derive New Attributes. 71
3.3.3 Outlier Analysis 71
3.3.4 Optimized Results 71
3.3.5 Final Algorithm of the Research Component 76
Chapter 4 : System Design 77
4.1 Component Diagram 77
4.1.1 Image Acquisition Subsystem 77
4.1.2 Data Extracting module 77
4.1.3 Decision Making Module 78
4.1.4 Actuator Controller 78
4.1.5 Controlling Module 78
4.1.6 Human Interface Module 78
4.1.7 Machine learning System 78
4.2 High Level Hardware Architecture 79
4.3 Detailed Architecture 80
4.3.1 Real Time Stem/Leaf Classification Module 80
4.3.2 Controlling module 80
4.4 Data Flow Diagram 81
3.5 Decision Making Flowchart 83
Chapter 5 : Implementation 84
5.1 Real time Stem/Leaf Classification Module 84
5.1.1 Hardware used 84
5.1.2 Implementation Details 85
5.2 Machine Learning Module 87
5.2.1.Pre Processing Unit 87
5.2.3.Feature Extraction Unit 88
5.2.4.Feature Selection and analytical unit 88
5.3 Human Machine Interaction Module and Controlling Module 91
5.3.1 Hardware Used 91
Chapter 6: Discussion 95
Chapter 7: Conclusions 97

v
 LIST OF FIGURES
Figure 1: Tea Manufacturing Process 9
Figure 2: Process of a Computer Vision Based System to Obtain the Tea Image Database 14
Figure 3 : Example Visualization of Good Cluster 20
Figure 4 : Example Visualization of Bad Cluster 21
Figure 5 : Basic concept of a machine learning system 23
Figure 6 : structure of the artificial neural network. 24
Figure 7 : Summary of ANN application for classification of food product done in [6]. 25
Figure 8 : maximum margin hyper plane for a SVM with two class data set 26
Figure 9 : SVM Classification Architecture of the literature [18] 27
Figure 10 : Tea samples from different areas of the country 30
Figure 11 : Y Component Histogram 30
Figure 12 : Cb Component Histogram 31
Figure 13 : Cr Component Histogram 31
Figure 14 : R Component Histogram 32
Figure 15 : G Component Histogram 32
Figure 16 : B Component Histogram 32
Figure 17 :L Component Histogram 33
Figure 18 : *a Component Histogram 33
Figure 19 : *b Component Histogram 33
Figure 20 : H Component Histogram 34
Figure 21 : S Component Histogram 34
Figure 22 : V Component Histogram 34
Figure 23 : Y Component Histogram 42
Figure 24 : Y Component Histogram 42
Figure 25 : Cr Component Histogram 42
Figure 26 : R Component Histogram 43
Figure 27 : G Component Histogram 43
Figure 28 : B Component Histogram 43
Figure 29 : H Component Histogram 44
Figure 30 : S Component Histogram 44
Figure 31 : V Component Histogram 44
Figure 32 : L Component Histogram 45
Figure 33 : a* Component Histogram 45
Figure 34 : b* Component Histogram 45
Figure 35 : Y Component Histogram 46
Figure 36 : Cb Component Histogram 46
Figure 37 : Cr Component Histogram 47
Figure 38 : R Component Histogram 47
Figure 39 ; G Component Histogram 47
Figure 40 : B Component Histogram 48
Figure 41 : Cr/Cb Component Histogram 48
Figure 42 : 1.5*Cr-2*Cb Component Histogram 49
Figure 43 : 9.3628*Y+8.9859*Cr+8.6208*Cb Component Histogram 49
Figure 44 : -0.0015Y+ -0.0035*Cr+0.0003*Cb Component Histogram 50
Figure 45 : -0.3339*Y -0.3069*Cr -0.1662*Cb Component Histogram 50
Figure 46 : -0.0003*Y+ 0.0002*Cr+ 0.0027*Cb Component Histogram 50

vi
Figure 47 : Multi linear regression to Y Cr Cb Components 51
Figure 48 : Multi linear regression to R G B Components 51
Figure 49 : H/S Component 51
Figure 50 : H-S Component Histogram 52
Figure 51 : b*Cb Component Histogram 52
Figure 52 : Cb-2*b3 Component Histogram 52
Figure 53 : Cb-2.5*b3-0.8*Cr Component Histogram 53
Figure 54 : -37.797*R-74.204*G+112*B Component Histogram 53
Figure 55 : 0.5687*R - 0.2813*G + 0.88*B Component Histogram 53
Figure 56 : R2 + B2 Component Histogram 54
Figure 57:Single Particle Identification Block Diagram 54
Figure 58;Motion Detection Block Diagram 55
Figure 59 : Object Detection Block Matching Diagram 55
Figure 60 : RGB Color Space 60
Figure 61 : YCbCr Color Space 61
Figure 62 : HSV Color Space 61
Figure 63 : L* a* b* Color Space 61
Figure 64 : Shape Features 62
Figure 65 : Textural Features 62
Figure 66 : ROC Curve of ANN 66
Figure 67: ROC Curve for perceptron 67
Figure 68 : ROC Curve of Naive Bayesian 67
Figure 69 : ROC Curve of SVM 69
Figure 70 : ROC Curve of Random Forest 70
Figure 71 : Component Diagram of the System 77
Figure 72 ; High Level Architecture Diagram 79
Figure 73 : Logic Diagram of the Classification Module 80
Figure 74 : Flow Diagram of Controlling Module 80
Figure 75 : Logic Diagram of H/W implementation 81
Figure 76 : Decision Making Flowchart 83
Figure 77 : Logic Diagram of ML Module 87
Figure 78 : UI of HMI Panel 93

vii
 LIST OF ABBREVIATIONS

Abbreviation Description

ML Machine Learning

FPGA Field Programmable Gate Array

PL Programmable Logic

PS Processing System

IPC Industrial PC

SoC System on Chip

viii
Chapter 1: INTRODUCTION

1.1 Background
Tea production is one of the main sources of foreign income in Sri Lanka. Sri Lankan tea
recognized as a best tea category in the world because of the unique taste and the high
quality. Production of high quality tea is not a simple task. After plucking tea leaves, they are
transported to the tea factory and go through various steps before producing the consumable
tea. Figure 1 shows the steps of tea producing process.

Figure 1: Tea Manufacturing Process

This project focuses on the last step of the process. Last step is removing stems from tea
mixture. Removing stems is a very critical step in producing high quality tea. Because tea
stems affect the taste and the quality of the tea. To produce high quality tea as much as
possible amount of stems should be removed from the mixture.

1.2 Problem statement

Color sorting machine is the latest method used to remove stems from the tea mixture. This
machine identifies stems using machine vision techniques and removes those stems using a
ejecting mechanism. Color sorters use threshold values to differentiate stems from leaves.
Initially those values need to configure manually by inspecting the machine output. Based on
the misclassifications operator has to tune those threshold values. The problem is color of tea
stems and leaves change due to various factors like whether conditions of the tea estate and
variations of firing temperature. If color of the mixture changed due to any reason the color
sorting machine need to reconfigure manually to match the new mixture. This is time
consuming and accuracy of the stem removing also affected if the selected threshold values
are not optimal. This is the main problem we are going to address in this project.
The other problem we are going to solve is misclassifications due to partially burned tea
stems and leaves. Current stem removing systems using only color data to identify stems.
Hence, if a stem or leaf is not burned completely the machine cant identify it correctly

1
because the machine is configured to identify the stems and leaves based on the color of a
completely burned stems and leaves. This project contains a research component on how
other characteristics such as shape and size of stems and leaves can be used to differentiate
tea stems from tea leaves.

1.3 Motivation
Stem removing is a very important process in tea industry. In our solution we are going to
improve the accuracy of the stem removing process as well as to automate the machine
configuration process which is currently done manually. This would be one of the greatest
contributions done by a Sri Lankan group of students to the Sri Lankan tea industry.

1.4 Requirement Specification

The tea sorting machine should be able to

Learn and adjust the sorting parameters based on the tea produced in a particular
factory.
Remove tea stems from tea with 95% confidence level

Chapter 2 : LITERATURE REVIEW

2.1 Abstract
Tea products are present in incredible varieties in terms of shape, size, color and as the
market grows more demanding, the tea products are subdivided in various categories and are

2
destined to different segments. The definition and characterization of different attributes of
tea are very important for the business and for the consumer, making it necessary to establish
norms of classification and standardization, thus making commercial trading more efficient
and allowing for higher awareness on the part of consumers. A computer vision system
(CVS) has proven very successful in the analysis on the basis of color, size, shape, texture
etc in food engineering. Computer vision is the science that develops the theoretical and
algorithmic basis by which useful information about an object or scene can be automatically
extracted and analyzed from an observed image, image set or image sequence.
An image can be defined as a spatial representation of an object or scene. Image processing is
base of computer vision system. Image acquisition, processing, segmentation are basic steps
for extracting information of image. Computer vision is an emerging subject in food sector.
CVS system is being used in tea industry for the detection of dust and for the categorization
based on the quality. It is a fast, non-invasive cheap method for evaluating quality.
The use of computers allows for increased efficiency with less manpower, reduces
dependence on trained experts which is costly and time consuming. Also, we can do
modeling of information obtained from image and can be used for future using machine
learning techniques.

2.2 Introduction
The great concern with product quality due to new market restrictions have become important
and the market competition become increasing and also new quality testing methods are
developing with new technologies. Over the past decade, significant advances in techniques
of tests have been observed, while extraordinary resources in electronics and informatics
were identified as important factors in this development. Automation has motivated the
development of testing equipment in production lines, and the evolution of sensor technology
has led to the establishment of new techniques of measurement of the products, thus allowing
permanent monitoring during the process, with the implementation of visual inspection
systems. In the tea industry, some quality evaluation is still performed manually by trained
inspectors, which is tedious, laborious, costly and inherently unreliable due to its subjective
nature. Increased demands for objectivity, consistency and efficiency have necessitated the
introduction of computer-based image processing techniques.
Recently, computer vision employing image processing techniques has been developed
rapidly, which can quantitatively characterize complex size, shape, color and texture

3
properties of foods. But less number of solutions for the tea grading and sorting systems is
available.
Computer vision based systems are basically a combination of a light source, a device for
capturing the image and a computational system for the extraction of characteristics. Such
systems are important for the product lines because it reduce the repetitive activities of the
human, increase the efficiency of the manufacturing process and since the less human
interaction , the accuracy and the quality of the output is increased. Different from the
problems present in visual inspection performed by people, these kinds of systems offer
accuracy and repeatability in measurements without contact, especially due to the elimination
of aspects such as subjectivity, tiredness, slowness and costs associated with human
inspection.
By automating the grading and the sorting process in tea industry has become an interesting
solution for the final analysis of the product quality and the assessed values or characteristics
involve not only dimensional aspects, but also characteristics of color, texture and shape.
Tea samples present a variety of shapes, sizes, colors and flavors, and as the market grows
more demanding, tea products are subdivided in various categories.
Over the past few years, the explosive growth in both computer hardware and software has
led to many significant advances in computer vision technology. Computer vision technology
provides a high level of flexibility and repeatability at relatively low cost. It also permits
fairly high plant throughput without compromising accuracy. Currently, computer vision
systems are being developed as an integral part of food processing plants for on-line, real-
time quality evaluation and quality control [11]
Therefore the computer vision based system can use for sorting and grading of agricultural
products such as tea.
In the case of detection of tea leaf, it is essential to correctly extract the object (leaf or stem)
from the background. Image segmentation is an important and perhaps the most difficult
image processing task. Segmentation refers to subdividing an image into regions exhibiting
similar characteristics. Subsequent image interpretation tasks, such as feature extraction
and object recognition, rely on the quality of the segmentation results.

2.3 Computer Vision System

Computer vision is the science that develops the theoretical and algorithmic basis by which
useful information about an object or scene can be automatically extracted and analyzed from
an observed image, image set or image sequence [11]. An image is the main component of a

4
computer vision based systems and it is defined as the spatial representation of an object.
Computer vision based systems rapidly used in the food engineering and they use to inspect
the quality, sorting and grading processes etc. Such systems are important because they
reduce the human interaction in food engineering and increase the accuracy of the results.
Research on tea industry was ranging from a simple evaluation of tea samples to a
complicated robot guidance application. In past decade, computer vision based systems have
been used in color measurement and sorting up to some accuracy level, including
classification and grading by shape detection and extracting the texture of the tea particles
[15]. Alternatively, computer vision system (CVS), a nondestructive method, offering
objective measures for color and other physical factors.

2.4 Image Processing System

An image is a matrix which consists of square pixels. Each pixel indicates two factors, i.e.
brightness value and the locations in the coordinates that are assigned to the image. The
former is the color feature while features extracted from the latter are known as size or shape
features. Image processing has become a main part of the tea sorting and grading process of
the currently available systems. This technique plays an important role in the tea sorting and
grading process while maintaining the accuracy and consistency. They offer flexibility in
application and can be reasonable substitutes for the human visual decision-making process.
In order to develop an automated system for tea sorting process, image processing techniques
are often combined with mechanical and instrumental devices to replace human manipulative
effort in the performance of a given process. In such a system, the image processing system is
the center, which controls the operation of the machinery.
Image acquisition
Digital image analysis offers an objective and quantitative method for the estimation of
morphological parameters. This process uses digital images to measure the size of individual
grains and mathematically extract features and shape related information from the images.

The study Identification of Green, Oolong and Black Teas in China via Wavelet Packet
Entropy and Fuzzy Support Vector Machine[18] had implemented the image acquiring
experiment as shown in figure 1. They had included five parts i.e. a digital camera, an
illumination platform, a capture board (digitizer or frame grabber), computer software, and
computer hardware.

5
 Figure 2: Process of a Computer Vision Based System to Obtain the Tea Image Database

2.5 Color Feature extraction

According to the diverse information stored in pixels, image features obtained can be
categorized into four types - color, size, shape, and texture.
In this project we need to extract the quantitative measurement such as color, shape and
texture to do the classification from tea leaves from the mixture.
In literature, Classification system for rain fed wheat grain cultivars using artificial neural
network[1], a variety of different methods have been developed to measure size, shape,
color, and texture features.
Color feature extraction
An RGB image, sometimes referred as a true color image, is stored as an m-by-n-by-3 data
array that defines red, green and blue color components for each individual pixel.
Using MATLAB and the image processing toolbox software extract the individual color
components like Y, Cb ,Cr, R, G, B etc in color spaces like RGB, YCbCR, HSV, L*a*b
which are most common color schemes use for color classification in grains and I1I2I3 color

6
space is defined in [4] which was calculated from RGB color space, using the following
equations

The study done for automated strawberry grading had used a technique called Dominant
Color Method[3]. In this study they had considered about the L*a*b* color space for their
experiment.
Generally, the human sight is more interested in main color of the image means that color
which appears frequently in the image. So this Dominant Color Method was used on a*
channel to extract the color feature from the image.
The methodology which was followed by them is as follows.
Step 1:
It first calculates the histogram of the image on a* channel.
Step 2:
Then it sets a window of width L, and then from the most left of the histogram the gray value
of the window is calculated. Then window is moved one pixel towards right and again the
gray value is calculated. This procedure continues until window is moved towards the most
right.
Step 3 : Then the major color window is found out i.e. the window having the largest
frequency. And then by using the Eq. (1) the average color value of the dominant window is
found out.

The L*a*b* model was also used in to detect the browning degree . The average L*a*b*
values were obtained from the image using the Histogram Window of Adobe Photoshop CS5.

7
In the histogram window the parameters L, a, b were not standard colour values so they were
converted as follows:

2.6 Morphological feature extraction

The following morphological features were extracted from labeled images of individual rain
fed wheat grains cultivars. Geometry related features including area, perimeter and major and
minor axis lengths were measured from the binary images[1]. Area (A): The area of a region
is defined as the number of pixels contained within its boundary; perimeter (P): the perimeter
is the length of its boundary. The length of the minor axis is the longest line that can be
drawn through the object perpendicular to the major axis.

Equivalent diameter (Eq): It was the diameter of a circle with the same area as the wheat
grain region.

Convex area (C): it is the number of pixels in the smallest convex polygon that can contain
the wheat grains region; solidity (S): the

8
proportion of the pixels in the grains region that are also in the convex hull; extent (Ex): the
proportion of the pixels in the bounding box which are also in the grains region; roundness
(R): this is given by:

Compactness (CO): the compactness provides a measure of the object's roundness:

2.7 Shape feature extraction

Size reflects the number of pixels and shape describes the boundary of food products. Size
features are usually obtained using measurements of area, perimeter, length, and width. Shape
is characterized mostly in two ways, i.e., size dependent measurements such as compactness
and elongation, and size independent measurements such as spatial moment and Fourier
descriptor (boundary shape).
In the study[2] , they had used the axis length and the area of the object to derive the shape
factors as follows.

9
2.8 Texture feature extraction
Texture is normally the dependency between pixels and their neighboring pixels or the
variation of intensity of pixel.

glcm is an m-by-n-by-p array of valid gray-level co-occurrence matrices. If glcm is an array

of GLCMs, stats is an array of statistics for each glcm.
graycoprops normalizes the gray-level co-occurrence matrix (GLCM) so that the sum of its
elements is equal to 1. Each element (r,c) in the normalized GLCM is the joint probability
occurrence of pixel pairs with a defined spatial relationship having gray level values r and c
in the image. graycoprops uses the normalized GLCM to calculate properties.
From the gray-level co-occurrence matrix glcm,Contrast,Homogeneity,Correlation,Energy
were derived[4] as follow:
Contrast: Returns a measure of the intensity contrast between a pixel and its neighbor over
the whole image.Range = [0 (size(GLCM,1)-1)^2] Contrast is 0 for a constant image.
The property Contrast is also known as variance and inertia.
Contrast=

Correlation:
Returns a measure of how correlated a pixel is to its neighbor over the whole image.
Range = [-1 1] Correlation is 1 or -1 for a perfectly positively or negatively correlated image.
Correlation=

Energy
Returns the sum of squared elements in the GLCM.
Range = [0 1] Energy is 1 for a constant image.
Energy=

10
Homogeneity
Returns a value that measures the closeness of the distribution of elements in the GLCM to
the GLCM diagonal.
Range = [0 1]
Homogeneity is 1 for a diagonal GLCM.
Homogeneity=

The feature vector was made from the earlier mentioned features and feed, used as an
artificial neural network for classification which in this case was used as a multi layer
perceptron (MLP) method.
Feature selection

Using UTA[3]
Feature selection is the problem of choosing a subset of features ideally necessary to perform
the classification task from a larger set of candidate features. There are several ways to
determine the best subset of features. UTA is a simple method which is based on trained
artificial neural network. In the basis of this method, average of one feature in all instances is
calculated. Then, the selected feature in all input vectors is replaced by the calculated mean
value. Then, the trained network is tested with the new features and data matrix . The
comparison error was defined in our strategy as follow:
E= (FP (new) + FN (new)) - (FP (old) + FN (old)) (14)
Where, FP(old) is the false positive and FN(old) is the false negative using the whole features
and FP(new) and FN(new) are those values when one of the feature are replaced by the mean
value. There are three states in this way: (1) one input is considered more relevant if E is
positive and higher according to the other features; (2) one input is ineffective if E is zero; (3)

11
one input is not only ineffective but also noisy and should be removed from the input vector
if E is negative.

Using data clustering against each other

Plot the data clusters against each features and identify the features that can be clustered most
efficiently and the standard deviation is low.

Figure 3 : Example Visualization of Good Cluster

Figure 4 : Example Visualization of Bad Cluster

12
 Principle Component Analysis
The research done in [16] had used the principle component analysis for the feature selection.
In this experiment they had extracted 80 features from the tea image. When there are number
of features extracted , the computation cost is increased. Therefore they had used PCA to
reduce the number of features.
Principle Component Analysis is an effective technique which can be used to reduce the
features and also can be used to retain the most substantial principle components.
Data pre-processing

Image Segmentation
The acquired images consist of both the object and the background. Therefore a technique
capable of removing the background from the images is needed.

Background Removing using K mean Clustering

Using Statistics and Machine Learning Toolbox of matlab we can segment colors in an
automated fashion using the L*a*b* color space and K-means clustering [5]

13
Figure 4-a : Original Image Figure 4-b: Segmented Image

Background Removing using Gray Level Thresholding

Using MATLAB image processing tool computes a global threshold (level) that can be used
to convert an intensity image to a binary image with im2bw. level is a normalized intensity
value that lies in the range [0, 1].The graythresh function uses Otsu's method, which chooses
the threshold to minimize the intraclass variance of the black and white pixels.
Multidimensional arrays are converted automatically to 2-D arrays using reshape. The
graythresh function ignores any nonzero imaginary part of I. [level EM] = graythresh(I)
returns the effectiveness metric, EM, as the second output argument. The effectiveness metric
is a value in the range [0 1] that indicates the effectiveness of the thresholding of the input
image. The lower bound is attainable only by images having a single gray level, and the
upper bound is attainable only by two-valued images.

14
2.9 Knowledge base Learning
Here the main purpose is to investigate the recent and relevant applications of learning
techniques in computer vision for food industry and how can them apply in tea sorting and
grading process.Figure 6 indicates the basic configuration of a knowledge base learning
system can be applied to any system. As shown in figure 6 the images of samples are
quantitatively characterized by a set of features such as size, shape, color and texture.
As discussed in feature extraction part a variety of different methods have been developed to
measure size, shape, color, and texture features. These features are identical data used to
represent the samples, which can also be used to form the training set. After the training set
has been developed, learning algorithm extracts the knowledge base necessary to make
decisions of test samples. Based on the knowledge, intelligent decision is made as output and
fed back to the knowledge base at the same time, which generalizes the way that inspectors
use to accomplish their tasks.

Figure 5 : Basic concept of a machine learning system

In literature, a variety of different learning techniques such as artificial neural network,

statistical learning, fuzzy logic, and decision tree have been used for learning, which have
been reviewed by [6].
The learning techniques are related to our project to do the classification in sorting/grading
process. Classification need to ensure whether the captured particle is a tea leaf or dust.
Neural network

15
 Figure 6 : structure of the artificial neural network.

Artificial neural network is a learning technique which behaves according to the theory of
human brain information processing system. Artificial Neural Network gathers information
by detecting and extracting the patterns and the relationship of the data set. An ANN is made
with hundreds of single unit which is called perceptron. This single unit has weighted inputs,
transfer function and one single output. A neural network consists of three main layers which
are called input layer, hidden layer and output layer. The behavior of the whole network
depends on the transfer functions of each neuron. The weights are adjustable hence a neural
network is a parameterized system. During the training the inter unit connections are
optimized until error of classification minimize and the classification becoming to a specified
accuracy level.
ANN technology allows the extension of computer vision technology into the areas of color,
content, shape, and texture inspection at near-human levels of performance, and can provide
the decision making and classification capabilities to succeed in these inspection tasks [9].
Currently the Artificial neural network has been used in food engineering for classification
and prediction purposes.
In literature, which has been reviewed in [6] has mentioned about food classification
experiment results which were done using artificial neural networks. This classification has

16
done based on the quality of the foods. To analyze the quality, they had used quantitative
measurements such as shape, color, texture etc. They had done this experiments under several
food categories i.e fishery, fruit, grain, meat and vegetable. As the result of this experiment
they present the following accuracy table.

Figure 7 : Summary of ANN application for classification of food product done in [6].

This study is related with our system because we also need to do a classification to separate
tea leaves from the mixture. And also the quantitative features such as size, shape, color,
texture are related with our process.

Support Vector machines

Support vector machine is a supervised learning method which is used in both classification
and regression [19]. In SVM, it minimizes the empirical classification error and maximizes
the geometric margin simultaneously. So the SVM is identified as the maximum margin
classifier. The input vector is mapped to a higher dimensional space where a maximal
separating hyperplane is constructed. There are two parallel hyperplanes are constructed on
each side of the hyperplane that separate the data. . An assumption is made that the larger
the margin or distance between these parallel hyperplanes the better the generalization
error of the classifier will be [19].

17
 Figure 8 : maximum margin hyper plane for a SVM with two class data set

Kernel of SVM:
A kernel is a similarity function. This kernel function provides an alternative way to the basic
machine learning concept. When considering some example classification system, instead of
defining slew of features, define a kernel function to find similarity between data points. The
rapidly used kernel functions are linear kernel, polynomial kernel, radial basis kernel and
sigmoid kernel.
In these popular kernel functions, RBF is the main kernel function because of
following reasons [19]:
1. The RBF kernel nonlinearly maps samples into a higher dimensional space unlike to linear
kernel.
2. The RBF kernel has less hyper parameters than the polynomial kernel.
3. The RBF kernel has less numerical difficulties.
A study done by [18] have been used support vector machines to classify tea into different
categories based on quantitative features i.e 64 color features and 16 wavelet features. In
this experiment they had used principle component analysis to identify important features and
then they had tested with the support vector machines. As the support vector machines cannot
deal with noise and the outliers i.e, its performance will decrease sharply when the data set
either contains outliers or was contaminated by noises they had used a variant of SVM which
is called Fuzzy SVM to overcome the issue. In this study, they had used an advanced
technique to calculate the out-of-sample performance of the proposed identification system:
the K-fold stratified cross validation (SCV) technique. The following figure 10 shows their

18
proposed knowledge based learning system to categorized tea based on the identified
quantitative measures.

Figure 9 : SVM Classification Architecture of the literature [18]

Bayesian Inference Technique

This technique has developed based on the Bayes theorem. An object of an image contains a
range of colors than a single color. So this technique has the ability to utilizing a color data
set containing a number of colors from the image and has used to find similar color data in
another image and has provide useful information for classification[17].
The Bayesian inference technique improves on the use of averages by capturing more
detailed information about the colours, and their distribution, in the image.
It calculates proportions of various ranges of colour by dividing the colour space of the
reference sample image into several regions.
By calculating the reference region probabilities, the process enables one to evaluate which of
the reference food images is the most likely match to the sample food image.
The study done by [17] has used Bayesian inference technique to do the food classification.
This classification has done using the average of CIE L*a*b* color space. Since this study is
color based classification system and color based classification is a major part of our system
this solution should be considerable while doing the experiments.

2.10 Hardware Used for Sorting

To identify the color, shape and texture of the particles in real time special hardware is
required .Since the image acquisition and processing should be done in real time.

Image Sensor

19
It is important to capture the image in a required quality to perform the classification. Both
line sensors as well as area sensors[14] are being used for this purpose. To obtain a higher
frame rate the region of interest of the camera can be defined to cover a small range. For
example in A machine vision system for high speed sorting of small spots on grains[14] a
color VGA sensor which operates in near linear scale s used. The image sensor that was used
is KAC-9628 manufactured by Eastman Kodak Co., Rochester, NY. It consists with
640x480 pixels of sensors and uses a color bayer filter to sense the Red , Green and Blue
colors .
Approximate spatial resolution is 0.06 mm per pixel. By reducing the region of interest of the
camera it is operated in near line scanner mode. Each line consists 640 pixels. This line is
divided to three parallel channels. From this line 3 different regions of interest are defined.
For this particular application 124 pixels from each channel is used.
Not only standard optical cameras, some optical sorting systems use X-ray and infrared
sensors to acquire the image. But these images requires long processing time compared to
standard optical cameras [12]

Clock
The FPGA can be clocked faster than the camera. But both camera and the FPGA is clocked
to synchronize both devices. The image sensor is configured to reduce the clock frequency
by dividing it by a factor of a power of 2. For example 48 MHz clock source can be used to
clock the FPGA in 48 MHz and the image sensor is clocked in 12 MHz [14]. Therefore the
mathematical operations can be performed while transferring pixel data from image sensor to
the FPGA.
EEPROM
EEPROM is used to store configuration data to be read by the FPGA at the system startup.
Data Transferring
Cameras boards use various color schemes to produce the output image. For example RGB
[python 1300] , YCrCb color schemes.
Lighting System
A very important factor of image acquisition is lighting condition. Most of the existing
sorting machines uses array of Light Emitting Diode as the source. A linear lense is used to
defuse the light from the lightning source and to focus to the imaging point .

20
Colors such as cool white [14] as well as various color combinations are used when capturing
the image[metaclorsorter]. To avoid the LED s being overheated an Aluminum heat sink is
used sometimes with a fan to dissipate the heat[14].
For special requirements special lighting sources are used. For example in mineral sorting X-
ray light source is used [12].
Processing Unit
In industrial optical sorting systems different kind of processing units are used depending on
the application. For high speed sorting such as sorting grains in free fall, Field Programmable
Gate Arrays are used [14] as the processing unit because the processing need to happen real
time. Some other sorting systems such as industrial mineral sorting systems where minerals
are being identified while they are on a convoy belt [12]industrial pcs are used as the
processing component. When the development cost needed to be get into the account without
going for complete hardware based solution like FPGA microprocessors have been used [8].
For example if a raspberry pi is used as a processing unit image processing libraries such as
opencv can be used. But due to limited processing power the convoy belt method have to be
used instead of free fall under gravity method.

21
Chapter 3: EXPERIMENTAL PROCEDURE

3.1 Preprocessing
First we collected the tea steam and leaf samples from different areas of country to analyses
their color variation. Then we could identify that Matara tea has high brown color than
Rakwana tea.

Figure 10 : Tea samples from different areas of the country

Then we plot the histograms of those tea samples which as shown below.

Histogram for tea Leaves from different area(Pelmadulla, Rakwana, Deniyaya and Balangoda
)

YCbCr Color Space

Figure 11 : Y Component Histogram

22
Figure 12 : Cb Component Histogram

Figure 13 : Cr Component Histogram

23
RGB Color Space

Figure 14 : R Component Histogram

Figure 15 : G Component Histogram

Figure 16 : B Component Histogram

24
L*a*b Color Space

Figure 17 :L Component Histogram

Figure 18 : *a Component Histogram

Figure 19 : *b Component Histogram

25
HSV color Space

Figure 20 : H Component Histogram

Figure 21 : S Component Histogram

Figure 22 : V Component Histogram

26
By that we could identify main existing problem of current method which is hard coded the
threshold values within the machine learning module. So for our system we try to built
dynamically changing thresholds which could able to train time to time.

And also we could identify that based on color space which we are using the classification
property of the tea samples are vastly vary. So we create a heuristic which we can classify tea
stem and leaf if we could able to find the proper color space. So we separate tea stem and
leaf samples of Pelmadulla tea. And the following steps are applied for data preprocessing.

Gathered samples are pasted on white background paper sample wise.

27
Crop the pasted samples to their bounding box. For this one we use moor neighbor algorithm.
The Moore neighborhood of a pixel, P, is the set of 8 pixels which share a vertex or edge
with that pixel. These pixels are namely pixels P1, P2, P3, P4, P5, P6, P7 and P8 shown in
Figure 1 below. The Moore neighborhood (also known as the 8-neighbors or indirect
neighbors) is an important concept that frequently arises in the literature.

Given a digital pattern i.e. a group of black pixels, on a background of white pixels i.e. a grid;
locate a black pixel and declare it as your "start" pixel. (Locating a "start" pixel can be done
in a number of ways; we'll start at the bottom left corner of the grid, scan each column of
pixels from the bottom going upwards -starting from the leftmost column and proceeding to
the right- until we encounter a black pixel. We'll declare that pixel as our "start" pixel.)

28
Next step is white balancing this is mainly for reducing the effect of background light while
we are taking training sample at different background. We basically tried 3 method.
Method 1
Taking the background RGB values and by assuming they are white and 255-Color value
And add that value to all pixels

29
Method 2: taking the mean intensity of full image
Color = color * meanGray / meanColor

Method 3: taking the mean intensity of background of image

Color = color * meanGray / meanColor

Method one does not gave much accurate result so we choose method second

30
For removing the background we use following algorithm:

31
 To convert an image in to binary we choose Otsu algorithm which involves iterating
through all the possible threshold values and calculating a measure of spread for the pixel
To convert
levels eachanside
image in to
of the binary we
threshold, i.e.choose Otsuthat
the pixels algorithm which
either fall involves iterating
in foreground or through
allbackground.
the possibleThe
threshold
aim is values
to find and calculating
the threshold a measure
value where of
thespread
sum offor the pixel levels
foreground and each
side of the threshold,
background spreads i.e. theminimum.
is its pixels that either fall in foreground or background. The aim is to
find
Thethe threshold
algorithm value
will where the sumusing
be demonstrated of foreground
the simpleand
6x6background
image shownspreads
below.is The
its
histogram for the image is shown next to it. To simplify the explanation, only 6 grey scale
minimum.
levels are used.

32
The algorithm will be demonstrated using the simple 6x6 image shown below. The histogram
for the image is shown next to it. To simplify the explanation, only 6 grey scale levels are use

And then for each sample extract the color component mean values color spaces such as
above mentioned. Then plot the histogram stem and leaf for get the idea how they behave on
those color spaces.

33
Histogram for Stem and leaf for Palmadulla tea Standard color spaces : Color mean of
object

Sample Size: 270

YCbCr Color Space

Figure 23 : Y Component Histogram

Figure 24 : Y Component Histogram

34
 Figure 25 : Cr Component Histogram

RGB Color Space

Figure 26 : R Component Histogram

Figure 27 : G Component Histogram

35
 Figure 28 : B Component Histogram

HSV Color Space

Figure 29 : H Component Histogram

Figure 30 : S Component Histogram

36
 Figure 31 : V Component Histogram

L*a*b Color Space

Figure 32 : L Component Histogram

Figure 33 : a* Component Histogram

37
 Figure 34 : b* Component Histogram

Then again we try to optimize those results by taking mean of top 50 percentile of the color
values. This is because stem may contain the partially burned area so those areas color is
black and if we take the mean value those feature may destroy.

Histogram for Stem and leaf for Palmadulla tea Standard color spaces : Color sum of Top 50
percentile

YCbCr Color Space

Figure 35 : Y Component Histogram

38
 Figure 36 : Cb Component Histogram

Figure 37 : Cr Component Histogram

RGB Color Space

Figure 38 : R Component Histogram

39
 Figure 39 ; G Component Histogram

Figure 40 : B Component Histogram

So finally we tried to build a derived color components by using existing color components
such as dividing, linear combinations. At this step we does not use any methodological
approach we just observe the behavior of histograms before finding the methodological
approach.

Histogram for Stem and leaf for Palmadulla tea Optimized color spaces

Sample count: 272

40
Cr/Cb Color Space

Figure 41 : Cr/Cb Component Histogram

Figure 42 : 1.5*Cr-2*Cb Component Histogram

Feature selection methods weights for Y, Cr, Cb components

41
Figure 43 : 9.3628*Y+8.9859*Cr+8.6208*Cb Component Histogram

Figure 44 : -0.0015Y+ -0.0035*Cr+0.0003*Cb Component Histogram

42
Figure 45 : -0.3339*Y -0.3069*Cr -0.1662*Cb Component Histogram

Figure 46 : -0.0003*Y+ 0.0002*Cr+ 0.0027*Cb Component Histogram

Figure 47 : Multi linear regression to Y Cr Cb Components

43
Figure 48 : Multi linear regression to R G B Components

Figure 49 : H/S Component

Figure 50 : H-S Component Histogram

44
 Figure 51 : b*Cb Component Histogram

Figure 52 : Cb-2*b3 Component Histogram

Figure 53 : Cb-2.5*b3-0.8*Cr Component Histogram

45
 Figure 54 : -37.797*R-74.204*G+112*B Component Histogram

Figure 55 : 0.5687*R - 0.2813*G + 0.88*B Component Histogram

Figure 56 : R2 + B2 Component Histogram

46
Finally at the end of the data analysis part we come across to the decision based on the color
space we could achieve significantly optimized result.
Based on these result we build a SIMULINK models and tested it with real time data.

Figure 57:Single Particle Identification Block Diagram

And another analysis we done is to try to find the detect the motion of a particle. Basically we
did this for ejecting particle at a correct place. So for that we tried following methods:
1) Detect Bounding box motion (Frame Differentiation)

47
 Figure 58;Motion Detection Block Diagram

2) Intensity difference

Figure 59 : Object Detection Block Matching Diagram

Next thing we tested is that textural and shape features. For the textural features analysis we
tried Gray Level Co-Occurrence Matrix features.GLCM_Features1 helps to calculate the
features from the different GLCMs that are input to the function. The GLCMs are stored in a
i x j x matrix, where n is the number of GLCMs calculated usually due to the different

48
orientation and displacements used in the algorithm. Usually the values i and j are equal to
'NumLevels' parameter of the GLCM computing function graycomatrix(). Under that we
tested following features:
Autocorrelation:
Contrast
Correlation
Cluster Prominence
Cluster Shade
Dissimilarity
Energy
Entropy
Homogeneity
Maximum probability:
Sum of squares: Variance
Sum average
Sum variance
Sum entropy
Difference variance
Difference entropy
Information measure of correlation
Inverse difference (INV) is holomorphic
Inverse difference normalized (INN)
Inverse difference moment normalized
And also we tried following as shape factors:
Area
Bounding Box ratio(width/height)
Perimeter
Minimum rectangle ratio
Major Axis to Minor axis ratio

Following are some plots shows the result we got from above

49
Implementation on SIMULINK
Case 1 : Width to height ratio of bounding box

Case 2 :Bounding box area of tea stem and leaf particles

50
Case 3 :Object pixel count of tea stem and leaf particles

3.1.2 Background removing

The acquired images consist of both the object and the background. Therefore a technique
capable of removing the background from the images is needed. We used Gray Level
Thresholding method to identify background and remove it.

3.1.3 White Balancing

To Collect data we captured series of images. While capturing images due to light condition
and due to dynamic nature of image sensor captured colors may vary in image to image. To
minimize that variation we did pixel selective white balancing. In pixel selective white
balancing, pixels which should be white (white places in actual scene) are selected . If those
places are not appear as white in image colors are changed by the image capturing process.
By selecting those pixels and calculating color correction for those pixels color correction for
image can be done.
We used a white background to capture images. After identifying the background, color
correction for image can be done using pixel selective white balancing method.

51
3.1.3 Data extraction
Color data of stem and leaves are extracted in RGB, HSV, YCrCb and Lab color spaces.

3.1.4 Data cleaning

3.1.4.1 Detect and remove noisy data

As a data cleaning we basically did the remove noisy data. The image acquiring devices and
background light conditions introduce artifacts to the images. This results in degradation of
the original content of the image. So as explained in image processing techniques we did the
white balancing and reduce the effect of color variations and when taking color value means
we remove the background and get mean color in the object only. And also we have try to
accurate result by instead of mean color value we try to take mean of top 50 percentile of
color values. This is because some stems are half burned and there may have half black
region but we need to categorize them to stem category.

3.1.4.2 Outliers
The data can have some waste materials like small plastic, nails added while processing to
sorting step. So here we use Grubbs Test for deleting outliers with the given significant
level.

52
3.1.5 Feature Selections
Attribute Sampling: Select only a subset of the most important attributes from the data.
Here we selected the color space using pairwise clustering of different color components in
different main color spaces and selected the most clearly clustered color space as Y Cb Cr.
This is specially because all the color spaces are derived from RGB color space and if we
feed all of them separately to ML models dependency in between attributes are increased.
About YCbCr color space:
To extract the color features of the image we used YCbCr color space.
About YCbCr Color Space :
Y: luminance
Cb : Blue Chromaticity
Cr : Red Chromaticity

Figure 60 : RGB Color Space

53
 Figure 61 : YCbCr Color Space

Figure 62 : HSV Color Space

Figure 63 : L* a* b* Color Space

From above color space graphs we select YCbCr color space since it gives more clear
clusters

54
 Figure 64 : Shape Features

Here we select Axis Ratios, Bounding Box Ratio, Minimum Rectangle Ratio and object Area
as the most important shape features

Figure 65 : Textural Features

Here we select Contrast, Correlation, Homogeneity only as textural features as the most
important features

55
3.1.6 Data Transformation
Create binary features : One-Hot-Encoding
Encode categorical integer features using a one-hot aka one-of-K scheme. The input to this
transformer should be a matrix of integers, denoting the values taken on by categorical
features. The output will be a sparse matrix where each column corresponds to one possible
value of one feature.
Overview of Sample Data Set

Train data set

Attribute of the Data Set
CompY: Y component of the color space
CompCb : Cb component of the color space
CompCr : Cr component of the color space
Entropy : A statistical measure of randomness that can be used to
characterize the texture of the input image.
Contrast : A measure of the intensity contrast between a pixel and its neighbor
over the whole image.
Homogeneity : A value that measures the closeness of the distribution of
elements in the gray-level co-occurrence matrix to the GLCM diagonal.
Correlation :A measure of how correlated a pixel is to its neighbor over the
whole image.
Energy :The sum of squared elements in the gray-level co-occurrence matrix
MajorMinorAxisRatios : Ratio between major axis length and minor axis
length
Roundness :Roundness of the particles

56
 BoundingBoxArea : Area of the object cropped by the bounding box
algorithm
BoundingBoxRatio : Height to width ratio of the cropped image from the
bounding box algorithm
MinRectangleRatio : Ratio between minimum rectangle height and minimum
rectangle width
ObjectArea : Area of the object
Label : Class Label. We used two class labels.
1 - tea leaf
0 tea stemp
Derived Attributes
CompCr:CompCb : Ratio between Cr and Cb color components.
CompCr:CompY: Ratio between Cr and Y color components.
CompCb:CompY: Ratio between Cb and Y color components.

3.2 Testing With Different Machine Learning Model & Evaluation of the Results

Evaluation Criteria

Basically when evaluate a machine learning model, following factors are considered.
They are Predictive accuracy, Speed and scalability which explained by time to construct
the model, time to use the model and efficiency in disk-resident databases, Robustness
which explain by handling noise, missing values and irrelevant features, streaming data
and Interpretability which explained by understanding and insight provided by the model

Evaluation technique that we used

In our problem we did binary classification using above mentioned machine learning
models. Therefore we mainly focused on the accuracy of the model. And also used the
ROC curve for model selection.

Weighting the Models

We first focused on the default attributes of the models. Then we change the default
parameters of the models and checked the accuracy variance. As an example in random

57
forest model first we trained with the arbitrary number of trees (100 trees) and we plot the
Out of Bagged error Vs number of trees. Then we were able to identify the error distribution
while increasing the number of trees as shown in below figure.

Using this graph we selected number of trees as the Out-of-Bag Classification error is
minimized. Likewise we use weighting and parameter changing in models SVM,ANN etc.
This data set consists of class labels for each data tuple. Therefore we needed to apply the
supervised learning approaches. There are three types of supervised learning i.e classification,
regression, structured prediction. Our problem belongs to the classification type. Therefore
we try out machine learning models which were used for classification.
Classifiers also mainly divided into two types.
1. Lazy Learners (Eg: K-Nearest Neighbor)
2. Eager Learners (Eg: SVM,ANN,Random Forest etc)

Here we have focused more towards eager learners. Eager learners are capable of classifying
instances quicker than lazy learners.

58
3.2.1 Artificial Neural Network

Figure 66 : ROC Curve of ANN

Confusion Matrix of ANN

3.2.2 Perceptron

59
 Figure 67: ROC Curve for perceptron

Confusion Matrix of perceptron

3.2.3 Naive Bayes

Figure 68 : ROC Curve of Nive Bayesian

Confusion matrix of Nave Bayesian

60
3.2.4 Support Vector Machine

In SVM, it minimizes the empirical classification error and maximizes the geometric
margin simultaneously. So the SVM is identified as the maximum margin classifier. The
input vector is mapped to a higher dimensional space where a maximal separating hyperplane
is constructed. There are two parallel hyperplanes are constructed on each side of the
hyperplane that separate the data. An assumption is made that the larger the margin or
distance between these parallel hyperplanes the better the generalization error of the
classifier will be.

Kernel - Linear

Accuracy = 88.67%

61
ROC - SVM Linear kernel

Figure 69 : ROC Curve of SVM

Kernel - rbf

Accuracy : 92.45%

ROC - SVM- rbf kernel

62
3.2.5 Random Forest

Accuracy : 90.56%

Figure 70 : ROC Curve of Random Forest

3.3 Result Optimization

3.3.1 Feature Selection

To optimize the result we can use feature selection of the data set. For that we can use feature
selection using following algorithms
1. Principle Component Analysis (PCA)
2. Filter Methods
3. Forward Selection
4. Backward Elimination
5. Bi-directional Search
6. MRMr
7. inffs
8. relieff
9. mutinffs
10. fsv
11. laplacian

63
 12. mcfs
13. rfe
14. fisher
And get the best accuracy level for each model

3.3.2 Derive New Attributes.

We can derive new attributes from the existing attributes. Sometimes this method helps to
optimize the result because the derived attributes can be more dependable on class labels than
existing attributes.

3.3.3 Outlier Analysis

Using the outlier analysis methods we can identify and treat to the outlier. For this we can
using following methods
1. K-medoid
2. Density based clustering
3. Ridge Regression
4. occam razor principle
5. Cross-Validation
And get the best accuracy level for each model

3.3.4 Optimized Results

3.3.4.1 Artificial Neural Network

Five features are selected using relieff feature selection algorithm which gave
maximum accurate features from feature selection methods mentioned in feature
selection section.

Confusion Matrix

64
 ROC Curve

3.3.4.2 Naive Bayes

Five features are selected using mutinffs feature selection algorithm which gave maximum
accurate features from feature selection methods mentioned in feature selection section.

Confusion Matrix

ROC Curve

65
3.3.4.2 Support Vector Machine
By Applying Principal Component Analysis
SVM:Kernel - Linear

ROC Curve

66
Accuracy : 92.45%
Increased Accuracy = 3.78%

SVM:Kernel - rbf

ROC Curve

67
Accuracy : 96.22%
Increased Accuracy : 3.77%

3.3.4.3 Random Forest

By Applying Principal Component Analysis

68
Accuracy : 94.33%
Increased accuracy : 3.77%

3.3.5 Final Algorithm of the Research Component

69
70
Chapter 4 : SYSTEM DESIGN

4.1 Component Diagram

Figure 71 : Component Diagram of the System

This diagram shows main components of the system and their interconnections.

4.1.1 Image Acquisition Subsystem

Main purpose of this sub-system is capture images using the image sensor. Captured images
will be send to data extracting module.

4.1.2 Data Extracting module

Get color data of tea leaves and stems from captured image is the main purpose of this
module. Color data is converted to suitable color spaces by this module. Those data will be
used by decision making module and machine learning module.

71
4.1.3 Decision Making Module

This module do the stem/leaf classification. Threshold values given from machine learning
system is used to do the classification. After classification is done this module will decide to
trigger or not to trigger actuators.

4.1.4 Actuator Controller

This module control actuators according to decision making modules commands.

4.1.5 Controlling Module

Controller is responsible for coordinating subsystems, change behavior of the system

according to user inputs and monitor status of the system.

4.1.6 Human Interface Module

This module handle all interactions with the user. Get user inputs and communicate them to
controller and show current status of the system to the user.

4.1.7 Machine learning System

Machine learning system Collects data from data extraction module find threshold values
using training data set.

4.2 High Level Hardware Architecture

72
 Figure 72 ; High Level Architecture Diagram

This diagram shows the high level hardware architecture of the system. MicroZed and
IPC(Industrial PC) are main two hardware components of this system. Those two
components communicate using TCP over Ethernet.
MicroZed contains the Real Time Stem/Leaf Classification module. Fig. xx shows the
detailed architecture of the Real Time Stem/Leaf Classification module implemented in
MicroZed. In MicroZed there are 2 main modules Processing System(PS) and Programmable
Logic(PL). Those two modules communicate using AXI bus. AXI is a protocol designed to
do the communication between module in same System on Chip (SoC). PS and PL are in the
same chip. PS is a general purpose CPU and PL is a FPGA.
IPC contains the Controlling Module, Human machine interaction module and the Machine
Learning Module. Fig xx shows the detailed architecture of those modules.

73
4.3 Detailed Architecture

4.3.1 Real Time Stem/Leaf Classification Module

Figure 73 : Logic Diagram of the Classification Module

This diagram shows where each module is implemented in ZYNQ7020 SoC and how
external inputs and outputs connect to the system.

4.3.2 Controlling module

Figure 74 : Flow Diagram of Controlling Module

74
4.4 Data Flow Diagram

Figure 75 : Logic Diagram of H/W implementation

This diagram shows the path of data flow which is originated from the image sensor. Image
sensor capture image frames and send row pixel data to the system. Those pixel data are used
for real time data processing and do the non-real time machine learning.
Raw pixel data taken from image sensor converted to AXI-4 video stream before doing any
modifications to those data. AXI-4 video protocol is used to the video data transferring
between IPs inside the FPGA. After the stream protocol conversion RAW pixel data will be
converted to 24 bit RGB data using Color filter array interpolation algorithm. This
algorithm is used to create RGB data from Bayer Pattern color data. Then the RGB data will
be converted to YCrCb 422 and to the custom color space which derived to do the stem/leaf
classification. YCrCb 422 data will be send to RAM of the microZed via a DMA controller.
Then those data will be transferred to Machine Learning module using TCP over Ethernet.
YCrCb 422 format is used instead of RGB because it require 16bits for one pixel. After
converting to YCrCb 422 minor color information will be lost. But this conversion is required
otherwise bandwidth of Ethernet connection will be not enough to transfer all pixel data
without missing them. This transferring has to be done real time because the RAM of
MicroZed is not sufficient for large buffer size. Transfer frames will be stored in the IPC and
will be used do the learning and find optimal threshold values to do the stem/leaf

75
classification. Machine learning task is non-real time task. Then those threshold values will
be send to CPU of the MicroZed using TCP over Ethernet connection. CPU send those
parameters to decision making module through Data Exchange Module which is located in
FPGA part of the SoC via the AXI-Lite interface. Data Exchange Module send those values
to Decision Making Module using their communication ports.
Other destination of image data is the decision making module. RGB data converted to the
custom color space and send to the decision making module to do the decision making
process. Those pixel data will be used to check whether the capture frame is contain a object
and the to decide that the object is a stem or a leaf. To identify object pixels and to take the
decision, threshold values which are send from machine learning module will be used. After
taking the decision signal will be sent to Actuators to activate them to or deactivate them
based on the decision.

76
3.5 Decision Making Flowchart

Figure 76 : Decision Making Flow chart

This diagram shows the flowchart of decision making process. After arriving new pixel data
to the decision making module it check whether that pixel is an object pixel or a background
pixel based on threshold values. If values of all three channels are less than threshold values
that pixel is considered as an object pixel. If it is not an object pixel discard the pixel and wait
for the next pixel. If it is an object check whether it is a stem or a leaf based on threshold
values. If values of all channels are greater than respective threshold value pixel is
considered as a stem. If the pixel is a stem pixel increment stemPixel counter else increment
leafPixel counter. This process continues until the last pixel of the frame arrives. After the
last pixel is arrive stem percentage is calculated using stemPixel count and leafPixel Count. If
the stem percentage is higher than the 100-Sensitivity. This sensitivity value can be
configured by the user. If the sensitivity value is 50 all the objects which stem percentage is
greater than 50% will be considered as a stem. This approach is used because of some stems
are partially burned and some pixels of those stems have color values of leaves. If the user set
the sensitivity to 100 this algorithm will consider object as a stem if there is at least one stem
pixel in the object.After take the decision actuators will be activated if the object is a stem.
Finally all the counters will be reset and start the process from the beginning for the new
frame.

77
 Chapter 5 : IMPLEMENTATION

5.1 Real time Stem/Leaf Classification Module

5.1.1 Hardware used

MicroZed Board

ZYNQ 7020 SoC

85k Logic Cells

Dual Core Arm Processor

1 GB Ram

Gigabit Ethernet

Python 1300 camera and Vision Carrier Card

78
1.3 Mega Pixel

1000 FPS (Maximum frame rate)

Manual Focusing

5.1.2 Implementation Details

We used microzed board to implement frame capturing module, decision making module,
communication module (to communicate with IPC) and actuator controller. SoC chip used in
microzed board is ZYNQ7020. This SoC contain both FPGA (Programmable Logic) and
CPU (Processing System) modules. To minimize the delay, frame capturing, decision making
and actuator controller was implemented in FPGA. Because in FPGA tasks can be executed
parallelly. Communication module which handle the communication between FPGA and the
IPC (which contains HMI panel and machine learning module) was implemented in
Processing System of the ZYNQ7020 SoC. To do the communication between MicroZed and
the IPC Gigabit ethernet port of the Microzed is used. Theoretical speed of it is 1Gbps.
Capturing images was done using Python 1300 camera module. That camera module was
connected to microzed board using the Vision Carrier Card. Fig. xx shows the camera with
the vision carrier card. 3rd Party IP cores are used to get raw pixel data from the image sensor
and convert them to AXI4-video stream which contain RGB data of pixels. An IP core was
designed to do color space conversion do the decision making. Fig xx shows the block view

79
of the decision making IP. Another IP was designed to do the communication between PS
and the PL. That IP use AXI4-Lite protocol to communicate with the PS. Communication
module implemented in PS write data to PL and read data from PL using address of that ip. 4
registers are used to write data to PL from PS. Those registers are used to send threshold
values and other configuration data like monitor enabling to decision making module. 12
registers are used to read to from PL to PS. Those registers are used to read current threshold
values and other configuration data from the decision making IP. All those IPs are written
using verilog.

5.2 Machine Learning Module

80
 Figure 77 : Logic Diagram of ML Module

5.2.1.Pre Processing Unit

First remove the blank frames and duplicate frames which are send through controlling
unit. Duplicate frame is identified by frame differentiation and blank frame deleted by object
detection using otusu algorithm.

Then using Moore-Neighbor tracing boundary detection method, contours of the object
was detected and cropped by its bounding box. This function was based on the boundaries
function presented in the first edition of Digital Image Processing Using MATLAB, by
Gonzalez, R. C., R. E. Woods, and S. L. Eddins, New Jersey, Pearson Prentice Hall, 2004 [8].
Fig. 1 shows the final result

These images consist of both object and the background. For the calculations, it was
needed to extract the color data pixels of the object. Therefore, the backgrounds of the images
should be removed. For this, the background was identified through the Gray threshold level
using otsu algorithm [9] and then the image was transformed into a binary image
(background should be black). Next, the original image was multiplied by all the pixel values
by the corresponding pixel value of a binary image. Since the binary image background
contains 0s pixel value, the background of resulting image will be 0 [10]. Fig.2 shows the
resulting image.

81
The remove the noise using mean filtering.

Data Sample Separation

Stems and Leafs data samples

5.2.3.Feature Extraction Unit

This unit is responsible for color space conversion and extracted the mean value of
each frame.

5.2.4.Feature Selection and analytical unit

This unit is responsible of finding the most accurate threshold using histogram plot method.

82
83
5.3 Human Machine Interaction Module and Controlling Module

5.3.1 Hardware Used

Industrial PC

84
Intel Atom 1.86 GHz

2 GB RAM

8 Touch Screen

Windows 7 operating system

85
This module was implemented inside the Industrial PC unit. The role of this module is to
build the communication between each system modules and the communication between user
and the machine.
This module consists of two sub modules. They are HMI Panel and the Communication
module.
HMI Panel : This sub module was implemented to build the interaction between machine and
the machine operator. This interaction is done through an interface which is shown in below
figure.

86
 Figure 78 : UI of HMI Panel

The special characteristic of this machine is the machine operator can train the machine
learning model and can derive thresholds value as specific to the factory. To do this the
machine operator no need to have any engineering knowledge. He can simply perform this
through the above interface. He can select the category of the tea using the drop down menu
and he can assign the training time through this UI and need to click on start button.
These user inputs are sent to the communication module. Then it calculates the number of
frames that can capture from the camera according to the frame rate and send it to the FPGA
module. Then machine operator need to enter stem and leaf samples within the assigned time
by the machine. The entered samples are capture by the camera in the FPGA module and
those captured images are sent to the communication module as binary image files through a
TCP connection.
After then those binary files are converted in to png image files and send them to the training
module.
After the training there are two thresholds are sent to the communication module. Then the
communication module store this two thresholds values permanently and send it to the FPGA
module.

87
After the training the machine operator can view the captured images by simply clicking on
Display button and also machine operator can view the statistics graphs by simply clicking
on the Statistic button.

88
Chapter 6: DISCUSSION
From starting with the native which are very commonly used color spaces our experiments
broaden to the extent to which is creating new hybrid color space which gave similar
accuracy to the native multilayer neural network. So after that our experiment is went through
the above mentioned textural and shape features for intending to create more optimized
hybrid color texture and shape space. But fortunately we ended up with following low
accuracies.

So as the 1st version of our product we limited our scope to hybrid color space until we found
the new texture value.

A. Partially burned Tea particles

B. Light Color Tea Leaves

89
C. Low resolution images

As to minimize errors comes under A and B we use histogram plotting method and to
remove the
Future work:
Tea Categorizing
In tea industry , tea is categorized under different grading levels. Those grading levels
are decided based on the size and the shape of the tea leaf. In current situation they use seiver
is used to do this grading process and this is done by labourers. The current process is time
consuming and required large space. And also it damage to the tea particles and it affect to
the quality of the tea. Therefore as the solution we propose to do this process in 95%
confident level using a computer vision based system.

90
Chapter 7: CONCLUSION
As a main part of this project we proposed the research which is a new criterion for tea stem
and leaf classification in the field of color image analysis.Through this research, the
advantages of using a color pixels classification algorithm in a Hybrid Color Space with
derived color components were presented. This new color space depends on the application.
This introduced a new method which is a lightweight solution for implementation and more
accurate than feed forward neural network. To increase the accuracy of result evaluation the
characteristic curve of histogram was used instead of histogram overlapping calculation
method which reduces the dependency on training sample and the accuracy level.

And as a production level we improve the accuracy of the system including using above
mentioned new color space and various other image processing and image enhancement
techniques and also we introduce dynamic thresholding by implementing separate training
module to the current tea industry.

91
 REFERENCES

[1] Alireza Pazoki and Zohreh Pazoki (2011).Classification system for rain fed wheat grain
cultivars using artificial neural network, in: proceeding of African Journal of Biotechnology
Vol. 10(41), pp. 8031-8038, 3 August, 2011 .

[2] Alireza Pazoki, Zohreh Pazoki and Behzad Sorkhilalehloo (2013).Rain Fed Barley Seed
Cultivars Identification Using Neural Network and Different Neurons Number, in:
proceeding of World Applied Sciences Journal 22.(5): 755-762, 2013

[3]. Ankur M Vyas ,Bijal Talati and Sapan Naik, Colour Feature Extraction Techniques
of Fruits, International Journal of Computer Applications (0975 8887), Sardar Vallabhbhai
Patel Institute of technology,2013.

[4] A. R. Pazoki , F. Farokhi , Z. Pazoki (2014).CLASSIFICATION OF RICE GRAIN

VARIETIES USING TWO ARTIFICIAL NEURAL NETWORKS (MLP AND NEURO-
FUZZY), in: proceeding of The Journal of Animal & Plant Sciences. Pp.336-343.

[5] Chang Chun1,Chen Yiqun1,Xiao Hongru1,Ren Shougang2,Song Weidong1(1.Nanjing

Research Institute for Agriculture Mechanization,Ministry of
Agriculture,Nanjing,210014,China; 2.Nanjing Agricultural
University,Nanjing,210095,China) (2013).Smart fresh tea sorter with neural network image
analysis in Journal of Changshu Institute of Technology.

[6]Cheng-Jin Du, Da-Wen Sun(2006). Learning techniques used in computer vision for food
quality evaluation.

[7] "Color-Based Segmentation Using K-Means Clustering - MATLAB & Simulink

Example", In.mathworks.com, 2016. [Online]. Available:
http://in.mathworks.com/help/images/examples/color-based-segmentation-using-k-means-
clustering.html. [Accessed: 26- Aug- 2016].

[8] Desai, Vasishth P., and Arjav Bavarva. "Image Processing Method for Embedded Optical
Peanut Sorting." International Journal of Image, Graphics and Signal Processing 8.2 (2016):
20.

[9]Domenico, S., & Gary, W. (1994). Machine vision and neural nets in food processing and
packagingnatural way combinations. In Food processing automation IIIProceedings of
the FPAC conference

[10]Du, C.-J., & Sun, D.-W. (2004). Recent developments in the applications of image
processing techniques for food quality evaluation. Trends in Food Science & Technology,
15(5), 230249.

[11]Gunasekaran. S. (1996.) Computer vision technology for food quality assurance. Trends
in Food Science & Technology , 7.

92
[12] "Global image threshold using Otsu's method - MATLAB graythresh",
In.mathworks.com, 2016. [Online]. Available:
http://in.mathworks.com/help/images/ref/graythresh.html. [Accessed: 26- Aug- 2016].

[13]Kolacz, Jacek. "Advanced sorting technologies and its potential in mineral processing."
AGH Journal of Mining and Geoengineering 36.4 (2012): 39-48.

[14] Pearson, Thomas. "High-speed sorting of grains by color and surface texture." Applied
engineering in agriculture 26.3 (2010): 499-505.

[15] "Properties Of Gray-Level Co-Occurrence Matrix - MATLAB Graycoprops".

In.mathworks.com. N.p., 2016. Web. 26 Aug. 2016.

[16]Shuihua Wang & Xiaojun Yang.(2015). Identification of Green, Oolong and Black Teas
in China via Wavelet Packet Entropy and Fuzzy Support Vector Machine

[17] S. Somatilake and A.N.Chalmers, Food Colour Measurement, BE(Hons) Engineering

Research Project Report, Manukau Institute of Technology,2007.
[18]Shuihua Wang & Xiaojun Yang.(2015). Identification of Green, Oolong and Black Teas
in China via Wavelet Packet Entropy and Fuzzy Support Vector Machine

[19] V. Vapnik (1995). The Nature of Statistical Learning Theory. NY: Springer-Verlag

93