Image Segmentation using Gaussian Mixture Models

Rahman Farnoosh, Gholamhossein Yari and Behnam Zarpak

Department of Applied Mathematics, University of Science and Technology, 16844, Narmak,Tehran, Iran Abstract. Recently stochastic models such as mixture models, graphical models, Markov random elds and hidden Markov models have key role in probabilistic data analysis. Also image segmentation means to divide one picture into different types of classes or regions, for example a picture of geometric shapes has some classes with different colors such as circle, rectangle, triangle and so on. Therefore we can suppose that each class has normal distribution with specify mean and variance. Thus in general a picture can be Gaussian mixture model. In this paper, we have learned Gaussian mixture model to the pixel of an image as training data and the parameter of the model are learned by EM-algorithm. Meanwhile pixel labeling corresponded to each pixel of true image is done by Bayes rule. This hidden or labeled image is constructed during of running EM-algorithm. In fact, we introduce a new numerically method of nding maximum a posterior estimation by using of EM-algorithm and Gaussians mixture model which we called EM-MAP algorithm. In this algorithm, we have made a sequence of the priors, posteriors and they then convergent to a posterior probability that is called the reference posterior probability. So Maximum a posterior estimation can be determined by this reference posterior probability which will make labeled image. This labeled image shows our segmented image with reduced noises. This method will show in several experiments. Keywords: Bayesian Rule, Gaussian Mixture Model (GMM),Maximum a Posterior (MAP), Expectation-Maximization (EM) Algorithm, Reference Analysis. PACS: 29. 85. +c, 87. 58. Mj , 07. 05. Tp

INTRODUCTION
Automatically image processing means image segmentation(i.e. dividing an image into different types of regions or classes), recognizing of objects and detecting of edges,etc by machine. All of these can be done after segmentation of a pictures. So image segmentation is the most important image problems. In addition noise removing and noise reduction of pictures always are important in classical image problems. In this paper, we do both segmentation and noise reduction with a probabilistic approach. There are many mathematical and statistical methods for image problems, but this paper argues about GMM as a general Gaussian distribution, EM-algorithm and Bayesian rule. But Bayesian framework usually has many difculties because the posterior probability has complex form. So we must use Markov Chain Monte Carlo algorithms or Variational methods with high computing to nd MAP estimation. These methods have worked well in the last decades [1,5]. In this paper, a new numerically EM-MAP algorithm base on Bayesian rule has constructed. We have used Bernardos theory about reference analysis [2] in practice and in image segmentation. it means that in reference analysis a sequence of priors and

posteriors are made and they convergent to a posterior probability which is called reference posterior probability. In this paper, we have used this idea. So we have modied EM-algorithm for our image segmentation. After nding reference posterior probability, MAP estimation and pixel labeling can easily make segmented of image. This paper organized as follows. It rst reviews of GMM and its properties. Then we introduce the EM-MAP algorithm for learning parameters for a given image as the training data. Choosing initial values of EM-algorithm have discussed in next section. EM -MAP algorithm never can not convergent without choosing a suitable starting points. But these initial information are made by histogram of image. Finally we show some experiments with simulated images.

GAUSSIAN MIXTURE MODELS

Image is a matrix which each element is a pixel. The value of the pixel is a number that shows intensity or color of the image. Let X is a random variable that takes these values. For a probability model determination, we can suppose to have mixture of Gaussian distribution as the following form f x

i 1

piN x i

i2
1 pi

(1) 1 (2)

Where k is the number of regions and pi N i i2

0 are weights such that k i

2 x 1 i exp 2i2 2pi

Wherei i are mean, standard deviation of class i. For a given image X, the lattice data are the values of pixels and GMM is our pixel base model. However, the parameters are 2 2 p1 pk 1 k 1 k and we can guess the number of regions in GMM by histogram of lattice data. This will show in experiments.

EM-MAP ALGORITHM
There is a popular EM algorithm for GMM in several papers [3,6]. We modify it to the following algorithm with the name of EM-MAP algorithm. 2. Initialize: 0 3. (E-step) 1. Input:Observed Image in a vector x j j
p0 1

12

n and i 1 2
0

k labels set

p0 10 k

2 k 0 1
j

2 k

pr1
ij

Pr1 i x

pr N x j ir i2 i f x j

(3)

4. (M-step)

pi r1 r1
i

1 n n j

j 1

prj i

(4)

n pi r1

r1 x 1 pi j j

(5)

r1 i2

n j

r1 x r1 i 1 pi j j
n pi r1

(6)

5. Iterate steps 3 and 4 until an specify error i.e. i e2 i 6. Compute pl j ArgMaxi pjf inal j 1 2 i

(7)

7. Construct labeled image corresponding of each true image pixel. This EM-MAP algorithm is a pixel labeling base method such that the labeled image shows each segment or object by different type of labels. Note that formula (3) is Bayes rule, pr is discrete prior probability in stage r and p r1 is discrete posterior probability i ij in the next stage. In this algorithm, we make a sequence of prior and then posterior until to get convergence. The labeled images chooses with MAP of the nal posterior.

CHOOSING A PRIOR WITH MAXIMUM ENTROPY PROPERTY

In EM-MAP algorithm, there are some difculties. How can we choose? the number of classes the weights the means the variances

A practical way that we can guess the prior of parameters is drawing the histogram of the observed image. Not only image histogram gives us four above parameters for using initial values in the EM algorithm, but also this extracted information usually has maximum entropy. This claim has shown in experiments. Besides, the nal posterior probability will get a stable entropy. We also compute the number of misclassications of results. This shows that how much our algorithm is well.

EXPERIMENTS
In rst example, we make three boxes in an image and add it white noise. The observed image and its histogram in the gure 1 and gure 2 are shown. Information extracted

1000 900 5 800 700 10 600 500 15 400 300 20 200 100 25 5 10 15 20 25 30 35 40 45 50 0 0 50 100 150 200 250

FIGURE 1. a)The Observed Image of Boxes,b)Histogram of the Observed Image

0.78

0.76

0.74 10 0.72 15 0.7 20 0.68

25 5 10 15 20 25 30 35 40 45 50

0.66

10

FIGURE 2. a)Labeled image with reduced noises,b)Entropy curve in each iteration

from the histogram are k 4 p0 0 0320 0 1344 0 0576 0 7760 or empirical prob40 75 210 220and 0 100 100 100 100. ability with entropy 0 7411 0 The stopping time occurs when L-2 norm of absolute error has very small value. After running EM-MAP, we had ten-times iteration in gure 2 and the entropy of each iteration which goes to a stable or maximum case. We see in segmented image that Blue l, cyan 2,Yellow 3 andRed 4. There is 0.0008 percentage misclassication that is only one red instead of yellow pixel is wrong. In this example, pixel labeling and noise removing are well done. In the second example, some different shapes such as circle, triangle, rectangle, etc have considered. The observed image and its histogram in gure 3 have shown. For EM-MAP nding, we need to get initial values by histogram of this image. We

3000 5 10 15 2000 20 25 30 35 40 500 45 50 10 20 30 40 50 60 70 80 90 100 0 50 1000 1500 2500

50

100

150

200

250

FIGURE 3. a)The Observed Image of Circle,b)Histogram of the Observed Image

0.9 5 10 15 20 25 30 35 40 45 50 10 20 30 40 50 60 70 80 90 100 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5 0.45 0.4 0 5 10 15 20 25

FIGURE 4. a)Labeled image with reduced noises,b)Entropy curve in each iteration

choose k 3 p 0 08 0 34 0 58 as empirical probability or relative frequency, 38 63 88 and 12 12 12 with norm of error less than 0.01. In gure 4, we made 3 classes Blue 1,Green 2 andRed 3. There are only 25 missing of classications or percentage of 0.005. If in EM-MAP algorithm, we compute the entropy of the posterior probability in Each stage of iteration, this entropy will be decreasing to reach a stable form. The third example is more complex. The number of components is great. In addition, there is dependent noise in image which noise reduction in this case is more difcult. The true image and observed image have shown in gure 5. Again,information extraction can nd by drawing histogram of observed image, it means

10

10

15

15

20

20

25

25

30 5 10 15 20 25 30

30 5 10 15 20 25 30

FIGURE 5. a)The True Image of Partition,b)The Observed Image of Partition

200 180 5 160 140 120 15 100 80 60 40 20 0 0 1 2 3 4 5 6 7 8 9 25 20 10

30 5 10 15 20 25 30

FIGURE 6. a)Histogram of Image,b)The Segmented Image of Partition

k p

10 0 0977 0 1377 0 0625 0 0693 0 0361 0 0361 0 1182 0 1904 0 1572 0 0947 1 2 2 5 3 5 4 5 5 5 6 7 7 5 8 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5

The results have shown in gure 6 with 20 times iteration. In gure 7 the results are shown in 50 times iteration and entropy has reached in a stable case.

2.19

2.185

10 2.18 15 2.175 20 2.17 25 2.165 30 5 10 15 20 25 30 2.16 0 10 20 30 40 50

FIGURE 7. a)Labeled image with reduced noises,b)Entropy curve in each iteration

CONCLUSIONS
In this paper, we make a new numerical EM-GMM-Map algorithm for image segmentation and noise reduction. This paper is used Bernardos idea about sequence of prior and posterior in reference analysis. We have used known EM-GMM algorithm and we added numerically MAP estimation. Also the initial values by histogram of image have suggested which is caused to convergence of EM-MAP method. After convergence of our algorithm, we had stability in entropy. EM-algorithm is iteration algorithm of rst order [3], so we had slow convergence. We used acceleration convergence such as Steffensen algorithm to have the second order convergence. But later we note that in EMMAP method, the number of classes will reduce to real classes of image. Finally, EMalgorithm is linear iteration method, so our method is suitable for simple images. It is important to note that "for segmentation of real images, the results depend critically on the features and feature models used" [4] that is not the focus of this paper.

ACKNOWLEDGMENTS
We have many thanks to prof.Mohammad-Djafari for his excellent ideas.

REFERENCES
1. C.Andrieu, N.D.Freitas, A.Doucet, M.I.Jordan, An Introduction to MCMC for Machine Learning, Journal of Machine Learning, 2003, 50, pp. 5-43. 2. J.M. Bernardo and A.F.M. Smith, Bayesian Theory, John Wiley & Sons, 2000. 3. L.Xu, M.I.Jordan, On Convergence Properties of the EM Algorithm for Gaussian Mixture, Neural Computation, 8, 1996, pp. 129-151. 4. M.A.T.Figueiredo, Bayesian Image Segmentation Using Gaussian Field Prior, EMMCVPR 2005, LNCS 3757, pp. 74-89.

5. M.I.Jordan, Z.Ghahramani, T.S.Jaakkola, L.K.Saul, An Introduction to Variational Methods for Graphical Models, Journal of Machine Learning, 1999, 37, pp. 183-233. 6. R.Farnoosh, B.Zarpak, Image Restoration with Gaussian Mixture Models, Wseas Trans. on Mathematics, 2004, 4, 3, pp.773-777.