Post-processing of JPEG image using MLP

Fall 2003 ECE539 Final Project Report Department of Electrical and Computer Engineering University of Wisconsin-Madison

_______________ Data Fok Submitted to: Professor Hu Dec 19th 2003

1. Introduction A picture is worth than a thousand words. We all should agree with that. There is an increasing demand of graphics in computer industry. However, the large file size is always a concern to the user. Although the widely adopted compression standard JPEG can pack an image to a rather small size, the image quality is not always acceptable. Especially in low bit rate (bpp) environment, the blocking artifacts affect the image quality very much. Other standard like JPEG 2000 can handle image in low bit rate very well, yet its popularity is far less than JPEG. Thus, this projects aim is to implement a method to improve the quality of the JPEG by eliminating the blocking artifacts. 2. Approach Blocking artifact is inherited from the coding scheme of JPEG, as the image will divide into blocks (typically 8x8) before coding. Different block will be processed on its own. Thus, the intensity level of the border pixels might have a large gradient as illustrated in Figure 1.

Figure 1: the intensity gradient between blocks. Two approaches were tested in the project to change the gradient of the block border pixels to the original gradient. Image quality improvement was then justified by peak signal to noise ratio (PSNR) and human eyes. 255 PSNR = 10 log 10 MSE

Mean Square Error, MSE =

[ I ( x, y) I( x, y )]
x, y

( MN ) 2

where M,N are the height and width of the image respectively. Multi-Layer Perceptron First approach is to use multi-layer perceptron model with 15 inputs and 6 outputs. The 15 inputs are the gradient of the pixels near the block border in the R,G, B plane. i.e. (refer to Figure 1), input 1 is the different between I 1 and I 2 in the R plane. input 2 is the different between I 1 and I 2 in the G plane. input 3 is the different between I and I in the B plane.
1 2

input 4 is the different between I 2 and I 3 in the R plane. input 5 is the different between I 2 and I 3 in the G plane. input 15 is the different between I 5 and I 6 in the B plane. The 6 outputs are the intensity different of the border pixels between the original image and the JPEG image. i.e. (refer to Figure 1), output 1 is the different between I 3 and I 3 in the R plane. output 2 is the different between I 3 and I in the G plane.
3

output 3 is the different between I 3 and I 3 in the B plane. output 4 is the different between I 4 and I in the R plane.
4

output 5 is the different between I 4 and I 4 in the G plane. output 6 is the different between I 4 and I 4 in the B plane. Experiments are performed in various images under different numbers of levels, numbers of hidden nodes, learning rate and momentum. Polynomial curve fitting Second approach is to use first order polynomial curve fitting apply to the block border ~ ~ pixels I 2 , I 3 , I 4 , I 5 of the JPEG image, and then replace I 3 , I 4 with the estimated I 3 , I 4 from the polynomial curve fitting. This was performed as a control experiment to the MLP postprocessing.

3. Experiments, Results & Discussions A program named jpegPost.m is written in matlab to model a back-pack propagate MLP and to implement the polynomial curve fitting. The MLP part of the program is based on the bp.m and bpconfig.m written by Professor Hu. Setting up the MLP Different numbers of layer of the MLP were tested. The result showed that a three level model is enough, as the PSNR and the quality do not improve much in adding more level.

Structure: 15-5-6

15-5-5-5-6

Different numbers of hidden nodes were tested. The result showed that 5 hidden nodes are enough to cheat the human eyes.

Structure: 15-5-6

Structure: 15-8-6

Different learning rates were tested. The result showed that 0.01 is sufficient.

Learning rate = 0.01

Learning rate = 0.05

Different momentums were applied. The result showed momentum = 0.5 yield the best result.

Momentum = 0.7

Momentum=0

Training sets were extracted from different images. One set of training data was extracted from each image. Then, they were fed to the MLP with the optimal structure obtained above. Comparison is made by the PSNR and the image quality from human eyes.

Experiment #1: grayscale image (train and test with the same image)

JPEG (0.14 bpp) PSNR = 41.2044 (dB)

MLP postprocessed PSNR = 40.2514 (dB)

Polynomial Fitting PSNR = 40.1896 (dB)

From PSNR and human eyes, we can see the quality of the image decreases after the MLP postprocess and the polynomial fit.

Experiment #2: color image (train and test with the same image)

JPEG (0.18 bpp) PSNR = 38.2464 (dB)

MLP postprocessed PSNR = 37.9718 (dB)

Polynomial Fitting PSNR = 37.6817 (dB)

The blocking artifacts decrease a little in the MLP image. However, the PSNR also decrease in the MLP image. It proves that the evaluation of the image quality is different in PSNR and human eyes. Experiment #3: grayscale image (train with a high bpp image, test with a low bpp image) Training JPEG bit rate = 0.255 bpp

JPEG (0.085 bpp) PSNR = 39.5696 (dB)

MLP postprocessed PSNR = 39.6552 (dB)

Polynomial Fitting PSNR = 39.2868 (dB)

Blocking artifact decreases by the MLP postprocess from human eyes. PSNR also increases after MLP postprocess. Polynomial fitting decreases blocking artifact also but it blurs the image which lead to the decrease in PSNR. Experiment #4: color image (train with a high bpp image, test with a low bpp image) Training JPEG image bit rate = 0.374 bpp

JPEG (0.065 bpp) PSNR = 37.4064 (dB)

MLP postprocessed PSNR = 37.3664 (dB)

Polynomial Fitting PSNR = 37.1856 (dB)

The case of color image is not as good as the one in grayscale image. Both the human eyes quality and the PSNR decreased after MLP postprocessing. On the other hand, polynomial fitting did a good job in decreasing the block artifact, yet the PSNR still decreases.

Experiment #5: train with a high bpp grayscale image, test with a low bpp color image Training JPEG image bit rate = 0.255 bpp

JPEG (0.065 bpp) PSNR = 37.4064 (dB)

MLP postprocessed PSNR = 37.4312 (dB)

Although PSNR is increased in the MLP postprocessed image, it is obvious that the blocking artifacts are still there. Experiment #6: train with a high bpp color image, test with a low bpp grayscale image Training JPEG image bit rate = 0.374 bpp

JPEG (0.085 bpp) PSNR = 39.5696 (dB)

MLP postprocessed PSNR = 39.125 (dB)

Not even blocking artifact was not improved; the MLP postprocess introduced more noise to the image making the PSNR decreases. Moreover, as the MLP was trained by color image, there was some color shift in the MLP postprocessed image. 5. Conclusion From experiment #3, we can see that the increases in the image quality and PSNR suggest the possibility of training the MLP by a high bit rate image can give a set of weight that can eliminate the blocking artifact. It gives an insight that further fine adjustments of the MLP structure might increase the image quality even in high bit rate image. As blocking artifacts mainly appears in low bit rate image, experiment #3 proves the ability of the MLP postprocess to decrease the blocking artifacts, which fulfill our main goal. However, experiment #4, 5 and 6 show that the MLP postprocess only work well if the train data is from grayscale image. It implies that the current MLP structure may not fit the training data from color image. Further study can be done on investigating a suitable MLP structure for color training data. 6. References

[1] W. B. Pennebaker and J. L. Mitchell, (1992) JPEG Still Image Compression Standard. New York: Van Nostrand Reinhold. [2] Martin Boliek, Charilaos Christopoulos, Eric Majani, (2000) JPEG 2000 Image Coding System, ISO/IEC JTCI/SC29 WGI, http://www.jpeg.org/CDs15444.html [3] Guoping Qiu, (2000) MLP for Adaptive Postprocessing Block-Coded Images. IEEE Transactions On Circuits And Systems For Video Technology, Vol. 10, No. 8, December 2000

Appendix A selected part of the test result is attached.