Analysis on Color Filter Array Image Compression Methods

Sung Hee Park

Electrical Engineering Stanford University Email: shpark7@stanford.edu

Albert No
Electrical Engineering Stanford University Email: albertno@stanford.edu

AbstractIn most imaging systems, image compression is carried out after demosaicing process and compression algorithms usually have focused on compressing threechannel color images. Recently, several image compression methods are proposed to directly handle CFA images that are captured before color interpolation. These CFA compression schemes exploit the fact that CFA images do not contain the redundant information introduced by demosaicing. In this paper, we will compare the performance of various CFA compression methods and describe how visual qualities are affected by the choice of demosaic algorithm. Index TermsCFA image compression, lossy compression, lossless compression

I. I NTRODUCTION The digital color imaging systems generally use a single image sensor to capture full color images using a color lter array (CFA). A color lter array enables each pixel to capture the intensity of light with different color spectrum. The most common design of CFA is the 22 Bayer pattern [1] which consists of two greens, one red, and one blue component. Since each pixel has only one color component, demosaic operation is performed to interpolate all three color components to estimate full color RGB images. Then, the demosaiced image can be sent to image display devices for display. Figure 1(a) shows the overall diagram of most widely used digital imaging system. When images are transmitted or stored from the image capture module, the connection between two systems is almost always bandwidth-limited and becomes the bottleneck of whole system performance. Thus, efcient image compression methods are employed to send less amount of data to the link and they are commonly applied after demosaicing operation [2], [3]. Recently, an alternative imaging system is proposed as shown in Figure 1(b) [4][7] . The main difference of this system is that it directly compresses CFA image and demosaic operation is done after decompression. By compressing the CFA data, the compressor can deal with only one-third of data compared to the case for compressing three-channel RGB images. Thus, it does not have to ght with the redundancy information that

(a)

(b) Fig. 1. Illustration of digital color imaging systems. (a) Conventional RGB image compression. (b) CFA image compression.

are introduced by color interpolation, which enables more efcient compression. In addition, CFA compression scheme ofoads computational burden to the decoder side that usually has much more computation power and memory. Not only reducing the load on the compressor, it also passes whole demosaic stage to the other side. Many image capturing systems have very strong constraints in power and processing time in order to work as real-time mobile devices. Thus, the proposed pipeline has great advantage in minimiz-

(a)

(b)

Fig. 2.

Illustration of the rearranging structure scheme.

ing the work load on power-limited image capturing module. From the motivations mentioned above, various approaches are suggested to compress CFA data. We would like to go through various approaches in CFA data compression and analyze their performances in the following sections. II. E XISTING CFA C OMPRESSION M ETHODS There are several different color lter arrays, but the Bayer CFA [1] is most popular. In this CFA, the green lters are placed in a quincunx pattern (interlaced), and red and blue lters are in the remaining locations. The compression methods assume the implementation precision is 8-bit and it can be expanded to other precisions. A. Naive Method The naive method is simply applying the existing compression algorithm, such as JPEG, to the original CFA images. Since the CFA image does not look like natural images and contains very specic Bayer structure, the JPEG compressor is not expected to work very well. B. Rearranging Structure
Fig. 4. Illustration of color conversion for the Bayer pattern. (c) Fig. 3. The rearranging structure scheme: (a) Simple merging, (b) Structure separation, (c) Structure conversion.

In order to handle the problem of the naive method, 128.6 0 25 65.5 Y 0 Y lr 0 128.6 25 65.5 several solutions have been proposed [4]. One main Y lr = 37.1 37.1 112 37.8 Y 128 + 0 Cb Cb approach is rearranging structure methods. Figure 2 46.9 46.9 18.2 112 128 Cr Cr shows basic pipeline of this method. Each color component of CFA image is separated into different chan- C. Filtering and Conversion nels, and reshaped into the rectangular form so that Another approach is to apply ltering followed by regular compression algorithm can deal with. However, downsampling [7]. The CFA image includes more high handling green patterns is not trivial because of its frequency components than the full color image data, quincunx pattern and numerous ways are proposed to which causes compression artifacts. By applying the reshape it into rectangular form. low pass lter, we can reduce these high frequency Figure 3 shows several rearranging structure meth- components. Figure 5 shows several low pass lters ods. Figure 3(a) shows the simple merging method which can be applied to CFA images. These methods which only merges every neighboring odd and even are different from the rearranging case because they are columns. Figure 3(b) shows the structure separation irreversible algorithms. Thus, we cannot apply them method which separates quincunx array into two ar- into lossless case. rays. Figure 3(c) describes the structure conversion III. I DEAL E NTROPY C ODING E STIMATION FOR method which converts two columns into a single L OSSLESS C ASES column by applying weighted-averaging. These methods can be also applied in different color Now we are going to nd theoretical bounds of spaces, such as both RGB and YCbCr. Even though rate in lossless case by considering the ideal entropy

the CFA image contains only one color component per pixel, we can still apply color conversion by considering a 22 block at the same time as described in Figure 4 [5]. The color conversion matrix from RGB to YCbCr can be represented as the following. ul ul

(a)

(a) (b) Fig. 6. The prediction methods : (a) Simple prediction, (b) Adaptive prediction.

(b) Fig. 5. The low-pass lters used in the ltering and conversion methods.

coding. These bounds will give idea about how much redundancy is removed by each compression scheme. Each method can be easily implemented by adopting efcient encoders that closely achieve theoretical bounds. A. Single Pixel Entropy Coding First, we can nd the entropy of each pixel. This case considers neither spatial nor color correlation between pixels. Thus, this entropy coding is expected to contain much redundant information. B. Joint 2 2 Bayer Block Entropy Coding In order to consider the color correlation between pixels, we can consider evaluating joint entropy of 2 2 Bayer block as following. However, as the order of joint entropy increases, the memory requirement explodes exponentially for 8-bit precision. Thus, we used the following to get approximate values for 4th order. H(Y1 , Y2 , Cb, Cr) = H(Y1 , Cb, Cr) + H(Y2 |Y1 , Cb, Cr) H(Y1 , Cb, Cr) + H(Y2 |Y1 ) Actually, it is reasonable to assume that Y2 and Cb, Cr components are independent given Y1 because the luminance component will be mainly dependent on the neighboring luminance value rather than chrominance value. C. Simple Prediction The simple prediction method is trying to use spatial correlation to reduce the rate. We applied the one used in lossless JPEG case as shown in Figure 6(a). The prediction is done on each color channel separately,
Fig. 7. Experimental results of lossy CFA image compression.

with two greens separated as the structure separation method described in section II-B. D. Adaptive Prediction To do better prediction, we tried a context matching based prediction [5]. This method predicts the value by evaluating the weighted sum of four candidate pixels that are shown in Figure 6(b). The weights are determined by how the support regions around the pixel are similar to the one around the pixel that you are predicting. The candidate pixel with the similar support will get higher weight. IV. E XPERIMENTAL R ESULTS FOR L OSSLESS CFA C OMPRESSION M ETHODS In this section, we will apply various lossless CFA image compression methods to 24 color images in the Kodak image set [8]. In the lossless case, reconstructed images will be exactly the same as the input images and only the rate, which can be measured in bits per pixel, will matter. Figure 7 shows the rate for several lossless compression methods. The red bars represent the cases of applying JPEG and PNG directly to the CFA image. We can see that these cases do not compress well

Three color channels are equally weighted in distortion calculation. A. Comparison between CFA Images Figure 9 shows the rate-PSNR curves evaluated by comparing CFA images before and after the compression. As we expected, the naive method, which applies JPEG directly to the CFA data, shows the worst tradeoff line. All other CFA methods usually work better than the naive method in most region. The ltering and conversion methods tend to give lower PSNR in high rate since the high-frequency details lost from low-pass ltering can not be recovered. B. Comparison between Demosaiced Images Since it is hard to evaluate the visual qualities between CFA images, we only compared with a quantitative measure in section V-A. In practice, it is the nal RGB image that the users will see and evaluate the visual qualities. Thus, we apply demosaic algorithm to both input CFA image and the reconstructed CFA image to get RGB images as described in Figure 8. We choose three demosaic algorithms to test: simple bilinear interpolation and two state-of-the-art demosaic methods, adaptive homogeneity method [9] and adaptive frequency domain method [10]. Before comparing the performance of each method, lets see what kinds of visual artifacts you get after CFA compression and demosaicing. When the naive method is used, we get very severe color artifacts because the JPEG compression is messing up inherent CFA structure as shown in Figure 11(a). The error in a pixel spreads to neighboring pixels, which causing undesired big color blobs. However, this artifact is not present in other CFA compression methods since they handle each channel separately. But, as in Figure 11(b), they still show blocking artifact at low rate because the CFA compression methods are designed based on JPEG compressor. The spatial CIELAB [11] is a measure that can be used to evaluate perceptual difference between two images. Figure 10(b) shows that the naive method gives signicant loss in visual quality while other methods perform relatively similar. Figure 10 shows the rate-distortion curves of each CFA compression methods for three different demosaic methods. In addition, the conventional method, which applies demosaic rst and then do RGB JPEG compression, is also drawn as the bright red line with circles. It is interesting to see that in Figure 10(a), the CFA methods work better than the conventional method for bilinear demosaicing while it is not true for the other sophisticated demosaic algorithms. It results from the fact that the conventional method does not work well for bilinear case as well as CFA methods work better for bilinear case.

Fig. 8.

The pipeline for evaluating error component.

Fig. 9. Experimental results of lossy CFA image compression. The error is estimated between input and reconstructed CFA images.

for CFA images. The blue bars represent the rearranging methods, such as simple merging, structure conversion and structure separation and they clearly work better than the naive methods. Especially, the structure conversion method performs the best among the rearranging methods. The green bars represent ideal entropy estimations. The single pixel entropy coding is the reference case and joint 2 2 Bayer entropy coding performs better, since they consider color correlation. Also, prediction methods show improvements by exploiting spatial correlation, and it is clear that adaptive prediction works better than simple prediction. V. E XPERIMENTAL R ESULTS FOR L OSSY CFA C OMPRESSION M ETHODS In this section, we will discuss about lossy CFA image compression cases. Six different CFA methods are implemented based on lossy JPEG compressor. By changing the Quality parameter in JPEG, the rate-distortion curves are obtained and the results are averaged through all 24 images in the Kodak set.

(a)

(b) Fig. 10. Experimental results of lossy CFA image compression. The error is estimated between demosaiced RGB images.

First, lets explain why bilinear demosaic method is not good for the conventional compression way. The bilinear method is a simple linear operation that causes severe color artifacts and zipper effects along strong edges. These artifacts result in very high frequency components in chrominance channels. However, these components are easily lost during the compression stage because JPEG assumes less high-frequency values in color and performs aggressive subsampling in the chrominance channels. This does not happen when complicated demosaic is used since those methods effectively suppress color artifacts from the beginning. In addition, we can explain why CFA compression methods give higher PSNR for bilinear interpolation. Good demosaic algorithms try to recover high frequency data from green channels and spread them to other color channels so that edges can be reconstructed sharply without causing color artifacts. However, the compression stage introduces various artifacts including blocking, which confuses demosaic algorithm. The demosaic algorithm can not tell whether the edge is from the image itself or it is an false edge caused by compression artifact. Thus, it amplies compression artifacts which results in lower PSNR.

As a result, the performance between the conventional method and CFA image compression methods is highly dependent on the choice of demosaic algorithm used afterwards. Many CFA methods [4], [7] assume bilinear demosaicing is used and claim victory over RGB image compression, but it may not be the case if we select better engineered demosaic methods. VI. C ONCLUSIONS We have shown that CFA data can be efciently compressed by using existing image compression algorithms such as JPEG. However, it is important to note that the overall performance of CFA compression methods is highly dependent on the choice of demosaic algorithm. When simple bilinear demosaic method is used, CFA image compression results in better images compared to conventional compression scheme. However, it may not be the case if you choose more sophisticated demosaic methods that minimize color artifacts. Additional studies will be required to come up with better solutions to improve the performance of CFA data compression. Another interesting direction of future works will be considering denoising, demosaicing, deblocking and image compression at the same time. There are several

are taken into account. All four operations are closely related and strongly affect the performance each other. We believe there is room for big improvement in this direction. VII. A PPENDIX : W ORK D ISTRIBUTION Sung Hee Park: ideal entropy estimation, pipeline simulation, presentation slides, nal report. Albert No: lossy CFA compression methods, presentation slides, nal report.
(a)

R EFERENCES
[1] B. Bayer, Color imaging array, U.S. Patent 3971065, July 1976. [2] A. Gentile and F. Sorbello, Image processing chain for digital still cameras based on the simpil architecture, in ICPPW 05: Proceedings of the 2005 International Conference on Parallel Processing Workshops. Washington, DC, USA: IEEE Computer Society, 2005, pp. 215222. [3] R. Ramanath, W. E. Snyder, Y. Yoo, and M. S. Drew, Color image processing pipeline, IEEE Signal Process. Mag, vol. 22, pp. 3443, 2005. [4] K. Nallaperumal, S. Christopher, S. S. Vinsley, and R. K. Selvakumar, New efcient image compression method for single sensor digital camera images, in ICCIMA 07: Proceedings of the International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007). Washington, DC, USA: IEEE Computer Society, 2007, pp. 113117. [5] K.-H. Chung and Y.-H. Chan, A lossless compression scheme for bayer color lter array images. IEEE Trans Image Process, vol. 17, no. 2, pp. 13444, 2008. [Online]. Available: http://www.biomedsearch.com/nih/lossless-compressionscheme-Bayer-color/18270106.html [6] S. Battiato, A. Buemi, L. D. Torre, and A. Vitali, Fast vector quantization engine for cfa data compression, in In Proceedings of IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing, NSIP, 2003. [7] X. Xie, G. Li, Z. Wang, C. Zhang, D. Li, and X. Li, A novel method of lossy image compression for digital image sensors with bayer color lter arrays, in ISCAS (5). IEEE, 2005, pp. 49954998. [8] Eastman Kodak Company, PhotoCD PCD0992, (http://r0k.us/graphics/kodak/). [9] K. Hirakawa and T. W. Parks, Adaptive homogeneity-directed demosaicing algorithm, IEEE Transactions on Image Processing, vol. 14, no. 3, pp. 360369, 2005. [10] E. Dubois, Frequency-domain methods for demosaicking of Bayer-sampled color images, IEEE Signal Process. Lett., vol. 12, no. 12, pp. 847850, 2005. [11] X. Zhang, B. A. Wandell, and B. A. W, A spatial extension of cielab for digital color image reproduction, in Proceedings of the SID Symposiums, 1996, pp. 731734.

(b)

(c) Fig. 11. Various artifacts after compression. (a) Color artifacts (Direct JPEG, Bilinear). (b) Blocking artifacts (Structure Conversion, Adaptive Frequency Domain). (c) Amplication of compression artifacts (Structure Conversion, Adaptive Homogeneity).

works on switching the order of some operations or trying some of them jointly, but not all of the operations