Pattern Recognition Letters 45 (2014) 115–120

Contents lists available at ScienceDirect

Pattern Recognition Letters

journal homepage: www.elsevier.com/locate/patrec

High resolution image fusion algorithm based on multi-focused region

extraction q
Xiaohua Xia ⇑, Suping Fang, Yan Xiao
School of Mechanical Engineering, State Key Laboratory for Manufacturing Systems Engineering, Xi’an Jiaotong University, Xi’an 710049, China

a r t i c l e i n f o a b s t r a c t

Article history: An efficient way to obtain the high resolution image of a scene is using a line scan camera in the manner
Received 18 November 2013 of macro photography. Due to the limited depth of field, only some portions of the image are focused. A
Available online 3 April 2014 common solution to this problem is utilizing the multi-focus image fusion technique, in which a series of
images with different focus settings is acquired and the images are fused to an all focused one. However,
Keywords: it is difficult to register these high resolution images. Firstly, the magnifications of different regions of an
High resolution image image are different because of the depths of the scene. Secondly, the accuracy of feature detection in the
Multi-focus image fusion
region out of focus is difficult to ensure. Misregistration of the multi-focus images leads to the misjudg-
Focused region extraction
ment of focus measures and the failure of image fusion. In this paper, we propose a novel high resolution
multi-focus image fusion algorithm to solve this problem. The focused regions of each image are
extracted for image registration and fusion, which improves the accuracy of image registration and the
quality of image fusion. Experimental results show the proposed method is superior to the traditional
methods in terms of both subjective evaluation and objective evaluation.
Ó 2014 Elsevier B.V. All rights reserved.

1. Introduction computational complexity and the image information directly

High resolution images not only have high definition but also
contain rich spectral, texture and shape information. They can be
used in many applications such as improving the performance of
pattern recognition in computer vision [19] and digital protection
of cultural relics [5]. These images are usually obtained by a line
scan camera in the manner of macro photography and the optical
resolution is usually more than 600 dpi. However, large magnifica-
tion causes the harsh compression of depth of field, which leads to
the image defect that only some regions of the image are in focus.
A common solution to this problem is to acquire several images
of a scene with focus on different depths and then fuse them in
such a manner that all regions of the scene are in focus [2,3,24].
In the past few years, a number of multi-focus image fusion meth-
ods have been developed. These methods are mainly divided into
two groups: spatial domain methods and transform domain
methods [16]. In spatial domain, focus measures such as Sum of
the Modified Laplacian (SML) [17], spatial frequency (SF) [4], etc.
are directly applied to the source images to select the best pixels
or regions. The major advantages of these methods are the low
q
This paper has been recommended for acceptance by Nappi Michele.
⇑ Corresponding author. Tel.: +86 15249200272; fax: +86 029 82665204.
E-mail address: qichishusheng@163.com (X. Xia).
originating from the source images [27]. However, focus measures
are sensitive to misregistration [13]. Misregistration leads to the
error selection of the best pixels, which degrades the image quality
by replacing focused pixels with defocused ones. Some burrs are
brought in the fused image, and they severely affect the visual
effects. Although various image registration methods have been
proposed (a survey can be found in [34]), it is a tough issue to
accurately register the multi-focus images of 3D objects. First,
the imaging of 3D objects is relevant to depth, which means the
magnifying powers of different depths are different. It is difficult
to establish the mapping among 2D images without depth infor-
mation. Second, feature detection is the first basic step of image
registration procedure [34]. The feature detection in the region
out of focus is difficult to ensure.
In transform domain, Multi-scale transforms (MST) provide
another useful manner for image fusion. The idea of MST-based
image fusion methods is to integrate the multi-resolution decom-
position of each source image into a composite fusion space, and
reconstruct the fused image by performing an inverse multi-
resolution transform. Common MST-based methods include pyra-
mid transform [25,28], discrete wavelet transform [11,6,18,
23,32], complex wavelet transform [26], redundant wavelet trans-
form [15], curvelet transform [14], contourlet transform [31].
Research results show that the MST-based methods possess many

http://dx.doi.org/10.1016/j.patrec.2014.03.018
0167-8655/Ó 2014 Elsevier B.V. All rights reserved.
116 X. Xia et al. / Pattern Recognition Letters 45 (2014) 115–120
advantages, including the improved signal-to-noise ratio, the
increased fusion quality, etc. [27]. However, MST-based methods
are not applicable to high resolution image fusion because they
are complicated and time-consuming [7,29]. Furthermore, MST-
based methods are also sensitive to misregistration [12,13].
Although some region-based methods [20,26] are proposed to
reduce the adverse effect of misregistration, the misregistration
on the boundary of regions still affects the image fusion quality.
In this paper, we propose a novel image fusion algorithm which
is based on focused region extraction. Different from other
methods, our method discards the majority of the defocused
regions. The focused regions of each image are extracted for image
registration and fusion, which improves the accuracy of image
registration and the quality of image fusion.
The paper is organized as follows. The methodology is
presented in detail in Section 2, including the imaging system we
use and the proposed image fusion algorithm. Then, experimental
results and performance evaluation are given to test the effective-
ness of the algorithm in Section 3. Finally, conclusion is drawn in
Section 4.
2. Methodology
2.1. The imaging system
There are three ways to obtain multi-focus images: moving the
sensor, moving the lens and moving the imaging system. We
choose the third one because the image distance is fixed. The mag-
nifications of the focused regions in different images can be seen as
the same. The schematic diagram of our high resolution imaging
system is shown in Fig. 1. The light emitted by the two white line
sources is projected onto the surface of the object. The reflected
light travels through the lens and enters the line scan camera
which converts the optical signal to the electrical signal. Finally,
images are acquired by the image grabber and they are saved to
the computer.
The line scan camera can only get a line image at a time. In
order to obtain a frame image, the imaging system should be
moved along the main scanning direction. Once the acquisition
of a frame image is over, the imaging system will be moved along
the height adjustment direction for a short distance to get the next
frame image. The moving distance should be less than the depth of
field. The procedure will continue until all images are obtained.
Note that the image distance is fixed in the process of image
acquisition and the depth of field is much less than the object dis-
tance. The magnifications of the focused regions in all images can
be considered as a constant, which provides a shortcut for the reg-
istration between the focused regions and the adjacent image.
Detailed registration procedure will be shown in Section 2.2.
Fig. 1. The schematic diagram of high resolution imaging system.
2.2. The image fusion algorithm
The flow diagram of our image fusion algorithm is shown in
Fig. 2. The algorithm consists of four steps: initial registration,
focused region extraction, accurate registration, and image fusion.
(a) Initial registration
According to the imaging characteristic of the line scan camera
in Fig. 1, we use the following global mapping model for initial
registration.
x0 ¼ ax þ t x
ð1Þ
y0 ¼ y þ t y
where (x, y) is the coordinate of the pixel in the reference image;
(x0 , y0 ) is the coordinate of the pixel in the target image; a is a
parameter for scaling transform; tx and ty are parameters for trans-
lation transform.
(b) Focused region extraction
After initial registration, a focus measure operator is used to
search for the best focused pixels in the sequence of images.
Experimental results of Huang and Jing [8] show SML focus
measure provides better performance, and we use it as the focus
measure in this paper. The best focused pixels are saved to a
new sequence of images and other pixels are discarded. However,
the new images are defective. The matching error in step (a) can
cause the misjudgment of the best focused pixels. Some defocused
pixels are introduced and some focused pixels are lost in the new
images. To solve this problem, probability filtering is used for fur-
ther judgment. Set a window on the pixel of the new image and
calculate the probability of ‘‘the best focused’’ pixels in the win-
dow. If the probability is greater than a threshold, the central pixel
will be considered to be the best focused. Our experience shows
that 50% is a good threshold and probability filtering can improve
the robustness of focus measures to misregistration. Nevertheless,
probability filtering is not perfect. A small number of focused pix-
els are still lost, especially in the misjudgment concentrated area.
To recover the lost focused pixels, we expand the focused regions.
Region expansion makes focused regions of adjacent images over-
lapped with each other. The defocused pixels introduced by region
expansion will be eliminated in the process of image fusion.
(c) Accurate registration
As shown in Fig. 3, the solid lines show the imaging geometry of
focused regions and the dotted lines show the imaging geometry
with a moving distance du. Since the depth of field is much less
than the object distance, the magnifying power can be seen as a
constant [17]. It can be expressed by:
v
m¼ ð2Þ
u
where v is the image distance and u is the object distance. With a
moving distance du, the magnifying power becomes:
v
m0 ¼ ð3Þ
u  du
A scaling is necessary for the registration between the focused
regions and the adjacent image:
m0 u
s¼ ¼ ð4Þ
m u  du
Fig. 2. The flow diagram of the proposed image fusion algorithm.
 X. Xia et al. / Pattern Recognition Letters 45 (2014) 115–120 117

(d) Image fusion

Based on the accurate registration established in step (c), SML
focus measure is applied to the overlapping areas of adjacent
focused regions, and then an all-focused image is fused.

3. Experiment and discussion

In this section, subjective and objective evaluations are given to

verify the validity of the proposed method.
Fig. 3. The imaging geometry of adjacent images.
3.1. Experiment setting

The mapping model for the focused regions and the adjacent NED’s NUCLi7300 line scan camera and RODENSTOCK’s Apo-
image can be expressed by: Rodagon-D lens are used to acquire the high resolution multi-focus
images. According to the specification supplied by the manufac-
x0 ¼ sx þ t x ture, the focal length of the lens is 120 mm, the pixel number of
ð5Þ the line scan camera is 7300  3, and the size of each pixel is
y0 ¼ y þ t y
10  10 lm. The aperture value we adopt is 5.6 and the optical
Extract the common features of the focused regions and the resolution of each image is 1282 dpi. The moving step we set to
adjacent image, and establish the mapping according to Eq. (5). the optical imaging system is 7 mm.
It’s also the mapping between the focused regions of the two adja- Four experimental objects of different materials are used to test
cent images. the proposed algorithm. They are respectively a Peking opera facial

Fig. 4. The multi-focus images of the experimental objects.

118 X. Xia et al. / Pattern Recognition Letters 45 (2014) 115–120
mask, a metal man, a Zisha cup and a wood frog. The materials are
respectively oil paint, metal, clay and wood. As shown in Fig. 4,
four multi-focus images of the facial mask, four multi-focus images
of the metal man, seven multi-focus images of the Zisha cup and
eight multi-focus images of the wood frog are obtained. The partial
enlarged view of the focused region in each image is also
presented, as marked with the rectangle in Fig. 4. We can see that
the multi-focus images are focused on different parts of the
objects.
3.2. Focus measure robustness analysis
Focus measure robustness comparisons are shown in Fig. 5(a)–
(m). Fig. 5(a)–(c) are from the image of the facial mask. Fig. 5(d)–(f)
are from the image of the metal man. Fig. 5(h)–(j) are from the
image of the Zisha cup. Fig. 5(k)–(m) are from the image of the
wood frog. Fig. 5(a), (d), (h), and (k) are the results of traditional
SML. Fig. 5(b), (e), (i) and (l) are the results of SML with probability
filtering. Fig. 5(c), (f), (j) and (m) are the results of SML with prob-
ability filtering and region expansion. In these twelve images, the
black regions are determined to be defocused by the focus measure
while the other regions are determined to be focused. From
Fig. 5(a), (d), (h) and (k), we can see that the traditional SML focus
measure brings out much focus misjudgment (the disorderly black
regions in the figure). After probability filtering (see Fig. 5(b), (e),
(i) and (l)), most of the focus misjudgment is eliminated. However,
some misjudgment still exists in the misjudgment relatively con-
centrated area. Fig. 5(c), (f), (j) and (m) show that region expansion
recovers the lost focused pixels. Although some defocused regions
are introduced by region expansion (see Fig. 5(n), (p), (r) and (t),
which are, respectively from the image of the facial mask, the
image of the metal man, the image of the Zisha cup, and the image
of the wood frog), they will be eliminated in the process of image
fusion (see Fig. 5(o), (q), (s) and (u)).
3.3. Subjective evaluation
To test the effectiveness of the proposed method, we compare
the proposed method with the traditional methods. The traditional
methods we use are respectively based on affine transformation
(Traditional 1) and radial basis function (Traditional 2). Affine
transformation is a widely used transformation for multi-focus
images [3]. Radial basis function is also commonly used in image
registration [34,9]. The image fusion results are shown in Fig. 6.
Fig. 6(a)–(d), which are, respectively the partial view of the facial
mask, the metal man, the Zisha cup and the wood frog, are the
Fig. 5. Focus measure robustness comparisons (a–m), and the introduction and elimination of defocused regions (n–u).
image fusion results by the proposed method. Fig. 6(e)–(h) are
the corresponding fusion results by Traditional 1. Fig. 6(i)–(l) are
the fusion results by Traditional 2. It can be seen that the fused
images by the traditional methods have many burrs, which are
marked with the ellipses in Fig. 6(e)–(l). The burrs are the defects
of the fused images and they severely affect the visual effects. In
contrast, there are no burr and visible defects in the fused images
by the proposed method. So the proposed method is more effective
than the traditional methods in visual inspection.
3.4. Objective evaluation
In this paper, focused regions are extracted for image registra-
tion and image fusion. So the objective evaluation includes two
aspects: image registration and image fusion.
Mean-square-error (MSE) is a common objective evaluation for
estimating the performance of registration [22]. The proposed
method establishes the mapping between the focused regions of
two adjacent images. The traditional methods establish the map-
ping between each image and the reference image(the last image
in each sequence of multi-focus images). The comparison results
are shown in Table 1. From Table 1, we can see that the registration
error of the proposed method is less than that of the traditional
methods. The idea of establishing the mapping among focused
regions can improve the accuracy of image registration.
Objective evaluation of image fusion is a difficult task because
there is no universally accepted metric for image fusion [33]. A
proper solution is using different metrics to test the fusion results
in different viewpoints. The metrics we use are QAB/F metric, MIF
metric and Entropy.
The QAB/F metric [30] considers the amount of edge information
transferred from the input images to the fused image. It uses a
Sobel edge detector to calculate the strength and orientation infor-
mation at each pixel in both source and fused images [10]. For
input images A and B, the QAB/F metric is defined as:
PM PN AF
i¼1
A
j¼1 Q ði; jÞw ði; jÞ þ Q BF ði; jÞwB ði; jÞ
Q AB=F ¼ PM PN A ð6Þ
i¼1 j¼1 w ði; jÞ þ wB ði; jÞ
where QAF(i, j) and QBF(i, j) are, respectively the preservation values
of images A and B; wA(i, j) and wB(i, j) are, respectively the weight
values of images A and B. More detailed descriptions can be found
in Xydeas and Petrovic [30]. The larger QAB/F value means the better
fused image.
MIF metric [21] is the sum of mutual information of each input
image and the fused image. It essentially computes how much
 X. Xia et al. / Pattern Recognition Letters 45 (2014) 115–120 119

Fig. 6. The comparison among different methods: the proposed (a–d), Traditional 1 (e–h), and Traditional 2 (i–l).

Table 1 Table 2
Objective evaluation of image registration. Objective evaluation of image fusion.

Objects Images Proposed Traditional 1 Traditional 2 Objects Metrics Proposed Traditional 1 Traditional 2

The facial mask Image 1 76.31 399.30 104.91 The facial mask QAB/F 0.8350 0.3167 0.3053
Image 2 105.59 240.33 224.20 MIF 22.6499 13.2982 12.8986
Image 3 41.41 42.58 42.54 EN 6.6928 6.6743 6.6914
The metal man Image 1 55.64 242.86 243.39 The metal man QAB/F 0.7474 0.3646 0.3676
Image 2 101.29 203.35 203.05 MIF 17.9456 12.8681 12.9355
Image 3 19.33 20.35 19.84 EN 6.1442 6.1387 6.1368
The Zisha cup Image 1 39.91 153.20 99.58 The Zisha cup QAB/F 0.3926 0.1957 0.1752
Image 2 16.84 110.86 61.92 MIF 29.3838 20.1143 19.6275
Image 3 13.94 90.06 72.36 EN 6.9809 6.9764 6.9754
Image 4 28.00 68.74 31.54
The wood frog QAB/F 0.8215 0.1656 0.1647
Image 5 24.83 54.44 43.27
MIF 34.1852 19.9021 19.8962
Image 6 34.58 36.75 36.25
EN 5.4901 5.4579 5.4579
The wood frog Image 1 2.47 8.68 8.68
Image 2 3.76 12.25 12.26
Image 3 4.70 12.46 12.57
Image 4 3.88 11.75 11.84 X
L1

Image 5 3.10 6.02 5.97 EN ¼  pðiÞlog2 pðiÞ ð9Þ

Image 6 4.74 7.74 7.76 i¼0
Image 7 3.95 3.98 3.96
where p(i) is the normalized histogram of the image. The larger EN
value indicates the better fusion result.
The values of QAB/F metric, MIF metric and EN of the fused
information from each of the fused images is transferred to the images are listed in Table 2. According to the evaluation rules of
fused image [10]. It is defined as: QAB/F metric, MIF metric and En, the better method of image fusion
X should have larger QAB/F, larger MIF and larger EN. We can see from
MIF ¼ MI ð7Þ Table 2 that the proposed method has larger QAB/F, larger MIF and
larger EN. So the proposed method provides better performance
where MI is the mutual information of each input image and the
and outperforms the traditional methods in image fusion.
fused image. It is defined as:
Based on the above analysis, we can see that the proposed
X
L1 X
L1
p ði; jÞ image fusion method is effective and it is superior to the traditional
MI ¼ pIF ði; jÞlog2 IF ð8Þ image fusion methods which are, respectively based on affine
i¼0 j¼0
pI ðiÞpF ðjÞ
transformation and radial basis function in terms of both subjec-
tive evaluation and objective evaluation.
where L is the number of gray levels; pIF(i, j) is the normalized joint
gray level histogram of the inpute image and the fused image; pI(i)
and pF(j) are the normalized marginal histograms of the two images. 4. Conclusion
The large MIF value signifies the better fusion result.
Entropy (EN) [1] measures the information quality of an image We present a new high resolution multi-focus image fusion
and it is defined as: algorithm. The algorithm consists of four steps: initial registration,
120 X. Xia et al. / Pattern Recognition Letters 45 (2014) 115–120
focused region extraction, accurate registration and image fusion.
Probability filtering is introduced in this paper to improve the
robustness of the focus measure. The focused regions of each image
are extracted for image registration and fusion, which improves
the accuracy of image registration and the quality of image fusion.
Experimental results show our method is effective and superior to
the widely used image fusion methods in terms of both subjective
evaluation and objective evaluation.
References
[1] S. Arivazhagan, L. Ganesan, T.G.S. Kumar, A modified statistical approach for
image fusion using wavelet transform, Signal Image Video Process. 3 (2009)
137–144.
[2] V. Aslantas, R. Kurban, Fusion of multi-focus images using differential
evolution algorithm, Expert Syst. Appl. 37 (2010) 8861–8870.
[3] I. De, B. Chanda, B. Chattopadhyay, Enhancing effective depth-of-field by image
fusion using mathematical morphology, Image Vision Comput. 24 (2006)
1278–1287.
[4] A.M. Eskicioglu, P.S. Fisher, Image quality measures and their performance,
IEEE Trans. Commun. 43 (1995) 2959–2965.
[5] S. Fang, X. Xia, Y. Xiao, A calibration method of lens distortion for line scan
cameras, Optik 124 (2013) 6749–6751.
[6] M.B.A. Haghighat, A. Aghagolzadeh, H. Seyedarabi, Multi-focus image fusion
for visual sensor networks in DCT domain, Comput. Electr. Eng. 37 (2011) 789–
797.
[7] W. Huang, Z. Jing, Multi-focus image fusion using pulse coupled neural
network, Pattern Recognit. Lett. 28 (2007) 1123–1132.
[8] W. Huang, Z. Jing, Evaluation of focus measures in multi-focus image fusion,
Pattern Recognit. Lett. 28 (2007) 493–500.
[9] R.J. Lapeer, S.K. Shah, R.S. Rowland, An optimized radial basis function
algorithm for fast non-rigid registration of medical images, Comput. Biol.
Med. 40 (2010) 1–7.
[10] J.J. Lewis, R.J. O’Callaghan, S.G. Nikolov, D.R. Bull, Pixel- and region-based
image fusion with complex wavelets, Inf. Fusion 8 (2007) 119–130.
[11] H. Li, B.S. Manjunath, S.K. Mitra, Multisensor image fusion using the wavelet
transform, Graph. Models Image Process. 57 (1995) 235–245.
[12] M. Li, W. Cai, Z. Tan, A region-based multi-sensor image fusion scheme using
pulse-coupled neural network, Pattern Recognit. Lett. 27 (2006) 1948–1956.
[13] S. Li, B. Yang, Multifocus image fusion using region segmentation and spatial
frequency, Image Vision Comput. 26 (2008) 971–979.
[14] S. Li, B. Yang, Multifocus image fusion by combining curvelet and wavelet
transform, Pattern Recognit. Lett. 29 (2008) 1295–1301.
[15] X. Li, M. He, M. Roux, Multifocus image fusion based on redundant wavelet
transform, IET Image Process. 4 (2010) 283–293.
[16] N. Mitianoudis, T. Stathaki, Pixel-based and region-based image fusion
schemes using ICA bases, Inf. Fusion 8 (2007) 131–142.
[17] S.K. Nayar, Y. Nakagawa, Shape from focus, IEEE Trans. Pattern Anal. Mach.
Intell. 16 (1994) 824–831.
[18] G. Pajares, J. Manuel de la Cruz, A wavelet-based image fusion tutorial, Pattern
Recognit. 37 (2004) 1855–1872.
[19] S.C. Park, M.K. Park, M.G. Kang, Super-resolution image reconstruction: a
technical overview, IEEE Signal Process. Mag. 20 (2003) 21–36.
[20] G. Piella, A general framework for multiresolution image fusion: from pixels to
regions, Inf. Fusion 4 (2003) 259–280.
[21] G. Qu, D. Zhang, P. Yan, Information measure for performance of image fusion,
Electron. Lett. 38 (2002) 313–315.
[22] D. Robinson, P. Milanfar, Fundamental performance limits in image
registration, IEEE Trans. Image Process. 13 (2004) 1185–1199.
[23] J. Tang, A contrast based image fusion technique in the DCT domain, Digital
Signal Process. 14 (2004) 218–226.
[24] J. Tian, L. Chen, L. Ma, W. Yu, Multi-focus image fusion using a bilateral
gradient-based sharpness criterion, Opt. Commun. 284 (2011) 80–87.
[25] A. Toet, Image fusion by a ratio of low-pass pyramid, Pattern Recognit. Lett. 9
(1989) 245–253.
[26] T. Wan, N. Canagarajah, A. Achim, Segmentation-driven image fusion based on
alpha-stable modeling of wavelet coefficients, IEEE Trans. Multimedia 11
(2009) 624–633.
[27] T. Wan, C. Zhu, Z. Qin, Multifocus image fusion based on robust principal
component analysis, Pattern Recognit. Lett. 34 (2013) 1001–1008.
[28] W. Wang, F. Chang, A multi-focus image fusion method based on Laplacian
pyramid, J. Comput. 6 (2011) 2559–2566.
[29] Z. Wang, Y. Ma, J. Gu, Multi-focus image fusion using PCNN, Pattern Recognit.
43 (2010) 2003–2016.
[30] C.S. Xydeas, V. Petrovic, Objective image fusion performance measure,
Electron. Lett. 36 (2000) 308–309.
[31] Q. Zhang, B. Guo, Multifocus image fusion using the nonsubsampled contourlet
transform, Signal Process. 89 (2009) 1334–1346.
[32] Y. Zheng, E.A. Essock, B.C. Hansen, Advanced discrete wavelet transform fusion
algorithm and its optimization by using the metric of image quality index, Opt.
Eng. 44 (2005). 037003-037003-12.
[33] H. Zhao, Z. Shang, Y.Y. Tang, B. Fang, Multi-focus image fusion based on the
neighbor distance, Pattern Recognit. 46 (2013) 1002–1011.
[34] B. Zitová, J. Flusser, Image registration methods: a survey, Image Vision
Comput. 21 (2003) 977–1000.