Digital Signal Processing 60 (2017) 200–210

Contents lists available at ScienceDirect

Digital Signal Processing

www.elsevier.com/locate/dsp

All-in-focus imaging using average filter-based relative focus measure

Oh-Jin Kwon, Seungcheol Choi ∗ , Dukhyun Jang, Hee-Suk Pang
Department of Electrical Engineering, Sejong University, 209 Neungdong-ro, Gwangjin-Gu, Seoul 05006, Republic of Korea

a r t i c l e i n f o a b s t r a c t

Article history: Digital images are normally taken by focusing on an object, resulting in defocused background regions.
Available online 20 September 2016 A popular approach to produce an all-in-focus image without defocused regions is to capture several
input images at varying focus settings, and then fuse them into an image using offline image processing
Keywords:
software. This paper describes an all-in-focus imaging method that can operate on digital cameras.
Multi-focus
All-in-focus
The proposed method consists of an automatic focus-bracketing algorithm that determines at which
Automatic focus bracketing focuses to capture images and an image-fusion algorithm that computes a high-quality all-in-focus image.
Image fusion While most previous methods use the focus measure calculated independently for each input image, the
Digital camera proposed method calculates the relative focus measure between a pair of input images. We note that
a well-focused region in an image shows better contrast, sharpness, and details than the corresponding
region that is defocused in another image. Based on the observation that the average filtered version of
a well-focused region in an image shows a higher correlation to the corresponding defocused region in
another image than the original well-focused version, a new focus measure is proposed. Experimental
results of various sample image sequences show the superiority of the proposed measure in terms of
both objective and subjective evaluation and the proposed method allows the user to capture all-in-focus
images directly on their digital camera without using offline image processing software.
© 2016 Elsevier Inc. All rights reserved.

1. Introduction
Many photographers are often frustrated when they try to cap-
ture a picture of an object in the foreground. When the foreground
object is in focus, the background is out of focus, and vice versa.
This is due to the limited depth-of-field of cameras. A common so-
lution to this problem is to capture images at various focuses and
fuse them together into an all-in-focus (AIF) image, where all visi-
ble objects are in focus [1–23].
A variety of multi-focus image fusion methods have been de-
veloped. So far, these methods are mainly classified into two types,
one of which is used in transform domain and the other in spatial
domain [1–3]. Transform domain methods transform the original
source images into transform coefficients, apply a unique fusion
rule, and reconstruct the fused image by performing the inverse
transform. Most commonly used transforms are multi-scale based
transforms. They include gradient pyramid transform [4], Lapla-
cian pyramid transform [5], discrete wavelet transform [6], dual
tree complex wavelet transform [7], contourlet transform [8,9], log-
Garbor transform [10], shearlet transform [11], curvelet transform
* Corresponding author.
E-mail addresses: ojkwon@sejong.ac.kr (O.-J. Kwon), choisc@sju.ac.kr (S. Choi),
jangdh777@sju.ac.kr (D. Jang), hspang@sejong.ac.kr (H.-S. Pang).
[12], and others. Even though these transform-based methods have
shown satisfactory fusion results for high quality images, there is
a known issue of considering limited spatial information. There-
fore, such methods often lose edge and texture information, which
leads to distortions in fused images [13–22]. Furthermore, the high
computing cost of these methods can be inhibitive.
Spatial domain methods can be divided into two categories:
region-based methods [13–18] and pixel-based methods [1,19–22].
In region-based methods, fusion results from two-pass process-
ing. These methods typically employ their own focus measure to
segment the source images into three regions: focused regions,
defocused regions, and transition regions between focused and de-
focused regions. Subsequently, different fusion rules are applied for
each region. Pixel-based methods, however, use one-pass process-
ing. Therefore, these methods are usually faster than region-based.
In pixel-based methods, the focus measure is designed to esti-
mate the focus of each pixel in an image. A focus weighting map
is generated for each source image to determine the pixel of the
fused image by calculating the weighted average of pixels from
source images or by selecting the pixel demonstrating optimal fo-
cus among source images.
Various focus measures have been proposed in the literature on
the subject, which include standard deviation or variance [13,23],
average gradient [13], energy of Laplacian [23], spatial frequency
[14,19], energy of gradient [23], Tenenbaum gradient [24], sum-

http://dx.doi.org/10.1016/j.dsp.2016.09.005
1051-2004/© 2016 Elsevier Inc. All rights reserved.
 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210
Fig. 1. Exemplified implementation of the proposed AIF imaging method.
modified Laplacian (SML) [25], frequency selective weighted me-
dian filter [26], and other measures. Comparisons of traditional fo-
cus measures for the multi-focus image fusion are provided in [13,
20,23,27]. More recently proposed measures for multi-focus image
fusion include multi-scale weighted gradient [15], morphological
filtering [17], bilateral gradient-based sharpness [20], homogeneity
similarity [21], neighbor distance [28], and other measures.
In this paper, we propose a pixel based AIF imaging method.
Professional photographers may create AIF images using popular
image editing software in post-processing. However, this offline ap-
proach is cumbersome for the ordinary photographer. We propose
a method for previewing and capturing AIF scenes directly on dig-
ital cameras without using offline image processing software. The
proposed method attempts to include the following features:
1) An automatic bracketing algorithm to determine the minimum
number of focus settings required to accurately capture AIF
scenes.
2) A real-time preview for the user to examine the AIF image and
freeze a shot.
3) Direct creation and review of the high-resolution AIF image on
the digital camera.
4) A new multi-focus image-fusion algorithm employed for fast
processing.
The paper is organized as follows. Section 2 describes the pro-
posed AIF imaging method. Section 3 presents the experimental
results. Finally, Section 4 concludes the paper.
2. Proposed AIF imaging method
Fig. 1 exemplifies an implementation of the proposed method.
One attractive feature of modern digital cameras is their video-
rate view-finding (VRVF) mode. The exemplified implementation
automatically starts focus bracketing when camera motion in the
VRVF mode is small for a few seconds and shows a fused image
201
in preview resolution size in real time. The camera switches be-
tween the VRVF mode and fused image preview (FIP) mode based
on detected camera motion. For fast processing, a simple frame
difference detecting method detects camera motion by threshold-
ing the sum of the absolute differences between centered blocks of
half the horizontal and vertical frame sizes of consecutive frames.
A high-resolution fused image is captured at the user’s demand in
the FIP mode.
In the following subsections, we first propose a new focus mea-
sure. We explain the focus-bracketing algorithm that implements
the proposed focus measure. Finally, we describe the multi-focus
image-fusion algorithm for previewing a reduced sized image in
real-time and capturing a high-resolution AIF image.
2.1. Proposed focus measure
The following attributes have been proposed for perceptual im-
age quality assessment: overall brightness, contrast, sharpness, de-
tails, naturalness, and colorfulness [29–31]. Researchers have in-
vestigated the effect of focus change on these attributes and have
found that focus change has little effect on overall brightness, nat-
uralness, and colorfulness [13–22]. Therefore, most focus measures
proposed have been designed mainly by utilizing contrast, sharp-
ness, and details. As these attributes may be reflected on the lu-
minance component of an image, the proposed focus measure is
calculated based on the luminance value.
Focus measures can be broadly organized into two main cate-
gories: measures calculated in the transform domain and the spa-
tial domain [1–3]. Even though transform domain measures result
in satisfactory fusion to produce high quality images, their com-
puting cost is too high for real-time processing in state-of-the-art
digital cameras. Spatial domain measures in multi-focus image fu-
sion were evaluated by Tian et al. [20] and Huang and Jing [23].
The authors reported that the SML-based measure produced the
best results among the real-time implementable measures.
Most spatial domain focus measures estimate the focus of a
pixel based on its neighboring pixels. For example, the SML is cal-
culated as follows:

f (m, n) = {|2x(i , j ) − x(i − 1, j ) − x(i + 1, j )|
(i , j )∈ B m,n (1)
+ |2x(i , j ) − x(i , j − 1) − x(i , j + 1)|},
where f (m, n) and x(m, n) denote the focus measure and the lu-
minance intensity of a pixel at (m, n), respectively, and B m,n is the
block centered at (m, n).
In this paper, we propose a new focus measure that results
in better quality fused images than the SML does with compara-
ble computing cost. To the authors’ knowledge, all focus measures
proposed to date calculate the absolute focus for each source im-
age independently, as exemplified in Eq. (1); the proposed measure
calculates the relative focus of an image compared to another. This
new approach improves fusion performance.
For two images taken at different focuses, a well-focused re-
gion in one image shows better contrast, sharpness, and detail than
the corresponding defocused region in the other image. Therefore,
average-filtering a well-focused region in one image increases the
correlation to the corresponding defocused region in the other im-
age. On the other hand, average-filtering a defocused region in one
image decreases its correlation to the corresponding well-focused
region in the other image. We verify this observation using an ex-
periment below.
Consider images 1 and 2 denoted by X (1) = {x(1) (m, n)} and
(1)
X (2) = {x(2) (m, n)}, with average-filtered versions X = {x(1) (m, n)}
(2) (2)
and X = {x (m, n)}, respectively. We define the correlation map
between X (1) and X (2) as follows:
202 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210

Fig. 2. Correlation map and AFRFM map between two differently focused images: (a) sample image X (1) , (b) sample image X (2) , (c) C M X (1) X (2) (m, n) and C M (1) (2) (m, n)
X X
map, and (d) f 1:2 (m, n) and f 2:1 (m, n) map.

C M X (1) X (2) (m, n) 2.2. Focus-bracketing algorithm


(i , j )∈ B m,n x(1) (i , j )x(2) (i , j ) Fig. 3 presents a graph of the AFRFM for a sequence of multi-
=      . (2)
x(1) (i , j )2 
focused images, and the images corresponding to AFRFM peaks.
x(2) (i , j )2
(i , j )∈ B m,n (i , j )∈ B m,n For focus bracketing, the AFRFM is calculated for preview resolu-
tion sized images as follows:
We observe the difference between C M X (1) X (2) (m, n) and
 2
C M (1) (2) (m, n) when X (1) and X (2) are two images taken at (k)
f AF RF M = x(k) (m, n) − x(k−1) (m, n)
X X
different focuses. For this observation, we set the window size (m,n)
of the averaging filter and the size of block B m,n to 7 × 7 and  2
5 × 5, respectively. Fig. 2(a) and Fig. 2(b) show two sample images × x(k+1) (m, n) − x(k) (m, n) , (4)
taken at different focuses. For visualizing the difference between
C M X (1) X (2) (m, n) and C M (1) (2) (m, n), we generate a map repre- where x(k) (m, n) denotes the kth focused image. As can be seen
X X from this figure, the AFRFM generates well-defined peaks.
senting C M X (1) X (2) (m, n) and C M (1) (2) (m, n) in RGB color space
X X Although the focus-bracketing algorithm is relatively simple,
in which C M (1) (2) (m, n) and C M X (1) X (2) (m, n) are represented by
X X its implementation in digital camera hardware platforms is not
R and G components, respectively, with B component set to be so straightforward. Depending on the type of hardware platform
zero. The resulting map is shown in Fig. 2(c). It is observed that available, some important implementation issues have to be taken
C M (1) (2) (m, n) (R component) is greater than C M X (1) X (2) (m, n) (G into consideration in order to achieve a balance between focus-
X X
component) in well-focused regions of X (1) . On the other hand, ing speed and accuracy [32]. For any focus measure, the choice
C M (1) (2) (m, n) is smaller than C M X (1) X (2) (m, n) in defocused re- of peak-finding procedure is a major concern, as each peak cor-
X X
gions of X (1) . responds to a specific object at a specific distance. Compared with
Based on this observation, we propose the average filter-based the well-studied problem of auto-focusing to find the best-focusing
relative focus measure (AFRFM) of image 1 to image 2 at (m, n) lens position [32–34], the problem of focus bracketing to deter-
calculated as follows: mine the minimum number of lens positions required for multi-
focus image fusion has received much less attention. However,
  2 some peak search algorithms used in auto-focusing systems can
 (1 ) (2 ) 
(i , j )∈ B m,n x (i , j ) − x (i , j ) also be applied to focus-bracketing algorithms.
f 1:2 (m, n) =  2 . (3)
  (2 ) (1 )  Three types of standard search algorithms have been adopted
(i , j )∈ B m,n  x ( i , j ) − x ( i , j  for auto-focusing: global search (GS), binary search (BS), and rule-
based search (RS) [32,34]. In GS, the peak position is obtained by
f 2:1 (m, n) = 1/ f 1:2 (m, n) and the AFRFM gives a relative value be- scanning the entire range of focus steps, eliminating any possibil-
tween two images, not the absolute value for each image. Fig. 2(d) ity of falsely obtaining a local peak. Whereas GS is generally slow,
shows a map representing f 1:2 (m, n) and f 2:1 (m, n) in RGB color it can be an effective approach if the number of steps within the
space in which f 1:2 (m, n) and f 2:1 (m, n) are represented by R and search range is small.
G components, respectively, with B component set to be zero. As BS efficiently reduces the number of searching steps relative
shown, this map reflects the relatively well-focused region of im- to GS by using a divide and conquer algorithm to find peaks.
age 1 to image 2. However, this algorithm requires frequent changes in search di-
 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210
Fig. 3. AFRFM measurements for a sequence of multi-focused images: (a) AFRFM measurements and (b) images corresponding to peaks.
rection to reach peak positions. In current digital camera hardware
platforms, most step-motors for adjusting focus steps have gear
backlash, which requires compensation procedures that increase
the focus lag time considerably. Reducing focusing lag time while
maintaining high accuracy and low power consumption is critical
for real-time operation, so the frequent direction changes limit the
utility of BS.
RS [34], which is adopted in the proposed method, lowers the
focus lag time by scanning the focus range in a sequential man-
ner in one direction and limiting the number of direction changes.
RS changes step size increments according to the rate of change
of the focus measurement. In RS, the focusing ranges are classified
into four types: initial, fine, mid, and coarse. A fine area has the
highest probability of containing a peak; in this area, every focus
step is examined. A mid area has a lower probability of contain-
ing a peak; every four focus steps are examined. A coarse area has
the lowest probability of containing a peak; every ten focus steps
are examined. The initial area is the position moved to just before
a new search routine begins after finding a local peak. The crite-
203
ria used for classifying the area are described in Kehtarnavaza and
Oh’s paper [34].
Once all local peaks are found, a limited number of peaks are
selected for fast fusion. The maximum number of selected peaks is
chosen based on the user’s allowable fusion time on the dedicated
hardware platform. The rule for selecting the limited number of
peaks is based on the height of the peak and the distance from the
nearest pre-selected peak. The global peak is chosen as the first
selected peak. The next ranked peak is selected by the following
score:
Height( p ) Step D ( p )
Score( p ) = + , (5)
GHieght FRange
where Score( p ), Height( p ), GHeight, StepD( p ), and FRange are the
score of the pth peak, the height of the pth peak, the height of
the global peak, the focus step distance from the pth peak to the
nearest pre-selected peak, and the total focus range, respectively.
This selecting procedure continues until all the peaks are selected,
or until the pre-defined maximum number of allowable selected
peaks is reached.
204 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210
2.3. Multi-focus image-fusion algorithm
Due to camera motion and focal-length change, the images
selected by the focus-bracketing procedure typically show small
geometric distortions such as locally nonlinear magnification and
displacement [25]. Therefore, an image alignment procedure is
demanded for the better fusion result. In our algorithm, image
alignment by enhanced correlation coefficient (ECC) maximization
[35] is performed following the focus-bracketing procedure. The
ECC algorithm is chosen because it provides a nonlinear warp-
ing transformation-based alignment suitable for the case of multi-
focused images and it is also fast, well-known, and widely used
in a wide range of applications, such as image registration, object
tracking, super-resolution, and visual surveillance by moving cam-
eras.
Fusion of multi-focused images for both FIP and high-resolution
fused images are performed on a pixel by pixel basis. Two types of
fusion rules were considered in developing the fusion algorithm:
the first rule selects the pixel with the best focus and the sec-
ond rule calculates the weighted sum for each pixel based on the
focus measure. The first rule is highly sensitive to noise and re-
sults in discontinuities between focused and defocused regions.
The second rule loses details in homogeneous regions. The pro-
posed fusion rule combines these two rules to create the fused
image X (1&2) = {x(1&2) (m, n)} of X (1) and X (2) as follows:
1) x(1&2) (m, n) = x(1) (m, n), if f 1:2 (m, n)/ f 2:1 (m, n) > λ;
2) x(1&2) (m, n) = x(2) (m, n), if f 2:1 (m, n)/ f 1:2 (m, n) > λ;
3) x(1&2) (m, n) = { f 1:2 (m, n)x(1) (m, n) + f 2:1 (m, n)x(2) (m, n)}/
{ f 1:2 (m, n) + f 2:1 (m, n)}, otherwise,
where λ is a threshold value that determines the balance be-
tween the selection and weighted sum based fusion rules. When
the number of multi-focus images is greater than two, this fusion
rule is applied recursively, as follows:
1) Given multi-focus images X (k) = {x(k) (m, n)}, k = 1, 2, · · · , K ,
set s = 1.
2) Using the above fusion rule, obtain X (s&(s+1)) and save the re-
sult as X (s+1) .
3) Repeat step 2) for s = 2, · · · , ( K − 1).
4) Declare X ( K ) as the final fusion result.
3. Experimental results
3.1. Experimental setup
The performance of the proposed method was evaluated on a
digital camera platform powered by a quad-core 2.1 GHz CPU and
3 GB RAM. The resolution size in the VRVF mode and FIP mode
was set to the video graphics array (VGA) resolution (640 × 480).
The resolution size for the high-resolution image fusion was set
to full high-definition (FHD) resolution (1920 × 1080). The cam-
era module installed in this platform consists of 1/2.8-inch CMOS
image sensor and a 20× zoom lens. The focal length of the lens
is 4.7 mm at its wide end and 94 mm at its tele end, respectively.
The maximum aperture at each end is 1.3 and 3.5, respectively. The
configurable minimum and maximum focus lengths of the camera
module are 15 cm and infinite, respectively.
The focus-bracketing procedure was rate-limited by the focus
step-motor lag time; the computing time of Eq. (4) for the AFRFM
used for bracketing was almost negligible compared to the lag
time. Therefore, the number of focus steps to be searched within
the search range was set based on the time needed to move the
step-motor and the user-defined allowable time for bracketing. The
size of the search range was chosen to limit bracketing times to
less than 1 second. In the platform used, this corresponded to a
search range of 350 steps. Reducing the focus search range further
improves imaging speed. Due to the optical relationship between
in-focus positions and zoom depths, it is not necessary to search
through every step-motor position. Peddigari et al. [32] performed
an experiment to determine the search range boundaries for vari-
able zoom depths by noting the absolute in-focus step-motor posi-
tion for an object at infinity and for an object closer to the camera.
Their experiment shows that the effective focus range for search-
ing can be reduced to be less than 50 percent of the total focus
range when the zoom depth is fixed. Based on this experiment,
we employed the reduced search range suggested by Peddigari et
al. [32] for each zoom depth.
The window size of the averaging filter and the size of block
B m,n were set to 7 × 7 and 5 × 5, respectively. Larger sizes may give
a better result, but at the expense of increased complexity. Exper-
iments on various test image sequences showed that larger filter
sizes negligibly improved image quality. The pre-defined maximum
number of allowable selected peaks in the focus-bracketing algo-
rithm was chosen to be 4 to satisfy the requirement that the fusion
time for FHD images be within 1 second. The threshold value λ for
the fusion rule was experimentally chosen to be 1.4.
3.2. Fusion results
We use seven sample image sequences to evaluate the perfor-
mance of the proposed method. We have shown the images fused
by the proposed method for each sequence in Fig. 4. We denoted
a region of interest (ROI) for each image sequence with a red box.
We will use these ROIs for our subjective tests later in this section.
We have also shown the images selected for fusion from each se-
quence as the result of our focus-bracketing algorithm in Fig. 5. For
displaying focus differences, we have cropped and enlarged parts
the selected images in Fig. 5. All image sequences included at least
two critical in-focus positions, which is typical of most scenes vi-
sualized by multi-focus image-fusion techniques.
Four fusion methods: “Z” [36], “H” [37], “C” [38], and the SML-
based fusion method [20], were selected for performance com-
parisons. “Z”, “H”, and “C” were selected as well-known offline
software used in image-fusion research. “Z” provides two fusion
rules: PMax and DMap. The PMax is a pyramidal style fusion rule
and the DMap is a depth map style fusion rule. “H” provides three
fusion rules: methods A, B, and C. The method A takes a weighted
average of all pixels from all source images. The method B selects
the sharpest pixel using a depth map, and the method C uses a
pyramidal style approach. On the other hand, “C” has a variety of
options: Do Stack, Do Soft Stack, Do Weighted Average, Pyramid
Weighted Average, Pyramid Do Stack, Pyramid Maximum Contrast.
For this paper’s experiment, we have chosen the DMap, method B,
and the Pyramid Do Stack for “Z”, “H”, and “C”, respectively, be-
cause the selected option has shown the best objective test result
for each set of sample image sequences. The SML-based method
was selected because it provided the best and most stable results
for the sample image sequences used here out of the methods re-
viewed in [20] and [23]. The SML-based method was implemented
as in Tian et al.’s evaluation paper [20]. Unfortunately, all the com-
pared methods do not provide their own focus-bracketing algo-
rithm. They perform the fusion for given input images. Therefore,
we compare fusion performances for the input images selected by
our focus-bracketing algorithm.
The mutual information-based quality metric M FX Y [39] was
used in our objective performance tests. The experiment is to be
performed on the original multi-focused images, not on the arti-
ficially generated ones. Therefore, it is assumed that the ground
truth fused image does not exist. M FX Y is the metric that does
 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210 205

Fig. 4. Images fused by the proposed method for each sample image sequence with ROIs. (For interpretation of the references to color in this figure, the reader is referred to
the web version of this article.)

not require the ground truth reference image for computation. where F is the fused image; I ( F , X (k) ) is the mutual information
M FX Y was chosen because it is the only metric known to the au- between F and X (k) ; H ( F ) and H ( X (k) ) are the entropy of F and
thors that can compare fusion results between more than two X (k) , respectively; and K is the number of source images. The val-
(K )
source images and is the most widely used fusion performance ues of M FX for the fused results are listed in Table 1. A larger
comparison test in recently published papers. The quality met- value corresponds to better performance. By this metric, the pro-
ric for a fused image of two source images X and Y , M FX Y , was posed method is superior.
extended for the case of more than two source images, X (k) , For subjective tests, images fused by compared methods are
k = 1, 2, · · · , K , as follows: shown in Fig. 6. Since performance differences are most clear in
textured regions and between adjacent in-focus regions, a portion
of each fused image including both types of regions was selected
(K ) 
K
2I ( F , X (k) )
M FX = , (6) as indicated as ROIs in Fig. 4, enlarged, and presented in Fig. 6.
H ( F ) + H ( X (k) ) Comparisons may be summarized as follows:
k =1
206 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210

Fig. 5. Focus bracketing selected images of sample image sequences.

1) For both object and background, the result of the SML-based 2) “Z”, “H”, and “C” tend to lose background details in the adja-
method is smoother than that of other methods, as observed cent areas with sharp edge, as indicated by the red arrows in
in all images in Fig. 6. Fig. 6(a), (b), (c), (d), (e), and (g).
 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210 207

Fig. 6. Subjective comparison with other methods for the ROIs in Fig. 4. (For interpretation of the references to color in this figure, the reader is referred to the web version
of this article.)

3) “Z”, “H”, and “C” show zigzag distortions on diagonal edges
between object and background, as indicated by the yellow
arrows in Fig. 6(a) and (c).
4) “Z”, “H”, and “C” lose thin structures such as hairs, as indicated
by the red arrow in Fig. 6(f).
5) “Z” and “H” tend to lose weak shadows, as indicated by the
yellow arrow in Fig. 6(f).
6) Compared to other methods, noticeable distortion is not found
in the images fused by the proposed method.
It is interesting that the SML-based method obtained almost
(K )
compatible M FX scores with “Z” and “H” but resulted in subjec-
tively poorer quality images. This suggests that further research is
needed to improve the state-of-the-art objective metrics for evalu-
208 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210

Fig. 6. (continued)

Table 1 fusing, so most of the fusion was performed by the selection rule
(K )
M FX Comparison with other methods. which required no computation. Table 2 shows the execution times
Image sequence “Z” “H” “C” SML Proposed for all sample image sequences and fusing methods. As shown in
P1 1.0361 0.9684 0.7930 0.9767 1.2369 the table, the proposed and SML-based methods are significantly
P2 1.1154 1.1527 0.8512 1.0804 1.1923 less complex than “Z”, “H”. and “C”. For sequences of less than
P3 1.4579 1.4246 1.3419 1.5564 1.5969 four input images, the complexity of the proposed method is com-
P4 0.9246 0.9117 0.8381 0.9793 1.1597
parable to that of the SML-based method. For sequences of four
P5 0.8883 0.9041 0.6240 0.7691 0.9975
P6 0.8706 0.8654 0.6731 0.8365 0.8739 input images, the complexity of the proposed method is slightly
P7 0.9234 0.9423 0.8935 1.1258 1.1318 higher than that of the SML-based method. When fusing four FHD-
sized source images, the proposed method performed the fusion in
Average 1.0309 1.0242 0.8593 1.0463 1.1699
about 1 second on the experimental platform.

ating multi-focus image fusion. Nevertheless, the proposed method 4. Conclusion

is superior in both objective and subjective tests.
Besides the quality of fused images, the computing efficien-
cies were evaluated by measuring execution times. The proposed
AFRFM was fast enough to be implemented in state-of-the-art dig-
ital cameras owing to the simplicity of the AFRFM average filter
computation and applied fusion rule. Furthermore, most fused pix-
els belonged to an in-focus region in one of the images selected for
In this paper, we propose an AIF imaging method that can be
implemented in digital cameras. A new focus measure, AFRFM,
was proposed. While most previous focus measures are calculated
independently for each input image, the AFRFM calculates the rel-
ative focus measure between a pair of input images. Based on the
observation that a well-focused region in an image contains bet-
 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210 209

Table 2 [10] R. Redondo, F. Šroubek, S. Fischer, G. Cristóbal, Multifocus image fusion using
Comparison of fusing method execution times (in seconds). the log-Gabor transform and a multisize windows technique, Inf. Fusion 10 (2)
(2009) 163–171.
Image size Image sequence “Z” “H” “C” SML Proposed
[11] Q. Miao, C. Shi, P. Xu, M. Yang, Y. Shi, Multi-focus image fusion algorithm based
VGA P1 1.423 0.531 1.102 0.070 0.069 on shearlets, Chin. Opt. Lett. 9 (4) (2011) 041001.
P2 2.023 0.703 1.169 0.089 0.115 [12] S. Zhenfeng, L. Jun, C. Qimin, Fusion of infrared and visible images based on
P3 2.314 0.669 1.103 0.087 0.110 focus measure operators in the curvelet domain, Appl. Opt. 51 (12) (2012)
P4 1.286 0.600 1.067 0.067 0.067 1910–1921.
P5 1.183 0.634 1.026 0.069 0.070 [13] X. Zhang, X. Li, Z. Liu, Y. Feng, Multi-focus image fusion using image-partition-
P6 1.485 0.517 1.025 0.070 0.069 based focus detection, Signal Process. 102 (2014) 64–76.
P7 2.606 0.771 1.099 0.106 0.163 [14] S. Li, B. Yang, Multifocus image fusion using region segmentation and spatial
Average 1.760 0.632 1.084 0.079 0.095 frequency, Image Vis. Comput. 26 (7) (2008) 971–979.
[15] Z. Zhou, S. Li, B. Wang, Multi-scale weighted gradient-based fusion for multi-
FHD P1 5.451 3.309 5.012 0.360 0.355
focus images, Inf. Fusion 20 (2014) 60–72.
P2 7.406 3.617 5.748 0.526 0.669
[16] J. Duan, G. Meng, S. Xiang, C. Pan, Multifocus image fusion via focus segmen-
P3 7.766 3.446 5.336 0.507 0.682
tation and region reconstruction, Neurocomputing 140 (2014) 193–209.
P4 5.023 3.103 5.672 0.357 0.341
[17] S. Li, X. Kang, J. Hu, B. Yang, Image matting for fusion of multi-focus images in
P5 5.503 3.394 5.10 0.358 0.348
dynamic scenes, Inf. Fusion 14 (2) (2013) 147–162.
P6 4.406 3.102 5.4 0.357 0.346
[18] L. Chen, J. Li, C.P. Chen, Regional multifocus image fusion using sparse repre-
P7 10.069 3.977 5.928 0.653 1.044
sentation, Opt. Express 21 (4) (2013) 5182–5197.
Average 6.518 3.421 5.457 0.445 0.541
[19] S. Li, J.T. Kwok, Y. Wang, Combination of images with diverse focuses using the
spatial frequency, Inf. Fusion 2 (3) (2001) 169–176.
[20] J. Tian, L. Chen, L. Ma, W. Yu, Multi-focus image fusion using a bilateral
ter contrast, sharpness, and details than the corresponding region gradient-based sharpness criterion, Opt. Commun. 284 (1) (2011) 80–87.
that is defocused in another image and the average filtered ver- [21] H. Li, Y. Chai, H. Yin, G. Liu, Multifocus image fusion and denoising scheme
sion of a well-focused region shows a higher correlation to the based on homogeneity similarity, Opt. Commun. 285 (2) (2012) 91–100.
corresponding defocused region in another image than the original [22] V. Naidu, Multi focus image fusion using the measure of focus, J. Opt. 41 (2)
(2012) 117–125.
well-focused version, the AFRFM was proposed. The AFRFM has [23] W. Huang, Z. Jing, Evaluation of focus measures in multi-focus image fusion,
shown that it allows for a simple calculation of the focus differ- Pattern Recognit. Lett. 28 (4) (2007) 493–500.
ence between two images and provides a recursive rule for fusing [24] T.T.E. Yeo, S. Ong Jayasooriah, R. Sinniah, Autofocusing for tissue microscopy,
more than two images. Image Vis. Comput. 11 (10) (1993) 629–639.
[25] S.K. Nayar, Y. Nakagawa, Shape from focus, IEEE Trans. Pattern Anal. Mach. In-
A focus bracketing algorithm using AFRFM was implemented
tell. 16 (8) (1994) 824–831.
for bracketing multi-focus image sequences and AIF imaging on a [26] K.-S. Choi, J.-S. Lee, S.-J. Ko, New autofocusing technique using the frequency
digital camera platform. The proposed method outperformed pre- selective weighted median filter for video cameras, IEEE Trans. Consum. Elec-
viously published methods in constructing high-resolution AIF im- tron. 45 (3) (1999) 820–827.
ages from several critical image sequences in both subjective and [27] V. Aslantas, R. Kurban, A comparison of criterion functions for fusion of multi-
focus noisy images, Opt. Commun. 282 (16) (2009) 3231–3242.
objective tests of quality and efficiency. [28] H. Zhao, Z. Shang, Y.Y. Tang, B. Fang, Multi-focus image fusion based on the
The proposed method works for still images. Future research neighbor distance, Pattern Recognit. 46 (3) (2013) 1002–1011.
will focus on extending this method for videos. [29] K. Panetta, C. Gao, S. Agaian, No reference color image contrast and quality
measures, IEEE Trans. Consum. Electron. 59 (3) (2013) 643–651.
[30] H. Yeganeh, Z. Wang, Objective quality assessment of tone-mapped images,
Acknowledgments IEEE Trans. Image Process. 22 (2) (2013) 657–667.
[31] M. Narwaria, M.P. Da Silva, P. Le Callet, R. Pepion, Tone mapping based HDR
This work was supported by the ICT Standardization program of compression: does it affect visual experience?, Signal Process. Image Commun.
29 (2) (2014) 257–273.
MSIP (The Ministry of Science, ICT & Future Planning).
[32] V. Peddigari, M. Gamadia, N. Kehtarnavaz, Real-time implementation issues in
passive automatic focusing for digital still cameras, J. Imaging Sci. Technol.
Appendix A. Supplementary material 49 (2) (2005) 114–123.
[33] J. Jeon, I. Yoon, D. Kim, J. Lee, J. Paik, Fully digital auto-focusing system with
automatic focusing region selection and point spread function estimation, IEEE
Supplementary material related to this article can be found on- Trans. Consum. Electron. 56 (3) (2010) 1204–1210.
line at http://dx.doi.org/10.1016/j.dsp.2016.09.005. [34] N. Kehtarnavaz, H.-J. Oh, Development and real-time implementation of a rule-
based auto-focus algorithm, Real-Time Imaging 9 (3) (2003) 197–203.
[35] G.D. Evangelidis, E.Z. Psarakis, Parametric image alignment using enhanced cor-
References
relation coefficient maximization, IEEE Trans. Pattern Anal. Mach. Intell. 30 (10)
(2008) 1858–1865.
[1] S. Pertuz, D. Puig, M.A. Garcia, A. Fusiello, Generation of all-in-focus images [36] ZereneStacker, http://www.zerenesystems.com.
by noise-robust selective fusion of limited depth-of-field images, IEEE Trans. [37] HeliconFocus, http://www.heliconsoft.com.
Image Process. 22 (3) (2013) 1242–1251. [38] CombineZP, http://www.hadleyweb.pwp.blueyonder.co.uk.
[2] A. Saha, G. Bhatnagar, Q.J. Wu, Mutual spectral residual approach for multifocus [39] M. Hossny, S. Nahavandi, D. Creighton, Comments on ‘information measure for
image fusion, Digit. Signal Process. 23 (4) (2013) 1121–1135. performance of image fusion’, Electron. Lett. 44 (18) (2008) 1066–1067.
[3] K.-L. Hua, H.-C. Wang, A.H. Rusdi, S.-Y. Jiang, A novel multi-focus image fusion
algorithm based on random walks, J. Vis. Commun. Image Represent. 25 (5)
(2014) 951–962. Oh-Jin Kwon received his M.S. degree from the University of Southern
[4] V.S. Petrović, C.S. Xydeas, Gradient-based multiresolution image fusion, IEEE California, Los Angeles in 1991, and his Ph.D. degree from the University of
Trans. Image Process. 13 (2) (2004) 228–237. Maryland, College Park in 1994; both were in electrical engineering. From
[5] W. Wang, F. Chang, A multi-focus image fusion method based on Laplacian 1984 to 1989, he was a researcher at the Agency for Defense Development
pyramid, J. Comput. 6 (12) (2011) 2559–2566. of Korea, and from 1995 to 1999, he was the head of Media Lab in Sam-
[6] G. Pajares, J.M. De La Cruz, A wavelet-based image fusion tutorial, Pattern
sung SDS Co., Ltd., Seoul, Korea. Since 1999, he has been a faculty member
Recognit. 37 (9) (2004) 1855–1872.
at Sejong University, Seoul, Korea, where he is currently an Associate Pro-
[7] J.J. Lewis, R.J. O’Callaghan, S.G. Nikolov, D.R. Bull, N. Canagarajah, Pixel- and
region-based image fusion with complex wavelets, Inf. Fusion 8 (2) (2007)
fessor. His research interests are image and video coding, watermarking,
119–130. analyzing, fusion, and processing.
[8] Q. Zhang, B.-l. Guo, Multifocus image fusion using the nonsubsampled con-
tourlet transform, Signal Process. 89 (7) (2009) 1334–1346. Seungcheol Choi received B.S. and M.S. degrees in computer science
[9] S. Yang, M. Wang, L. Jiao, R. Wu, Z. Wang, Image fusion based on a new con- from Sejong University, Seoul, Korea in 1998 and 2001, respectively. He is
tourlet packet, Inf. Fusion 11 (2) (2010) 78–84. currently pursuing his Ph.D. in electronics engineering at the same univer-
210 O.-J. Kwon et al. / Digital Signal Processing 60 (2017) 200–210

sity. From 2001 to 2013, he was a researcher at Galaxia Communications.
His research interests are image and video coding, high dynamic range
imaging, image processing, image fusion, and JPEG.
Dukhyun Jang is currently an M.S. student in electronics engineering
at Sejong University, Seoul, Korea. His research interests are image pro-
cessing and fusion.
Hee-Suk Pang received the B.S., M.S., and Ph.D. degrees in electronics
engineering from Seoul National University in 1994, 1996, and 2001, re-
spectively. From 2001 to 2008, he was with DVC Gr., DM Research Lab., LG
Electronics as a chief research engineer. Since 2008, he has been a faculty
member with Sejong University, Seoul, Korea, where he is currently an
Associate Professor. His research interests are signal processing and audio
coding systems.