School of Electrical Engineering and Computer Science

Kyungpook National Univ.

Multiscale
Multiscale
image sharpening
image sharpening
adaptive to edge profile
adaptive to edge profile
Journal of Electronic Imaging,
Journal of Electronic Imaging,
v
v
ol. 14, no. 1,
ol. 14, no. 1,
Jan.
Jan.
-
-
Mar. 2004
Mar. 2004
Hiroaki Kotera and Hui Wang
2 / 39
New image sharpening method
Properties
Adaptation to the local edge slopes
Suppression of background noises
Method
Transforming RGB to YIQ space for only managing
luminance image
Prescanning of image with GD (Gaussian Derivative) filters
Generating of edge map consisted of hard, medium, and soft
edges from the edge image
Applying GD filters to separated area in an edge map
Using a Gaussian smoothing filter for flat area
Inversing YIQ to RGB area
Abstract
Abstract
3 / 39
Introduction
Introduction
Purpose
Sharpening the blurred image taken by a
conventional digital camera or scanner with normal
sensor noise
Properties of classical sharpening models
Example : USM (unsharp masking)
Sensitivity to noise
Overshoot artifacts due to enhancing high-contrast
area
Inappropriateness for RGB
Insufficient denoising function
4 / 39
Diffusion model analogous to thermal image
Blurring
Thermal image
Time varying image :
Blurred image :
First order approximation
Original sharp image
The same as Laplacian or simplest USM method
Review of Image sharpening models
Review of Image sharpening models
t t f f t f + ) ( ) 0 ( ) (
) (
2 2 2 2 2
y f x f k f k t f + = =
g g y f x f g f
2 2 2 2 2
) ( +
where : the diffusion constant
k
(1)
(2)
(3)
. ) )( 2 1 ( ) ( ) 0 ( ) (
2 2 2
L + + + = t t f t t f f t f
(4)
5 / 39
High-pass filter model and denosing operator
L-USM (linear USM)
Models to suppress noises in uniform area
A-USM (adaptive USM) model
Two directional Laplacian operators
Sensitivity for detail area with medium contrast
Insensitivity for uniform area
) , ( ) , ( ) , ( ) , ( ) , ( ) , ( y x z y x y x z y x y x g y x f
y y x x
+ + =
) , 1 ( ) , 1 ( ) , ( 2 ) , ( y x g y x g y x g y x z
x
+ =
) 1 , ( ) 1 , ( ) , ( 2 ) , ( + = y x g y x g y x g y x z
y
where
) , ( ) , ( ) , ( y x Z y x g y x f
t
+ =
) , ( ) , ( ) , ( y x z y x g y x f + =
where : a positive scaling factor
: a blurred input image

) 1 , ( ) 1 , ( ) , 1 ( ) , 1 ( ) , ( 4 ) , ( + + = y x g y x g y x g y x g y x g y x z
) , ( y x g
(5)
(6)
(7)
(8)
6 / 39
Cubic USM (C-USM) model
Two types of operators
Sensitivity to high gradient edge
Less sensitivity to slow gradient edge
Separable cubic (SC-USM)
Nonseparable cubic (NSC-USM)
SNSC-USM
Average of the SC-USM and the NSC-USM
)] 1 , ( ) 1 , (
) , ( 2 [ )] 1 , ( ) 1 , ( [ )] , 1 (
) , 1 ( ) , ( 2 [ )] , 1 ( ) , 1 ( [ ) , (
2
2
+
+ + +
+ =

y x g y x g
y x g y x g y x g y x g
y x g y x g y x g y x g y x z
USM SC
)] 1 , (
) 1 , ( ) , 1 ( ) , 1 ( ) , ( 4 [
)] 1 , ( ) 1 , ( ) , 1 ( ) , 1 ( [ ) , (
2
+
+ +
+ + =

y x g
y x g y x g y x g y x g
y x g y x g y x g y x g y x z
USM NSC
(9)
(10)
7 / 39
Rational USM(R-USM) model
Higher enhancement in the detail zone
Lower enhancement in the uniform zone
Wavelet USM (W-USM) model
Multi-scale gradients of the wavelet transform
Independence of the variance of images and noises
method in DIP book
FWT of the noise image with a wavelet function
Threshold detail coefficient with hard or soft thresholding
Reconstruction with original approximation coefficient
)] , ( ) , ( ) , ( ) , ( [ ) , ( ) , ( y x z y x c y x z y x c y x g y x f
y y x x
+ + =
(11)
where ,
) , (
) , (
) , (
2
h y x kg
y x g
y x c
x
x
x
+
= ,
) , (
) , (
) , (
2
h y x kg
y x g
y x c
y
y
y
+
=
(12)
, )] , 1 ( ) , 1 ( [ ) , (
2
y x g y x g y x g
x
+ =
. )] 1 , ( ) 1 , ( [ ) , (
2
+ = y x g y x g y x g
x
(13)
8 / 39
Lower-upper-middle (LUM) filter
Usage for both smoothing and sharpening
Insensitivity to additive noise and removing impulse noise
9 / 39
Vision
Vision
-
-
Based Edge
Based Edge
-
-
Sharpening Operator
Sharpening Operator
Sharpening models
Gaussian derivative (=Hermite polynomialGaussian)
Gabor(=cosineGaussian)
DOG (Difference of Gaussian)
Difference-of-offset-Gaussian
Difference-of-offset-(DOG)
Fig. 1. Typical edge-sharpening operator
based on the human visual field model.
10 / 39
Basic equations of GD
Basic form of Gaussian
Second derivative
Edge signals and result of prescanning
. ,
2
exp
2
1
) (
2 2 2
2
2
2
y x r
r
r G + =
|
|
.
|

\
|
=

|
|
.
|

\
|

|
|
.
|

\
|
=
+ =
2
2
2
2
2
2 2 2 2 2
2
exp 1
2
1
) ( ) ( ) (

r r
y r G x r G r G
) , ( ) , ( ) , (
2
y x g y x G y x =
(14)
(15)
(16)
11 / 39
Multi
Multi
-
-
scale Adaptive Sharpening by Edge
scale Adaptive Sharpening by Edge
Classification
Classification
Fig. 2-1. Multi-scale edge adaptive image
sharpening process.
Procedure of multi-scale adaptive sharpening
Transform the RGB image to
YIQ image
Different characteristics edges
depending on each channel
Preservation of gray balance on
the edge by using only luminance
part
Extract edge area by
prescanning GD filter with
Design the edge histogram
Calculation
H

3 2 1

12 / 39
Fig. 2-2. Multi-scale edge adaptive image
sharpening process.
Segment into multiple edge
zones with hard, medium, soft
and flat gradients with
for making an edge map
from prescanned image
Apply GD filter with each
to sharpen the edge map
and Apply Gaussian
smoothing operator for flat
area (denoising)
Transform YIQ to RGB
H

) , , (
3 2 1

13 / 39
RGB
Edge image
edge histogram
Y
RGB
RGB to YIQ transform matrix
Prescanning with a GD filter and
S

Calculation
S

+
=
) , ( ) , (
) , ( ) , (
) , (
y x g y x G
y x y x g
y x f
m

for edge area

for flat area
0 ) , ( y x M
0 ) , ( = y x M
edge map ) , ( y x M
= ) , ( y x M
) , ( 3
) , ( 2
) , ( 1
) , ( 0 0
4
4 2
2 1
1
y x for
y x for
y x for
y x for
S
S
S
S

<
<
<
: gentle or flat areas
: soft edges
: medium edges
: hard edges
YIQ to RGB transform matrix
Calculation
H

H H H
a a a 3 : 2 : ) , , (
3 2 1

) , ( ) , ( ) , (
2
y x g y x G y x =
|
|
.
|

\
|

|
|
.
|

\
|
=
2
2
2
2
2
2
2
exp 1
2
1
) (

r r
r G
(
(
(

(
(
(

=
(
(
(

B
G
R
Q
I
Y
311135 . 0 522591 . 0 211456 . 0
321263 . 0 274453 . 0 595716 . 0
114 . 0 587 . 0 299 . 0

8 . 0 ~ 5 . 0 = a where
) , ( y x
Y from YIQ
14 / 39
GD filter design
Conditions
The filter is approximated by an square matrix.
The weights of the GD filters should be equivalent to the
local integral of continuous GD functions in between
discrete lattice points.
The sum of GD filter weights is to be equal to zero so as
not to respond to flat signals.
M M
15 / 39
Decision of a matrix for the first condition
Decision the size of
Dependence on
Sufficient size to describe the receptive field :
Matrix of square

8 M
] [
ij
w = W
M M
Fig. 3. GD filter design in the zero sum condition.
zero cross point
minimum peak
16 / 39
Decision of weights for the second condition
Choice an odd integer
Calculation weights between the lattice points
Compensation of weights for the third condition
Omission the negative weights outside of
Unsuitability for the zero sum condition
) 1 2 ( + = m M
] , [ j i

+

+
+
=
5 . 0
5 . 0
5 . 0
5 . 0
2
)] , ( [
j
j
i
i
ij
dxdy y x G w
8 M

= =
+
> + = =
m
m i
m
m j
ij
W W w W 0
where : the sum of the weights
: the positive entries of
: the positive entries of
W
,

= =
+ +
=
m
m i
m
m j
ij
w W

= =

=
m
m i
m
m j
ij
w W
+
ij
w

ij
w
] [
ij
w
] [
ij
w
(20)
(19)
17 / 39
Modification the weights
New weight
Correction of only part

= =
=
m
m i
m
m j
ij
w 0
'
where : corrected weights
] [ ] [ ] [
' ' +
+ =
ij ij ij
w w w
0 = +
+
kW W

=
ij ij
kw w
'

= W W k / 1
] [

ij
w
] [ ] [ ] [
' ' +
+ = =
ij ij ij
w w w
'
W
(21)
(22)
(23)
18 / 39
Design of edge map
Decision of
Calculation of from prescanned image
Results from the experimental with the dozens of images
Best choice : is around to
Ratio of :
)) , ( ( y x M
= ) , ( y x M
) , ( 3
) , ( 2
) , ( 1
) , ( 0 0
3
3 2
2 1
1
y x for
y x for
y x for
y x for
S
S
S
S

<
<
< : gentle or flat areas
: soft edges
: medium edges
: hard edges
where : edges signals
) , ( y x
H
5 . 0
1

H
8 . 0

3 : 2 : 1 ) , , (
3 2 1

H

) , , (
3 2 1

(17)
19 / 39
Application filters to an edge map
Transformation YIQ to RGB

+
=
) , ( ) , (
) , ( ) , (
) , (
y x g y x G
y x y x g
y x f
m

for edge area

for flat area
0 ) , ( y x M
0 ) , ( = y x M
where is edge signals with
is a Gaussian smoothing filter
) , ( y x
m

) , ( y x G
H H H
a a a 3 : 2 : ) , , (
3 2 1

(
(
(

(
(
(

=
(
(
(

B
G
R
Q
I
Y
311135 . 0 522591 . 0 211456 . 0
321263 . 0 274453 . 0 595716 . 0
114 . 0 587 . 0 299 . 0
(18)
20 / 39
Experimental Results
Experimental Results
Edge map tuning
Prescanning with
Example of histogram
from prescanning image
Decision the range of edge map
Calculation of from prescanning image
Choice of : around to
Usage of
Fig. 4. Typical dispersion in an edge
histogram dependent on image content.
6 . 0 =
S

H H H
a a a 3 : 2 : ) , , (
3 2 1

1

H
5 . 0
H
8 . 0
H

21 / 39
Fig. 5. Map of standard deviation in an edge histogram for typical images.
H

standard deviation of edge histogram

H

Variety of standard deviations depending on images

22 / 39
Fig. 6(a). Edge map and edge histogram for image swallowtail with
. 5 . 35 =
H

H H H


9 . 0 : 6 . 0 : 3 . 0
) , , (
3 2 1

H H H


5 . 1 : 0 . 1 : 5 . 0
) , , (
3 2 1

H H H


4 . 2 : 6 . 1 : 8 . 0
) , , (
3 2 1

Test images to find optimal
1

23 / 39
H H H


9 . 0 : 6 . 0 : 3 . 0
) , , (
3 2 1

H H H


5 . 1 : 0 . 1 : 5 . 0
) , , (
3 2 1

H H H


4 . 2 : 6 . 1 : 8 . 0
) , , (
3 2 1

Fig. 6(b). Edge map and edge histogram for image bride with
. 8 . 7 =
H

24 / 39
Comparison with improved USM methods for
a black and white image
Test images
Lena1 : normal image with a small background
Lena2 : a blurred image with Gaussian noise
Lena3 : a degraded image corrupted by heavy impulse
noise
Value of parameters

for smoothing by Gaussian filter

6 . 0 =
S

8 . 1 : 2 . 1 : 6 . 0 ) , , (
3 2 1

8 . 0 =
f

25 / 39
Result
Better enhancement in the facial close-up with smoothed
surface and less noise
Fig. 7. Comparison with typical improved USM methods for black and white Lena1.
26 / 39
Fig. 8. Comparison with typical improved USM methods for black and white Lena2.
Better denoising in the facial area
27 / 39
Addition a well-known median filter as a preprocessor
MGD is not good reducing impulse noise
Better sharpened details in M-Russo
Better smoothed in facial area with proposed method
Fig. 9. Comparison with typical improved USM methods for black and white Lena3.
28 / 39
Sharpness Factor
Sharpness Factor
ES (edge sharpness)
Measurement the enhanced edge component
existing only in the edge areas
Calculation the integrated absolute amplitude of
edge enhancing signal divided by edge area
Obtainment
Counting the pixel numbers with in flat area of
edge map
E
E
A
dxdy y x s y x f
ES


=
) , ( ) , (
filt
where is sharpening filter
is the amount of edge area
filt
s
E
A
E
A
0 ) , ( = y x M
(24)
29 / 39
FS (frequency sharpness)
Enhancement Fourier spectra after sharpening
MSE (mean square error)

=


d V G
d V G F
FS
) ( ) (
) ( ) ( ) (
where is the original Fourier spectra
is the sharpened Fourier spectra
is the human visual function
) ( G
) ( F
) ( V

= dxdy y x g y x f MSE
2
] ) , ( ) , ( [
(25)
(26)
30 / 39
(noise power in flat area)
Noise power only in flat area
Result of experimental
f
N

= dxdy y x g y x f y x E N
flat f
2
] ) , ( ) , ( )[ , (
where is a gate function to pass only flat area signals ) , ( y x E
flat
Table. 1. Comparison of image quality measure for Lena1.
(27)
31 / 39
Table. 2. Comparison of image quality measure for Lena2.
Table. 2. Comparison of image quality measure for Lena3.
32 / 39
Sharpening for color image
Sharpening for color image
Procedure from a RGB image to YIQ image
Obtaining a gamma-corrected camera image
Transforming RGB into a linear sRGB and
operated on an RGB-to-YIQ linear matrix
Fig. 10. Edge coloring in RGB-independent sharpening versus Luminance Y sharpening.
(a) Original (b) RGB independent (c) Sharpening Y in YIQ
33 / 39
(a) Original (b) Single GD (c) Proposed Multi GD (d) Edge map
(e) Edge histogram
(a) Original (b) Single GD (c) Proposed Multi GD (d) Edge map
(f) Edge histogram
Fig. 11. Tuned edge map and sharpened images.
34 / 39
Comparison of color sharpened images
(a) Original
(d) Original
(a) Original
(d) Original
(b) L-USM (c) Laplacian
(b) L-USM (c) Laplacian
(e) Single GD (f) Proposed Multi GD
(e) Single GD (f) Proposed Multi GD
Fig. 12. Comparison in sharpened color images.
35 / 39
(a) Original image A (b) Edge map (c) Edge histogram
(d) Original foreground
(g) Original background
(e) Single GD (f) Proposed Multi GD
(h) Single GD (i) Proposed Multi GD
Fig. 13-1. Adaptive sharpening effect with smoothing.
36 / 39
Fig. 13-2. Adaptive sharpening effect with smoothing.
(j) Original image B
(k) Edge map
(l) Single GD (m) Proposed Multi GD
37 / 39
(a) Selected scan line in image (b) scan line on the edge map
(a) Enlarge image profiles before and after sharpening on scan line
Fig. 14. Adaptive sharpening effect with smoothing.
Edge Profile Comparison
Edge Profile Comparison
38 / 39
Sharpness Factors in Color Images
Sharpness Factors in Color Images
Fig. 15. Estimated quality factors.
Estimation
ES
Single GD > Multi-GD
Over-sharpening in single GD
FS
Lift up the spatial frequency
components in the visible spatial
frequency range

Reducing noise
f
N
39 / 39
A novel adaptive multi-filtering method
Work on the YIQ space
Usage of GD-filter with and considering
edge map, and Gaussian filter for flat area in edge
map to reduce noises
Excellent results of sharpness and denoising
Drawback
Time-consuming
Discussion and Conclusion
Discussion and Conclusion
S

40 / 39
41 / 39