IMAGE PROCESSING

Construction of holographic images using cross

sample modified laplacian method

Abstract—This paper presents a important issues in computer vision.

shape-from-focus method,which is Techniques executing this task are often
improved with regard to the referred to as shape-from-x, where x is a
mathematical operator used for visual cue such as shading, texture,
contrast measurement, the selection of contour, focus, stereo, and motion [1].
the neighborhood size, surface Blurring phenomena due to defocusing
refinement through interpolation, and are among the important cues for depth
surface postprocessing. Three- recovery. Shape-from focus is a method
dimensional models of living human where a sequence of images is taken
faces are presented with such a high while changing the focus setting in small
resolution that single hairs are visible. steps. One image of such a sequence,
Key words—Autofocusing, depth further on referred to as focus series,
estimation, depth-from focus, facial contains a focused region of the imaged
measurement, focus analysis, focusing, object as well as an unfocused
focus measure, holography, shape background. By determining the focus
estimation, shape-from-focus, 3-D setting, that locally optimizes image
reconstruction. focus, 3-D information about the imaged
object is retrieved [2]. If the object is
moving and cannot be expected to
I. INTRODUCTION remain in a certain position during the
capturing time, an alternative method for
THE acquisition of 3-D information out 3-D measurement is holography, where
of 2-D images is one of the most part of the 3-D information about the
recorded object is stored by This is done optically, if thehologram
superimposing the back-scattered light was stored on an analogue recording
coherently with a reference wave. material. It is realized by illuminating
Through this, the amplitude and the the hologram with the complex
phase information of the scattered light conjugate reference beam, which
field are stored as an interference produces the so-called real image, a 3-D
pattern, for example in an analogue light field with one-to-one
photosensitive material. This kind of correspondence to therecorded object. It
acquisition of 3-D data has the has to be digitized into a set of 2-D
tremendous advantage of storing the projections, because of the 2-D nature of
high-resolved information in an digital recording devices.The result of
extremely short amount of time, namely the holographic reconstruction is a set of
in the duration of the recording laser images containing focused and
pulse (35 ns for the setup described unfocused regions similar to the focus
here). However, the only way to get series gained through subsequent
access to the stored information is recording. This is why the same methods
reconstructing the hologram. designed for shape-from-focus in
microscopy are applicable for
holographic reconstructions [3]–[7].

This paper discusses improvements of

shape-from-focus methods, applicable to
all kind of focus series. The application,
Fig. 1. Optical reconstruction of the that provides the exemplary material for
hologram and digitalization of a set of the discussion, is the analogue
2-D projections at different axial holographic recording of living human
positions. faces. For that kind of application, the
full advantage of the extremely short
recording time arises since motion
artifacts caused by changes of the face
due to mimic, breathing, or heartbeat can
be avoided. The completely focused
image, extracted from the focus series in
combination with the 3-D information,
serves as precisely fitting texture. Such
highly resolved textured 3-D models of
living human faces are of. great interest
in many kinds of applications, among
others in medicine and forensicscience
[8]–[14].
Fig. 1(a) shows exemplarily one image
from the focus series and (b) screenshot
of the resulting, textured 3-D computer
model of the face.
. registration between the model and the
SHAPE-FROM-FOCUS
The considered focus series may consist
of gray-scale images, each characterized
by their intensity distribution in the
plane with the coordinate . The
distinction between focused and
unfocused image parts is realized with a
so-called focus criterion, a mathematical
operator which measures the local image
contrast of each pixel in each image with
the axial position with the aid of a local
pixel neighborhood. These values are
further on referred to as focus values. A
maximization of the focus values along
the direction localizes the focused
regions and produces information about
the 3-D geometry of the imaged object
(see Fig. 2). This process is further on
called surface extraction. In addition to
the topometrical data, the procedure can
be used to generate an image with
unlimited depth of field by compounding
the detected sharp patches from each
image into one image described by . This
so-called texture image can be used for
texturing the resulting 3-D computer
models. The pixel-to-pixel
correspondence between texture and
geometry, which is inherent to the
described process, makes any further
texture expendable [see Fig. 1(b)].
Pioneering work in microscopic shape-
from-focus was done by Nayar [2]. It is
used for a wide range of applications
like the measurement of firearm bullets
and cartridge cases and for machine
tools for inspection purposes [15], for
material structure analysis in
mineralogical research [16] or for micro
structures such as a micro-cogwheel[17].
The optimization of the depth-form-
focus procedure depends strongly on
thenatural frequency content of the
texture of the object to be recorded. For
objects with low intrinsic contrast, active
illumination can be used [18]. In the case
of human faces, the intrinsic contrast of
the skin is sufficient for shape-from
focus.
OPTIMIZED FOCUS CRITERIA
The quintessence of the depth-from-
focus procedure is thedetection of
focused image parts, the same task that
is performed by auto focus algorithms
used in many different applications. The
performance of the different auto focus
algorithms depends strongly on the
properties of the object to be focused on,
as well as those of the imaging
systems.Alot of research was performed
for finding the suitable algorithm for the
different applications [19]–[26]. The
core of an autofocusing algorithm is a
mathematical operator, a focus criterion,
working on the pixel representation of
the digital image and assigning a
quantitative value to a whole image or a
specified part. In shape-from-focus, a
sharpness value is assigned to every
pixel. It is calculated using a local pixel
neighborhood. To find a suitable focus
criterion for holographic facial
measurement, a quantitative comparison
of the extracted facial surface was
performed with the help of a simulated
focus series with different amounts of
added noise. The evaluated criteria
were among others Tenengrad [31],
[19], Squared Gradient, Absolute
Gradient, Sum Modified Laplacian
(SML) [2], Variance, and Cross Sum
Modified Laplacian (XSML). The latter
is a newly developed modification of
Nayars SML operator, presented for the
first time in this paper. The Laplacian in
the SML takes only direct neighbors in
`the x- and y-directions into account.
Additionally, also the diagonal
neighbours could be included, weighted
with a factor of to compensate for their
larger distance to the central point where
is a square pixel neighborhood of the
parameter can be adjusted to the
characteristic size of texture elements in
the recorded scene.
A detailed discussion on the criteria and
their performance would go beyond the
scope of the paper. An in-depth study
can be found in [7]. It demonstrates that
the newly invented XSML criteria Fig. 2. For each fixed lateral point (x,y),
produces the lowest deviation between the focus values F(x,y,z) are
the extracted and a reference surface for maximized,
any amount of added noise. The XSML which leads to 3-D geometry
operator shows a slightly higher information about the object.
robustness against noise than the SML- On the other hand, small neighborhoods
criterion due to the increased number of are very sensitive to noise. The lower the
points used for the calculation. signal-to-noise ratio, the larger the
ADAPTIVE SELECTION OF neighborhood should be. Even in the
NEIGHBORHOOD SIZE absence of noise, the minimal
The choice of the neighborhood size is a neighborhood size is limited to a certain
critical step in the procedure of surface extent by the feature size of the texture
extraction. The lateral resolution of the of the recorded object. With a
ex- tracted surface is limited by the size neighborhood smaller than any feature it
of the neighborhood, since structures is impossible to distinguish between
smaller than this size cannot be resolved. focused or defocused image regions.
Thus, the neighborhood size should be When choosing a neighborhood size,one
chosen as small as possible. always has to compromise between
lateral resolution and robustness against
noise. . In regions of high intrinsic
contrast, like the mustache of the
proband, a high-quality surface can be
extracted using a neighborhood size of 3
 neighborhood size d along the direction
Z and finding the

Fig. 4. Histograms of confidence values

for a typical facial data set calculated
with a neighborhood size of 3 (a), 5 (b),

Fig. 3. Exemplary height maps extracted and 11 pixels (c). An increasing

from a holographic data set with a neighborhood size leads to the

neighborhood size of 3 (a) and 17 (b) appearance of a second peak belonging

to reliable surface points.

pixelspixels with a lateral resolution so maximum. The higher the maximum in

high that even single hairs can be comparison to the mean sharpness value,

distinguished. In regions of lower the more reliable is the generated surface

intrinsic contrast, however, the surface point.is defined as follows:

extracted with a neighborhood size of 3 (1)

is affected by noise. Using a with

neighborhood size of 17 pixels on the (2)

other hand allows for a noise resistant The normalization leads to confidence

surface extraction in regions of lower values between zero and one, a higher

intrinsic contrast, while the lateral value indicates a more reliable surface

resolution is reduced. To overcome this point. The confidence values of one

tradeoff, an adaptive algorithm was surface point differ for different

developed choosing locally the smallest neighborhood sizes. The concept of the

possible neighborhood size so that the adaptive algorithm is to start with the

surface is not corrupted by noise. It is smallest neighborhood size and accept

based on the definition of a confidence extracted surface points only if they are

value for each found surface point and reliable. If not, a larger neighborhood is

several neighborhood sizes . The surface considered. Reliability can be defined

extraction is done by comparing with the help of the confidence values by

sharpness values calculated with a

selecting a threshold . The thresholds are
determined by regarding the histogram
of the confidence values created with
different neighborhood sizes for a
typical data set of a recorded face.
Fig. 4 shows such histograms for a
neighborhood size of 3 pixels (a), 5
pixels (b), and 11 pixels (c). For a small
neighborhood size, very few reliable
surface points are present. The histogram
shows mainly a normal distribution of
confidence values of unreliable points
with a slight asymmetry caused by the
reliable points. Increasing the size of the
neighborhood, the number of reliable
points increases,so that a second peak
appears with its center shifting to higher
confidence values. Unreliable points can
be excluded if the threshold is chosen
such that the first peak is eliminated. In
this manner, the neighborhood size
depending thresholds were determined
for 25 different data sets manually.4
Fig. 5 shows the mean thresholds, as
well as an exponential fit. The resulting
curve is further on used for automated
threshold selection and the distinction
between reliable and unreliable surface
points without manually analyzing the
histograms.Limiting the maximal
neighborhood size to a certain value,
there are still points where no reliable
surface value could be found. In order to
fill these gaps, the surface extracted with
the maximal neighborhood can be
smoothed by weighted averaging.
The confidence values act thereby as
weights. These values are used to fill the
gaps if desired. The result can be seen in
Fig. 6. It shows a surface not corrupted
by noise with the maximal achievable
lateral resolution. 4An automated
selection is also possible, but is not
necessary, as is explained in the
following.
. Fig. 5. Threshold � against
neighborhood size with exponential fit.
The thresholds were averaged over 25
values, the error bars represent the
standard deviation.
Fig. 6. Height map created with the
presented algorithm corresponding to the
height maps created with a fixed represents the threshold for points to be
neighborhood size shown in Fig. 3. considered.
tions. Thus, focus information is also
available only at discrete positions and
with it surface information. Hence, the
choice of the interslice distance limits
the maximal achievable axial resolution.
An interpolation of the discrete focus
Fig. 7. Section of the texture extracted
values would provide continuous
from a holographically recorded face
information and consequently a
digitized with a resolution of 40 _m (a).
potentially higher resolution. Nayar [2]
A screenshot of the 3-D computer model
proposed an interpolation method
of the marked section is shown in (b).
based on a Gaussian distribution (3)
Another example can be seen in Fig. 7. It
By choosing the maximal focus value
shows a section of the texture image
and its two neighbors, the unknown
extracted form a holographic recording,
parameters and can be calculated just by
digitized with a resolution of 40 m. The
solving a system of algebraic equations.
adaptive approach presented in this
With this, continuous values are
paper permitted the creation of a closed
produced that are not restricted anymore
surface with a local resolution so high
to thediscreteslicepositions.An
that single hairs are visible in the 3-D
alternative approach is to fit the
model.
Gaussian model mentioned in (3) to a
SURFACE INTERPOLATION
particular chosen number of points. For
The set of 2-D images, from which the
the selection of points used for the fit it
surface information is extracted,
was found best to first determine the
represents the holographical image at
height of the maximum above the
discrete position.
average focus values and consider only
points in the upper half of that interval
(dashedline in Fig. 8). Additionally, the
Fig. 8. Exemplary focus profile with
points have to lie in an interval of 20
fitted Gaussian curve. The dashed line
tomograms before and behind the
maximal value, to eliminate noisy data
points coincidentally lying above the
threshold but far away from the
maximum. The fitting algorithm was
implemented using a downhill simplex
method in multidimensions[27, p. 408]
proposed by Nelder and Mead [28]. Fig.
8 shows the whole profile with the fitted
curve. The center of the Gaussian serves
as new coordinate.The interpolation
method is based upon three focus values
exactly met by the curve, while the fit
method takes up to twenty points into
account and finds thecurve by
minimizing the overalldeviation. That
makes the fit more accurate and more
stable in the presence of noise. However,
its computational effort is much higher
since an iterative optimization step is
involved.In holographic facial
measurement, where computational time
is not as important as in real time
applications, the Gaussian fit proved
to be a useful tool for surface
refinement. A quantitative comparison
of the interpolation methods mentioned
above and an additional one can be
found in [7].Additionally to the
increased robustness against noise, the
fit also eliminates step like structures in
the extracted surface,which especially
appear if using large neighborhood sizes.
An example can be seen in the height
profile in Fig. 9. The steps in the height
map are much larger than the interslice
distance.These steps are lessened
considerably through the Gaussian fit, as
can be seen in Fig. 9. The height profile
gained with the fit is shifted for better
comparability.
SURFACE POSTPROCESSING
The previous sections presented new
methods invented for the improvement
of the classical depth-from-focus
method.However, despite sophisticated
surface extraction methods,little noisy
surface fluctuation cannot be avoided
entirely.Often denoising is done with
low-pass filtering, which reduces
noise, but also blurs sharp features and
details. We found that a smoothing
method proposed by Tasdizen [29] based
on anisotropic diffusion of normals
worked perfectly for the postprocessing
of facial models.Anisotropic diffusion of
normals is the natural expansion of the
anisotropic diffusion method for 2-D
images introduced by Perona and Malik
[30] to curved surfaces in the 3-D space.
Smoothing in regions of high total
curvature is suppressed through the
introduction of a nonconstant diffusion
coefficient depending on the total This paper demonstrated that
curvature. the classical shape-from-focus
method is perfectly suited for
the extraction of 3-D geometry
and texture information from
holographic reconstructions,
especially
Fig. 10 shows a facial model gained with for the creation of highly
the presented adaptive algorithm and resolved facial models.We
Gaussian fitting without any evaluated several mathematical
postprocessing (a), with isotropic operators for contrast
smoothing (b) and with anisotropic measurement and found that
diffusion of normals.5 While small the best criterion for this
features are almost obliterated in (b) they application is the newly
are nearly as pronounced in (c) as in the
developed XSML criterion, a
raw data.The generation of continuous
modification of SML operator
surface data through the Gaussian fit or
proposed by Nayar [2].
Gaussian interpolation is obligatory for
Additionally, we presented an
the appliance of the anisotropic diffusion
algorithm for the automated
of normals. For discrete surfaces, as they
adaption of the neighborhood
originate from classic shape-from-focus,
size to the conditions of the
the anisotropic smoothing does not
data.
behave stable.With all improvements
presented in this paper applied, the
computation time on a standard PC for
facial models as shown in this paper
increases from 1 min for the standard
procedure to approximately 30 min,
whereas the Gaussian fitting requires the
most of the additional time

Conclusions
The neighborhood size is chosen
as small as possible while
preventing
corruption through noise. With
this, an optimal lateral
resolution can be achieved. It
was demonstrated that in facial
measurement single hairs can
be visualized through this
procedure. It was also shown that a Fig. 11. Schematic top view, side view,
Gaussian fit to the focus profile and photograph of the holographic
improves the quality of the surface camera for portrait holography.
tremendously. Artifacts are
eliminated and the influence of noise is REFERENCES
reduced, while at the same time a
continuous surface not restricted to the 1.IEEE journals and magazines&website
digitization positions is created. This is 2.google search engine
compulsory for the appliance of
anisotropic smoothing methods, which
were also presented in this paper.
The holographic method for facial
measurement is superior to conventional
measurement techniques since it
captures 3-D information of the face
with one single laser pulse of 35-ns
duration.In that extremely short amount
of time, no movement artifacts can occur
and highly resolved facial models can be
created for various applications.