Sunteți pe pagina 1din 14

ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

Contents lists available at ScienceDirect

ISPRS Journal of Photogrammetry and Remote Sensing


journal homepage: www.elsevier.com/locate/isprsjprs

Aerial multi-camera systems: Accuracy and block triangulation issues


Ewelina Rupnik ⇑, Francesco Nex, Isabella Toschi, Fabio Remondino
3D Optical Metrology Unit (3DOM), Bruno Kessler Foundation (FBK), Trento, Italy

a r t i c l e i n f o a b s t r a c t

Article history: Oblique photography has reached its maturity and has now been adopted for several applications. The
Received 28 August 2014 number and variety of multi-camera oblique platforms available on the market is continuously growing.
Received in revised form 22 December 2014 So far, few attempts have been made to study the influence of the additional cameras on the behaviour of
Accepted 23 December 2014
the image block and comprehensive revisions to existing flight patterns are yet to be formulated. This
Available online 16 January 2015
paper looks into the precision and accuracy of 3D points triangulated from diverse multi-camera oblique
platforms. Its coverage is divided into simulated and real case studies. Within the simulations, different
Keywords:
imaging platform parameters and flight patterns are varied, reflecting both current market offerings and
Oblique images
Simulation
common flight practices. Attention is paid to the aspect of completeness in terms of dense matching algo-
Flight planning rithms and 3D city modelling – the most promising application of such systems. The experimental part
Orientation demonstrates the behaviour of two oblique imaging platforms in real-world conditions. A number of
Bundle adjustment Ground Control Point (GCP) configurations are adopted in order to point out the sensitivity of tested
Accuracy imaging networks and arising block deformations. To stress the contribution of slanted views, all scenar-
Automation ios are compared against a scenario in which exclusively nadir images are used for evaluation.
Ó 2014 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). Published by Elsevier
B.V. All rights reserved.

1. Introduction Thanks to the rapid progress in military photography from air-


planes, kites were then abandoned in favour of powered flight,
Oblique aerial images, alone or in combination with vertical which gained importance for military reconnaissance during
aerial images, have always been seen as a very attractive data World War I. Simultaneously, single and multiple lens cameras
source thanks to their intuitive nature. Documented use of oblique were produced for oblique-only or combined configurations in
imagery dates back to the nineteenth century. Gaspard Félix Tour- the UK, Germany, France, Italy, Switzerland and USA (Manual of
nachon acquired the first aerial photograph in 1858 by raising a Photogrammetry, 1952). In the 1930s for instance, the U.S. Geolog-
balloon over the village of Petit Bicêtre in France (Colwell, 1997). ical Survey and the U.S. Army used a Fairchild T-3A five-lens film
Two years later, the oldest surviving oblique aerial image was camera for mapping, surveillance and reconnaissance purposes.
taken by J.W. Black and S. King in Boston, USA, from a balloon. In Finally, during World War II, oblique aerial photography was
1887 a balloon again was employed by a German forester to employed for reconnaissance purposes before and after bombing
acquire aerial photos over forests in order to identify and measure missions (Nocerino et al., 2012), thus further stimulating the rapid
stands of trees (Colwell, 1997). The same year also saw the birth of development of new cameras, lenses, films and camera mounting
aerial photography from a kite, thanks to a British meteorologist, systems (Aber et al., 2010).
E.D. Archibald (Colwell, 1997). At about the same time in France, The concept of fitting several imaging sensors into a unique
the Tissandier brothers conducted other experiments of kite and camera housing re-emerged for civil applications with the intro-
balloon aerial photography. Thereafter this practice of acquiring duction of digital imaging technology (Petrie, 2010). However,
images from kites moved across the Atlantic and advanced rapidly. the true revival of oblique imaging systems for geospatial applica-
In 1906 G.R. Lawrence, using between nine and seventeen large tions occurred in 2000 when Pictometry International introduced
kites to lift a huge camera, took some oblique aerial photographs their five-lens camera system that incorporated vertical and slant
of San Francisco (USA) after an enormous earthquake in the area. views (Patent Ser. No. 60/425,275, filed November 8, 2002). Pic-
tometry imagery has also, more recently, been disseminated
⇑ Corresponding author. through the Bird’s Eye view function of current Bing Maps. Over
E-mail addresses: rupnik@fbk.eu (E. Rupnik), franex@fbk.eu (F. Nex), toschi@fbk.eu
the past decade, the geospatial industry has enthusiastically taken
(I. Toschi), remondino@fbk.eu (F. Remondino). up oblique imaging technology. New businesses have been born

http://dx.doi.org/10.1016/j.isprsjprs.2014.12.020
0924-2716/Ó 2014 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). Published by Elsevier B.V. All rights reserved.
234 E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

(e.g. Icaros, Simplex Mapping Ltd., VisionMap) and a growing num- results, what has been largely overlooked until now is any revision
ber of existing companies (e.g. IGI, Leica/Hexagon, Track’Air/ of traditionally adopted flight patterns. Open questions include: (i)
MIDAS, Vexcel/Microsoft, BSF Swissphoto) now include oblique How do different overlap scenarios influence the precision of com-
camera systems in their portfolios. As competition has grown, puted object points? (ii) What is the right balance between aerial
the market has evolved and expanded its offerings. To date, most survey precision and productivity? (iii) How should a flight be
prominent cities in the world have been covered with georefer- planned in terms of overlap and camera configuration parameters
enced oblique images and these datasets are usually updated every (e.g. tilt) to achieve most complete results? (iv) What, after all, is
few years to ensure that they are current. This also gives rise to the influence of the oblique views on the image block triangulation
valuable multi-temporal databases of image data (Karbo and from the point of view of obtainable accuracy?
Simmons, 2007). In the following sections of the paper, the authors address the
Today, the oblique camera systems are available in a variety of above problems related to oblique multi-camera systems in terms
configurations, the differences between designs being mainly in of image triangulation (self-calibration is not the primary scope of
the imaging sensors (number, arrangement, format and sensitivity) the paper). Section 2 reports on existing flight planning routines
and the mode of acquisition, which include: and simulation studies of various block configurations. The influ-
ences of varying sensor size, tilt angle and overlap (in nadir and
– maltese-cross systems comprising five different imaging sensors, oblique cameras) are considered regarding the obtainable accu-
one pointing vertically and the other four pointing symmetri- racy, with attention being given to the issue of completeness of
cally in the four cardinal directions. Camera formats range from point cloud retrieval in urban environments. Section 3 briefly
small to large depending upon the intended applications, and describes the image orientation task related to oblique imagery.
the tilt angles normally lie between 30° and 45°. These oblique Lastly, Section 4 reports the results of two real-case studies. The
camera systems are suitable for large areas mapping of 3D city behaviour of image blocks is evaluated for different Ground Con-
modelling. trol Point (GCP) configurations, different image overlaps, and GNSS
– fan systems, which mainly come as twin cameras, with the sen- assisted image triangulation.
sors positioned linearly along- or across- track, making them
very suitable for corridor mapping. A quite innovative fan solu-
tion is offered by VisionMap with the A3 Edge sensor, a stepping 2. Flight planning – simulated case studies
frame system that captures up to 64 images per sweep, corre-
sponding to a field of view of 109°. Flight planning refers to the initial determination of flight
geometry, given the area of interest and the required end-product,
Most of today’s oblique camera systems acquire RGB images, and thus the desired accuracy. In the case of traditional nadir imag-
though acquisitions in the NIR band are becoming standard in ery, the parameters that are optimized are the flying height, the
the newer commercial systems. camera with its optics and the overlap pattern. Depending upon
On the application side, the actual interest in oblique photogra- the type of photogrammetric application, adopted lenses can vary
phy for mapping purposes is due to its primary quality: the imag- from wide to narrow angles. As a general rule, long focal lengths
ing of entire building façades and, normally, their footprints as are used when large height variations are present (cities, moun-
well. Possible applications based on oblique aerial views are multi- tainous areas), whereas short focal lengths are preferred in rural
ple: dense point cloud extraction for 3D city modelling (Wang areas, for overview flights or when better height accuracy is tar-
et al., 2008; Fritsch et al., 2012), monitoring situations during mass geted. Extensive studies on optimization of flight pattern scenarios
events and environmental accidents (Kurz et al., 2007; were carried out during the second half of the 20th century
Grenzdörffer et al., 2008; Petrie, 2008), building detection and (Förstner, 1985; Ackermann, 1992; Kraus, 1997, 2007). At that
reconstruction (Nex et al., 2013), urban area classification (Gerke time, digital photogrammetry was still an unrealised concept and
and Xiao, 2013), identification of structural damage to buildings manual stereoscopic observations were commonplace. Hence, the
(Nyaruhuma et al., 2012), updating of road databases (Mishra attention remained around vertical imaging while combined verti-
et al., 2008) and administration services (Lemmens et al., 2008). cal and oblique photography was conceived merely for pictorial
From a practical point of view, the extracted geometric information representation. The prevailing rule-of-thumb suggested that flying
should be complete, reliable, generated with minimal interaction, scenarios with 60% forward overlap would ensure a 3-fold cover-
all while keeping the survey cost at the optimal level. However, age for points down each side of the image, providing for strip for-
although the abundance and practical utility of this oblique image mation. Furthermore this also afforded good stereoscopic viewing
data is clear, both dealing with the resulting very large datasets with a good base length for 3D measurement. At the same time,
and automated processing remain a challenge. This is primarily the 20–40% side overlap provided a sufficiently strong geometry
because complex image configurations are imposed by the oblique to tie the neighbouring strips of photography together. Reduced
image acquisitions and because of shortcomings in on-board direct processing times and compilation costs were a top priority; there-
orientation sensors that do not satisfy the strict requirements of fore the fewer images covering the area of interest the better the
geomatics applications. economy of a project.
The prospect of information extraction from oblique imagery The costs of mapping with photogrammetry started to signifi-
has also given impetus to scientific initiatives (e.g. EuroSDR Bench- cantly reduce as the technology matured and the digital era was
mark on Image Matching; ISPRS Scientific Initiatives) aimed at embarked upon. This brought about automated image measure-
evaluating the current status of available tools and at boosting fur- ment, which raised the compilation efficiency and reduced human
ther research into automated orientation (Wiedemann and More, operator interaction. Automation gave rise to a new tendency to fly
2012; Rupnik et al., 2013), dense image matching (Gerke, 2009; denser patterns, in particular in the in-flight direction. Also, think-
Rupnik et al., 2014), object detection and 3D reconstruction (Xiao ing in terms of single stereo models gave way to thinking in terms
et al., 2012). These efforts target developers and users in delivering of multi-image bundles of rays (McGlone, 2004; Aber et al., 2010).
comparative performance analyses on the provided datasets, in Within this context of ‘‘digital evolution’’, oblique multi-camera
particular for dense matching and 3D reconstruction (Cavegn systems started again to be developed and employed for different
et al., 2014). However, in spite of new data processing develop- application fields and market niches. They began to be thought
ments and comprehensive evaluation of block triangulation as instruments for cartographic mapping, rather than only for
E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246 235

visualization purposes or reconnaissance, as was the case in the reflect the situation where observations from all cameras are
past. The imaging geometry of oblique camera systems is analo- accepted, while in the constrained scenarios only observations in
gous to close-range applications, with all their characteristics, the nadir cameras and oblique cameras of same look direction
e.g. greatly varying scale within the images, illumination changes, are considered, as discussed in Section 3. In order to account for
multiple viewing directions and wide baselines and hence large possible occlusions that influence point observability, normally
perspective differences between the views. To fully exploit the prevailing in urban environments, a complex of buildings
oblique cameras on-board, be it for texture mapping or image (Fig. 4a, street width of 5 m and 10 m, building height 20 m) was
dense matching, the flight planning must thus more cautiously embedded in the simulated scene. Every point back-projected to
adapt to the situation on the ground, and the overlaps must be sep- a camera is verified whether obstructing with the scene or not. Fur-
arately calculated for nadir, backward/forward, and side cameras. ther constraint is imposed on the oblique cameras, specifying that
Apart from the flying height and the camera optics, the effective object points can be observed only from cameras that look the
overlaps are conditioned on the camera tilt angles. As will be same direction (see also Section 3). This tolerance is full-fledged
shown later, only a just combination of all the parameters will pro- given the large perspective distortion between slanted views that
duce images that can serve for precise, and complete 3D scene hamper the matching of homologous points (Rupnik et al., 2013).
reconstruction (Fritsch and Rothermel, 2013; Rupnik et al., 2013).
In the following paragraphs, the authors evaluate the quality of 2.2. Precision, productivity, completeness
3D point triangulation under several imaging configurations. Sim-
ilar investigations using multi-view nadir and stereo oblique 2.2.1. Precision
geometry have been presented in Förstner (1998) and Gerke According to the data contained in Table 1, presented case stud-
(2009), respectively. ies involve the maltese-cross system with small (A), medium (B)
and large format (C) cameras. Results presented in Fig. 3 show
2.1. Simulation procedure the relationship between object point precision (computed from
the covariance matrix) and factors such as overlap, camera tilt
The purpose of simulation is to provide a priori variances of tri- angle, and imaging ray redundancy. Due to the fact that the scale
angulated points from the inversion of the normal equation matrix of the nadir images is kept constant for all case studies, the figures
of the bundle adjustment, without performing a real measurement. of precision do not vary significantly between the cases.
It is a way of optimizing object point accuracy when minimum Within the different imaging configurations, the overlap and tilt
information about the scene is available, a process known within angle of the oblique cameras, and the imposed constraint, largely
close-range photogrammetry as network design. The design prob- influence the results. It can be noted (Fig. 3) that the increased tilt
lem (Fraser, 1984, 1996; Mason, 1995) typically divides into Zero- angle contributes to:
order (the datum definition problem), First-order (the network
configuration problem), Second-order (the weight problem) and – a larger point redundancy (observations),
Third-order (the densification problem) design. In the following – improved precision, i.e. a better standard deviation of 3D points,
we consider only First-order design, i.e. the optimization of obser- – a homogenous standard deviation in all three dimensions.
vation configurations by (i) simulating a flight, (ii) generating
image observations by back-projecting 3D points to all cameras/ However, imposition of the look direction constraint causes:
images and (iii) performing a bundle adjustment to retrieve the
standard deviations from the diagonal elements of the symmetric – degradation in precision along the Z axis through a decrease in
covariance matrix. Object points, camera positions and orienta- both the number of observations and the intersection angles,
tions are regarded as free parameters. The estimated precision of – inhomogeneous planimetric precision (especially noticeable for
object points is based on the following standard errors – image large tilt angles) due to the different intersecting angles in the
coordinate 1 pixel, camera position 0.1 m, camera orientation XZ and YZ planes.
0.001°. Collinearity equations constitute the basis for the 3D
restitution. Compared with the classical scenario of exclusively nadir
The simulated scenarios are imaged with a maltese-cross system images (Fig. 3), it can be observed that while the planimetric pre-
and vary according to the following parameters (Table 1 and cision is only slightly improved, the standard deviation in the Z
Fig. 1): sensor size, forward and side overlap, camera focal length direction (height) is 2–3 or 4–5 times better, depending whether
and field of view. The flying height is adapted in order to keep the look direction constraint is enabled.
the Ground Sampling Distance (GSD) computed in the nadir
images constant. The choice and values of particular parameters 2.2.2. Productivity
reflect the current market offerings and the most common flight Increasing overlap between subsequent acquisitions and strips
practices. To distinguish between the sensors, they are referred is costly in the image collection time and post-processing but
to as small, medium and large although the size of the latter would causes substantial gain in precision, as proves Fig. 2. The figure dis-
normally be classified as medium. The unconstrained scenarios plays the precision differences between the overlap 60/40 and 80/

Table 1
Sensor specifications for the three simulated case studies, A, B, and C, which correspond, respectively, to small, medium and large format cameras of maltese-cross design. Field of
view (FOV) of oblique cameras in along direction refers to forward/backward looking views while the across-track direction refers to side-looking views. 30° and 45° oblique
angles (indicated as ‘‘tilt’’ in the table) are tested. The tilt refers to the viewing along cameras’ line-of-sight. The flying heights selected for the case studies are: 1200 m (A), 1000 m
(B) and 1600 m (C).

Case A Case B Case C


Nadir Oblique Nadir Oblique Nadir Oblique
Sensor size (mm/pix) 36  24/7360  4912 49  36/8176  6132 70  45/11,674  7514 53  40/8900  6650
Focal length (mm) 50 80 50 80 80 120
FOV (°) along/across 26/39 tilt ± 8.5/8.5 39/52 tilt ± 12.5/12.5 31/47 tilt ± 9.5/(+9.5–15.7)
236 E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

Fig. 1. Schematic representation of the simulated camera systems in cases A, B and C for 45° tilt angle (a–c). Overlapped FOVs for the different systems (d). Top row
represents the along-track direction, the bottom row is the across-track configuration. The angles reported for every camera system show the maximum achievable incidence
angle.

Fig. 2. Differences in precision counted in percent for scenarios A, B and C in different tilt angle configuration (a–c). The plain line refers tothe comparison of 60/40 and 80/60
flight pattern. The crossed line refers to the comparison of 80/60 and 80/80 flight pattern. The percentage is computed as a ratio e.g. r60=40 x  rx80=60 =r60=40
x . Number of
acquisitions necessary to cover a 4 km  5 km area of interest in various overlap scenarios, with the camera systems A, B, C (d).

60 (plain line) as well as between 80/60 and 80/80 (crossed line). to the figures, they should be ‘‘weighted’’ by the additional number
Throughout all the scenarios, flying with a 80/60 overlap pattern, of acquisitions needed to densify the data collection, especially
rather than 60/40, delivers standard deviations better by ca. 50% that the number grows exponentially (Fig. 2d). Confronting the
along X, Y, Z directions. On the other hand, when comparing the two, the rationality suggests that the 80% along- and 60% across-
standard deviations between 80/60 and 80/80 overlap, a precision track patterns would be the golden mean as the return in precision
increase of only about 20–30% is observed. To give more meaning for the cost of extra labour is substantial. Having said this, it must
E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246 237

be underlined that dense matching in oblique datasets is feasible

and camera tilt angle. The numbers within the diagram boxes correspond to point redundancy according to the different overlap. See Section 3 for explanation of constrained and unconstrained cases. The flight trajectory is along
Fig. 3. For the three case studies A, B, C (small, medium and large format, respectively), standard deviations (m) of computed object coordinates in X (red line), Y (green line), Z (blue line) are reported with respect to image overlap
from images of the same perspective. This is analogous to the
automatic tie point extraction step where SIFT-like features are
matched across views of similar look direction (see Section 3).
It implies that both across- and along-track overlaps are equiva-
45° Constrained

lent and favourably equal to 80%.


With the GSD size equal for all cases, the variable sensor size
affects only the survey productivity. In Fig. 3 the redundancy fig-
ures of the C camera system are in the majority of cases 20–30%
smaller than in A and B. Notably, the measures of precisions do
not drop accordingly (see also Fig. 2).

2.2.3. Completeness
Besides the precision and productivity aspects, sensor size and
tilt angle are also important from the standpoint of the complete-
ness of the reconstructed scene. Increasing the value of both
parameters enlarges the camera’s field of view (FOV) and pro-
vides for a bigger GSD, respectively (assuming fixed focal length).
But, larger FOVs contribute to more severe occlusions – an unde-
45°

sirable artefact in image-based 3D city modelling (Haala and


Kada, 2010). Because of this, as a part of the project planning,
attention should be paid to careful adaptation of the camera plat-
form parameters to the situation in the field, i.e. to the topogra-
phy of the city to be flown over.
Fig. 4b–e exemplifies the effect of the tilt angle on the visibility
of two selected façades. Both façades are oriented parallel to the
flight direction, and the width of the street in front of them are
the Y axis. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

10 m (building E10) and 5 m (building E5). The simulated example


30 ° Constrained

was derived from the flight with 80/60 overlap, surveyed with the
camera system of type A. Due to the occlusion effects, the tilt
angle of 45° introduces inhomogeneous distribution of redun-
dancy along the façades. The 30° tilt angle results in fewer but
uniform redundancies. The width of the street and the building
height play a major role in planning a successful urban survey
campaign. The taller the city architecture the lower the camera
incidence angle should be. Therefore, a compromise ought to be
found between the camera tilt setting, given focal length, sensor
size, overlap and the geometry of the surveyed area.

3. Image block orientation

The processing workflow for oblique imagery can be divided in


30°

sequential steps (Rupnik et al., 2014), namely image orientation,


dense matching and point cloud filtering, feature extraction,
orthoimage generation and monoplotting (Murtiyoso et al.,
2014). While this follows the traditional photogrammetric pro-
cessing pipeline, the algorithms for the orientation and matching
phases need to be slightly adjusted in order to cope with oblique
image datasets. The orientation task is recapped below. Dense
matching and feature extraction issues are discussed in Nex
et al. (2013).
In the image orientation process, the interior and exterior
parameters are nowadays often known a priori, as retrieved with
a prior calibration procedure or measured directly with on-board
Nadir

sensors (GNSS/IMU), respectively. Nonetheless, these parameters


are generally regarded as approximate if one has metric and auto-
matic applications in mind. An adjustment in a least squares
sense is therefore mandatory, requiring linearization of the equa-
tions and good approximate values of unknown parameters. On
the basis of image data (homologous points), these parameters
are (i) retrieved with direct methods (e.g. resection, perspec-
tive-n-point problem) individually for each image, (ii) concate-
A

C
B

nated to a common reference system with the help of the


fundamental/essential matrix or trifocal tensor, and eventually
(iii) refined inside the bundle adjustment. The homologous points
238 E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

E10 , 45° E10, 30° E5, 45° E5, 30°

(a) (b) (c) (d) (e)


Fig. 4. The simulated complex of buildings (20 m high) (a). Given the two building façades (E5, E10), figures (b–e) show a color coded map with the number of images where a
point on the façade is observed. E10 (10 m width street) and E5 (5 m width street) are observed with the A camera system in 80% along-, 60% across-track overlap. Oblique
cameras are tilted by 30° (c and e) and 45° (b and d). The flight direction is along the Y (green) axis. (For interpretation of the references to color in this figure legend, the
reader is referred to the web version of this article.)

Fig. 5. Camera network of the Milan dataset (left) and five different configurations of the ground control information. The Y axis is parallel to the flight direction (Y runs
vertically).

Fig. 6. Camera network of the Graz dataset (left) and four different configurations of the ground control information. The Y axis is orthogonal to the flight direction (Y runs
horizontally).

are found automatically across multiple aerial views with standard Pierrot-Deseilligny and Clery, 2011) and the generation of putative
area-based image matching techniques (such as Normalized Cross correspondences between images is therefore prone to a greater
Correlation and Least Squares Matching) or feature-based methods number of mismatches. Indeed, the literature reports that tradi-
(SIFT, SURF, etc.) (Apollonio et al., 2014) coupled with robust esti- tional photogrammetric software is optimized for nadir image
mators to remove possible wrong correspondences (RANSAC, Least acquisition and generally performs poorly for other imaging geom-
Median of the Squares, etc.). etries (Jacobsen, 2008; Gerke and Nyaruhuma, 2009).
Oblique imagery is richer in content compared to nadir imag- In the developed processing pipeline, the computation of
ery. Objects (building, roads, etc.) are recorded at different scales, approximate orientation parameters is guided with the help of a
and object features are back-project under different projective connectivity graph (Rupnik et al., 2013). The connectivity between
transformations. Also, due to different viewing directions grave images refers to a graph with nodes and edges being representa-
occlusions are exhibited. Such configurations, are characteristic tions of images and their relationships. Two images are linked with
of unordered terrestrial image networks (Barazzetti et al., 2010; an edge if and only if they are spatially compatible. GNSS/IMU data
E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246 239

Table 2
Dataset specifications (N and O stand for nadir and oblique). The given overlap is calculated on the nadir images.

Data Camera Sensor size (mm) Focal N/O FOV along/ Image GSD N # images Y/X Area
(mm) across N (°) scale (m) N/O overlap (%) (km)
Milan Midas 5 36  24 80 17/25 15,000 0.10 125/375 70/30 35
100
Graz Vexcel Osprey I 70  45 (nadir); 23.5  36 (left/right) 71.5  23.5 51 48/70 21,000 0.12 20/160 75/65 3  1.5
(for-/backward)
80

Table 3
A summary of all performed tests. ID refers to a name given to a particular test. The columns indicate the adjustment parameters used as constants, unknowns, observed
unknowns, and observations. FX means fixed IOR, FR refers to self-calibrating bundle adjustment (i.e. free IOR).

ID Input images Constants (C) unknowns (U) Exterior orientation Interior orientation GCPs, observed in GCPs, observed in Tie
observations (O) EORa IOR + AP nadir oblique pointsb
1FX Nadir C U
U U U U
O U U U
2FX Nadir + oblique C U
U U U U
O U U U
3FX Nadir + oblique C U
U U U U U
O U U U U
1FR Nadir C
U U U U U
O U U U
2FR Nadir + oblique C
U U U U U
O U U U
3FR Nadir + oblique C
U U U U U U
O U U U U
a
GNSS observations are used inside the bundle block adjustment as observed unknowns only in the GNSS-assisted scenario.
b
Tie points in image space treated as observed and in object space as unknown.

MIDAS 5 UltraCam Osprey I


Forward and backward oblique views Forward and backward oblique views

along-track overlap across-track overlap along-track overlap across-track overlap


Side oblique views Side oblique views

along-track overlap across-track overlap along-track overlap across-track overlap

Fig. 7. Overlaps in forward/backward and side oblique cameras for Midas 5 (left) and UltraCam Osprey I (right) systems corresponding to 70/30 and 75/65 overlaps in nadir
views. The overlap in the oblique views is given as a ratio of the complete length of the median of a trapezoid (forward/backward camera in cross-track overlap and side
camera in along-track overlap) or the axis of symmetry (forward/backward camera in along-track overlap and side camera in cross-track overlap), to its length between
subsequent acquisitions (red line). It is computed in along- (horizontal red line) and cross-track (vertical red line). (For interpretation of the references to color in this figure
legend, the reader is referred to the web version of this article.)

beneficially provides the input information for establishing the reducing the number of possible outliers. There are three condi-
image associations. Traversing across the connectivity graph helps tions (constraints) to be fulfilled for an image pair to be regarded
in speeding up the extraction of image correspondences and in as compatible:
240 E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

Table 4 4.2. Tests


The computed a posteriori standard error of unit weight r0 (pixel), for the different
configurations of ground control information reported in Figs. 5 and 6. Note that the
sigma values are the same for both, with and w/o the GNSS support.
Two pieces of control information (GCPs and GNSS observations)
were incorporated in the computations, and regarded as soft con-
Data GCPs config. 1FX 2FX 3FX 1FR 2FR 3FR straints, i.e. treated as unknowns and observations (so-called
Milan 0–4 0.49 0.47 0.55 0.47 0.45 0.52 observed unknowns), therefore assigned corrections within the
Graz 1–4 0.38 0.36 0.39 0.31 0.29 0.31 bundle adjustment. The blocks were oriented using different con-
figurations of GCPs (yellow circles in Figs. 5 and 6) and independent
CPs (red markers in Figs. 5 and 6).1 For both datasets, configuration 1
shows a good GCP distribution along the borders of the block,
a Their ground footprints coincide by a given percent.
whereas configurations 2, 3 and 4 are examples of unfavourable con-
b Look directions of the images are similar or one is nadir-looking
trol point distributions. Although the latter may be seen as unrealistic
(similarity can be then violated).
cases, they are shown for the sake of completeness and to stress the
c The number of extracted homologous points for the pair is
role of ground control information in the bundle solution. Thanks to
above a given threshold.
the large number of points surveyed in the Milan test area, an addi-
tional configuration was added (configuration 0). It comprises a good
Given a graph complying with conditions a and b, feature
GCPs distribution across the whole area of interest. A summary of the
extraction is allowed between images with at least two edges.
performed tests is provided in Tabs. 3 and 5, where abbreviated
Next, the edges of the graph are enriched with another attribute,
descriptions of the different processing scenarios are listed as well.
i.e. the number of extracted tie points, and all images are concate-
The initial values of internal orientation parameters (IOR) and
nated. Concatenating images for the initial solution is understood
additional parameters (AP) were taken from a calibration certifi-
to (i) find triplets of images within the connectivity graph to be
cate provided by the company executing the flight. These input
used for computing approximate orientations, and (ii) providing
data can be either held fixed within the bundle adjustment process
all the triplets a sequence in the concatenation order. A bundle
or included as unknowns in a self-calibration procedure. In our
adjustment controlled in this way minimizes the risk of divergence
tests, the scene contains significant height differences – leading
by maximizing the similarity of images within particular triplets,
to variable scale across images, hence both possibilities were
hence maximizing the ratio of good to bad matches and ensuring
tested. A version of the Fraser calibration model (Fraser, 1997),
the block’s cohesion.
involving three coefficients of radial distortion and two coefficients
As a last step, the control information in form of GNSS/IMU data
of decentering distortion, was used in the experiments. While for
and/or GCPs is included in the adjustment computations. With this,
middle-format cameras this could not be sufficient and more
the data is transformed to the coordinate system of the control
sophisticated approaches are available (Cramer, 2009), the camera
information. The quality of the resulting orientation is strongly
calibration issue was not the primary scope of this article.
coupled to the spatial distribution of the control information, its
In the following, two measures of adjustment quality are
‘‘observability’’ in the images as well as the importance given in
reported, the sigma naught (r0) and the root mean square values
the adjustment. It should be remarked that if the surveyed area
(RMS), defined as the square root of the mean squared difference
surpasses a certain size, the Earth curvature effects must be
between nominal values and corresponding adjusted observations.
considered.
Moreover, to give a more global impression of the possible deforma-
tions within the image block, the resulting sparse point clouds are
4. Real case studies compared to a reference point cloud, which was chosen to be the
one exhibiting the optimal measures of accuracy against CPs. Net-
4.1. Datasets work 3FR in GCPs configurations 0 (Milan) and 1 (Graz), i.e. self-cal-
ibrating condition and good distribution of GCPs was selected. The
The first dataset (Milan) comprised 550 images and was imaging geometry corresponds to the constrained case of Section
acquired with a Midas 5 multi-camera system (one nadir and four 3, i.e. tie point observations were allowed from all nadir images
oblique looking cameras tilted at 45°). The image block covered an and those oblique images with the same viewing directions.
area of ca. 3 km by 5 km that was flown with 70/30 (Y/X) overlap The purpose of the performed tests is to investigate how the
related to nadir images (Fig. 5). In order to evaluate the quality accuracy of triangulated 3D points is affected by the following fac-
of the block orientation results, 25 GCPs were GNSS-surveyed with tors: (i) introduction of oblique imagery in the image block, (ii)
a mean positional accuracy less than 5 cm. self-calibration, (iii) spatial distribution of GCPs and (iv) GNSS sup-
The second dataset (Graz) comprised 180 images and was port. All tests were conducted using MicMac, an open-source soft-
acquired using an UltraCam Osprey I system (one nadir camera ware suite for bundle block adjustment and dense image matching
with six oblique-looking cameras tilted at 45° along the four car- (Pierrot-Deseilligny and Paparoditis, 2006; Pierrot-Deseilligny and
dinal directions – see Gruber and Wolfgang (2014)). The area Clery, 2011).
covered ca. 3 km by 1.5 km and was flown with an overlap of
75/65 (Y/X) related to nadir images (Fig. 6). In a similar manner
to the Milan case study, a metric evaluation of the achieved 4.3. Bundle adjustment results
results was performed using ground truth information in form
of GCPs and check points (CP) GNSS-surveyed to a mean accu- MicMac solves the orientation task within a bundle adjustment
racy of 5 cm. by means of weighted least mean squares. Classically, the imaging
In both datasets the approximate exterior orientation param- equation f (see Eq. (1)), i.e. the functional model of the adjustment,
eters (EOR) were available as GNSS/IMU observations. The is defined in terms of the following parameters: rotations R, per-
dataset specifications are reported in Table 2. The structure of spective centers C, camera interior parameters I, and object coordi-
the flights resemble the comprehensive block specification nates of the tie points P. The optimal values of the parameters are
of Honkavaara et al. (2006a), without crossing flight lines. Corre-
sponding overlaps computed for oblique images are presented in 1
For interpretation of color in Figs. 5 and 6, the reader is referred to the web
Fig. 7. version of this article.
E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246 241

Table 5
RMS errors computed at CPs. The rows represent 5 different GCPs configurations; the column labels are explained in Table 3. The values correspond to the geo-referencing and
block adjustment based on GCP information while the values in bracket correspond to the GNSS-assisted triangulation.

GCP RMS No. No. MILAN dataset No. No. GRAZ dataset
config. (m) GCPs CPs GCPs CPs
1FR 2FR 3FR 1FX 2FX 3FX 1FR 2FR 3FR 1FX 2FX 3FX
x 0.12 0.12 0.15 0.10 0.07 0.13
(0.16) (0.14) (0.15) (0.16) (0.13) (0.16)
0 y 7 12 0.12 0.10 0.15 0.18 0.28 0.30 – – – –
(0.18) (0.12) (0.14) (0.17) (0.13) (0.14)
z 0.42 0.30 0.13 0.45 0.55 0.20
(0.32) (0.30) (0.20) (0.34) (0.31) (0.20)
x 0.23 0.22 0.18 0.36 0.16 0.21 0.04 0.04 0.05 0.04 0.04 0.03
(0.26) (0.21) (0.16) (0.26) (0.20) (0.17) (0.04) (0.04) (0.05) (0.05) (0.06) (0.05)
1 y 4 15 0.22 0.14 0.16 0.27 0.48 0.45 4 3 0.03 0.06 0.04 0.04 0.06 0.05
(0.19) (0.13) (0.14) (0.18) (0.14) (0.15) (0.03) (0.04) (0.03) (0.03) (0.04) (0.04)
z 6.85 1.39 0.26 10.68 3.68 0.64 0.08 0.08 0.02 0.17 0.13 0.13
(0.76) (0.54) (0.22) (0.77) (0.56) (0.20) (0.04) (0.10) (0.04) (0.20) (0.18) (0.16)
x 0.55 1.01 0.33 1.07 1.28 0.42 0.03 0.05 0.06 0.03 0.04 0.06
(0.26) (0.21) (0.22) (0.31) (0.20) (0.24) (0.03) (0.05) (0.05) (0.03) (0.09) (0.05)
2 y 3 16 0.29 0.35 0.21 0.68 1.81 0.92 3 4 0.03 0.03 0.03 0.04 0.03 0.04
(0.23) (0.15) (0.17) (0.30) (0.15) (0.19) (0.03) (0.03) (0.03) (0.03) (0.05) (0.07)
z 12.22 11.92 0.80 19.75 15.40 1.63 0.34 0.27 0.21 0.49 0.61 0.62
(0.77) (0.56) (0.29) (2.47) (0.58) (0.27) (0.08) (0.09) (0.04) (0.33) (0.34) (0.41)
x 0.31 0.80 0.75 1.79 3.29 3.33 0.04 0.14 0.13 0.11 1.36 1.35
(0.12) (0.13) (0.14) (0.17) (0.10) (0.13) (0.04) (0.14) (0.13) (0.11) (0.53) (0.52)
3 y 3 16 0.78 2.14 1.81 4.46 7.28 6.84 3 4 0.09 0.11 0.05 0.07 0.49 0.49
(0.38) (0.35) (0.35) (0.80) (0.35) (0.39) (0.09) (0.05) (0.05) (0.08) (0.19) (0.20)
z 6.67 2.29 1.77 11.97 3.81 2.10 2.85 2.54 2.76 4.14 2.75 2.90
(0.36) (0.34) (0.24) (0.79) (0.40) (0.54) (0.09) (0.24) (0.17) (0.24) (0.14) (0.26)
x 0.28 0.20 0.19 0.63 1.16 1.46 0.05 0.04 0.04 0.04 0.05 0.04
(0.28) (0.24) (0.23) (0.27) (0.22) (0.25) (0.04) (0.06) (0.06) (0.06) (0.15) (0.13)
4 y 3 16 0.21 0.23 0.28 0.72 2.02 2.50 4 3 0.05 0.05 0.06 0.06 0.05 0.06
(0.19) (0.09) (0.13) (0.18) (0.10) (0.11) (0.04) (0.04) (0.03) (0.03) (0.03) (0.06)
z 4.30 2.24 1.59 7.62 4.75 2.30 1.07 2.47 2.86 2.22 2.86 3.32
(0.72) (0.52) (0.22) (0.70) (0.52) (0.24) (0.07) (0.13) (0.18) (0.21) (0.28) (0.42)

found by minimizing an objective function (Eq. (3)) which is the Here, the hat variables (e.g. ^l) indicate the estimates whereas the
sum of squared discrepancies multiplied by a weighting function bar variables (e.g. ^l) indicate the original observations. Also, r is
q(v), see Eq. (4). The weight function is composed of three terms: the number of degrees of freedom (redundancy), Nobs the number
1
r 2 that weights observations given their a priori accuracy,
of observations within a given group of observations (e.g. GCPs)
obs
Nmax and Nmax a threshold that limits the influence of a group of observa-
Nmax þN obs
that limits the influence of a particular group of observa-
tion below the specified threshold value.
tions on the solution, and pffiffiffiffiffiffiffiffiffiffi
1
v 2
that ‘‘penalizes’’ observations In our evaluations, the tie points are measured with the SIFT
1þðlÞ
operator and the a priori image measurement accuracy (robs) is
with large residuals making it a robust estimation. The normalizing
set to 0.5 pixel. The rejection parameter B is set to be ten times
factor l is dataset-dependent and should be set according to the
estimated observation accuracy. As a general rule, its value will
robs. Estimated accuracies of the remaining observations are cho-
sen in the range of 0.05–0.1 m for GCPs and 0.5–1.0 m for GNSS
change from iteration to iteration. The normalizing factor should
observations. The Nmax parameters are chosen to balance the influ-
be high when unknowns are poorly approximated and recede
ence of all groups of observations within the adjustment, and to
along the iterations. If a residual exceeds the user-defined B
avoid the situation where only the abundant tie points or GNSS
parameter, the weight takes on the value zero, hence eliminating
data have a dominant influence upon the solution. The quality
the influence of that particular observation in the adjustment. An
and quantity of all groups of observations must be taken into
in-depth explanation of the adopted weighting scheme can be
account when assigning values to these parameters (e.g. 550 GNSS
found in (Pierrot-Deseilligny, 2014).
observations at 1.0 m and 7 GCPs at 0.1 m). In our tests, the Nmax
^l ¼ f ð R; b bI; PÞ
b C; b ¼ l þ v ð1Þ were assigned the value of 50 for the group of tie points and GNSS
data. The GCPs were not weighted due to its scarcity compared
V ¼ l  f ð R; b bI; PÞ
b C; b ð2Þ with other groups of observations. In other words, the second term
of Eq. (4) is equal to 1 for each GCP observation.
X
qðv Þv 2 ! minimum ð3Þ Table 4 reports the a posteriori standard error of unit weight of
the bundle adjustment (r0 in Eq. (5)) computed in the aforemen-
tioned tests. Introducing oblique images into the network, but
1 Nmax 1
qðv Þ ¼ if ðv > BÞ 0 else   qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð4Þ allowing the image measurements of GCPs only in nadir images
r2obs Nmax þ Nobs 1 þ ðlv Þ2 (2FX, 2FR), brings about a slight improvement in the re-projection
error. On the contrary, when GCPs are measured both in nadir and
P oblique views (3FR, 3FX), the re-projection error grows. It is an
qðv Þv 2
r20 ¼ ð5Þ expected outcome given that point identification and precise mea-
r
surement in slanted views are more difficult. Also, the ground sam-
242 E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

MILAN dataset
GCPs
conf. 1FR 2FR 3FR 1FX 2FX 3FX

. GRAZ dataset

Fig. 8. Discrepancy vectors at GCPs (red triangles) and CPs (open circles). Above: Milan dataset, GCP configurations 0 and 3, with GNSS. Below: Graz dataset, GCP
configurations 1 and 3, with GNSS. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

pling distance can be larger than that for the nadir looking camera. nario to scenario, r0 remains nearly constant and a plausible expla-
Furthermore, although freeing camera parameters can help to nation for this is that even if the GCPs introduce ‘‘strain’’ into the
compensate for systematic lens- and sensor-related errors, the block, resulting in locally larger residuals, the robust weight func-
improvement in terms of a reduced r0 is minor. Finally, from sce- tion (Eq. (4)) diminishes their effect on the sigma naught value.
E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246 243

1FX - 3FX 1FR - 3FR 1FX - 1FR 3FX - 3FR


MILAN dataset

no-GNSS

μ = 0.89 m μ = 0.53 m μ = 0.68 m μ = 0.23 m


σ = 0.76 m σ = 0.48 m σ = 0.46 m σ = 0.15 m

GNSS

μ = 0.35 m μ = 0.36 m μ = 0.07 m μ = 0.10 m


σ = 0.26 m σ = 0.28 σ =0.06 σ =0.09

GRAZ dataset

no-GNSS

μ = 0.08 m μ = 0.06 m μ = 0.18 m μ = 0.25 m


σ = 0.11 m σ = 0.06 m σ = 0.11 m σ = 0.21 m

GNSS

μ = 0.10 m μ = 0.06 m μ = 0.15 m μ = 0.22 m


σ = 0.08 m σ = 0.05 m σ = 0.09 m σ = 0.18 m

Fig. 9. l – arithmetic mean Euclidean distance, and r – standard deviation between sparse point clouds computed in GCP configurations 0 (Milan) and 1 (Graz). E.g. 1FX – 3FX
refers to the difference between sparse point clouds resulting from 1FX and 3FX scenarios. Red dots mark the GCPs, the black dots are the CP’s. Note that the legends should be
decoded separately for each dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

4.3.1. Influence of oblique views length of Midas 5 (see Table 2), and (ii) different overlap of the
It is evident from the RMS values in Table 5 that for the Milan two image blocks (75/65 versus 70/30 related to nadir cameras).
dataset, observations in oblique views improve object point accu-
racy in height (compare 1FR and 1FX with 2FR/3FR and 2FX/3FX), 4.3.2. Influence of self-calibration
which is consistent with expectations from the simulations pre- For the Milan dataset, the bundle solutions without GNSS gen-
sented in Section 2. For the Graz dataset the advantage of oblique erally show a gain in accuracy when IO parameters are treated as
views is less obvious but nevertheless visible especially in the FR unknowns (self-calibration). In the GNSS-assisted solutions this
examples of GCP configurations 1 and 2, where it leads to better evidence is less pronounced and the RMS errors at independent
accuracies in the Z direction. What is symptomatic is the fact that CPs are comparable, suggesting that (i) the available camera cali-
in GCP configurations 3 and 4, i.e. the unfavourable point distribu- bration is valid and self-calibration is superfluous, and (ii) the
tions, nadir images (1FR) deliver either better or the same Z-accu- newly adjusted camera calibration parameters in the GCP-only
racy results as 2FR or 3FR with the oblique views included. The cases do not correct for systematic errors stemming from physical
exclusively nadir block seems to be more stable in the event of deficiencies in the additional parameters model but, due to projec-
deficient ground control information, which is opposite to the tive coupling with the EOR, do compensate for the suboptimal esti-
results in the Milan dataset. The different behaviour could be due mates of camera rotations and positions (see Section 4.4 for more
to (i) different parameters of the nadir camera models, i.e. large evidence). For the Graz dataset, freeing self-calibration parameters
sensor and shorter focal length of the Osprey I system provide in the bundle adjustment improved results in the Z coordinates for
for better imaging geometry than the small sensor and long focal all GCPs configurations (compare all FX versus FR). The provided
244 E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

1 2 3 4 2 3 4
3FX (GCPs config. 1(2)-4) - 3FX (GCPs config 0(1))

no- no-
GNSS GNSS

μ = 0.34 m μ = 0.83 m μ = 2.09 m μ = 1.56 μ = 0.33 m μ = 1.43 m μ = 1.15 m


σ = 0.18 m σ = 0.47 m σ = 1.18 m σ = 0.86 m σ = 0.23 m σ = 0.92 m σ = 0.83 m

GNSS GNSS

μ = 0.08 m μ = 0.19 m μ = 0.45 m μ = 0.18 m μ = 0.16 m μ = 0.35 m μ = 0.19 m


σ = 0.13 m σ = 0.21 m σ = 0.15 m
σ =0.03 m σ = 0.09 m σ = 0.25 m σ = 0.08 m
3FR (GCPs config. 1(2)-4) - 3FR (GCPs config 0(1))

no-
no-
GNSS
GNSS

μ = 0.07 m μ = 1.28 m μ = 1.07 m


μ = 0.18 m μ = 0.50 m μ = 1.10 m μ = 0.98 m σ = 0.05 m σ = 0.94 m σ = 0.75 m
σ = 0.09 m σ = 0.27 m σ = 0.51 m σ = 0.63 m

GNSS GNSS

μ = 0.09 m μ = 0.18 m μ = 0.28 m μ = 0.14 m μ = 0.01 m μ = 0.11 m μ = 0.07 m


σ = 0.01 m σ = 0.07 m σ = 0.06 m
σ = 0.04 m σ = 0.09 m σ = 0.15 m σ = 0.07 m

Fig. 10. l – arithmetic mean Euclidean distance, and r – standard deviation between point clouds in selected scenarios and the reference cloud. The reference is unique for all
comparisons – 3FR in GCP configuration 0 in Milan, and GCP configuration 1 in Graz. Left: Milan, right: Graz datasets. Note that the legends should be decoded separately for
each dataset.

camera calibration, unlike that of the Milan dataset, was not suffi- weights for the GNSS data should be used within the bundle
cient to achieve the most optimal result and self-calibration affor- adjustment.
ded superior results. As seen in the discrepancy plots in Fig. 8, the
vectors in Milan dataset do not change between the cases of fixed 4.4. Evaluation of derived sparse point clouds
or freed. Their lengths in the Graz dataset, however, clearly recede
if camera calibration parameters are re-adjusted. No ground truth in the form of DSM data was available for
either the Milan or Graz networks. Possible deformations and the
4.3.3. Influence of GCPs distribution and GNSS observations level of noise in the derived 3D object coordinates (i.e. sparse point
The GCP distribution plays a crucial role in determining CP cloud) were evaluated for the cases of with and without oblique
accuracy. So long as the distribution of control information is even images, GNSS/IMU data and when GCP configurations changed.
and appropriately dense, the constraint on CPs is satisfactory (GCP Conclusions were drawn from comparisons in Figs. 9 and 10.
configuration 0 and 1 in Table 5). As soon as the GCP configuration In Fig. 9, in order to ascertain the influence of the inclusion of
gets sparser, extrapolation effects emerge and are reflected in lar- oblique images, scenarios 1FX and 1FR were compared with sce-
ger RMS error values (GCPs configuration 2, 3 and 4 in Table 5 and narios 3FX and 3FR. To free the analyses from the impact of GCP
Figs. 8 and 10). The inclusion of GNSS constraints largely mitigates locations inasmuch as possible, results obtained in GCPs configura-
the problem (GCPs configuration 2, 3 and 4 in Table 5, values in tion 0 (Milan) and 1 (Graz) were used. As a side-product, also the
brackets). Under poor GCP distributions, the Z coordinate suffers influence of self-calibration on the triangulated 3D points was
the most, and this is the case for both datasets. In GCP configura- devised from the differences 1FX with 1FR and 3FX with 3FR.
tion 3 the blocks ‘‘wobble’’ in counter directions (compare the In Fig. 10, in order to determine the influence of the GCPs distri-
RMS in X and Y in Table 5). Nonetheless, although the Milan dataset bution, the oblique image blocks 3FR and 3FX, in all GCP configu-
appears highly sensitive to GCP variations, the sensitivity of the rations, were compared against a reference result. The reference
Graz network is minor, especially when GNSS constraints are is unique for all comparisons and was chosen to be the one report-
included. Again, the imaging geometry and extent of image overlap ing the optimal RMS error values at CPs, namely 3FR in GCPs con-
are the most probable factors influencing this sensitivity. In the figuration 0 (Milan) and 1 (Graz). The comparisons quantified the
Milan case study, switching on the GNSS constraints in one of mean differences and standard deviations. Also color coded maps
the ‘‘good’’ scenarios (3FR, GCPs configuration 1) leads to larger of the geometric differences (Euclidean distance) were produced.
coordinate discrepancies at CPs (0.20 m against 0.13 m). This The distance values were color coded such that they ranged from
behaviour could be a sign of ‘‘tension’’ between ground control 0 to 6 times (Milan dataset) or 3 times (Graz dataset) the standard
information and perspective centers. As a remedy, less optimistic deviation determined in each test.
E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246 245

4.4.1. Influence of oblique views and self-calibration duces substantial enhancement in precision over the
The sparse point differences confirm and complement the con- corresponding block with nadir images only. The precision in
clusions of the previous subsection. Adopting oblique cameras height determination is increased some two to five times com-
reduces the magnitude of noise in 3D object space and homogenis- pared to the nadir image case. While increasing the overlap to an
es the distribution of discrepancies within the block. Blocks con- 80/60 scenario can bring about further improvements, the change
taining the full set of 5 or 7 imaging cones are less sensitive to in precision between 80/60, 80/80 and 90/90 is less dramatic. To
the GCPs arrangement and shorten the growth of discrepancies grasp the actual value of the increased overlap, the gain in preci-
outside the region containing the GCPs. Bundle adjustment with sion ought to be confronted with the extra labour imposed by den-
self-calibration brings about further profits with regard to noise sified data collection. It has been shown that the latter grows
and deformations (see the patterns in 1FX – 1FR and 3FX – 3FR). exponentially. On the other hand, having in mind the application
The ‘‘dooming’’ effect that is characteristic for un-modelled radial of 3D city modelling from imagery and knowing that the content
distortion error in 1FX – 1FR of the Graz dataset (Fig. 9, bottom), of urban scenes is complex, with visibility restricted due to occlu-
provides further evidence of an outdated camera calibration and sions, the overlap should be kept large. It is advised that as a part of
justifies the freeing of additional parameters in the bundle adjust- the project planning camera system and flight parameters are
ment. James and Robson (2014) found that such systematic errors adapted to the topography of the city. The building height to street
can be mitigated if oblique images are acquired in the block. Based width ratio is an important factor in the planning phase.
on real and simulated studies, they showed that in the presence of If productivity is considered as the number of images needed to
radial distortion, with an assumed error-free camera model, (i) a conduct a survey over an area, evidently larger sensors and greater
nadir block will produce the ‘‘dome’’ in the DEM (see 1FX – 1FR) tilt angles will outperform smaller sensors and tilt angles (pro-
and (ii) an oblique block will decrease the magnitude of discrepan- vided that there is no change in focal lengths). Nonetheless, it
cies (no ‘‘dome’’ effect in 3FX – 3 FR). It can be validly claimed that has been shown that although the 45° tilt angle gives best results
under conditions of self-calibration, the systematic vertical DEM from the perspective of object point precision, it is more prone to
error is eliminated, for all practical purposes, which explains the occlusions.
small discrepancies obtained in 1FR – 3FR. Similar conclusions The empirical part of the paper evaluated two datasets, one
are reported in Wackrow and Chandler (2011), Nocerino et al. from a Midas 5 camera system with small sensors (Milan dataset)
(2014). Besides the radial distortion, another cause of the and the other recorded with an UltraCam Osprey I with large sen-
systematic block deformations in Figs. 9 and 10 is possibly the sor sizes (Graz dataset). The UltraCam Osprey I dataset outper-
un-modelled out-of-plane distortion, i.e. unflatness. Fraser (1997) forms the Midas 5 dataset in accuracy as a consequence of (i)
warns that systematic errors caused by the sensor unflatness are larger along- and cross-strip overlap, (ii) different imaging
critical in that the image space perturbations might be concealed geometry that stems from different camera internal parameters,
while their effect on photogrammetric triangulation precision is i.e. large sensor and short focal length versus small sensor and long
severe. The induced radial displacement depends on the value of focal length, and (iii) the camera being designed for metrically
the incidence angle thus wide-angle lenses are especially subject accurate mapping purposes rather than comprising integrated
to manifest the errors. The standard 8-parameter and the extended cameras designed more for the consumer market. The advantage
10-parameter models are insufficient to compensate for these of the UltraCam system is noticeable when considering CP accu-
misclosures. Yet, they are present in today’s photogrammetric sys- racy, which was in the same range as the GCPs measurement accu-
tems. In theory higher degree polynomial functions could remove racy. Furthermore, the magnitude of the noise in the sparse point
these systematisms, but in practise their application is not advised cloud comparisons was smaller. The tests conducted, however,
for the arising correlations between the parameters. Development do not constitute a basis for direct comparison of the two oblique
of new mathematical models is therefore encouraged (Honkavaara camera systems due to (i) the different flight patterns, (ii) different
et al., 2006b; Tang et al., 2012). extent of the regions covered, and (iii) it was not the aim of this
paper to comment upon which of the two systems might be
superior.
4.4.2. Influence of GCPs distribution and GNSS observations
Both evaluations indicate that inclusion of oblique imagery into
In both datasets the ‘‘strain’’ that the GCPs induce on the entire
the bundle adjustment brings a significant gain in height accuracy,
block is striking in all the block adjustment solutions that do not
as well as improved stability of the image block when (i) the over-
include GNSS constraints. The discrepancies are kept small within
lap is small and (ii) no GNSS or well distributed GCPs are available.
the general region enclosed by GCPs (marked with red dots in
Likewise, the height accuracy is improved in the self-calibrating
Fig. 9) whereas extrapolation errors increased away from this
bundle adjustment. The characteristic deformation patterns arising
region. When GNSS constraints are introduced into the bundle
in point cloud comparisons revealed the remaining sensor-related
adjustment, the magnitude of these discrepancies is substantially
errors. The applied 8-parameter camera model failed to correct all
reduced, as is the extrapolation error. GNSS observational con-
remaining systematic errors. For this reason, future work will
straints stabilize the bundle adjustment solution, due to projection
involve the study of other additional parameter models able to
centers behaving like control points. The visible elongated defor-
remove all inaccuracies and thus more suitable for the employed
mation pattern (Fig. 9, top) arising in the comparison of the nadir
sensors.
image blocks (1FX – 1FR) with GNSS constraints was most likely
be due to dependencies between camera calibration parameters
Acknowledgements
and exterior orientation parameters.

This work is partially funded by the RAPIDMAP project, a Euro-


5. Conclusions pean Union (EU) funded project in the International Cooperation
Activities under the Capacities Programme of the EU 7th Frame-
This paper has presented an analysis of oblique multi-camera work Programme for Research and Technology Development
systems employed for mapping applications. The simulated and (CONCERT-Japan).
real case studies demonstrate that increasing the overlap, and thus The authors are thankful to Blom-CGR S.p.A. (Parma, Italy) and
point redundancy, improves 3D precision and accuracy. The tradi- Microsoft UltraCam (Graz, Austria) for providing the oblique imag-
tional 60/40 flight pattern applied to an oblique image block pro- ery datasets processed in the presented work.
246 E. Rupnik et al. / ISPRS Journal of Photogrammetry and Remote Sensing 101 (2015) 233–246

References Kraus, K., 1997. Photogrammetry – Advanced Methods and Applications. Dümmler
Verlag, 466pages.
Kraus, K., 2007. Photogrammetry: Geometry from Images and Laser Scans. Walter
Aber, J.S., Marzolff, I., Ries, J., 2010. Small-Format Aerial Photography: Principles,
de Gruyter, 459 pages.
Techniques and Geoscience Applications. Elsevier, 268pages.
Kurz, F., Müller, R., Stephani, M., Reinartz, P., Schröder, M., 2007. Calibration of
Ackermann, F., 1992. Kinematic GPS control for photogrammetry. Photogramm.
wide-angle digital camera system for near real time scenarios. International
Record 14 (80), 261–276.
Archives of Photogrammetry and Remote Sensing 36(1/W51), Hannover,
Apollonio, F., Ballabeni, A., Gaiani, M., Remondino, F., 2014. Evaluation of feature-
Germany.
based methods for automated network orientation. Int. Arch. Photogramm.
Lemmens, M., Lemmen, C., Wubbe, M., 2008. Pictometry: Potentials for land
Remote Sens. Spatial Inform. Sci. 40 (5), 47–54.
administration. In: International Federation of Surveyors, 6th FIG Regional
Barazzetti, L., Scaioni, M., Remondino, F., 2010. Orientation and 3D modeling from
Conference, San Jose, Costa Rica.
markerless terrestrial images: combining accuracy with automation.
Manual of Photogrammetry, second ed., Chapter II, Part1, Aerial Cameras and
Photogramm. Rec. 25 (132), 356–381.
Accessories, ASPRS, 1952.
Cavegn, S., Haala, N., Nebiker, S., Rothermel, M., Tutzauer, P., 2014. Benchmarking
Mason, S.O., 1995. Conceptual model of the convergent multistation network
high density image matching for oblique airborne imagery. Int. Arch.
configuration task. Photogram. Rec. 15 (86), 277–299.
Photogramm. Remote Sens. Spatial Inform. Sci. 40 (3), 45–52.
McGlone, J.C., 2004. Manual of Photogrammetry. American Society for
Colwell, R.N., 1997. History and place of photographic interpretation. In: Philipson,
Photogrammetry and Remote Sensing, 1151pages.
W.R. (Ed.), Manual of Photographic Interpretation, second ed. American Society
Mishra, P., Ofek, E., Kimchi, G., 2008. Validation of vector data using oblique images.
for Photogrammetry and Remote Sensing, Bethesda, Maryland, United States,
In: Advances in Geographic Information Systems, 16th ACM SIGSPATIAL
pp. 3–47.
International Conference, Irvine, CA.
Cramer, M., 2009. Digital Camera Calibration – Report of EuroSDR Project. 262
Murtiyoso, A., Remondino, F., Rupnik, E., Nex, F., 2014. Oblique aerial photography
pages. Frankfurt a.M. 2009.
for building façade inspection and damage assessment. Int. Arch. Photogram.
Förstner, W., 1985. The reliability of block triangulation. Photogramm. Eng. Remote
Remote Sens. Spatial Inform. Sci., 40–41
Sens. 51, 1137–1149.
Nex, F., Rupnik, E., Remondino, F., 2013. Building Footprints Extraction from Oblique
Förstner, W., 1998. On the theoretical accuracy of multi image matching,
Imagery. ISPRS Ann. Photogramm. Remote Sens. Spatial Inform. Sci. II-3/W3, pp.
restoration and triangulation. In: Festschrift zum 65. Geburtstag von Prof. Dr.-
61–66.
Ing. mult. G. Konecny. Institut für Photogrammetrie, Universität Hannover.
Nocerino, E., Menna, F., Remondino, F., 2012. Multi-temporal analysis of landscapes
Fraser, C.S., 1984. Network design considerations for non-topographic
and urban areas. Int. Arch. Photogram. Remote Sens. Spatial Inform. Sci. 39 (4),
photogrammetry. Photogramm. Eng. Remote Sens. 50 (8), 1115–1126.
85–90.
Fraser, C.S., 1996. Network Design. In: Atkinson, K.B. (Ed.), Close Range
Nocerino, E., Menna, F., Remondino, F., 2014. Accuracy of typical photogrammetric
Photogrammetry and Machine Vision, Whittles, Scotland, 371p., pp. 256–281.
networks in cultural heritage 3D modeling projects. Int. Arch. Photogram.
Fraser, C.S., 1997. Digital camera self-calibration. ISPRS J. Photogramm. Remote
Remote Sens. Spatial Inform. Sci. 40-5, 465–472.
Sens. 52 (4), 149–159.
Nyaruhuma, J., Gerke, M., Vosselman, G., 2012. Verification of 3D building models
Fritsch, D., Rothermel, M., 2013. Oblique image data processing: potential,
using mutual information in airborne oblique images. ISPRS Ann. Photogram.
experiences and recommendations. Proc. 54th Photogrammetric Week, pp.
Remote Sens. Spatial Inform. Sci., I-3.
73–88.
Petrie, G., 2008. Systematic aerial oblique photography using multiple digital
Fritsch, D., Kremer, J., Grimm, A., 2012. Towards all-in-one photogrammetry. GIM
cameras. In: VIII International Scientific & Technical Conference. Porec, Croatia.
Int. 26 (4), 18–23.
Petrie, G., 2010. Current developments in airborne digital frame cameras: as
Gerke, M., 2009. Dense matching in high resolution oblique airborne images. Int.
displayed in the Intergeo 2010 exhibition. GeoInformatics 13 (8), 34–40.
Arch. Photogram. Remote Sens. Spatial Inform. Sci. 38 (3/W4), 77–82.
Pierrot-Deseilligny, M., 2014. MicMac, Apero, Pastis and Other Beverages in a
Gerke, M., Nyaruhuma, A., 2009. Incorporating scene constraints into the
Nutshell. Manual http://logiciels.ign.fr/?Telechargement,20 (Last access
triangulation of airborne oblique images. International Archives of
October, 2014).
Photogrammetry, Remote Sensing and Spatial Information Sciences 38(1-4-7/
Pierrot-Deseilligny, M., Clery, I., 2011. APERO – open source bundle adjustment
W5), Hannover, Germany.
software for automatic calibration and orientation of a set of images. Int. Arch.
Gerke, M., Xiao, J., 2013. Supervised and unsupervised MRF based 3D scene
Photogramm. Remote Sens. Spatial Inform. Sci. 38 (5/W16), 269–277.
classification in multiple view airborne oblique views. ISPRS Ann. Photogramm.,
Pierrot-Deseilligny, M., Paparoditis, N., 2006. A multiresolution and optimization –
Remote Sens. Spatial Inform. Sci., II-3/W3, pp. 25–30.
based image matching approach: an application to surface reconstruction from
Grenzdörffer, G., Guretzki, M., Friedlander, I., 2008. Photogrammetric image
SPOT 5 – HRS stereo imagery. Int. Arch. Photogramm., Remote Sens. Spatial
acquisition and image analysis of oblique imagery. Photogram. Rec. 23 (124),
Inform. Sci. 36 (1/W41).
372-368.
Rupnik, E., Nex, F., Remondino, F., 2013. Automatic orientation of large blocks of
Gruber, M., Wolfgang, W., 2014. Calibrating the new ultracam osprey oblique aerial
oblique images. Int. Arch. Photogram. Remote Sens. Spatial Inform. Sci. 40 (1/
sensor. Int. Arch. Photogram. Remote Sens. Spatial Inform. Sci. 40–3 (W1), 47–
W1), 299–304.
52.
Rupnik, E., Nex, F., Remondino, F., 2014. Oblique multi-camera systems –
Haala, N., Kada, M., 2010. An update on automatic 3D building reconstruction. ISPRS
orientation and dense matching issues. Int. Arch. Photogramm. Remote Sens.
J. Photogram. Remote Sens. 65 (6), 570–580.
Spatial Inform. Sci. 50 (3/W1), 107–114.
Honkavaara, E., Ahokas, E., Hyyppä, J., Jaakkola, J., Kaartinen, H., Kuittinen, R.,
Tang, R., Fritsch, D., Cramer, M., 2012. New rigorous and flexible Fourier self-
Markelin, L., Nurminen, K., 2006a. Geometric test field calibration of digital
calibration models for airborne camera calibration. ISPRS J. Photogram. Remote
photogrammetric sensors. ISPRS J. Photogram. Remote Sens. 60 (6), 387–399.
Sens. 71, 76–85.
Honkavaara, E., Jaakkola, J., Markelin, L., Nurminen, K., Ahokas, E., 2006b.
Wackrow, R., Chandler, J.H., 2011. Minimising systematic error surfaces in digital
Theoretical and empirical evaluation of geometric performance of multi-head
elevation models using oblique convergent imagery. Photogram. Rec. 26 (133),
large format photogrammetric sensors. Int. Arch. Photogram. Remote Sens.
16–31.
Spatial Inform. Sci. 36 (1), 56–61.
Wang, Y., Schultz, S., Giuffrida, F., 2008. Pictometry’s proprietary airborne digital
Jacobsen, K., 2008. Geometry of vertical and oblique image combination. In: Remote
imaging system and its application in 3D city modelling. Int. Arch. Photogram.
Sensing for a Changing Europe: Proceedings of the 28th Symposium of the
Remote Sens. Spatial Inform. Sci. 10 (B1), 1065–1070.
European Association of Remote Sensing Laboratories, Istanbul, Turkey.
Wiedemann, A., More, J., 2012. Orientation strategies for aerial oblique images. Int.
James, M.R., Robson, S., 2014. Mitigating systematic error in topographic models
Arch. Photogram. Remote Sens. Spatial Inform. Sci. 39 (B1), 185–189.
derived from UAV and ground-based image networks. Earth Surf. Proc. Land. 39
Xiao, J., Gerke, M., Vosselman, G., 2012. Building extraction from oblique airborne
(10), 1413–1420.
imagery based on robust facade detection. ISPRS J. Photogram. Remote Sens. 68,
Karbo, N., Simmons, G., 2007. Aerial imagery from different angles. Prof. Surveyor
56–68.
Mag. 27, 8–11.