Drone-based reconstruction for 3D geospatial data processing
Jason D. Renwick Levente J. Klein
IBM TJ Watson Research Center IBM TJ Watson Research Center
Yorktown Heights, NY 10598 Yorktown Heights, NY 10598
jdrenwic@us.ibm.com kleinl@us.ibm.com
Abstract—Urban air quality affects health and well-being
of more than half of the world's population. Measurement,
modeling and real time action on pollution sources can alleviate
their impact. A preliminary study for drone-based image
acquisition and processing steps required for 3D reconstruction
of potential pollution sources is presented. The 3D surface
terrain models combined with sensor data are inputs into air
pollution models for visualization, understanding and potential
mitigation of methane plumes. Scaling these technologies across
large areas requires the integration of big geospatial data with
modeling, machine learning, and image processing.
Keywords-Drones; Unmanned Air Vehicle; Internet of
Things; Smart Cities; Open Source; IoT; UAV; Air Quality
Monitoring.
I. I NTRODUCTION
Cities are a multi system operating environment with
sensors continuously acquiring data from traffic, water dis-
tribution networks, electric grids, and air quality stations.
Such data is used to improve the quality of life for citizens,
increase operational efficiencies, and reduce waste. Recently,
the quantity of data from mobile devices, social media
and crowdsourcing has increased exponentially, providing
snippets of information about the surrounding environment.
Many data sets are hosted in proprietary databases where
access and data formats are controlled. There is a clear need
for data fusion of heterogeneous data sets, data curation and
advanced analytics so that all data can be easily accessed
across multiple computational platforms and be queryable
for any location and for any time interval.
Internet of Things (IoT) can be envisioned as an in-
teroperable collection of cyber-physical devices comprising
electronics, sensors/actuators, software and network connec-
tivity, enabling such devices to dynamically connect and
exchange data over the Internet. Traditionally, IoT was
associated with sensor networks which could be expanded
over large areas by scaling the number of sensing nodes.
Alternatively, satellites offer a large-area observation of the
Earth's surface that can track spatial and temporal changes
which are considered as important for many applications [1].
However, neither sensor networks nor satellite observations
can fulfill the demand for simultaneous large-area coverage
or high temporal resolution observations.
Drones are low-cost sensor platforms which can be flown
easily using either manual control or preprogrammed routes.
729
Hendrik F. Hamann
IBM TJ Watson Research Center
Yorktown Heights, NY 10598
hendrikh@us.ibm.com
Drones have multiple advantages, such as, being easily
deployable, allowing reprogramming of flight paths during
operation, and having sufficient computational power to
carry out calculations during data acquisition. Drone usage
has been extended to geospatial mapping, as an alternative to
satellite observations, for quick observation and inspection
of small areas. This can be especially important for situations
like disaster management when rapid data acquisition can
save human lives and protect properties.
In recent times, natural gas (methane) exploration sites
have been developed in close proximity to urban areas.
In the Dallas/Fort Worth area the gas exploration sites
are interweaved with homes. Lately, pollution surveys have
established that these sites may have contributed to increased
greenhouse emission and potentially impact human health
[2]. We describe in this paper a potential framework (Figure
1) for air pollution assessment based on the integration of
drone imagery, data fusion, and physical modeling.
II. D RONE T ECHNOLOGY
Here, we explore the applicability of drones for air quality
monitoring, especially for situations where gas emissions
sites are clustered. High concentrations of methane gas,
emitted from a single well pad, may be harmless but close
proximity of multiple emission sources can create hazardous
situations. Drones are ideal to inspect the gas plumes due
to low operating cost, flexibility in flying at different alti-
tudes, and the possibility to schedule a flight at almost any
moment. The drone-acquired high resolution imagery can
be processed using open source tools and machine learning
techniques. The confluence of easy drone operations and
image analytics make the drone-based implementations a
very attractive solution to address these challenges in the
oil and gas industry.
While drones are most commonly associated with high
quality image processing, they can be fitted with application-
specific sensors like Light Detection And Ranging (LiDAR),
spectral cameras, acoustic or chemical sensors. The onboard
sensor can complement Global Positioning System (GPS),
compass, barometric pressure sensors which are already part
of the drones navigation system. Mobile measurements using
drone-attached sensors can acquire measurements at any
location of the 3D space and adjust flight path in real time
 Data
Drones
Images GDAL
Sensor data Python
OpenCV
PAIRS
Satellite Weather
Topography Census
Figure 1. Framework for drone imagery processing, analytics and integration with PAIRS
to sample the spatial and temporal variations of gas plume
dispersion.
Drones may also act as a mobile information transmission
link between remote sensing points having no Internet con-
nectivity and data processing cloud platforms. The drone can
download sensor information from locations with no Internet
connectivity, move the data to a location where Internet
access exists, and upload the data to a cloud platform.
Furthermore, if the acquired data results into an actionable
command, drones can take the actuation command from the
cloud platform, transport it to the relevant sensing point
and transmit the information to the control system that can
execute the actuation. Such commands can, for example, be
turning off a gas valve or shut off the well pad operation.
Visual observation by drones can ensure that the command
was carried out successfully. In such scenarios, drones
enable remote operation of sites where direct communication
with control systems may not exist.
Apart from data collection, drones can be used as edge
devices for real time processing using attached computers
like a Raspberry Pi [3]. On board data processing can enable
independent decision making, the precursor for cognitive
processes and autonomy. Proactive decisions based on image
processing and modeling can enable machine learning and
actuation. Until drones evolve to a state where they can take
independent decisions, multiple issues related to automatic
flight control and avoiding obstacles and collision have to be
addressed. In this paper, we present the first steps in image
acquisition and processing e.g. stitching, orthorectification,
geo referencing, and object recognition and discuss the
necessary steps to integrate 3D reconstruction of gas well
pad sites with Computational Fluid Dynamics models.
Processing Analytics
Orthorectification Machine learning
Stitching
3D reconstruction
GeoReferencing
Segmentation
Data Fusion
Video Analytics
Data Object recognition
curation
III. H ARDWARE AND S OFTWARE
The drone used for this setup was the DJI Phantom 3
Standard which had a built-in camera, GPS module and
sensor array. This drone weighs 1.22 kg and can carry
an estimated maximum payload of 1.5 kg. DJI provides
an android/ios app for inflight viewing through the built-
in camera, image capture configurations and drone flight
configurations. For the flights outlined in this paper, the
image acquisition mode was set to a 5-second interval.
The exposure settings were manually set to keep uniform
exposure between the images.
A. Flight Limitations
The 4480 mAh battery operated nominally at 15.2V and
allowed for a maximum flight time of 20 minutes under
normal conditions. The maximum speed the drone can fly
at under no wind conditions is 16 m/s limiting the safe
operation of the drone to horizontal wind conditions of
16 m/s or less. Legally, drones can fly up to 100 mph.
The current state-of-the-art drone technology allows for
drones to fly at 80 mph while still recorded high resolution
imagery. The legal altitude limit of an unmanned air vehicles
(UAVs) is 400 feet. The remote controller used with the
drone allowed for a maximum signal distance of 1000 m
with direct line of sight. For the image stabilization during
flight, the camera was attached to a gimbal that allowed for
stabilization between -90 to 30 degrees of flexibility.
B. Camera
The drone is outfitted with a camera with a 1/2.3-inch
12 megapixel sensor. The images acquired were 4000x3000
pixels. The permanently attached rectilinear prime lens has
a 20mm focal length (35mm equivalent). The rectilinear
construction of the lens produced a 94-degree field of view
730
with little barrel distortion. During the stitching process, the
images were undistorted using a calculated camera calibra-
tion matrix in OpenCV [4]. For small areas of stitching, the
distortion was considered negligible and the calibration step
was ignored.

C. Georeferencing
The process of achieving orthorectification and georefer-
encing were combined into a single transform with the use
of Geospatial Data Abstraction Library (GDAL) [5]. The
open-source GDAL package supplied for Python allows for
geospatial processing of images. To warp the mosaic into
orthorectification with accurate georeferencing, a polyno-
mial transform was applied to the image. The georeferencing
was tested with linear, perspective, Helmert and projective
transform. These tests produced subpar results compared to
the polynomial transform. To apply the polynomial trans-
form, at least 10 Ground Control Points (GCPs) had to
be matched between the stitched mosaic and known GPS
coordinates. For the experimental flights, there were no
physical GCPs planted in the area covered by the drone. To
acquire the GCPs needed for the transform, a comparison
of key image features was done between the mosaic and a
known orthorectified satellite image. The transform would
yield fairly good results (Figure 2) once at least 15 GCPs
were acquired. Since the process involved applying a finite
transform, the resulting orthomosaic was limited by 2 major
factors: i) the accuracy of the compared satellite imagery and
ii) the quantity of GCPs acquired. For visualization, quick
and simple comparison, Quantum Geographic Information
System (QGIS) was used to assist with the orthorectification
and georeferencing [6]. The georeferencer plugin in QGIS
also allowed for GDAL script generation so results could be
reproduced in a Python shell without the use of external
software. The use of the generated Python script could
restrict the entire image processing framework to a single
Python shell.
D. Methane Leak Detection
Traditional methane leak detection has been done with a
thermal/infrared camera. These cameras have been retrofitted
to drones to suit additional applications such as power grid
inspection [7]. The drone hardware described previously can
be combined with current infrared imaging technology to
provide a detailed thermal survey of a large area. Thermal
imaging can be combined with the RGB georeferenced
orthomosaic to produce an accurate multi-spectral high reso-
lution geospatial layer. The resulting images can guide more
precised inspection, detection or compliance enforcement of
chemical leaks [8]. The spectral imagery obtained from the
infrared camera can be supplemented by chemical sensors
on the drone to corroborate the potential leaks detected.
Figure 2. Georeferenced stitched orthomosaic (middle left) overlaid on a
satelitte map (right) showing accuracy of 1 m in georeferencing
E. Software
Open source tools, Python along with OpenCV 2.4.13 was
primarily used in producing the stitched mosaic of aerial
images. The image stitching algorithm (Figure 3) used a
combination of native OpenCV functions. It is a simplified
version of the panoramic invariant stitching algorithm [9].
All of the images are matched to a reference image. After
the keypoints are detected and matched, the second image
is warped towards the reference image using a perspective
transform. After every iteration, the reference image is
updated and the remaining images are matched to the new
reference. This is repeated as long as sufficient matches are
found between an image in the pool and the reference image
for that iteration. For small scale applications, the Python
wrapper was most convenient to use with strong online
support. For larger applications, the OpenCV framework
would be better used with C++ as it allows for easier access
to parallel Graphics Processing Unit (GPU) processing and
memory allocation.
IV. E XPERIMENTAL F LIGHTS AND DATA
A. Flight 1
The drone was flown along the outside of the IBM’s TJ
Watson Research Center to acquire aerial imagery at a height
of approximately 115 m. The image capture was done man-
ually and images were captured approximately 30 m apart.
While in flight the camera was aimed perpendicular to the
Earth's surface or -90 degrees from the drone. These images
were used in the construction of the stitched orthomosaic.
B. Flight 2
The drone was flown at varying heights between 3-15 m
around an object or structure to capture terrestrial imagery.

731
 Image Pool
Remove distortion
[CameraCalibrate]
Resize Image
[Resize]
Find Key Points Match Features
[FeatureDetect] [Matcher]
Obtain
Warp/Combine
Homography
Images
[FindHomography]
Figure 3. Algorithm for image stitching
While in flight, the camera was always pointed at the
object and images were acquired at angles between -90 to
0 degrees to the object. These images were used for the 3D
reconstruction (Figure 4) [10].
V. DATA P ROCESSING
Drones can acquire images at a 10 cm or higher spatial
resolution thus generating a few 10s of Gbyte of data
during a single flight. During drone flight, apart from
image acquisition, the on-board sensor data can also be
acquired; longitude, latitude, altitude, velocity, ground speed,
accelerometer, gyroscope, barometer, quaternion, roll, pitch,
yaw and compass readings. These point data offer contextual
information for the imagery, e.g. flight path can be used to
determine the best future flight route. Data management is
a critical aspect in applications where multiple data sets are
acquired across the same geographical region.
To host the drone imagery, we are using a big data
geospatial platform called, Physical Analytics Integrated
Repository and Services (PAIRS). It provides access to more
than 20 spatially and temporarily aligned data layers [1].
Presently, PAIRS hosts more than 500 TByte of geospatial
data with more than 5 TByte of new data added weekly.
All data layers in PAIRS are curated and indexed. Indexing
each data point allows searching every geospatial data layer
732
Figure 4. Examples of 3D reconstruction with 2D drone images
and extracting only information of interest, e.g. give me all
houses that are within 30 m from street and have vegetation
in close proximity while the terrain is on a slope of 5%.
Traditionally, these calculations require massive amounts of
data download and processing but in our case the PAIRS
database handles all these cross layer queries.
Additionally, PAIRS is a computational platform based on
a highly parallelized processing implemented on Hadoop and
Spark for leveraging built-in machine learning algorithms.
To train machine learning algorithms, access to histori-
cal data sets is required. PAIRS hosts such geospatially
referenced datalayers to train object detection and feature
recognition. We integrated drone imagery in PAIRS by
creating contiguous large area images. Once included in
PAIRS, the drone images are aligned with all existing data
layers such as satellite, weather and topography layers.
PAIRS also serves as the computation platform for drone
image classification and object recognition using Python,
GDAL, and OpenCV. Currently, image segmentation and
classification are implemented to identify and delineate
features in the drone imagery, like vegetation, buildings, cars
and/or road networks. The drone-extracted information can
be validated against PAIRS's hosted data like road networks,
population census, or building layouts [1].

Figure 5. Overlay of multiple georeferenced data layers in PAIRS
VI. A IR P OLLUTION M ONITORING
We propose drone imagery as a quick and precise way
to improve air pollution modeling in urban areas. There are
more than 200 well pads with methane emission potential
positioned in and around the city perimeter of Fort Worth,
TX. Since prevalent winds in the area can move the methane
leak plumes towards schools, homes for the elderly, and
hospitals any modeling needs to consider the geospatial
component of the detection. Surface terrain models extracted
from drone imagery help in modeling wind turbulence
around buildings and infrastructure. These are necessary
inputs for pollution models like QUIC, AERMOD, and
CALPUFF [11], [12], [13]. The 3D reconstruction of a city
neighborhood or well pads can capture the current ground
conditions. Furthermore, wind and methane concentration
from IoT sensor networks deployed around that area can
be used as input into dispersion models to constrain and to
ensure accuracy in methane level modeling. Computational
Fluid Dynamics combined with accurate 3D reconstruction
of the region of interest and IoT sensor network measure-
ments can provide precise estimates of city level pollution.
VII. D ISCUSSION
Some of the steps involved in using drones as low-cost
data acquisition tools were discussed, however, there are
limitations that restrict the versatility of such a system.
One obvious limitation is the physical constraints of a
UAV, particularly, flight constraints. Current state-of-the-art
drone technology allows for a maximum of 20-30 minutes
of flight time on a full battery. The limitations in battery
lifetime, requires careful planning of the drone flight. In
manual or semi-automatic flights, excessive data is acquired
since during flight it is difficult to estimate if consecutive
images have sufficient overlap for stitching or if images
are suitable for 3D reconstruction. Also, the autonomous
capabilities of drones are not standard across all available
UAVs. This implies most drones in flight have different
decision making processes when faced with an obstacle. This
lack of predictability may restrict the ability to have multiple
concurrent drone data acquisition networks. Work related
733
to drone autonomy involves enhancing drones with sensors
and onboard processing capabilities [14], [15]. In the future,
drones may be capable of automatically determining the best
route, number of snapshots, and viewing angle to acquire
images for specific applications. Such cognitive decision
making capabilities are currently being investigated.
Additionally, having a uniform standard regarding the
operation of drones is needed. Recently, the FAA released
some regulations outlining the restrictions related to UAV
operation but it does not describe, in detail, the standard
for drone control in low altitude airspace. One can argue
that the rules for operating a drone in a city should differ
to the operation of a drone in the countryside. Solutions
or implementations related to this limitation are presently
being explored with traffic management systems for UAV
operation [16].
VIII. C ONCLUSION
In the paper, we presented the steps involved in process-
ing aerial images captured by drones through open source
software. Additionally, the large volume of data collected
was also shown to be handled and curated by IBM's PAIRS
platform. These steps are necessary towards integrating
drones into air pollution monitoring frameworks and models.
Furthering this work will involve outfitting a drone with
additional sensors and reconstructing methane plumes.
ACKNOWLEDGMENT
The authors would like to thank Fernando Marianno,
Conrad Albrecht, Xiaoyan Shao, Michael Schappert and
other researchers at IBM’s Thomas J Watson center for their
assistance in processing and understanding geospatial data,
guidance through the progression of the drone technology
work and intellectually engaging discussions. Norma Sosa
for facilitating and supporting the Geospatial UAV project.
Nicholas Fuller for orchestrating the UWI-IBM internship.
Shamin Renwick for her guidance and support in culminat-
ing this paper.
Part of This report was prepared as an account of work
sponsored by an agency of the United States government.
Neither the United States government nor any agency
thereof, nor any of their employees, makes any warranty,
expressed or implied, or assumes any legal liability or
responsibility for the accuracy, completeness, or usefulness
of any information, apparatus, product, or process disclosed,
or represented that its use would not infringe privately owned
rights. Reference herein to any specific commercial product,
process, or service by trade name, trademark, manufacturer,
or otherwise does not necessarily constitute or imply its
endorsement, recommendation, or favoring by the United
States government or any agency thereof. The views and
opinions of authors expressed herein do not necessarily state
or reflect those of the United States government or any
agency thereof.
 R EFERENCES
[1] L. J. Klein, F. J. Marianno, C. M. Albrecht, M. Freitag, S. Lu,
N. Hinds, X. Shao, S. B. Rodriguez, and H. F. Hamann,
“PAIRS: A scalable geo-spatial data analytics platform,” in
2015 IEEE International Conference on Big Data (IEEE
BigData 2015). IEEE, 2015, pp. 1290–1298.
[2] D. R. Lyon, D. Zavala-Araiza, R. A. Alvarez, R. Harriss,
V. Palacios, X. Lan, R. Talbot, T. Lavoie, P. Shepson, T. I.
Yacovitch et al., “Constructing a spatially resolved methane
emission inventory for the barnett shale region,” Environmen-
tal Science & Technology, vol. 49, no. 13, pp. 8147–8157,
2015.
[3] W. Zhou, D. Nair, O. Gunawan, T. van Kessel, and H. F.
Hamann, “A testing platform for on-drone computation,”
in 2015 33rd IEEE International Conference on Computer
Design (ICCD). IEEE, 2015, pp. 732–735.
[14] A. Bürkle, F. Segor, and M. Kollmann, “Towards autonomous
micro uav swarms,” Journal of Intelligent & Robotic Systems,
vol. 61, no. 1, pp. 339–353, 2011. [Online]. Available:
http://dx.doi.org/10.1007/s10846-010-9492-x.
[15] G. Vásárhelyi, C. Virágh, G. Somorjai, N. Tarcai, T. Szörényi,
T. Nepusz, and T. Vicsek, “Outdoor flocking and formation
flight with autonomous aerial robots,” in 2014 IEEE/RSJ
International Conference on Intelligent Robots and Systems.
IEEE, 2014, pp. 3866–3873.
[16] T. Prevot, J. Rios, P. Kopardekar, J. E. Robinson III, M. John-
son, and J. Jung, “UAS traffic management (UTM) concept of
operations to safely enable low altitude flight operations,” in
16th AIAA Aviation Technology, Integration, and Operations
Conference. AIAA, 2016.

[4] (n.d.) Welcome to openCV documentation. OpenCV (Open

Source Computer Vision). Accessed Jul. 30, 2016. [Online].
Available: http://docs.opencv.org/2.4/index.html.

[5] (n.d.) GDAL - geospatial data abstraction library. Open

Source Geospatial Foundation OSGeo. Accessed Jul. 30,
2016. [Online]. Available: http://www.gdal.org.

[6] (n.d.) Discover QGIS. QGIS Development Team.

Accessed Jul. 30, 2016. [Online]. Available:
http://www.qgis.org/en/site/about/features.html.

[7] J. Zhang, L. Liu, B. Wang, X. Chen, Q. Wang, and T. Zheng,

“High speed automatic power line detection and tracking for
a uav-based inspection,” in 2012 International Conference on
Industrial Control and Electronics Engineering ICICEE, Aug
2012, pp. 266–269.

[8] A. P. Pacsi, N. S. Alhajeri, D. Zavala-Araiza, M. D. Webster,

and D. T. Allen, “Regional air quality impacts of increased
natural gas production and use in Texas,” Environmental
Science & Technology, vol. 47, no. 7, pp. 3521–3527, 2013.

[9] M. Brown and D. G. Lowe, “Automatic panoramic image

stitching using invariant features,” International Journal of
Computer Vision, vol. 74, no. 1, pp. 59–73, 2007.

[10] (n.d.) Pix4DmapperDiscovery. Pix4D. Accessed Jul. 30, 2016.

[Online]. Available: https://pix4d.com/product/pix4dmapper-
discovery.

[11] Quick Urban & Industrial Complex (QUIC) Dispersion

Modeling System. (n.d.). Los Alamos National
Laboratory. Accessed Jul. 29, 2016. [Online]. Available:
http://www.lanl.gov/projects/quic.

[12] A. J. Cimorelli, S. G. Perry, A. Venkatram, J. C. Weil,

R. J. Paine, and W. D. Peters, “AERMOD: Description of
model formulation,” Aerosp. Corp., Los Angeles, CA, Tech.
Rep. EPA-454/R-03-004, Sep. 2004. [Online]. Available:
http://www.epa.gov/scram001/7thconf/aermod/aermod mfd.pdf.

[13] Official CALPUFF modeling system. (n.d.). Exponent:

Engineering and Scientific Consulting. Accessed Jul. 29,
2016. [Online]. Available: http://www.src.com.

734