Robotics and Autonomous Systems 116 (2019) 80–97

Contents lists available at ScienceDirect

Robotics and Autonomous Systems

journal homepage: www.elsevier.com/locate/robot

A deep learning gated architecture for UGV navigation robust to sensor

failures
∗
Naman Patel, Anna Choromanska, Prashanth Krishnamurthy, Farshad Khorrami
Control/Robotics Research Laboratory (CRRL), Department of Electrical and Computer Engineering, NYU Tandon School of Engineering, 5 MetroTech
Center, USA

highlights

• An end-to-end learning system for UGV indoor corridor tracking using camera and LiDAR.
• A deep learning framework for sensor fusion that is robust to sensor failure.
• Gating based dropout regularization to enable robustness to various data corruptions.
• Experimental demonstration of the efficacy of the proposed system on our in-house UGV.

article info a b s t r a c t

Article history: In this paper, we introduce a novel methodology for fusing sensors and improving robustness to
Received 6 July 2018 sensor failures in end-to-end learning based autonomous navigation of ground vehicles in unknown
Received in revised form 22 January 2019 environments. We propose the first learning based camera–LiDAR fusion methodology for autonomous
Accepted 7 March 2019
in-door navigation. Specifically, we develop a multimodal end-to-end learning system, which maps raw
Available online 18 March 2019
depths and pixels from LiDAR and camera, respectively, to the steering commands. A novel gating based
Keywords: dropout regularization technique is introduced which effectively performs multimodal sensor fusion
Robustness to sensor failures and reliably predicts steering commands even in the presence of various sensor failures. The robustness
Deep learning for autonomous navigation of our network architecture is demonstrated by experimentally evaluating its ability to autonomously
Vision/LiDAR based navigation navigate in the indoor corridor environment. Specifically, we show through various empirical results
Learning from demonstration
that our framework is robust to sensor failures, partial image occlusions, modifications of the camera
Sensor fusion
image intensity, and the presence of noise in the camera or LiDAR range images. Furthermore, we show
Autonomous vehicles
that some aspects of obstacle avoidance are implicitly learned (while not being specifically trained for
it); these learned navigation capabilities are shown in ground vehicle navigation around static and
dynamic obstacles.
© 2019 Published by Elsevier B.V.

1. Introduction resilient to data corruption for self-driving vehicles equipped

There have been significant advances in machine learning
based approaches for robotic applications in recent years due
to the rapid developments in deep learning techniques. Deep
learning approaches have the ability to leverage large amounts of
labeled and contextually rich data to give desired outputs. Some
recent applications of deep learning that are relevant to this paper
include autonomous car driving systems [1–12].
In this work, we address the sensor fusion problem (Fig. 1)
in the context of developing end-to-end learning frameworks
∗ Corresponding author.
E-mail addresses: naman.patel@nyu.edu (N. Patel), ac5455@nyu.edu
(A. Choromanska), prashanth.krishnamurthy@nyu.edu (P. Krishnamurthy),
khorrami@nyu.edu (F. Khorrami).
with multiple sensors (mainly camera and LiDAR, although the
same methodology may be utilized for fusing a larger number of
sensors). Our previous work [7] deals with the problem of deep
learning based sensor fusion by modifying the training method-
ology to include cases wherein either of the two modalities was
randomly shut off to deal with the possibility of sensor failure.
This work is motivated by three primary objectives. The first
objective is to design an architecture which inherently performs
robust sensor fusion leveraging the complementary strengths of
the two sensors. The second objective is to design a system frame-
work, which is robust to corrupted data (e.g., partial occlusion,
noisy images, and varying intensity) without explicitly/separately
being trained for this purpose; thus, requiring less training data.
The third primary objective is to study the possibility of obtaining
better performance characteristics with the multimodal fused

https://doi.org/10.1016/j.robot.2019.03.001
0921-8890/© 2019 Published by Elsevier B.V.
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
Fig. 1. End-to-end modality fusion framework for learning autonomous
navigation policy in indoor environments.
system than with either sensor modality separately (e.g., see
Fig. 2).
Our work focuses on the problem of navigation of an au-
tonomous unmanned ground vehicle (UGV) using vision from a
monocular camera and depth measurements from a LiDAR in
an indoor environment with deep learning. A novel approach
to modality fusion is presented to generate steering commands
for autonomous navigation of a ground vehicle. The proposed
methodology naturally extends to the setting with multiple sen-
sors, where one or more sensor’s data might be corrupted.
Previous approaches for navigating in an unknown environ-
ment are based on classical simultaneous localization and map-
ping (SLAM) based frameworks where a 3D map is obtained using
photometric and depth information from sensors such as camera
and LiDAR and then path planning is performed on these recon-
structed maps. The limitations of these approaches are that the
reconstructed maps have to be accurate to navigate safely. Addi-
tionally, these approaches only use geometric information from
the map and nothing about the planning decision can be inferred
before the environment is explicitly observed, even when there
are obvious patterns. Thus, learning based approaches are a good
alternative as these frameworks learn the planning decisions
from past observations.
In this work, we propose a deep learning architecture for
the sensor fusion problem that consists of two convolutional
neural networks (CNNs), each processing one of the input modal-
ities, which are then fused with a dropout regularization aided
gating mechanism. The gating mechanism is realized as a fully-
connected network that is trained to generate environment-
appropriate scalar weights for LiDAR and camera using the CNN-
generated feature vectors. The scalar weights are then passed
through a dropout layer [13], which randomly makes the scalar
weights zero. These scalar weights are then utilized to obtain the
fused embedding including both modalities. The fused embedding
is then passed through additional network layers to generate the
steering command for the vehicle. The training of the proposed
network does not rely on the introduction of corrupted data in the
training batches (to mimic sensor failures) as introduced in our
previous work [7]. The novel aspects of this paper are as follows:
• an end-to-end learning system for the problem of indoor
corridor tracking with a ground vehicle registering camera
and LiDAR data,
• a new deep learning architecture for sensor fusion that
leads to a system for autonomous driving of ground vehicles
indoors that is robust to the presence of partial data from a
single modality,
• a gating based dropout regularization technique which ef-
fectively performs sensor fusion and makes the network
robust to various data corruptions without retraining (thus
reducing the required training data),
81
Fig. 2. Instances of undesirable behavior of the UGV when using only camera
or LiDAR for autonomous navigation.
• experimental demonstration of the efficacy of the proposed
system on our in-house developed ground vehicle that in-
cludes a real-time autopilot, single board computer with
graphics processing unit (GPU), and integrated camera and
LiDAR sensors.
The paper is organized as follows. Related literature is briefly
summarized in Section 2. The problem formulation is presented
in Section 3. The architectures for sensor fusion developed in this
paper are discussed in Section 5, including an architecture based
on a gating mechanism as well as two other architectures more
similar to prior literature (although in a different context than in
this paper). The training mechanisms are also discussed in Sec-
tion 5. Empirical verification studies are presented in Section 6.
Finally, concluding remarks are provided in Section 7.
2. Related work
Autonomous robot navigation in a variety of environments has
been extensively studied in prior works [14–17]. Many traditional
methods for autonomous navigation use geometric approaches
such as SLAM for path planning in both indoor and outdoor envi-
ronments using vision and depth based sensors, such as camera,
stereo camera, and LiDAR [18–20]. Vision processing for indoor
wall detection and corridor following has also been studied using
optical flow [21], visual landmark [22], and object recognition
based techniques [23,24]. Obstacle avoidance and navigation in
uncertain environments using these traditional approaches have
also been applied to aerial vehicles [25], humanoids [26,27], and
unmanned sea surface vehicles [28].
These traditional approaches use geometry of the environ-
ment without considering the semantic information about the
landmarks. These approaches only take the observed states into
consideration for autonomous navigation and do not use any
information learned from similar environments. They might also
fail to localize in environments such as long corridors with tex-
tureless walls, which are common in indoor environments. To
resolve these issues, learning based approaches are being used for
autonomous navigation. These learning based approaches use two
different paradigms to learn from data, namely imitation learning
and reinforcement learning.
An early work addressed end-to-end navigation of a car using
laser range finder and a camera [29] using a single-layer neural
network to process 30 × 32 camera images and 8 × 32 LiDAR
depth range images. Neural networks have been used for learning
driving decision rules (by extracting key indicators from sensor
data) for autonomous navigation using camera [30,31]. End-to-
end navigation of cars has been addressed using stereo camera [2]
as well as a single front-facing camera [3,4,32,33] (also, some
82 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
debugging tools were developed for these autonomous systems
to understand the visual cues that the network uses to produce
a steering command, e.g., [34]). Model predictive control based
approach which learn navigation from demonstration has also
been proposed [35]. Self-supervised learning methods have also
been introduced to learn autonomous navigation safely [1,9,36–
38] using stereo and monocular cameras. Learning based sensor
fusion techniques have also been proposed for navigation of a
ground vehicle in an indoor [7,8,39] as well as in simulation [40]
environment. Conditioned imitation learning [12] based naviga-
tion framework where the vehicle can be guided to move in
specific directions has also been introduced. These learning based
approaches have also been used for autonomous aerial vehicle
navigation [41–43]. It is to be noted the particular problem con-
sidered in this paper (i.e., learning based indoor navigation using
vision and LiDAR) has not been considered in prior literature.
Furthermore, unlike the prior approaches discussed above, the
primary focus in this paper is the development of a network
architecture and regularization and training methodologies to
achieve robustness to sensor noise/non-idealities including inter-
mittent loss of sensor modalities, occlusions, lighting variations,
and sensor noise.
Reinforcement learning techniques have also been utilized to
teach the mobile robot to avoid obstacles and navigate through
the corridor using sensors such as laser range finder [44]. Policy
search using reinforcement learning through an auxiliary task like
estimating depth from monocular images to estimate steering
directions was introduced in [45]. Deep reinforcement learn-
ing (DRL) based approaches addressing the problem of integrat-
ing path planning and learning have been presented in simu-
lated [46,47] as well as real-world environments [6]. DRL was
used to develop efficient policies for socially aware motion plan-
ning [48] enabling the vehicle to navigate safely and efficiently in
pedestrian-rich environments. Deep inverse reinforcement learn-
ing based approaches have also been proposed to learn cost
functions for path planning [10,11,49], which can be used to
refine manually handcrafted cost functions. Additionally, end-to-
end memory based mapping and planning frameworks different
from reactive approaches have been introduced recently [50–53].
In the deep learning literature, the fusion of different modal-
ities has been studied for various other applications (i.e., in con-
texts other than the ground vehicle application considered here)
in recent years such as in [54] for indoor scene recognition
and in [55] for object detection and segmentation using images
and their corresponding depth map representation. A framework
to embed point-cloud and natural language instruction combi-
nations and manipulation trajectories in the same semantically
meaningful space was introduced to perform end-to-end exper-
iments on a PR2 robot in [56]. A recurrent neural network [57]
was applied to implicitly learn the dependencies between RGB
images and depth map to perform semantic segmentation. In [58],
RGB image and its corresponding 3D point cloud are used as
inputs for 3D object detection. RGB image, optical flow, and
LiDAR range images are combined to form a six channel input
to a deep neural network for object detection [59]. Various other
multimodal learning frameworks have also been proposed for
object detection, recognition and segmentation [60–63].
Our proposed system introduces several novel aspects (out-
lined in the introduction) compared to the prior works sum-
marized above. Specifically, we introduce a new gating mecha-
nism based dropout regularization that enables modality fusion
for robust end-to-end learning of autonomous corridor driving.
The efficacy of the proposed approach is demonstrated through
experimental studies on our UGV platform (Fig. 3).
Fig. 3. Our unmanned ground vehicle system with integrated LiDAR (VLP-16)
and camera sensors (LI-OV5640).
3. Problem formulation
Our goal is to learn in an end-to-end manner, autonomous
navigation policies for a ground vehicle in an indoor environ-
ment using camera and LiDAR sensors. The visual RGB image
gives information about the texture and color of objects in the
nearby environment as well as the overall type of environment.
The depth range image provides geometric information which
complements visual information provided by the camera. Within
this context, we aim to explore effective deep neural network ar-
chitectures and novel training techniques for fusion of the camera
and LiDAR modalities to obtain policies robust to sensor failures
and data corruption as well as achieve performance superior
to what is achieved by either sensor separately (e.g., Fig. 2).
These policies are learned using data recorded under human
teleoperation of the UGV as described in detail in Section 4.
The aforementioned approach is applicable when multiple
cameras or LiDARs are utilized as well. It should be pointed out
that a single modality system will have less robustness to sensor
failures/corruptions than a multi-modal sensor system. Under a
single sensor scenario (either a camera or LiDAR), the control sys-
tem is more likely to fail when partial or complete sensor failure
is present. As illustrated in Fig. 2, each sensor separately can have
limitations in environment perception (e.g., lack of resolution,
blind spots, lighting conditions, or data corruption) causing poor
performance or collision of the vehicle. Additionally, generating
training dataset even under no sensor failure condition requires
a reasonable amount of training dataset. The NetGatedDropout
architecture presented here alleviates these issues.
Classical methods for navigation [14] are based on the SLAM
framework where sensory measurements from camera, LiDAR,
etc are fused using traditional filtering and estimation methods
to obtain vehicle pose which is used for path planning. As shown
in Fig. 4, maps based on geometry and no semantic information
(in textureless environment), using a framework such as ORB-
SLAM2 [64] and maps based on vision from camera, using a
framework such as ORB-SLAM [65] do not reconstruct an accurate
map of the environment. It is to be noted that the results shown
in Fig. 4 are when all the sensors used by the framework are
operating normally. The performance of these SLAM frameworks
will further degrade significantly during sensor failures or sensor
data corruptions (e.g., occlusions, sensor noise, etc.) which our
proposed framework is robust to. Additionally, from Fig. 4, it
is noticed that in challenging indoor environments with scarce
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
Fig. 4. Reconstructed ground vehicle trajectory using monocular ORB-SLAM
(yellow) [65] and ORB-SLAM2 (black) [64] (with camera and LiDAR) tracking
the center (blue) of a rectangular indoor corridor environment (anti-clockwise
trajectory) . (For interpretation of the references to color in this figure legend,
the reader is referred to the web version of this article.)
features, monocular SLAM frameworks such as ORB-SLAM are
ineffective and have to be replaced by frameworks using mul-
tiple modalities. Thus, sensor fusion is a crucial component for
autonomous navigation.
One approach in navigating in an unknown environment is
to plan a path based on a constructed map of the environment
and localize with respect to that map (i.e., simultaneous local-
ization and mapping — SLAM). A controller ensures tracking of
the desired path using current position of the robot obtained
from SLAM. However, in challenging environments where the
number of features is small (e.g., a long textureless hallway) or
varying (objects are moved around), typical SLAM algorithms do
not perform well (Fig. 4). Moreover, the control loop is entirely
based on perceived geometry. The decoupled state estimation and
execution of the control loop makes the system less adaptive due
to vulnerability to sensor failures and data corruption. Our work
addresses these issues by forming the estimation and control
problem as an end-to-end learning problem where both geometry
and semantic cues are simultaneously utilized.
4. Experimental testbed
The proposed system is implemented on our in-house inte-
grated UGV (Fig. 3) with differential drive and Tegra TX1 onboard
GPU for implementation of the deep networks. The vehicle is
equipped with Leopard Imaging LI-OV5640 camera running at
30 Hz (down sampled to 10 Hz with the same time stamp as the
LiDAR), Velodyne ‘Puck’ VLP-16 LiDAR running at 10 Hz, an in-
house designed autopilot that translates generated velocity and
heading commands (differential speed to left and right wheels)
to PWM signals driving the motor controllers commutating the
actual command signals to the motors for the given speed and
heading. The PWM signals for both speed and heading are gen-
erated by the autopilot with a period of 20 ms and an ON period
varying from 1 ms to 2 ms. The training dataset (camera image,
LiDAR range image and steering commands) is collected from
UGV while a human driver operates it (teleoperation).
The front and rear wheels are all driven independently and
the middle wheels are passive and carry encoders. Additional
sensors are built into the platform (i.e., inertial navigation units
and wheel encoders) but are not used in this paper. The outer-
loop commands generated by the end-to-end network driving
83
the motors are generated at a rate of 10 Hz. Both the velocity
command, v (kept constant – 0.5 m/s), and the network steering
command predictions, ω, lie in the interval [−100, 100] and are
mapped to their respective PWM ON periods (in seconds) as
shown in Eqs. (1) and (2).
v v
ton = 1.5 + 0.5 × (1)
100
ω ω
ton = 1.5 + 0.5 × . (2)
100
The mapping from differential PWM command to angular
velocity of the vehicle has its own dynamics due to power ampli-
fiers, motor dynamics, and gear train and ground friction. Hence,
the plots are in terms of the percentage of the differential PWM
from its neutral value (0% corresponds to no turn, i.e., moving
straight forward) given to the motors driving the wheels on the
opposite sides of the vehicle. For example, the vehicle’s angular
velocity corresponding to differential PWM commands (steering
commands) 4%, 8%, and 16% in steady state are 0.15, 0.56, and
1.89 rad/s.
5. Proposed sensor fusion framework
In this section, our proposed framework and the correspond-
ing deep learning based network architectures with their imple-
mentation details are presented.
5.1. Network architectures
Various network architectures are considered for the sensor
fusion task. The primary architecture (Fig. 5), denoted as Net-
GatedDropout, is a gating based architecture aided by dropout
regularization described further below. We also consider three
other architectures (denoted as NetEmb, NetConEmb and Net-
Gated) that are more similar to prior literature; these networks
are also described further below.
The architectures of NetEmb, NetConEmb, NetGated, and Net-
GatedDropout are described in Tables 1, 2, 3, and 4, respectively.
In NetEmb (which shares the same first 20 layers as NetConEmb,
NetGated, and NetGatedDropout), feature maps from RGB image
and LiDAR depth range image are extracted through a series of
convolution layers. Next, the features extracted from the convo-
lution layers in both the parallel networks are embedded into
a feature vector using a fully connected network. The intuition
behind embedding features is that the features extracted from
camera image and depth range image will have the same di-
mension. This ensures that one modality does not have a greater
effect on the result than the other due to unequal size. The
number of parameters in NetEmb are 4,754,357. In NetConEmb,
the convolution feature maps are passed into a fully connected
network. As shown in Table 1, the network architecture con-
sists of 8 convolution layers and 1 fully connected network for
each modality and 2 fully connected networks for information
fusion from the two modalities. Each convolution layer consists
of 3 × 3 kernels which convolve through the input with a stride
of 1 to generate feature maps which are then passed through
Rectification (ReLU) non-linearity. The inputs are padded during
convolution to preserve the spatial resolution. The feature maps
are downsampled after every two convolution layers by max-
pooling operation with a window size and stride of 2 × 2. All
hidden layers including the fully connected layers are equipped
with Rectification (ReLU) non-linearity. NetConEmb has 9,243,573
parameters. The network learns its parameters by minimizing the
Huber loss (δ = 1) between the predicted steering command and
the command of the human driver.
84 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97

Table 1
NetEmb: Deep learning based modality fusion architecture using embeddings. The left side of the table is for processing of the RGB image from the camera and
the right side of the table is for processing of the depth range image from the LiDAR. The feature vectors (of length 512) constructed from camera and LiDAR are
concatenated at layer 24.
Layer name Layer input (for Layer output Kernel Stride No. Layer name Layer input (for Layer output Kernel Stride No.
RGB image) size kernels LiDAR range size kernels
image)
1 Spatial 3 × 120 × 160 16 × 120 × 160 3 × 3 1 16 Spatial 1 × 900 × 16 16 × 900 × 16 3 × 3 1 16
convolution convolution
2 Rectified linear 16 × 120 × 160 16 × 120 × 160 – – – Rectified linear 16 × 900 × 16 16 × 900 × 16 – – –
unit unit
3 Spatial 16 × 120 × 160 16 × 120 × 160 3 × 3 1 16 Spatial 16 × 900 × 16 16 × 900 × 16 3 × 3 1 16
convolution convolution
4 Rectified linear 16 × 120 × 160 16 × 120 × 160 – – – Rectified linear 16 × 900 × 16 16 × 900 × 16 – – –
unit unit
5 Max pooling 16 × 120 × 160 16 × 60 × 80 2 × 2 2 – Max pooling 16 × 900 × 16 16 × 450 × 8 2 × 2 2 –
6 Spatial 16 × 60 × 80 32 × 60 × 80 3 × 3 1 32 Spatial 16 × 450 × 8 32 × 450 × 8 3 × 3 1 32
convolution convolution
7 Rectified linear 32 × 60 × 80 32 × 60 × 80 – – – Rectified linear 32 × 450 × 8 32 × 450 × 8 – – –
unit unit
8 Spatial 32 × 60 × 80 32 × 60 × 80 3 × 3 1 32 Spatial 32 × 450 × 8 32 × 450 × 8 3 × 3 1 32
convolution convolution
9 Rectified linear 32 × 60 × 80 32 × 60 × 80 – – – Rectified linear 32 × 450 × 8 32 × 450 × 8 – – –
unit unit
10 Max pooling 32 × 60 × 80 32 × 30 × 40 2 × 2 2 – Max pooling 32 × 450 × 8 32 × 225 × 4 2 × 2 2 –
11 Spatial 32 × 30 × 40 48 × 30 × 40 3 × 3 1 48 Spatial 32 × 225 × 4 48 × 225 × 4 3 × 3 1 48
convolution convolution
12 Rectified linear 48 × 30 × 40 48 × 30 × 40 – – – Rectified linear 48 × 225 × 4 48 × 225 × 4 – – –
unit unit
13 Spatial 48 × 30 × 40 48 × 30 × 40 3 × 3 1 48 Spatial 48 × 225 × 4 48 × 225 × 4 3 × 3 1 48
convolution convolution
14 Rectified linear 48 × 30 × 40 48 × 30 × 40 – – – Rectified linear 48 × 225 × 4 48 × 225 × 4 – – –
unit unit
15 Max pooling 48 × 30 × 40 48 × 15 × 20 2 × 2 2 – Max pooling 48 × 225 × 4 48 × 113 × 2 2 × 2 2 –
16 Spatial 48 × 15 × 20 64 × 15 × 20 3 × 3 1 64 Spatial 48 × 113 × 2 64 × 113 × 2 3 × 3 1 64
convolution convolution
17 Rectified linear 64 × 15 × 20 64 × 15 × 20 – – – Rectified linear 64 × 113 × 2 64 × 113 × 2 – – –
unit unit
18 Spatial 64 × 15 × 20 64 × 15 × 20 3 × 3 1 64 Spatial 64 × 113 × 2 64 × 113 × 2 3 × 3 1 64
convolution convolution
19 Rectified linear 64 × 15 × 20 64 × 15 × 20 – – – Rectified linear 64 × 113 × 2 64 × 113 × 2 – – –
unit unit
20 Max pooling 64 × 15 × 20 64 × 8 × 10 2 × 2 2 – Max pooling 64 × 113 × 2 64 × 57 × 1 2 × 2 2 –
21 Flatten 64 × 8 × 10 5120 – – – Flatten 64 × 57 × 1 3648 – – –
22 Fully connected 5120 512 – – – Fully connected 3648 512 – – –
23 Rectified linear 512 512 – – – Rectified linear 512 512 – – –
unit unit
24 Concatenate 512,512 1024 – – – – – – – – –
25 Fully connected 1024 32 – – – – – – – – –
26 Rectified linear 32 32 – – – – – – – – –
unit
27 Fully connected 32 10 – – – – – – – – –
28 Rectified linear 10 10 – – – – – – – – –
unit
29 Fully connected 10 1 – – – – – – – – –

Table 2
NetConEmb: Fusion architecture where the convolution feature maps are directly passed through
a fully connected network instead of first converting them into feature embeddings as done in
NetEmb. The first 20 layers and layers 25 to 29 are identical to NetEmb and the layers in bold are
the unique part of the network.
Layer name Layer input Layer output Layer name Layer input Layer output
1 .. 20 Same as Table 1
21 Flatten 64 × 8 × 10 5120 Flatten 64 × 57 × 1 3648
22 Concatenate 5120,3648 8768 – – –
23 Fully connected 8768 1024 – – –
24 Rectified linear unit 1024 1024 – – –
25 Fully connected 1024 32 – – –
26 Rectified linear unit 32 32 – – –
27 Fully connected 32 10 – – –
28 Rectified linear unit 10 10 – – –
29 Fully connected 10 1 – – –
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97 85

Table 3
NetGated: Fusion architecture with gating mechanism based on computing scalar weights from the feature embeddings and then
constructing a combination of the feature embeddings based on the scalar weights. The first 24 layers are identical to NetEmb and
layers in bold are the unique part of the architecture.
Layer name Layer input Layer output Layer name Layer input Layer output
1 .. 20 Same as Table 1
21 Flatten 64 × 8 × 10 5120 Flatten 64 × 57 × 1 3648
22 Fully connected 5120 512 Fully connected 3648 512
23 Rectified linear unit 512 512 Rectified linear unit 512 512
24 Concatenate 512,512 1024 – – –
25 Fully connected 1024 64 – – –
26 Rectified linear unit 64 64 – – –
27 Fully connected 64 2 – – –
28 Split 2 1,1 – – –
29 Multiplication with output 23 1 512 Multiplication with output 23 1 512
30 Addition 512,512 512 – – –
31 Fully connected 512 32 – – –
32 Rectified linear unit 32 32 – – –
33 Fully connected 32 1 – – –

Table 4
NetGatedDropout: Fusion architecture with gating mechanism based on computing scalar weights from the feature embeddings and
then constructing a combination of the feature embeddings based on the scalar weights. This network architecture is the same as
NetGated except that one additional layer (layer 28 shown in bold) is introduced.
Layer name Layer input Layer output Layer name Layer input Layer output
1 .. 20 Same as Table 1
21 Flatten 64 × 8 × 10 5120 Flatten 64 × 57 × 1 3648
22 Fully connected 5120 512 Fully connected 3648 512
23 Rectified linear unit 512 512 Rectified linear unit 512 512
24 Concatenate 512,512 1024 – – –
25 Fully connected 1024 64 – – –
26 Rectified linear unit 64 64 – – –
27 Fully connected 64 2 – – –
28 Dropout (with p = 0.5) 2 2 – – –
29 Split 2 1,1 – – –
30 Multiplication with output 23 1 512 Multiplication with output 23 1 512
31 Addition 512, 512 512 – – –
32 Fully connected 512 32 – – –
33 Rectified linear unit 32 32 – – –
34 Fully connected 32 1 – – –

In NetGated, the feature embedding architecture is similar to
NetEmb. The embeddings are passed through a gating network
to fuse the information from both the modalities to generate the
steering command. The gating network takes the two embeddings
obtained from RGB image and range image as input and outputs
the corresponding two weights. These weights are then used
to perform a weighted sum of the embeddings. This weighted
sum is then passed through two fully connected networks to
obtain the steering command. Each of the considered network
architectures is an end-to-end deep learning system that takes
an RGB image and a LiDAR depth range image as input and
fuses the modalities using a deep neural network to predict
the appropriate steering command of the ground vehicle for
autonomous navigation. NetGated has 4,802,945 parameters. In
NetGatedDropout, we add an extra dropout layer to the two
weights which randomly makes them zero during training. This
essentially regularizes the network by making the network not
dependent on necessarily having both the modalities to predict
the steering command and thus essentially results in the network
learning to perform robust sensor fusion. The proposed dropout
regularization is different from the usually utilized dropout. The
usually utilized dropout layer is based on Bernouilli distribution
whereas the proposed dropout regularization is based on gen-
eralized Bernoulli distribution where a configuration is selected
based on the given probability. The number of configurations (for
generalized Bernouilli distribution) are 2N − 1 for N sensors as
we do not include the configuration corresponding to when both
modalities are turned off.
5.2. Implementation and training
The inputs to the networks are the normalized RGB image with
a field of view of 72◦ and the LiDAR range image which is cropped
such that the front half with a field of view of 180◦ is visible.
Both the modalities are normalized by making each channel of
the modality in the training dataset zero mean with a standard
deviation of 1. At testing time, the mean and standard deviation
calculated during training are used to normalize the input.
To train the networks, camera and LiDAR datasets were ob-
tained by manually driving the vehicle (with constant speed)
through the corridor environment obtaining approximately the
same amount of training data for straight motion, left turns,
and right turns. The network was trained on a dataset of 14 456
images and the corresponding range images (around 24 min of
data). It was trained using Adagrad optimizer with a base learning
rate of 0.01. Bias terms for all the layers in the networks are
disabled. The network is implemented using the PyTorch frame-
work [66] and trained using an NVIDIA Titan X Pascal GPU based
workstation.
86 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
Fig. 5. NetGatedDropout: Our proposed end-to-end learning based architecture for fusion of camera and LiDAR sensors.
Our end-to-end learning framework learns to predict the ap-
propriate steering command by learning the weights of the net-
work which minimize the Huber loss between the predicted
steering commands and the recorded human steering commands.
We use the Huber loss instead of mean squared error since an
instability due to the divergence of the gradients was noted with
mean squared error loss. The Huber loss [67] for bounding box
regression and is given by
⎧
1 (namely NetEmb, NetConEmb, and NetGated as described in Ta-
⎨ (y − f (x))2 ,
⎪ for ∥y − f (x)∥ ≤ δ
L(y, f (x)) = 2 (3)
⎩ ∥y − f (x)∥ − 1 δ 2 ,
⎪
other w ise.
2 fusion architecture impartially, all the networks have similar
We follow the training procedure explained in our previous
work [8] for NetEmb, NetConEmb, and NetGated. To train the
network to utilize either sensors when available and also to be
robust to the possibility of sensor failure, the training of these
networks was performed in two stages. In the first stage of
training, the network is trained with the corresponding LiDAR
depth range images and camera RGB images for each time step
as inputs. In the second stage, the training of the network is
continued with corrupted data (i.e., with one modality shut down
to mimic sensor failure). Specifically, the network is trained with
60% corrupted data for each epoch out of which 50% of the data
is with the camera shut off (i.e., zero values for all elements in
the RGB image) and 50% is with the LiDAR shut off.
Due to the architecture of NetGatedDropout which consists of
the dropout layer, modality embeddings are randomly dropped
(to zero) which is similar to randomly shutting down one of the
networks. As a result, retraining is not required for the NetGated-
Dropout architecture and it learns to fuse modalities through
end-to-end learning. The probability of dropping either weight is
set to 50% for the dropout layer.
It is seen in Section 6 that the proposed network architecture
provides robust performance under sensor failures and various
data corruptions and implicitly learns to use the relevant infor-
mation from both modalities to generate steering commands. We
compare the NetGated and NetGatedDropout networks trained
only on the original dataset and the NetGated network retrained
with the corrupted dataset and show that the NetGatedDropout
network provides superior performance than both the original
NetGated network and the retrained NetGated network (retrained
with the corrupted dataset) when one of the sensor modalities
fails or is partially occluded or noisy. Also, the NetGatedDropout
network retains the performance of the original/retrained Net-
Gated networks when both the sensors are present.
6. Experimental studies
In this section, experimental results are presented for the de-
scribed architectures (NetEmb, NetConEmb, NetGated, NetGated-
Dropout), which are trained using both camera and LiDAR. We
also compare the performance of the original NetGated, Net-
Gated retrained with training procedure described in the previ-
ous section, and NetGatedDropout for various conditions of the
modalities (e.g., sensor data corruption).
6.1. Performance of different network architectures
To evaluate the performance of the original architectures
bles 1, 2, and 3, respectively), steering command predictions of
each network were compared with the steering commands of
a human operator. To evaluate the performance of the above
structure except for their respective fusion mechanisms. This
evaluation was done using a different dataset in a different cor-
ridor environment (test dataset) than the one used for training.
The results of each of the architectures compared to the human
operator are shown in Fig. 6 where the steering commands
given by a human operator (during teleoperation of the UGV)
are denoted as the ground truth. Additionally, Fig. 7 also shows
the error (between network steering command prediction and
ground truth) frequency plots of the steering command to better
interpret the performance of each architecture. The left and
right plot of both the figures show clockwise (right turns) and
counterclockwise (left turns) navigations of a complete floor of a
corridor environment.
As shown in Figs. 6 and 7, the utilization in NetEmb of an
equal-size embedding (constructed using a fully connected layer)
for each modality after the last convolution layer provides better
performance than NetConEmb as hypothesized in Section 5.1. The
NetEmb architecture performs better when one of the modalities
is switched off and also oscillates less compared to the Net-
ConEmb architecture. As discussed in Section 5.1, for NetConEmb,
the number of features for the camera after the last convolution
layer is much larger than for the LiDAR. This causes the output to
become more dependent on one modality resulting in unbalanced
fusion. As a result, the steering commands oscillate more, similar
to the behavior of the camera-only network. Motivated by the
observations above, fully connected layer based embeddings for
each modality were also used in the NetGated architecture. An
additional advantage of using an equal-size embedding for each
modality is that it is then easier and more natural to fuse the
embeddings by the learned gated weights by simply taking a
weighted linear combination. As shown in Fig. 6, the NetGated
architecture based network learns to move straight with fewer
oscillations than even the human operator. The fusion of camera
and LiDAR results in a smoother output than a LiDAR-only system
as shown in Fig. 6.
Since a desirable characteristic of motion in the indoor corri-
dor environment is that the ground vehicle should approximately
track the center of the corridor and should not come too close
to walls when turning, a useful metric for the performance of
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97 87

Fig. 6. Steering command predictions of NetEmb, NetConEmb, and NetGated architecture when both camera and LiDAR are working.

Fig. 7. Error frequency plots for steering commands of NetEmb, NetConEmb, and NetGated architecture when both camera and LiDAR are working.

the system is the distance of the vehicle to the left side and Table 5
Standard deviation of minimum distances to the wall for a clockwise trajectory
right side walls/objects. The closest distances on the left and
under fully autonomous mode.
right sides vary quite significantly even for an ‘‘ideal’’ motion
Network type Network Left wall distance Right wall distance
due to several objects such as trash cans, boxes, empty spaces input standard deviation standard deviation
and open office doors at some locations. To remove such ‘‘noise’’ (in m) (in m)
effects, an effective performance metric is the standard deviation 1 NetConEmb Camera and 0.48 0.43
(rather than mean) of distances to the left side and right side network LiDAR
2 NetEmb Camera and 0.38 0.35
walls/objects. In addition to standard deviation, Fig. 8 shows the network LiDAR
histogram plots of left and right distances (computed from LiDAR 3 NetGated Camera and 0.32 0.31
range images) for human teleoperation and different architecture. network LiDAR
4 Human Camera and 0.34 0.3
These standard deviations were recorded under fully
Teleoperation LiDAR
autonomous mode (i.e., with the network providing the com-
mands to the autopilot) with the different networks for both
clockwise and counterclockwise directions. The measured stan-
dard deviations for a clockwise motion through the building 6.2. Saliency map visualization
corridor environment (one complete floor of the building) are
shown in Table 5 and it is noted that the NetGated network A saliency analysis of the proposed framework was performed
architecture provides the best (lowest) standard deviation; a sim- to determine which parts of the camera image and LiDAR range
ilar observation was also noted for a counterclockwise motion. image were important for prediction. These salient features of the
NetGatedDropout performs similarly to NetGated in the absence image can be visualized as a grayscale image (as shown in Figs. 9
of sensor noise/dropouts. The performance of NetGatedDropout and 10) with the same dimension as the input. For both camera
image and LiDAR range image, saliency maps are shown in Figs. 9
is shown in Table 6.
and 10. The brightness of each pixel of the saliency map is directly
For all the considered network architectures, it is noted that
proportional to its importance in determining the output of the
the system trained on a dataset with both camera and LiDAR
network.
data is not directly robust to the possibility of a sensor failing
The saliency map for a particular input is computed through
(i.e., only one sensor modality available and the other zeroed
gradient of the output with respect to the input as it determines
out). For example, NetGated places much more trust on the the change in output value with respect to the change in input.
LiDAR input than on the camera input and does not provide any The saliency maps were computed based on the guided back-
reasonable performance in the event of a LiDAR failure. Hence, in propagation method [68,69] where during backpropagation for
order to achieve robustness to sensor failure, we introduce the computing gradients, only positive gradients for positive acti-
training strategy described in Section 5.2 to continue retraining vations are propagated. Thus, to compute the saliency map for
of the network with corrupted data generated by synthetically each input, the ReLU activations are determined in the forward
turning off either of the two modalities. The performance of the pass and positive gradients are determined during the back-
retrained NetGated network (after retraining with this corrupted propagation. Next, using these positive gradients and positive
data based technique) is compared in Section 6.3 with the original activation as switches in backpropagation, the gradient of the
trained NetGated network and the human operator. output with respect to the inputs are determined. These gradients
88 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
Fig. 8. Histogram of distances of the ground vehicle from the left wall and right wall for trajectories of different networks and human teleoperation in the corridor
environment.
Fig. 9. Saliency map visualization of activations of the NetGatedDropout architecture for various camera images. We have camera image inputs for the UGV in
various scenarios on the top and their corresponding saliency maps in the bottom.
are visualized by normalizing it through subtracting and dividing
the minimum and maximum gradient value from each pixel.
In the top row of Fig. 14, we have the camera image inputs
for the UGV in various scenarios and the corresponding activation
saliency maps in the bottom. The most salient parts of the images
are near the intersection of walls and floors and edges of objects
in the corridor environment like trashcans and doors. The LiDAR
range images (during straight, left and right turns) and their
corresponding saliency maps are shown in Fig. 15 in the left and
right column respectively. As shown in the figure, the salient
parts of the range image are around the wall and obstacles near
the UGV. Thus, from Figs. 14 and 15 it can be inferred that the
proposed framework intuitively attends to nearby walls, floor and
obstacles to decide the output.
6.3. Performance of networks during data corruption
We compare the performance of the original NetGated archi-
tecture, the NetGated architecture when retrained with corrupted
data as explained in 5.2, and the NetGatedDropout architecture
for various conditions of sensor failures as described below.
6.3.1. Either modality turned off
As shown in Figs. 11 and 12, when both camera and LiDAR
are working, all the network architectures perform well and
have very similar performance; but, when one of the modalities
is shut off, the original NetGated architecture gives erroneous
predictions. In Fig. 11, the predictions of the various architectural
frameworks when only camera is working, only LiDAR is working,
and when both sensor modalities are working are shown and
compared with ground truth. Fig. 12 shows the error frequency
plots for the above conditions. We simulate the conditions of
camera or LiDAR being shut off by turning all the pixels of that
modality to zero. The left and right plot of both the figures
show clockwise (right turns) and counterclockwise (left turns)
navigations of a complete floor of a corridor environment.
The performance characteristics (in terms of trajectory tra-
versed) of the retrained NetGated network and NetGatedDropout
were also evaluated (under the possibilities of both camera and
LiDAR available, only camera available, and only LiDAR available)
using the distance standard deviation based metric as discussed
above under fully autonomous operation of the UGV (Fig. 13). It is
noted in Table 6 that both the NetGatedDropout and the retrained
NetGated network achieve autonomous navigation through the
corridor although the camera-only and LiDAR-only situations pro-
vide lower performance (i.e., higher distance standard deviation)
than the camera + LiDAR situation. We also observe that Net-
GatedDropout has lower standard deviation than the retrained
NetGated network in all the cases.
We also compare the predictions of all three networks (Net-
Gated, NetGatedDropout, and retrained NetGated) with the hu-
man teleoperation based ground truths under the conditions of
only camera working, only LiDAR working, and both modalities
working using root mean squared error and discretized accu-
racy metric. As discussed before, all three network architectures
predict the duty cycle of the PWM signal, which in turn is con-
verted to the steering command. To compare ‘‘correctness’’ of
the predicted outputs, we discretize the outputs as described
below. We set a threshold of 10% to differentiate between straight
movement, left turn, and right turn. A duty cycle between −10%
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97 89

Fig. 10. Saliency map visualization of the LiDAR range image for the activations of the NetGatedDropout network. We have the LiDAR range image input for the
UGV in three different scenarios on the left and their corresponding saliency maps on the right.

Fig. 11. Steering command predictions using the NetGated and NetGatedDropout architectures trained only with the camera + LiDAR dataset and the NetGated
retrained with corrupted data under cases of only camera working (top), only LiDAR working (middle), and both camera and LiDAR working (bottom).

and 10% corresponds to the ground vehicle moving straight. If
the duty cycle is greater than 10%, then it is equivalent to a right
turn and if the duty cycle is less than −10%, it is equivalent to a
left turn. Thus, by using the above methodology, the predictions
and ground truth are discretized into left/straight/right. The dis-
cretized accuracy metrics of all three networks on the test dataset
are shown in Table 7. The root mean squared error between
the network predictions and human teleoperation ground truth
for the same trajectory are shown in Table 8. It is observed
from the tables that NetGatedDropout performs better than both
the retrained NetGated and the original NetGated architecture.
The NetGated architecture fails to predict the steering command
when either of the modalities is not present.
6.3.2. Partial occlusion
One of the most common instances of failures in vision and
LiDAR-based navigation is when there are partial occlusions of
camera or LiDAR range image. Our framework takes into consid-
eration performance of the network when either of the modalities
is partially occluded. We show that our proposed framework pro-
vides robustness to partial occlusion of the sensor data without
ever being specifically trained for it.
For a camera image, various types of partial occlusions were
considered and some of the worst-case scenarios are shown in the
last two images of the bottom row of Fig. 14. To test our network,
we selected a part of the camera image and made the values of
the pixels in that part zero. The steering command predictions
compared to the ground truth on a test dataset in a different
corridor environment for the three networks (NetGated, retrained
NetGated, and NetGatedDropout) are shown in the top row of
90 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97

Fig. 12. Error frequency plots for steering commands of the NetGated and NetGatedDropout architectures trained only with the camera + LiDAR dataset and the
NetGated retrained with corrupted data under cases of only camera working (top), only LiDAR working (middle), and both camera and LiDAR working (bottom).

Fig. 13. Distances of the ground vehicle from the left wall and right wall (in m) for counterclockwise (left turns) navigations in the corridor environment for
NetGatedDropout architecture.

Fig. 16. The top row of Fig. 17 shows the error frequency metric
for the same. As shown in the figures, the NetGatedDropout
architecture performs the best out of the three architectures.
The performance of the three architectures was also tested
for partially occluded LiDAR range image and some of the test
cases of occlusion are shown in the middle row of Fig. 15. Partial
occlusion is simulated for a LiDAR range image during testing by
making the pixels of a part of the range image zero. We observe
from the graphs in the bottom row of Fig. 16 and error frequency
graph in Fig. 17 that both NetGated and the retrained NetGated
fail to give accurate steering command predictions on the test
dataset and only NetGatedDropout is able to correctly predict
steering commands.
To empirically verify the results, we also compare the predic-
tions of all three networks with the ground truth from human
teleoperation using discretized accuracy metric and root mean
squared error metric as explained in the previous subsection. The
results are shown in Tables 7 and 8 from which we can ascertain
that the NetGatedDropout architecture performs better than both
the retrained NetGated and original NetGated architecture when
the camera image or the range image is partially occluded.
6.3.3. Camera image with varying image formations
As shown in the top row of Fig. 14, the pixel intensities of the
camera image are varied by changing brightness, contrast, and
saturation. The brightness of the image is varied by alpha blend-
ing the original camera image with an image with all its pixels
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97 91

Fig. 14. Examples of various types of data corruption for a camera image. In the top row, we have the original camera image, image with brightness changed, image
with varying contrast, and image with varying saturation, respectively. In the bottom row, we have the image with additive random noise, multiplicative random
noise, and partially occluded images in different directions, respectively.

Fig. 15. Examples of various types of data corruption for a LiDAR range image. In the top row, we have the original LiDAR range image. We have partially occluded
LiDAR range image in various directions in the second row and image with additive random noise and multiplicative random noise in the bottom row.

Fig. 16. Steering command predictions using the different network architectures when camera images are partially occluded (top) and when LiDAR range images
are partially occluded (bottom).

zero. The saturation of the image is changed by alpha blending the
original camera image with its own grayscale image. The contrast
of the image is modified by alpha blending the original camera
image with an image whose pixel values are equal to the mean
of its own grayscale image. Alpha used for alpha blending or alpha
compositing is randomly generated between 0 and 1 for each
camera image.
92 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97

Fig. 17. Error frequency plots for steering commands of the different network architectures when camera images are partially occluded (top) and when LiDAR range
images are partially occluded (bottom).

Fig. 18. Steering command predictions using the different network architectures when camera brightness is changed.

Fig. 19. Error frequency plots for steering commands of the different network architectures when camera brightness is changed.

The performance of each network is shown in Fig. 18 and
error frequency plot in Fig. 19, when the brightness of the camera
image is modified. As observed from the figures, we see that
the original NetGated network gives oscillating and inaccurate
steering command predictions. The retrained NetGated performs
better than the original NetGated but does not perform as well
as the NetGatedDropout network. Similar results were observed
when contrast or saturation was varied.
6.3.4. Data corruption by random noise
Inaccurate predictions due to image noise are one of the
prevalent issues for any vision-based method. The noise is usually
caused by problems in the electronic circuitry of the sensors. We
test the proposed network for various types of noise, namely
additive random noise, multiplicative random noise, and salt and
pepper noise.
For the camera image, we generate an image with noise am-
plitude of half the maximum intensity and add it to the original
image, resulting in an image such as the first image in the bottom
row of Fig. 14. We also test our network on images whose pixels
are multiplied by random numbers generated between 0 and 1.
An example of this multiplicative random noise is shown in the
second image in the bottom row of Fig. 14. The other type of noise
that we experimented with is the salt and pepper noise wherein
randomly selected sets of pixels in the image are made black or
white.
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97 93

Fig. 20. Steering command predictions using the different network architectures when random noise is added to the camera images (top) and when random noise
is added to LiDAR range images (bottom).

Fig. 21. Error frequency plots for steering commands of the different network architectures when random noise is added to the camera images (top) and when
random noise is added to LiDAR range images (bottom).

Similar to the generation of the noisy camera images, three
different types of noisy LiDAR images were generated: (1) by
adding random noise with amplitude of half the maximum range
value to each pixel of the range image, (2) with multiplica-
tive random noise, (3) with salt and pepper noise. The random
noise in 2nd and 3rd cases are generated following the same
procedures as for generating camera images with multiplicative
random noise and with salt and pepper noise, respectively.
The performance of the three architectures (NetGated, re-
trained NetGated, and NetGatedDropout) when the camera and
LiDAR images are corrupted with additive random noise is shown
in Fig. 20 (top and bottom rows, respectively). The error fre-
quency plots for the same, are shown in top and bottom rows
of Fig. 21 for camera and LiDAR range images, respectively. It
is observed from both the figures that the NetGatedDropout
architecture performs the best out of all the considered archi-
tectures. This observation is empirically verified by comparing
the predictions of all three networks with the ground truth using
discretized accuracy metric as explained in the previous subsec-
tion and root mean square error metric. The results are shown in
Tables 7 and 8 from which we can validate that the NetGated-
Dropout architecture provides the best results in both the cases
of the noisy camera images and the noisy LiDAR range images.
Similar results were observed for the cases when the camera and
LiDAR range images were corrupted with multiplicative random
noise and salt and pepper noise.
94 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97

Table 6
Standard deviation of minimum distances to the walls using the retrained Net-
Gated and NetGatedDropout architectures when various modalities are turned
off for a counterclockwise trajectory.
Network type Network Left wall distance Right wall distance
input standard deviation standard deviation
(in m) (in m)
1 NetGated Camera 0.43 0.39
(Retrained)
architecture
2 NetGated- Camera 0.33 0.36
Dropout
architecture
3 NetGated LiDAR 0.37 0.32
(Retrained)
architecture
4 NetGated- LiDAR 0.31 0.32
Dropout
architecture
5 NetGated Camera and 0.33 0.32
(Retrained) LiDAR
architecture
6 NetGated- Camera and 0.31 0.3
Dropout LiDAR
architecture
Fig. 22. Example trajectories of a UGV autonomously navigating in an indoor
environment: left turn (top row), straight motion (middle row), right turn
(bottom row). These pictures were taken from behind the UGV.
6.4. Autonomous indoor navigation of the ground vehicle

The NetGatedDropout architecture based framework is able
to successfully navigate through the indoor corridor environ-
ment and is robust to sensor failures and various other data
corruptions. Autonomous navigation through corridors is shown
in Fig. 22. The ground vehicle is able to appropriately make
turns at corners enabling it to be equidistant from the walls
after the turn. It is also able to navigate through narrower spaces
(e.g., between trash cans) as shown in the middle two rows of
Fig. 22.
Furthermore, the system implicitly learns to avoid static as
well as dynamic obstacles as shown in Fig. 23 without ever being
specifically trained for this purpose, i.e., the training dataset did
not include any specific demonstrations of moving around obsta-
cles. Also, the fused camera + LiDAR network performs better in
several scenarios than the camera-only or LiDAR-only situations.
While a LiDAR-only network can enable avoiding of obstacles
such as humans, it does not typically detect small (low-profile)
objects since these register only a few points in the LiDAR scan.
In such situations, the camera image enables the fused network
to avoid the obstacle. When approaching a visually featureless
wall, a camera-only system cannot disambiguate between left
and right turns while the LiDAR enables the fused network to
detect the appropriate turn. When passing an open door or other
open spaces (such as a short corridor leading to a dead end),
the LiDAR being a more geometric sensor measuring distances
to points tends to make a LiDAR-only system move towards the
open space. However, the visual features implicitly detected from
the camera enable the fused network to completely ignore such
an ‘‘unintended’’ open space and remain at the center of the
corridor (Fig. 24).
7. Conclusion
A convolutional neural network based architecture was intro-
duced for fusing vision and depth measurements from camera
and LiDAR, respectively, for learning an autonomous navigation
policy for a ground robot operating in an indoor environment.
Multiple network architectures were considered including a novel
dropout regularization aided gating based network architecture.
This architecture enables the ground vehicle navigation to be
robust to the possibility of sensor failure or data corruption and to

Table 7
Comparison of accuracy metric values for the steering command predictions of NetGated, NetGated retrained by randomly switching off modalities, and
NetGatedDropout with the steering command ground truth (UGV driven by human operator).
Network type Both modalities Camera LiDAR additive Camera only LiDAR only Camera image LiDAR range
(in %) additive noise noise (in %) (in %) (in %) partially image partially
(in %) occluded (in %) occluded (in %)
NetGated 92.25 88.83 86.93 8.81 1.76 72.57 31.42
NetGated (Retrained) 92.24 88.85 80.83 91.84 88.09 90.09 45.39
NetGatedDropout 93.65 93.40 86.95 95.234 90.235 94.22 92.89

Table 8
Comparison of root mean squared error metric values for the steering command predictions of NetGated, NetGated retrained by randomly switching off modalities,
and NetGatedDropout with the steering command ground truth (UGV driven by human operator).
Network type Both modalities Camera LiDAR Camera only LiDAR only Camera image LiDAR range image
(in PWM %) additive noise additive noise (in PWM %) (in PWM %) partially occluded partially occluded
(in PWM %) (in PWM %) (in PWM %) (in PWM %)
NetGated 4.71 5.28 4.96 14.41 25.04 10.16 15.56
NetGated (Retrained) 4.88 5.40 5.54 5.35 8.24 7.02 13.35
NetGatedDropout 4.40 4.63 5.39 4.67 5.85 4.88 4.59
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
Fig. 23. Examples of trajectories of the UGV autonomously navigating in the
presence of static (top row) and dynamic obstacles (bottom row). These pictures
were taken from behind the UGV.
Fig. 24. Trajectory comparison of LiDAR-only network and fused network when
passing through a corridor with misleading open spaces (e.g., open doors).
Ground vehicle navigating with LiDAR-only network shows a tendency to deviate
(from the center of the corridor) towards open doors or other open spaces as
compared to the fusion network.
properly leverage the complementary strengths of the two sen-
sors so as to achieve better performance than with either sensor
separately. It was experimentally demonstrated that the pro-
posed deep learning based system is able to fully autonomously
navigate in the indoor environment with robustness to the failure
of either the camera or the LiDAR and to various types of sensor
data corruption.
Acknowledgment
This work was funded in part by ONR grant number N00014-
15-12-182.
References
[1] R. Hadsell, P. Sermanet, J. Ben, A. Erkan, M. Scoffier, K. Kavukcuoglu,
U. Muller, Y. LeCun, Learning long-range vision for autonomous off-road
driving, J. Field Robotics 26 (2) (2009) 120–144.
[2] Y. LeCun, U. Muller, J. Ben, E. Cosatto, B. Flepp, Off-road obstacle avoidance
through end-to-end learning, in: Proceedings of the Advances in Neural
Information Processing Systems, Vancouver, Canada, 2006, pp. 739–746.
[3] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L.D.
Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, K. Zieba, End to
end learning for self-driving cars, CoRR, abs/1604.07316, 2016.
[4] H. Xu, Y. Gao, F. Yu, T. Darrell, End-to-end learning of driving models
from large-scale video datasets, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, 2017, pp.
3530–3538.
[5] N. Patel, P. Krishnamurthy, F. Khorrami, Semantic segmentation guided
SLAM using vision and LIDAR, in: Proceedings of the 50th International
Symposium on Robotics, Munich, Germany, 2018, pp. 352–358.
[6] J. Bruce, N. Suenderhauf, P. Mirowski, R. Hadsell, M. Milford, One-shot
reinforcement learning for robot navigation with interactive replay, in:
Proceedings of the NIPS Workshop on Acting and Interacting in the Real
World: Challenges in Robot Learning, Long Beach, USA, 2017.
[7] N. Patel, A. Choromanska, P. Krishnamurthy, F. Khorrami, Sensor modality
fusion with CNNs for UGV autonomous driving in indoor environments, in:
Proceedings of the IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), Vancouver, Canada, 2017, pp. 1531–1536.
[8] N. Patel, P. Krishnamurthy, Y. Fang, F. Khorrami, Reducing operator
workload for indoor navigation of autonomous robots via multimodal
sensor fusion, in: Proceedings of the Companion of the 2017 ACM/IEEE
International Conference on Human-Robot Interaction, Vienna, Austria,
2017, pp. 253–254.
95
[9] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, S. Levine, Self-supervised deep
reinforcement learning with generalized computation graphs for robot
navigation, in: Proceedings of the NIPS Workshop on Acting and Interacting
in the Real World: Challenges in Robot Learning, Long Beach, USA, 2017.
[10] M. Wulfmeier, D.Z. Wang, I. Posner, Watch this: Scalable cost-function
learning for path planning in urban environments, in: Proceedings of the
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
Daejeon, South Korea, 2016, pp. 2089–2095.
[11] M. Wulfmeier, D. Rao, D.Z. Wang, P. Ondruska, I. Posner, Large-scale cost
function learning for path planning using deep inverse reinforcement
learning, Int. J. Robot. Res. 36 (10) (2017) 1073–1087.
[12] F. Codevilla, M. Müller, A. Dosovitskiy, A. López, V. Koltun, End-to-end
driving via conditional imitation learning, CoRR, abs/1710.02410, 2017.
[13] N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov,
Dropout: a simple way to prevent neural networks from overfitting, J.
Mach. Learn. Res. 15 (1) (2014) 1929–1958.
[14] S. Thrun, W. Burgard, D. Fox, Probabilistic Robotics (Intelligent Robotics
and Autonomous Agents), The MIT Press, 2005.
[15] G.N. DeSouza, A.C. Kak, Vision for mobile robot navigation: A survey, IEEE
Trans. Pattern Anal. Mach. Intell. 24 (2) (2002) 237–267.
[16] B.D. Argall, S. Chernova, M. Veloso, B. Browning, A survey of robot learning
from demonstration, Robot. Auton. Syst. 57 (5) (2009) 469–483.
[17] B. Paden, M. Čáp, S.Z. Yong, D. Yershov, E. Frazzoli, A survey of motion
planning and control techniques for self-driving urban vehicles, IEEE Trans.
Intell. Veh. 1 (1) (2016) 33–55.
[18] C. Thorpe, M.H. Hebert, T. Kanade, S.A. Shafer, Vision and navigation for
the Carnegie-Mellon Navlab, IEEE Trans. Pattern Anal. Mach. Intell. 10 (3)
(1988) 362–373.
[19] C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. Clark, J. Dolan,
D. Duggins, T. Galatali, C. Geyer, et al., Autonomous driving in urban
environments: Boss and the urban challenge, J. Field Robotics 25 (8) (2008)
425–466.
[20] J.J. Leonard, H.F. Durrant-Whyte, Simultaneous map building and local-
ization for an autonomous mobile robot, in: Proceedings of the IEEE/RSJ
International Workshop on Intelligent Robots and Systems, Osaka, Japan,
1991, pp. 1442–1447.
[21] A. Dev, B.J.A. Kröse, F.C.A. Groen, Navigation of a mobile robot on the
temporal development of the optic flow, in: Proceedings of the IEEE/RSJ In-
ternational Conference on Intelligent Robots and Systems (IROS), Grenoble,
France, 1997, pp. 558–563.
[22] S. Se, D. Lowe, J. Little, Mobile robot localization and mapping with
uncertainty using scale-invariant visual landmarks, Int. J. Robot. Res. 21
(8) (2002) 735–758.
[23] Z. Zheng, X. He, J. Weng, Approaching camera-based real-world navigation
using object recognition, Procedia Comput. Sci. 53 (2015) 428–436.
[24] H. Cheng, H. Chen, Y. Liu, Topological indoor localization and navigation
for autonomous mobile robot, IEEE Trans. Autom. Sci. Eng. 12 (2) (2015)
729–738.
[25] S. Shen, N. Michael, V. Kumar, Autonomous multi-floor indoor navigation
with a computationally constrained mav, in: Proceedings of the IEEE
International Conference on Robotics and Automation (ICRA), Shanghai,
China, 2011, pp. 20–25.
[26] P. Krishnamurthy, F. Khorrami, GODZILA: A low-resource algorithm for
path planning in unknown environments, J. Intell. Robot. Syst. 48 (3)
(2007) 357–373.
[27] G. Brooks, P. Krishnamurthy, F. Khorrami, Humanoid robot navigation and
obstacle avoidance in unknown environments, in: Proceedings of the Asian
Control Conference, Washington, DC, USA, 2013, pp. 1–6.
[28] P. Krishnamurthy, F. Khorrami, A hierarchical control and obstacle avoid-
ance system for Unmanned Sea Surface Vehicles, in: Proceedings of the
IEEE Conference on Decision and Control/ European Control Conference,
Orlando, USA, 2011, pp. 2070–2075.
[29] D. Pomerleau, Alvinn: An autonomous land vehicle in a neural network,
in: Proceedings of the Advances in Neural Information Processing Systems,
Denver, USA, 1988, pp. 305–313.
[30] D. Silver, J.A. Bagnell, A. Stentz, Learning from demonstration for au-
tonomous navigation in complex unstructured terrain, Int. J. Robot. Res.
29 (12) (2010) 1565–1592.
[31] C. Chen, A. Seff, A. Kornhauser, J. Xiao, Deepdriving: Learning affordance
for direct perception in autonomous driving, in: Proceedings of the IEEE
International Conference on Computer Vision (ICCV), Santiago, Chile, 2015,
pp. 2722–2730.
[32] V.N. Murali, S.T. Birchfield, Autonomous navigation and mapping us-
ing monocular low-resolution grayscale vision, in: Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Workshops, Anchorage, USA, 2008, pp. 1–8.
96 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
[33] I. Kostavelis, A. Gasteratos, Learning spatially semantic representations for
cognitive robot navigation, Robot. Auton. Syst. 61 (12) (2013) 1460–1475.
[34] M. Bojarski, A. Choromanska, K. Choromanski, B. Firner, L.D. Jackel, U.
Muller, K. Zieba, Visualbackprop: visualizing CNNs for autonomous driving,
CoRR, abs/1611.05418, 2016.
[35] S. Lefevre, A. Carvalho, F. Borrelli, A learning-based framework for velocity
control in autonomous driving, IEEE Trans. Autom. Sci. Eng. 13 (1) (2016)
32–42.
[36] C. Richter, N. Roy, Learning to plan for visibility in navigation of un-
known environments, in: Proceedings of the International Symposium on
Experimental Robotics, Springer, 2016, pp. 387–398.
[37] C. Richter, N. Roy, Safe visual navigation via deep learning and novelty
detection, in: Proceedings of the Robotics: Science and Systems Conference,
Cambridge, Massachusetts, 2017.
[38] A. Dairi, F. Harrou, M. Senouci, Y. Sun, Unsupervised obstacle detection
in driving environments using deep-learning-based stereovision, Robot.
Auton. Syst. 100 (2018) 287–301.
[39] H.F.M. Zaki, F. Shafait, A.S. Mian, Learning a deeply supervised multi-modal
RGB-D embedding for semantic scene and object category recognition,
Robot. Auton. Syst. 92 (2017) 41–52.
[40] G. Liu, A. Siravuru, S. Prabhakar, M.M. Veloso, G. Kantor, Learning end-to-
end multimodal sensor policies for autonomous navigation, in: Proceedings
of the 1st Annual Conference on Robot Learning (CoRL), in: ser. Proceedings
of Machine Learning Research, vol. 78, PMLR, California, USA, 2017, pp.
249–261.
[41] A. Giusti, J. Guzzi, D.C. Cireşan, F.-L. He, J.P. Rodríguez, F. Fontana, M.
Faessler, C. Forster, J. Schmidhuber, G. Di Caro, et al., A machine learning
approach to visual perception of forest trails for mobile robots, IEEE Robot.
Automat. Lett. 1 (2) (2016) 661–667.
[42] F. Sadeghi, S. Levine, CAD2RL: Real single-image flight without a single real
image, in: Proceedings of the Robotics: Science and Systems Conference,
Cambridge, Massachusetts, 2017.
[43] D. Gandhi, L. Pinto, A. Gupta, Learning to fly by crashing, in: Proceedings
of the IEEE/RSJ International Conference on Intelligent Robots and Systems
(IROS), Vancouver, Canada, 2017, pp. 3948–3955.
[44] W.D. Smart, L.P. Kaelbling, Effective reinforcement learning for mobile
robots, in: Proceedings of the IEEE International Conference on Robotics
and Automation (ICRA), Washington, DC, USA, 2002, pp. 3404–3410.
[45] J. Michels, A. Saxena, A.Y. Ng, High speed obstacle avoidance using
monocular vision and reinforcement learning, in: Proceedings of the
International Conference on Machine Learning (ICML), in: ser. ACM In-
ternational Conference Proceeding Series, vol. 119, ACM, Bonn, Germany,
2005, pp. 593–600.
[46] Y. Zhu, R. Mottaghi, E. Kolve, J.J. Lim, A. Gupta, L. Fei-Fei, A. Farhadi,
Target-driven visual navigation in indoor scenes using deep reinforcement
learning, in: Proceedings of the IEEE International Conference on Robotics
and Automation (ICRA), Singapore, Singapore, 2017, pp. 3357–3364.
[47] W. Gao, D. Hsu, W.S. Lee, S. Shen, K. Subramanian, Intention-net: Integrat-
ing planning and deep learning for goal-directed autonomous navigation,
in: Proceedings of the 1st Annual Conference on Robot Learning (CoRL), in:
ser. Proceedings of Machine Learning Research, vol. 78, PMLR, California,
USA, 2017, pp. 185–194.
[48] Y.F. Chen, M. Everett, M. Liu, J.P. How, Socially aware motion planning with
deep reinforcement learning, in: Proceedings of the IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada,
2017, pp. 1343–1350.
[49] C. Xia, A.E. Kamel, Neural inverse reinforcement learning in autonomous
navigation, Robot. Auton. Syst. 84 (2016) 1–14.
[50] A. Khan, C. Zhang, N. Atanasov, K. Karydis, V. Kumar, D.D. Lee, Mem-
ory augmented control networks, in: Proceedings of the International
Conference on Learning Representations (ICLR), 2018.
[51] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, J. Malik, Cognitive mapping
and planning for visual navigation, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2017.
[52] J. Zhang, L. Tai, J. Boedecker, W. Burgard, M. Liu, Neural SLAM, CoRR,
abs/1706.09520, 2017.
[53] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, F. Shentu, E.
Shelhamer, J. Malik, A.A. Efros, T. Darrell, Zero-shot visual imitation, in:
International Conference on Learning Representations (ICLR), 2018.
[54] H. Zhu, J.-B. Weibel, S. Lu, Discriminative multi-modal feature fusion for
RGBD indoor scene recognition, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 2016,
pp. 2969–2976.
[55] S. Gupta, R. Girshick, P. Arbeláez, J. Malik, Learning rich features from RGB-
D images for object detection and segmentation, in: Proceedings of the
European Conference on Computer Vision, Springer, Zurich, Switzerland,
2014, pp. 345–360.
[56] J. Sung, I. Lenz, A. Saxena, Deep multimodal embedding: manipulat-
ing novel objects with point-clouds, language and trajectories, in: 2017
IEEE International Conference on Robotics and Automation, ICRA 2017,
Singapore, Singapore, May 29 - June 3, 2017, 2017, pp. 2794–2801.
[57] C. Hazirbas, L. Ma, C. Domokos, D. Cremers, Fusenet: Incorporating depth
into semantic segmentation via fusion-based CNN architecture, in: Pro-
ceedings of the Asian Conference on Computer Vision, Taipei, Taiwan,
2016, pp. 213–228.
[58] S. Song, J. Xiao, Deep sliding shapes for amodal 3D object detection in
RGB-D images, in: Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), Las Vegas, USA, 2016, pp. 808–816.
[59] M. Giering, V. Venugopalan, K. Reddy, Multi-modal sensor registration for
vehicle perception via deep neural networks, in: Proceedings of the High
Performance Extreme Computing Conference (HPEC), Waltham, USA, 2015,
pp. 1–6.
[60] A. Eitel, J.T. Springenberg, L. Spinello, M.A. Riedmiller, W. Burgard, Multi-
modal deep learning for robust RGB-D object recognition, in: 2015 IEEE/RSJ
International Conference on Intelligent Robots and Systems, IROS 2015,
Hamburg, Germany, September 28 - October 2, 2015, 2015, pp. 681–687.
[61] A. Valada, G.L. Oliveira, T. Brox, W. Burgard, Deep multispectral semantic
scene understanding of forested environments using multimodal fusion,
in: International Symposium on Experimental Robotics, ISER 2016, Tokyo,
Japan, October (2016) 3-6, 2016, pp. 465–477.
[62] O. Mees, A. Eitel, W. Burgard, Choosing smartly: Adaptive multimodal fu-
sion for object detection in changing environments, CoRR, abs/1707.05733,
2017.
[63] A. Valada, J. Vertens, A. Dhall, W. Burgard, Adapnet: Adaptive semantic
segmentation in adverse environmental conditions, in: 2017 IEEE Inter-
national Conference on Robotics and Automation, ICRA 2017, Singapore,
Singapore, May 29 - June 3, 2017, 2017, pp. 4644–4651.
[64] R. Mur-Artal, J.D. Tardós, ORB-SLAM2: an open-source SLAM system for
monocular, stereo, and RGB-D cameras, IEEE Trans. Robot. 33 (5) (2017)
1255–1262.
[65] R. Mur-Artal, J.M.M. Montiel, J.D. Tardos, Orb-slam: a versatile and accurate
monocular slam system, IEEE Trans. Robot. 31 (5) (2015) 1147–1163.
[66] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A.
Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch, 2017.
[67] R.B. Girshick, Fast R-CNN, in: Proceedings of the IEEE International
Conference on Computer Vision (ICCV), Santiago, Chile, 2015, pp.
1440–1448.
[68] M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional net-
works, in: Proceedings of the European Conference on Computer Vision,
Zurich, Switzerland, 2014, pp. 818–833.
[69] J.T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller, Striving for
simplicity: The all convolutional net, in: Proceedings of the International
Conference on Learning Representations (ICLR) Workshop, San Diego, USA,
2015.
Naman Patel received his Master degree in Electrical
Engineering from NYU Tandon School of Engineering,
Brooklyn, New York, in 2016, where he is cur-
rently working towards his Ph.D. degree with the
Control/Robotics Research Laboratory (CRRL) headed
by Professor Farshad Khorrami. His research inter-
ests include developing algorithms for autonomous
unmanned ground and aerial vehicles, simultaneous
localization and mapping systems, computer vision and
artificial intelligence.
Anna Choromonska received her Ph.D. degree from
the Department of Electrical Engineering at Columbia
University in the City of New York in 2014, and a
M.Sc. degree with distinctions from the Department
of Electronics and Information Technology at Warsaw
University of Technology. She did her Post-Doctoral
studies in the Computer Science Department at Courant
Institute of Mathematical Sciences in NYU. She joined
the Department of Electrical and Computer Engineering
at NYU Tandon School of Engineering in the Spring
2017 as an Assistant Professor. Prof. Choromanska’s
research interests focus on numerical optimization, deep learning, and large
data analysis with applications to autonomous car driving. She is also working
on learning from data streams, learning with expert advice, supervised and
unsupervised online learning, clustering, and structured prediction. Prof. Choro-
manska has co-authored several international conference papers and refereed
 N. Patel, A. Choromanska, P. Krishnamurthy et al. / Robotics and Autonomous Systems 116 (2019) 80–97
journal publications, as well as book chapters. She is also a contributor to
the open source fast out-of-core learning system Vowpal Wabbit (aka VW).
Prof. Choromanska gave over 50 invited and conference talks and serves as a
book editor (MIT Press volume), organizer of machine learning events at top
venues like NIPS, and a reviewer and area chair for several top machine learning
conferences (e.g., ICML, NIPS, and AISTATS) and journals (e.g., Transactions on
Pattern Analysis and Machine Intelligence and Machine Learning).
Prashanth Krishnamurthy received his B.Tech. degree
in electrical engineering from Indian Institute of Tech-
nology, Chennai in 1999, and M.S. and Ph.D. degrees in
electrical engineering from Polytechnic University (now
NYU), Brooklyn, NY in 2002 and 2006, respectively. He
is currently a Research Scientist and Adjunct Faculty
with the Department of Electrical and Computer En-
gineering at NYU Tandon School of Engineering, NY,
and a Senior Researcher with FarCo Technologies, NY.
He has co-authored over 110 journal and conference
papers in the broad areas of autonomous systems,
robotics, and control systems. He has also co-authored the book ‘‘Modeling and
Adaptive Nonlinear Control of Electric Motors’’ published by Springer Verlag
in 2003. His research interests include robust and adaptive nonlinear control,
resilient control, autonomous vehicles and robotic systems, path planning and
obstacle avoidance, sensor data fusion, machine learning, real-time embedded
systems, electromechanical systems modeling and control, cyber–physical sys-
tems and cyber-security, decentralized and large-scale systems, high-fidelity and
hardware-in-the-loop simulation, real-time software implementations for robotic
systems, and distributed multi-agent systems.
97
Farshad Khorrami received his Bachelors degrees in
Mathematics and Electrical Engineering in 1982 and
1984 respectively from The Ohio State University. He
also received his Master’s degree in Mathematics and
Ph.D. in Electrical Engineering in 1984 and 1988 from
The Ohio State University. Dr. Khorrami is currently a
professor of Electrical & Computer Engineering Depart-
ment at NYU where he joined as an assistant professor
in Sept. 1988. His research interests include adaptive
and nonlinear controls, robotics and automation, un-
manned vehicles (fixed-wing and rotary wing aircrafts
as well as underwater vehicles and surface ships), resilient control for industrial
control systems, cyber security for cyber–physical systems, large-scale systems
and decentralized control, and real-time embedded instrumentation and control.
Prof. Khorrami has published more than 240 refereed journal and conference
papers in these areas and holds thirteen U.S. patents. His book on ‘‘Modeling and
adaptive nonlinear control of electric motors’’ was published by Springer Verlag
in 2003. He also has thirteen U.S. patents on novel smart micropositioners and
actuators, control systems, and wireless sensors and actuators. He has developed
and directed the Control/Robotics Research Laboratory at Polytechnic University
(Now NYU). His research has been supported by the Army Research Office,
National Science Foundation, Office of Naval Research, DARPA, Air Force Research
Laboratory, Sandia National Laboratory, Army Research Laboratory, NASA, and
several corporations. Prof. Khorrami has served as general chair and conference
organizing committee member of several international conferences.