Ten Trending Academic Papers on the Future of

Computer Vision

These papers provide a breadth of information about Computer Vision that is generally useful and interesting from a

data science perspective.

 Contents

 Learning Individual Styles of Conversational Gesture

 Textured Neural Avatars

 DSFD: Dual Shot Face Detector

 GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face

Reconstruction

 DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation,

Segmentation and Re-Identification of Clothing Images

 Inverse Cooking: Recipe Generation from Food Images

 ArcFace: Additive Angular Margin Loss for Deep Face Recognition

 Fast Online Object Tracking and Segmentation: A Unifying Approach

 Revealing Scenes by Inverting Structure from Motion Reconstructions

 Semantic Image Synthesis with Spatially-Adaptive Normalization

 Learning Individual Styles of Conversational Gesture

Shiry Ginosar∗ Amir Bar∗ Gefen Kohavi Caroline Chan

UC Berkeley Zebra Medical Vision UC Berkeley MIT
Andrew Owens Jitendra Malik
UC Berkeley UC Berkeley
arXiv:1906.04160v1 [cs.CV] 10 Jun 2019

Figure 1: Speech-to-gesture translation example. In this paper, we study the connection between conversational gesture and speech.
Here, we show the result of our model that predicts gesture from audio. From the bottom upward: the input audio, arm and hand pose
predicted by our model, and video frames synthesized from pose predictions using [10]. (See http://people.eecs.berkeley.
edu/˜shiry/speech2gesture for video results.)

Abstract we spontaneously emit when we speak [34]. They comple-

Human speech is often accompanied by hand and arm
gestures. Given audio speech input, we generate plausi-
ble gestures to go along with the sound. Specifically, we
perform cross-modal translation from “in-the-wild” mono-
logue speech of a single speaker to their hand and arm mo-
tion. We train on unlabeled videos for which we only have
noisy pseudo ground truth from an automatic pose detec-
tion system. Our proposed model significantly outperforms
baseline methods in a quantitative comparison. To support
research toward obtaining a computational understanding
of the relationship between gesture and speech, we release
a large video dataset of person-specific gestures.
1. Introduction
When we talk, we convey ideas via two parallel channels
of communication—speech and gesture. These conversa-
tional, or co-speech, gestures are the hand and arm motions
∗ Indicates equal contribution.
ment speech and add non-verbal information that help our
listeners comprehend what we say [6]. Kendon [23] places
conversational gestures at one end of a continuum, with sign
language, a true language, at the other end. In between the
two extremes are pantomime and emblems like “Italianite”,
with an agreed-upon vocabulary and culture-specific mean-
ings. A gesture can be subdivided into phases describing
its progression from the speaker’s rest position, through the
gesture preparation, stroke, hold and retraction back to rest.
Is the information conveyed in speech and gesture corre-
lated? This is a topic of ongoing debate. The hand-in-hand
hypothesis claims that gesture is redundant to speech when
speakers refer to subjects and objects in scenes [43]. In
contrast, according to the trade-off hypothesis, speech and
gesture are complementary since people use gesture when
speaking would require more effort and vice versa [15]. We
approach the question from a data-driven learning perspec-
tive and ask to what extent can we predict gesture motion
from the raw audio signal of speech.
We present a method for temporal cross-modal transla-
tion. Given an input audio clip of a spoken statement (Fig-

1
 Almaram Angelica Kubinec Covach Kagan

Conan Oliver Stewart Meyers Ellen

Figure 2: Speaker-specific gesture dataset. We show a representative video frame for each speaker in our dataset. Below each one is a
heatmap depicting the frequency that their arms and hands appear in different spatial locations (using the skeletal representation of gestures
shown in Figure 1). This visualization reveals the speaker’s resting pose, and how they tend to move—for example, Angelica tends to keep
her hands folded, whereas Kubinec frequently points towards the screen with his left hand. Note that some speakers, like Kagan, Conan
and Ellen, alternate between sitting and standing and thus the distribution of their arm positions is bimodal.

ure 1 bottom), we generate a corresponding motion of the
speaker’s arms and hands which matches the style of the
speaker, despite the fact that we have never seen or heard
this person say this utterance in training (Figure 1 middle).
We then use an existing video synthesis method to visualize
what the speaker might have looked like when saying these
words (Figure 1 top).
To generate motion from speech, we must learn a map-
ping between audio and pose. While this can be formu-
lated as translation, in practice there are two inherent chal-
lenges to using the natural pairing of audio-visual data in
this setting. First, gesture and speech are asynchronous, as
gesture can appear before, after or during the correspond-
ing utterance [4]. Second, this is a multimodal prediction
task as speakers may perform different gestures while say-
ing the same thing on different occasions. Moreover, ac-
quiring human annotations for large amounts of video is in-
feasible. We therefore need to get a training signal from
pseudo ground truth of 2D human pose detections on unla-
beled video.
Nevertheless, we are able to translate speech to gesture
in an end-to-end fashion from the raw audio to a sequence
of poses. To overcome the asynchronicity issue we use a
large temporal context (both past and future) for prediction.
Temporal context also allows for smooth gesture prediction
despite the noisy automatically-annotated pseudo ground
truth. Due to multimodality, we do not expect our predicted
motion to be the same as the ground truth. However, as this
is the only training signal we have, we still use automatic
pose detections for learning through regression. To avoid
regressing to the mean of all modes, we apply an adversar-
ial discriminator [19] to our predicted motion. This ensures
that we produce motion that is “real” with respect to the
current speaker.
Gesture is idiosyncratic [34], as different speakers tend
to use different styles of motion (see Figure 2). It is there-
fore important to learn a personalized gesture model for
each speaker. To address this, we present a large, 144-hour
person-specific video dataset of 10 speakers that we make
publicly available1 . We deliberately pick a set of speakers
for which we can find hours of clean single-speaker footage.
Our speakers come from a diverse set of backgrounds: tele-
vision show hosts, university lecturers and televangelists.
They span at least three religions and discuss a large range
of topics from commentary on current affairs through the
philosophy of death, chemistry and the history of rock mu-
sic, to readings in the Bible and the Qur’an.
1 http://people.eecs.berkeley.edu/ shiry/
˜
speech2gesture
2. Related Work
Conversational Gestures McNeill [34] divides gestures
into several classes [34]: emblematics have specific conven-
tional meanings (e.g. “thumbs up!”); iconics convey physi-
cal shapes or direction of movements; metaphorics describe
abstract content using concrete motion; deictics are point-
ing gestures, and beats are repetitive, fast hand motions that
provide a temporal framing to speech.
Many psychologists have studied questions related to co-
speech gestures [34, 23] (See [46] for a review). This vast
body of research has mostly relied on studying a small num-
ber of individual subjects using recorded choreographed
story retelling in lab settings. Analysis in these studies was
a manual process. Our goal, instead, is to study conversa-
tional gestures in the wild using a data-driven approach.
Conditioning gesture prediction on speech is arguably an
ambiguous task, since gesture and speech may not be syn-
chronous. While McNeill [34] suggests that gesture and
speech originate from a common source and thus should co-
occur in time according to well-defined rules, Kendon [23]
suggests that gesture starts before the corresponding utter-
ance. Others even argue that the temporal relationships be-
tween speech and gesture are not yet clear and that gesture
can appear before, after or during an utterance [4].
Sign language and emblematic gesture recognition
There has been a great deal of computer vision work geared
towards recognizing sign language gestures from video.
This includes methods that use video transcripts as a weak
source of supervision [3], as well as recent methods based
on CNNs [37, 26] and RNNs [13]. There has also been work
that recognizes emblematic hand and face gestures [17, 14],
head gestures [35], and co-speech gestures [38]. By con-
trast, our goal is to predict co-speech gestures from audio.
Conversational agents Researchers have proposed a
number of methods for generating plausible gestures, par-
ticularly for applications with conversational agents [8]. In
early work, Cassell et al. [7] proposed a system that guided
arm/hand motions based on manually defined rules. Sub-
sequent rule-based systems [27] proposed new ways of ex-
pressing gestures via annotations.
More closely related to our approach are methods that
learn gestures from speech and text, without requiring an
author to hand-specify rules. Notably, [9] synthesized ges-
tures using natural language processing of spoken text, and
Neff [36] proposed a system for making person-specific
gestures. Levine et al. [30] learned to map acoustic prosody
features to motion using a HMM. Later work [29] extended
this approach to use reinforcement learning and speech
recognition, combined acoustic analysis with text [33], cre-
ated hybrid rule-based systems [40], and used restricted
Boltzmann machines for inference [11]. Since the goal of
these methods is to generate motions for virtual agents, they
use lab-recorded audio, text, and motion capture. This al-
lows them to use simplifying assumptions that present chal-
lenges for in-the-wild video analysis like ours: e.g., [30]
requires precise 3D pose and assumes that motions occur
on syllable boundaries, and [11] assumes that gestures are
initiated by an upward motion of the wrist. In contrast
with these methods, our approach does not explicitly use
any text or language information during training—it learns
gestures from raw audio-visual correspondences—nor does
it use hand-defined gesture categories: arm/hand pose are
predicted directly from audio.
Visualizing predicted gestures One of the most common
ways of visualizing gestures is to use them to animate a 3D
avatar [45, 29, 20]. Since our work studies personalized
gestures for in-the-wild videos, where 3D data is not avail-
able, we use a data-driven synthesis approach inspired by
Bregler et al. [2]. To do this, we employ the pose-to-video
method of Chan et al. [10], which uses a conditional gen-
erative adversarial network (GAN) to synthesize videos of
human bodies from pose.
Sound and vision Aytar et al. [1] use the synchronization
of visual and audio signals in natural phenomena to learn
sound representations from unlabeled in-the-wild videos.
To do this, they transfer knowledge from trained discrim-
inative models in the visual domain, to the audio domain.
Synchronization of audio and visual features can also be
used for synthesis. Langlois et al. [28] try to optimize for
synchronous events by generating rigid-body animations of
objects falling or tumbling that temporally match an input
sound wave of the desired sequence of contact events with
the ground plane. More recently, Shlizerman et al. [42]
animated the hands of a 3D avatar according to input mu-
sic. However, their focus was on music performance, rather
than gestures, and consequently the space of possible mo-
tions was limited (e.g., the zig-zag motion of a violin bow).
Moreover, while music is uniquely defined by the motion
that generates it (and is synchronous with it), gestures are
neither unique to, nor synchronous with speech utterances.
Several works have focused on the specific task of
synthesizing videos of faces speaking, given audio input.
Chung et al. [12] generate an image of a talking face from
a still image of the speaker and an input speech segment
by learning a joint embedding of the face and audio. Simi-
larly, [44] synthesizes videos of Obama saying novel words
by using a recurrent neural network to map speech audio to
mouth shapes and then embedding the synthesized lips in
ground truth facial video. While both methods enable the
creation of fake content by generating faces saying words
taken from a different person, we focus on single-person
models that are optimized for animating same-speaker ut-
terances. Most importantly, generating gesture, rather than
lip motion, from speech is more involved as gestures are
asynchronous with speech, multimodal and person-specific.
3. A Speaker-Specific Gesture Dataset
We introduce a large 144-hour video dataset specifically
tailored to studying speech and gesture of individual speak-
ers in a data-driven fashion. As shown in Figure 2, our
dataset contains in-the-wild videos of 10 gesturing speak-
ers that were originally recorded for television shows or
university lectures. We collect several hours of video per
speaker, so that we can individually model each one. We
chose speakers that cover a wide range of topics and ges-
turing styles. Our dataset contains: 5 talk show hosts, 3
lecturers and 2 televangelists. Details about data collection
and processing as well as an analysis of the individual styles
of gestures can be found in the supplementary material.
Gesture representation and annotation We represent
the speakers’ pose over time using a temporal stack of 2D
skeletal keypoints, which we obtain using OpenPose [5].
From the complete set of keypoints detected by OpenPose,
we use the 49 points corresponding to the neck, shoulders,
elbows, wrists and hands to represent gestures. Together
with the video footage, we provide the skeletal keypoints
for each frame of the data at a 15fps. Note, however, that
these are not ground truth annotations, but a proxy for the
ground truth from a state-of-the-art pose detection system.
Quality of dataset annotations All ground truth,
whether from human observers or otherwise, has associated
error. The pseudo ground truth we collect using automatic
pose detection may have much larger error than human an-
notations, but it enables us to train on much larger amounts
of data. Still, we must estimate whether the accuracy of the
pseudo ground truth is good enough to support our quantita-
tive conclusions. We compare the automatic pose detections
to labels obtained from human observers on a subset of our
training data and find that the pseudo ground truth is close
to human labels and that the error in the pseudo ground truth
is small enough for our task. The full experiment is detailed
in our supplementary material.
4. Method
Given raw audio of speech, our goal is to generate the
speaker’s corresponding arm and hand gesture motion. We
approach this task in two stages—first, since the only sig-
nal we have for training are corresponding audio and pose
detection sequences, we learn a mapping from speech to
gesture using L1 regression to temporal stacks of 2D key-
points. Second, to avoid regressing to the mean of all pos-
sible modes of gesture, we employ an adversarial discrim-
inator that ensures that the motion we produce is plausible
with respect to the typical motion of the speaker.
Audio G(t1), . . . , G(tT )
G L1 regression loss
Time
D
Real or Fake
Motion Sequence?
Frequency
Figure 3: Speech to gesture translation model. A convolutional
audio encoder downsamples the 2D spectrogram and transforms
it to a 1D signal. The translation model, G, then predicts a corre-
sponding temporal stack of 2D poses. L1 regression to the ground
truth poses provides a training signal, while an adversarial dis-
criminator, D, ensures that the predicted motion is both temporally
coherent and in the style of the speaker.
4.1. Speech-to-Gesture Translation
Any realistic gesture motion must be temporally coher-
ent and smooth. We accomplish smoothness by learning an
audio encoding which is a representation of the whole ut-
terance, taking into account the full temporal extent of the
input speech, s, and predicting the whole temporal sequence
of corresponding poses, p, at once (rather than recurrently).
Our fully convolutional network consists of an audio en-
coder followed by a 1D UNet [39, 22] translation architec-
ture, as shown in Figure 3. The audio encoder takes a 2D
log-mel spectrogram as input, and downsamples it through
a series of convolutions, resulting in a 1D signal with the
same sampling rate as our video (15 Hz). The UNet transla-
tion architecture then learns to map this signal to a temporal
stack of pose vectors (see Section 3 for details of our gesture
representation) via an L1 regression loss:
LL1 (G) = Es,p [||p − G(s)||1 ]. (1)
We use a UNet architecture for translation since its bot-
tleneck provides the network with past and future tempo-
ral context, while the skip connections allow for high fre-
quency temporal information to flow through, enabling pre-
diction of fast motion.
4.2. Predicting Plausible Motion
While L1 regression to keypoints is the only way we
can extract a training signal from our data, it suffers from
the known issue of regression to the mean which produces
overly smooth motion. This can be seen in our supplemen-
tary video results. To combat the issue and ensure that we
produce realistic motion, we add an adversarial discrimi-
nator [22, 10] D, conditioned on the difference of the pre-
dicted sequence of poses. i.e. the input to the discriminator
is the vector m = [p2 −p1 , . . . , pT −pT −1 ] where pi are 2D
pose keypoints and T is the temporal extent of the input au-
dio and predicted pose sequence. The discriminator D tries
to maximize the following objective while the generator G
(translation architecture, Section 4.1) tries to minimize it:
LGAN (G, D) = Em [log D(m)] + Es [log(1 − G(s))], (2)
where s is the input audio speech segment and m is the mo-
tion derivative of the predicted stack of poses. Thus, the
generator learns to produce real-seeming speaker motion
while the discriminator learns to classify whether a given
motion sequence is real. Our full objective is therefore:
min max LGAN (G, D) + λLL1 (G). (3)
G D
4.3. Implementation Details
We obtain translation invariance by subtracting (per
frame) the neck keypoint location from all other keypoints
in our pseudo ground truth gesture representation (section
3). We then normalize each keypoint (e.g. left wrist) across
all frames by subtracting the per-speaker mean and divid-
ing by the standard deviation. During training, we take as
input spectrograms corresponding to about 4 seconds of au-
dio and predict 64 pose vectors, which correspond to about
4 seconds at a 15Hz frame-rate. At test time we can run
our network on arbitrary audio durations. We optimize us-
ing Adam [24] with a batch size of 32 and a learning rate of
10−4 . We train for 300K/90K iterations with and without an
adversarial loss, respectively, and select the best performing
model on the validation set.
5. Experiments
We show that our method produces motion that quanti-
tatively outperforms several baselines, as well as a previous
method that we adapt to the problem.
5.1. Setup
We describe our experimental setup including our base-
lines for comparison and evaluation metric.
5.1.1 Baselines
We compare our method to several other models.
Always predict the median pose Speakers spend most of
their time in rest position [23], so predicting the speaker’s
median pose can be a high-quality baseline. For a visualiza-
tion of each speaker’s rest position, see Figure 2.
Predict a randomly chosen gesture In this baseline, we
randomly select a different gesture sequence (which does
not correspond to the input utterance) from the training set
of the same speaker, and use this as our prediction. While
we would not expect this method to perform well quantita-
tively, there is reason to think it would generate qualitatively
appealing motion: these are real speaker gestures—the only
way to tell they are fake is to evaluate how well they corre-
sponds to the audio.
Nearest neighbors Instead of selecting a completely ran-
dom gesture sequence from the same speaker, we can use
audio as a similarity cue. For an input audio track, we find
its nearest neighbor for the speaker using pretrained audio
features, and transfer its corresponding motion. To repre-
sent the audio, we use the state-of-the-art VGGish feature
embedding [21] pretrained on AudioSet [18], and use co-
sine distance on normalized features.
RNN-based model [42] We further compare our motion
prediction to an RNN architecture proposed by Shlizerman
et al. Similar to us, Shlizerman et al. predict arm and hand
motion from audio in a 2D skeletal keypoint space. How-
ever, while our model is a convolutional neural network
with log-mel spectrogram input, theirs uses a 1-layer LSTM
model that takes MFCC features (a low-dimensional, hand-
crafted audio feature representation) as input. We evaluated
both feature types and found that for [42], MFCC features
outperform the log-mel spectrogram features on all speak-
ers. We therefore use their original MFCC features in our
experiments. For consistency with our own model, instead
of measuring L2 distance on PCA features, as they do, we
add an extra hidden layer and use L1 distance.
Ours, no GAN Finally, as an ablation, we compare our
full model to the prediction of the translation architecture
alone, without the adversarial discriminator.
5.1.2 Evaluation Metrics
Our main quantitative evaluation metric is the L1 regres-
sion loss of the different models in comparison. We ad-
ditionally report results according to the percent of correct
keypoints (PCK) [47], a widely accepted metric for pose de-
tection. Here, a predicted keypoint is defined as correct if
it falls within α max(h, w) pixels of the ground truth key-
point, where h and w are the height and width of the person
bounding box, respectively.
We note that PCK was designed for localizing object
parts, whereas we use it here for a cross-modal prediction
task (predicting pose from audio). First, unlike L1 , PCK is
not linear and correctness scores fall to zero outside a hard
threshold. Since our goal is not to predict the ground truth
motion but rather to use it as a training signal, L1 is more
suited to measuring how we perform on average. Second,
PCK is sensitive to large gesture motion as the correctness
radius depends on the width of the span of the speaker’s
arms. While [47] suggest α = 0.1 for data with full people
and α = 0.2 for data where only half the person is visi-
ble, we take an average over α = 0.1, 0.2 and show the full
results in the supplementary.
5.2. Quantitative Evaluation
We compare the results of our method to the baselines
using our quantitative metrics. To assess whether our re-
sults are perceptually convincing, we conduct a user study.
Finally, we ask whether the gestures we predict are person-
specific and whether the input speech is indeed a better pre-
dictor of motion than the initial pose of the gesture.
5.2.1 Numerical Comparison
We compare to all baselines on 2,048 randomly chosen test
set intervals per speaker and display the results in Table 1.
We see that on most speakers, our model outperforms all
others, where our no-GAN condition is slightly better than
the GAN one. This is expected, as the adversarial dis-
criminator pushes the generator to snap to a single mode
of the data, which is often further away from the actual
ground truth than the mean predicted by optimizing L1 loss
alone. Our model outperforms the RNN-based model on
most speakers. Qualitatively, we find that this baseline pre-
dicts relatively small motions on our data, which may be
due to the fact that it has relatively low capacity compared
to our UNet model.
5.2.2 Human Study
To gain insight into how synthesized gestures perceptually
compare to real motion, we conducted a small-scale real
vs. fake perceptual study on Amazon Mechanical Turk.
We used two speakers who are always shot from the same
camera viewpoint: Oliver, whose gestures are relatively dy-
namic and Meyers, who is relatively stationary. We visu-
alized gesture motion using videos of skeletal wire frames.
To provide participants with additional context, we included
the ground truth mouth and facial keypoints of the speaker
in the videos. We show examples of skeletal wire frame
videos in our video supplementary material.
Participants watched a series of video pairs. In each pair,
one video was produced from a real pose sequence; the
other was generated by an algorithm—our model or a base-
line. Participants were then asked to identify the video con-
taining the motion that corresponds to the speech sound (we
did not verify that they in fact listened to the speech while
answering the question). Videos of 4 seconds or 12 seconds
each of resolution 400×226 (downsampled from 910×512
in order to fit two videos side-by-side on different screen
sizes) were shown, and after each pair, participants were
given unlimited time to respond. We sampled 100 input
audio intervals at random and predicted from them a 2D-
keypoint motion sequence using each method. Each task
consisted of 20 pairs of videos and was performed by 300
different participants. Each participant was given a short
training set of 10 video pairs before the start of the task,
and was given feedback indicating whether they had cor-
rectly identified the ground-truth motion.
We compared all the gesture-prediction models (Sec-
tion 5.1.1) and assessed the quality of each method using
Figure 4: Our trained models are person-specific. For every
speaker audio input (row) we apply all other individually trained
speaker models (columns). Color saturation corresponds to L1
loss values on a held out test set (lower is better). For each row,
the entry on the diagonal is lightest as models work best using the
input speech of the person they were trained on.
the rate at which its output fooled the participants. Inter-
estingly, we found that for the dynamic speaker all meth-
ods that generate realistic motion fooled humans at similar
rates. As shown in Table 2, our results for this speaker were
comparable to real motion sequences, whether selected by
an audio-based nearest neighbor approach or randomly. For
the stationary speaker who spends most of the time in rest
position, real motion was more often selected as there is
no prediction error associated with it. While the nearest
neighbor and random motion models are significantly less
accurate quantitatively (Table 1), they are perceptually con-
vincing because their components are realistic.
5.2.3 The Predicted Gestures are Person-Specific
For every speaker’s speech input (Figure 4 rows), we pre-
dict gestures using all other speakers’ trained models (Fig-
ure 4 columns). We find that on average, predicting using
our model trained on a different speaker performs better nu-
merically than predicting random motion, but significantly
worse than always predicting the median pose of the input
speaker (and far worse than the predictions from the model
trained on the input speaker). The diagonal structure of the
confusion matrix in Figure 4 exemplifies this.
5.2.4 Speech is a Good Predictor for Gesture
Seeing the success of our translation model, we ask how
much does the audio signal help when the initial pose of the
gesture sequence is known. In other words, how much can
sound tell us beyond what can be predicted from motion dy-
namics. To study this, we augment our model by providing
it the pose of the speaker directly preceding their speech,
which we incorporate into the bottleneck of the UNet (Fig-
ure 3). We consider the following conditions: Predict me-
dian pose, as in the baselines above. Predict the input initial
 Model Meyers Oliver Conan Stewart Ellen Kagan Kubinec Covach Angelica Almaram Avg. L1 Avg. PCK
Median 0.66 0.69 0.79 0.63 0.75 0.80 0.80 0.70 0.74 0.76 0.73 38.11
Random 0.93 1.00 1.10 0.94 1.07 1.11 1.12 1.00 1.04 1.08 1.04 26.55
NN [21] 0.88 0.96 1.05 0.93 1.02 1.11 1.10 0.99 1.01 1.06 1.01 27.92
RNN [42] 0.61 0.66 0.76 0.62 0.71 0.74 0.73 0.72 0.72 0.75 0.70 39.69
Ours, no GAN 0.57 0.60 0.63 0.61 0.71 0.72 0.68 0.69 0.75 0.76 0.67 44.62
Ours, GAN 0.77 0.63 0.64 0.68 0.81 0.74 0.70 0.72 0.78 0.83 0.73 41.95

Table 1: Quantitative results for the speech to gesture translation task using L1 loss (lower is better) on the test set. The rightmost column
is the average PCK value (higher is better) over all speakers and α = 0.1, 0.2 (See full results in supplementary).

Oliver Meyers Model Avg. L1 Avg. PCK

Model 4 seconds 12 seconds 4 seconds 12 seconds

Pred.
Predict the median pose 0.73 38.11
Median 12.1 ± 2.8 6.7 ± 2.0 34.0 ± 4.2 25.8 ± 3.9 Predict the input initial pose 0.53 60.50
Random 34.2 ± 4.0 29.1 ± 3.7 40.9 ± 4.6 34.3 ± 4.4 Speech input 0.67 44.62

NN [21] 36.9 ± 3.9 26.4 ± 3.8 43.5 ± 4.5 33.3 ± 4.4
RNN [42] 18.2 ± 3.2 10.0 ± 2.5 37.5 ± 4.6 19.4 ± 3.6
Ours, no GAN 25.0 ± 3.8 19.8 ± 3.4 36.1 ± 4.3 33.1 ± 4.2
Ours, GAN 35.4 ± 4.0 27.8 ± 3.9 33.2 ± 4.4 22.0 ± 4.0
Table 2: Human study results for the speech to gesture translation
task on 4 and 12-second video clips of two speakers—one dy-
namic (Oliver) and one relatively stationary (Meyers). As a metric
for comparison, we use the percentage of times participants were
fooled by the generated motions and picked them as real over the
ground truth motion in a two-alternative forced choice. We found
that humans were not sensitive to the alignment of speech and
gesture. For the dynamic speaker, gestures with realistic motion—
whether randomly selected from another video of the same speaker
or generated by our GAN-based model—fooled humans at equal
rates (no statistically significant difference between the bolded
numbers). Since the stationary speaker is usually at rest position,
real unaligned motion sequences look more realistic as they do not
suffer from prediction noise like the generated ones.
Input
Initial pose input 0.49 61.24
Speech & initial pose input 0.47 62.39
Table 3: How much information does sound provide once we
know the initial pose of the speaker? We see that the initial pose
of the gesture sequence is a good predictor for the rest of the
4-second motion sequence (second to last row), but that adding
audio improves the prediction (last row). We use both average
L1 loss (lower is better) and average PCK over all speakers and
α = 0.1, 0.2 (higher is better) as metrics of comparison. We com-
pare two baselines and three conditions of inputs.
5.3. Qualitative Results
We qualitatively compare our speech to gesture transla-
tion results to the baselines and the ground truth gesture
sequences in Figure 5. Please refer to our supplementary
video results which better convey temporal information.

pose, a model that simply repeats the input initial ground-
truth pose as its prediction. Speech input, our model. Initial
pose input, a variation of our model in which the audio in-
put is ablated and the network predicts the future pose from
only an initial ground-truth pose input, and Speech & initial
pose input, where we condition the prediction on both the
speech and the initial pose.
Table 3 displays the results of the comparison for our
model trained without the adversarial discriminator (no
GAN). When comparing the Initial pose input and Speech
& initial pose input conditions, we find that the addition
of speech significantly improves accuracy when we average
the loss across all speakers (p < 10−3 using a two sided
t-test). Interestingly, we find that most of the gains come
from a small number of speakers (e.g. Oliver) who make
large motions during speech.
6. Conclusion
Humans communicate through both sight and sound,
yet the connection between these modalities remains un-
clear [23]. In this paper, we proposed the task of predict-
ing person-specific gestures from “in-the-wild” speech as a
computational means of studying the connections between
these communication channels. We created a large person-
specific video dataset and used it to train a model for pre-
dicting gestures from speech. Our model outperforms other
methods in an experimental evaluation.
Despite its strong performance on these tasks, our model
has limitations that can be addressed by incorporating in-
sights from other work. For instance, using audio as in-
put has its benefits compared to using textual transcriptions
as audio is a rich representation that contains information
about prosody, intonation, rhythm, tone and more. How-
ever, audio does not directly encode high-level language se-
Figure 5: Speech to gesture translation qualitative results. We show the input audio spectrogram and the predicted poses overlaid on the
ground-truth video for Dr. Kubinec (lecturer) and Conan O’Brien (show host). See our supplementary material for more results.
mantics that may allow us to predict certain types of gesture
(e.g. metaphorics), nor does it separate the speaker’s speech
from other sounds (e.g. audience laughter). Additionally,
we treat pose estimations as though they were ground truth,
which introduces significant amount of noise—particularly
on the speakers’ fingers.
We see our work as a step toward a computational anal-
ysis of conversational gesture, and opening three possible
directions for further research. The first is in using gestures
as a representation for video analysis: co-speech hand and
arm motion make a natural target for video prediction tasks.
The second is using in-the-wild gestures as a way of train-
ing conversational agents: we presented one way of visual-
izing gesture predictions, based on GANs [10], but, follow-
ing classic work [8], these predictions could also be used
to drive the motions of virtual agents. Finally, our method
is one of only a handful of initial attempts to predict mo-
tion from audio. This cross-modal translation task is fertile
ground for further research.
Acknowledgements: This work was supported, in part, by the
AWS Cloud Credits for Research and the DARPA MediFor pro-
grams, and the UC Berkeley Center for Long-Term Cybersecu-
rity. Special thanks to Alyosha Efros, the bestest advisor, and to
Tinghui Zhou for his dreams of late-night talk show stardom.
References
[1] Y. Aytar, C. Vondrick, and A. Torralba. Soundnet: Learning
sound representations from unlabeled video. In Advances in
Neural Information Processing Systems, 2016. 3
[2] C. Bregler, M. Covell, and M. Slaney. Video rewrite: Driv-
ing visual speech with audio. In Computer Graphics and
Interactive Techniques, SIGGRAPH, pages 353–360. ACM,
1997. 3
[3] P. Buehler, A. Zisserman, and M. Everingham. Learning
sign language by watching tv (using weakly aligned subti-
tles). In Computer Vision and Pattern Recognition (CVPR),
pages 2961–2968. IEEE, 2009. 3
[4] B. Butterworth and U. Hadar. Gesture, speech, and compu-
tational stages: A reply to McNeill. Psychological Review,
 96:168–74, Feb. 1989. 2, 3
[5] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime multi-
person 2d pose estimation using part affinity fields. In Com-
puter Vision and Pattern Recognition (CVPR). IEEE, 2017.
4 A. Jansen, C. Moore, M. Plakal, D. Platt, R. A. Saurous,
[6] J. Cassell, D. McNeill, and K.-E. McCullough. Speech-
gesture mismatches: Evidence for one underlying represen-
tation of linguistic and nonlinguistic information. Pragmat-
ics and Cognition, 7(1):1–34, 1999. 1
[7] J. Cassell, C. Pelachaud, N. Badler, M. Steedman,
B. Achorn, T. Becket, B. Douville, S. Prevost, and M. Stone.
Animated conversation: Rule-based generation of facial ex-
pression, gesture & spoken intonation for multiple conversa-
tional agents. In Computer Graphics and Interactive Tech-
niques, SIGGRAPH, pages 413–420. ACM, 1994. 3
[8] J. Cassell, J. Sullivan, E. Churchill, and S. Prevost. Embod-
ied conversational agents. MIT press, 2000. 3, 8
[9] J. Cassell, H. H. Vilhjálmsson, and T. Bickmore. Beat: the
behavior expression animation toolkit. In Life-Like Charac-
ters, pages 163–185. Springer, 2004. 3
[10] C. Chan, S. Ginosar, T. Zhou, and A. A. Efros. Everybody
Dance Now. ArXiv e-prints, Aug. 2018. 1, 3, 4, 8
[11] C.-C. Chiu and S. Marsella. How to train your avatar:
A data driven approach to gesture generation. In Interna-
tional Workshop on Intelligent Virtual Agents, pages 127–
140. Springer, 2011. 3
[12] J. S. Chung, A. Jamaludin, and A. Zisserman. You said that?
In British Machine Vision Conference, 2017. 3
[13] N. Cihan Camgoz, S. Hadfield, O. Koller, H. Ney, and
R. Bowden. Neural sign language translation. In Computer
Vision and Pattern Recognition (CVPR). IEEE, June 2018. 3
[14] T. J. Darrell, I. A. Essa, and A. P. Pentland. Task-specific
gesture analysis in real-time using interpolated views. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
18(12):1236–1242, Dec. 1996. 3
[15] J. P. de Ruiter, A. Bangerter, and P. Dings. The interplay
between gesture and speech in the production of referring
expressions: Investigating the tradeoff hypothesis. Topics in
Cognitive Science, 4(2):232–248, Mar. 2012. 1
[16] D. F. Fouhey, W.-c. Kuo, A. A. Efros, and J. Malik. From
lifestyle vlogs to everyday interactions. arXiv preprint
arXiv:1712.02310, 2017. 10
[17] W. T. Freeman and M. Roth. Orientation histograms for hand
gesture recognition. In Workshop on Automatic Face and
Gesture Recognition. IEEE, June 1995. 3
[18] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen,
W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Au-
dio set: An ontology and human-labeled dataset for audio
events. In International Conference on Acoustics, Speech
and Signal Processing, pages 776–780, Mar. 2017. 5
[19] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Gen-
erative adversarial nets. In Advances in Neural Information
Processing Systems, pages 2672–2680, 2014. 2
[20] A. Hartholt, D. Traum, S. C. Marsella, A. Shapiro, G. Stra-
tou, A. Leuski, L.-P. Morency, and J. Gratch. All Together
Now: Introducing the Virtual Human Toolkit. In 13th In-
ternational Conference on Intelligent Virtual Agents, Edin-
burgh, UK, Aug. 2013. 3
[21] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke,
B. Seybold, M. Slaney, R. Weiss, and K. Wilson. CNN ar-
chitectures for large-scale audio classification. In Interna-
tional Conference on Acoustics, Speech and Signal Process-
ing. 2017. 5, 7
[22] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image
translation with conditional adversarial networks. In Com-
puter Vision and Pattern Recognition (CVPR), 2017. 4
[23] A. Kendon. Gesture: Visible Action as Utterance. Cam-
bridge University Press, 2004. 1, 3, 5, 7, 10, 11
[24] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. CoRR, abs/1412.6980, 2014. 5
[25] M. Kipp, M. Neff, K. H. Kipp, and I. Albrecht. Towards
natural gesture synthesis: Evaluating gesture units in a data-
driven approach to gesture synthesis. In C. Pelachaud, J.-C.
Martin, E. André, G. Chollet, K. Karpouzis, and D. Pelé,
editors, Intelligent Virtual Agents, pages 15–28, Berlin, Hei-
delberg, 2007. Springer Berlin Heidelberg. 10
[26] O. Koller, H. Ney, and R. Bowden. Deep hand: How to train
a cnn on 1 million hand images when your data is contin-
uous and weakly labelled. In Computer Vision and Pattern
Recognition (CVPR), pages 3793–3802. IEEE, 2016. 3
[27] S. Kopp, B. Krenn, S. Marsella, A. N. Marshall,
C. Pelachaud, H. Pirker, K. R. Thórisson, and
H. Vilhjálmsson. Towards a common framework for
multimodal generation: The behavior markup language. In
International workshop on intelligent virtual agents, pages
205–217. Springer, 2006. 3
[28] T. R. Langlois and D. L. James. Inverse-foley animation:
Synchronizing rigid-body motions to sound. ACM Transac-
tions on Graphics, 33(4):41:1–41:11, July 2014. 3
[29] S. Levine, P. Krähenbühl, S. Thrun, and V. Koltun. Gesture
controllers. In ACM Transactions on Graphics, volume 29,
page 124. ACM, 2010. 3
[30] S. Levine, C. Theobalt, and V. Koltun. Real-time prosody-
driven synthesis of body language. In ACM Transactions on
Graphics, volume 28, page 172. ACM, 2009. 3
[31] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-
manan, P. Dollr, and C. L. Zitnick. Microsoft coco: Common
objects in context. In European Conference on Computer Vi-
sion (ECCV), Zrich, 2014. Oral. 10
[32] R. C. B. Madeo, S. M. Peres, and C. A. de Moraes Lima.
Gesture phase segmentation using support vector machines.
Expert Systems with Applications, 56:100 – 115, 2016. 11
[33] S. Marsella, Y. Xu, M. Lhommet, A. Feng, S. Scherer, and
A. Shapiro. Virtual character performance from speech.
In Symposium on Computer Animation, SCA, pages 25–35.
ACM, 2013. 3
[34] D. McNeill. Hand and Mind: What Gestures Reveal about
Thought. University of Chicago Press, Chicago, 1992. 1, 2,
3, 10
[35] L.-P. Morency, A. Quattoni, and T. Darrell. Latent-dynamic
discriminative models for continuous gesture recognition. In
 Computer Vision and Pattern Recognition (CVPR), pages 1–
8. IEEE, 2007. 3
[36] M. Neff, M. Kipp, I. Albrecht, and H.-P. Seidel. Gesture
modeling and animation based on a probabilistic re-creation
of speaker style. ACM Transactions on Graphics, 27(1):5:1–
5:24, Mar. 2008. 3
[37] T. Pfister, K. Simonyan, J. Charles, and A. Zisserman. Deep
convolutional neural networks for efficient pose estimation
in gesture videos. In Asian Conference on Computer Vision,
pages 538–552. Springer, 2014. 3
[38] F. Quek, D. McNeill, R. Bryll, S. Duncan, X.-F. Ma, C. Kir-
bas, K. E. McCullough, and R. Ansari. Multimodal hu-
man discourse: gesture and speech. ACM Transactions
on Computer-Human Interaction (TOCHI), 9(3):171–193,
2002. 3
[39] O. Ronneberger, P.Fischer, and T. Brox. U-net: Convolu-
tional networks for biomedical image segmentation. In Med-
ical Image Computing and Computer-Assisted Intervention
(MICCAI), volume 9351 of LNCS, pages 234–241. Springer,
2015. 4
[40] N. Sadoughi and C. Busso. Retrieving target gestures to-
ward speech driven animation with meaningful behaviors. In
Proceedings of the 2015 ACM on International Conference
on Multimodal Interaction, ICMI ’15, pages 115–122. ACM,
2015. 3
[41] H. Sakoe and S. Chiba. Dynamic programming algorithm
optimization for spoken word recognition. IEEE Transac-
tions on Acoustics, Speech, and Signal Processing, 26(1):43–
49, Feb. 1978. 11
[42] E. Shlizerman, L. Dery, H. Schoen, and I. Kemelmacher-
Shlizerman. Audio to body dynamics. In Computer Vision
and Pattern Recognition (CVPR). IEEE, 2018. 3, 5, 7
[43] W. C. So, S. Kita, and S. Goldin-Meadow. Using the hands
to identify who does what to whom: Gesture and speech go
hand-in-hand. Cognitive Science, 33(1):115–125, Feb. 2009.
1 since 0.25 = ||prediction − µi || >> σi = 0.06. Note
[44] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-
Shlizerman. Synthesizing obama: Learning lip sync from
audio. ACM Transactions on Graphics, 36(4):95:1–95:13,
July 2017. 3
[45] M. Thiebaux, S. Marsella, A. N. Marshall, and M. Kall-
mann. Smartbody: Behavior realization for embodied con-
versational agents. In International Joint Conference on Au-
tonomous Agents and Multiagent Systems, volume 1, pages
151–158. International Foundation for Autonomous Agents
and Multiagent Systems, 2008. 3
[46] P. Wagner, Z. Malisz, and S. Kopp. Gesture and speech in
interaction: An overview. Speech Communication, 57:209 –
232, 2014. 3
[47] Y. Yang and D. Ramanan. Articulated human detection with
flexible mixtures of parts. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 35(12):2878–2890, Dec.
2013. 5
Figure 6: A segmented gesture unit.
7. Appendix
7.1. Dataset
Data collection and processing We collected internet
videos by querying YouTube for each speaker, and de-
duplicated the data using the approach of [16]. We then
used out-of-the-box face recognition and pose detection
systems to split each videos into intervals in which only the
subject appears in frame and all detected keypoints are vis-
ible. Our dataset consists of 60,000 such intervals with an
average length of 8.7 seconds and a standard deviation of
11.3 seconds. In total, there are 144 hours of video. We
split the data into 80% train, 10% validation, and 10% test
sets, such that each source video only appears in one set.
Quality of dataset annotations We estimate whether the
accuracy of the pseudo ground truth is good enough to sup-
port our quantitative conclusions via the following experi-
ment. We took a 200-frame subset of the pseudo ground
truth used for training and had it labeled by 3 human ob-
servers with neck and arm keypoints. We quantified the
consensus between annotators via, σi , a standard devia-
tion per keypoint-type i, as is typical in COCO [31] eval-
uation. We also computed ||opi − µi ||, the distance between
the OpenPose detection and the mean of the annotations,
and ||prediction − µi || the distance between our audio-
to-motion prediction and the annotation mean. We found
that the pseudo ground truth is close to human labels, since
0.14 = E[||opi − µi ||] ≈ E[σi ] = 0.06; And that the er-
ror in the pseudo ground truth is small enough for our task,
that this is a lower bound on the prediction error since it is
computed on training data samples.
7.2. Learning Individual Gesture Dictionaries
Gesture unit segmentation We use an unsupervised
method for building a dictionary of an individual’s ges-
tures. We segment motion sequences into gesture units,
propose an appropriate descriptor and similarity metric and
then cluster the gestures of an individual.
A gesture unit is a sequence of gestures that starts from
a rest position and returns to a rest position only after the
last gesture [23]. While [34] observed that most of their
subjects usually perform one gesture at a time, a study of an
18-minute video dataset of TV speakers reported that their
gestures were often strung together in a sequence [25]. We
treat each gesture unit – from rest position to rest position –
as an atomic segment.
 Individual styles of gesture These clusters represent an
unsupervised definition of the typical gestures that an in-
dividual performs. For each dictionary element cluster we
Cluster 3

define the central point as the point that is closest on aver-

age to all datapoints in the cluster. We sort the gesture units
in each cluster by their distance to the central point and pick
the most central ones for display. We visualize some exam-
ples of the dictionary of gestures we learn for Jon Stewart
in Figure 7.
Cluster 4
Cluster 9

Figure 7: Individual styles of gesture. Examples from Jon Stew-

art’s gesture dictionary.

We use an unsupervised approach to the temporal seg-

mentation of gesture units based on prediction error (by
contrast, [32] use a supervised approach). Given a motion
sequence of keypoints (Section 3) from time t0 to tT , we try
to predict the tT +1 pose. A low prediction error may signal
that the speaker is at rest, or that they are in the middle of a
gesture that the model has frequently seen during training.
Since speakers spend most of the time in rest position [23],
a high prediction error may indicate that a new gesture has
begun. We segment gesture units at points of high predic-
tion error (without defining a rest position per person). An
example of a segmented gesture unit is displayed in Fig-
ure 6. We train a segmentation model per subject and do
not expect it to generalize across speakers.

Dictionary learning We use the first 5 principal compo-

nents of the keypoints computed over all static frames as
a gesture unit descriptor. This reduces the dimensionality
while capturing 93% of the variance. We use dynamic time
warping [41] as our distance metric to account for temporal
variations in the execution of similar gestures. Since this
is not a Euclidean norm, we must compute the distance be-
tween each pair of datapoints. We precompute a distance
matrix for a randomly chosen sample of 1, 000 training ges-
ture units and use it to hierarchically cluster the datapoints.
 Textured Neural Avatars

Aliaksandra Shysheya 1,2 Egor Zakharov 1,2 Kara-Ali Aliev 1 Renat Bashirov 1
Egor Burkov 1,2 Karim Iskakov 1 Aleksei Ivakhnenko 1 Yury Malkov 1
Igor Pasechnik 1 Dmitry Ulyanov 1,2 Alexander Vakhitov 1,2 Victor Lempitsky 1,2
1 2
Samsung AI Center, Moscow Skolkovo Institute of Science and Technology, Moscow
arXiv:1905.08776v1 [cs.CV] 21 May 2019

Figure 1: We propose a new model for neural rendering of humans. The model is trained for a single person and can produce
renderings of this person from novel viewpoints (top) or in the new body pose (bottom) unseen during training. To improve
generalization, our model retains explicit texture representation, which is learned alongside the rendering neural network.

Abstract system is capable of learning to generate realistic render-

We present a system for learning full-body neural avatars,
i.e. deep networks that produce full-body renderings of a
person for varying body pose and camera position. Our
system takes the middle path between the classical graph-
ics pipeline and the recent deep learning approaches that
generate images of humans using image-to-image transla-
tion. In particular, our system estimates an explicit two-
dimensional texture map of the model surface. At the same
time, it abstains from explicit shape modeling in 3D. In-
stead, at test time, the system uses a fully-convolutional
network to directly map the configuration of body feature
points w.r.t. the camera to the 2D texture coordinates of in-
dividual pixels in the image frame. We show that such a
ings while being trained on videos annotated with 3D poses
and foreground masks. We also demonstrate that maintain-
ing an explicit texture representation helps our system to
achieve better generalization compared to systems that use
direct image-to-image translation.
1. Introduction
Capturing and rendering human body in all of its com-
plexity under varying pose and imaging conditions is one
of the core problems of both computer vision and com-
puter graphics. Recently, there is a surge of interest that
involves deep convolutional networks (ConvNets) as an al-
ternative to traditional computer graphics means. Realistic

1
neural rendering of body fragments e.g. faces [37, 43, 62],
eyes [24], hands [47] is now possible. Very recent works
have shown the abilities of such networks to generate views
of a person with a varying body pose but with a fixed cam-
era position, and using an excessive amount of training
data [1, 12, 42, 67]. In this work, we focus on the learn-
ing of neural avatars, i.e. generative deep networks that
are capable of rendering views of individual people under
varying body pose defined by a set of 3D positions of the
body joints and under varying camera positions (Figure 1).
We prefer to use body joint positions to represent the hu-
man pose, as joint positions are often easier to capture using
marker-based or marker-less motion capture systems.
Generally, neural avatars can serve as an alternative to
classical (“neural-free”) avatars based on a standard com-
puter graphics pipeline that estimates a user-personalized
body mesh in a neutral position, performs skinning (defor-
mation of the neutral pose), and projects the resulting 3D
surface onto the image coordinates, while superimposing
person-specific 2D texture. Neural avatars attempt to short-
cut the multiple stages of the classical pipeline and to re-
place them with a single network that learns the mapping
from the input (the location of body joints) to the output (the
2D image). As a part of our contribution, we demonstrate
that, however appealing for its conceptual simplicity, exist-
ing pose-to-image translation networks generalize poorly to
new camera views, and therefore new architectures for neu-
ral avatars are required.
Towards this end, we present a neural avatar system that
does full-body rendering and combines the ideas from the
classical computer graphics, namely the decoupling of ge-
ometry and texture, with the use of deep convolutional neu-
ral networks. In particular, similarly to the classic pipeline,
our system explicitly estimates the 2D textures of body
parts. The 2D texture within the classical pipeline effec-
tively transfers the appearance of the body fragments across
camera transformations and body articulations. Keeping
this component within the neural pipeline boosts general-
ization across such transforms. The role of the convolu-
tional network in our approach is then confined to predict-
ing the texture coordinates of individual pixels in the out-
put 2D image given the body pose and the camera parame-
ters (Figure 2). Additionally, the network predicts the body
foreground/background mask.
In our experiments, we compare the performance of our
textured neural avatar with a direct video-to-video trans-
lation approach [67], and show that explicit estimation of
textures brings additional generalization capability and im-
proves the realism of the generated images for new views
and/or when the amount of training data is limited.
2
2. Related work
Our approach is closely related to a vast number of pre-
vious works, and below we discuss a small subset of these
connections.
Building full-body avatars from image data has long
been one of the main topics of computer vision research.
Traditionally, an avatar is defined by a 3D geometric mesh
of a certain neutral pose, a texture, and a skinning mecha-
nism that transforms the mesh vertices according to pose
changes. A large group of works has been devoted to
body modeling from 3D scanners [51], registered multi-
view sequences [53] as well as from depth and RGB-D
sequences [7, 69, 74]. On the other extreme are methods
that fit skinned parametric body models to single images
[6, 8, 30, 35, 49, 50, 59]. Finally, research on building full-
body avatars from monocular videos has started [3, 4]. Sim-
ilarly to the last group of works, our work builds an avatar
from a video or a set of unregistered monocular videos. The
classical (computer graphics) approach to modeling human
avatars requires explicit physically-plausible modeling of
human skin, hair, sclera, clothing surface, as well as mo-
tion under pose changes. Despite considerable progress in
reflectivity modeling [2, 18, 38, 70, 72] and better skin-
ning/dynamic surface modeling [23, 44, 60], the computer
graphics approach still requires considerable “manual” ef-
fort of designers to achieve high realism [2] and to pass the
so-called uncanny valley [46], especially if real-time ren-
dering of avatars is required.
Image synthesis using deep convolutional neural net-
works is a thriving area of research [20, 27] and a lot of
recent effort has been directed onto synthesis of realistic hu-
man faces [15, 36, 61]. Compared to traditional computer
graphics representations, deep ConvNets model data by fit-
ting an excessive number of learnable weights to training
data. Such ConvNets avoid explicit modeling of the sur-
face geometry, surface reflectivity, or surface motion under
pose changes, and therefore do not suffer from the lack of
realism of the corresponding components. On the flipside,
the lack of ingrained geometric or photometric models in
this approach means that generalizing to new poses and in
particular to new camera views may be problematic. Still
a lot of progress has been made over the last several years
for the neural modeling of personalized talking head mod-
els [37, 43, 62], hair [68], hands [47]. Notably, the recent
system [43] has achieved very impressive results for neural
face rendering, while decomposing view-dependent texture
and 3D shape modeling.
Over the last several months, several groups have pre-
sented results of neural modeling of full bodies [1, 12, 42,
67]. While the presented results are very impressive, the ap-
proaches still require a large amount of training data. They
also assume that the test images are rendered with the same
camera views as the training data, which in our experience
makes the task considerably simpler than modeling body
appearance from an arbitrary viewpoint. In this work, we
aim to expand the neural body modeling approach to tackle
the latter, harder task. The work [45] uses a combination of
classical and neural rendering to render human body from
new viewpoints, but does so based on depth scans and there-
fore with a rather different algorithmic approach.
A number of recent works warp a photo of a person to a
new photorealistic image with modified gaze direction [24],
modified facial expression/pose [9, 55, 64, 71], or modified
body pose [5, 48, 56, 64], whereas the warping field is esti-
mated using a deep convolutional network (while the origi-
nal photo effectively serves as a texture). These approaches
are however limited in their realism and/or the amount of
change they can model, due to their reliance on a single
photo of a given person for its input. Our approach also
disentangles texture from surface geometry/motion mod-
eling but trains from videos, therefore being able to han-
dle harder problem (full body multi-view setting) and to
achieve higher realism.
Our system relies on the DensePose body surface param-
eterization (UV parameterization) similar to the one used in
the classical graphics-based representation. Part of our sys-
tem performs a mapping from the body pose to the surface
parameters (UV coordinates) of image pixels. This makes
our approach related to the DensePose approach [28] and
the earlier works [29, 63] that predict UV coordinates of
image pixels from the input photograph. Furthermore, our
approach uses DensePose results [28] for pretraining.
Our system is related to approaches that extract textures
from multi-view image collections [26, 39] or multi-view
video collections [66] or a single video [52]. Our approach
is also related to free-viewpoint video compression and ren-
dering systems, e.g. [11, 16, 21, 66]. Unlike those works,
ours is restricted to scenes containing a single human. At
the same time, our approach aims to generalize not only
a specific image location, e.g. Bij [x, y] denotes the scalar
element in the j-th map of the stack Bi located at location
(x, y), and Bi [x, y] denotes the vector of elements corre-
sponding to all maps sampled at location (x, y).
Input and output. In general, we are interested in syn-
thesizing images of a certain person given her/his pose. We
assume that the pose for the i-th image comes in the form of
3D joint positions defined in the camera coordinate frame.
As an input to the network, we then consider a map stack
Bi , where each map Bij contains the rasterized j-th segment
(bone) of the “stickman” (skeleton) projected on the camera
plane. To retain the information about the third coordinate
of the joints, we linearly interpolate the depth value between
the joints defining the segments, and use the interpolated
values to define the values in the map Bij corresponding to
the bone pixels (the pixels not covered by the j-th bone are
set to zero). Overall, the stack Bi incorporates the informa-
tion about the person and the camera pose.
As an output of the whole system, we expect an RGB
image (a three-channel stack) Ii and a single channel mask
Mi , defining the pixels that are covered by the avatar. Be-
low, we consider two approaches: the direct translation
baseline, which directly maps Bi into {Ii , Mi } and the tex-
tured neural avatar approach that performs such mapping
indirectly using texture mapping.
In both cases, at training time, we assume that for each
input frame i, the input joint locations and the “ground
truth” foreground mask are estimated, and we use 3D body
pose estimation and human semantic segmentation to ex-
tract them from raw video frames. At test time, given a
real or synthetic background image I˜i , we generate the fi-
nal view by first predicting Mi and Ii from the body pose
and then linearly blending the resulting avatar into an im-
to new camera views but also to new user poses unseen in
the training videos. The work of [73] is the most related
to ours in this group, as they warp the individual frames of
the multi-view video dataset according to the target pose to
generate new sequences. The poses that they can handle,
however, are limited by the need to have a close match in
the training set, which is a strong limitation given the com-
binatorial nature of the human pose configuration space.
3. Methods
Notation. We use the lower index i to denote objects that
are specific to the i-th training or test image. We use up-
percase notation, e.g. Bi to denote a stack of maps (a third-
order tensor/three-dimensional array) corresponding to the
i-th training or test image. We use the upper index to denote
a specific map (channel) in the stack, e.g. Bij . Furthermore,
we use square brackets to denote elements corresponding to
3
age: Iˆi = Ii  Mi + I˜i  (1 − Mi ) (where  defines a
“location-wise” product, i.e. the RGB values at each loca-
tion are multiplied by the mask value at this location).
Direct translation baseline. The direct approach that we
consider as a baseline to ours is to learn an image trans-
lation network that maps the map stack Bik to the map
stacks Ii and Mi (usually the two output stacks are pro-
duced within two branches that share the initial stage of the
processing [20]). Generally, mappings between stacks of
maps can be implemented using fully-convolutional archi-
tectures. Exact architectures and losses for such networks
is an active area of research [14, 31, 33, 65]. Very recent
works [1, 12, 42, 67] have used direct translation (with var-
ious modifications) to synthesize the view of a person for
a fixed camera. We use the video-to-video variant of this
approach [67] as a baseline for our method.
 Part assignments Predicted mask Ground truth mask
Cross-entropy
loss
Input pose Generator

Perceptual
loss
Render

Part coordinates Predicted RGB Ground truth RGB

Texture stack

Figure 2: The overview of the textured neural avatar system. The input pose is defined as a stack of ”bone” rasterizations
(one bone per channel; here we show it as a skeleton image). The input is processed by the fully-convolutional network
(generator) to produce the body part assignment map stack and the body part coordinate map stack. These stacks are then
used to sample the body texture maps at the locations prescribed by the part coordinate stack with the weights prescribed by
the part assignment stack to produce the RGB image. In addition, the last body assignment stack map corresponds to the
background probability. During learning, the mask and the RGB image are compared with ground-truth and the resulting
losses are backpropagated through the sampling operation into the fully-convolutional network and onto the texture, resulting
in their updates.

Textured neural avatar. The direct translation approach
relies on the generalization ability of ConvNets and incor-
porates very little domain-specific knowledge into the sys-
tem. As an alternative, we suggest the textured avatar ap-
proach, that explicitly estimates the textures of body parts,
thus ensuring the similarity of the body surface appearance
under varying pose and cameras.
Following the DensePose approach [28], we subdivide
the body into n=24 parts, where each part has a 2D param-
eterization. Each body part also has the texture map T k ,
which is a color image of a fixed pre-defined size (256×256
in our implementation). The training process for the tex-
tured neural avatar estimates personalized part parameteri-
zations and textures.
Again, following the DensePose approach, we assume
that each pixel in an image of a person is (soft)-assigned
to one of n parts or to the background and with a specific
location on the texture of that part (body part coordinates).
Unlike DensePose, where part assignments and body part
coordinates are induced from the image, our approach at
test time aims to predict them based solely on the pose Bi .
The introduction of the body surface parameterization
outlined above changes the translation problem. For a
given pose defined by Bi , the translation network now has
to predict the stack Pi of body part assignments and the
stack Ci of body part coordinates, where Pi contains n+1
maps
Pn of knon-negative numbers that sum to identity (i.e.
k=0 Pi [x, y] = 1 for any position (x, y)), and Ci con-
tains 2n maps of real numbers between 0 and w, where w is
the spatial size (width and height) of the texture maps T k .
The map channel Pik for k = 0, . . . , n−1 is then in-
terpreted as the probability of the pixel to belong to the k-
th body part, and the map channel Pin corresponds to the
probability of the background. The coordinate maps Ci2k
and Ci2k+1 correspond to the pixel coordinates on the k-th
body part. Specifically, once the part assignments Pi and
body part coordinates Ci are predicted, the image Ii at each
pixel (x, y) is reconstructed as a weighted combination of
texture elements, where the weights and texture coordinates
are prescribed by the part assignment maps and the coordi-
nate maps correspondingly:
n−1
X
s(Pi , Ci , T )[x, y] = Pik [x, y]·
k=0
T k Ci2k [x, y], Ci2k+1 [x, y] ,
 
(1)
where s(·, ·, ·) is the sampling function (layer) that outputs
the RGB map stack given the three input arguments. In (1),
the texture maps T k are sampled at non-integer locations
(Ci2k [x, y], Ci2k+1 [x, y]) in a piecewise-differentiable man-
ner using bilinear interpolation [32].
When training the neural textured avatar, we learn a con-
volutional network gφ with learnable parameters φ to trans-
late the input map stacks Bi into the body part assignments
and the body part coordinates. As gφ has two branches
(“heads”), we denote with gφP the branch that produces the
body part assignments stack, and with gφC the branch that
produces the body part coordinates. To learn the parameters
of the textured neural avatar, we optimize the loss between
the generated image and the ground truth image I¯i :


Limage (φ, T ) = dImage I¯i , s gφP (Bi ), gφC (Bi ), T (2)
where dImage (·, ·) is a loss used to compare two images.
In our current implementation we use a simple perceptual

4
loss [25, 33, 65], which computes the maps of activations
within pretrained fixed VGG network [58] for both im-
ages and evaluates the L1-norm between the resulting maps
(Conv1,6,11,20,29 of VGG19 were used). More ad-
vanced adversarial losses [27] popular in image translation
[19, 31] can also be used here.
During the stochastic optimization, the gradient of the
loss (2) is backpropagated through (1) both into the trans-
lation network gφ and onto the texture maps T k , so that
minimizing this loss updates not only the network param-
eters but also the textures themselves. As an addition, the
learning also optimizes the mask loss that measures the dis-
crepancy between the ground truth background mask 1−M̄i
and the background mask prediction:
Figure 3: The impact of the learning on the texture (top,
shown for the same subset of maps T k ) and on the convolu-
 
Lmask (φ, T ) = dBCE 1̄ − Mi , gφP (Bi )n (3)
tional network gφC predictions (bottom, shown for the same
pair of input poses). Left part shows the starting state (af-
where dBCE is the binary cross-entropy loss, and gφP (Bi )n
ter initialization), while the right part shows the final state,
corresponds to the n-th (i.e. background) channel of the pre-
which is considerably different from the start.
dicted part assignment map stack. After backpropagation
of the weighted combination of (2) and (3), the network
parameters φ and the textures maps T k are updated. As person, and they change significantly during the end-to-end
the training progresses, the texture maps change (Figure 2), learning (Figure 3).
and so does the body part coordinate predictions, so that the
learning is free to choose the appropriate parameterization
of body part surfaces. 4. Experiments
Below, we discuss the details of the experimental vali-
Initialization of textured neural avatar. The success of dation, provide comparison with baseline approaches, and
our network depends on the initialization strategy. When show qualitative results. The project webpage1 contains
training from multiple video sequences, we use the Dense- more videos of the learned avatars.
Pose system [28] to initialize the textured neural avatar.
Specifically, we run DensePose on the training data and pre-
train gφ as a translation network between the pose stacks Bi Architecture. We input 3D pose via bone rasterizations,
and the DensePose outputs. where each bone, hand and face are drawn in separate
An alternative way that is particularly attractive when channels. We then use standard image translation archi-
training data is scarce is to initialize the avatar is through tecture [33] to perform a mapping from these bones’ ras-
transfer learning. In this case, we simply take gφ from an- terizations to texture assignments and coordinates. This ar-
other avatar trained on abundant data. The explicit decou- chitecture consists of downsampling layers, stack of resid-
pling of geometry from appearance in our method facilitates ual blocks, operating at low dimensional feature representa-
transfer learning, as the geometrical mapping provided by tions, and upsampling layers. We then split the network into
the network gφ usually does not need to change much be- two roughly equal parts: encoder and decoder, with texture
tween two people, especially if the body types are not too assignments and coordinates having separate decoders. We
dissimilar. use 4 downsampling and upsampling layers with initial 32
channels in the convolutions and 256 channels in the resid-
Once the mapping gφ has been initialized, the texture
ual blocks. The ConvNet gφ has 17 million parameters.
maps T k are initialized as follows. Each pixel in the train-
ing image is assigned to a single body part (according to the
prediction of the pretrained gφP ) and to a particular texture Datasets. We train neural avatars on several types of
pixel on the texture of the corresponding part (according datasets. First, we consider collections of multi-view videos
to the prediction of the pretrained gφC ). Then, the value of registered in time and space, where 3D pose estimates can
each texture pixel is initialized to the mean of all image pix- be obtained via triangulation of 2D poses. We use two sub-
els assigned to it (the texture pixels assigned zero pixels are sets (corresponding to two persons from the 171026 pose2
initialized to black). The initialized texture T and gφ usu-
ally produce images that are only coarsely reminding the 1 https://saic-violet.github.io/texturedavatar/

5
Figure 4: Renderings produced by multiple textured neural avatars (for all people in our study). All renderings are produced
from the new viewpoints unseen during training.

(a) User study (b) SSIM score (c) Frechet distance

Ours-v-V2V Ours-v-Direct V2V Direct Ours V2V Direct Ours
CMU1-16 0.56 0.75 0.908 0.899 0.919 6.7 7.3 8.8
CMU2-16 0.54 0.74 0.916 0.907 0.922 7.0 8.8 10.7
CMU1-6 0.50 0.92 0.905 0.896 0.914 7.7 10.7 8.9
CMU2-6 0.53 0.71 0.918 0.907 0.920 7.0 9.7 10.4

Table 1: Quantitative comparison of the three models operating on different datasets (see text for discussion).

scene) from the CMU Panoptic dataset collection [34], re-
ferring to them as CMU1 and CMU2 (both subsets have ap-
proximately four minutes / 7,200 frames in each camera
view). We consider two regimes: training on 16 cameras
(CMU1-16 and CMU2-16) or six cameras (CMU1-6 and
CMU2-6). The evaluation is done on the hold-out cameras
and hold-out parts of the sequence (no overlap between train
and test in terms of the cameras or body motion).
We have also captured our own multi-view sequences
of three subjects using a rig of seven cameras, spanning
approximately 30◦ . In one scenario, the training sets in-
cluded six out of seven cameras, where the duration of each
video was approximately six minutes (11,000 frames). We
show qualitative results for the hold-out camera as well as
from new viewpoints. In the other scenario described below,
training was done based on a video from a single camera.
Finally, we evaluate on two short monocular sequences
from [4] and a Youtube video in Figure 7.
Pre-processing. Our system expects 3D human pose as
input. For non-CMU datasets, we used the OpenPose-
compatible [10, 57] 3D pose formats, represented by
25 body joints, 21 joints for each hand and 70 facial land-
marks. For the CMU Panoptic datasets, we use the available
3D pose annotation as input (which has 19 rather than 25
body joints). To get a 3D pose for non-CMU sequences we
first apply the OpenPose 2D pose estimation engine to five
consecutive frames of the monocular RGB image sequence.
Then we concatenate and lift the estimated 2D poses to infer
the 3D pose of the last frame by using a multi-layer percep-
tron model. The perceptron is trained on the CMU 3D pose
annotations (augmented with position of the feet joints by
triangulating the output of OpenPose) in orthogonal projec-
tion.
For foreground segmentation we use DeepLabv3+ with
Xception-65 backbone [13] initially trained on PAS-
CAL VOC 2012 [22] and fine-tuned on HumanParsing
dataset [40, 41] to predict initial human body segmentation
masks. We additionally employ GrabCut [54] with back-
ground/foreground model initialized by the masks to refine
object boundaries on the high-resolution images. Pixels
covered by the skeleton rasterization were always added to
the foreground mask.
Baselines. In the multi-video training scenario, we con-
sider two other systems, against which ours is compared.
First, we take the video-to-video (V2V) system [67], using
the authors’ code with minimal modifications that lead to
improved performance. We provide it with the same input
as ours, and we use images with blacked-out background
(according to our segmentation) as desired output. On the
CMU1-6 task, we have also evaluated a model with Dense-
Pose results computed on the target frame given as input
(alongside keypoints). Despite much stronger (oracle-type)

6
 GT Direct V2V Proposed
Figure 5: Comparison of the rendering quality for the Direct, V2V and proposed methods on the CMU1-6 and CMU2-6
sequences. Images from six arbitrarily chosen cameras were used for training. We generate the views onto the hold-out
cameras which were not used during training. The pose and camera in the lower right corner are in particular difficult for all
the systems.
conditioning, the performance of this model in terms of con-
sidered metrics has not improved in comparison with V2V
that uses only body joints as input.
The video-to-video system employs several adversarial
losses and an architecture different from ours. Therefore we
consider a more direct ablation (Direct), which has the same
network architecture that predicts RGB color and mask di-
rectly, rather than via body part assignments/coordinates.
The Direct system is trained using the same losses and in
the same protocol as ours.
As for the single video case, two baseline systems,
against which ours is compared, were considered. On our
own captured sequences, we compare our system against
video-to-video (V2V) system [67], whereas on sequences
from [4] we provide a qualitative comparison against the
system of [4].
Multi-video comparison. We compare the three systems
(ours, V2V, Direct) in CMU1-16, CMU2-16, CMU1-6,
CMU2-6. Using the hold-out sequences/motions, we then
evaluated two popular metrics, namely structured self-
similarity (SSIM) and Frechet Inception Distance (FID) be-
tween the results of each system and the hold-out frames
(with background removed using our segmentation algo-
rithm). Our method outperforms the other two in terms of
SSIM and underperforms V2V in terms of FID. Represen-
tative examples are shown in Figure 5.
We have also performed user study using a crowd-
sourcing website, where the users were shown the results of
ours and one of the other two systems on either side of the
ground truth image and were asked to pick a better match to
the middle image. In the side-by-side comparison, the re-
sults of our method were always preferred by the majority
of crowd-sourcing users. We note that our method suffers
7
GT Direct V2V Proposed
from a disadvantage both in the quantitative metrics and in
the user comparison, since it averages out lighting from dif-
ferent viewpoints. The more detailed quantitative compari-
son is presented in Table 1.
We show more qualitative examples of our method for a
variety of models in Figure 4 and some qualitative compar-
isons with baselines in Figure 6.
Single video comparisons. We also evaluate our system
in a single video case. We consider the scenario, where we
train the model and transfer it to a new person by fitting it
to a single video. We use single-camera videos from one
of the cameras in our rig. We then evaluate the model (and
V2V baseline) on a hold-out set of poses projected onto the
camera from the other side of the rig (around 30◦ away).
We thus demonstrate that new models can be obtained us-
ing a single monocular video. For our models, we consider
transferring from CMU1-16.
We thus pretrain V2V and our system on CMU1-16 and
use the obtained weights of gφ as initialization for fine-
tuning to the single video in our dataset. The texture maps
are initialized from scratch as described above. Evaluating
on hold-out camera and motion highlighted strong advan-
tage of our method. In the user study on two subjects, the
result of our method has been preferred to V2V in 55% and
65% of the cases. We further compare our method and the
system of [4] on the sequences from [4]. The qualitative
comparison is shown in Figure 7. In addition, we gener-
ate an avatar from a YouTube video. In this set of exper-
iments, the avatars were obtained by fine-tuning from the
same avatar (shown in Figure 6–left). Except for the con-
siderable artefacts on hand parts, our system has generated
avatars that can generalize to new pose despite very short
video input (300 frames in the case of [4]).
 GT Proposed V2V GT Proposed V2V

Figure 6: Results comparison for our multi-view sequences using a hold-out camera. Textured Neural Avatars and the images
produced by the video-to-video (V2V) system correspond to the same viewpoint. Both systems use a video from a single
viewpoint for training. Electronic zoom-in recommended.

Figure 7: Results on external monocular sequences. Rows 1-2: avatars for sequences from [4] in an unseen pose (left – ours,
right – [4]). Row 3 – the textured avatar computed from a popular YouTube video (’PUMPED UP KICKS DUBSTEP’). In
general, our system is capable of learning avatars from monocular videos.

8
5. Summary and Discussion
We have presented textured neural avatar approach to
model the appearance of humans for new camera views and
new body poses. Our system takes the middle path between
the recent generation of methods that use ConvNets to map
the pose to the image directly, and the traditional approach
that uses geometric modeling of the surface and superim-
pose the personalized texture maps. This is achieved by
learning a ConvNet that predicts texture coordinates of pix-
els in the new view jointly with the texture within the end-
to-end learning process. We demonstrate that retaining an
explicit shape and texture separation helps to achieve better
generalization than direct mapping approaches.
Our method suffers from certain limitations. The gen-
eralization ability is still limited, as it does not generalize
well when a person is rendered at a scale that is consid-
erably different from the training set (which can be par-
tially addressed by rescaling prior to rendering followed by
cropping/padding postprocessing). Furthermore, textured
avatars exhibit strong artefacts in the presence of pose es-
timation errors on hands and faces. Finally, our method as-
sumes constancy of the surface color and ignores lighting
effects. This can be potentially addressed by making our
textures view- and lighting-dependent [17, 43].
References
[1] Kfir Aberman, Mingyi Shi, Jing Liao, Dani Lischinski, Bao-
quan Chen, and Daniel Cohen-Or. Deep video-based perfor-
mance cloning. arXiv preprint arXiv:1808.06847, 2018. 2,
3
[2] Oleg Alexander, Mike Rogers, William Lambeth, Jen-Yuan
Chiang, Wan-Chun Ma, Chuan-Chang Wang, and Paul De-
bevec. The Digital Emily project: Achieving a photorealistic
digital actor. IEEE Computer Graphics and Applications,
30(4):20–31, 2010. 2
[3] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian
Theobalt, and Gerard Pons-Moll. Detailed human avatars
from monocular video. In 2018 International Conference on
3D Vision (3DV), pages 98–109. IEEE, 2018. 2
[4] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian
Theobalt, and Gerard Pons-Moll. Video based reconstruction
of 3d people models. In Proc. CVPR, June 2018. 2, 6, 7, 8
[5] Guha Balakrishnan, Amy Zhao, Adrian V. Dalca, Frédo Du-
rand, and John V. Guttag. Synthesizing images of humans in
unseen poses. In Proc. CVPR, pages 8340–8348, 2018. 3
[6] Alexandru O Bălan and Michael J Black. The naked truth:
Estimating body shape under clothing. In Proc. ECCV, pages
15–29. Springer, 2008. 2
[7] Federica Bogo, Michael J Black, Matthew Loper, and Javier
Romero. Detailed full-body reconstructions of moving peo-
ple from monocular RGB-D sequences. In Proc. ICCV,
pages 2300–2308, 2015. 2
[8] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter
Gehler, Javier Romero, and Michael J Black. Keep it smpl:
9
Automatic estimation of 3d human pose and shape from a
single image. In Proc. ECCV, pages 561–578. Springer,
2016. 2
[9] Jie Cao, Yibo Hu, Hongwen Zhang, Ran He, and Zhenan
Sun. Learning a high fidelity pose invariant model
for high-resolution face frontalization. arXiv preprint
arXiv:1806.08472, 2018. 3
[10] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.
Realtime multi-person 2d pose estimation using part affinity
fields. In Proc. CVPR, 2017. 6
[11] Dan Casas, Marco Volino, John Collomosse, and Adrian
Hilton. 4d video textures for interactive character appear-
ance. In Computer Graphics Forum, volume 33, pages 371–
380. Wiley Online Library, 2014. 3
[12] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and
Alexei A Efros. Everybody dance now. arXiv preprint
arXiv:1808.07371, 2018. 2, 3
[13] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian
Schroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
Proc. ECCV, 2018. 6
[14] Qifeng Chen and Vladlen Koltun. Photographic image syn-
thesis with cascaded refinement networks. In Proc. ICCV,
pages 1520–1529, 2017. 3
[15] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha,
Sunghun Kim, and Jaegul Choo. Stargan: Unified genera-
tive adversarial networks for multi-domain image-to-image
translation. In Proc. CVPR, June 2018. 2
[16] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Den-
nis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk,
and Steve Sullivan. High-quality streamable free-viewpoint
video. ACM Transactions on Graphics (TOG), 34(4):69,
2015. 3
[17] Paul E. Debevec, Yizhou Yu, and George Borshukov. Effi-
cient view-dependent image-based rendering with projective
texture-mapping. In Rendering Techniques ’98, Proceedings
of the Eurographics Workshop in Vienna, Austria, June 29 -
July 1, 1998, pages 105–116, 1998. 9
[18] Craig Donner, Tim Weyrich, Eugene d’Eon, Ravi Ra-
mamoorthi, and Szymon Rusinkiewicz. A layered, heteroge-
neous reflectance model for acquiring and rendering human
skin. In ACM Transactions on Graphics (TOG), volume 27,
page 140. ACM, 2008. 2
[19] Alexey Dosovitskiy and Thomas Brox. Generating images
with perceptual similarity metrics based on deep networks.
In Proc. NIPS, pages 658–666, 2016. 5
[20] Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas
Brox. Learning to generate chairs with convolutional neural
networks. In Proc. CVPR, pages 1538–1546, 2015. 2, 3
[21] Mingsong Dou, Philip Davidson, Sean Ryan Fanello, Sameh
Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir
Tankovich, and Shahram Izadi. Motion2fusion: real-time
volumetric performance capture. ACM Transactions on
Graphics (TOG), 36(6):246, 2017. 3
[22] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I.
Williams, J. Winn, and A. Zisserman. The pascal visual ob-
ject classes challenge: A retrospective. International Journal
of Computer Vision, 111(1):98–136, Jan. 2015. 6
[23] Andrew Feng, Dan Casas, and Ari Shapiro. Avatar reshap-
ing and automatic rigging using a deformable model. In Pro-
ceedings of the 8th ACM SIGGRAPH Conference on Motion
in Games, pages 57–64. ACM, 2015. 2
[24] Yaroslav Ganin, Daniil Kononenko, Diana Sungatullina, and
Victor Lempitsky. Deepwarp: Photorealistic image resynthe-
sis for gaze manipulation. In Proc. ECCV, pages 311–326.
Springer, 2016. 2, 3
[25] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.
Image style transfer using convolutional neural networks. In
Proc. CVPR, pages 2414–2423, 2016. 5
[26] Bastian Goldlücke and Daniel Cremers. Superresolution
texture maps for multiview reconstruction. In Proc. ICCV,
pages 1677–1684, 2009. 3
[27] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and
Yoshua Bengio. Generative adversarial nets. In Proc. NIPS,
pages 2672–2680, 2014. 2, 5
[28] Riza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.
DensePose: Dense human pose estimation in the wild. In
Proc. CVPR, June 2018. 3, 4, 5
[29] Riza Alp Güler, George Trigeorgis, Epameinondas Anton-
akos, Patrick Snape, Stefanos Zafeiriou, and Iasonas Kokki-
nos. DenseReg: Fully convolutional dense shape regression
in-the-wild. In Proc. CVPR, volume 2, page 5, 2017. 3
[30] Nils Hasler, Hanno Ackermann, Bodo Rosenhahn, Thorsten
Thormählen, and Hans-Peter Seidel. Multilinear pose and
body shape estimation of dressed subjects from image sets.
In Proc. CVPR, pages 1823–1830. IEEE, 2010. 2
[31] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A.
Efros. Image-to-image translation with conditional adver-
sarial networks. In Proc. CVPR, pages 5967–5976, 2017. 3,
5 capture with real-time neural re-rendering. ACM Trans.
[32] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and
Koray Kavukcuoglu. Spatial transformer networks. In Proc.
NIPS, pages 2017–2025, 2015. 4
[33] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual
losses for real-time style transfer and super-resolution. In
Proc. ECCV, pages 694–711, 2016. 3, 5
[34] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei
Tan, Lin Gui, Sean Banerjee, Timothy Scott Godisart, Bart
Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and
Yaser Sheikh. Panoptic studio: A massively multiview sys-
tem for social interaction capture. IEEE Transactions on Pat-
tern Analysis and Machine Intelligence, 2017. 6
[35] Angjoo Kanazawa, Michael J Black, David W Jacobs, and
Jitendra Malik. End-to-end recovery of human shape and
pose. In Proc. CVPR, 2018. 2
[36] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of GANs for improved quality, stabil-
ity, and variation. In International Conference on Learning
Representations, 2018. 2
[37] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng
Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian
Richardt, Michael Zollhöfer, and Christian Theobalt. Deep
video portraits. arXiv preprint arXiv:1805.11714, 2018. 2
10
[38] Oliver Klehm, Fabrice Rousselle, Marios Papas, Derek
Bradley, Christophe Hery, Bernd Bickel, Wojciech Jarosz,
and Thabo Beeler. Recent advances in facial appearance
capture. In Computer Graphics Forum, volume 34, pages
709–733. Wiley Online Library, 2015. 2
[39] Victor S. Lempitsky and Denis V. Ivanov. Seamless mosaic-
ing of image-based texture maps. In Proc. CVPR, 2007. 3
[40] Xiaodan Liang, Si Liu, Xiaohui Shen, Jianchao Yang, Luoqi
Liu, Jian Dong, Liang Lin, and Shuicheng Yan. Deep human
parsing with active template regression. Pattern Analysis and
Machine Intelligence, IEEE Transactions on, 37(12):2402–
2414, Dec 2015. 6
[41] Xiaodan Liang, Chunyan Xu, Xiaohui Shen, Jianchao Yang,
Si Liu, Jinhui Tang, Liang Lin, and Shuicheng Yan. Iccv.
2015. 6
[42] Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo
Kim, Florian Bernard, Marc Habermann, Wenping Wang,
and Christian Theobalt. Neural animation and reenactment
of human actor videos. arXiv preprint arXiv:1809.03658,
2018. 2, 3
[43] Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser
Sheikh. Deep appearance models for face rendering. ACM
Transactions on Graphics (TOG), 37(4):68, 2018. 2, 9
[44] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard
Pons-Moll, and Michael J Black. Smpl: A skinned multi-
person linear model. ACM Transactions on Graphics (TOG),
34(6):248, 2015. 2
[45] Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel
Pidlypenskyi, Jonathan Taylor, Julien P. C. Valentin, Sameh
Khamis, Philip L. Davidson, Anastasia Tkach, Peter Lin-
coln, Adarsh Kowdle, Christoph Rhemann, Dan B. Gold-
man, Cem Keskin, Steven M. Seitz, Shahram Izadi, and
Sean Ryan Fanello. LookinGood: enhancing performance
Graph., 37(6):255:1–255:14, 2018. 3
[46] Masahiro Mori. The uncanny valley. Energy, 7(4):33–35,
1970. 2
[47] Franziska Mueller, Florian Bernard, Oleksandr Sotny-
chenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and
Christian Theobalt. GANerated hands for real-time 3d hand
tracking from monocular RGB. In Proc. CVPR, June 2018.
2
[48] Natalia Neverova, Riza Alp Güler, and Iasonas Kokkinos.
Dense pose transfer. In Proc. ECCV, September 2018. 3
[49] Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Pe-
ter V. Gehler, and Bernt Schiele. Neural body fitting: Uni-
fying deep learning and model-based human pose and shape
estimation. Verona, Italy, 2018. 2
[50] Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas
Daniilidis. Learning to estimate 3d human pose and shape
from a single color image. In Proc. CVPR, June 2018. 2
[51] Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and
Michael J Black. Dyna: A model of dynamic human shape in
motion. ACM Transactions on Graphics (TOG), 34(4):120,
2015. 2
[52] Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, and An-
drew W. Fitzgibbon. Unwrap mosaics: a new representation
 for video editing. ACM Trans. Graph., 27(3):17:1–17:11, [69] Alexander Weiss, David Hirshberg, and Michael J Black.
2008. 3 Home 3d body scans from noisy image and range data. In
[53] Nadia Robertini, Dan Casas, Edilson De Aguiar, and Chris- Proc. ICCV, pages 1951–1958. IEEE, 2011. 2
tian Theobalt. Multi-view performance capture of sur- [70] Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd
face details. International Journal of Computer Vision, Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho
124(1):96–113, 2017. 2 Lee, Addy Ngan, Henrik Wann Jensen, et al. Analysis of
[54] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. human faces using a measurement-based skin reflectance
”grabcut”: interactive foreground extraction using iterated model. In ACM Transactions on Graphics (TOG), vol-
graph cuts. ACM Trans. Graph., 23(3):309–314, 2004. 6 ume 25, pages 1013–1024. ACM, 2006. 2
[55] Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris [71] Olivia Wiles, A. Sophia Koepke, and Andrew Zisserman.
Samaras, Nikos Paragios, and Iasonas Kokkinos. Deform- X2face: A network for controlling face generation using im-
ing autoencoders: Unsupervised disentangling of shape and ages, audio, and pose codes. In Proc. ECCV, September
appearance. In Proc. ECCV, September 2018. 3 2018. 3
[56] Aliaksandr Siarohin, Enver Sangineto, Stphane Lathuilire, [72] Erroll Wood, Tadas Baltrusaitis, Xucong Zhang, Yusuke
and Nicu Sebe. Deformable gans for pose-based human im- Sugano, Peter Robinson, and Andreas Bulling. Rendering
age generation. In Proc. CVPR, June 2018. 3 of eyes for eye-shape registration and gaze estimation. In
[57] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Proc. ICCV, pages 3756–3764, 2015. 2
Sheikh. Hand keypoint detection in single images using mul- [73] Feng Xu, Yebin Liu, Carsten Stoll, James Tompkin, Gau-
tiview bootstrapping. In CVPR, 2017. 6 rav Bharaj, Qionghai Dai, Hans-Peter Seidel, Jan Kautz,
[58] Karen Simonyan and Andrew Zisserman. Very deep convo- and Christian Theobalt. Video-based characters: creating
lutional networks for large-scale image recognition. CoRR, new human performances from a multi-view video database.
abs/1409.1556, 2014. 5 ACM Transactions on Graphics (TOG), 30(4):32, 2011. 3
[59] J Starck and A Hilton. Model-based multiple view recon- [74] Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai
struction of people. In Proc. ICCV, pages 915–922, 2003. Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu. Doublefu-
2 sion: Real-time capture of human performances with inner
[60] Ian Stavness, C Antonio Sánchez, John Lloyd, Andrew Ho, body shapes from a single depth sensor. In Proc. CVPR,
Johnty Wang, Sidney Fels, and Danny Huang. Unified skin- pages 7287–7296. IEEE Computer Society, 2018. 2
ning of rigid and deformable models for anatomical simu-
lations. In SIGGRAPH Asia 2014 Technical Briefs, page 9.
ACM, 2014. 2
[61] Diana Sungatullina, Egor Zakharov, Dmitry Ulyanov, and
Victor Lempitsky. Image manipulation with perceptual dis-
criminators. In Proc. ECCV, September 2018. 2
[62] Supasorn Suwajanakorn, Steven M Seitz, and Ira
Kemelmacher-Shlizerman. Synthesizing Obama: learning
lip sync from audio. ACM Transactions on Graphics (TOG),
36(4):95, 2017. 2
[63] Jonathan Taylor, Jamie Shotton, Toby Sharp, and Andrew
Fitzgibbon. The vitruvian manifold: Inferring dense corre-
spondences for one-shot human pose estimation. In Proc.
CVPR, pages 103–110. IEEE, 2012. 3
[64] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan
Kautz. Mocogan: Decomposing motion and content for
video generation. In Proc. CVPR, June 2018. 3
[65] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Vic-
tor S. Lempitsky. Texture networks: Feed-forward synthesis
of textures and stylized images. In Proc. ICML, pages 1349–
1357, 2016. 3, 5
[66] Marco Volino, Dan Casas, John P Collomosse, and Adrian
Hilton. Optimal representation of multi-view video. In Proc.
BMVC, 2014. 3
[67] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu,
Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-
video synthesis. arXiv preprint arXiv:1808.06601, 2018. 2,
3, 6, 7
[68] Lingyu Wei, Liwen Hu, Vladimir Kim, Ersin Yumer, and
Hao Li. Real-time hair rendering using sequential adversarial
networks. In Proc. ECCV, September 2018. 2

11
 DSFD: Dual Shot Face Detector

Jian Li† Yabiao Wang‡ Changan Wang‡ Ying Tai‡

Jianjun Qian†∗ Jian Yang†∗ Chengjie Wang‡ Jilin Li‡ Feiyue Huang‡
†
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education
†
Jiangsu Key Lab of Image and Video Understanding for Social Security
†
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China
arXiv:1810.10220v3 [cs.CV] 6 Apr 2019

‡
Youtu Lab, Tencent
†
lijiannuist@gmail.com, {csjqian, csjyang}@njust.edu.cn
‡
{casewang, changanwang, yingtai, jasoncjwang, jerolinli, garyhuang}@tencent.com

Scale Blurry Illumination

Pose & Occlusion Reflection Makeup

Figure 1: Visual results. Our method is robust to various variations on scale, blurry, illumination, pose, occlusion, reflection and makeup.

Abstract mentation to provide better initialization for the regressor.

In this paper, we propose a novel face detection network
with three novel contributions that address three key aspects
of face detection, including better feature learning, progres-
sive loss design and anchor assign based data augmenta-
tion, respectively. First, we propose a Feature Enhance
Module (FEM) for enhancing the original feature maps to
extend the single shot detector to dual shot detector. Sec-
ond, we adopt Progressive Anchor Loss (PAL) computed by
two different sets of anchors to effectively facilitate the fea-
tures. Third, we use an Improved Anchor Matching (IAM)
by integrating novel anchor assign strategy into data aug-
∗ Jianjun
Qian and Jian Yang are corresponding authors. This work
was supported by the National Science Fund of China under Grant Nos.
61876083, U1713208, and Program for Changjiang Scholars. This work
was done when Jian Li was an intern at Tencent Youtu Lab.
Since these techniques are all related to the two-stream de-
sign, we name the proposed network as Dual Shot Face De-
tector (DSFD). Extensive experiments on popular bench-
marks, WIDER FACE and FDDB, demonstrate the superi-
ority of DSFD over the state-of-the-art face detectors.
1. Introduction
Face detection is a fundamental step for various facial
applications, like face alignment [26], parsing [3], recog-
nition [34], and verification [6]. As the pioneering work
for face detection, Viola-Jones [29] adopts AdaBoost algo-
rithm with hand-crafted features, which are now replaced by
deeply learned features from the convolutional neural net-
work (CNN) [10] that achieves great progress. Although

1
the CNN based face detectors have being extensively stud-
ied, detecting faces with high degree of variability in scale,
pose, occlusion, expression, appearance and illumination in
real-world scenarios remains a challenge.
Previous state-of-the-art face detectors can be roughly
divided into two categories. The first one is mainly based
on the Region Proposal Network (RPN) adopted in Faster
RCNN [24] and employs two stage detection schemes [30,
33, 36]. RPN is trained end-to-end and generates high-
quality region proposals which are further refined by Fast
R-CNN detector. The other one is Single Shot Detec-
tor (SSD) [20] based one-stage methods, which get rid of
RPN, and directly predict the bounding boxes and confi-
dence [4, 27, 39]. Recently, one-stage face detection frame-
work has attracted more attention due to its higher inference
efficiency and straightforward system deployment.
Despite the progress achieved by the above methods,
there are still some problems existed in three aspects:
Feature learning Feature extraction part is essential for
a face detector. Currently, Feature Pyramid Network
(FPN) [17] is widely used in state-of-the-art face detectors
for rich features. However, FPN just aggregates hierarchi-
cal feature maps between high and low-level output layers,
which does not consider the current layer’s information, and
the context relationship between anchors is ignored.
Loss design The conventional loss functions used in object
detection include a regression loss for the face region and
a classification loss for identifying if a face is detected or
not. To further address the class imbalance problem, Lin et
al. [18] propose Focal Loss to focus training on a sparse set
of hard examples. To use all original and enhanced features,
Zhang et al. propose Hierarchical Loss to effectively learn
the network [37]. However, the above loss functions do not
consider progressive learning ability of feature maps in both
of different levels and shots.
Anchor matching Basically, pre-set anchors for each fea-
ture map are generated by regularly tiling a collection of
boxes with different scales and aspect ratios on the image.
Some works [27, 39] analyze a series of reasonable anchor
scales and anchor compensation strategy to increase posi-
tive anchors. However, such strategy ignores random sam-
pling in data augmentation, which still causes imbalance be-
tween positive and negative anchors.
In this paper, we propose three novel techniques to ad-
dress the above three issues, respectively. First, we intro-
duce a Feature Enhance Module (FEM) to enhance the dis-
criminability and robustness of the features, which com-
bines the advantages of the FPN in PyramidBox and Re-
ceptive Field Block (RFB) in RFBNet [19]. Second, moti-
vated by the hierarchical loss [37] and pyramid anchor [27]
in PyramidBox, we design Progressive Anchor Loss (PAL)
that uses progressive anchor sizes for not only different lev-
els, but also different shots. Specifically, we assign smaller
anchor sizes in the first shot, and use larger sizes in the
second shot. Third, we propose Improved Anchor Match-
ing (IAM), which integrates anchor partition strategy and
anchor-based data augmentation to better match anchors
and ground truth faces, and thus provides better initializa-
tion for the regressor. The three aspects are complementary
so that these techniques can work together to further im-
prove the performance. Besides, since these techniques are
all related to two-stream design, we name the proposed net-
work as Dual Shot Face Detector (DSFD). Fig. 1 shows the
effectiveness of DSFD on various variations, especially on
extreme small faces or heavily occluded faces.
In summary, the main contributions of this paper include:
• A novel Feature Enhance Module to utilize different
level information and thus obtain more discriminability and
robustness features.
• Auxiliary supervisions introduced in early layers via a
set of smaller anchors to effectively facilitate the features.
• An improved anchor matching strategy to match an-
chors and ground truth faces as far as possible to provide
better initialization for the regressor.
• Comprehensive experiments conducted on popular
benchmarks FDDB and WIDER FACE to demonstrate the
superiority of our proposed DSFD network compared with
the state-of-the-art methods.
2. Related work
We review the prior works from three perspectives.
Feature Learning Early works on face detection mainly
rely on hand-crafted features, such as Harr-like fea-
tures [29], control point set [1], edge orientation his-
tograms [13]. However, hand-crafted features design is lack
of guidance. With the great progress of deep learning, hand-
crafted features have been replaced by Convolutional Neu-
ral Networks (CNN). For example, Overfeat [25], Cascade-
CNN [14], MTCNN [38] adopt CNN as a sliding window
detector on image pyramid to build feature pyramid. How-
ever, using an image pyramid is slow and memory ineffi-
cient. As the result, most two stage detectors extract fea-
tures on single scale. R-CNN [7, 8] obtains region propos-
als by selective search [28], and then forwards each nor-
malized image region through a CNN to classify. Faster
R-CNN [24], R-FCN [5] employ Region Proposal Network
(RPN) to generate initial region proposals. Besides, ROI-
pooling [24] and position-sensitive RoI pooling [5] are ap-
plied to extract features from each region.
More recently, some research indicates that multi-scale
features perform better for tiny objects. Specifically,
SSD [20], MS-CNN [2], SSH [23], S3FD [39] predict
boxes on multiple layers of feature hierarchy. FCN [22],
Hypercolumns [9], Parsenet [21] fuse multiple layer fea-
tures in segmentation. FPN [15, 17], a top-down architec-
ture, integrate high-level semantic information to all scales.
 (a) Original Feature Shot
Input Image conv3_3 conv4_3 conv5_3 conv_fc7 conv6_2 conv7_2

First Shot PAL

(b) Feature Enhance Module

Second Shot PAL

640x640 160x160 80x80 40x40 20x20 10x10 5x5

(c) Enhanced Feature Shot

Figure 2: Our DSFD framework uses a Feature Enhance Module (b) on top of a feedforward VGG/ResNet architecture to generate the
enhanced features (c) from the original features (a), along with two loss layers named first shot PAL for the original features and second
shot PAL for the enchanted features.
Current feature map
FPN-based methods, such as FAN [31], PyramidBox [27]
N/3
achieve significant improvement on detection. However,
product N/3 concat
these methods do not consider the current layers informa-
tion. Different from the above methods that ignore the con- 1x1 N
conv N/3
text relationship between anchors, we propose a feature en-
Up feature map

hance module that incorporates multi-level dilated convolu-

tional layers to enhance the semantic of the features.
Loss Design Generally, the objective loss in detection is a
weighted sum of classification loss (e.g. softmax loss) and
box regression loss (e.g. L2 loss). Girshick et al. [7] pro-
pose smooth L1 loss to prevent exploding gradients. Lin
et al. [18] discover that the class imbalance is one obsta-
cle for better performance in one stage detector, hence they
propose focal loss, a dynamically scaled cross entropy loss.
Besides, Wang et al. [32] design RepLoss for pedestrian de-
tection, which improves performance in occlusion scenar-
ios. FANet [37] create a hierarchical feature pyramid and
presents hierarchical loss for their architecture. However,
the anchors used in FANet are kept the same size in dif-
ferent stages. In this work, we adaptively choose different
anchor sizes in different stages to facilitate the features.
Anchor Matching To make the model more robust, most
detection methods [20,35,39] do data augmentation, such as
color distortion, horizontal flipping, random crop and multi-
scale training. Zhang et al. [39] propose an anchor compen-
sation strategy to make tiny faces to match enough anchors
during training. Wang et al. [35] propose random crop to
generate large number of occluded faces for training. How-
ever, these methods ignore random sampling in data aug-
mentation, while ours combines anchor assign to provide
better data initialization for anchor matching.
1x1
conv upsample dilation conv,kernel=3x3,rate=3
Figure 3: Illustration on Feature Enhance Module, in which
the current feature map cell interactives with neighbors in current
feature maps and up feature maps.
3. Dual Shot Face Detector
We firstly introduce the pipeline of our proposed frame-
work DSFD, and then detailly describe our feature enhance
module in Sec. 3.2, progressive anchor loss in Sec. 3.3 and
improved anchor matching in Sec. 3.4, respectively.
3.1. Pipeline of DSFD
The framework of DSFD is illustrated in Fig. 2. Our
architecture uses the same extended VGG16 backbone as
PyramidBox [27] and S3FD [39], which is truncated be-
fore the classification layers and added with some aux-
iliary structures. We select conv3 3, conv4 3, conv5 3,
conv fc7, conv6 2 and conv7 2 as the first shot detec-
tion layers to generate six original feature maps named
of1 , of2 , of3 , of4 , of5 , of6 . Then, our proposed FEM trans-
fers these original feature maps into six enhanced feature
maps named ef1 , ef2 , ef3 , ef4 , ef5 , ef6 , which have the
same sizes as the original ones and are fed into SSD-style
head to construct the second shot detection layers. Note that
the input size of the training image is 640, which means the Table 1: The stride size, feature map size, anchor scale, ratio, and
feature map size of the lowest-level layer to highest-level number of six original/enhanced features for two shots.
Feature Stride Size Scale Ratio Number
layer is from 160 to 5. Different from S3FD and Pyramid- ef 1 (of 1) 4 160 × 160 16 (8) 1.5 : 1 25600
Box, after we utilize the receptive field enlargement in FEM ef 2 (of 2) 8 80 × 80 32 (16) 1.5 : 1 6400
and the new anchor design strategy, its unnecessary for the ef 3 (of 3) 16 40 × 40 64 (32) 1.5 : 1 1600
three sizes of stride, anchor and receptive field to satisfy ef 4 (of 4) 32 20 × 20 128 (64) 1.5 : 1 400
ef 5 (of 5) 64 10 × 10 256 (128) 1.5 : 1 100
equal-proportion interval principle. Therefore, our DSFD is ef 6 (of 6) 128 5×5 512 (256) 1.5 : 1 25
more flexible and robustness. Besides, the original and en-
hanced shots have two different losses, respectively named
First Shot progressive anchor Loss (FSL) and Second Shot vs. background), and Lloc is the smooth L1 loss between the
progressive anchor Loss (SSL). parameterizations of the predicted box ti and ground-truth
box gi using the anchor ai . When p∗i = 1 (p∗i = {0, 1}),
3.2. Feature Enhance Module the anchor ai is positive and the localization loss is acti-
vated. β is a weight to balance the effects of the two terms.
Feature Enhance Module is able to enhance original fea-
Compared to the enhanced feature maps in the same level,
tures to make them more discriminable and robust, which
the original feature maps have less semantic information for
is called FEM for short. For enhancing original neuron cell
classification but more high resolution location information
oc(i,j,l) , FEM utilizes different dimension information in-
for detection. Therefore, we believe that the original feature
cluding upper layer original neuron cell oc(i,j,l) and current
maps can detect and classify smaller faces. As the result, we
layer non-local neuron cells: nc(i−ε,j−ε,l) , nc(i−ε,j,l) , ...,
propose the First Shot multi-task Loss with a set of smaller
nc(i,j+ε,l) , nc(i+ε,j+ε,l) . Specially, the enhanced neuron
anchors as follows:
cell ec(i,j,l) can be mathematically defined as follow:
1
Σi Lconf (pi , p∗i )
LF SL (pi , p∗i , ti , gi , sai ) =
ec(i,j,l) = fconcat (fdilation (nc(i,j,l) )) Nconf
(1)
nci,j,l = fprod (oc(i,j,l) , fup (oc(i,j,l+1) )) β
+ Σi p∗i Lloc (ti , gi , sai ),
Nloc
where ci,j,l is a cell located in (i, j) coordinate of the feature (3)
maps in the l-th layer, f denotes a set of basic dilation con- where sa indicates the smaller anchors in the first shot lay-
volution, elem-wise production, up-sampling or concatena- ers, and the two shots losses can be weighted summed into
tion operations. Fig. 3 illustrates the idea of FEM, which is a whole Progressive Anchor Loss as follows:
inspired by FPN [17] and RFB [19]. Here, we first use 1×1
convolutional kernel to normalize the feature maps. Then, LP AL = LF SL (sa) + λLSSL (a). (4)
we up-sample upper feature maps to do element-wise prod-
Note that anchor size in the first shot is half of ones in the
uct with the current ones. Finally, we split the feature maps
second shot, and λ is weight factor. Detailed assignment
to three parts, followed by three sub-networks containing
on the anchor size is described in Sec. 3.4. In prediction
different numbers of dilation convolutional layers.
process, we only use the output of the second shot, which
3.3. Progressive Anchor Loss means no additional computational cost is introduced.
Different from the traditional detection loss, we design 3.4. Improved Anchor Matching
progressive anchor sizes for not only different levels, but
Current anchor matching method is bidirectional be-
also different shots in our framework. Motivated by the
tween the anchor and ground-truth face. Therefore, an-
statement in [24] that low-level features are more suitable
chor design and face sampling during augmentation are col-
for small faces, we assign smaller anchor sizes in the first
laborative to match the anchors and faces as far as pos-
shot, and use larger sizes in the second shot. First, our Sec-
sible for better initialization of the regressor. Our IAM
ond Shot anchor-based multi-task Loss function is defined
targets on addressing the contradiction between the dis-
as:
crete anchor scales and continuous face scales, in which
1 the faces are augmented by Sinput ∗ Sf ace /Sanchor (S in-
(Σi Lconf (pi , p∗i )
LSSL (pi , p∗i , ti , gi , ai ) =
Nconf dicates the spatial size) with the probability of 40% so as
β to increase the positive anchors, stabilize the training and
+ Σi p∗i Lloc (ti , gi , ai )), thus improve the results. Table 1 shows details of our an-
Nloc
(2) chor design on how each feature map cell is associated to
where Nconf and Nloc indicate the number of positive and the fixed shape anchor. We set anchor ratio 1.5:1 based
negative anchors, and the number of positive anchors re- on face scale statistics. Anchor size for the original fea-
spectively, Lconf is the softmax loss over two classes (face ture is one half of the enhanced feature. Additionally, with
Table 2: Effectiveness of Feature Enhance Module on the AP
performance.
Component Easy Medium Hard
FSSD+VGG16 92.6% 90.2% 79.1%
FSSD+VGG16+FEM 93.0% 91.4% 84.6%

Table 3: Effectiveness of Progressive Anchor Loss on the AP

performance.
Component Easy Medium Hard
FSSD+RES50 93.7% 92.2% 81.8%
FSSD+RES50+FEM 95.0% 94.1% 88.0% Figure 5: Comparisons on number distribution of matched
FSSD+RES50+FEM+PAL 95.3% 94.4% 88.6% anchor for ground truth faces between traditional anchor match-
ing (blue line) and our improved anchor matching (red line). we
actually set the IoU threshold to 0.35 for the traditional version.
That means even with a higher threshold (i.e., 0.4), using our IAM,
we can still achieve more matched anchors. Here, we choose a
slightly higher threshold in IAM so that to better balance the num-
ber and quality of the matched faces.

4.2. Analysis on DSFD

Figure 4: The number distribution of different scales of faces
compared between traditional anchor matching (Left) and our im-
proved anchor matching (Right).
probability of 2/5, we utilize anchor-based sampling like
data-anchor-sampling in PyramidBox, which randomly se-
lects a face in an image, crops sub-image containing the
face, and sets the size ratio between sub-image and selected
face to 640/rand (16, 32, 64, 128, 256, 512). For the remain-
ing 3/5 probability, we adopt data augmentation similar to
SSD [20]. In order to improve the recall rate of faces and
ensure anchor classification ability simultaneously, we set
Intersection-over-Union (IoU) threshold 0.4 to assign an-
chor to its ground-truth faces.
4. Experiments
4.1. Implementation Details
First, we present the details in implementing our net-
work. The backbone networks are initialized by the pre-
trained VGG/ResNet on ImageNet. All newly added con-
volution layers’ parameters are initialized by the ‘xavier’
method. We use SGD with 0.9 momentum, 0.0005 weight
decay to fine-tune our DSFD model. The batch size is set to
16. The learning rate is set to 10−3 for the first 40k steps,
and we decay it to 10−4 and 10−5 for two 10k steps.
During inference, the first shot’s outputs are ignored
and the second shot predicts top 5k high confident detec-
tions. Non-maximum suppression is applied with jaccard
overlap of 0.3 to produce top 750 high confident bound-
ing boxes per image. For 4 bounding box coordinates, we
round down top left coordinates and round up width and
height to expand the detection bounding box. The offi-
cial code has been released at: https://github.com/
TencentYoutuResearch/FaceDetection-DSFD.
In this subsection, we conduct extensive experiments and
ablation studies on the WIDER FACE dataset to evaluate
the effectiveness of several contributions of our proposed
framework, including feature enhance module, progressive
anchor loss, and improved anchor matching. For fair com-
parisons, we use the same parameter settings for all the ex-
periments, except for the specified changes to the compo-
nents. All models are trained on the WIDER FACE training
set and evaluated on validation set. To better understand
DSFD, we select different baselines to ablate each compo-
nent on how this part affects the final performance.
Feature Enhance Module First, We adopt anchor designed
in S3FD [39], PyramidBox [27] and six original feature
maps generated by VGG16 to perform classification and re-
gression, which is named Face SSD (FSSD) as the baseline.
We then use VGG16-based FSSD as the baseline to add
feature enchance module for comparison. Table 2 shows
that our feature enhance module can improve VGG16-based
FSSD from 92.6%, 90.2%, 79.1% to 93.0%, 91.4%, 84.6%.
Progressive Anchor Loss Second, we use Res50-based
FSSD as the baseline to add progressive anchor loss for
comparison. We use four residual blocks’ ouputs in
ResNet to replace the outputs of conv3 3, conv4 3, conv5 3,
conv fc7 in VGG. Except for VGG16, we do not perform
layer normalization. Table 3 shows our progressive an-
chor loss can improve Res50-based FSSD using FEM from
95.0%, 94.1%, 88.0% to 95.3%, 94.4%, 88.6%.
Improved Anchor Matching To evaluate our improved
anchor matching strategy, we use Res101-based FSSD
without anchor compensation as the baseline. Table 4 shows
that our improved anchor matching can improve Res101-
based FSSD using FEM from 95.8%, 95.1%, 89.7% to
96.1%, 95.2%, 90.0%. Finally, we can improve our DSFD
to 96.6%, 95.7%, 90.4% with ResNet152 as the backbone.
 Val: easy Val: medium Val: hard

Test: easy Test: medium Test: hard

Figure 6: Precision-recall curves on WIDER FACE validation and testing subset.
Table 4: Effectiveness of Improved Anchor Matching on the AP performance.
Component Easy Medium Hard
FSSD+RES101 95.1% 93.6% 83.7%
FSSD+RES101+FEM 95.8% 95.1% 89.7%
FSSD+RES101+FEM+IAM 96.1% 95.2% 90.0%
FSSD+RES101+FEM+IAM+PAL 96.3% 95.4% 90.1%
FSSD+RES152+FEM+IAM+PAL 96.6% 95.7% 90.4%
FSSD+RES152+FEM+IAM+PAL+LargeBS 96.4% 95.7% 91.2%

Table 5: Effectiveness of different backbones.

Component Params ACC@Top-1 Easy Medium Hard
FSSD+RES101+FEM+IAM+PAL 399M 77.44% 96.3% 95.4% 90.1%
FSSD+RES152+FEM+IAM+PAL 459M 78.42% 96.6% 95.7% 90.4%
FSSD+SE-RES101+FEM+IAM+PAL 418M 78.39% 95.7% 94.7% 88.6%
FSSD+DPN98+FEM+IAM+PAL 515M 79.22% 96.3% 95.5% 90.4%
FSSD+SE-RESNeXt101 32×4d+FEML+IAM+PA 416M 80.19% 95.7% 94.8% 88.9%

Table 6: FEM vs. RFB on WIDER FACE. Comparison with RFB Our FEM differs from RFB in two
Backbone - ResNet101 (%) Easy Medium Hard
DSFD (RFB) 96.0 94.5 87.2
aspects. First, our FEM is based on FPN to make full use of
DSFD (FPN) / (FPN+RFB) 96.2 / 96.2 95.1 / 95.3 89.7 / 89.9 feature information from different spatial levels, while RFB
DSFD (FEM) 96.3 95.4 90.1 ignores. Second, our FEM adopts stacked dilation convolu-
tions in a multi-branch structure, which efficiently leads to
Besides, Fig. 4 shows that our improved anchor match- larger Receptive Fields (RF) than RFB that only uses one
ing strategy greatly increases the number of ground truth dilation layer in each branch, e.g., R3 in FEM compared to
faces that are closed to the anchor, which can reduce the R in RFB where indicates the RF of one dilation convolu-
contradiction between the discrete anchor scales and con- tion. Tab. 6 clearly demonstrates the superiority of our FEM
tinuous face scales. Moreover, Fig. 5 shows the number dis- over RFB, even when RFB is equipped with FPN.
tribution of matched anchor number for ground truth faces, From the above analysis and results, some promising
which indicates our improved anchor matching can signif- conclusions can be drawn: 1) Feature enhance is crucial.
icantly increase the matched anchor number, and the aver- We use a more robust and discriminative feature enhance
aged number of matched anchor for different scales of faces module to improve the feature presentation ability, espe-
can be improved from 6.4 to about 6.9. cially for hard face. 2) Auxiliary loss based on progressive
 Discontinous ROC curves
Discontinous ROC curves
Figure 7: Comparisons with popular state-of-the-art methods on the FDDB dataset. The first row shows the ROC results without
additional annotations, and the second row shows the ROC results with additional annotations.
anchor is used to train all 12 different scale detection feature
maps, and it improves the performance on easy, medium
and hard faces simultaneously. 3) Our improved anchor
matching provides better initial anchors and ground-truth
faces to regress anchor from faces, which achieves the im-
provements of 0.3%, 0.1%, 0.3% on three settings, respec-
tively. Additionally, when we enlarge the training batch size
(i.e., LargeBS), the result in hard setting can get 91.2% AP.
Effects of Different Backbones To better understand
our DSFD, we further conducted experiments to examine
how different backbones affect classification and detection
performance. Specifically, we use the same setting ex-
cept for the feature extraction network, we implement SE-
ResNet101, DPN−98, SE-ResNeXt101 32×4d following
the ResNet101 setting in our DSFD. From Table 5, DSFD
with SE-ResNeXt101 32×4d got 95.7%, 94.8%, 88.9%, on
easy, medium and hard settings respectively, which indi-
cates that more complexity model and higher Top-1 Ima-
geNet classification accuracy may not benefit face detection
AP. Therefore, in our DSFD framework, better performance
on classification are not necessary for better performance
on detection, which is consistent to the conclusion claimed
in [11, 16]. Our DSFD enjoys high inference speed bene-
fited from simply using the second shot detection results.
Continous ROC curves
Continous ROC curves
For VGA resolution inputs to Res50-based DSFD, it runs
22 FPS on NVIDA GPU P40 during inference.
4.3. Comparisons with State-of-the-Art Methods
We evaluate the proposed DSFD on two popular face
detection benchmarks, including WIDER FACE [35] and
Face Detection Data Set and Benchmark (FDDB) [12]. Our
model is trained only using the training set of WIDER
FACE, and then evaluated on both benchmarks without any
further fine-tuning. We also follow the similar way used
in [31] to build the image pyramid for multi-scale testing
and use more powerful backbone similar as [4].
WIDER FACE Dataset It contains 393, 703 annotated
faces with large variations in scale, pose and occlusion in
total 32, 203 images. For each of the 60 event classes, 40%,
10%, 50% images of the database are randomly selected
as training, validation and testing sets. Besides, each sub-
set is further defined into three levels of difficulty: ’Easy’,
’Medium’, ’Hard’ based on the detection rate of a baseline
detector. As shown in Fig. 6, our DSFD achieves the best
performance among all of the state-of-the-art face detectors
based on the average precision (AP) across the three sub-
sets, i.e., 96.6% (Easy), 95.7% (Medium) and 90.4% (Hard)
on validation set, and 96.0% (Easy), 95.3% (Medium) and
 Scale Pose Occlusion Blurry

Makeup Illumination Modality Reflection

Figure 8: Illustration of our DSFD to various large variations on scale, pose, occlusion, blurry, makeup, illumination, modality and
reflection. Blue bounding boxes indicate the detector confidence is above 0.8.

90.0% (Hard) on test set. Fig. 8 shows more examples to 5. Conclusions

demonstrate the effects of DSFD on handling faces with
various variations, in which the blue bounding boxes indi-
cate the detector confidence is above 0.8.
FDDB Dataset It contains 5, 171 faces in 2, 845 images
taken from the faces in the wild data set. Since WIDER
FACE has bounding box annotation while faces in FDDB
are represented by ellipses, we learn a post-hoc ellipses re-
gressor to transform the final prediction results. As shown
in Fig. 7, our DSFD achieves state-of-the-art performance
on both discontinuous and continuous ROC curves, i.e.
99.1% and 86.2% when the number of false positives equals
to 1, 000. After adding additional annotations to those un-
labeled faces [39], the false positives of our model can be
further reduced and outperform all other methods.
This paper introduces a novel face detector named Dual
Shot Face Detector (DSFD). In this work, we propose a
novel Feature Enhance Module that utilizes different level
information and thus obtains more discriminability and ro-
bustness features. Auxiliary supervisions introduced in
early layers by using smaller anchors are adopted to ef-
fectively facilitate the features. Moreover, an improved an-
chor matching method is introduced to match anchors and
ground truth faces as far as possible to provide better initial-
ization for the regressor. Comprehensive experiments are
conducted on popular face detection benchmarks, FDDB
and WIDER FACE, to demonstrate the superiority of our
proposed DSFD compared with the state-of-the-art face de-
tectors, e.g., SRN and PyramidBox.
References
[1] Yotam Abramson, Bruno Steux, and Hicham Ghorayeb. Yet
even faster (yef) real-time object detection. International
Journal of Intelligent Systems Technologies and Applica-
tions, 2(2-3):102–112, 2007. 2
[2] Zhaowei Cai, Quanfu Fan, Rogerio S Feris, and Nuno Vas-
concelos. A unified multi-scale deep convolutional neural
network for fast object detection. In Proceedings of Euro-
pean Conference on Computer Vision (ECCV), 2016. 2
[3] Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian
Yang. Fsrnet: End-to-end learning face super-resolution with
facial priors. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2018. 1
[4] Cheng Chi, Shifeng Zhang, Junliang Xing, Zhen Lei, Stan Z
Li, and Xudong Zou. Selective refinement network for high
performance face detection. In Proceedings of Association
for the Advancement of Artificial Intelligence (AAAI), 2019.
2, 7
[5] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object
detection via region-based fully convolutional networks. In
Proceedings of Advances in Neural Information Processing
Systems (NIPS), 2016. 2
[6] Jiankang Deng, Jia Guo, and Stefanos Zafeiriou. Arc-
face: Additive angular margin loss for deep face recognition.
arXiv:1801.07698v1, 2018. 1
[7] Ross Girshick. Fast r-cnn. In Proceedings of IEEE Inter-
national Conference on Computer Vision (ICCV), 2015. 2,
3 2, 3, 5
[8] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra
Malik. Rich feature hierarchies for accurate object detection
and semantic segmentation. In Proceedings of IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
pages 580–587, 2014. 2
[9] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Ji-
tendra Malik. Hypercolumns for object segmentation and
fine-grained localization. In Proceedings of IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
2015. 2
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceed-
ings of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016. 1
[11] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu,
Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wo-
jna, Yang Song, Sergio Guadarrama, and Kevin Murphy.
Speed/accuracy trade-offs for modern convolutional object
detectors. In Proceedings of the IEEE Conference on Com-
puter Vision and Pattern Recognition, 2017. 7
[12] Vidit Jain and Erik Learned-Miller. Fddb: A benchmark
for face detection in unconstrained settings. Technical
report, Technical Report UM-CS-2010-009, University of
Massachusetts, Amherst, 2010. 7
[13] Kobi Levi and Yair Weiss. Learning object detection from a
small number of examples: the importance of good features.
In Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2004. 2
[14] Haoxiang Li, Zhe Lin, Xiaohui Shen, Jonathan Brandt, and
Gang Hua. A convolutional neural network cascade for face
detection. In Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2015. 2
[15] Jian Li, Jianjun Qian, and Jian Yang. Object detection via
feature fusion based single network. In IEEE International
Conference on Image Processing, 2017. 2
[16] Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong
Deng, and Jian Sun. Detnet: A backbone network for object
detection. In Proceedings of European Conference on Com-
puter Vision, 2018. 7
[17] Tsung-Yi Lin, Piotr Dollár, Ross B Girshick, Kaiming He,
Bharath Hariharan, and Serge J Belongie. Feature pyra-
mid networks for object detection. In Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), 2017. 2, 4
[18] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and
Piotr Dollár. Focal loss for dense object detection. In Pro-
ceedings of IEEE International Conference on Computer Vi-
sion (ICCV), 2017. 2, 3
[19] Songtao Liu, Di Huang, and Yunhong Wang. Receptive field
block net for accurate and fast object detection. In Proceed-
ings of European Conference on Computer Vision, 2018. 2,
4
[20] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian
Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C
Berg. Ssd: Single shot multibox detector. In Proceedings
of European conference on computer vision (ECCV), 2016.
[21] Wei Liu, Andrew Rabinovich, and Alexander Berg. Parsenet:
Looking wider to see better. In Proceedings of International
Conference on Learning Representations Workshop, 2016. 2
[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In Pro-
ceedings of IEEE Conference on Computer Vision and Pat-
tern Recognition (CVPR), 2015. 2
[23] Mahyar Najibi, Pouya Samangouei, Rama Chellappa, and
Larry S Davis. Ssh: Single stage headless face detector. In
Proceedings of IEEE International Conference on Computer
Vision (ICCV), 2017. 2
[24] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region
proposal networks. In Proceedings of Advances in Neural
Information Processing Systems (NIPS), 2015. 2, 4
[25] Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Math-
ieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated
recognition, localization and detection using convolutional
networks. In Proceedings of International Conference on
Learning Representations (ICLR), 2014. 2
[26] Ying Tai, Yicong Liang, Xiaoming Liu, Lei Duan, Jilin Li,
Chengjie Wang, Feiyue Huang, and Yu Chen. Towards
highly accurate and stable face alignment for high-resolution
videos. In The AAAI Conference on Artificial Intelligence
(AAAI), 2019. 1
[27] Xu Tang, Daniel K Du, Zeqiang He, and Jingtuo Liu. Pyra-
midbox: A context-assisted single shot face detector. In
Proceedings of European Conference on Computer Vision
(ECCV), 2018. 2, 3, 5
[28] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gev-
ers, and Arnold WM Smeulders. Selective search for ob-
ject recognition. International Journal of Computer Vision,
104(2):154–171, 2013. 2
[29] Paul Viola and Michael J Jones. Robust real-time face detec-
tion. International Journal of Computer Vision, 57(2):137–
154, 2004. 1, 2
[30] Hao Wang, Zhifeng Li, Xing Ji, and Yitong Wang. Face r-
cnn. arXiv preprint arXiv:1706.01061, 2017. 2
[31] Jianfeng Wang, Ye Yuan, and Gang Yu. Face attention net-
work: An effective face detector for the occluded faces.
arXiv preprint arXiv:1711.07246, 2017. 3, 7
[32] Xinlong Wang, Tete Xiao, Yuning Jiang, Shuai Shao, Jian
Sun, and Chunhua Shen. Repulsion loss: Detecting pedes-
trians in a crowd. In Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2018. 3
[33] Yitong Wang, Xing Ji, Zheng Zhou, Hao Wang, and Zhifeng
Li. Detecting faces using region-based fully convolutional
networks. arXiv preprint arXiv:1709.05256, 2017. 2
[34] Jian Yang, Lei Luo, Jianjun Qian, Ying Tai, Fanlong Zhang,
and Yong Xu. Nuclear norm based matrix regression with
applications to face recognition with occlusion and illumi-
nation changes. IEEE Transactions on Pattern Analysis and
Machine Intelligence (TPAMI), 39(1):156–171, 2017. 1
[35] Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang.
Wider face: A face detection benchmark. In Proceedings of
IEEE Conference on Computer Vision and Pattern Recogni-
tion (CVPR), 2016. 3, 7
[36] Changzheng Zhang, Xiang Xu, and Dandan Tu. Face
detection using improved faster rcnn. arXiv preprint
arXiv:1802.02142, 2018. 2
[37] Jialiang Zhang, Xiongwei Wu, Jianke Zhu, and Steven CH
Hoi. Feature agglomeration networks for single stage face
detection. arXiv preprint arXiv:1712.00721, 2017. 2, 3
[38] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.
Joint face detection and alignment using multitask cascaded
convolutional networks. IEEE Signal Processing Letters,
23(10):1499–1503, 2016. 2
[39] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo
Wang, and Stan Z Li. Sˆ 3fd: Single shot scale-invariant face
detector. In Proceedings of IEEE International Conference
on Computer Vision (ICCV), 2017. 2, 3, 5, 8
 GANFIT: Generative Adversarial Network Fitting
for High Fidelity 3D Face Reconstruction

Baris Gecer1,2 , Stylianos Ploumpis1,2 , Irene Kotsia3 , and Stefanos Zafeiriou1,2

1
Imperial College London
2
FaceSoft.io
3
University of Middlesex
arXiv:1902.05978v2 [cs.CV] 6 Apr 2019

{b.gecer, s.ploumpis, s.zafeiriou}@imperial.ac.uk , drkotsia@gmail.com

Figure 1: The proposed deep fitting approach can reconstruct high quality texture and geometry from a single image with
precise identity recovery. The reconstructions in the figure and the rest of the paper are represented by a vector of size 700
floating points and rendered without any special effects. We would like to highlight that the depicted texture is reconstructed
by our model and none of the features taken directly from the image.

Abstract
In the past few years, a lot of work has been done to-
wards reconstructing the 3D facial structure from single
images by capitalizing on the power of Deep Convolutional
Neural Networks (DCNNs). In the most recent works, differ-
entiable renderers were employed in order to learn the rela-
tionship between the facial identity features and the param-
eters of a 3D morphable model for shape and texture. The
texture features either correspond to components of a lin-
ear texture space or are learned by auto-encoders directly
from in-the-wild images. In all cases, the quality of the fa-
cial texture reconstruction of the state-of-the-art methods is
still not capable of modeling textures in high fidelity. In this
paper, we take a radically different approach and harness
the power of Generative Adversarial Networks (GANs) and
DCNNs in order to reconstruct the facial texture and shape
from single images. That is, we utilize GANs to train a very
powerful generator of facial texture in UV space. Then, we
revisit the original 3D Morphable Models (3DMMs) fitting
approaches making use of non-linear optimization to find
the optimal latent parameters that best reconstruct the test
image but under a new perspective. We optimize the param-
eters with the supervision of pretrained deep identity fea-
tures through our end-to-end differentiable framework. We
demonstrate excellent results in photorealistic and identity
preserving 3D face reconstructions and achieve for the first
time, to the best of our knowledge, facial texture reconstruc-
tion with high-frequency details.1
1. Introduction
Estimation of the 3D facial surface and other intrinsic
components of the face from single images (e.g., albedo,
etc.) is a very important problem at the intersection of
computer vision and machine learning with countless ap-
plications (e.g., face recognition, face editing, virtual real-
ity). It is now twenty years from the seminal work of Blanz
and Vetter [4] which showed that it is possible to recon-
struct shape and albedo by solving a non-linear optimiza-
1 Project page: https://github.com/barisgecer/ganfit

1
tion problem that is constrained by linear statistical models
of facial texture and shape. This statistical model of tex-
ture and shape is called a 3D Morphable Model (3DMM).
Arguably the most popular publicly available 3DMM is the
Basel model built from 200 people [21]. Recently, large
scale statistical models of face and head shape have been
made publicly available [7, 10].
For many years 3DMMs and its variants were the meth-
ods of choice for 3D face reconstruction [33, 46, 22].
Furthermore, with appropriate statistical texture models
on image features such as Scale Invariant Feature Trans-
form (SIFT) and Histogram Of Gradients (HOG), 3DMM-
based methodologies can still achieve state-of-the-art per-
formance in 3D shape estimation on images captured un-
der unconstrained conditions [6]. Nevertheless, those meth-
ods [6] can reconstruct only the shape and not the facial tex-
ture. Another line of research in [45, 34] decouples texture
and shape reconstruction. A standard linear 3DMM fitting
strategy [41] is used for face reconstruction followed by a
number of steps for texture completion and refinement. In
these papers [34, 45], the texture looks excellent when ren-
dered under professional renderers (e.g., Arnold), neverthe-
less when the texture is overlaid on the images the quality
significantly drops 2 .
In the past two years, a lot of work has been con-
ducted on how to harness Deep Convolutional Neural Net-
works (DCNNs) for 3D shape and texture reconstruction.
The first such methods either trained regression DCNNs
from image to the parameters of a 3DMM [42] or used
a 3DMM to synthesize images [30, 18] and formulate an
image-to-image translation problem using DCNNs to es-
timate the depth3 [36]. The more recent unsupervised
DCNN-based methods are trained to regress 3DMM param-
eters from identity features by making use of differentiable
image formation architectures [9] and differentiable render-
ers [16, 40, 31].
The most recent methods such as [39, 43, 14] use both
the 3DMM model, as well as additional network structures
(called correctives) in order to extend the shape and texture
representation. Even though the paper [39] shows that the
reconstructed facial texture has indeed more details than a
texture estimated from a 3DMM [42, 40], it is still unable to
capture high-frequency details in texture and subsequently
many identity characteristics (please see the Fig. 4). Fur-
thermore, because the method permits the reconstructions
to be outside the 3DMM space, it is susceptible to outliers
(e.g., glasses etc.) which are baked in shape and texture. Al-
though rendering networks (i.e. trained by VAE [26]) gen-
erates outstanding quality textures, each network is capable
of storing up to few individuals whom should be placed in a
2 Please see the supplementary materials for a comparison with [34, 45].
3 The depth was afterwards refined by fitting a 3DMM and then chang-
ing the normals by using image features.
controlled environment to collect ∼20 millions of images.
In this paper, we still propose to build upon the success
of DCNNs but take a radically different approach for 3D
shape and texture reconstruction from a single in-the-wild
image. That is, instead of formulating regression method-
ologies or auto-encoder structures that make use of self-
supervision [39, 16, 43], we revisit the optimization-based
3DMM fitting approach by the supervision of deep iden-
tity features and by using Generative Adversarial Networks
(GANs) as our statistical parametric representation of the
facial texture.
In particular, the novelties that this paper brings are:
• We show for the first time, to the best of our knowl-
edge, that a large-scale high-resolution statistical re-
construction of the complete facial surface on an un-
wrapped UV space can be successfully used for recon-
struction of arbitrary facial textures even captured in
unconstrained recording conditions4 .
• We formulate a novel 3DMM fitting strategy which is
based on GANs and a differentiable renderer.
• We devise a novel cost function which combines vari-
ous content losses on deep identity features from a face
recognition network.
• We demonstrate excellent facial shape and texture re-
constructions in arbitrary recording conditions that are
shown to be both photorealistic and identity preserving
in qualitative and quantitative experiments.
2. History of 3DMM Fitting
Our methodology naturally extends and generalizes the
ideas of texture and shape 3DMM using modern methods
for representing texture using GANs, as well as defines loss
functions using differentiable renderers and very powerful
publicly available face recognition networks [12]. Before
we define our cost function, we will briefly outline the his-
tory of 3DMM representation and fitting.
2.1. 3DMM representation
The first step is to establish dense correspondences be-
tween the training 3D facial meshes and a chosen template
with fixed topology in terms of vertices and triangulation.
2.1.1 Texture
Traditionally 3DMMs use a UV map for representing tex-
ture. UV maps help us to assign 3D texture data into 2D
4 In the very recent works, it was shown that it is feasible to reconstruct
the non-visible parts a UV space for facial texture completion[11] and that
GANs can be used to generate novel high-resolution faces[38]. Neverthe-
less, our work is the first one that demonstrates that a GAN can be used
as powerful statistical texture prior and reconstruct the complete texture of
arbitrary facial images.
 PCA Shape Model Input Image
.
Differentiable Renderer (Sec.3.2) Landmark Detector (Sec.3.3.4)
.
. Camera and Lighting
Parameters
p s= .
.
.
.
.
.
.
.
.
c= .
i= .
. .
.
.
.
.
.
= 2
Expression Blend Shapes
.
.
(Eq.6)
=| - |
.
p e= .
. Face Recognition CNN (Sec.3.3.1)
. . . . .
. . . .
.

Coloured mesh
= .
.
.
*
.
.
.
+
.
.
.
*
.
.
.
Sampling . . . .
. . . .
Texture GAN (Sec.3.1)
. (Eq.7)
. . .
. . .

pt= .
.
Random
pe, c, i
= -
.
.
.
.
.
.
. . .
. .
.
2

Figure 2: Detailed overview of the proposed approach. A 3D face reconstruction is rendered by a differentiable renderer
(shown in purple). Cost functions are mainly formulated by means of identity features on a pretrained face recognition
network (shown in gray) and they are optimized by flowing the error all the way back to the latent parameters (ps , pe , pt , c, i,
shown in green) with gradient descent optimization. End-to-end differentiable architecture enables us to use computationally
cheap and reliable first order derivatives for optimization thus making it possible to employ deep networks as a generator
(i.e,. statistical model) or as a cost function.

planes with universal per-pixel alignment for all textures. A
commonly used UV map is built by cylindrical unwrapping
the mean shape into a 2D flat space formulation, which we
use to create an RGB image IU V . Each vertex in the 3D
space has a texture coordinate tcoord in the UV image plane
in which the texture information is stored. A universal func-
tion exists, where for each vertex we can sample the texture
information from the UV space as T = P(IU V , tcoord ).
In order to define a statistical texture representation, all
the training texture UV maps are vectorized and Principal
Component Analysis (PCA) is applied. Under this model
any test texture T0 is approximated as a linear combination
of the mean texture mt and a set of bases Ut as follows:
T(pt ) ≈ mt + Ut pt
where pt is the texture parameters for the text sample T0 .
In the early 3DMM studies, the statistical model of the tex-
ture was built with few faces captured in strictly controlled
conditions and was used to reconstruct the test albedo of
the face. Since, such texture models can hardly represent
faces captured in uncontrolled recording conditions (in-the-
wild). Recently it was proposed to use statistical models
of hand-crafted features such as SIFT or HoG [6] directly
from in-the-wild faces. The interested reader is referred to
[5, 32] for more details on texture models used in 3DMM
fitting algorithms.
The recent 3D face fitting methods [39, 43, 14] still make
use of similar statistical models for the texture. Hence, they
can naturally represent only the low-frequency components
of the facial texture (please see Fig. 4).
2.1.2 Shape
The method of choice for building statistical models of fa-
cial or head 3D shapes is still PCA [23]. Assuming that the
3D shapes in correspondence comprise of N vertexes, i.e.
 T T
 T
s = xT 1 , . . . , xN = [x1 , y1 , z1 , . . . , xN , yN , zN ] . In
order to represent both variations in terms of identity and
expression, generally two linear models are used. The first
(1) is learned from facial scans displaying the neutral expres-
sion (i.e., representing identity variations) and the second
is learned from displacement vectors (i.e., representing ex-
pression variations). Then a test facial shape S(ps,e ) can be
written as
S(ps,e ) ≈ ms,e + Us,e ps,e (2)
where ms,e in the mean shape vector, Us,e ∈ R3N ×ns,e
is Us,e = [Us , Ue ] where the Us are the bases that cor-
respond to identity variations, and Ue the bases that cor-
respond to expression. Finally, ps,e are the ns,e shape pa-
rameters which can be split accordingly to the identity and
expression bases: ps,e = [ps , pe ].
2.2. Fitting
3D face and texture reconstruction by fitting a 3DMM
is performed by solving a non-linear energy based cost op-
timization problem that recovers a set of parameters p =
[ps,e , pt , pc , pl ] where pc are the parameters related to a
camera model and pl are the parameters related to an illu-
mination model. The optimization can be formulated as:
min E(p) = ||I0 (p) − W(p)||22 + Reg({ps,e , pt }) (3)
p
where I0 is the test image to be fitted and W is a vector
produced by a physical image formation process (i.e., ren-
dering) controlled by p. Finally, Reg is the regularization
term that is mainly related to texture and shape parameters.
Various methods have been proposed for numerical op-
timization of the above cost functions [19, 2]. A notable
recent approach is [6] which uses handcrafted features (i.e.,
H) for texture representation simplified the cost function as:
min
r
E(pr ) = ||H(I0 (pr ))−H(W(pr ))||2A+Reg(ps,e ) (4)
p
where ||a||2A = aT Aa, A is the orthogonal space to the
statistical model of the texture and pr is the set of reduced
parameters pr = {ps,e , pc }. The optimization problem in
Eq. 4 is solved by Gauss-Newton method. The main draw-
back of this method is that the facial texture in not recon-
structed.
In this paper, we generalize the 3DMM fittings and in-
troduce the following novelties:
• We use a GAN on high-resolution UV maps as our sta-
tistical representation of the facial texture. That way
we can reconstruct textures with high-frequency de-
tails.
• Instead of other cost functions used in the literature
such as low-level `1 or `2 loss (e.g., RGB values [29],
edges [33]) or hand-crafted features (e.g., SIFT [6]),
we propose a novel cost function that is based on fea-
ture loss from the various layers of publicly available
face recognition embedding network [12]. Unlike oth-
ers, deep identity features are very powerful at preserv-
ing identity characteristics of the input image.
• We replace physical image formation stage with a dif-
ferentiable renderer to make use of first order deriva-
tives (i.e., gradient descent). Unlike its alternatives,
gradient descent provides computationally cheaper and
more reliable derivatives through such deep architec-
tures (i.e., above-mentioned texture GAN and identity
DCNN).
3. Approach
We propose an optimization-based 3D face reconstruc-
tion approach from a single image that employs a high fi-
delity texture generation network as statistical prior as il-
lustrated in Fig. 2. To this end, the reconstruction mesh
is formed by 3D morphable shape model; textured by the
generator network’s output UV map; and projected into 2D
image by a differentiable renderer. The distance between
the rendered image and the input image is minimized in
terms of a number of cost functions by updating the latent
parameters of 3DMM and the texture network with gradi-
ent descent. We mainly formulate these functions based on
rich features of face recognition network [12, 35, 28] for
smoother convergence and landmark detection network [13]
for alignment and rough shape estimation.
The following sections introduce firstly our novel texture
model that employs a generator network trained by progres-
sive growing GAN framework. After describing the proce-
dure for image formation with differentiable renderer, we
formulate our cost functions and the procedure for fitting
our shape and texture models onto a test image.
3.1. GAN Texture Model
Although conventional PCA is powerful enough to build
a decent shape and texture model, it is often unable to cap-
ture high frequency details and ends up having blurry tex-
tures due to its Gaussian nature. This becomes more appar-
ent in texture modelling which is a key component in 3D
reconstruction to preserve identity as well as photo-realism.
GANs are shown to be very effective at capturing such
details. However, they suffer from preserving 3D co-
herency [17] of the target distribution when the training im-
ages are semi-aligned. We found that a GAN trained with
UV representation of real textures with per pixel alignment
avoids this problem and is able to generate realistic and co-
herent UVs from 99.9% of its latent space while at the same
time generalizing well to unseen data.
In order to take advantage of this perfect harmony, we
train a progressive growing GAN [24] to model distribu-
tion of UV representations of 10,000 high resolution tex-
tures and use the trained generator network
G(pt ) : R512 → RH×W ×C (5)
as texture model that replaces 3DMM texture model in
Eq. 1.
While fitting with linear models, i.e. 3DMM, is as sim-
ple as linear transformation, fitting with a generator net-
work can be formulated as an optimization that minimizes
per-pixel Manhattan distance between target texture in UV
space Iuv and the network output G(pt ) with respect to the
latent parameter pt , i.e. minpt |G(pt ) − Iuv |.
3.2. Differentiable Renderer
Following [16], we employ a differentiable renderer to
project 3D reconstruction into a 2D image plane based on
deferred shading model with given camera and illumination
parameters. Since color and normal attributes at each vertex
are interpolated at the corresponding pixels with barycen-
tric coordinates, gradients can be easily backpropagated
through the renderer to the latent parameters.
A 3D textured mesh at the center of Cartesian origin
[0, 0, 0] is projected onto 2D image plane by a pinhole cam-
era model with the camera standing at [xc , yc , zc ], directed
towards [x0c , yc0 , zc0 ] and with the focal length fc . The il-
lumination is modelled by phong shading given 1) direct
light source at 3D coordinates [xl , yl , zl ] with color values
[rl , gl , bl ], and 2) color of ambient lighting [ra , ga , ba ].
Finally, we denote the rendered image given
geometry (ps,e ), texture (pt ), camera (pc
[xc , yc , zc , x0c , yc0 , zc0 , fc ]) and lighting parameters
(pl = [xl , yl , zl , rl , gl , bl , ra , ga , ba ] by the following:
IR = R(S(ps , pe ), P(G(pt )), pc , pl )
where we construct shape mesh by 3DMM as given in Eq. 2
and texture by GAN generator network as in Eq. 5. Since
our differentiable renderer supports only color vectors, we
sample from our generated UV map to get vectorized color
representation as explained in Sec. 2.1.1.
Additionally, we render a secondary image with random
expression, pose and illumination in order to generalize
identity related parameters well with those variations. We
sample expression parameters from a normal distribution as
pˆe ∼ N (µ = 0, σ = 0.5) and sample camera and illumina-
tion parameters from the Gaussian distribution of 300W-3D
dataset as p̂c ∼ N (µˆc , σˆc ) and p̂l ∼ N (µ̂l , σ̂l ). This ren-
dered image of the same identity as IR (i.e., with same ps
and pt parameters) is expressed by the following:
ÎR = R(S(ps , pˆe ), P(G(pt )), pˆc , p̂l )
3.3. Cost Functions
Given an input image I0 , we optimize all of the afore-
mentioned parameters simultaneously with gradient descent
updates. In each iteration, we simply calculate the forth-
coming cost terms for the current state of the 3D recon-
struction, and take the derivative of the weighted error with
respect to the parameters using backpropagation.
3.3.1 Identity Loss
With the availability of large scale datasets, CNNs have
shown incredible performance on many face recognition
benchmarks. Their strong identity features are robust to
many variations including pose, expression, illumination,
age etc. These features are shown to be quite effective at
many other tasks including novel identity synthesizing [15],
face normalization [9] and 3D face reconstruction [16]. In
our approach, we take advantage of an off-the-shelf state-
of-the-art face recognition network [12]5 in order to capture
identity related features of an input face image and optimize
the latent parameters accordingly. More specifically, given a
pretrained face recognition network F n (I) : RH×W ×C →
R512 consisting of n convolutional filters, we calculate the
cosine distance between the identity features (i.e., embed-
dings) of the real target image and our rendered images as
following:
F n (I0 ).F n (IR )
Lid = 1 − (8)
||F n (I0 )||2 ||F n (IR )||2
We formulate an additional identity loss on the rendered im-
age ÎR that is rendered with random pose, expression and
= lighting. This loss ensures that our reconstruction resembles
the target identity under different conditions. We formulate
it by replacing IR by ÎR in Eq. 8 and it is denoted as L̂id .
(6)
3.3.2 Content Loss
Face recognition networks are trained to remove all kinds
of attributes (e.g. expression, illumination, age, pose) other
than abstract identity information throughout the convolu-
tional layers. Despite their strength, the activations in the
very last layer discard some of the mid-level features that
are useful for 3D reconstruction, e.g. variations that depend
on age. Therefore we found it effective to accompany iden-
tity loss by leveraging intermediate representations in the
face recognition network that are still robust to pixel-level
deformations and not too abstract to miss some details. To
this end, normalized euclidean distance of intermediate ac-
tivations, namely content loss, is minimized between input
and rendered image with the following loss term:
n
||F j (I0 ) − F j (IR )||2
X
(7) Lcon = (9)
j
HF j × WF j × CF j
3.3.3 Pixel Loss
While identity and content loss terms optimize albedo of
the visible texture, lighting conditions are optimized based
on pixel value difference directly. While this cost function
is relatively primitive, it is sufficient to optimize lighting
parameters such as ambient colors, direction, distance and
color of a light source. We found that optimizing illumina-
tion parameters jointly with others helped to improve albedo
of the recovered texture. Furthermore, pixel loss support
identity and content loss with fine-grained texture as it sup-
ports highest available resolution while images needs to be
5 We empirically deduced that other face recognition networks work
almost equally well and this choice is orthogonal to the proposed approach.
Figure 3: Example fits of our approach for the images from various datasets. Please note that our fitting approach is robust
to occlusion (e.g., glasses), low resolution and black-white in the photos and generalizes well with ethnicity, gender and
age. The reconstructed textures are very well at capturing high frequency details of the identities; likewise, the reconstructed
geometries from 3DMM are surprisingly good at identity preservation thanks to the identity features used, e.g. crooked nose
at bottom-left, dull eyes at bottom-right and chin dimple at top-left

downscaled to 112 × 112 before identity and content loss.
The pixel loss is defined by pixel level `1 loss function as:
0 R
Lpix = ||I − I ||1 (10)
3.3.4 Landmark Loss
The face recognition network F is pre-trained by the im-
ages that are aligned by similarity transformation to a fixed
landmark template. To be compatible with the network, we
align the input and rendered images under the same settings.
However, this process disregards the aspect ratio and scale
of the reconstruction. Therefore, we employ a deep face
alignment network [13] M(I) : RH×W ×C → R68×2 to
detect landmark locations of the input image and align the
rendered geometry onto it by updating the shape, expression
and camera parameters. That is, camera parameters are op-
timized to align with the pose of image I and geometry pa-
rameters are optimized for the rough shape estimation. As
a natural consequence, this alignment drastically improves
the effectiveness of the pixel and content loss, which are
sensitive to misalignment between the two images.
The alignment error is achieved by point-to-point eu-
clidean distances between detected landmark locations of
the input image and 2D projection of the 3D reconstruc-
tion landmark locations that is available as meta-data of the
shape model. Since landmark locations of the reconstruc-
tion heavily depend on camera parameters, this loss is great
a source of information the alignment of the reconstruction
onto input image and is formulated as following:
Llan = ||M(I0 ) − M(IR )||2 (11)
3.4. Model Fitting
We first roughly align our reconstruction to the input im-
age by optimizing shape, expression and camera parame-
ters by: minpr E(pr ) = λlan Llan . We then simultaneously
optimize all of our parameters with gradient descent and
backpropagation so as to minimize weighted combination
of above loss terms in the following:
min E(p) = λid Lid + λ̂id L̂id + λcon Lcon +λpix Lpix
p
+λlan Llan + λreg Reg({ps,e , pl })
(12)
where we weight each of our loss terms with λ parame-
ters. In order to prevent our shape and expression mod-
els and lighting parameters from exaggeration to arbitrar-
ily bias our loss terms, we regularize those parameters by
Reg({ps,e , pl }).
Fitting with Multiple Images (i.e. Video): While the
proposed approach can fit a 3D reconstruction from a single
image, one can take advantage of more images effectively
when available, e.g. from a video recording. This often
helps to improve reconstruction quality under challenging
conditions, e.g. outdoor, low resolution. While state-of-
the-art methods follow naive approaches by averaging ei-
ther the reconstruction [42] or features-to-be-regressed [16]
 Input Images

Ours

Genova
[16]

A.T.Tran et al.
[42]

Tewari et al.
[39]

Ours
Geometry

Tewari et al.
[39]

L. Tran et al.
[43]

Figure 4: Comparison of our qualitative results with other state-of-the-art methods in MoFA-Test dataset. Rows 2-5 show
comparison with textured geometry and rows 6-8 compare only shapes. The Figure is best viewed in colored and under zoom.
before making a reconstruction, we utilize the power of iter- 4.1. Implementation Details
ative optimization by averaging identity reconstruction pa-
rameters (ps , pt ) after every iteration. For an image set For all of our experiments, a given face image is aligned
I = {I0 , I1 , . . . , Ii , . . . , Ini }, we reformulate our param- to our fixed template using 68 landmark locations detected
eters as p = [ps , pie , pt , pic , pil ] in which we average shape by an hourglass 2D landmark detection [13]. For the iden-
and texture parameters by the following: tity features, we employ ArcFace [12] network’s pretrained
n
X n
X models. For the generator network G, we train a progres-
ps = pis , pt = pit (13) sive growing GAN [24] with around 10,000 UV maps from
i i [7] at the resolution of 512 × 512. We use the Large Scale
4. Experiments Face Model [7] for 3DMM shape model with ns = 158
and the expression model learned from 4DFAB database [8]
This section demonstrates the excellent performance of with ne = 29. During fitting process, we optimize pa-
the proposed approach for 3D face reconstruction and shape rameters using Adam Solver [25] with 0.01 learning rate.
recovery. We verify this by qualitative results in Fig- And we set our balancing factors as the following: λid :
ures 1, 3, qualitative comparisons with the state-of-the-art 2.0, λ̂id : 2.0, λcon : 50.0, λpix : 1.0, λlan : 0.001, λreg :
in Sec. 4.2 and quantitative shape reconstruction experiment {0.05, 0.01}. The Fitting converges in around 30 seconds
on a database with ground truth in Sec. 4.3. on an Nvidia GTX 1080 TI GPU for a single image.
 Cooperative Indoor Outdoor
Method Mean Std. Mean Std. Mean Std.
Tran et al. [42] 1.93 0.27 2.02 0.25 1.86 0.23
Booth et al. [6] 1.82 0.29 1.85 0.22 1.63 0.16
Genova et al. [16] 1.50 0.13 1.50 0.11 1.48 0.11
Ours 0.95 0.107 0.94 0.106 0.94 0.106
(a) I0 (b) IR (c) IR albedo
Table 1: Accuracy results for the meshes on the MICC
Dataset using point-to-plane distance. The table reports the
mean error (Mean), the standard deviation (Std.).

4.2. Qualitative Comparison to the State-of-the-art

Fig. 4 compares our results with the most recent face
reconstruction studies [40, 39, 16, 42, 43] on a subset of (d) IR \ Lid (e) IR \ L̂id (f) IR \ Lcon
MoFA test-set. The first four rows after input images show
a comparison of our shape and texture reconstructions to
[16, 42, 39] and the last three rows show our reconstructed
geometries without texture compared to [39, 43]. All in all,
our method outshines all others with its high fidelity pho-
torealistic texture reconstructions. Both of our texture and
shape reconstructions manifest strong identity characteris- (g) IR \ Lpix (h)IR\{Lid,L̂id,Lcon} (i) IR with T(pt )
tics of the corresponding input images from the thickness
and shape of the eyebrows to wrinkles around the mouth Figure 5: Contributions of the components or loss terms of
and forehead. the proposed approach with an leave-one-out ablation study.
4.3. 3D shape recovery on MICC dataset something that suggests that each of our components signif-
icantly contributes towards a good reconstruction. Fig. 5(c)
We evaluate the shape reconstruction performance of our
indicates albedo is well disentangled from illumination and
method on MICC Florence 3D Faces dataset (MICC) [1] in
our model capture the light direction accurately.
Table 1. The dataset provides 3D scans of 53 subjects as
While Fig. 5(d-f) shows each of the identity terms con-
well as their short video footages under three difficulty set-
tributes to preserve identity, Fig. 5(h) demonstrates the sig-
tings: ’cooperative’, ’indoor’ and ’outdoor’. Unlike [16, 42]
nificance identity features altogether. Still, overall recon-
which processes all the frames in a video, we uniformly
struction utilizes pixel intensities to capture better albedo
sample only 5 frames from each video regardless of their
and illumination as shown in Fig. 5(g). Finally, Fig. 5(i)
zoom level. And, we run our method with multi-image sup-
shows the superiority of our textures over PCA-based ones.
port for these 5 frames for each video separately as shown
in Eq. 13. Each test mesh is cropped at a radius of 95mm
around the tip of the nose according to [42] in order to eval-
5. Conclusion
uate the shape recovery of the inner facial mesh. We per- In this paper, we revisit optimization-based 3D face re-
form dense alignment between each predicted mesh and its construction under a new perspective, that is, we utilize the
corresponding ground truth mesh, by implementing an iter- power of recent machine learning techniques such as GANs
ative closest point (ICP) method [3]. As evaluation metric, and face recognition network as statistical texture model
we follow [16] to measure the error by average symetric and as energy function respectively.
point-to-plane distance. To the best of our knowledge, this is the first time that
Table 1 reports the normalized point-to-plain errors in GANs are used for model fitting and they have shown excel-
millimeters. It is evident that we have improved the abso- lent results for high quality texture reconstruction. The pro-
lute error compared to the other two state-of-the-art meth- posed approach shows identity preserving high fidelity 3D
ods by 36%. Our results are shown to be consistent across reconstructions in qualitative and quantitative experiments.
all different settings with minimal standard deviation from
the mean error. Acknowledgements: Baris Gecer is funded by the Turk-
4.4. Ablation Study ish Ministry of National Education. Stefanos Zafeiriou
acknowledges support by EPSRC Fellowship DEFORM
Fig. 5 shows an ablation study on our method where the (EP/S010203/1) and a Google Faculty Award.
full model reconstructs the input face better than its variants,
References
[1] Andrew D Bagdanov, Alberto Del Bimbo, and Iacopo Masi.
The florence 2d/3d hybrid face dataset. In Proceedings of the
2011 joint ACM workshop on Human gesture and behavior
understanding, pages 79–80. ACM, 2011. 8
[2] Anil Bas, William AP Smith, Timo Bolkart, and Stefanie
Wuhrer. Fitting a 3d morphable model to edges: A com-
parison between hard and soft correspondences. In ACCV,
2016. 4
[3] Paul J Besl and Neil D McKay. Method for registration of 3-
d shapes. In Sensor Fusion IV: Control Paradigms and Data
Structures, volume 1611, pages 586–607, 1992. 8
[4] Volker Blanz and Thomas Vetter. A morphable model for
the synthesis of 3d faces. In Proceedings of the 26th an-
nual conference on Computer graphics and interactive tech-
niques, pages 187–194. ACM Press/Addison-Wesley Pub-
lishing Co., 1999. 1
[5] Volker Blanz and Thomas Vetter. Face recognition based
on fitting a 3d morphable model. TPAMI, 25(9):1063–1074,
2003. 3
[6] James Booth, Epameinondas Antonakos, Stylianos
Ploumpis, George Trigeorgis, Yannis Panagakis, Stefanos
Zafeiriou, et al. 3d face morphable models in-the-wild. In
CVPR, 2017. 2, 3, 4, 8
[7] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan
Ponniah, and David Dunaway. A 3d morphable model learnt
from 10,000 faces. In CVPR, 2016. 2, 7
[8] Shiyang Cheng, Irene Kotsia, Maja Pantic, and Stefanos
Zafeiriou. 4dfab: a large scale 4d facial expression database
for biometric applications. arXiv preprint arXiv:1712.01443,
2017. 7
[9] Forrester Cole, David Belanger, Dilip Krishnan, Aaron
Sarna, Inbar Mosseri, and William T Freeman. Synthesiz-
ing normalized faces from facial identity features. In CVPR,
2017. 2, 5
[10] Hang Dai, Nick Pears, William Smith, and Christian Dun-
can. A 3d morphable model of craniofacial shape and tex-
ture variation. In 2017 IEEE International Conference on
Computer Vision (ICCV), 2017. 2
[11] Jiankang Deng, Shiyang Cheng, Niannan Xue, Yuxiang
Zhou, and Stefanos Zafeiriou. Uv-gan: Adversarial facial uv
map completion for pose-invariant face recognition. CVPR,
2018. 2
[12] Jiankang Deng, Jia Guo, and Stefanos Zafeiriou. Arcface:
Additive angular margin loss for deep face recognition. arXiv
preprint arXiv:1801.07698, 2018. 2, 4, 5, 7
[13] Jiankang Deng, Yuxiang Zhou, Shiyang Cheng, and Stefanos
Zaferiou. Cascade multi-view hourglass model for robust 3d
face alignment. In Automatic Face & Gesture Recognition
(FG), pages 399–403. IEEE, 2018. 4, 6, 7
[14] Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Val-
gaerts, Kiran Varanasi, Patrick Pérez, and Christian
Theobalt. Reconstruction of personalized 3d face rigs from
monocular video. ACM Transactions on Graphics (TOG),
35(3):28, 2016. 2, 3
[15] Baris Gecer, Binod Bhattarai, Josef Kittler, and Tae-Kyun
Kim. Semi-supervised adversarial learning to generate pho-
torealistic face images of new identities from 3d morphable
model. ECCV, 2018. 5
[16] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron
Sarna, Daniel Vlasic, and William T Freeman. Unsupervised
training for 3d morphable model regression. In CVPR, 2018.
2, 5, 6, 7, 8, 11
[17] Ian Goodfellow. Nips 2016 tutorial: Generative adversarial
networks. arXiv preprint arXiv:1701.00160, 2016. 4
[18] Yudong Guo, Juyong Zhang, Jianfei Cai, Boyi Jiang, and
Jianmin Zheng. Cnn-based real-time dense face reconstruc-
tion with inverse-rendered photo-realistic face images. IEEE
transactions on pattern analysis and machine intelligence,
2018. 2
[19] Guosheng Hu, Fei Yan, Josef Kittler, William Christmas,
Chi Ho Chan, Zhenhua Feng, and Patrik Huber. Efficient 3d
morphable face model fitting. Pattern Recognition, 67:366–
379, 2017. 4
[20] Gary B. Huang, Marwan Mattar, Honglak Lee, and Erik
Learned-Miller. Learning to align from scratch. In NIPS,
2012. 11
[21] IEEE. A 3D Face Model for Pose and Illumination Invariant
Face Recognition, 2009. 2
[22] Luo Jiang, Juyong Zhang, Bailin Deng, Hao Li, and Lig-
ang Liu. 3d face reconstruction with geometry details from
a single image. IEEE Transactions on Image Processing,
27(10):4756–4770, 2018. 2
[23] Ian Jolliffe. Principal component analysis. In Interna-
tional encyclopedia of statistical science, pages 1094–1096.
Springer, 2011. 3
[24] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of GANs for improved quality, stability,
and variation. In ICLR, 2018. 4, 7
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for
stochastic optimization. arXiv preprint arXiv:1412.6980,
2014. 7
[26] Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser
Sheikh. Deep appearance models for face rendering. ACM
Transactions on Graphics (TOG), 37(4):68, 2018. 2
[27] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face
recognition. In British Machine Vision Conference, 2015.
11
[28] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, et al.
Deep face recognition. In BMVC, 2015. 4
[29] Marcel Piotraschke and Volker Blanz. Automated 3d face
reconstruction from multiple images using quality measures.
In CVPR, 2016. 4
[30] Elad Richardson, Matan Sela, and Ron Kimmel. 3d face re-
construction by learning from synthetic data. In 2016 Fourth
International Conference on 3D Vision (3DV), pages 460–
469. IEEE, 2016. 2
[31] Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel.
Learning detailed face reconstruction from a single image.
In CVPR, 2017. 2
[32] Sami Romdhani, Volker Blanz, and Thomas Vetter. Face
identification by fitting a 3d morphable model using linear
shape and texture error functions. In ECCV, 2002. 3
[33] Sami Romdhani and Thomas Vetter. Estimating 3d shape
and texture using pixel intensity, edges, specular highlights,
texture constraints and a prior. In CVPR, 2005. 2, 4
[34] Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and
Hao Li. Photorealistic facial texture inference using deep
neural networks. In CVPR, 2017. 2, 12
[35] Florian Schroff, Dmitry Kalenichenko, and James Philbin.
Facenet: A unified embedding for face recognition and clus-
tering. In CVPR, 2015. 4
[36] Matan Sela, Elad Richardson, and Ron Kimmel. Unre-
stricted facial geometry reconstruction using image-to-image
translation. In ICCV, 2017. 2
[37] Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli,
Eli Shechtman, and Dimitris Samaras. Neural face edit-
ing with intrinsic image disentangling. In Computer Vision
and Pattern Recognition (CVPR), 2017 IEEE Conference on,
pages 5444–5453. IEEE, 2017. 11
[38] Ron Slossberg, Gil Shamai, and Ron Kimmel. High quality
facial surface and texture synthesis via generative adversarial
networks. ECCVW, 2018. 2
[39] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian
Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian
Theobalt. Self-supervised multi-level face model learning
for monocular reconstruction at over 250 hz. 2018. 2, 3, 7, 8
[40] Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo
Garrido, Florian Bernard, Patrick Pérez, and Christian
Theobalt. Mofa: Model-based deep convolutional face au-
toencoder for unsupervised monocular reconstruction. In
ICCV, 2017. 2, 8
[41] Justus Thies, Michael Zollhofer, Marc Stamminger, Chris-
tian Theobalt, and Matthias Nießner. Face2face: Real-time
face capture and reenactment of rgb videos. In CVPR, pages
2387–2395, 2016. 2
[42] Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard
Medioni. Regressing robust and discriminative 3d mor-
phable models with a very deep neural network. In CVPR,
2017. 2, 6, 7, 8, 11
[43] Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable
model. In CVPR, 2018. 2, 3, 7, 8
[44] Michael J Wilber, Chen Fang, Hailin Jin, Aaron Hertzmann,
John Collomosse, and Serge J Belongie. Bam! the behance
artistic media dataset for recognition beyond photography. In
ICCV, pages 1211–1220, 2017. 11
[45] Shuco Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie
Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and
Hao Li. High-fidelity facial reflectance and geometry infer-
ence from an unconstrained image. ACM Transactions on
Graphics (TOG), 37(4):162, 2018. 2, 11
[46] Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and
Stan Z Li. Face alignment across large poses: A 3d solu-
tion. In CVPR, 2016. 2
Appendix A. Experiments on LFW
In order to evaluate identity preservation capacity of the
proposed method, we run two face recognition experiments
on Labelled Faces in the Wild (LFW) dataset [20]. Follow-
ing [16], we feed real LFW images and rendered images
of their 3D reconstruction by our method to a pretrained
face recognition network, namely VGG-Face[27]. We then
compute the activations at the embedding layer and measure

Genova et al.
cosine similarity between 1) real and rendered images and
2) renderings of same/different pairs.
In Fig. 6 and 7, we have quantitatively showed that our
method is better at identity preservation and photorealism
(i.e., as the pretrained network is trained by real images)
than other state-of-the-art deep 3D face reconstruction ap-
proaches [16, 42].

Ours.
Rendering-to-photo cosine similarity on LFW

Genova et al.
Tran et al.
Ours

Figure 8: Our results on BAM dataset[44] compared to [16].

Our method is robust to many image deformations and even
capable of recovering identities from paintings thanks to
strong identity features.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Figure 6: Cosine similarity distributions of rendered and

real images LFW based on activations at the embedding
layer of VGG-Face network[27]. Our method achieves
more than 0.5 similarity on average which [16] has 0.35
average similarity and [42] 0.16 average similarity. Camera
and lighting parameters are fixed for all renderings.
Input Image Shu et al. Yamaguchi et al. Ours

Cosine similarity of same/different pairs of LFW

Genova et al. same
Genova et al. different
Ours same
Ours different
Figure 9: Qualitative comparison with [45, 37] by overlay-
ing the reconstructions on the input images. Our method
can generate high fidelity texture with accurate shape, cam-
era and illumination fitting.

Appendix B. More Qualitative Results

Figures 8, 9, 10, and 11 illustrate the reconstructions of
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
our method under different settings in comparison to the
other state-of-the-art methods. Please see figure captions
Figure 7: Our method successfully preserve identity so that for detailed explanation.
distribution of cosine similarity of same/different pairs is
separable by thresholding. Camera and lighting parameters
are fixed for all renderings.
Input Images Saito et al. Ours

Figure 10: Qualitative comparison with [34] by means of texture maps, whole and partial face renderings. Please note that
while our method does not require any particular renderer for special effects, e.g., lighting, [34] produce these renderings
with a commercial renderer called Arnold.
 (a) I0 (b) IR (c) IR
alb. (d) IR−IR
alb. (e) S

Figure 11: Results under more challenging conditions, i.e. strong illuminations, self-occlusions and facial hair. (a) Input
image. (b) Estimated fitting overlayyed including illumination estimation. (c) Overlayyed fitting without illumination. (d)
Pixel-wise intensity difference of (b) to (c). (e) Estimated shape mesh
 DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation,
Segmentation and Re-Identification of Clothing Images
Yuying Ge1 , Ruimao Zhang1 , Lingyun Wu2 , Xiaogang Wang1 , Xiaoou Tang1 , and Ping Luo1
1
The Chinese University of Hong Kong
2
SenseTime Research
arXiv:1901.07973v1 [cs.CV] 23 Jan 2019
Abstract
DeepFashion
Understanding fashion images has been advanced by
benchmarks with rich annotations such as DeepFashion,
whose labels include clothing categories, landmarks, and
consumer-commercial image pairs. However, DeepFash-
ion has nonnegligible issues such as single clothing-item
per image, sparse landmarks (4∼8 only), and no per-pixel
masks, making it had significant gap from real-world sce-
DeepFashion2
narios. We fill in the gap by presenting DeepFashion2 to
address these issues. It is a versatile benchmark of four
tasks including clothes detection, pose estimation, segmen-
tation, and retrieval. It has 801K clothing items where
each item has rich annotations such as style, scale, view-
point, occlusion, bounding box, dense landmarks (e.g. 39
for ‘long sleeve outwear’ and 15 for ‘vest’), and masks.
There are also 873K Commercial-Consumer clothes pairs.
The annotations of DeepFashion2 are much larger than
its counterparts such as 8× of FashionAI Global Chal-
lenge. A strong baseline is proposed, called Match R-
CNN, which builds upon Mask R-CNN to solve the above
four tasks in an end-to-end manner. Extensive evalu-
ations are conducted with different criterions in Deep-
Fashion2. DeepFashion2 Dataset will be released at :
https://github.com/switchablenorms/DeepFashion2
1. Introduction
Fashion image analyses are active research topics in re-
cent years because of their huge potential in industry. With
the development of fashion datasets [20, 5, 7, 3, 14, 12, 21,
1], significant progresses have been achieved in this area
[2, 19, 17, 18, 9, 8].
However, understanding fashion images remains a chal-
lenge in real-world applications, because of large deforma-
tions, occlusions, and discrepancies of clothes across do-
mains between consumer and commercial images. Some
1
(a)
tank top
cardigan
cardigan
tank top
(b)
long sleeve
long sleeve outwear
vest top
vest
shorts
skirt
long sleeve
trousers outwear
shorts
Figure 1. Comparisons between (a) DeepFashion and (b) Deep-
Fashion2. (a) only has single item per image, which is annotated
with 4 ∼ 8 sparse landmarks. The bounding boxes are estimated
from the labeled landmarks, making them noisy. In (b), each im-
age has minimum single item while maximum 7 items. Each item
is manually labeled with bounding box, mask, dense landmarks
(20 per item on average), and commercial-customer image pairs.
challenges can be rooted in the gap between the recent
benchmark and the practical scenario. For example, the
existing largest fashion dataset, DeepFashion [14], has its
own drawbacks such as single clothing item per image,
sparse landmark and pose definition (every clothing cate-
gory shares the same definition of 4 ∼ 8 keypoints), and no
per-pixel mask annotation as shown in Fig.1(a).
To address the above drawbacks, this work presents
DeepFashion2, a large-scale benchmark with comprehen-
sive tasks and annotations of fashion image understanding.
DeepFashion2 contains 491K images of 13 popular cloth-
ing categories. A full spectrum of tasks are defined on
them including clothes detection and recognition, landmark
and pose estimation, segmentation, as well as verification
and retrieval. All these tasks are supported by rich annota-
 2
tions. For instance, DeepFashion2 totally has 801K cloth- hion et I hion
Fas nA Fas
ing items, where each item in an image is labeled with scale, TB
I RN eep daN hio eep
W DA D Mo Fas D
occlusion, zooming, viewpoint, bounding box, dense land- year 2015[5] 2015[7] 2016[14] 2018[21] 2018[1] now
#images 425K 182K 800K 55K 357K 491K
marks, and per-pixel mask, as shown in Fig.1(b). These #categories 11 20 50 13 41 13
items can be grouped into 43.8K clothing identities, where #bboxes 39K 7K × × × 801K
a clothing identity represents the clothes that have almost #landmarks × × 120K × 100K 801K
#masks × × × 119K × 801K
the same cutting, pattern, and design. The images of the #pairs 39K 91K 251K × × 873K
same identity are taken by both customers and commercial Table 1. Comparisons of DeepFashion2 with the other clothes
shopping stores. An item from the customer and an item datasets. The rows represent number of images, bounding boxes,
from the commercial store forms a pair. There are 873K landmarks, per-pixel masks, and consumer-to-shop pairs respec-
pairs that are 3.5 times larger than DeepFashion. The above tively. Bounding boxes inferred from other annotations are not
thorough annotations enable developments of strong algo- counted.
rithms to understand fashion images.
This work has three main contributions. (1) We build as well as 873K pairs. It is the most comprehensive bench-
a large-scale fashion benchmark with comprehensive tasks mark of its kinds to date.
and annotations, to facilitate fashion image analysis. Deep- Fashion Image Understanding. There are various
Fashion2 possesses the richest definitions of tasks and the tasks that analyze clothing images such as clothes detec-
largest number of labels. Its annotations are at least 3.5× of tion [2, 14], landmark prediction [15, 19, 17], clothes seg-
DeepFashion [14], 6.7× of ModaNet [21], and 8× of Fash- mentation [18, 20, 13], and retrieval [7, 5, 14]. However,
ionAI [1]. (2) A full spectrum of tasks is carefully defined a unify benchmark and framework to account for all these
on the proposed dataset. For example, to our knowledge, tasks is still desired. DeepFashion2 and Match R-CNN fill
clothing pose estimation is presented for the first time in the in this blank. We report extensive results for the above
literature by defining landmarks and poses of 13 categories tasks with respect to different variations, including scale,
that are more diverse and fruitful than human pose. (3) With occlusion, zoom-in, and viewpoint. For the task of clothes
DeepFashion2, we extensively evaluate Mask R-CNN [6] retrieval, unlike previous methods [5, 7] that performed
that is a recent advanced framework for visual perception. image-level retrieval, DeepFashion2 enables instance-level
A novel Match R-CNN is also proposed to aggregate all the retrieval of clothing items. We also present a new fashion
learned features from clothes categories, poses, and masks task called clothes pose estimation, which is inspired by
to solve clothing image retrieval in an end-to-end manner. human pose estimation to predict clothing landmarks and
DeepFashion2 and implementations of Match R-CNN will skeletons for 13 clothes categories. This task helps improve
be released. performance of fashion image analysis in real-world appli-
cations.
1.1. Related Work
2. DeepFashion2 Dataset and Benchmark
Clothes Datasets. Several clothes datasets have been
proposed such as [20, 5, 7, 14, 21, 1] as summarized in Overview. DeepFashion2 has four unique characteris-
Table 1. They vary in size as well as amount and type of tics compared to existing fashion datasets. (1) Large Sam-
annotations. For example, WTBI [5] and DARN [7] have ple Size. It contains 491K images of 43.8K clothing iden-
425K and 182K images respectively. They scraped cat- tities of interest (unique garment displayed by shopping
egory labels from metadata of the collected images from stores). On average, each identity has 12.7 items with dif-
online shopping websites, making their labels noisy. In ferent styles such as color and printing. DeepFashion2 con-
contrast, CCP [20], DeepFashion [14], and ModaNet [21] tained 801K items in total. It is the largest fashion database
obtain category labels from human annotators. Moreover, to date. Furthermore, each item is associated with various
different kinds of annotations are also provided in these annotations as introduced above.
datastes. For example, DeepFashion labels 4∼8 landmarks (2) Versatility. DeepFashion2 is developed for multiple
(keypoints) per image that are defined on the functional re- tasks of fashion understanding. Its rich annotations support
gions of clothes (e.g. ‘collar’). The definitions of these clothes detection and classification, dense landmark and
sparse landmarks are shared across all categories, making pose estimation, instance segmentation, and cross-domain
them difficult to capture rich variations of clothing images. instance-level clothes retrieval.
Furthermore, DeepFashion does not have mask annotations. (3) Expressivity. This is mainly reflected in two aspects.
By comparison, ModaNet [21] has street images with masks First, multiple items are present in a single image, unlike
(polygons) of single person but without landmarks. Unlike DeepFashion where each image is labeled with at most one
existing datasets, DeepFashion2 contains 491K images and item. Second, we have 13 different definitions of landmarks
801K instances of landmarks, masks, and bounding boxes, and poses (skeletons) for 13 different categories. There is

2
 Commercial Customer

Scale
（1）
Short sleeve top

Occlusion
Zoom-in （2）
Shorts

Viewpoint （3）
Long sleeve6 outwear
12
7 37
5
3 28
8 16 4 36
15 17 27 29
9 14
30 35
10 13 18 26
31 34
11 12 19 25 32 33

20

（4）
24

21 22 23
Long sleeve dress
Figure 2. Examples of DeepFashion2. The first column shows definitions of dense landmarks and skeletons of four categories. From (1)
to (4), each row represents clothes images with different variations including ‘scale’, ‘occlusion’, ‘zoom-in’, and ‘viewpoint’. At each row,
we partition the images into two groups, the left three columns represent clothes from commercial stores, while the right three columns are
from customers. In each group, the three images indicate three levels of difficulty with respect to the corresponding variation, including (1)
‘small’, ‘moderate’, ‘large’ scale, (2) ‘slight’, ‘medium’, ‘heavy’ occlusion, (3) ‘no’, ‘medium’, ‘large’ zoom-in, (4) ‘not on human’, ‘side’,
‘back’ viewpoint. Furthermore, at each row, the items in these two groups of images are from the same clothing identity but from two
different domains, that is, commercial and customer. The items of the same identity may have different styles such as color and printing.
Each item is annotated with landmarks and masks.

23 defined landmarks for each category on average. Some
definitions are shown in the first column of Fig.2. These
representations are different from human pose and are not
presented in previous work. They facilitate learning of
strong clothes features that satisfy real-world requirements.
(4) Diversity. We collect data by controlling their vari-
ations in terms of four properties including scale, occlu-
sion, zoom-in, and viewpoint as illustrated in Fig.2, making
DeepFashion2 a challenging benchmark. For each property,
each clothing item is assigned to one of three levels of dif-
ficulty. Fig.2 shows that each identity has high diversity
where its items are from different difficulties.
Data Collection and Cleaning. Raw data of DeepFash-
ion2 are collected from two sources including DeepFashion
[14] and online shopping websites. In particular, images
of each consumer-to-shop pair in DeepFashion are included
in DeepFashion2, while the other images are removed. We
further crawl a large set of images on the Internet from both
commercial shopping stores and consumers. To clean up
the crawled set, we first remove shop images with no corre-
sponding consumer-taken photos. Then human annotators
are asked to clean images that contain clothes with large oc-
clusions, small scales, and low resolutions. Eventually we
have 491K images of 801K items and 873K commercial-
consumer pairs.
Variations. We explain the variations in DeepFashion2.
Their statistics are plotted in Fig.3. (1) Scale. We divide all
clothing items into three sets, according to the proportion
of an item compared to the image size, including ‘small’
(< 10%), ‘moderate’ (10% ∼ 40%), and ‘large’ (> 40%).
Fig.3(a) shows that only 50% items have moderate scale.
(2) Occlusion. An item with occlusion means that its re-
gion is occluded by hair, human body, accessory or other
items. Note that an item with its region outside the im-

3
 (a) small slight
moderate 6% medium
24% 26% large heavy
47%
47%
50%
(1) Scale (2) Occlusion
(b) (c)
200000
Instance Number
150000
100000
50000
1000
Figure 3. (a) shows the statistics of different variations in DeepFashion2. (b) is the numbers of items of the 13 categories in DeepFashion2.
(c) shows that categories in DeepFashion [14] have ambiguity. For example, it is difficult to distinguish between ‘cardigan’ and ‘coat’, and
between ‘joggers’ and ‘sweatpants’. They result in ambiguity when labeling data. (d) Top: masks may be inaccurate when complex poses
are presented. Bottom: the masks will be refined by human.
age does not belong to this case. Each item is categorized
by the number of its landmarks that are occluded, includ-
ing ‘partial occlusion’(< 20% occluded keypoints), ‘heavy
occlusion’ (> 50% occluded keypoints), ‘medium occlu-
sion’ (otherwise). More than 50% items have medium or
heavy occlusions as summarized in Fig.3. (3) Zoom-in. An
item with zoom-in means that its region is outside the im-
age. This is categorized by the number of landmarks out-
side image. We define ‘no’, ‘large’ (> 30%), and ‘medium’
zoom-in. We see that more than 30% items are zoomed in.
(4) Viewpoint. We divide all items into four partitions in-
cluding 7% clothes that are not on people, 78% clothes on
people from frontal viewpoint, 15% clothes on people from
side or back viewpoint.
2.1. Data Labeling
Category and Bounding Box. Human annotators are
asked to draw a bounding box and assign a category label
for each clothing item. DeepFashion [14] defines 50 cat-
egories but half of them contain less than 5‰ number of
images. Also, ambiguity exists between 50 categories mak-
ing data labeling difficult as shown in Fig.3(c). By grouping
categories in DeepFashion, we derive 13 popular categories
without ambiguity. The numbers of items of 13 categories
are shown in Fig.3(b).
Clothes Landmark, Contour, and Skeleton. As differ-
ent categories of clothes (e.g. upper- and lower-body gar-
ment) have different deformations and appearance changes,
we represent each category by defining its pose, which is a
set of landmarks as well as contours and skeletons between
landmarks. They capture shapes and structures of clothes.
Pose definitions are not presented in previous work and are
significantly different from human pose. For each clothing
item of a category, human annotations are asked to label
no no wear
(d)
7%
12% medium frontal
7%
large 8% side
21% back
67%
78%
(3) Zoom-in (4) Viewpoint
Cardigan Coat Joggers Sweatpants
landmarks following these instructions.
Moreover, each landmark is assigned one of the two
modes, ‘visible’ or ‘occluded’. We then generate contours
and skeletons automatically by connecting landmarks in a
certain order. To facilitate this process, annotators are also
asked to distinguish landmarks into two types, that is, con-
tour point or junction point. The former one refers to key-
points at the boundary of an item, while the latter one is
assigned to keypoints in conjunction e.g. ‘endpoint of strap
on sling’. The above process controls the labeling quality,
because the generated skeletons help the annotators reex-
amine whether the landmarks are labeled with good quality.
In particular, only when the contour covers the entire item,
the labeled results are eligible, otherwise keypoints will be
refined.
Mask. We label per-pixel mask for each item in a semi-
automatic manner with two stages. The first stage automat-
ically generates masks from the contours. In the second
stage, human annotators are asked to refine the masks, be-
cause the generated masks may be not accurate when com-
plex human poses are presented. As shown in Fig.3(d), the
mark is inaccurate when an image is taken from side-view
of people crossing legs. The masks will be refined by hu-
man.
Style. As introduced before, we collect 43.8K different
clothing identities where each identity has 13 items on av-
erage. These items are further labeled with different styles
such as color, printing, and logo. Fig.2 shows that a pair
of clothes that have the same identity could have different
styles.
2.2. Benchmarks
We build four benchmarks by using the images and la-
bels from DeepFashion2. For each benchmark, there are
4
391K images for training, 34K images for validation and
𝐼" FN RoIAlign 14x14 14x14 28x28
landmark
PN
67K images for test. x256 x512 x32

Clothes Detection. This task detects clothes in an im- class

ResNet RoIAlign 7x7x
age by predicting bounding boxes and category labels. The 256 1024 1024
FPN
evaluation metrics are the bounding box’s average preci- box
RoIAlign 28x28
sion APbox , APIoU=0.50
box , and APIoU=0.75
box by following 14x14
x256
14x14
x256 x256 mask
COCO [11].
Landmark Estimation. This task aims to predict land- z
256 𝑣" MN
marks for each detected clothing item in an each image. NxN NxNx
1024 matching score
x256 1024
Similarly, we employ the evaluation metrics used by COCO Square FC
𝐼" Sub
for human pose estimation by calculating the average pre-
NxNx
cision for keypoints APpt , APOKS=0.50 , and APOKS=0.75 , NxN 1024 not matching score
pt pt FN x256
1024
256 𝑣#
where OKS indicates the object landmark similarity.
Segmentation. This task assigns a category label 𝐼# PN
(including background label) to each pixel in an item.
The evaluation metrics is the average precision includ- Figure 4. Diagram of Match R-CNN that contains three main
IoU=0.50
ing APmask , APmask , and APIoU=0.75
mask computed over components including a feature extraction network (FN), a per-
masks. ception network (PN), and a match network (MN).
Commercial-Consumer Clothes Retrieval. Given a
detected item from a consumer-taken photo, this task aims
to search the commercial images in the gallery for the items ture with lateral connections to build a pyramid of feature
that are corresponding to this detected item. This setting maps. RoIAlign extracts features from different levels of
is more realistic than DeepFashion [14], which assumes the pyramid map.
ground-truth bounding box is provided. In this task, top-k In the second stage, PN contains three streams of net-
retrieval accuracy is employed as the evaluation metric. We works including landmark estimation, clothes detection,
emphasize the retrieval performance while still consider the and mask prediction as shown in Fig.4. The extracted RoI
influence of detector. If a clothing item fails to be detected, features after the first stage are fed into three streams in
this query item is counted as missed. In particular, we have PN separately. The clothes detection stream has two hidden
more than 686K commercial-consumer clothes pairs in the fully-connected (fc) layers, one fc layer for classification,
training set. In the validation set, there are 10, 990 con- and one fc layer for bounding box regression. The stream of
sumer images with 12, 550 items as a query set, and 21, 438 landmark estimation has 8 ‘conv’ layers and 2 ‘deconv’ lay-
commercial images with 37, 183 items as a gallery set. In ers to predict landmarks. Segmentation stream has 4 ‘conv’
the test set, there are 21, 550 consumer images with 24, 402 layers, 1 ‘deconv’ layer, and another ‘conv’ layer to predict
items as queries, while 43, 608 commercial images with masks.
75, 347 items in the gallery.
In the third stage, MN contains a feature extractor and
a similarity learning network for clothes retrieval. The
3. Match R-CNN learned RoI features after the FN component are highly
We present a strong baseline model built upon Mask R- discriminative with respect to clothes category, pose, and
CNN [6] for DeepFashion2, termed Match R-CNN, which mask. They are fed into MN to obtain features vectors
is an end-to-end training framework that jointly learns for retrieval, where v1 and v2 are passed into the similar-
clothes detection, landmark estimation, instance segmenta- ity learning network to obtain the similarity score between
tion, and consumer-to-shop retrieval. The above tasks are the detected clothing items in I1 and I2 . Specifically, the
solved by using different streams and stacking a Siamese feature extractor has 4 ‘conv’ layers, one pooling layer, and
module on top of these streams to aggregate learned fea- one fc layer. The similarity learning network consists of
tures. subtraction and square operator and a fc layer, which esti-
As shown in Fig.4, Match R-CNN employs two images mates the probability of whether two clothing items match
I1 and I2 as inputs. Each image is passed through three or not.
main components including a Feature Network (FN), a Per- Loss Functions. The parameters Θ of the Match R-CNN
ception Network (PN), and a Matching Network (MN). In are optimized by minimizing five loss functions, which are
the first stage, FN contains a ResNet-FPN [10] backbone, formulated as minΘ L = λ1 Lcls + λ2 Lbox + λ3 Lpose +
a region proposal network (RPN) [16] and RoIAlign mod- λ4 Lmask + λ5 Lpair , including a cross-entropy (CE) loss
ule. An image is first fed into ResNet50 to extract features, Lcls for clothes classification, a smooth loss [4] Lbox for
which are then fed into a FPN that uses a top-down architec- bounding box regression, a CE loss Lpose for landmark es-

5
 scale occlusion zoom-in viewpoint overall
small moderate large slight medium heavy no medium large no wear frontal side or back
APbox 0.604 0.700 0.660 0.712 0.654 0.372 0.695 0.629 0.466 0.624 0.681 0.641 0.667
APIoU=0.50
box 0.780 0.851 0.768 0.844 0.810 0.531 0.848 0.755 0.563 0.713 0.832 0.796 0.814
APIoU=0.75
box 0.717 0.809 0.744 0.812 0.768 0.433 0.806 0.718 0.525 0.688 0.791 0.744 0.773

Table 2. Clothes detection of Mask R-CNN [6] on different validation subsets, including scale, occlusion, zoom-in, and viewpoint. The
evaluation metrics are APbox , APIoU=0.50
box , and APIoU=0.75
box . The best performance of each subset is bold.

(a)
long sleeve dress long sleeve outwear long sleeve top
long sleeve outwear
long sleeve top
shorts
short sleeve dress
long sleeve dress 0.80

(b) vest vest

outwear sling

skirt vest dress

Figure 5. (a) shows failure cases in clothes detection while (b) shows failure cases in clothes segmentation. In (a) and (b), the missing
bounding boxes are drawn in red while the correct category labels are also in red. Inaccurate masks are also highlighted by arrows in (b).
For example, clothes fail to be detected or segmented in too small scale, too large scale, large non-rigid deformation, heavy occlusion, large
zoom-in, side or back viewpoint.

timation, a CE loss Lmask for clothes segmentation, and a
CE loss Lpair for clothes retrieval. Specifically, Lcls , Lbox ,
Lpose , and LPmask are identical as defined in [6]. We have
n
Lpair = − n1 i=1 [yi log(ŷi ) + (1 − yi )log(1 − ŷi )], where
yi = 1 indicates the two items of a pair are matched, other-
wise yi = 0.
Implementations. In our experiments, each training im-
age is resized to its shorter edge of 800 pixels with its longer
edge that is no more than 1333 pixels. Each minibatch has
two images in a GPU and 8 GPUs are used for training.
For minibatch size 16, the learning rate (LR) schedule starts
at 0.02 and is decreased by a factor of 0.1 after 8 epochs
and then 11 epochs, and finally terminates at 12 epochs.
This scheduler is denoted as 1x. Mask R-CNN adopts 2x
schedule for clothes detection and segmentation where ‘2x’
is twice as long as 1x with the LR scaled proportionally.
Then It adopts s1x for landmark and pose estimation where
s1x scales the 1x schedule by roughly 1.44x. Match R-
CNN uses 1x schedule for consumer-to-shop clothes re-
trieval. The above models are trained by using SGD with
a weight decay of 10−5 and momentum of 0.9.
In our experiments, the RPN produces anchors with 3 as-
pect rations on each level of the FPN pyramid. In clothes
detection stream, an RoI is considered positive if its IoU
with a ground truth box is larger than 0.5 and negative oth-
erwise. In clothes segmentation stream, positive RoIs with
foreground label are chosen while in landmark estimation
stream, positive RoIs with visible landmarks are selected.
We define ground truth box of interest as clothing items
whose style number is > 0 and can constitute matching
pairs. In clothes retrieval stream, RoIs are selected if their
IoU with a ground truth box of interest is larger than 0.7. If
RoI features are extracted from landmark estimation stream,
RoIs with visible landmarks are also selected.
Inference. At testing time, images are resized in the
same way as the training stage. The top 1000 proposals with
detection probabilities are chosen for bounding box classi-
fication and regression. Then non-maximum suppression is
applied to these proposals. The filtered proposals are fed
into the landmark branch and the mask branch separately.
For the retrieval task, each unique detected clothing item in
consumer-taken image with highest confidence is selected
as query.
4. Experiments
We demonstrate the effectiveness of DeepFashion2 by
evaluating Mask R-CNN [6] and Match R-CNN in multiple
tasks including clothes detection and classification, land-
mark estimation, instance segmentation, and consumer-to-

6
 scale occlusion zoom-in viewpoint overall
small moderate large slight medium heavy no medium large no wear frontal side or back
0.587 0.687 0.599 0.669 0.631 0.398 0.688 0.559 0.375 0.527 0.677 0.536 0.641
APpt
0.497 0.607 0.555 0.643 0.530 0.248 0.616 0.489 0.319 0.510 0.596 0.456 0.563
0.780 0.854 0.782 0.851 0.813 0.534 0.855 0.757 0.571 0.724 0.846 0.748 0.820
APOKS=0.50
pt
0.764 0.839 0.774 0.847 0.799 0.479 0.848 0.744 0.549 0.716 0.832 0.727 0.805
0.671 0.779 0.678 0.760 0.718 0.440 0.786 0.633 0.390 0.571 0.771 0.610 0.728
APOKS=0.75
pt
0.551 0.703 0.625 0.739 0.600 0.236 0.714 0.537 0.307 0.550 0.684 0.506 0.641

Table 3. Landmark estimation of Mask R-CNN [6] on different validation subsets, including scale, occlusion, zoom-in, and viewpoint.
Results of evaluation on visible landmarks only and evaluation on both visible and occlusion landmarks are separately shown in each row.
OKS=0.50
The evaluation metrics are APpt , APpt , and APOKS=0.75
pt . The best performance of each subset is bold.

(a)
(a) shop clothes retrieval. To further show the large variations
of DeepFashion2, the validation set is divided into three
subsets according to their difficulty levels in scale, occlu-
sion, zoom-in, and viewpoint. The settings of Mask R-CNN
and Match R-CNN follow Sec.3. All models are trained in
the training set and evaluated in the validation set.
(b) The following sections from 4.1 to 4.4 report results for
different tasks, showing that DeepFashion2 imposes signif-
icant challenges to both Mask R-CNN and Match R-CNN,
which are the recent state-of-the-art systems for visual per-
ception.

4.1. Clothes Detection

(c)
Table 2 summarizes the results of clothes detection on
different difficulty subsets. We see that the clothes of mod-
erate scale, slight occlusion, no zoom-in, and frontal view-
point have the highest detection rates. There are several
observations. First, detecting clothes with small or large
scale reduces detection rates. Some failure cases are pro-
vided in Fig.5(a) where the item could occupy less than 2%
of the image while some occupies more than 90% of the
image. Second, in Table 2, it is intuitively to see that heavy
occlusion and large zoom-in degenerate performance. In
these two cases, large portions of the clothes are invisible
as shown in Fig.5(a). Third, it is seen in Table 2 that the
(d)
(1) (2) clothing items not on human body also drop performance.
Retrieval Accuracy

Retrieval Accuracy

0.6 0.6

0.4 0.4
This is because they possess large non-rigid deformations as
0.2
visualized in the failure cases of Fig.5(a). These variations
0.2

are not presented in previous object detection benchmarks

0 0
1 5 10
Retrieved Instance
15 20 1 5 10
Retrieved Instance
15 20 such as COCO. Fourth, clothes with side or back viewpoint,
class pose mask class pose mask
are much more difficult to detect as shown in Fig.5(a).
pose+class mask+class pose+class mask+class

Figure 6. (a) shows results of landmark and pose estimation. (b) 4.2. Landmark and Pose Estimation
shows results of clothes segmentation. (c) shows queries with top-
5 retrieved clothing items. The first column is the image from the Table 3 summarizes the results of landmark estimation.
customer with bounding box predicted by detection module, and The evaluation of each subset is performed in two settings,
the second to the sixth columns show the retrieval results from the including visible landmark only (the occluded landmarks
store. (d) is the retrieval accuracy of overall query validation set are not evaluated), as well as both visible and occluded
with (1) detected box (2) ground truth box. Evaluation metrics are landmarks. As estimating the occluded landmarks is more
top-1, -5, -10, -15, and -20 retrieval accuracy. difficult than visible landmarks, the second setting generally
provides worse results than the first setting.
In general, we see that Mask R-CNN obtains an overall

7
 scale occlusion zoom-in viewpoint overall
small moderate large slight medium heavy no medium large no wear frontal side or back
APmask 0.634 0.700 0.669 0.720 0.674 0.389 0.703 0.627 0.526 0.695 0.697 0.617 0.680
APIoU=0.50
mask 0.831 0.900 0.844 0.900 0.878 0.559 0.899 0.815 0.663 0.829 0.886 0.843 0.873
APIoU=0.75
mask 0.765 0.838 0.786 0.850 0.813 0.463 0.842 0.740 0.613 0.792 0.834 0.732 0.812

Table 4. Clothes segmentation of Mask R-CNN [6] on different validation subsets, including scale, occlusion, zoom-in, and viewpoint.
The evaluation metrics are APmask , APIoU=0.50
mask , and APIoU=0.75
mask . The best performance of each subset is bold.

scale occlusion zoom-in viewpoint overall

small moderate large slight medium heavy no medium large no wear frontal side or back top-1 top-10 top-20
0.513 0.619 0.547 0.580 0.556 0.503 0.608 0.557 0.441 0.555 0.580 0.533 0.122 0.363 0.464
class
0.445 0.558 0.515 0.542 0.514 0.361 0.557 0.514 0.409 0.508 0.529 0.519 0.104 0.321 0.417
0.695 0.775 0.729 0.752 0.729 0.698 0.769 0.742 0.618 0.725 0.755 0.705 0.255 0.555 0.647
pose
0.619 0.695 0.688 0.704 0.668 0.559 0.700 0.693 0.572 0.682 0.690 0.654 0.234 0.495 0.589
0.641 0.705 0.663 0.688 0.656 0.645 0.708 0.670 0.556 0.650 0.690 0.653 0.187 0.471 0.573
mask
0.584 0.656 0.632 0.657 0.619 0.512 0.663 0.630 0.541 0.628 0.645 0.602 0.175 0.421 0.529
0.752 0.786 0.733 0.754 0.750 0.728 0.789 0.750 0.620 0.726 0.771 0.719 0.268 0.574 0.665
pose+class
0.691 0.730 0.705 0.725 0.706 0.605 0.746 0.709 0.582 0.699 0.723 0.684 0.244 0.522 0.617
0.679 0.738 0.685 0.711 0.695 0.651 0.742 0.699 0.569 0.677 0.719 0.678 0.214 0.510 0.607
mask+class
0.623 0.696 0.661 0.685 0.659 0.568 0.708 0.667 0.566 0.659 0.676 0.657 0.200 0.463 0.564
Table 5. Consumer-to-Shop Clothes Retrieval of Match R-CNN on different subsets of some validation consumer-taken images. Each
query item in these images has over 5 identical clothing items in validation commercial images. Results of evaluation on ground truth box
and detected box are separately shown in each row. The evaluation metrics are top-20 accuracy. The best performance of each subset is
bold.

AP of just 0.563, showing that clothes landmark estimation
could be even more challenging than human pose estima-
tion in COCO. In particular, Table 3 exhibits similar trends
as those from clothes detection. For example, the cloth-
ing items with moderate scale, slight occlusion, no zoom-
in, and frontal viewpoint have better results than the others
subsets. Moreover, heavy occlusion and zoom-in decreases
performance a lot. Some results are given in Fig.6(a).
4.3. Clothes Segmentation
Table 4 summarizes the results of segmentation. The
performance declines when segmenting clothing items with
small and large scale, heavy occlusion, large zoom-in, side
or back viewpoint, which is consistent with those trends in
the previous tasks. Some results are given in Fig.6(b). Some
failure cases are visualized in Fig.5(b).
increases the accuracy. In particular, the learned features
from pose and class achieve better results than the other
features. When comparing learned features from pose and
mask, we find that the former achieves better results, indi-
cating that landmark locations can be more robust across
scenarios.
As shown in Table 5, the performance declines when
small scale, heavily occluded clothing items are presented.
Clothes with large zoom-in achieved the lowest accuracy
because only part of clothes are displayed in the image and
crucial distinguishable features may be missing. Compared
with clothes on people from frontal view, clothes from side
or back viewpoint perform worse due to lack of discrim-
inative features like patterns on the front of tops. Exam-
ple queries with top-5 retrieved clothing items are shown in
Fig.6(c).

4.4. Consumer-to-Shop Clothes Retrieval 5. Conclusions

Table 5 summarizes the results of clothes retrieval. The
retrieval accuracy is reported in Fig. 6(d), where top-1, -
5, -10, and -20 retrieval accuracy are shown. We evaluate
two settings in (c.1) and (c.2), when the bounding boxes
are predicted by the detection module in Match R-CNN and
are provided as ground truths. Match R-CNN achieves a
top-20 accuracy of less than 0.7 with ground-truth bounding
boxes provided, indicating that the retrieval benchmark is
challenging. Furthermore, retrieval accuracy drops when
using detected boxes, meaning that this is a more realistic
setting.
In Table 5, different combinations of the learned features
are also evaluated. In general, the combination of features
This work represented DeepFashion2, a large-scale fash-
ion image benchmark with comprehensive tasks and an-
notations. DeepFashion2 contains 491K images, each of
which is richly labeled with style, scale, occlusion, zoom-
ing, viewpoint, bounding box, dense landmarks and pose,
pixel-level masks, and pair of images of identical item from
consumer and commercial store. We establish benchmarks
covering multiple tasks in fashion understanding, including
clothes detection, landmark and pose estimation, clothes
segmentation, consumer-to-shop verification and retrieval.
A novel Match R-CNN framework that builds upon Mask
R-CNN is proposed to solve the above tasks in end-to-end
manner. Extensive evaluations are conducted in DeepFash-

8
ion2.
The rich data and labels of DeepFashion2 will defi-
nitely facilitate the developments of algorithms to under-
stand fashion images in future work. We will focus on
three aspects. First, more challenging tasks will be explored
with DeepFashion2, such as synthesizing clothing images
by using GANs. Second, it is also interesting to explore
multi-domain learning for clothing images, because fashion
trends of clothes may change frequently, making variations
of clothing images changed. Third, we will introduce more
evaluation metrics into DeepFashion2, such as size, run-
time, and memory consumptions of deep models, towards
understanding fashion images in real-world scenario.
[17] W. Wang, Y. Xu, J. Shen, and S.-C. Zhu. Attentive fashion
grammar network for fashion landmark detection and cloth-
ing category classification. In CVPR, 2018.
[18] K. Yamaguchi, M. Hadi Kiapour, and T. L. Berg. Paper doll
parsing: Retrieving similar styles to parse clothing items. In
ICCV, 2013.
[19] S. Yan, Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Un-
constrained fashion landmark detection via hierarchical re-
current transformer networks. In ACM Multimedia, 2017.
[20] W. Yang, P. Luo, and L. Lin. Clothing co-parsing by joint
image segmentation and labeling. In CVPR, 2014.
[21] S. Zheng, F. Yang, M. H. Kiapour, and R. Piramuthu.
Modanet: A large-scale street fashion dataset with polygon
annotations. In ACM Multimedia, 2018.

References
[1] Fashionai dataset. http://fashionai.alibaba.
com/datasets/.
[2] H. Chen, A. Gallagher, and B. Girod. Describing clothing by
semantic attributes. In ECCV, 2012.
[3] Q. Chen, J. Huang, R. Feris, L. M. Brown, J. Dong, and
S. Yan. Deep domain adaptation for describing people based
on fine-grained clothing attributes. In CVPR, 2015.
[4] R. Girshick. Fast r-cnn. In ICCV, 2015.
[5] M. Hadi Kiapour, X. Han, S. Lazebnik, A. C. Berg, and T. L.
Berg. Where to buy it: Matching street clothing photos in
online shops. In ICCV, 2015.
[6] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn.
In ICCV, 2017.
[7] J. Huang, R. S. Feris, Q. Chen, and S. Yan. Cross-domain
image retrieval with a dual attribute-aware ranking network.
In ICCV, 2015.
[8] X. Ji, W. Wang, M. Zhang, and Y. Yang. Cross-domain image
retrieval with attention modeling. In ACM Multimedia, 2017.
[9] L. Liao, X. He, B. Zhao, C.-W. Ngo, and T.-S. Chua. Inter-
pretable multimodal retrieval for fashion products. In ACM
Multimedia, 2018.
[10] T.-Y. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and
S. J. Belongie. Feature pyramid networks for object detec-
tion. In CVPR, 2017.
[11] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-
manan, P. Dollár, and C. L. Zitnick. Microsoft coco: Com-
mon objects in context. In ECCV, 2014.
[12] K.-H. Liu, T.-Y. Chen, and C.-S. Chen. Mvc: A dataset for
view-invariant clothing retrieval and attribute prediction. In
ACM Multimedia, 2016.
[13] S. Liu, X. Liang, L. Liu, K. Lu, L. Lin, X. Cao, and S. Yan.
Fashion parsing with video context. IEEE Transactions on
Multimedia, 17(8):1347–1358, 2015.
[14] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfashion:
Powering robust clothes recognition and retrieval with rich
annotations. In CVPR, 2016.
[15] Z. Liu, S. Yan, P. Luo, X. Wang, and X. Tang. Fashion land-
mark detection in the wild. In ECCV, 2016.
[16] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards
real-time object detection with region proposal networks. In
NIPS, 2015.

9
 Inverse Cooking: Recipe Generation from Food Images
Amaia Salvador1∗ Michal Drozdzal2 Xavier Giro-i-Nieto1 Adriana Romero2
1
Universitat Politecnica de Catalunya 2 Facebook AI Research
{amaia.salvador, xavier.giro}@upc.edu, {adrianars, mdrozdzal}@fb.com
arXiv:1812.06164v2 [cs.CV] 15 Jun 2019
Abstract
People enjoy food photography because they appreciate
food. Behind each meal there is a story described in a com-
plex recipe and, unfortunately, by simply looking at a food
image we do not have access to its preparation process.
Therefore, in this paper we introduce an inverse cooking
system that recreates cooking recipes given food images.
Our system predicts ingredients as sets by means of a novel
architecture, modeling their dependencies without impos-
ing any order, and then generates cooking instructions by
attending to both image and its inferred ingredients simul-
taneously. We extensively evaluate the whole system on the
large-scale Recipe1M dataset and show that (1) we improve
performance w.r.t. previous baselines for ingredient predic-
tion; (2) we are able to obtain high quality recipes by lever-
aging both image and ingredients; (3) our system is able to
produce more compelling recipes than retrieval-based ap-
proaches according to human judgment. We make code and
models publicly available1 .
1. Introduction
Food is fundamental to human existence. Not only does
it provide us with energy—it also defines our identity and
culture [10, 34]. As the old saying goes, we are what we eat,
and food related activities such as cooking, eating and talk-
ing about it take a significant portion of our daily life. Food
culture has been spreading more than ever in the current
digital era, with many people sharing pictures of food they
are eating across social media [31]. Querying Instagram for
#food leads to at least 300M posts; similarly, searching for
#foodie results in at least 100M posts, highlighting the un-
questionable value that food has in our society. Moreover,
eating patterns and cooking culture have been evolving over
time. In the past, food was mostly prepared at home, but
nowadays we frequently consume food prepared by third-
parties (e.g. takeaways, catering and restaurants). Thus,
the access to detailed information about prepared food is
∗ Work done during internship at Facebook AI Research
1 https://github.com/facebookresearch/inversecooking
Title: Biscuits
Ingredients:
Flour, butter, sugar, egg, milk, salt.
Instructions:
- Preheat oven to 450 degrees.
- Cream butter and sugar.
- Add egg and milk.
- Sift flour and salt together.
- Add to creamed mixture.
- Roll out on floured board to 1/4
inch thickness.
- Cut with biscuit cutter.
- Place on ungreased cookie sheet.
- Bake for 10 minutes.
Figure 1: Example of a generated recipe, composed of a
title, ingredients and cooking instructions.
limited and, as a consequence, it is hard to know precisely
what we eat. Therefore, we argue that there is a need for
inverse cooking systems, which are able to infer ingredients
and cooking instructions from a prepared meal.
The last few years have witnessed outstanding improve-
ments in visual recognition tasks such as natural image clas-
sification [47, 14], object detection [42, 41] and semantic
segmentation [27, 19]. However, when comparing to natu-
ral image understanding, food recognition poses additional
challenges, since food and its components have high intra-
class variability and present heavy deformations that occur
during the cooking process. Ingredients are frequently oc-
cluded in a cooked dish and come in a variety of colors,
forms and textures. Further, visual ingredient detection re-
quires high level reasoning and prior knowledge (e.g. cake
will likely contain sugar and not salt, while croissant will
presumably include butter). Hence, food recognition chal-
lenges current computer vision systems to go beyond the
merely visible, and to incorporate prior knowledge to en-
able high-quality structured food preparation descriptions.
Previous efforts on food understanding have mainly fo-
cused on food and ingredient categorization [1, 39, 24].
However, a system for comprehensive visual food recog-
nition should not only be able to recognize the type of meal
or its ingredients, but also understand its preparation pro-
cess. Traditionally, the image-to-recipe problem has been
formulated as a retrieval task [54, 3, 4, 45], where a recipe
is retrieved from a fixed dataset based on the image similar-
ity score in an embedding space. The performance of such
systems highly depends on the dataset size and diversity, as
well as on the quality of the learned embedding. Not sur-
prisingly, these systems fail when a matching recipe for the
image query does not exist in the static dataset.
An alternative to overcome the dataset constraints of re-
trieval systems is to formulate the image-to-recipe problem
as a conditional generation one. Therefore, in this paper, we
present a system that generates a cooking recipe containing
a title, ingredients and cooking instructions directly from
an image. Figure 1 shows an example of a generated recipe
obtained with our method, which first predicts ingredients
from an image and then conditions on both the image and
the ingredients to generate the cooking instructions. To the
best of our knowledge, our system is the first to generate
cooking recipes directly from food images. We pose the in-
struction generation problem as a sequence generation one
conditioned on two modalities simultaneously, namely an
image and its predicted ingredients. We formulate the in-
gredient prediction problem as a set prediction, exploiting
their underlying structure. We model ingredient dependen-
cies while not penalizing for prediction order, thus revising
the question of whether order matters [51]. We extensively
evaluate our system on the large-scale Recipe1M dataset
[45] that contains images, ingredients and cooking instruc-
tions, showing satisfactory results. More precisely, in a hu-
man evaluation study, we show that our inverse cooking sys-
tem outperforms previously introduced image-to-recipe re-
trieval approaches by a large margin. Moreover, using a
small set of images, we show that food image-to-ingredient
prediction is a hard task for humans and that our approach
is able to surpass them.
The contributions of this paper can be summarized as:
– We present an inverse cooking system, which gener-
ates cooking instructions conditioned on an image and
its ingredients, exploring different attention strategies
to reason about both modalities simultaneously.
– We exhaustively study ingredients as both a list and a
set, and propose a new architecture for ingredient pre-
diction that exploits co-dependencies among ingredi-
ents without imposing order.
– By means of a user study we show that ingredient pre-
diction is indeed a difficult task and demonstrate the
superiority of our proposed system against image-to-
recipe retrieval approaches.
2. Related Work
Food Understanding. The introduction of large scale
food datasets, such as Food-101 [1] and Recipe1M [45], to-
gether with a recently held iFood challenge2 has enabled
significant advancements in visual food recognition, by
providing reference benchmarks to train and compare ma-
chine learning approaches. As a result, there is currently
a vast literature in computer vision dealing with a variety
of food related tasks, with special focus in image classifi-
cation [26, 39, 38, 33, 6, 24, 30, 60, 16, 17]. Subsequent
works tackle more challenging tasks such as estimating the
number of calories given a food image [32], estimating food
quantities [5], predicting the list of present ingredients [3, 4]
and finding the recipe for a given image [54, 3, 4, 45, 2].
Additionally, [34] provides a detailed cross-region anal-
ysis of food recipes, considering images, attributes (e.g.
style and course) and recipe ingredients. Food related tasks
have also been considered in the natural language process-
ing literature, where recipe generation has been studied in
the context of generating procedural text from either flow
graphs [13, 36, 35] or ingredients’ checklists [21].
Multi-label classification. Significant effort has been
devoted in the literature to leverage deep neural networks
for multi-label classification, by designing models [49, 8,
56, 37, 53] and studying loss functions [12] well suited for
this task. Early attempts exploit single-label classification
models coupled with binary logistic loss [3], assuming the
independence among labels and dropping potentially rele-
vant information. One way of capturing label dependen-
cies is by relying on label powersets [49]. Powersets con-
sider all possible label combinations, which makes them in-
tractable for large scale problems. Another expensive alter-
native consists in learning the joint probability of the labels.
To overcome this issue, probabilistic classifier chains [8]
and their recurrent neural network-based [53, 37] counter-
parts propose to decompose the joint distribution into con-
ditionals, at the expense of introducing intrinsic ordering.
Note that most of these models require to make a predic-
tion for each of the potential labels. Moreover, joint input
and label embeddings [57, 25, 61] have been introduced to
preserve correlations and predict label sets. As an alterna-
tive, researchers have attempted to predict the cardinality of
the set of labels [43, 44]; however, assuming the indepen-
dence of labels. When it comes to multi-label classification
objectives, binary logistic loss [3], target distribution cross-
entropy [12, 29], target distribution mean squared error [56]
and ranking-based losses [12] have been investigated and
compared. Recent results on large scale datasets outline the
potential of the target distribution loss [29].
Conditional text generation. Conditional text genera-
tion with auto-regressive models has been widely studied in
the literature using both text-based [48, 11, 50, 9] as well
as image-based conditionings [52, 59, 28, 20, 23, 7, 46]. In
neural machine translation, where the goal is to predict the
translation for a given source text into another language, dif-
2 https://www.kaggle.com/c/ifood2018

Image
Encoder Ingredient
eI Decoder
beef
onion
𝜃I 𝜃L tomato
beans
Figure 2: Recipe generation model. We extract image features eI with the image encoder, parametrized by θI . Ingredients
are predicted by θL , and encoded into ingredient embeddings eL with θe . The cooking instruction decoder, parametrized by
θR generates a recipe title and a sequence of cooking steps by attending to image embeddings eI , ingredient embeddings eL ,
and previously predicted words (r0 , ..., rt−1 ).
ferent architecture designs have been studied, including re-
current neural networks [48], convolutional models [11] and
attention based approaches [50]. More recently, sequence-
to-sequence models have been applied to more open-ended
generation tasks, such as poetry [55] and story generation
[23, 9]. Following neural machine translation trends, auto-
regressive models have exhibited promising performance in
image captioning [52, 59, 28, 20, 7, 46], where the goal is to
provide a short description of the image contents, opening
the doors to less constrained problems such as generating
descriptive paragraphs [23] or visual storytelling [18].
3. Generating recipes from images
Generating a recipe (title, ingredients and instructions)
from an image is a challenging task, which requires a si-
multaneous understanding of the ingredients composing the
dish as well as the transformations they went through, e.g.
slicing, blending or mixing with other ingredients. Instead
of obtaining the recipe from an image directly, we argue that
a recipe generation pipeline would benefit from an interme-
diate step predicting the ingredients list. The sequence of
instructions would then be generated conditioned on both
the image and its corresponding list of ingredients, where
the interplay between image and ingredients could provide
additional insights on how the latter were processed to pro-
duce the resulting dish.
Figure 2 illustrates our approach. Our recipe genera-
tion system takes a food image as an input and outputs a
sequence of cooking instructions, which are generated by
means of an instruction decoder that takes as input two em-
beddings. The first one represents visual features extracted
from an image, while the second one encodes the ingre-
dients extracted from the image. We start by introducing
our transfomer-based instruction decoder in Subsection 3.1.
This allows us to formally review the transformer, which we
then study and modify to predict ingredients in an orderless
manner in Subsection 3.2. Finally, we review the optimiza-
tion details in Subsection 3.3.
Add onion and cook until tender
r0 r1 r2 r3 r4 r5
Ingredient
Encoder eL
𝜃R 𝜃R 𝜃R 𝜃R 𝜃R 𝜃R
𝜃E
Instruction Decoder
3.1. Cooking Instruction Transformer
Given an input image with associated ingredients, we
aim to produce a sequence of instructions R = (r1 , ..., rT )
(where rt denotes a word in the sequence) by means of
an instruction transformer [50]. Note that the title is pre-
dicted as the first instruction. This transformer is condi-
tioned jointly on two inputs: the image representation eI
and the ingredient embedding eL . We extract the image
representation with a ResNet-50 [15] encoder and obtain the
ingredient embedding eL by means of a decoder architec-
ture to predict ingredients, followed by a single embedding
layer mapping each ingredient into a fixed-size vector.
The instruction decoder is composed of transformer
blocks, each of them containing two attention layers fol-
lowed by a linear layer [50]. The first attention layer applies
self-attention over previously generated outputs, whereas
the second one attends to the model conditioning in order
to refine the self-attention output. The transformer model
is composed of multiple transformer blocks followed by a
linear layer and a softmax nonlinearity that provides a dis-
tribution over recipe words for each time step t. Figure 3a
illustrates the transformer model, which traditionally is con-
ditioned on a single modality. However, our recipe gen-
erator is conditioned on two sources: the image features
eI ∈ RP ×de and ingredients embeddings eL ∈ RK×de
(P and K denote the number of image and ingredient fea-
tures, respectively, and de is the embedding dimensional-
ity). Thus, we want our attention to reason about both
modalities simultaneously, guiding the instruction genera-
tion process. To that end, we explore three different fusion
strategies (depicted in Figure 3):
– Concatenated attention. This strategy first concate-
nates both image eI and ingredients eL embeddings
over the first dimension econcat ∈ R(K+P )×de . Then,
attention is applied over the combined embeddings.
– Independent attention. This strategy incorporates
two attention layers to deal with the bi-modal condi-
tioning. In this case, one layer attends over the image
embedding eI , whereas the other attends over the in-
 Output probabilities
Softmax
Linear

Add
xN Add & Norm
Feed-forward

Attention
Add & Norm

Attention

e Add & Norm Add & Norm Add & Norm eI/eL Add & Norm
Self-Attention
Attention Attention Attention Attention
Add & Norm

Positional encoding
Embedding eL/eI
[eL eI] eI eL
Outputs (shifted right)

(a) Transformer model [50] (b) Concatenated (c) Independent (d) Sequential

Figure 3: Attention strategies for the instruction decoder. In our experiments, we replace the attention module in the
transformer (a), with three different attention modules (b-d) for cooking instruction generation using multiple conditions.

gredient embeddings eL . The output of both attention
layers is combined via summation operation.
– Sequential attention. This strategy sequentially at-
tends over the two conditioning modalities. In our de-
sign, we consider two orderings: (1) image first where
the attention is first computed over image embeddings
eI and then over ingredient embeddings eL ; and (2)
ingredients first where the order is flipped and we first
attend over ingredient embeddings eL followed by im-
age embeddings eI .
3.2. Ingredient Decoder
Which is the best structure to represent ingredients? On
the one hand, it seems clear that ingredients are a set, since
permuting them does not alter the outcome of the cooking
recipe. On the other hand, we colloquially refer to ingredi-
ents as a list (e.g. list of ingredients), implying some order.
Moreover, it would be reasonable to think that there is some
information in the order in which humans write down the
ingredients in a recipe. Therefore, in this subsection we
consider both scenarios and introduce models that work ei-
ther with a list of ingredients or with a set of ingredients.
A list of ingredients is a variable sized, ordered collec-
tion of unique meal constituents. More precisely, let us de-
fine a dictionary of ingredients of size N as D = {di }N i=0 ,
from which we can obtain a list of ingredients L by select-
ing K elements from D: L = [li ]K i=0 . We encode L as a
binary matrix L of dimensions K × N , with Li,j = 1 if
dj ∈ D is selected and 0 otherwise (one-hot-code represen-
tation). Thus, our training data consists of M image and
ingredient list pairs {(x(i) , L(i) )}M
i=0 . In this scenario, the
goal is to predict L̂ from an image x by maximizing the
following objective:
M
X
arg max log p(L̂(i) = L(i) |x(i) ; θI , θL ),
θI ,θL i=0
where θI and θL represent the learnable parameters of the
image encoder and ingredient decoder, respectively. Since
L denotes a list, we can factorize p(L̂(i) = L(i) |x(i) )
PK (i) (i) (i)
into K conditionals: k=0 log p(L̂k = Lk |x(i) , L<k ) 3
(i) (i)
and parametrize p(L̂k |x(i) , L<k ) as a categorical distribu-
tion. In the literature, these conditionals are usually mod-
eled with auto-regressive (recurrent) models. In our experi-
ments, we choose the transformer model as well. It is worth
mentioning that a potential drawback of this formulation is
that it inherently penalizes for order, which might not nec-
essarily be relevant for ingredients.
A set of ingredients is a variable sized, unordered col-
lection of unique meal constituents. We can obtain a set of
ingredients S by selecting K ingredients from the dictio-
nary D: S = {si }K i=0 . We represent S as a binary vector s
of dimension N , where si = 1 if si ∈ S and 0 otherwise.
Thus, our training data consists of M image and ingredient
set pairs: {(x(i) , s(i) )}Mi=0 . In this case, the goal is to predict
ŝ from an image x by maximizing the following objective:
M
X
arg max log p(ŝ(i) = s(i) |x(i) ; θI , θL ). (2)
θI ,θL i=0
Assuming independence among elements, we can fac-
PN (i) (i)
torize p(ŝ(i) = s(i) |x(i) ) as j=0 log p(ŝj = sj |x(i) ).
However, the ingredients in the set are not necessarily inde-
pendent, e.g. salt and pepper frequently appear together.
To account for element dependencies in the set, we
model the set as a list, i.e. as a product of conditional prob-
abilities, by means of an auto-regressive model such as the
transformer. The transformer predicts ingredients in a list-
(i) (i)
like fashion p(L̂k |x(i) , L<k ), until the end of sequence eos
token is encountered. As mentioned previously, the draw-
back of this approach is that such model design penalizes
(i) (i)
3 Lk denotes the k-th row of L(i) and L<k represents all rows of
(1)
L(i) up to, but not including, the k-th one.

salt onion beans rice eos
l0 l1 l2 l3 l4
pool
𝜃L 𝜃L 𝜃L 𝜃L 𝜃L
Figure 4: Set transformer (TFset ). Softmax probabilities
are pooled across time to avoid penalizing for order.
for order. In order to remove the order in which ingre-
dients are predicted, we propose to aggregate the outputs
across different time-steps by means of a max pooling op-
eration (see Figure 4). Moreover, to ensure that the ingre-
dients in L̂(i) are selected without repetition, we force the
(i) (i)
pre-activation of p(L̂k |x(i) , L<k ) to be −∞ for all previ-
ously selected ingredients at time-steps < k. We train this
model by minimizing the binary cross-entropy between the
predicted ingredients (after pooling) and the ground truth.
Including the eos in the pooling operation would result in
loosing the information of where the token appears. There-
fore, in order to learn the stopping criteria of the ingredient
prediction, we introduce an additional loss accounting for
it. The eos loss is defined as the binary cross-entropy loss
between the predicted eos probability at all time-steps and
the ground truth (represented as a unit step function, whose
value is 0 for the time-steps corresponding to ingredients
and 1 otherwise). In addition to that, we incorporate a car-
dinality `1 penalty, which we found empirically useful. At
inference time, we directly sample from the transformer’s
output. We refer to this model as set transformer.
Alternatively, we could use target distribution
P (i)
p(s(i) |x(i) ) = s(i) / j sj [12, 29] to model the
joint distribution of set elements and train a model by
minimizing the cross-entropy loss between p(s(i) |x(i) ) and
the model’s output distribution p(ŝ(i) |x(i) ). Nonetheless,
it is not clear how to convert the target distribution back to
the corresponding set of elements with variable cardinality.
In this case, we build a feed forward network and train it
with the target distribution cross-entropy loss. To recover
the ingredient set, we propose to greedily sample elements
from a cumulative distribution of sorted output probabil-
ities p(ŝ(i) |x(i) ) and stop the sampling once the sum of
probabilities of selected elements is above a threshold. We
refer to this model as feed forward (target distribution).
3.3. Optimization
We train our recipe transfomer in two stages. In the first
stage, we pre-train the image encoder and ingredients de-
coder as presented in Subsection 3.2. Then, in the second
stage, we train the ingredient encoder and instruction de-
coder (following Subsection 3.1) by minimizing the neg-
ative log-likelihood and adjusting θR and θE . Note that,
while training, the instruction decoder takes as input the
ground truth ingredients. All transformer models are trained
with teacher forcing [58] except for the set transformer.
4. Experiments
This section is devoted to the dataset and the descrip-
tion of implementation details, followed by an exhaustive
analysis of the proposed attention strategies for the cooking
instruction transformer. Further, we quantitatively compare
the proposed ingredient prediction models to previously in-
troduced baselines. Finally, a comparison of our inverse
cooking system with retrieval-based models as well as a
comprehensive user study is provided.
4.1. Dataset
We train and evaluate our models on the Recipe1M
dataset [45], composed of 1 029 720 recipes scraped from
cooking websites. The dataset contains 720 639 training,
155 036 validation and 154 045 test recipes, containing a ti-
tle, a list of ingredients, a list of cooking instructions and
(optionally) an image. In our experiments, we use only
the recipes containing images, and remove recipes with less
than 2 ingredients or 2 instructions, resulting in 252 547
training, 54 255 validation and 54 506 test samples.
Since the dataset was obtained by scraping cooking web-
sites, the resulting recipes are highly unstructured and con-
tain frequently redundant or very narrowly defined cooking
ingredients (e.g. olive oil, virgin olive oil and spanish olive
oil are separate ingredients). Moreover, the ingredient vo-
cabulary contains more than 400 different types of cheese,
and more than 300 types of pepper. As a result, the original
dataset contains 16 823 unique ingredients, which we pre-
process to reduce its size and complexity. First, we merge
ingredients if they share the first or last two words (e.g. ba-
con cheddar cheese is merged into cheddar cheese); then,
we cluster the ingredients that have same word in the first or
in the last position (e.g. gorgonzola cheese or cheese blend
are clustered together into the cheese category); finally we
remove plurals and discard ingredients that appear less than
10 times in the dataset. Altogether, we reduce the ingredi-
ent vocabulary from over 16k to 1 488 unique ingredients.
For the cooking instructions, we tokenize the raw text and
remove words that appear less than 10 times in the dataset,
and replace them with unknown word token. Moreover, we
add special tokens for the start and the end of recipe as well
 Model IoU F1 fluence of visual features on recipe quality, we adapt our
model by removing visual features and predicting instruc-
FFBCE 17.85 30.30
FFIOU 26.25 41.58 tions directly from ingredients (L2R). Our system achieves
Model ppl FFDC 27.22 42.80 a test set perplexity of 8.51, improving both I2R and L2R
FFT D 28.84 44.11 baselines, and highlighting the benefits of using both image
Independent 8.59
and ingredients when generating recipes. L2R surpasses
Seq. img. first 8.53 TFlist 29.48 45.55
I2R with a perplexity of 8.67 vs. 9.66, demonstrating the
Seq. ing. first 8.61 TFlist + shuf. 27.86 43.58
Concatenated 8.50 TFset 31.80 48.26 usefulness of having access to concepts (ingredients) that
are essential to the cooking instructions. Finally, we greed-
Table 1: Model selection (val). Left: Recipe perplexity ily sample instructions from our model and analyze the re-
(ppl). Right: Global ingredient IoU & F1. sults. We notice that generated instructions have an average
of 9.21 sentences containing 9 words each, whereas real,
ground truth instructions have an average of 9.08 sentences
as the end of instruction. This process results in a recipe
of length 12.79. See supplementary material for qualitative
vocabulary of 23 231 unique words.
examples of generated recipes.
4.2. Implementation Details
4.4. Ingredient Prediction
We resize images to 256 pixels in their shortest side and
take random crops of 224 × 224 for training and we select In this section, we compare the proposed ingredient pre-
central 224 × 224 pixels for evaluation. For the instruc- diction approaches to previously introduced models, with
tion decoder, we use a transformer with 16 blocks and 8 the goal of assessing whether ingredients should be treated
multi-head attentions, each one with dimensionality 64. For as lists or sets. We consider models from the multilabel
the ingredient decoder, we use a transformer with 4 blocks classification literature as baselines, and tune them for our
and 2 multi-head attentions, each one with dimensionality purposes. On the one hand, we have models based on feed
of 256. To obtain image embeddings we use the last convo- forward convolutional networks, which are trained to pre-
lutional layer of ResNet-50 model. Both image and ingredi- dict sets of ingredients. We experiment with several losses
ents embedings are of dimension 512. We keep a maximum to train these models, namely binary cross-entropy, soft in-
of 20 ingredients per recipe and truncate instructions to a tersection over union as well as target distribution cross-
maximum of 150 words. The models are trained with Adam entropy. Note that binary cross-entropy is the only one not
optimizer [22] until early-stopping criteria is met (using pa- taking into account dependencies among elements in the set.
tience of 50 and monitoring validation loss). All models are On the other hand, we have sequential models that predict
implemented with PyTorch4 [40]. Additional implementa- lists, imposing order and exploiting dependencies among
tion details are provided in the supplementary material. elements. Finally, we consider recently proposed models
which couple set prediction with cardinality prediction to
4.3. Recipe Generation determine which elements to include in the set [44].
Table 1 (right) reports the results on the validation set
In this section, we compare the proposed multi-modal
for the state-of-the-art baselines as well as the proposed
attention architectures described in Section 3.1. Table 1
approaches. We evaluate the models in terms of Intersec-
(left) reports the results in terms of perplexity on the val-
tion over Union (IoU) and F1 score, computed for accumu-
idation set. We observe that independent attention exhibits
lated counts of T P , F N and F P over the entire dataset
the lowest results, followed by both sequential attentions.
split (following Pascal VOC convention). As shown in the
While the latter have the capability to refine the output with
table, the feed forward model trained with binary cross-
either ingredient or image information consecutively, inde-
entropy [3] (FFBCE ) exhibits the lowest performance on
pendent attention can only do it in one step. This is also
both metrics, which could be explained by the assumed in-
the case of concatenated attention, which achieves the best
dependence among ingredients. These results are already
performance. However, concatenated attention is flexible
notably improved by the method that learns to predict the set
enough to decide whether to give more focus to one modal-
cardinality (FFDC ). Similarly, the performance increases
ity, at the expense of the other, whereas independent atten-
when training the model with structured losses such as soft
tion is forced to include information from both modalities.
IoU (FFIOU ). Our feed forward model trained with tar-
Therefore, we use the concatenated attention model to re-
get distribution (FFT D ) and sampled by thresholding (th
port results on the test set. We compare it to a system go-
= 0.5) the sum of probabilities of selected ingredients out-
ing directly from image-to-sequence of instructions with-
performs all feed forward baselines, including recently pro-
out predicting ingredients (I2R). Moreover, to assess the in-
posed alternatives for set prediction such as [44] (FFDC ).
4 https://pytorch.org/ Note that target distribution models dependencies among
 Card. error # pred. ingrs
FFBCE 5.67 ± 3.10 2.37 ± 1.58
FFDC 2.68 ± 2.07 9.18 ± 2.06
FFIOU 2.46 ± 1.95 7.86 ± 1.72
FFT D 3.02 ± 2.50 8.02 ± 3.24
TFlist 2.49 ± 2.11 7.05 ± 2.77
TFlist + shuffle 3.24 ± 2.50 5.06 ± 1.85
TFset 2.56 ± 1.93 9.43 ± 2.35

Table 2: Ingredient Cardinality. Figure 5: Ingredient prediction results: P@K and F1 per ingredient.

elements in a set and implicitly captures cardinality infor- IoU F1

mation. Following recent literature modeling sets as lists
RI2L [45] 18.92 31.83
[37], we train a transformer network to predict ingredients RI2LR [45] 19.85 33.13 Rec. Prec.
given an image by minimizing the negative log-likelihood
loss (TFlist ). Moreover, we train the same transformer FFT D (ours) 29.82 45.94 RIL2R 31.92 28.94
TFset (ours) 32.11 48.61 Ours 75.47 77.13
by randomly shuffling the ingredients (thus, removing or-
der from the data). Both models exhibit competitive results
when compared to feed forward models, highlighting the Table 3: Test performance against retrieval. Left: Global
importance of modeling dependencies among ingredients. ingredient IoU and F1 scores. Right: Precision and Recall
Finally, our proposed set transformer TFset , which models of ingredients in cooking instructions.
ingredient co-occurrences exploiting the auto-regressive na-
IoU F1 Success %
ture of the model yet satisfying order invariance, achieves
the best results, emphasizing the importance of modeling Human 21.36 35.20 Real 80.33
dependencies, while not penalizing for any given order. Retrieved 18.03 30.55 Retrieved 48.81
The average number of ingredients per sample in Ours 32.52 49.08 Ours 55.47
Recipe1M is 7.99 ± 3.21 after pre-processing. We report
the cardinality prediction errors as well as the average num- Table 4: User studies. Left: IoU & F1 scores for ingredi-
ber of predicted ingredients for each of the tested models in ents obtained with retrieval [45], our approach and humans.
Table 2. TFset is the third best method in terms of cardi- Right: Recipe success rate according to human judgment.
nality error (after FFIOU and TFlist ), while being superior
to all methods in terms of F1 and IoU. Further, Figure 5 task, we use the image embeddings to retrieve the closest
(left) shows the precision score at different values of K. As recipe and report metrics for the ingredients of the retrieved
observed, the plot follows similar trends as Table 1 (right), recipe. We further consider an alternative retrieval archi-
with FFT D being among the most competitive models and tecture, which learns joint embeddings between images and
TFset outperforming all previous baselines for most values ingredients list (ignoring title and instructions). We refer
of K. Figure 5 (right) shows the F1 per ingredient, where to this model as RI2L . Table 3 (left) reports the obtained
the ingredients in the horizontal axes are sorted by score. results on the Recipe1M test set. The RI2LR model outper-
Again, we see that models that exploit dependencies con- forms the RI2L one, which indicates that instructions con-
sistently improve ingredient’s F1 scores, strengthening the tain complementary information that is useful when learn-
importance of modeling ingredient co-occurrences. ing effective embeddings. Furthermore, both of our pro-
posed methods outperform the retrieval-baselines by a large
4.5. Generation vs Retrieval
margin (e.g. TFset outperforms the RI2LR retrieval base-
In this section, we compare our proposed recipe genera- line by 12.26 IoU points and 15.48 F1 score points), which
tion system with retrieval baselines, which we use to search demonstrates the superiority of our models. Finally, Figure
recipes in the entire test set for fair comparison. 6 presents some qualitative results for image-to-ingredient
Ingredient prediction evaluation. We use the retrieval prediction for our model as well as for the retrieval based
model in [45] as a baseline and compare it with our best system. We use blue to highlight the ingredients that are
ingredient predictions models, namely FFT D and FFset . present in the ground truth annotation and red otherwise.
The retrieval model, which we refer to as RI2LR , learns Recipe generation evaluation. We compare our pro-
joint embeddings of images and recipes (title, ingredients posed instruction decoder (which generates instructions
and instructions). Therefore, for the ingredient prediction given an image and ingredients) with a retrieval variant. For
 Ours Retrieved Real and F1 ingredient scores obtained by humans, the retrieval
cheese onion
baseline and our method. Results are included in Table 4
potato butter milk water
pepper soup soup cheese butter potato
(left), underlining the complexity of the task. As shown in
cream salt milk onion corn cheese the table, humans outperform the retrieval baseline (F1 of
butter cream corn onion
35.20% vs 30.55%, respectively). Furthermore, our method
outperforms both human baseline and retrieval based sys-
lemon zucchini oil
shrimp butter tems obtaining F1 of 49.08%. Qualitative comparisons
lemon salt clove pepper shrimp
garlic zucchini
catfish seasoning between generated and human-written recipes (including
pepper soy_sauce juice salt garlic
carrot parsley
juice parsley onion recipes from average and expert users) are provided in the
supplementary material.
sugar
strawberries juice tart_shell sugar
butter vanilla The second study aims at quantifying the quality of the
strawberries sugar
water raspberries
cornstarch juice
wine vinegar generated recipes (ingredients and instructions) with respect
strawberries
cream cream to (1) the real recipes in the dataset, and (2) the ones ob-
tained with the retrieval baseline [45]. With this purpose,
cheese cheese
tomato cracker
muffin we randomly select 150 recipes with their associated im-
cheese
cracker miracle_whip
broccoli ages from the test set and, for each image, we collect the
broccoli lettuce
muffin tomato
tomato corresponding real recipe, the top-1 retrieved recipe and
our generated recipe. We present the users with 15 image-
recipe pairs (randomly chosen among the real, retrieved and
Figure 6: Ingredient prediction examples. We compare
generated ones) asking them to indicate whether the recipe
obtained ingredients with our method and the retrieval base-
matches the image. In the study, we collected answers from
line. Ingredients are displayed in blue if they are present in
105 different users, resulting in an average of 10 responses
the real sample and red otherwise. Best viewed in color.
for each image. Table 4 (right) presents the results of this
study, reporting the success rate of each recipe type. As
a fair comparison, we retrain the retrieval system to find the
it can be observed, the success rate of generated recipes is
cooking instructions given both image and ingredients. In
higher than the success rate of retrieved recipes, stressing
our evaluation, we consider the ground truth ingredients as
the benefits of our approach w.r.t. retrieval.
reference and compute recall and precision w.r.t. the ingre-
dients that appear in the obtained instructions. Thus, recall
computes the percentage of ingredients in the reference that 5. Conclusion
appear in the output instructions, whereas precision mea- In this paper, we introduced an image-to-recipe genera-
sures the percentage of ingredients appearing in the instruc- tion system, which takes a food image and produces a recipe
tions that also appear in the reference. Table 3 (right) dis- consisting of a title, ingredients and sequence of cooking
plays comparison between our model and the retrieval sys- instructions. We first predicted sets of ingredients from
tem. Results show that ingredients appearing in generated food images, showing that modeling dependencies matters.
instructions have better recall and precision scores than the Then, we explored instruction generation conditioned on
ingredients in retrieved instructions. images and inferred ingredients, highlighting the impor-
4.6. User Studies tance of reasoning about both modalities at the same time.
Finally, user study results confirm the difficulty of the task,
In this section, we quantify the quality of predicted in- and demonstrate the superiority of our system against state-
gredients and generated instructions with user studies. In of-the-art image-to-recipe retrieval approaches.
the first study, we compare the performance of our model
against human performance in the task of recipe genera- 6. Acknowledgements
tion (including ingredients and recipe instructions). We ran-
domly select 15 images from the test set, and ask users to We are grateful to Nicolas Ballas, Lluis Castrejon,
select up to 20 distinct ingredients as well as write a recipe Zizhao Zhang and Pascal Vincent for their fruitful com-
that would correspond with the provided image. To re- ments and suggestions. We also want to express our grat-
duce the complexity of the task for humans, we reduced itude to Joelle Pineau for her unwavering support to this
the ingredient vocabulary from 1 488 to 323, by increas- project. Finally, we wish to thank everyone who anony-
ing the frequency threshold from 10 to 1k. We collected mously participated in the user studies.
answers from 31 different users, altogether collecting an This work has been partially developed in the framework
average of 5.5 answers for each image. For fair compar- of projects TEC2013-43935-R and TEC2016-75976-R, fi-
ison, we re-train our best ingredient prediction model on nanced by the Spanish Ministerio de Economa y Competi-
the reduced vocabulary of ingredients. We compute IoU tividad and the European Regional Development Fund.
References
[1] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.
Food-101–mining discriminative components with random
forests. In ECCV, 2014.
[2] Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier,
Nicolas Thome, and Matthieu Cord. Cross-modal retrieval in
the cooking context: Learning semantic text-image embed-
dings. In SIGIR, 2018.
[3] Jing-Jing Chen and Chong-Wah Ngo. Deep-based ingredient
recognition for cooking recipe retrieval. In ACM Multimedia.
ACM, 2016.
[4] Jing-Jing Chen, Chong-Wah Ngo, and Tat-Seng Chua.
Cross-modal recipe retrieval with rich food attributes. In
ACM Multimedia. ACM, 2017.
[5] Mei-Yun Chen, Yung-Hsiang Yang, Chia-Ju Ho, Shih-Han
Wang, Shane-Ming Liu, Eugene Chang, Che-Hua Yeh, and
Ming Ouhyoung. Automatic chinese food identification and
quantity estimation. In SIGGRAPH Asia 2012 Technical
Briefs, 2012.
[6] Xin Chen, Hua Zhou, and Liang Diao. Chinesefoodnet:
A large-scale image dataset for chinese food recognition.
CoRR, abs/1705.02743, 2017.
[7] Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. To-
wards diverse and natural image descriptions via a condi-
tional gan. ICCV, 2017.
[8] Krzysztof Dembczyński, Weiwei Cheng, and Eyke
Hüllermeier. Bayes optimal multilabel classification via
probabilistic classifier chains. In ICML, 2010.
[9] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical
neural story generation. In ACL, 2018.
[10] Claude Fischler. Food, self and identity. Information (Inter-
national Social Science Council), 1988.
[11] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats,
and Yann N. Dauphin. Convolutional sequence to sequence
learning. CoRR, abs/1705.03122, 2017.
[12] Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander To-
shev, and Sergey Ioffe. Deep convolutional ranking for mul-
tilabel image annotation. CoRR, abs/1312.4894, 2013.
[13] Kristian J. Hammond. CHEF: A model of case-based plan-
ning. In AAAI, 1986.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Delving deep into rectifiers: Surpassing human-level perfor-
mance on imagenet classification. In CVPR, 2015.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In CVPR,
2016.
[16] Luis Herranz, Shuqiang Jiang, and Ruihan Xu. Modeling
restaurant context for food recognition. IEEE Transactions
on Multimedia, 2017.
[17] Shota Horiguchi, Sosuke Amano, Makoto Ogawa, and Kiy-
oharu Aizawa. Personalized classifier for food image recog-
nition. IEEE Transactions on Multimedia, 2018.
[18] Qiuyuan Huang, Zhe Gan, Asli Çelikyilmaz, Dapeng Oliver
Wu, Jianfeng Wang, and Xiaodong He. Hierarchically struc-
tured reinforcement learning for topically coherent visual
story generation. CoRR, abs/1805.08191, 2018.
[19] Simon Jégou, Michal Drozdzal, David Vazquez, Adriana
Romero, and Yoshua Bengio. The one hundred layers
tiramisu: Fully convolutional densenets for semantic seg-
mentation. In CVPR-W, 2017.
[20] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align-
ments for generating image descriptions. In CVPR, 2015.
[21] Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. Globally
coherent text generation with neural checklist models. In
EMNLP, 2016.
[22] Diederik P. Kingma and Jimmy Ba. Adam: A method for
stochastic optimization. CoRR, abs/1412.6980, 2014.
[23] Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li
Fei-Fei. A hierarchical approach for generating descriptive
image paragraphs. In CVPR, 2017.
[24] Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun
Yang. Cleannet: Transfer learning for scalable image classi-
fier training with label noise. In CVPR, 2018.
[25] Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang.
Multi-label classification via feature-aware implicit label
space encoding. In ICML, 2014.
[26] Chang Liu, Yu Cao, Yan Luo, Guanling Chen, Vinod
Vokkarane, and Yunsheng Ma. Deepfood: Deep learning-
based food image recognition for computer-aided dietary as-
sessment. In ICOST, 2016.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In
CVPR, 2015.
[28] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.
Knowing when to look: Adaptive attention via a visual sen-
tinel for image captioning. In CVPR, 2017.
[29] Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan,
Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe,
and Laurens van der Maaten. Exploring the limits of weakly
supervised pretraining. CoRR, abs/1805.00932, 2018.
[30] Niki Martinel, Gian Luca Foresti, and Christian Micheloni.
Wide-slice residual networks for food recognition. In WACV,
2018.
[31] Sara McGuire. Food Photo Frenzy: Inside the Instagram
Craze and Travel Trend. https://www.business.
com/articles/food-photo-frenzy-inside-
the-instagram-craze-and-travel-trend/,
2017. [Online; accessed Nov-2018].
[32] Austin Meyers, Nick Johnston, Vivek Rathod, Anoop Korat-
tikara, Alex Gorban, Nathan Silberman, Sergio Guadarrama,
George Papandreou, Jonathan Huang, and Kevin P Murphy.
Im2calories: towards an automated mobile vision food diary.
In ICCV, 2015.
[33] Simon Mezgec and Barbara Koroui Seljak. Nutrinet: A deep
learning food and drink image recognition system for dietary
assessment. Nutrients, 9(7), 2017.
[34] Weiqing Min, Bing-Kun Bao, Shuhuan Mei, Yaohui Zhu,
Yong Rui, and Shuqiang Jiang. You are what you eat: Ex-
ploring rich recipe information for cross-region food analy-
sis. IEEE Transactions on Multimedia, 2018.
[35] Shinsuke Mori, Hirokuni Maeta, Tetsuro Sasada, Koichiro
Yoshino, Atsushi Hashimoto, Takuya Funatomi, and Yoko
Yamakata. Flowgraph2text: Automatic sentence skeleton
 compilation for procedural text generation. In INLG. The [53] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang
Association for Computer Linguistics, 2014. Huang, and Wei Xu. CNN-RNN: A unified framework for
[36] Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, and Tet- multi-label image classification. In CVPR, 2016.
suro Sasada. Flow graph corpus from recipe texts. In LREC. [54] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord,
European Language Resources Association (ELRA), 2014. and Frederic Precioso. Recipe recognition with large multi-
[37] Jinseok Nam, Eneldo Loza Mencı́a, Hyunwoo J Kim, and modal food dataset. In ICMEW, 2015.
Johannes Fürnkranz. Maximizing subset accuracy with re- [55] Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng
current neural networks in multi-label classification. In Wang, and Enhong Chen. Chinese poetry generation with
NeurIPS. 2017. planning based neural network. CoRR, abs/1610.09889,
[38] Chong-Wah Ngo. Deep learning for food recognition. In 2016.
SoICT, 2017. [56] Yunchao Wei, Wei Xia, Junshi Huang, Bingbing Ni, Jian
[39] Ferda Ofli, Yusuf Aytar, Ingmar Weber, Raggi al Hammouri, Dong, Yao Zhao, and Shuicheng Yan. CNN: single-label
and Antonio Torralba. Is saki# delicious?: The food percep- to multi-label. CoRR, abs/1406.5726, 2014.
tion gap on instagram and its relation to health. In ICWWW, [57] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie:
2017. Scaling up to large vocabulary image annotation. In IJCAI,
[40] Adam Paszke, Sam Gross, Soumith Chintala, Gregory 2011.
Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- [58] Ronald J. Williams and David Zipser. A learning algorithm
ban Desmaison, Luca Antiga, and Adam Lerer. Automatic for continually running fully recurrent neural networks. Neu-
differentiation in pytorch. In NeurIPS-W, 2017. ral Comput., 1(2), June 1989.
[41] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali [59] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron
Farhadi. You only look once: Unified, real-time object de- Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua
tection. In CVPR, 2016. Bengio. Show, attend and tell: Neural image caption gen-
[42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. eration with visual attention. In ICML, 2015.
Faster R-CNN: towards real-time object detection with re- [60] Ruihan Xu, Luis Herranz, Shuqiang Jiang, Shuang Wang,
gion proposal networks. In NeurIPS, 2015. Xinhang Song, and Ramesh Jain. Geolocalized modeling for
[43] S Hamid Rezatofighi, Anton Milan, Ehsan Abbasnejad, An- dish recognition. IEEE Transactions on Multimedia, 2015.
thony Dick, Ian Reid, et al. Deepsetnet: Predicting sets with [61] Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-
deep neural networks. In ICCV, 2017. Chiang Frank Wang. Learning deep latent spaces for multi-
[44] S Hamid Rezatofighi, Anton Milan, Qinfeng Shi, Anthony label classification. CoRR, abs/1707.00418, 2017.
Dick, and Ian Reid. Joint learning of set cardinality and state
distribution. AAAI, 2018.
[45] Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin,
Ferda Ofli, Ingmar Weber, and Antonio Torralba. Learning
cross-modal embeddings for cooking recipes and food im-
ages. CVPR, 2017.
[46] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu
Soricut. Conceptual captions: A cleaned, hypernymed, im-
age alt-text dataset for automatic image captioning. In ACL,
2018.
[47] Karen Simonyan and Andrew Zisserman. Very deep convo-
lutional networks for large-scale image recognition. CoRR,
abs/1409.1556, 2014.
[48] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to
sequence learning with neural networks. In NeurIPS, 2014.
[49] Grigorios Tsoumakas and Ioannis Vlahavas. Random k-
labelsets: An ensemble method for multilabel classification.
In Joost N. Kok, Jacek Koronacki, Raomon Lopez de Man-
taras, Stan Matwin, Dunja Mladenič, and Andrzej Skowron,
editors, ECML, 2007.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In NeurIPS, 2017.
[51] Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order
matters: Sequence to sequence for sets. In ICLR, 2016.
[52] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du-
mitru Erhan. Show and tell: A neural image caption gen-
erator. In CVPR, 2015.
7. Supplementary Material
This supplementary material intends to provide further
details as well as qualitative results. In Section 7.1, we de-
scribe additional implementation and training details. Sec-
tion 7.2 presents an analysis of our ingredient vocabulary
before and after its pre-processing. Examples of generated
recipes, displayed together with real ones from the dataset,
are presented in Section 7.3. Section 7.4 includes screen-
shots of the two forms that were used to collect data for the
user studies. Section 7.5 includes examples of human writ-
ten recipes compared to real and generated ones. Finally, in
Section 7.6, we provide examples of generated recipes for
out-of-dataset pictures taken by authors.
7.1. Training Details
Ingredient Prediction. Feed-forward models FFBCE ,
FFT D and FFIOU were trained with a mini-batch size of
300, whereas FFDC was trained with a mini-batch size of
256. All of them were trained with a learning rate of 0.001.
The learning rate for pre-trained ResNet layers was scaled
for each model as follows: 0.01× for FFBCE , FFIOU and
FFDC and 0.1× for FFT D . Transformer list-based models
TFlist were trained with mini-batch size 300 and learning
rate 0.001, scaling the learning rate of ResNet layers with
a factor of 0.1×. Similarly, the set transformer TFset was
trained with mini-batch size of 300 and a learning rate of
0.0001, scaling the learning rate of pre-trained ResNet lay-
ers with a factor of 1.0×. The optimization of TFset mini-
mizes a cost function composed of three terms, namely the
ingredient prediction loss Lingr and the end-of-sequence
loss Leos and the cardinality penalty Lcard . We set the con-
tribution of each term with weights 1000.0 and 1.0 and 1.0,
respectively. We use a label smoothing factor of 0.1 for
all models trained with BCE loss (FFBCE , FFDC , TFset ),
which we found experimentally useful.
Instruction Generation. We use a batch size of 256 and
learning rate of 0.001. Parameters of the image encoder
module are taken from the ingredient prediction model and
frozen during training for instruction generation.
All models are trained with Adam optimizer (β1 = 0.9,
β1 = 0.99 and  =1e-8), exponential decay of 0.99 after
each epoch, dropout probability 0.3 and a maximum num-
ber of 400 epochs (if early stopping criterion is not met).
During training we randomly flip (p = 0.5), rotate (±10
degrees) and translate images (±10% image size on each
axis) for augmentation.
7.2. Ingredient Analysis
We provide visualizations of the ingredient vocabulary
used to train our models. Figure 7 displays each unique
ingredient in the vocabulary before and after our pre-
processing stage. The size of each ingredient word indi-
cates its frequency in the dataset (e.g. butter and salt appear
in many recipes). After filtering and clustering ingredients,
the distribution slightly changes (e.g. pepper becomes the
most frequent ingredient, and popular ingredients such as
olive oil or vegetable oil are clustered into oil). Addition-
ally, we illustrate the high ingredient overlap in the dataset
with an example of the different types of cheese that appear
as different ingredients before pre-processing.
7.3. Generated Recipes
Figure 8 shows additional examples of generated recipes
obtained with our method. We also provide the real recipe
for completeness. Although sometimes far from the real
recipe, our system is able to generate plausible and struc-
tured recipes for the input images. Common mistakes in-
clude failures in ingredient recognition (e.g. stuffed toma-
toes are confused with stuffed peppers in Figure 8b), in-
consistencies between ingredients and instructions (e.g. cu-
cumber is predicted as an ingredient but unused in Figure
8d, and meat is mentioned in the title and instructions but is
not predicted as an ingredient in Figure 8e), and repetitions
in ingredient enumeration (e.g. Stir in tomato sauce, tomato
paste, tomato paste, ... in Figure 8c).
7.4. User Study Forms
We provide screenshots of the two forms used to collect
data for user studies. Figure 9 shows the interface used by
users to select image ingredients (each ingredient was se-
lected using a drop-down menu), and write recipes (as free-
form text). Figure 10 shows the form we used to assess
whether a recipe matched the provided image according to
human judgment.
7.5. Human-written Recipes
In Figure 11 we show examples of recipes written by
humans, which were collected using the form in Figure 9.
We also display the real and generated recipes for complete-
ness. Recipes written by humans tend to be shorter, with an
average of 5.29 instructions of 9.03 words each. In contrast,
our model generates recipes that contain an average of in-
structions 9.21 of 9 words each, which closely matches the
real distribution (9.08 sentences of length 12.79).
7.6. Dine Out: A case study
We test the capabilities of our model to generalize for
out-of-dataset food images. Figure 12 shows recipes ob-
tained for food images taken by authors at their homes or in
restaurants during the weeks prior to the submission.
Figure 7: Ingredient word clouds. The size of each ingredient word is proportional to the frequency of appearance in the
dataset. We display word clouds for ingredients before (7a) and after (7b) our pre-processing step. In 7c we show the different
types of cheese that are clustered together after pre-processing.

(a) Before pre-processing.

(b) After pre-processing.

(c) Types of cheese before pre-processing.

Figure 8: Recipe examples. We show both real and generated recipes for different test images.

(a)
(b)
(c)
(d)
(e)
(f)
Figure 9: User Study 1. Interface for writing recipes and selecting ingredients.
Figure 10: User Study 2. Recipe quality assessment form.
 Figure 11: Written Recipes. Real, generated and human written recipes collected with our user study.

Ingredients Ingredients Ingredients

hamburger, beans, tomato, soup, onion, macaroni, chili, pineapple, greens, stick, tomato, vinaigrette beef, onion, garlic, salt, pepper, parsley, basil, sugar, rice,
sugar, ketchup, broth, butter, pepper broth
Real

Instructions
-Combine all ingredients and cook over medium heat until
potatoes are just tender.
-Turn down heat to low and simmer at least 1.5 hours.
Instructions
-Preheat greased grill to medium-high heat. Instructions
-Grill fruit 3 min. -Brown the beef with spices and onion when browned add
-On each side or until lightly browned on both sides. paste
-Cut fruit into 2-inch sticks; place in large salad bowl. -Bring broth to boil add tomato sauce and spices let boil 3
-Add greens, jicama and tomatoes; toss lightly. minutes add rice bring back to boil cover and let sit off heat
-Drizzle with dressing just before serving. 7 minutes
-Mix rice and sauce enjoy

Ingredients Ingredients Ingredients

chuck, tomato, water, soup greens, cheese, tomato, ranch_dressing, chicken, rice, onion, pepper, oil, salt, cheese, tomato, clove, broth
Generated

italian_dressing Instructions
-In a large skillet, heat oil over medium heat.
-Add onion and garlic and cook until onion is translucent.
Instructions Instructions -Add rice and cook until rice is lightly browned.
-Brown ground chuck in a large dutch oven. -Toss greens with chicken, tomatoes, cheese and dressing in -Add chicken broth, tomatoes, salt, pepper and cayenne
-Drain any grease and add the crushed tomatoes, onion soup large bowl. pepper.
mix and water. -Add dressing; mix lightly. -Bring to a boil.
-Simmer 5 minutes. -Reduce heat and cover.
-Add velveeta shells and cheese, mix well and serve with hot -Simmer for 20 minutes.
rolls. -Remove from heat and let stand covered for 5 minutes.
-Sprinkle with cheese and serve.

Ingredients Ingredients Ingredients

tomato, chili, salt, beef, cheese, oil, pepper tomato, spinach, sprouts, arugula, vinaigrette, zucchini rice, meat, parsley, salt, oil, cheese
Written (1)

Instructions Instructions Instructions

-Wash the tomato -Chop the tomatoes. -Cook the minced meat in a big pan. Put salt and oil into it
-Cut the tomato. -Mix all the ingredients together. -When it is cooked, add the rice and water so it can be boiled
-Put some oil in a pan -Add the vinaigrette on top. -Boil it for 20 minutes
-Add the beef - When all the water has evaporated, add some cheese and
-Add the tomato parsley to enhance the flavor.
-Add the chili and the cheese
-Add some salt and pepper

Ingredients Ingredients Ingredients

beans, seasoning, tomato, beef, spices, hot_sauce, cheese, arugula, tomato, kale, zucchini, onion, oil rice, parsley, broth, meat, salt, water
Written (2)

onion

Instructions Instructions
-Cook the beef in a frying pan till brown. Instructions -Put the sauteed meat in a casserole with hot oil.
-Dice the onions. -Wash and rinse the kale -After it is quite cooked, add the rice.
-Drain the beans. -Wash, cut and slice the onion, the yellow zucchini and the -Add the water.
-Take a slow-cooker and put the beef, tomatoes, beans and cherry tomatoes -Add the broth.
onions. -Add all the ingredients in a bowl and mix -Add salt
-Add the seasoning and spices. -Add some olive oil and your favorite vinagrette -Let it cook until the rice it is done.
-Turn the slow-cooker on medium and cook for around 6 -Add the parsley
hrs.
Figure 12: Dine Out Study. Generated recipes for food images taken by authors.

(a)
(b)

(c)
(d)
 ArcFace: Additive Angular Margin Loss for Deep Face Recognition

Jiankang Deng * Jia Guo ∗ Niannan Xue

Imperial College London InsightFace Imperial College London
j.deng16@imperial.ac.uk guojia@gmail.com n.xue15@imperial.ac.uk

Stefanos Zafeiriou
Imperial College London
arXiv:1801.07698v3 [cs.CV] 9 Feb 2019

s.zafeiriou@imperial.ac.uk

Abstract
One of the main challenges in feature learning using
Deep Convolutional Neural Networks (DCNNs) for large-
scale face recognition is the design of appropriate loss func-
tions that enhance discriminative power. Centre loss pe-
nalises the distance between the deep features and their cor-
responding class centres in the Euclidean space to achieve
intra-class compactness. SphereFace assumes that the lin-
ear transformation matrix in the last fully connected layer
can be used as a representation of the class centres in an
angular space and penalises the angles between the deep
features and their corresponding weights in a multiplicative
way. Recently, a popular line of research is to incorporate
margins in well-established loss functions in order to max-
imise face class separability. In this paper, we propose an
Additive Angular Margin Loss (ArcFace) to obtain highly
discriminative features for face recognition. The proposed
ArcFace has a clear geometric interpretation due to the ex-
act correspondence to the geodesic distance on the hyper-
sphere. We present arguably the most extensive experimen-
tal evaluation of all the recent state-of-the-art face recog-
nition methods on over 10 face recognition benchmarks in-
cluding a new large-scale image database with trillion level
of pairs and a large-scale video dataset. We show that Ar-
cFace consistently outperforms the state-of-the-art and can
be easily implemented with negligible computational over-
head. We release all refined training data, training codes,
pre-trained models and training logs1 , which will help re-
produce the results in this paper.
1. Introduction
Face representation using Deep Convolutional Neural
Network (DCNN) embedding is the method of choice for
∗ denotes equal contribution to this work.
1 https://github.com/deepinsight/insightface
Figure 1. Based on the centre [18] and feature [37] normalisation,
all identities are distributed on a hypersphere. To enhance intra-
class compactness and inter-class discrepancy, we consider four
kinds of Geodesic Distance (GDis) constraint. (A) Margin-Loss:
insert a geodesic distance margin between the sample and cen-
tres. (B) Intra-Loss: decrease the geodesic distance between the
sample and the corresponding centre. (C) Inter-Loss: increase the
geodesic distance between different centres. (D) Triplet-Loss: in-
sert a geodesic distance margin between triplet samples. In this
paper, we propose an Additive Angular Margin Loss (ArcFace),
which is exactly corresponded to the geodesic distance (Arc) mar-
gin penalty in (A), to enhance the discriminative power of face
recognition model. Extensive experimental results show that the
strategy of (A) is most effective.
face recognition [32, 33, 29, 24]. DCNNs map the face im-
age, typically after a pose normalisation step [45], into a
feature that has small intra-class and large inter-class dis-
tance.
There are two main lines of research to train DCNNs
for face recognition. Those that train a multi-class clas-
sifier which can separate different identities in the train-
ing set, such by using a softmax classifier [33, 24, 6], and
those that learn directly an embedding, such as the triplet
loss [29]. Based on the large-scale training data and the
elaborate DCNN architectures, both the softmax-loss-based
methods [6] and the triplet-loss-based methods [29] can ob-
tain excellent performance on face recognition. However,

1
both the softmax loss and the triplet loss have some draw-
backs. For the softmax loss: (1) the size of the linear trans-
formation matrix W ∈ Rd×n increases linearly with the
identities number n; (2) the learned features are separable
for the closed-set classification problem but not discrimina-
tive enough for the open-set face recognition problem. For
the triplet loss: (1) there is a combinatorial explosion in the
number of face triplets especially for large-scale datasets,
leading to a significant increase in the number of iteration
steps; (2) semi-hard sample mining is a quite difficult prob-
lem for effective model training.
Several variants [38, 9, 46, 18, 37, 35, 7, 34, 27] have
been proposed to enhance the discriminative power of the
softmax loss. Wen et al. [38] pioneered the centre loss, the
Euclidean distance between each feature vector and its class
centre, to obtain intra-class compactness while the inter-
class dispersion is guaranteed by the joint penalisation of
the softmax loss. Nevertheless, updating the actual centres
during training is extremely difficult as the number of face
classes available for training has recently dramatically in-
creased.
By observing that the weights from the last fully con-
nected layer of a classification DCNN trained on the soft-
max loss bear conceptual similarities with the centres of
each face class, the works in [18, 19] proposed a multiplica-
tive angular margin penalty to enforce extra intra-class com-
pactness and inter-class discrepancy simultaneously, lead-
ing to a better discriminative power of the trained model.
Even though Sphereface [18] introduced the important idea
of angular margin, their loss function required a series of ap-
proximations in order to be computed, which resulted in an
unstable training of the network. In order to stabilise train-
ing, they proposed a hybrid loss function which includes the
standard softmax loss. Empirically, the softmax loss dom-
inates the training process, because the integer-based mul-
tiplicative angular margin makes the target logit curve very
precipitous and thus hinders convergence. CosFace [37, 35]
directly adds cosine margin penalty to the target logit, which
obtains better performance compared to SphereFace but ad-
mits much easier implementation and relieves the need for
joint supervision from the softmax loss.
In this paper, we propose an Additive Angular Margin
Loss (ArcFace) to further improve the discriminative power
of the face recognition model and to stabilise the training
process. As illustrated in Figure 2, the dot product be-
tween the DCNN feature and the last fully connected layer
is equal to the cosine distance after feature and weight nor-
malisation. We utilise the arc-cosine function to calculate
the angle between the current feature and the target weight.
Afterwards, we add an additive angular margin to the tar-
get angle, and we get the target logit back again by the co-
sine function. Then, we re-scale all logits by a fixed feature
norm, and the subsequent steps are exactly the same as in
the softmax loss. The advantages of the proposed ArcFace
can be summarised as follows:
Engaging. ArcFace directly optimises the geodesic dis-
tance margin by virtue of the exact correspondence between
the angle and arc in the normalised hypersphere. We in-
tuitively illustrate what happens in the 512-D space via
analysing the angle statistics between features and weights.
Effective. ArcFace achieves state-of-the-art performance
on ten face recognition benchmarks including large-scale
image and video datasets.
Easy. ArcFace only needs several lines of code as given
in Algorithm 1 and is extremely easy to implement in the
computational-graph-based deep learning frameworks, e.g.
MxNet [8], Pytorch [25] and Tensorflow [4]. Furthermore,
contrary to the works in [18, 19], ArcFace does not need
to be combined with other loss functions in order to have
stable performance, and can easily converge on any training
datasets.
Efficient. ArcFace only adds negligible computational
complexity during training. Current GPUs can easily sup-
port millions of identities for training and the model parallel
strategy can easily support many more identities.
2. Proposed Approach
2.1. ArcFace
The most widely used classification loss function, soft-
max loss, is presented as follows:
T
N
1 X eWyi xi +byi
L1 = − log Pn T , (1)
N i=1 eWj xi +bj
j=1
where xi ∈ Rd denotes the deep feature of the i-th sample,
belonging to the yi -th class. The embedding feature dimen-
sion d is set to 512 in this paper following [38, 46, 18, 37].
Wj ∈ Rd denotes the j-th column of the weight W ∈ Rd×n
and bj ∈ Rn is the bias term. The batch size and the class
number are N and n, respectively. Traditional softmax loss
is widely used in deep face recognition [24, 6]. However,
the softmax loss function does not explicitly optimise the
feature embedding to enforce higher similarity for intra-
class samples and diversity for inter-class samples, which
results in a performance gap for deep face recognition under
large intra-class appearance variations (e.g. pose variations
[30, 48] and age gaps [22, 49]) and large-scale test scenarios
(e.g. million [15, 39, 21] or trillion pairs [2]).
For simplicity, we fix the bias bj = 0 as in [18]. Then,
we transform the logit [26] as WjT xi = kWj k kxi k cos θj ,
where θj is the angle between the weight Wj and the fea-
ture xi . Following [18, 37, 36], we fix the individual weight
kWj k = 1 by l2 normalisation. Following [28, 37, 36, 35],
we also fix the embedding feature kxi k by l2 normalisation
and re-scale it to s. The normalisation step on features and
Figure 2. Training a DCNN for face recognition supervised by the ArcFace loss. Based on the feature xi and weight W normalisation, we
get the cos θj (logit) for each class as WjT xi . We calculate the arccosθyi and get the angle between the feature xi and the ground truth
weight Wyi . In fact, Wj provides a kind of centre for each class. Then, we add an angular margin penalty m on the target (ground truth)
angle θyi . After that, we calculate cos(θyi + m) and multiply all logits by the feature scale s. The logits then go through the softmax
function and contribute to the cross entropy loss.
Algorithm 1 The Pseudo-code of ArcFace on MxNet
Input: Feature Scale s, Margin Parameter m in Eq. 3, Class Number n, Ground-Truth ID gt.
1. x = mx.symbol.L2Normalization (x, mode = ’instance’)
2. W = mx.symbol.L2Normalization (W, mode = ’instance’)
3. fc7 = mx.sym.FullyConnected (data = x, weight = W, no bias = True, num hidden = n)
4. original target logit = mx.sym.pick (fc7, gt, axis = 1)
5. theta = mx.sym.arccos (original target logit)
6. marginal target logit = mx.sym.cos (theta + m)
7. one hot = mx.sym.one hot (gt, depth = n, on value = 1.0, off value = 0.0)
8. fc7 = fc7 + mx.sym.broadcast mul (one hot, mx.sym.expand dims (marginal target logit - original target logit, 1))
9. fc7 = fc7 * s
Output: Class-wise affinity score f c7.

weights makes the predictions only depend on the angle be- while the proposed ArcFace loss can obviously enforce a
tween the feature and the weight. The learned embedding more evident gap between the nearest classes.
features are thus distributed on a hypersphere with a radius
of s.
N
1 X es cos θyi
L2 = − log s cos θy Pn s cos θj
. (2)
N i=1 e i +
j=1,j6=yi e

As the embedding features are distributed around each

feature centre on the hypersphere, we add an additive angu- (a) Softmax (b) ArcFace
lar margin penalty m between xi and Wyi to simultaneously Figure 3. Toy examples under the softmax and ArcFace loss on
enhance the intra-class compactness and inter-class discrep- 8 identities with 2D features. Dots indicate samples and lines re-
ancy. Since the proposed additive angular margin penalty is fer to the centre direction of each identity. Based on the feature
equal to the geodesic distance margin penalty in the nor- normalisation, all face features are pushed to the arc space with
malised hypersphere, we name our method as ArcFace. a fixed radius. The geodesic distance gap between closest classes
becomes evident as the additive angular margin penalty is incor-
N porated.
1 X es(cos(θyi +m))
L3 = − log s(cos(θ +m)) Pn . 2.2. Comparison with SphereFace and CosFace
N i=1 e yi
+ j=1,j6=yi es cos θj
(3) Numerical Similarity. In SphereFace [18, 19], ArcFace,
We select face images from 8 different identities contain- and CosFace [37, 35], three different kinds of margin
ing enough samples (around 1,500 images/class) to train 2- penalty are proposed, e.g. multiplicative angular margin
D feature embedding networks with the softmax and Ar- m1 , additive angular margin m2 , and additive cosine mar-
cFace loss, respectively. As illustrated in Figure 3, the gin m3 , respectively. From the view of numerical analysis,
softmax loss provides roughly separable feature embedding different margin penalties, no matter add on the angle [18]
but produces noticeable ambiguity in decision boundaries, or cosine space [37], all enforce the intra-class compactness
and inter-class diversity by penalising the target logit [26].
In Figure 4(b), we plot the target logit curves of SphereFace,
ArcFace and CosFace under their best margin settings. We
only show these target logit curves within [20◦ , 100◦ ] be-
cause the angles between Wyi and xi start from around 90◦
(random initialisation) and end at around 30◦ during Arc-
Face training as shown in Figure 4(a). Intuitively, there are
three factors in the target logit curves that affect the perfor-
mance, i.e. the starting point, the end point and the slope.
10000
start
middle
end
8000
SphereFace [18] employs an annealing optimisation strat-
egy. To avoid divergence at the beginning of training, joint
supervision from softmax is used in SphereFace to weaken
the multiplicative margin penalty. We implement a new ver-
sion of SphereFace without the integer requirement on the
margin by employing the arc-cosine function instead of us-
ing the complex double angle formula. In our implementa-
tion, we find that m = 1.35 can obtain similar performance
compared to the original SphereFace without any conver-
gence difficulty.
1
0.8 2.3. Comparison with Other Losses
0.6

0.4

0.2 Other loss functions can be designed based on the angu-

Target Logit

6000
Numbers

4000
-0.2 lar representation of features and weight-vectors. For exam-
Softmax (1.00, 0.00, 0.00)
-0.4

-0.6
SphereFace(m=4, =5)
SphereFace(1.35, 0.00, 0.00)
ArcFace (1.00, 0.50, 0.00)
ples, we can design a loss to enforce intra-class compact-
2000 -0.8

-1
CosFace (1.00, 0.00, 0.35)
CM1
CM2
(1.00, 0.30, 0.20)
(0.90, 0.40, 0.15)
ness and inter-class discrepancy on the hypersphere. As
0
20 30 40 50 60 70 80 90
Angle between the Feature and Target Center
100
-1.2
20 30 40 50 60 70 80
Angle between the Feature and Target Center
90 100 shown in Figure 1, we compare with three other losses in
this paper.
(a) θj Distributions (b) Target Logits Curves
Intra-Loss is designed to improve the intra-class compact-
Figure 4. Target logit analysis. (a) θj distributions from start to ness by decreasing the angle/arc between the sample and
end during ArcFace training. (2) Target logit curves for softmax, the ground truth centre.
SphereFace, ArcFace, CosFace and combined margin penalty
(cos(m1 θ + m2 ) − m3 ). 1 X
N

By combining all of the margin penalties, we implement L5 = L2 + θy . (5)

SphereFace, ArcFace and CosFace in an united framework
with m1 , m2 and m3 as the hyper-parameters.
N
1 X es(cos(m1 θyi +m2 )−m3 )
L4 = − log s(cos(m θ +m )−m ) Pn
N i=1 e 1 yi 2 3 +
j=1,j6=yi e
As shown in Figure 4(b), by combining all of the above-
motioned margins (cos(m1 θ + m2 ) − m3 ), we can easily
get some other target logit curves which also have high per-
formance.
Geometric Difference. Despite the numerical similarity
between ArcFace and previous works, the proposed ad-
ditive angular margin has a better geometric attribute as
the angular margin has the exact correspondence to the
geodesic distance. As illustrated in Figure 5, we compare
the decision boundaries under the binary classification case.
The proposed ArcFace has a constant linear angular margin
throughout the whole interval. By contrast, SphereFace and
CosFace only have a nonlinear angular margin.
Figure 5. Decision margins of different loss functions under bi-
nary classification case. The dashed line represents the decision
boundary, and the grey areas are the decision margins.
The minor difference in margin designs can have “butter-
fly effect” on the model training. For example, the original
πN i=1 i
Inter-Loss targets at enhancing inter-class discrepancy by
increasing the angle/arc between different centres.
s cos θj
. N n
1 X X
(4) L6 = L2 − arccos(WyTi Wj ). (6)
πN (n − 1) i=1
j=1,j6=yi
The Inter-Loss here is a special case of the Minimum
Hyper-spherical Energy (MHE) method [17]. In [17], both
hidden layers and output layers are regularised by MHE. In
the MHE paper, a special case of loss function was also pro-
posed by combining the SphereFace loss with MHE loss on
the last layer of the network.
Triplet-loss aims at enlarging the angle/arc margin between
triplet samples. In FaceNet [29], Euclidean margin is ap-
plied on the normalised features. Here, we employ the
triplet-loss by the angular representation of our features as
arccos(xposi xi ) + m ≤ arccos(xi
neg
xi ).
3. Experiments
3.1. Implementation Details
Datasets. As given in Table 1, we separately employ CA-
SIA [43], VGGFace2 [6], MS1MV2 and DeepGlint-Face
(including MS1M-DeepGlint and Asian-DeepGlint) [2] as
our training data in order to conduct fair comparison with
other methods. Please note that the proposed MS1MV2 is a
semi-automatic refined version of the MS-Celeb-1M dataset
[10]. To best of our knowledge, we are the first to em-
ploy ethnicity-specific annotators for large-scale face image
 Datasets #Identity #Image/Video ResNet50 and 250MB for ResNet100) and extract the 512-
CASIA [43] 10K 0.5M D features (8.9 ms/face for ResNet50 and 15.4 ms/face for
VGGFace2 [6] 9.1K 3.3M ResNet100) for each normalised face. To get the embed-
MS1MV2 85K 5.8M ding features for templates (e.g. IJB-B and IJB-C) or videos
MS1M-DeepGlint [2] 87K 3.9M (e.g. YTF and iQIYI-VID), we simply calculate the feature
Asian-DeepGlint [2] 94 K 2.83M centre of all images from the template or all frames from
LFW [13] 5,749 13,233 the video. Note that, overlap identities between the training
CFP-FP [30] 500 7,000 set and the test set are removed for strict evaluations, and
AgeDB-30 [22] 568 16,488 we only use a single crop for all testing.
CPLFW [48] 5,749 11,652
CALFW [49] 5,749 12,174 3.2. Ablation Study on Losses
YTF [40] 1,595 3,425
In Table 2, we first explore the angular margin setting
MegaFace [15] 530 (P) 1M (G)
for ArcFace on the CASIA dataset with ResNet50. The best
IJB-B [39] 1,845 76.8K
margin observed in our experiments was 0.5. Using the pro-
IJB-C [21] 3,531 148.8K
posed combined margin framework in Eq. 4, it is easier to
Trillion-Pairs [2] 5,749 (P) 1.58M (G)
set the margin of SphereFace and CosFace which we found
iQIYI-VID [20] 4,934 172,835
to have optimal performance when setting at 1.35 and 0.35,
Table 1. Face datasets for training and testing. “(P)” and “(G)” respectively. Our implementations for both SphereFace and
refer to the probe and gallery set, respectively. CosFace can lead to excellent performance without observ-
ing any difficulty in convergence. The proposed ArcFace
annotations, as the boundary cases (e.g. hard samples and achieves the highest verification accuracy on all three test
noisy samples) are very hard to distinguish if the annotator sets. In addition, we performed extensive experiments with
is not familiar with the identity. During training, we explore the combined margin framework (some of the best perfor-
efficient face verification datasets (e.g. LFW [13], CFP-FP mance was observed for CM1 (1, 0.3, 0.2) and CM2 (0.9,
[30], AgeDB-30 [22]) to check the improvement from dif- 0.4, 0.15)) guided by the target logit curves in Figure 4(b).
ferent settings. Besides the most widely used LFW [13] and The combined margin framework led to better performance
YTF [40] datasets, we also report the performance of Ar- than individual SphereFace and CosFace but upper-bounded
cFace on the recent large-pose and large-age datasets(e.g. by the performance of ArcFace.
CPLFW [48] and CALFW [49]). We also extensively test Besides the comparison with margin-based methods, we
the proposed ArcFace on large-scale image datasets (e.g. conduct a further comparison between ArcFace and other
MegaFace [15], IJB-B [39], IJB-C [21] and Trillion-Pairs losses which aim at enforcing intra-class compactness (Eq.
[2]) and video datasets (iQIYI-VID [20]). 5) and inter-class discrepancy (Eq. 6). As the baseline
Experimental Settings. For data prepossessing, we follow we have chosen the softmax loss and we have observed
the recent papers [18, 37] to generate the normalised face performance drop on CFP-FP and AgeDB-30 after weight
crops (112 × 112) by utilising five facial points. For the and feature normalisation. By combining the softmax with
embedding network, we employ the widely used CNN ar- the intra-class loss, the performance improves on CFP-FP
chitectures, ResNet50 and ResNet100 [12, 11]. After the and AgeDB-30. However, combining the softmax with the
last convolutional layer, we explore the BN [14]-Dropout inter-class loss only slightly improves the accuracy. The
[31]-FC-BN structure to get the final 512-D embedding fea- fact that Triplet-loss outperforms Norm-Softmax loss in-
ture. In this paper, we use ([training dataset, network struc- dicates the importance of margin in improving the perfor-
ture, loss]) to facilitate understanding of the experimental mance. However, employing margin penalty within triplet
settings. samples is less effective than inserting margin between sam-
We follow [37] to set the feature scale s to 64 and choose ples and centres as in ArcFace. Finally, we incorporate the
the angular margin m of ArcFace at 0.5. All experiments in Intra-loss, Inter-loss and Triplet-loss into ArcFace, but no
this paper are implemented by MXNet [8]. We set the batch improvement is observed, which leads us to believe that Ar-
size to 512 and train models on four NVIDIA Tesla P40 cFace is already enforcing intra-class compactness, inter-
(24GB) GPUs. On CASIA, the learning rate starts from 0.1 class discrepancy and classification margin.
and is divided by 10 at 20K, 28K iterations. The training To get a better understanding of ArcFace’s superiority,
process is finished at 32K iterations. On MS1MV2, we di- we give the detailed angle statistics on training data (CA-
vide the learning rate at 100K,160K iterations and finish at SIA) and test data (LFW) under different losses in Table
180K iterations. We set momentum to 0.9 and weight decay 3. We find that (1) Wj is nearly synchronised with em-
to 5e − 4. During testing, we only keep the feature embed- bedding feature centre for ArcFace (14.29◦ ), but there is
ding network without the fully connected layer (160MB for an obvious deviation (44.26◦ ) between Wj and the em-
 Loss Functions LFW CFP-FP AgeDB-30 NS ArcFace IntraL InterL TripletL
ArcFace (0.4) 99.53 95.41 94.98 W-EC 44.26 14.29 8.83 46.85 -
ArcFace (0.45) 99.46 95.47 94.93 W-Inter 69.66 71.61 31.34 75.66 -
ArcFace (0.5) 99.53 95.56 95.15 Intra1 50.50 38.45 17.50 52.74 41.19
ArcFace (0.55) 99.41 95.32 95.05 Inter1 59.23 65.83 24.07 62.40 50.23
SphereFace [18] 99.42 - - Intra2 33.97 28.05 12.94 35.38 27.42
SphereFace (1.35) 99.11 94.38 91.70 Inter2 65.60 66.55 26.28 67.90 55.94
CosFace [37] 99.33 - -
CosFace (0.35) 99.51 95.44 94.56 Table 3. The angle statistics under different losses ([CASIA,
CM1 (1, 0.3, 0.2) 99.48 95.12 94.38 ResNet50, loss*]). Each column denotes one particular loss. “W-
CM2 (0.9, 0.4, 0.15) 99.50 95.24 94.86 EC” refers to the mean of angles between Wj and the correspond-
ing embedding feature centre. “W-Inter” refers to the mean of
Softmax 99.08 94.39 92.33
minimum angles between Wj ’s. “Intra1” and “Intra2” refer to the
Norm-Softmax (NS) 98.56 89.79 88.72 mean of angles between xi and the embedding feature centre on
NS+Intra 98.75 93.81 90.92 CASIA and LFW, respectively. “Inter1” and “Inter2” refer to the
NS+Inter 98.68 90.67 89.50 mean of minimum angles between embedding feature centres on
NS+Intra+Inter 98.73 94.00 91.41 CASIA and LFW, respectively.
Triplet (0.35) 98.98 91.90 89.98 Method #Image LFW YTF
ArcFace+Intra 99.45 95.37 94.73 DeepID [32] 0.2M 99.47 93.20
ArcFace+Inter 99.43 95.25 94.55 Deep Face [33] 4.4M 97.35 91.4
ArcFace+Intra+Inter 99.43 95.42 95.10 VGG Face [24] 2.6M 98.95 97.30
ArcFace+Triplet 99.50 95.51 94.40 FaceNet [29] 200M 99.63 95.10
Table 2. Verification results (%) of different loss functions ([CA- Baidu [16] 1.3M 99.13 -
SIA, ResNet50, loss*]). Center Loss [38] 0.7M 99.28 94.9
Range Loss [46] 5M 99.52 93.70
Marginal Loss [9] 3.8M 99.48 95.98
bedding feature centre for Norm-Softmax. Therefore, the
SphereFace [18] 0.5M 99.42 95.0
angles between Wj cannot absolutely represent the inter-
SphereFace+ [17] 0.5M 99.47 -
class discrepancy on training data. Alternatively, the em-
CosFace [37] 5M 99.73 97.6
bedding feature centres calculated by the trained network
MS1MV2, R100, ArcFace 5.8M 99.83 98.02
are more representative. (2) Intra-Loss can effectively com-
press intra-class variations but also brings in smaller inter- Table 4. Verification performance (%) of different methods on
class angles. (3) Inter-Loss can slightly increase inter-class LFW and YTF.
discrepancy on both W (directly) and the embedding net- 3.3. Evaluation Results
work (indirectly), but also raises intra-class angles. (4) Ar-
cFace already has very good intra-class compactness and Results on LFW, YTF, CALFW and CPLFW. LFW [13]
inter-class discrepancy. (5) Triplet-Loss has similar intra- and YTF [40] datasets are the most widely used benchmark
class compactness but inferior inter-class discrepancy com- for unconstrained face verification on images and videos. In
pared to ArcFace. In addition, ArcFace has a more distinct this paper, we follow the unrestricted with labelled outside
margin than Triplet-Loss on the test set as illustrated in Fig- data protocol to report the performance. As reported in Ta-
ure 6. ble 4, ArcFace trained on MS1MV2 with ResNet100 beats
the baselines (e.g. SphereFace [18] and CosFace [37]) by
a significant margin on both LFW and YTF, which shows
104 104
3 2.5

2.5
Negative
Positive
Negative
Positive
that the additive angular margin penalty can notably en-
2

2
hance the discriminative power of deeply learned features,
Pair Numbers

Pair Numbers

1.5

1.5 demonstrating the effectiveness of ArcFace.

1
1
Besides on LFW and YTF datasets, we also report the
0.5
0.5
performance of ArcFace on the recently introduced datasets
0 0
0 10 20 30 40 50 60 70 80 90 100 110 120
Angles Between Positive and Negative Pairs
0 10 20 30 40 50 60 70 80 90 100 110 120
Angles Between Positive and Negative Pairs (e.g. CPLFW [48] and CALFW [49]) which show higher
(a) ArcFace (b) Triplet-Loss pose and age variations with same identities from LFW.
Figure 6. Angle distributions of all positive pairs and random neg- Among all of the open-sourced face recognition models, the
ative pairs (∼ 0.5M) from LFW. Red area indicates positive pairs ArcFace model is evaluated as the top-ranked face recog-
while blue indicates negative pairs. All angles are represented in nition model as shown in Table 5, outperforming coun-
degree. ([CASIA, ResNet50, loss*]). terparts by an obvious margin. In Figure 7, we illustrate
the angle distributions (predicted by ArcFace model trained
 Method LFW CALFW CPLFW Methods Id (%) Ver (%)
HUMAN-Individual 97.27 82.32 81.21 Softmax [18] 54.85 65.92
HUMAN-Fusion 99.85 86.50 85.24 Contrastive Loss[18, 32] 65.21 78.86
Center Loss [38] 98.75 85.48 77.48 Triplet [18, 29] 64.79 78.32
SphereFace [18] 99.27 90.30 81.40 Center Loss[38] 65.49 80.14
VGGFace2 [6] 99.43 90.57 84.00 SphereFace [18] 72.729 85.561
MS1MV2, R100, ArcFace 99.82 95.45 92.08 CosFace [37] 77.11 89.88
AM-Softmax [35] 72.47 84.44
Table 5. Verification performance (%) of open-sourced face recog-
nition models on LFW, CALFW and CPLFW. SphereFace+ [17] 73.03 -
CASIA, R50, ArcFace 77.50 92.34
on MS1MV2 with ResNet100) of both positive and nega- CASIA, R50, ArcFace, R 91.75 93.69
tive pairs on LFW, CFP-FP, AgeDB-30, YTF, CPLFW and FaceNet [29] 70.49 86.47
CALFW. We can clearly find that the intra-variance due CosFace [37] 82.72 96.65
to pose and age gaps significantly increases the angles be- MS1MV2, R100, ArcFace 81.03 96.98
tween positive pairs thus making the best threshold for face MS1MV2, R100, CosFace 80.56 96.56
verification increasing and generating more confusion re- MS1MV2, R100, ArcFace, R 98.35 98.48
gions on the histogram. MS1MV2, R100, CosFace, R 97.91 97.91
250 180 200
Table 6. Face identification and verification evaluation of different
Negative Negative Negative

200
Positive 160

140
Positive

150
Positive
methods on MegaFace Challenge1 using FaceScrub as the probe
120

set. “Id” refers to the rank-1 face identification accuracy with 1M

Pair Numbers

Pair Numbers

Pair Numbers

150
100
100

distractors, and “Ver” refers to the face verification TAR at 10−6

80
100
60
50
40
50

0
20

0 0
FAR. “R” refers to data refinement on both probe set and 1M dis-
tractors. ArcFace obtains state-of-the-art performance under both
0 10 20 30 40 50 60 70 80 90 100 110 120 0 10 20 30 40 50 60 70 80 90 100 110 120 0 10 20 30 40 50 60 70 80 90 100 110 120
Angles Between Positive and Negative Pairs Angles Between Positive and Negative Pairs Angles Between Positive and Negative Pairs

(a) LFW (99.83%) (b) CFP-FP (98.37%) (c) AgeDB (98.15%) small and large protocols.
350

300
Negative
Positive
250
Negative
Positive
180

160
Negative
Positive
affects the performance. Therefore, we manually refined
200
140
250
120 the whole MegaFace dataset and report the correct perfor-
Pair Numbers

Pair Numbers

Pair Numbers

150
200 100

150

100
100
80

60
mance of ArcFace on MegaFace. On the refined MegaFace,
50
50
40

20 ArcFace still clearly outperforms CosFace and achieves the

0 0
0 10 20 30 40 50 60 70 80 90 100 110 120
Angles Between Positive and Negative Pairs
(d) YTF (98.02%)
Figure 7. Angle distributions of both positive and negative pairs on
LFW, CFP-FP, AgeDB-30, YTF, CPLFW and CALFW. Red area
indicates positive pairs while blue indicates negative pairs. All an-
gles are represented in degree. ([MS1MV2, ResNet100, ArcFace])
Results on MegaFace. The MegaFace dataset [15] includes
1M images of 690K different individuals as the gallery set
and 100K photos of 530 unique individuals from FaceScrub
[23] as the probe set. On MegaFace, there are two testing
scenarios (identification and verification) under two proto-
0
0 10 20 30 40 50 60 70 80 90 100 110 120
Angles Between Positive and Negative Pairs
0 10 20 30 40 50 60 70 80 90 100 110 120
Angles Between Positive and Negative Pairs
best performance on both verification and identification.
(e) CPLFW (92.08%) (f) CALFW (95.45%) Under large protocol, ArcFace surpasses FaceNet [29]
by a clear margin and obtains comparable results on iden-
tification and better results on verification compared to
CosFace [37]. Since CosFace employs a private training
data, we retrain CosFace on our MS1MV2 dataset with
ResNet100. Under fair comparison, ArcFace shows supe-
riority over CosFace and forms an upper envelope of Cos-
Face under both identification and verification scenarios as
shown in Figure 8.
100 100

cols (large or small training set). The training set is defined 95

99

98

as large if it contains more than 0.5M images. For the fair

True Positive Rate (%)
Identification Rate (%)

97
90
96
comparison, we train ArcFace on CAISA and MS1MV2 85 95

under the small protocol and large protocol, respectively. 80 CASIA, ResNet50, ArcFace, Original
CASIA, ResNet50, ArcFace, Refine
94

93
CASIA, ResNet50, ArcFace, Original
CASIA, ResNet50, ArcFace, Refine
MS1MV2, ResNet100, ArcFace, Original MS1MV2, ResNet100, ArcFace, Original
In Table 6, ArcFace trained on CASIA achieves the best 75 MS1MV2, ResNet100, ArcFace, Refine
MS1MV2, ResNet100, CosFace, Original
92

91
MS1MV2, ResNet100, ArcFace, Refine
MS1MV2, ResNet100, CosFace, Original
MS1MV2, ResNet100, CosFace, Refine MS1MV2, ResNet100, CosFace, Refine

single-model identification and verification performance, 70

100 101 102 103 104 105 106
90
10-6 10-5 10-4 10-3 10-2 10-1 100

not only surpassing the strong baselines (e.g. SphereFace Rank False Positive Rate

[18] and CosFace [37]) but also outperforming other pub- (a) CMC (b) ROC
lished methods [38, 17]. Figure 8. CMC and ROC curves of different models on MegaFace.
As we observed an obvious performance gap between Results are evaluated on both original and refined MegaFace
identification and verification, we performed a thorough dataset.
manual check in the whole MegaFace dataset and found Results on IJB-B and IJB-C. The IJB-B dataset [39]
many face images with wrong labels, which significantly contains 1, 845 subjects with 21.8K still images and 55K
 Method IJB-B IJB-C Method Id (@FPR=1e-3) Ver(@FPR=1e-9)
ResNet50 [6] 0.784 0.825 CASIA 26.643 21.452
SENet50 [6] 0.800 0.840 MS1MV2 80.968 78.600
ResNet50+SENet50 [6] 0.800 0.841 DeepGlint-Face 80.331 78.586
MN-v [42] 0.818 0.852 MS1MV2+Asian 84.840 (1st) 80.540
MN-vc [42] 0.831 0.862 CIGIT IRSEC 84.234 (2nd) 81.558 (1st)
ResNet50+DCN(Kpts) [41] 0.850 0.867
Table 8. Identification and verification results (%) on the Trillion-
ResNet50+DCN(Divs) [41] 0.841 0.880 Pairs dataset. ([Dataset*, ResNet100, ArcFace])
SENet50+DCN(Kpts) [41] 0.846 0.874
SENet50+DCN(Divs) [41] 0.849 0.885
VGG2, R50, ArcFace 0.898 0.921 set. Every pair between gallery and probe set is used
MS1MV2, R100, ArcFace 0.942 0.956 for evaluation (0.4 trillion pairs in total). In Table 8,
we compare the performance of ArcFace trained on dif-
Table 7. 1:1 verification TAR (@FAR=1e-4) on the IJB-B and IJB- ferent datasets. The proposed MS1MV2 dataset obvi-
C dataset.
ously boosts the performance compared to CASIA and even
frames from 7, 011 videos. In total, there are 12, 115 slightly outperforms the DeepGlint-Face dataset, which has
templates with 10, 270 genuine matches and 8M impos- a double identity number. When combining all identities
tor matches. The IJB-C dataset [39] is a further extension from MS1MV2 and Asian celebrities from DeepGlint, Arc-
of IJB-B, having 3, 531 subjects with 31.3K still images Face achieves the best identification performance 84.840%
and 117.5K frames from 11, 779 videos. In total, there (@FPR=1e-3) and comparable verification performance
are 23, 124 templates with 19, 557 genuine matches and compared to the most recent submission (CIGIT IRSEC)
15, 639K impostor matches. from the lead-board.
On the IJB-B and IJB-C datasets, we employ the VGG2 Results on iQIYI-VID. The iQIYI-VID challenge [20]
dataset as the training data and the ResNet50 as the embed- contains 565,372 video clips (training set 219,677, valida-
ding network to train ArcFace for the fair comparison with tion set 172,860, and test set 172,835) of 4934 identities
the most recent methods [6, 42, 41]. In Table 7, we compare from iQIYI variety shows, films and television dramas. The
the TAR (@FAR=1e-4) of ArcFace with the previous state- length of each video ranges from 1 to 30 seconds. This
of-the-art models [6, 42, 41]. ArcFace can obviously boost dataset supplies multi-modal cues, including face, cloth,
the performance on both IJB-B and IJB-C (about 3 ∼ 5%, voice, gait and subtitles, for character identification. The
which is a significant reduction in the error). Drawing sup- iQIYI-VID dataset employs MAP@100 as the evaluation
port from more training data (MS1MV2) and deeper neu- indicator. MAP (Mean Average Precision) refers to the
ral network (ResNet100), ArcFace can further improve the overall average accuracy rate, which is the mean of the av-
TAR (@FAR=1e-4) to 94.2% and 95.6% on IJB-B and IJB- erage accuracy rate of the corresponding videos of person
C, respectively. In Figure 9, we show the full ROC curves of ID retrieved in the test set for each person ID (as the query)
the proposed ArcFace on IJB-B and IJB-C 2 , and ArcFace in the training set.
achieves impressive performance even at FAR=1e-6 setting As shown in Table 9, ArcFace trained on combined
a new baseline. MS1MV2 and Asian datasets with ResNet100 sets a high
baseline (MAP=(79.80%)). Based on the embedding fea-
1
ROC on IJB-B
1
ROC on IJB-C ture for each training video, we train an additional three-
0.9

0.8
0.9

0.8
layer fully connected network with a classification loss to
0.7 0.7
get the customised feature descriptor on the iQIYI-VID
True Positive Rate

True Positive Rate

0.6 0.6

0.5 0.5 dataset. The MLP learned on the iQIYI-VID training set
0.4 0.4

0.3 0.3 significantly boosts the MAP by 6.60%. Drawing support

0.2 0.2

0.1
MS1MV2, ResNet100, ArcFace
VGG2, ResNet50, ArcFace
0.1
MS1MV2, ResNet100, ArcFace
VGG2, ResNet50, ArcFace
from the model ensemble and context features from the off-
0
10-6 10 -5
10 -4
10 -3

False Positive Rate

10 -2
10 -1
10 0
0
10-6 10-5 10-4 10-3
False Positive Rate
10-2 10-1 100 the-shelf object and scene classifier [1], our final result sur-
passes the runner-up by a clear margin ( 0.99%).
(a) ROC for IJB-B (b) ROC for IJB-C
Figure 9. ROC curves of 1:1 verification protocol on the IJB-B and
IJB-C dataset.
4. Conclusions

Results on Trillion-Pairs. The Trillion-Pairs dataset [2] In this paper, we proposed an Additive Angular Margin
provides 1.58M images from Flickr as the gallery set and Loss function, which can effectively enhance the discrimi-
274K images from 5.7k LFW [13] identities as the probe native power of feature embeddings learned via DCNNs for
face recognition. In the most comprehensive experiments
2 https://github.com/deepinsight/insightface/tree/master/Evaluation/IJB reported in the literature we demonstrate that our method
 Method MAP(%) 25

MS1MV2+Asian, R100, ArcFace 79.80

+ MLP 86.40

GPU Memory Consumption (GB)

20

+ Ensemble 88.26
+ Context 88.65 (1st) 15

Other Participant 87.66 (2nd)

10
Table 9. MAP of our method on the iQIYI-VID test set. “MLP”
refers to a three-layer fully connected network trained on the
iQIYI-VID training data. 5
Parallel Acceleration on Feature (x) -- Data Parallel
consistently outperforms the state-of-the-art. Code and de- Parallel Acceleration on Feature (x) and Center (W)
0
tails have been released under the MIT license. 0 0.5 1 1.5 2 2.5 3
Identity Number in the Training Data 6
10

5. Appendix (a) GPU Memory

5.1. Parallel Acceleration 1000

900
Can we apply ArcFace on large-scale identities? Yes,

Training Speed (samples/second)

800
millions of identities are not a problem.
700
The concept of Centre (W ) is indispensable in ArcFace,
600
but the parameter size of Centre (W ) is proportional to the
500
number of classes. When there are millions of identities
400
in the training data, the proposed ArcFace confronts with
300
substantial training difficulties, e.g. excessive GPU mem-
200
ory consumption and massive computational cost, even at a Parallel Acceleration on Feature (x) -- Data Parallel
100
prohibitive level. Parallel Acceleration on Feature (x) and Center (W)
0
In our implementation 3 , we employ a parallel acceler- 0 0.5 1 1.5 2 2.5 3

ation strategy [44] to relieve this problem. We optimise Identity Number in the Training Data 106

our training code to easily and efficiently support million
level identities on a single machine by parallel accelera-
tion on both feature x (it known as the general data parallel
strategy) and centre W (we named it as the centre parallel
strategy). As shown in Figure 10, our parallel acceleration
on both feature x and centre W can significantly decrease
the GPU memory consumption and accelerate the training
speed. Even for one million identities trained on 8*1080ti
(11GB), our implementation (ResNet 50, batch size 8*64,
feature dimension 512 and float point 32) can still run at
800 samples per second. Compared to the approximate ac-
celeration method proposed in [47], our implementation has
no performance drop.
In Figure 11, we illustrate the main calculation steps of
the parallel acceleration by simple matrix partition, which
can be easily grasped and reproduced by beginners [3].
(1) Get feature (x). Face embedding features are aggre-
gated into one feature matrix (batch size 8*64 × feature
dimension 512) from 8 GPU cards. The size of the aggre-
gated feature matrix is only 1MB, and the communication
cost is negligible when we transfer the feature matrix.
(2) Get similarity score matrix (score = xW ). We copy
the feature matrix into each GPU, and concurrently multi-
ply the feature matrix by the centre sub-matrix (feature di-
mension 512 × identity number 1M/8) to get the similarity
3 https://github.com/deepinsight/insightface/tree/master/recognition
(b) Training Speed
Figure 10. Parallel acceleration on both feature x and centre W .
Setting: ResNet 50, batch size 8*64, feature dimension 512, float
point 32, GPU 8*P40 (24GB).
score sub-matrix (batch size 512 × identity number 1M/8)
on each GPU. The similarity score matrix goes forward to
calculate the ArcFace loss and the gradient. Here, we con-
duct a simple matrix partition on the centre matrix and the
similarity score matrix along the identity dimension, and
there is no communication cost on the centre and similarity
score matrix. Both the centre sub-matrix and the similarity
score sub-matrix are only 256MB on each GPU.
(3) Get gradient on centre (dW ). We transpose the fea-
ture matrix on each GPU, and concurrently multiply the
transposed feature matrix by the gradient sub-matrix of the
similarity score.
(4) Get gradient on feature (x). We concurrently multi-
ply the gradient sub-matrix of similarity score by the trans-
posed centre sub-matrix and sum up the outputs from 8
GPU cards to get the gradient on feature x.
Considering the communication cost (MB level), our
implementation of ArcFace can be easily and efficiently
trained on millions of identities by clusters.
 nearest neighbour separation[5] is
2 1 Γ( d2 ) 1
E[θ(Wj )] → n− d−1 Γ(1 + )( √ d−1
)− d−1 ,
d − 1 2 π(d − 1)Γ( 2 )
(7)
where d is the space dimension, n is the identity number,
and θ(Wj ) = min1≤i,j≤n,i6=j arccos(Wi , Wj )∀i, j. In Fig-
ure 12, we give E[θ(Wj )] in the 128-d, 256-d and 512-d
space with the class number ranging from 10K to 100M .
(a) x
The high-dimensional space is so large that E[θ(Wj )] de-
creases slowly when the class number increases exponen-
tially.
90

85

Minimum Angles Between Individuals

80

75

70
(b) score = xW
65

60

128-d
55 256-d
512-d
50
4 5 6 7 8
10 10 10 10 10
Random Individual Numbers

Figure 12. The high-dimensional space is so large that the mean

of the nearest angles decreases slowly when the class number in-
(c) dW = xT dscore creases exponentially.

References
(d) dx = dscoreW T
Figure 11. Parallel calculation by simple matrix partition. Setting:
ResNet 50, batch size 8*64, feature dimension 512, float point
32, identity number 1 Million, GPU 8 * 1080ti (11GB). Com-
munication cost: 1MB (feature x). Training speed: 800 sam-
ples/second.
5.2. Feature Space Analysis
Is the 512-d hypersphere space large enough to hold
large-scale identities? Theoretically, Yes.
We assume that the identity centre Wj ’s follow a realis-
tically spherical uniform distribution, the expectation of the
[1] http://data.mxnet.io/models/. 8
[2] http://trillionpairs.deepglint.com/overview. 2, 4, 5, 8
[3] Stanford cs class cs231n: Convolutional neural networks
for visual recognition. http://cs231n.github.io/
neural-networks-case-study/. 9
[4] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,
C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al.
Tensorflow: Large-scale machine learning on heterogeneous
distributed systems. arXiv:1603.04467, 2016. 2
[5] J. S. Brauchart, A. B. Reznikov, E. B. Saff, I. H. Sloan,
Y. G. Wang, and R. S. Womersley. Random point sets on
the spherehole radii, covering, and separation. Experimental
Mathematics, 2018. 10
[6] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman.
Vggface2: A dataset for recognising faces across pose and
age. In FG, 2018. 1, 2, 4, 5, 7, 8
[7] B. Chen, W. Deng, and J. Du. Noisy softmax: improving
the generalization ability of dcnn via postponing the early
softmax saturation. In CVPR, 2017. 2
[8] T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao,
B. Xu, C. Zhang, and Z. Zhang. Mxnet: A flexible and effi-
cient machine learning library for heterogeneous distributed
systems. arXiv:1512.01274, 2015. 2, 5
[9] J. Deng, Y. Zhou, and S. Zafeiriou. Marginal loss for deep
face recognition. In CVPR Workshop, 2017. 2, 6
[10] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb-1m:
A dataset and benchmark for large-scale face recognition. In
ECCV, 2016. 4
[11] D. Han, J. Kim, and J. Kim. Deep pyramidal residual net-
works. arXiv:1610.02915, 2016. 5
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR, 2016. 5
[13] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller.
Labeled faces in the wild: A database for studying face
recognition in unconstrained environments. Technical report,
2007. 5, 6, 8
[14] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. In
ICML, 2015. 5
[15] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and
E. Brossard. The megaface benchmark: 1 million faces for
recognition at scale. In CVPR, 2016. 2, 5, 7
[16] J. Liu, Y. Deng, T. Bai, Z. Wei, and C. Huang. Targeting
ultimate accuracy: Face recognition via deep embedding.
arXiv:1506.07310, 2015. 6
[17] W. Liu, R. Lin, Z. Liu, L. Liu, Z. Yu, B. Dai, and L. Song.
Learning towards minimum hyperspherical energy. In NIPS,
2018. 4, 6, 7
[18] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song.
Sphereface: Deep hypersphere embedding for face recogni-
tion. In CVPR, 2017. 1, 2, 3, 4, 5, 6, 7
[19] W. Liu, Y. Wen, Z. Yu, and M. Yang. Large-margin softmax
loss for convolutional neural networks. In ICML, 2016. 2, 3
[20] Y. Liu, P. Shi, B. Peng, H. Yan, Y. Zhou, B. Han, Y. Zheng,
C. Lin, J. Jiang, and Y. Fan. iqiyi-vid: A large dataset for
multi-modal person identification. arXiv:1811.07548, 2018.
5, 8
[21] B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller,
C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, and J. Ch-
eney. Iarpa janus benchmark–c: Face dataset and protocol.
In ICB, 2018. 2, 5
[22] S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kot-
sia, and S. Zafeiriou. Agedb: The first manually collected
in-the-wild age database. In CVPR Workshop, 2017. 2, 5
[23] H.-W. Ng and S. Winkler. A data-driven approach to clean-
ing large face datasets. In ICIP, 2014. 7
[24] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face
recognition. In BMVC, 2015. 1, 2, 6
[25] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De-
Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Au-
tomatic differentiation in pytorch. In NIPS Workshop, 2017.
2 [46] X. Zhang, Z. Fang, Y. Wen, Z. Li, and Y. Qiao. Range loss
[26] G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hin-
ton. Regularizing neural networks by penalizing confident
output distributions. arXiv:1701.06548, 2017. 2, 3
[27] X. Qi and L. Zhang. Face recognition via centralized coor-
dinate learning. arXiv:1801.05678, 2018. 2
[28] R. Ranjan, C. D. Castillo, and R. Chellappa. L2-
constrained softmax loss for discriminative face verification.
arXiv:1703.09507, 2017. 2
[29] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A uni-
fied embedding for face recognition and clustering. In CVPR,
2015. 1, 4, 6, 7
[30] S. Sengupta, J.-C. Chen, C. Castillo, V. M. Patel, R. Chel-
lappa, and D. W. Jacobs. Frontal to profile face verification
in the wild. In WACV, 2016. 2, 5
[31] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov. Dropout: a simple way to prevent neural
networks from overfitting. JML, 2014. 5
[32] Y. Sun, Y. Chen, X. Wang, and X. Tang. Deep learning face
representation by joint identification-verification. In NIPS,
2014. 1, 6, 7
[33] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:
Closing the gap to human-level performance in face verifica-
tion. In CVPR, 2014. 1, 6
[34] W. Wan, Y. Zhong, T. Li, and J. Chen. Rethinking fea-
ture distribution for loss functions in image classification.
arXiv:1803.02988, 2018. 2
[35] F. Wang, W. Liu, H. Liu, and J. Cheng. Additive margin
softmax for face verification. IEEE Signal Processing Let-
ters, 2018. 2, 3, 7
[36] F. Wang, X. Xiang, J. Cheng, and A. L. Yuille. Norm-
face: l 2 hypersphere embedding for face verification.
arXiv:1704.06369, 2017. 2
[37] H. Wang, Y. Wang, Z. Zhou, X. Ji, Z. Li, D. Gong, J. Zhou,
and W. Liu. Cosface: Large margin cosine loss for deep face
recognition. In CVPR, 2018. 1, 2, 3, 5, 6, 7
[38] Y. Wen, K. Zhang, Z. Li, and Y. Qiao. A discriminative fea-
ture learning approach for deep face recognition. In ECCV,
2016. 2, 6, 7
[39] C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. C.
Adams, T. Miller, N. D. Kalka, A. K. Jain, J. A. Duncan, and
K. Allen. Iarpa janus benchmark-b face dataset. In CVPR
Workshop, 2017. 2, 5, 7, 8
[40] L. Wolf, T. Hassner, and I. Maoz. Face recognition in un-
constrained videos with matched background similarity. In
CVPR, 2011. 5, 6
[41] W. Xie, S. Li, and A. Zisserman. Comparator networks. In
ECCV, 2018. 8
[42] W. Xie and A. Zisserman. Multicolumn networks for face
recognition. In BMVC, 2018. 8
[43] D. Yi, Z. Lei, S. Liao, and S. Z. Li. Learning face represen-
tation from scratch. arXiv:1411.7923, 2014. 4, 5
[44] D. Zhang. A distributed training solution for face recogni-
tion. DeepGlint, 2018. 9
[45] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. Joint face detec-
tion and alignment using multitask cascaded convolutional
networks. IEEE Signal Processing Letters, 2016. 1
for deep face recognition with long-tail. In ICCV, 2017. 2, 6
[47] X. Zhang, L. Yang, J. Yan, and D. Lin. Accelerated train-
ing for massive classification via dynamic class selection. In
AAAI, 2018. 9
[48] T. Zheng and W. Deng. Cross-pose lfw: A database for
studying cross-pose face recognition in unconstrained envi-
ronments. Technical Report, 2018. 2, 5, 6
[49] T. Zheng, W. Deng, and J. Hu. Cross-age lfw: A database
for studying cross-age face recognition in unconstrained en-
vironments. arXiv:1708.08197, 2017. 2, 5, 6
 Fast Online Object Tracking and Segmentation: A Unifying Approach

Qiang Wang∗ Li Zhang∗ Luca Bertinetto∗

CASIA University of Oxford FiveAI
qiang.wang@nlpr.ia.ac.cn lz@robots.ox.ac.uk luca@robots.ox.ac.uk

Weiming Hu Philip H.S. Torr

CASIA University of Oxford
arXiv:1812.05050v2 [cs.CV] 5 May 2019

wmhu@nlpr.ia.ac.cn philip.torr@eng.ox.ac.uk

Abstract

In this paper we illustrate how to perform both visual object

tracking and semi-supervised video object segmentation,
in real-time, with a single simple approach. Our method,
dubbed SiamMask, improves the offline training procedure
of popular fully-convolutional Siamese approaches for ob-
ject tracking by augmenting their loss with a binary seg-
mentation task. Once trained, SiamMask solely relies on a
single bounding box initialisation and operates online, pro-
ducing class-agnostic object segmentation masks and ro-
tated bounding boxes at 55 frames per second. Despite its
simplicity, versatility and fast speed, our strategy allows us Init Estimates
to establish a new state of the art among real-time track- Figure 1. Our method aims at the intersection between the tasks
ers on VOT-2018, while at the same time demonstrating of visual tracking and video object segmentation to achieve high
competitive performance and the best speed for the semi- practical convenience. Like conventional object trackers, it relies
supervised video object segmentation task on DAVIS-2016 on a simple bounding box initialisation (blue) and operates online.
Differently from state-of-the-art trackers such as ECO [12] (red),
and DAVIS-2017. The project website is http://www.
SiamMask (green) is able to produce binary segmentation masks,
robots.ox.ac.uk/˜qwang/SiamMask.
which can more accurately describe the target object.

reason about the current position of the object [26]. This

1. Introduction
Tracking is a fundamental task in any video applica-
tion requiring some degree of reasoning about objects of
interest, as it allows to establish object correspondences be-
tween frames [34]. It finds use in a wide range of scenarios
such as automatic surveillance, vehicle navigation, video la-
belling, human-computer interaction and activity recogni-
tion. Given the location of an arbitrary target of interest in
the first frame of a video, the aim of visual object tracking
is to estimate its position in all the subsequent frames with
the best possible accuracy [48].
For many applications, it is important that tracking can
be performed online, while the video is streaming. In other
words, the tracker should not make use of future frames to
∗ Equal contribution.
is the scenario portrayed by visual object tracking bench-
marks, which represent the target object with a simple axis-
aligned (e.g. [56, 52]) or rotated [26, 27] bounding box.
Such a simple annotation helps to keep the cost of data la-
belling low; what is more, it allows a user to perform a quick
and simple initialisation of the target.
Similar to object tracking, the task of semi-supervised
video object segmentation (VOS) requires estimating the
position of an arbitrary target specified in the first frame
of a video. However, in this case the object represen-
tation consists of a binary segmentation mask which ex-
presses whether or not a pixel belongs to the target [40].
Such a detailed representation is more desirable for appli-
cations that require pixel-level information, like video edit-
ing [38] and rotoscoping [37]. Understandably, produc-
ing pixel-level estimates requires more computational re-

1
sources than a simple bounding box. As a consequence,
VOS methods have been traditionally slow, often requir-
ing several seconds per frame (e.g. [55, 50, 39, 1]). Very
recently, there has been a surge of interest in faster ap-
proaches [59, 36, 57, 8, 7, 22, 21]. However, even the fastest
still cannot operate in real-time.
In this paper, we aim at narrowing the gap between ar-
bitrary object tracking and VOS by proposing SiamMask,
a simple multi-task learning approach that can be used
to address both problems. Our method is motivated by
the success of fast tracking approaches based on fully-
convolutional Siamese networks [3] trained offline on mil-
lions of pairs of video frames (e.g. [28, 63, 15, 60]) and by
the very recent availability of YouTube-VOS [58], a large
video dataset with pixel-wise annotations. We aim at retain-
ing the offline trainability and online speed of these meth-
ods while at the same time significantly refining their rep-
resentation of the target object, which is limited to a simple
axis-aligned bounding box.
To achieve this goal, we simultaneously train a Siamese
network on three tasks, each corresponding to a different
strategy to establish correspondances between the target ob-
ject and candidate regions in the new frames. As in the
fully-convolutional approach of Bertinetto et al. [3], one
task is to learn a measure of similarity between the target
object and multiple candidates in a sliding window fashion.
The output is a dense response map which only indicates the
location of the object, without providing any information
about its spatial extent. To refine this information, we si-
multaneously learn two further tasks: bounding box regres-
sion using a Region Proposal Network [46, 28] and class-
agnostic binary segmentation [43]. Notably, binary labels
are only required during offline training to compute the seg-
mentation loss and not online during segmentation/tracking.
In our proposed architecture, each task is represented by
a different branch departing from a shared CNN and con-
tributes towards a final loss, which sums the three outputs
together.
Once trained, SiamMask solely relies on a single bound-
ing box initialisation, operates online without updates and
produces object segmentation masks and rotated bound-
ing boxes at 55 frames per second. Despite its simplicity
and fast speed, SiamMask establishes a new state-of-the-art
on VOT-2018 for the problem of real-time object tracking.
Moreover, the same method is also very competitive against
recent semi-supervised VOS approaches on DAVIS-2016
and DAVIS-2017, while being the fastest by a large mar-
gin. This result is achieved with a simple bounding box
initialisation (as opposed to a mask) and without adopting
costly techniques often used by VOS approaches such as
fine-tuning [35, 39, 1, 53], data augmentation [23, 30] and
optical flow [50, 1, 39, 30, 8].
The rest of this paper is organised as follows. Section 2
2
briefly outlines some of the most relevant prior work in vi-
sual object tracking and semi-supervised VOS; Section 3
describes our proposal; Section 4 evaluates it on four bench-
marks and illustrates several ablative studies; Section 5 con-
cludes the paper.
2. Related Work
In this section, we briefly cover the most representative
techniques for the two problems tackled in this paper.
Visual object tracking. Arguably, until very recently,
the most popular paradigm for tracking arbitrary objects
has been to train online a discriminative classifier exclu-
sively from the ground-truth information provided in the
first frame of a video (and then update it online).
In the past few years, the Correlation Filter, a sim-
ple algorithm that allows to discriminate between the tem-
plate of an arbitrary target and its 2D translations, rose
to prominence as particularly fast and effective strategy
for tracking-by-detection thanks to the pioneering work of
Bolme et al. [4]. Performance of Correlation Filter-based
trackers has then been notably improved with the adop-
tion of multi-channel formulations [24, 20], spatial con-
straints [25, 13, 33, 29] and deep features (e.g. [12, 51]).
Recently, a radically different approach has been intro-
duced [3, 19, 49]. Instead of learning a discrimative clas-
sifier online, these methods train offline a similarity func-
tion on pairs of video frames. At test time, this function
can be simply evaluated on a new video, once per frame.
In particular, evolutions of the fully-convolutional Siamese
approach [3] considerably improved tracking performance
by making use of region proposals [28], hard negative min-
ing [63], ensembling [15] and memory networks [60].
Most modern trackers, including all the ones mentioned
above, use a rectangular bounding box both to initialise the
target and to estimate its position in the subsequent frames.
Despite its convenience, a simple rectangle often fails to
properly represent an object, as it is evident in the examples
of Figure 1. This motivated us to propose a tracker able to
produce binary segmentation masks while still only relying
on a bounding box initialisation.
Interestingly, in the past it was not uncommon for track-
ers to produce a coarse binary mask of the target object
(e.g. [11, 42]). However, to the best of our knowledge, the
only recent tracker that, like ours, is able to operate on-
line and produce a binary mask starting from a bounding
box initialisation is the superpixel-based approach of Yeo et
al. [61]. However, at 4 frames per seconds (fps), its fastest
variant is significantly slower than our proposal. Further-
more, when using CNN features, its speed is affected by a
60-fold decrease, plummeting below 0.1 fps. Finally, it has
not demonstrated to be competitive on modern tracking or
VOS benchmarks. Similar to us, the methods of Perazzi et
al. [39] and Ci et al. [10] can also start from a rectangle and
output per-frame masks. However, they require fine-tuning
at test time, which makes them slow.
Semi-supervised video object segmentation. Bench-
marks for arbitrary object tracking (e.g. [48, 26, 56]) as-
sume that trackers receive input frames in a sequential fash-
ion. This aspect is generally referred to with the attributes
online or causal [26]. Moreover, methods are often focused
on achieving a speed that exceeds the ones of typical video
framerates [27]. Conversely, semi-supervised VOS algo-
rithms have been traditionally more concerned with an ac-
curate representation of the object of interest [38, 40].
In order to exploit consistency between video frames,
several methods propagate the supervisory segmentation
mask of the first frame to the temporally adjacent ones via
graph labeling approaches (e.g. [55, 41, 50, 36, 1]). In
particular, Bao et al. [1] recently proposed a very accurate
method that makes use of a spatio-temporal MRF in which
temporal dependencies are modelled by optical flow, while
spatial dependencies are expressed by a CNN.
Another popular strategy is to process video frames in-
dependently (e.g. [35, 39, 53]), similarly to what happens
in most tracking approaches. For example, in OSVOS-S
Maninis et al. [35] do not make use of any temporal in-
formation. They rely on a fully-convolutional network pre-
trained for classification and then, at test time, they fine-
tune it using the ground-truth mask provided in the first
frame. MaskTrack [39] instead is trained from scratch on
individual images, but it does exploit some form of tempo-
rality at test time by using the latest mask prediction and
optical flow as additional input to the network.
Aiming towards the highest possible accuracy, at test
time VOS methods often feature computationally intensive
techniques such as fine-tuning [35, 39, 1, 53], data augmen-
tation [23, 30] and optical flow [50, 1, 39, 30, 8]. Therefore,
these approaches are generally characterised by low fram-
erates and the inability to operate online. For example, it
is not uncommon for methods to require minutes [39, 9] or
even hours [50, 1] for videos that are just a few seconds
long, like the ones of DAVIS.
Recently, there has been an increasing interest in the
VOS community towards faster methods [36, 57, 8, 7, 22,
21]. To the best of our knowledge, the fastest approaches
with a performance competitive with the state of the art
are the ones of Yang et al. [59] and Wug et al. [57]. The
former uses a meta-network “modulator” to quickly adapt
the parameters of a segmentation network during test time,
while the latter does not use any fine-tuning and adopts an
encoder-decoder Siamese architecture trained in multiple
stages. Both these methods run below 10 frames per sec-
ond, while we are more than six times faster and only rely
on a bounding box initialisation.
3
3. Methodology
To allow online operability and fast speed, we adopt
the fully-convolutional Siamese framework [3]. Moreover,
to illustrate that our approach is agnostic to the specific
fully-convolutional method used as a starting point (e.g. [3,
28, 63, 60, 16]), we consider the popular SiamFC [3] and
SiamRPN [28] as two representative examples. We first in-
troduce them in Section 3.1 and then describe our approach
in Section 3.2.
3.1. Fully-convolutional Siamese networks
SiamFC. Bertinetto et al. [3] propose to use, as a fun-
damental building block of a tracking system, an offline-
trained fully-convolutional Siamese network that compares
an exemplar image z against a (larger) search image x to
obtain a dense response map. z and x are, respectively, a
w×h crop centered on the target object and a larger crop
centered on the last estimated position of the target. The
two inputs are processed by the same CNN fθ , yielding two
feature maps that are cross-correlated:
gθ (z, x) = fθ (z) ? fθ (x). (1)
In this paper, we refer to each spatial element of the re-
sponse map (left-hand side of Eq. 1) as response of a can-
didate window (RoW). For example, gθn (z, x), encodes a
similarity between the examplar z and n-th candidate win-
dow in x. For SiamFC, the goal is for the maximum value of
the response map to correspond to the target location in the
search area x. Instead, in order to allow each RoW to en-
code richer information about the target object, we replace
the simple cross-correlation of Eq. 1 with depth-wise cross-
correlation [2] and produce a multi-channel response map.
SiamFC is trained offline on millions of video frames with
the logistic loss [3, Section 2.2], which we refer to as Lsim .
SiamRPN. Li et al. [28] considerably improve the perfor-
mance of SiamFC by relying on a region proposal network
(RPN) [46, 14], which allows to estimate the target location
with a bounding box of variable aspect ratio. In particular,
in SiamRPN each RoW encodes a set of k anchor box pro-
posals and corresponding object/background scores. There-
fore, SiamRPN outputs box predictions in parallel with
classification scores. The two output branches are trained
using the smooth L1 and the cross-entropy losses [28, Sec-
tion 3.2]. In the following, we refer to them as Lbox and
Lscore respectively.
3.2. SiamMask
Unlike existing tracking methods that rely on low-
fidelity object representations, we argue the importance of
producing per-frame binary segmentation masks. To this
aim we show that, besides similarity scores and bound-
ing box coordinates, it is possible for the RoW of a fully-

𝑓" 15*15*256
127*127*3
17*17*256
⋆'
RoW: 1*1*256
𝑓" 31*31*256
255*255*3
(a) three-branch variant architecture
Figure 2. Schematic illustration of SiamMask variants: (a) three-branch architecture (full), (b) two-branch architecture (head). ?d denotes
depth-wise cross correlation. For simplicity, upsampling layer and mask refinement module are omitted here and detailed in Appendix A.
convolutional Siamese network to also encode the informa-
tion necessary to produce a pixel-wise binary mask. This
can be achieved by extending existing Siamese trackers with
an extra branch and loss.
We predict w×h binary masks (one for each RoW) using
a simple two-layers neural network hφ with learnable pa-
rameters φ. Let mn denote the predicted mask correspond-
ing to the n-th RoW,
mn = hφ (gθn (z, x)). (2)
From Eq. 2 we can see that the mask prediction is a function
of both the image to segment x and the target object in z.
In this way, z can be used as a reference to guide the seg-
mentation process: given a different reference image, the
network will produce a different segmentation mask for x.
Loss function. During training, each RoW is labelled with
a ground-truth binary label yn ∈ {±1} and also associated
with a pixel-wise ground-truth mask cn of size w×h. Let
cij
n ∈ {±1} denote the label corresponding to pixel (i, j) of
the object mask in the n-th candidate RoW. The loss func-
tion Lmask (Eq. 3) for the mask prediction task is a binary
logistic regression loss over all RoWs:
X 1 + yn X ij ij
Lmask (θ, φ) = ( log(1 + e−cn mn )). (3)
n
2wh ij
Thus, the classification layer of hφ consists of w×h classi-
fiers, each indicating whether a given pixel belongs to the
object in the candidate window or not. Note that Lmask is
considered only for positive RoWs (i.e. with yn = 1).
Mask representation. In contrast to semantic segmen-
tation methods in the style of FCN [32] and Mask R-
CNN [17], which maintain explicit spatial information
throughout the network, our approach follows the spirit
of [43, 44] and generates masks starting from a flat-
tened representation of the object. In particular, in our
17*17*(63*63)
ℎ %
17*17*(63*63)
127*127*1 ℎ % mask
1*1*(63*63) mask
1*1*(63*63)
17*17*256
𝑏) 17*17*4k box
RoW: 1*1*256 𝑝- score
17*17*1
𝑠+ 17*17*2k score
(b) two-branch variant head
case this representation corresponds to one of the (17×17)
RoWs produced by the depth-wise cross-correlation be-
tween fθ (z) and fθ (x). Importantly, the network hφ of
the segmentation task is composed of two 1×1 convolu-
tional layers, one with 256 and the other with 632 chan-
nels (Figure 2). This allows every pixel classifier to utilise
information contained in the entire RoW and thus to have
a complete view of its corresponding candidate window in
x, which is critical to disambiguate between instances that
look like the target (e.g. last row of Figure 4), often referred
to as distractors. With the aim of producing a more accurate
object mask, we follow the strategy of [44], which merges
low and high resolution features using multiple refinement
modules made of upsampling layers and skip connections
(see Appendix A).
Two variants. For our experiments, we augment the ar-
chitectures of SiamFC [3] and SiamRPN [28] with our seg-
mentation branch and the loss Lmask , obtaining what we
call the two-branch and three-branch variants of SiamMask.
These respectively optimise the multi-task losses L2B and
L3B , defined as:
L2B = λ1 · Lmask + λ2 · Lsim , (4)
L3B = λ1 · Lmask + λ2 · Lscore + λ3 · Lbox . (5)
We refer the reader to [3, Section 2.2] for Lsim and to [28,
Section 3.2] for Lbox and Lscore . For L3B , a RoW is con-
sidered positive (yn = 1) if one of its anchor boxes has
IOU with the ground-truth box of at least 0.6 and negative
(yn = −1) otherwise. For L2B , we adopt the same strat-
egy of [3] to define positive and negative samples. We did
not search over the hyperparameters of Eq. 4 and Eq. 5 and
simply set λ1 = 32 like in [43] and λ2 = λ3 = 1. The task-
specific branches for the box and score outputs are consti-
tuted by two 1×1 convolutional layers. Figure 2 illustrates
the two variants of SiamMask.
Box generation. Note that, while VOS benchmarks re-
quire binary masks, typical tracking benchmarks such as
4

Figure 3. In order to generate a bounding box from a binary mask
(in yellow), we experiment with three different methods. Min-
max: the axis-aligned rectangle containing the object (red); MBR:
the minimum bounding rectangle (green); Opt: the rectangle ob-
tained via the optimisation strategy proposed in VOT-2016 [26]
(blue).
VOT [26, 27] require a bounding box as final representation
of the target object. We consider three different strategies
to generate a bounding box from a binary mask (Figure 3):
(1) axis-aligned bounding rectangle (Min-max), (2) rotated
minimum bounding rectangle (MBR) and (3) the optimisa-
tion strategy used for the automatic bounding box gener-
ation proposed in VOT-2016 [26] (Opt). We empirically
evaluate these alternatives in Section 4 (Table 1).
3.3. Implementation details
Network architecture. For both our variants, we use a
ResNet-50 [18] until the final convolutional layer of the 4-th
stage as our backbone fθ . In order to obtain a high spatial
resolution in deeper layers, we reduce the output stride to 8
by using convolutions with stride 1. Moreover, we increase
the receptive field by using dilated convolutions [6]. In our
model, we add to the shared backbone fθ an unshared adjust
layer (1×1 conv with 256 outputs). For simplicity, we omit
it in Eq. 1. We describe the network architectures in more
detail in Appendix A.
Training. Like SiamFC [3], we use examplar and search
image patches of 127×127 and 255×255 pixels respec-
tively. During training, we randomly jitter examplar and
search patches. Specifically, we consider random transla-
tions (up to ±8 pixels) and rescaling (of 2±1/8 and 2±1/4
for examplar and search respectively).
The network backbone is pre-trained on the
ImageNet-1k classification task. We use SGD with a
first warmup phase in which the learning rate increases
linearly from 10−3 to 5×10−3 for the first 5 epochs
and then descreases logarithmically until 5×10−4 for 15
more epochs. We train all our models using COCO [31],
ImageNet-VID [47] and YouTube-VOS [58].
Inference. During tracking, SiamMask is simply evalu-
ated once per frame, without any adaptation. In both our
variants, we select the output mask using the location attain-
ing the maximum score in the classification branch. Then,
after having applied a per-pixel sigmoid, we binarise the
output of the mask branch at the threshold of 0.5. In the
two-branch variant, for each video frame after the first one,
we fit the output mask with the Min-max box and use it as
5
reference to crop the next frame search region. Instead, in
the three-branch variant, we find more effective to exploit
the highest-scoring output of the box branch as reference.
4. Experiments
In this section, we evaluate our approach on two related
tasks: visual object tracking (on VOT-2016 and VOT-2018)
and semi-supervised video object segmentation (on DAVIS-
2016 and DAVIS-2017). We refer to our two-branch and
three-branch variants with SiamMask-2B and SiamMask
respectively.
4.1. Evaluation for visual object tracking
Datasets and settings. We adopt two widely used bench-
marks for the evaluation of the object tracking task: VOT-
2016 [26] and VOT-2018 [27], both annotated with rotated
bounding boxes. We use VOT-2016 to understand how dif-
ferent types of representation affect the performance. For
this first experiment, we use mean intersection over union
(IOU) and Average Precision (AP)@{0.5, 0.7} IOU. We
then compare against the state-of-the-art on VOT-2018, us-
ing the official VOT toolkit and the Expected Average Over-
lap (EAO), a measure that considers both accuracy and ro-
bustness of a tracker [27].
How much does the object representation matter?
Existing tracking methods typically predict axis-aligned
bounding boxes with a fixed [3, 20, 13, 33] or variable [28,
19, 63] aspect ratio. We are interested in understanding to
which extent producing a per-frame binary mask can im-
prove tracking. In order to focus on representation accuracy,
for this experiment only we ignore the temporal aspect and
sample video frames at random. The approaches described
in the following paragraph are tested on randomly cropped
search patches (with random shifts within ±16 pixels and
scale deformations up to 21±0.25 ) from the sequences of
VOT-2016.
In Table 1, we compare our three-branch variant using
the Min-max, MBR and Opt approaches (described at the
end of Section 3.2 and in Figure 3). For reference, we also
report results for SiamFC and SiamRPN as representative
of the fixed and variable aspect-ratio approaches, together
with three oracles that have access to per-frame ground-
truth information and serve as upper bounds for the dif-
ferent representation strategies. (1) The fixed aspect-ratio
oracle uses the per-frame ground-truth area and center loca-
tion, but fixes the aspect reatio to the one of the first frame
and produces an axis-aligned bounding box. (2) The Min-
max oracle uses the minimal enclosing rectangle of the ro-
tated ground-truth bounding box to produce an axis-aligned
bounding box. (3) Finally, the MBR oracle uses the rotated
minimum bounding rectangle of the ground-truth. Note that
(1), (2) and (3) can be considered, respectively, the per-
 mIOU (%) mAP@0.5 IOU mAP@0.7 IOU
Fixed a.r. Oracle 73.43 90.15 62.52
Min-max Oracle 77.70 88.84 65.16
MBR Oracle 84.07 97.77 80.68
SiamFC [3] 50.48 56.42 9.28
SiamRPN [63] 60.02 76.20 32.47
SiamMask-Min-max 65.05 82.99 43.09
SiamMask-MBR 67.15 85.42 50.86
SiamMask-Opt 71.68 90.77 60.47
Table 1. Performance for different bounding box representation
strategies on VOT-2016.
formance upper bounds for the representation strategies of
SiamFC, SiamRPN and SiamMask.
Table 1 shows that our method achieves the best mIOU,
no matter the box generation strategy used (Figure 3). Al-
beit SiamMask-Opt offers the highest IOU and mAP, it re-
quires significant computational resources due to its slow
optimisation procedure [54]. SiamMask-MBR achieves a
mAP@0.5 IOU of 85.4, with a respective improvement of
+29 and +9.2 points w.r.t. the two fully-convolutional
baselines. Interestingly, the gap significantly widens when
considering mAP at the higher accuracy regime of 0.7 IOU:
+41.6 and +18.4 respectively. Notably, our accuracy re-
sults are not far from the fixed aspect-ratio oracle. More-
over, comparing the upper bound performance represented
by the oracles, it is possible to notice how, by simply chang-
ing the bounding box representation, there is a great room
for improvement (e.g. +10.6% mIOU improvement be-
tween the fixed aspect-ratio and the MBR oracles).
Overall, this study shows how the MBR strategy to obtain
a rotated bounding box from a binary mask of the object
offers a significant advantage over popular strategies that
simply report axis-aligned bounding boxes.
Results on VOT-2018 and VOT-2016. In Table 2 we
compare the two variants of SiamMask with MBR strategy
and SiamMask–Opt against five recently published state-
of-the-art trackers on the VOT-2018 benchmark. Unless
stated otherwise, SiamMask refers to our three-branch vari-
ant with MBR strategy. Both variants achieve outstanding
performance and run in real-time. In particular, our three-
branch variant significantly outperforms the very recent
and top performing DaSiamRPN [63], achieving a EAO of
0.380 while running at 55 frames per second. Even with-
out box regression branch, our simpler two-branch vari-
ant (SiamMask-2B) achieves a high EAO of 0.334, which
is in par with SA Siam R [15] and superior to any other
real-time method in the published literature. Finally, in
SiamMask–Opt, the strategy proposed in [54] to find the op-
timal rotated rectangle from a binary mask brings the best
overall performance (and a particularly high accuracy), but
comes at a significant computational cost.
Our model is particularly strong under the accuracy met-
6
ric, showing a significant advantage with respect to the Cor-
relation Filter-based trackers CSRDCF [33], STRCF [29].
This is not surprising, as SiamMask relies on a richer object
representation, as outlined in Table 1. Interestingly, sim-
ilarly to us, He et al. (SA Siam R) [15] are motivated to
achieve a more accurate target representation by consider-
ing multiple rotated and rescaled bounding boxes. However,
their representation is still constrained to a fixed aspect-ratio
box.
Table 3 gives further results of SiamMask with dif-
ferent box generation strategies on VOT-2018 and -2016.
SiamMask-box means the box branch of SiamMask is
adopted for inference despite the mask branch has been
trained. We can observe clear improvements on all evalua-
tion metrics by using the mask branch for box generation.
4.2. Evaluation for semi-supervised VOS
Our model, once trained, can also be used for the task
of VOS to achieve competitive performance without requir-
ing any adaptation at test time. Importantly, differently to
typical VOS approaches, ours can operate online, runs in
real-time and only requires a simple bounding box initiali-
sation.
Datasets and settings. We report the performance of
SiamMask on DAVIS-2016 [40], DAVIS-2017 [45] and
YouTube-VOS [58] benchmarks. For both DAVIS datasets,
we use the official performance measures: the Jaccard index
(J ) to express region similarity and the F-measure (F) to
express contour accuracy. For each measure C ∈ {J , F},
three statistics are considered: mean CM , recall CO , and
decay CD , which informs us about the gain/loss of per-
formance over time [40]. Following Xu et al. [58], for
YouTube-VOS we report the mean Jaccard index and F-
measure for both seen (JS , FS ) and unseen categories (JU ,
FU ). O is the average of these four measures.
To initialise SiamMask, we extract the axis-aligned
bounding box from the mask provided in the first frame
(Min-max strategy, see Figure 3). Similarly to most VOS
methods, in case of multiple objects in the same video
(DAVIS-2017) we simply perform multiple inferences.
Results on DAVIS and YouTube-VOS. In the semi-
supervised setting, VOS methods are initialised with a
binary mask [38] and many of them require computa-
tionally intensive techniques at test time such as fine-
tuning [35, 39, 1, 53], data augmentation [23, 30], infer-
ence on MRF/CRF [55, 50, 36, 1] and optical flow [50, 1,
39, 30, 8]. As a consequence, it is not uncommon for VOS
techniques to require several minutes to process a short se-
quence. Clearly, these strategies make the online applicabil-
ity (which is our focus) impossible. For this reason, in our
comparison we mainly concentrate on fast state-of-the-art
approaches.
 SiamMask-Opt SiamMask SiamMask-2B DaSiamRPN [63] SiamRPN [28] SA Siam R [15] CSRDCF [33] STRCF [29]
EAO ↑ 0.387 0.380 0.334 0.326 0.244 0.337 0.263 0.345
Accuracy ↑ 0.642 0.609 0.575 0.569 0.490 0.566 0.466 0.523
Robustness ↓ 0.295 0.276 0.304 0.337 0.460 0.258 0.318 0.215
Speed (fps) ↑ 5 55 60 160 200 32.4 48.9 2.9
Table 2. Comparison with the state-of-the-art under the EAO, Accuracy, and Robustness metrics on VOT-2018.

VOT-2018 VOT-2016 AN RN EAO ↑ JM↑ FM↑ Speed

EAO ↑ A ↑ R↓ EAO ↑ A ↑ R↓ Speed SiamFC 4 0.188 - - 86
SiamMask-box 0.363 0.584 0.300 0.412 0.623 0.233 76 SiamFC 4 0.251 - - 40
SiamMask 0.380 0.609 0.276 0.433 0.639 0.214 55 SiamRPN 4 0.243 - - 200
SiamMask-Opt 0.387 0.642 0.295 0.442 0.670 0.233 5 SiamRPN 4 0.359 - - 76
SiamMask-2B w/o R 4 0.326 62.3 55.6 43
Table 3. Results on VOT-2016 and VOT-2018. SiamMask w/o R 4 0.375 68.6 57.8 58
SiamMask-2B-score 4 0.265 - - 40
FT M JM↑ JO↑ JD↓ FM↑ FO↑ FD↓ Speed
SiamMask-box 4 0.363 - - 76
OnAVOS [53] 4 4 86.1 96.1 5.2 84.9 89.7 5.8 0.08 SiamMask-2B 4 0.334 67.4 63.5 60
MSK [39] 4 4 79.7 93.1 8.9 75.4 87.1 9.0 0.1 SiamMask 4 0.380 71.7 67.8 55
MSKb [39] 4 8 69.6 - - - - - 0.1
Table 7. Ablation studies on VOT-2018 and DAVIS-2016.
SFL [9] 4 4 76.1 90.6 12.1 76.0 85.5 10.4 0.1
FAVOS [8] 8 4 82.4 96.5 4.5 79.5 89.4 5.5 0.8
RGMP [57] 8 4 81.5 91.7 10.9 82.0 90.8 10.1 8 SiamMask achieves a very low decay [40] for both region
PML [7] 8 4 75.5 89.6 8.5 79.3 93.4 7.8 3.6 similarity (JD ,) and contour accuracy (FD ). This suggests
OSMN [59] 8 4 74.0 87.6 9.0 72.9 84.0 10.6 8.0 that our method is robust over time and thus it is indicated
PLM [62] 8 4 70.2 86.3 11.2 62.5 73.2 14.7 6.7 for particularly long sequences.
VPN [22] 8 4 70.2 82.3 12.4 65.5 69.0 14.4 1.6 Qualitative results of SiamMask for both VOT and
SiamMask 8 8 71.7 86.8 3.0 67.8 79.8 2.1 55 DAVIS sequences are shown in Figure 4, 9 and 10. Despite
Table 4. Results on DAVIS 2016 (validation set). FT and M respec- the high speed, SiamMask produces accurate segmentation
tively denote if the method requires fine-tuning and whether it is masks even in presence of distractors.
initialised with a mask (4) or a bounding box (8).
4.3. Further analysis
FT M JM↑ JO↑ JD↓ FM↑ FO↑ FD↓ Speed In this section, we illustrate ablation studies, failure
OnAVOS [53] 4 4 61.6 67.4 27.9 69.1 75.4 26.6 0.1 cases and timings of our methods.
OSVOS [5] 4 4 56.6 63.8 26.1 63.9 73.8 27.0 0.1 Network architecture. In Table 7, AN and RN denote
FAVOS [8] 8 4 54.6 61.1 14.1 61.8 72.3 18.0 0.8 whether we use AlexNet or ResNet-50 as the shared back-
OSMN [59] 8 4 52.5 60.9 21.5 57.1 66.1 24.3 8.0 bone fθ (Figure 2), while with “w/o R” we mean that the
SiamMask 8 8 54.3 62.8 19.3 58.5 67.5 20.9 55
method does not use the refinement strategy of Pinheiro et
Table 5. Results on DAVIS 2017 (validation set). al. [44]. From the results of Table 7, it is possible to make
several observations. (1) The first set of rows shows that,
FT M JS↑ JU ↑ FS↑ FU ↑ O↑ Speed by simply updating the architecture of fθ , it is possible to
OnAVOS [53] 4 4 60.1 46.6 62.7 51.4 55.2 0.1
achieve an important performance improvement. However,
OSVOS [5] 4 4 59.8 54.2 60.5 60.7 58.8 0.1
this comes at the cost of speed, especially for SiamRPN. (2)
OSMN [59] 8 4 60.0 40.6 60.1 44.0 51.2 8.0 SiamMask-2B and SiamMask considerably improve over
SiamMask 8 8 60.2 45.1 58.2 47.7 52.8 55 their baselines (with same fθ ) SiamFC and SiamRPN. (3)
Table 6. Results on YouTube-VOS (validation set).
Interestingly, the refinement approach of Pinheiro et al. [44]
is very important for the contour accuracy FM , but less so
for the other metrics.
Table 4, 5 and 6 show how SiamMask can be considered Multi-task training. We conducted two further experi-
as a strong baseline for online VOS. First, it is almost two ments to disentangle the effect of multi-task training. Re-
orders of magnitude faster than accurate approaches such sults are reported in Table 7. To achieve this, we modified
as OnAVOS [53] or SFL [9]. Second, it is competitive with the two variants of SiamMask during inference so that, re-
recent VOS methods that do not employ fine-tuning, while spectively, they report an axis-aligned bounding box from
being four times more efficient than the fastest ones (i.e. the score branch (SiamMask-2B-score) or the box branch
OSMN [59] and RGMP [57]). Interestingly, we note that (SiamMask-box). Therefore, despite having been trained,

7
 Basketball
Nature
Car-Shadow
Dogs-Jump
Pigs
Figure 4. Qualitative results of our method for sequences belonging to both object tracking and video object segmentation benchmarks.
Basketball and Nature are from VOT-2018 [27]; Car-Shadow is from DAVIS-2016 [40]; Dogs-Jump and Pigs are from DAVIS-2017 [45].
Multiple masks are obtained from different inferences (with different initialisations).
Figure 5. Failure cases: motion blur and “non-object” instance.
the mask branch is not used during inference. We can ob-
serve how both variants obtain a modest but meaningful im-
provement with respect to their counterparts (SiamFC and
SiamRPN): from 0.251 to 0.265 EAO for the two-branch
and from 0.359 to 0.363 for the three-branch on VOT2018.
Timing. SiamMask operates online without any adap-
tation to the test sequence. On a single NVIDIA RTX
2080 GPU, we measured an average speed of 55 and 60
frames per second, respectively for the two-branch and
three-branch variants. Note that the highest computational
burden comes from the feature extractor fθ .
Failure cases. Finally, we discuss two scenarios in which
SiamMask fails: motion blur and “non-object” instance
(Figure 5). Despite being different in nature, these two
1
cases arguably arise from the complete lack of similar train-
ing samples in a training sets, which are focused on objects
8
that can be unambiguously discriminated from the fore-
ground.
5. Conclusion
In this paper we introduced SiamMask, a simple ap-
proach that enables fully-convolutional Siamese trackers to
produce class-agnostic binary segmentation masks of the
target object. We show how it can be applied with success
to both tasks of visual object tracking and semi-supervised
video object segmentation, showing better accuracy than
state-of-the-art trackers and, at the same time, the fastest
speed among VOS methods. The two variants of SiamMask
we proposed are initialised with a simple bounding box, op-
erate online, run in real-time and do not require any adapta-
tion to the test sequence. We hope that our work will inspire
further studies that consider the two problems of visual ob-
ject tracking and video object segmentation together.
Acknowledgements. This work was supported by
the ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC
grant Seebibyte EP/M013774/1 and EPSRC/MURI grant
EP/N019474/1. We would also like to acknowledge the sup-
port of the Royal Academy of Engineering and FiveAI Ltd.
Qiang Wang is partly supported by the NSFC (Grant No.
61751212, 61721004 and U1636218).
References
[1] L. Bao, B. Wu, and W. Liu. Cnn in mrf: Video object seg-
mentation via inference in a cnn-based higher-order spatio-
temporal mrf. In IEEE Conference on Computer Vision and
Pattern Recognition, 2018. 2, 3, 6
[2] L. Bertinetto, J. F. Henriques, J. Valmadre, P. H. S. Torr, and
A. Vedaldi. Learning feed-forward one-shot learners. In Ad-
vances in Neural Information Processing Systems, 2016. 3
[3] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and
P. H. Torr. Fully-convolutional siamese networks for object
tracking. In European Conference on Computer Vision work-
shops, 2016. 2, 3, 4, 5, 6
[4] D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui.
Visual object tracking using adaptive correlation filters. In
IEEE Conference on Computer Vision and Pattern Recogni-
tion, 2010. 2
[5] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé,
D. Cremers, and L. Van Gool. One-shot video object seg-
mentation. In IEEE Conference on Computer Vision and
Pattern Recognition, 2017. 7
[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Deeplab: Semantic image segmentation with
deep convolutional nets, atrous convolution, and fully con-
nected crfs. IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence, 2018. 5, 11
[7] Y. Chen, J. Pont-Tuset, A. Montes, and L. Van Gool. Blaz-
ingly fast video object segmentation with pixel-wise metric
learning. In IEEE Conference on Computer Vision and Pat-
tern Recognition, 2018. 2, 3, 7
[8] J. Cheng, Y.-H. Tsai, W.-C. Hung, S. Wang, and M.-H. Yang.
Fast and accurate online video object segmentation via track-
ing parts. In IEEE Conference on Computer Vision and Pat-
tern Recognition, 2018. 2, 3, 6, 7
[9] J. Cheng, Y.-H. Tsai, S. Wang, and M.-H. Yang. Segflow:
Joint learning for video object segmentation and optical
flow. In IEEE International Conference on Computer Vision,
2017. 3, 7
[10] H. Ci, C. Wang, and Y. Wang. Video object segmentation by
learning location-sensitive embeddings. In European Con-
ference on Computer Vision, 2018. 2
[11] D. Comaniciu, V. Ramesh, and P. Meer. Real-time tracking
of non-rigid objects using mean shift. In IEEE Conference
on Computer Vision and Pattern Recognition, 2000. 2
[12] M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg. Eco:
Efficient convolution operators for tracking. In IEEE Con-
ference on Computer Vision and Pattern Recognition, 2017.
1, 2
[13] M. Danelljan, G. Häger, F. S. Khan, and M. Felsberg. Learn-
ing spatially regularized correlation filters for visual track-
ing. In IEEE International Conference on Computer Vision,
2015. 2, 5
[14] C. Feichtenhofer, A. Pinz, and A. Zisserman. Detect to track
and track to detect. In IEEE International Conference on
Computer Vision, 2017. 3
[15] A. He, C. Luo, X. Tian, and W. Zeng. Towards a better match
in siamese network based visual object tracker. In European
Conference on Computer Vision workshops, 2018. 2, 6, 7
9
[16] A. He, C. Luo, X. Tian, and W. Zeng. A twofold siamese
network for real-time object tracking. In IEEE Conference
on Computer Vision and Pattern Recognition, 2018. 3
[17] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-
cnn. In IEEE International Conference on Computer Vision,
2017. 4
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In IEEE Conference on Computer
Vision and Pattern Recognition, 2016. 5, 11
[19] D. Held, S. Thrun, and S. Savarese. Learning to track at 100
fps with deep regression networks. In European Conference
on Computer Vision, 2016. 2, 5
[20] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. High-
speed tracking with kernelized correlation filters. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
2015. 2, 5
[21] Y.-T. Hu, J.-B. Huang, and A. G. Schwing. Videomatch:
Matching based video object segmentation. In European
Conference on Computer Vision, 2018. 2, 3
[22] V. Jampani, R. Gadde, and P. V. Gehler. Video propagation
networks. In IEEE Conference on Computer Vision and Pat-
tern Recognition, 2017. 2, 3, 7
[23] A. Khoreva, R. Benenson, E. Ilg, T. Brox, and B. Schiele.
Lucid data dreaming for object tracking. In IEEE Con-
ference on Computer Vision and Pattern Recognition work-
shops, 2017. 2, 3, 6
[24] H. Kiani Galoogahi, T. Sim, and S. Lucey. Multi-channel
correlation filters. In IEEE International Conference on
Computer Vision, 2013. 2
[25] H. Kiani Galoogahi, T. Sim, and S. Lucey. Correlation filters
with limited boundaries. In IEEE Conference on Computer
Vision and Pattern Recognition, 2015. 2
[26] M. Kristan, A. Leonardis, J. Matas, M. Felsberg,
R. Pflugfelder, L. Čehovin, T. Vojı́r, G. Häger, A. Lukežič,
G. Fernández, et al. The visual object tracking vot2016 chal-
lenge results. In European Conference on Computer Vision,
2016. 1, 3, 5
[27] M. Kristan, A. Leonardis, J. Matas, M. Felsberg,
R. Pfugfelder, L. C. Zajc, T. Vojir, G. Bhat, A. Lukezic,
A. Eldesokey, G. Fernandez, and et al. The sixth visual object
tracking vot-2018 challenge results. In European Conference
on Computer Vision workshops, 2018. 1, 3, 5, 8, 12
[28] B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu. High performance
visual tracking with siamese region proposal network. In
IEEE Conference on Computer Vision and Pattern Recogni-
tion, 2018. 2, 3, 4, 5, 7
[29] F. Li, C. Tian, W. Zuo, L. Zhang, and M.-H. Yang. Learn-
ing spatial-temporal regularized correlation filters for visual
tracking. In IEEE Conference on Computer Vision and Pat-
tern Recognition, 2018. 2, 6, 7
[30] X. Li and C. C. Loy. Video object segmentation with joint
re-identification and attention-aware mask propagation. In
European Conference on Computer Vision, 2018. 2, 3, 6
[31] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-
manan, P. Dollár, and C. L. Zitnick. Microsoft coco: Com-
mon objects in context. In European Conference on Com-
puter Vision, 2014. 5
[32] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In IEEE Conference on
Computer Vision and Pattern Recognition, 2015. 4
[33] A. Lukezic, T. Vojir, L. C. Zajc, J. Matas, and M. Kristan.
Discriminative correlation filter with channel and spatial reli-
ability. In IEEE Conference on Computer Vision and Pattern
Recognition, 2017. 2, 5, 6, 7
[34] T. Makovski, G. A. Vazquez, and Y. V. Jiang. Visual learning
in multiple-object tracking. PLoS One, 2008. 1
[35] K.-K. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Leal-
Taixé, D. Cremers, and L. Van Gool. Video object segmen-
tation without temporal information. In IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2017. 2, 3, 6
[36] N. Märki, F. Perazzi, O. Wang, and A. Sorkine-Hornung. Bi-
lateral space video segmentation. In IEEE Conference on
Computer Vision and Pattern Recognition, 2016. 2, 3, 6
[37] O. Miksik, J.-M. Pérez-Rúa, P. H. Torr, and P. Pérez. Roam:
a rich object appearance model with application to rotoscop-
ing. In IEEE Conference on Computer Vision and Pattern
Recognition, 2017. 1
[38] F. Perazzi. Video Object Segmentation. PhD thesis, ETH
Zurich, 2017. 1, 3, 6
[39] F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and
A. Sorkine-Hornung. Learning video object segmentation
from static images. In IEEE Conference on Computer Vision
and Pattern Recognition, 2017. 2, 3, 6, 7
[40] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool,
M. Gross, and A. Sorkine-Hornung. A benchmark dataset
and evaluation methodology for video object segmentation.
In IEEE Conference on Computer Vision and Pattern Recog-
nition, 2017. 1, 3, 6, 7, 8, 13
[41] F. Perazzi, O. Wang, M. Gross, and A. Sorkine-Hornung.
Fully connected object proposals for video segmentation. In
IEEE International Conference on Computer Vision, 2015. 3
[42] P. Pérez, C. Hue, J. Vermaak, and M. Gangnet. Color-Based
Probabilistic Tracking. In European Conference on Com-
puter Vision, 2002. 2
[43] P. O. Pinheiro, R. Collobert, and P. Dollár. Learning to seg-
ment object candidates. In Advances in Neural Information
Processing Systems, 2015. 2, 4
[44] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollár. Learn-
ing to refine object segments. In European Conference on
Computer Vision, 2016. 4, 7, 11
[45] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-
Hornung, and L. Van Gool. The 2017 davis chal-
lenge on video object segmentation. arXiv preprint
arXiv:1704.00675, 2017. 6, 8, 13
[46] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards
real-time object detection with region proposal networks. In
Advances in Neural Information Processing Systems, 2015.
2, 3
[47] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge. In-
ternational Journal of Computer Vision, 2015. 5
[48] A. W. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara,
A. Dehghan, and M. Shah. Visual tracking: An experimental
10
survey. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 2014. 1, 3
[49] R. Tao, E. Gavves, and A. W. Smeulders. Siamese instance
search for tracking. In IEEE Conference on Computer Vision
and Pattern Recognition, 2016. 2
[50] Y.-H. Tsai, M.-H. Yang, and M. J. Black. Video segmenta-
tion via object flow. In IEEE Conference on Computer Vision
and Pattern Recognition, 2016. 2, 3, 6
[51] J. Valmadre, L. Bertinetto, J. Henriques, A. Vedaldi, and
P. H. S. Torr. End-to-end representation learning for correla-
tion filter based tracking. In IEEE Conference on Computer
Vision and Pattern Recognition, 2017. 2
[52] J. Valmadre, L. Bertinetto, J. F. Henriques, R. Tao,
A. Vedaldi, A. Smeulders, P. H. S. Torr, and E. Gavves.
Long-term tracking in the wild: A benchmark. In European
Conference on Computer Vision, 2018. 1
[53] P. Voigtlaender and B. Leibe. Online adaptation of convo-
lutional neural networks for video object segmentation. In
British Machine Vision Conference, 2017. 2, 3, 6, 7
[54] T. Vojir and J. Matas. Pixel-wise object segmentations for
the vot 2016 dataset. Research Report CTU-CMP-2017–01,
Center for Machine Perception, Czech Technical University,
Prague, Czech Republic, 2017. 6
[55] L. Wen, D. Du, Z. Lei, S. Z. Li, and M.-H. Yang. Jots: Joint
online tracking and segmentation. In IEEE Conference on
Computer Vision and Pattern Recognition, 2015. 2, 3, 6
[56] Y. Wu, J. Lim, and M.-H. Yang. Online object tracking: A
benchmark. In IEEE Conference on Computer Vision and
Pattern Recognition, 2013. 1, 3
[57] S. Wug Oh, J.-Y. Lee, K. Sunkavalli, and S. Joo Kim. Fast
video object segmentation by reference-guided mask propa-
gation. In IEEE Conference on Computer Vision and Pattern
Recognition, 2018. 2, 3, 7
[58] N. Xu, L. Yang, Y. Fan, J. Yang, D. Yue, Y. Liang, B. Price,
S. Cohen, and T. Huang. Youtube-vos: Sequence-to-
sequence video object segmentation. In European Confer-
ence on Computer Vision, 2018. 2, 5, 6
[59] L. Yang, Y. Wang, X. Xiong, J. Yang, and A. K. Katsaggelos.
Efficient video object segmentation via network modulation.
In IEEE Conference on Computer Vision and Pattern Recog-
nition, June 2018. 2, 3, 7
[60] T. Yang and A. B. Chan. Learning dynamic memory net-
works for object tracking. In European Conference on Com-
puter Vision, 2018. 2, 3
[61] D. Yeo, J. Son, B. Han, and J. H. Han. Superpixel-based
tracking-by-segmentation using markov chains. In IEEE
Conference on Computer Vision and Pattern Recognition,
2017. 2
[62] J. S. Yoon, F. Rameau, J. Kim, S. Lee, S. Shin, and I. S.
Kweon. Pixel-level matching for video object segmentation
using convolutional neural networks. In IEEE International
Conference on Computer Vision, 2017. 7
[63] Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, and W. Hu.
Distractor-aware siamese networks for visual object track-
ing. In European Conference on Computer Vision, 2018. 2,
3, 5, 6, 7
A. Architectural details block score box mask
conv5 1 × 1, 256 1 × 1, 256 1 × 1, 256
Network backbone. Table 8 illustrates the details of our conv6 1 × 1, 2k 1 × 1, 4k 1 × 1, (63 × 63)
backbone architecture (fθ in the main paper). For both vari-
ants, we use a ResNet-50 [18] until the final convolutional Table 9. Architectural details of the three-branch head. k denotes
the number of anchor boxes per RoW.
layer of the 4-th stage. In order to obtain a higher spatial
resolution in deep layers, we reduce the output stride to block score mask
8 by using convolutions with stride 1. Moreover, we in- conv5 1 × 1, 256 1 × 1, 256
crease the receptive field by using dilated convolutions [6]. conv6 1 × 1, 1 1 × 1, (63 × 63)
Specifically, we set the stride to 1 and the dilation rate to
2 in the 3×3 conv layer of conv4 1. Differently to the Table 10. Architectural details of the two-branch head.
original ResNet-50, there is no downsampling in conv4 x.
We also add to the backbone an adjust layer (a 1×1 con- 61*61*8 2x, up + 31*31*16
conv,
3*3, 16
conv,
3*3, 16 31*31*16

volutional layer with 256 output channels). Examplar and

search patches share the network’s parameters from conv1 31*31*16
to conv4 x, while the parameters of the adjust layer are
not shared. The output features of the adjust layer are then conv,
+ Element-wise
3*3, 16 sum
depth-wise cross-correlated, resulting a feature map of size ReLU
17×17. conv,
3*3, 32

Network heads. The network architecture of the branches conv,

Refinement module
3*3, 64
of both variants are shows in Table 9 and 10. The conv5
block in both variants contains a normalisation layer and 31*31*256
conv 2
ReLU non-linearity while conv6 only consists of a 1×1
convolutional layer. Figure 6. Example of a refinement module U3 .
Mask refinement module. With the aim of producing a
more accurate object mask, we follow the strategy of [44],
which merges low and high resolution features using multi-
ple refinement modules made of upsampling layers and skip
connections. Figure 6 gives an example of refinement mod- Target
ule U3 , while Figure 8 illustrates how a mask is generated Search
with stacked refinement modules.

block examplar output size search output size backbone

conv1 61×61 125×125 7×7, 64, stride 2
3×3 max pool, stride 2
 
1×1, 64
conv2 x 31×31 63×63  3×3, 64 ×3
1×1, 256
 
1×1, 128 Figure 7. Score maps from the mask branch at different locations.
conv3 x 15×15 31×31  3×3, 128 ×4
1×1, 512
 
1×1, 256 score branch to select the final output mask (using the loca-
conv4 x 15×15 31×31  3×3, 256 ×6
1×1, 1024 tion attaining the maximum score). The example of Figure 7
adjust 15×15 31×31 1×1, 256 illustrates the multiple output masks produced by the mask
xcorr 17 × 17 depth-wise branch, each corresponding to a different RoW.
Benchmark sequences. More qualitative results for VOT
Table 8. Backbone architecture. Details of each building block are
and DAVIS sequences are shown in Figure 9 and 10.
shown in square brackets.

B. Further qualitative results

Different masks at different locations. Our model gener-
ates a mask for each RoW. During inference, we rely on the

11
 conv, deconv,
Sigmoid 127*127*4 𝑈$ 61*61*8 𝑈# 31*31*16 𝑈" 15*15*32 32
3*3, 1

127*127*1
mask

61*61*64 31*31*256 15*15*512 15*15*1024 15*15*256 17*17*256

adjust 1*1*256
conv 1 conv 2 conv 3 con 4
(RoW)

127*127*3

ResNet-50

31*31*256
conv 1 conv 2 conv 3 conv 4 adjust

255*255*3

Figure 8. Schematic illustration of the stacked refinement modules.

butterfly
crabs1
iceskater1
iceskater2
motocross1
singer2
soccer1

Figure 9. Further qualitative results of our method on sequences from the visual object tracking benchmark VOT-2018 [27].

12
dog
drift-straight
goat
Libby
motocross-jump
parkour
Gold-Fish

Figure 10. Further qualitative results of our method on sequences from the semi-supervised video object segmentation benchmarks DAVIS-
2016 [40] and DAVIS-2017 [45]. Multiple masks are obtained from different inferences (with different initialisations).

13
 Revealing Scenes by Inverting Structure from Motion Reconstructions

Francesco Pittaluga1 Sanjeev J. Koppal1 Sing Bing Kang2 Sudipta N. Sinha2

1 2
University of Florida Microsoft Research
arXiv:1904.03303v1 [cs.CV] 5 Apr 2019

(a) SfM point cloud (top view) (b) Projected 3D points (c) Synthesized Image (d) Original Image

Figure 1: S YNTHESIZING IMAGERY FROM A S F M POINT CLOUD : From left to right: (a) Top view of a SfM reconstruction
of an indoor scene, (b) 3D points projected into a viewpoint associated with a source image, (c) the image reconstructed using
our technique, and (d) the source image. The reconstructed image is very detailed and closely resembles the source image.

Abstract workplaces, and other sensitive environments. Image-based

Many 3D vision systems localize cameras within a scene
using 3D point clouds. Such point clouds are often obtained
using structure from motion (SfM), after which the images
are discarded to preserve privacy. In this paper, we show,
for the first time, that such point clouds retain enough in-
formation to reveal scene appearance and compromise pri-
vacy. We present a privacy attack that reconstructs color
images of the scene from the point cloud. Our method is
based on a cascaded U-Net that takes as input, a 2D multi-
channel image of the points rendered from a specific view-
point containing point depth and optionally color and SIFT
descriptors and outputs a color image of the scene from
that viewpoint. Unlike previous feature inversion meth-
ods [46, 9], we deal with highly sparse and irregular 2D
point distributions and inputs where many point attributes
are missing, namely keypoint orientation and scale, the de-
scriptor image source and the 3D point visibility. We evalu-
ate our attack algorithm on public datasets [24, 39] and an-
alyze the significance of the point cloud attributes. Finally,
we show that novel views can also be generated thereby en-
abling compelling virtual tours of the underlying scene.
1. Introduction
Emerging AR technologies on mobile devices based on
ARCore [2], ARKit [3], 3D mapping APIs [1], and new
devices such as HoloLens [15] have set the stage for de-
ployment of devices with always-on cameras in our homes,
localization techniques allow such devices to estimate their
precise pose within the scene [18, 37, 23, 25]. However,
these localization methods requires persistent storage of 3D
models of the scene which contains sparse 3D point clouds
reconstructed using images and SfM algorithms [38].
SfM source images are usually discarded to safeguard
privacy. Surprisingly, however, we show that the SfM point
cloud and the associated attributes such as color and SIFT
descriptors contain enough information to reconstruct de-
tailed comprehensible images of the scene (see Fig. 1 and
Fig. 3). This suggests that the persistent point cloud storage
poses serious privacy risks that have been widely ignored
so far but will become increasingly relevant as localization
services are adopted by a larger user community.
While privacy issues for wearable devices have been
studied [16], to the best of our knowledge, a systematic
analysis of privacy risk of storing 3D point cloud maps has
never been reported. We illustrate the privacy concerns by
proposing the problem of synthesizing color images from
an SfM model of a scene. We assume that the reconstructed
model contains a sparse 3D point cloud with optional at-
tributes such as descriptors, color, point visibility and asso-
ciated camera poses but not the source images.
We make the following contributions: (1) We intro-
duce the problem of inverting a sparse SfM point cloud
and reconstructing detailed views of the scene from arbi-
trary viewpoints. This problem differs from the previously
studied single-image feature inversion problem due to the
need to deal with highly sparse point distributions and a
higher degree of missing information in the input, namely
unknown keypoint orientation and scale, unknown image
source of descriptors, and unknown 3D point visibilities.
(2) We present a new approach based on three neural net-
works where the first network performs visibility estima-
tion, the second network reconstructs the image and the
third network uses an adversarial framework to further re-
fine the image quality. (3) We systematically analyze vari-
ants of the inversion attack that exploits additional attributes
that may be available, namely per-point descriptors, color
and information about the source camera poses and point
visibility and show that even the minimalist representation
(descriptors only) are prone to the attack. (4) We demon-
strate the need for developing privacy preserving 3D repre-
sentations, since the reconstructed images reveal the scene
in great details and confirm the feasibility of the attack in a
wide range of scenes. We also show that novel views of the
scene can be synthesized without any additional effort and
a compelling virtual tour of a scene can be easily generated.
The three networks in our cascade are trained on 700+
indoor and outdoor SfM reconstructions generated from
500k+ multi-view images taken from the NYU2 [39] and
MegaDepth [24] datasets. The training data for all three
networks including the visibility labels were generated au-
tomatically using COLMAP [38]. Next we compare our
approach to previous work on inverting image features
[46, 9, 8] and discuss how the problem of inverting SfM
models poses a unique set of challenges.
2. Related Work
In this section, we review existing work on inverting im-
age features and contrast them to inverting SfM point cloud
models. We then broadly discuss image-to-image transla-
tion, upsampling and interpolation, and privacy attacks.
Inverting features. The task of reconstructing images from
features has been explored to understand what is encoded
by the features, as was done for SIFT features by Weinza-
epfel et al. [46], HOG features by Vondrick et al. [45] and
bag-of-words by Kato and Harada [20]. Recent work on
the topic has been primarily focused on inverting and inter-
preting CNN features [49, 48, 29]. Dosovitskiy and Brox
proposed encoder-decoder CNN architectures for inverting
many different features (DB1) [9] and later incorporated ad-
versarial training with perceptual loss functions (DB2) [8].
While DB1 [9] showed some qualitative results on inverting
sparse SIFT, both papers focused primarily on dense fea-
tures. In contrast to these feature inversion approaches, we
focus solely on inverting SIFT descriptors stored along with
SfM point clouds. While the projected 3D points on a cho-
sen viewpoint may resemble single image SIFT features,
there are some key differences. First, our input 2D point
distributions can be highly sparse and irregular, due to the
typical inherent sparsity of SfM point clouds. Second, the
SIFT keypoint scale and orientation are unknown since SfM
methods retain only the descriptors for the 3D points. Third,
each 3D point typically has only one descriptor sampled
from an arbitrary source image whose identity is not stored
either, entailing descriptors with unknown perspective dis-
tortions and photometric inconsistencies. Finally, the 3D
point visibilities are also unknown and we will demonstrate
the importance of visibility reasoning in the paper.
Image-to-Image Translation. Various methods such as
Pix2Pix [19], CycleGan [50], CoGAN [27] and related un-
supervised approaches [7, 26, 34] use conditional adversar-
ial networks to transform between 2D representations, such
as edge to color, label to color, and day to night images.
While such networks are typically dense (without holes)
and usually low-dimensional (single channel or RGB), Con-
tour2Im [5] takes sparse 2D points sampled along gradients
along with low-dimensional input features. In contrast to
our work, these approaches are trained on specific object
categories and semantically similar images. While we use
similar building blocks to these methods (encoder-decoder
networks, U-nets, adversarial loss, and perceptual loss), our
networks can generalize to arbitrary images, and are trained
on large scale indoor and outdoor SfM datasets.
Upsampling. When the input and output domains are iden-
tical, deep networks have shown excellent results on up-
sampling and superresolution tasks for images, disparity,
depth maps and active range maps [4, 28, 43, 36, 17]. How-
ever, prior upsampling methods typically focus on inputs
with uniform sparsity. Our approach differs due to the non-
uniform spatial sampling in the input data which also hap-
pens to be high dimensional and noisy since the input de-
scriptors are from different source images and viewpoints.
Novel view synthesis and image-based rendering. Deep
networks can significantly improve photorealism in free
viewpoint image-based rendering [12, 14]. Additionally,
several works have also explored monocular depth estima-
tion and novel view synthesis using U-Nets [11, 24, 31].
Our approach arguably provides similar photorealistic visu-
ally quality – remarkably, from sparse SfM reconstructions
instead of images. This is disappointing news from a pri-
vacy perspective but could be useful in other settings for
generating photorealistic images from 3D reconstructions.
CNN-based privacy attacks and defense techniques. Re-
cently, McPherson et al. [30] and Vasiljevic et al. [44]
showed that deep models could defeat existing image obfus-
cation methods. Further more, many image transformations
can be considered as adding noise and undoing them as de-
noising, and here deep networks have been quite success-
ful [47]. To defend against CNN-based attacks, attempts at
learning CNN-resistant transformations have shown some
promise [33, 10, 35, 13]. Concurrent to our work, Speciale
et al. [41] introduced the privacy preserving image-based
localization problem to address the privacy issues we have
brought up. They proposed a new camera pose estimation
technique using an obfuscated representation of the map ge-
ometry which can defend against our inversion attack.
3. Method
The input to our pipeline is a feature map generated from
a SfM 3D point cloud model given a specific viewpoint i.e.
a set of camera extrinsic parameters. We obtain this fea-
ture map by projecting the 3D points on the image plane
and associating the 3D point attributes (SIFT descriptor,
color, etc.) with the discrete 2D pixel where the 3D point
projects in the image. When multiple points project to the
same pixel, we retain the attributes for the point closest to
the camera and store its depth. We train a cascade of three
encoder-decoder neural networks for visibility estimation,
coarse image reconstruction and the final refinement step
which recovers fine details in the reconstructed image.
Visibility Estimation. Since SfM 3D point clouds are of-
ten quite sparse and the underlying geometry and topology
of the surfaces in the scene are unknown, it is not possi-
ble to easily determine which 3D points should be consid-
ered as visible from a specific camera viewpoint just us-
ing z-buffering. This is because a sufficient number of 3D
points may not have been reconstructed on the foreground
occluding surfaces. This produces 2D pixels in the input
feature maps which are associated with 3D points in the
background i.e. lie on surfaces which are occluded from
that viewpoint. Identifying and removing such points from
the feature maps is critical to generating high-quality im-
ages and avoiding visual artifacts. We propose to recover
point visibility using a data-driven neural network-based
approach, which we refer to as V ISIB N ET. We also evaluate
two geometric methods which we refer to as V ISIB S PARSE
and V ISIB D ENSE. Both geometric methods however re-
quire additional information which might be unavailable.
Coarse Image Reconstructon and Refinement. Our tech-
nique for image synthesis from feature maps consists of a
coarse image reconstruction step followed by a refinement
step. C OARSE N ET is conditioned on the input feature map
and produces an RGB image of the same width and height
as the feature map. R EFINE N ET outputs the final color im-
age which has the same size, given the input feature map
along with the image output of C OARSE N ET as its input.
3.1. Visibility Estimation
If we did not perform explicit visibility prediction in
our pipeline, some degree of implicit visibility reasoning
would still be carried out by the image synthesis network
C OARSE N ET. In theory, this network has access to the input
depths and could learn to reason about visibility. However,
in practice, we found that this approach to be inaccurate,
especially in regions where the input feature maps contain
a low ratio of visible to occluded points. Qualitative exam-
ples of these failure cases are shown in Figure 5. Therefore
we explored explicit visibility estimation approaches based
on geometric reasoning as well as learning.
VisibSparse. We explored a simple geometric method that
we refer to as V ISIB S PARSE. It is based on the “point splat-
ting” paradigm used in computer graphics. By considering
only the depth channel in the input, we apply a min filter
with a k × k kernel on the feature map to obtain a filtered
depth map. Here, we used k = 3 based on empirical test-
ing. Each entry in the feature map whose depth value is no
greater than 5% of the depth value in the filtered depth map
is retained as visible. Otherwise, the point is considered
occluded and the associated entry in the input is removed.
VisibDense. When the camera poses for the source im-
ages computed during SfM and the image measurements
are stored along with the 3D point cloud, it is often possible
to exploit that data to compute a dense scene reconstruc-
tion. Labatut et al. [21] proposed such a method to com-
pute a dense triangulated mesh by running space carving on
the tetrahedral cells of the 3D Delaunay triangulation of the
sparse SfM points. We used this method, implemented in
COLMAP [38] and computed 3D point visibility based on
the reconstructed mesh model using traditional z-buffering.
VisibNet. A geometric method such as V ISIB D ENSE can-
not be used when the SfM cameras poses and image mea-
surements are unavailable. We therefore propose a general
regression-based approach that directly predicts the visibil-
ity from the input feature maps, where the predictive model
is trained using supervised learning. Specifically, we train
an encoder-decoder neural network which we refer to as
V ISIB N ET to classify each input point as either “visible”
or “occluded”. Ground truth visibility labels were gener-
ated automatically by leveraging V ISIB D ENSE on all train,
test, and validation scenes. Using V ISIB N ET’s predictions
to “cull” occluded points from the input feature maps prior
to running C OARSE N ET significantly improves the quality
of the reconstructed images, especially in regions where the
input feature map contains fewer visible points compared to
the number of points that are actually occluded.
3.2. Architecture
A sample input feature map as well as our complete net-
work architecture consisting of V ISIB N ET, C OARSE N ET,
and R EFINE N ET is shown in Figure 2. The input to our
network is an H × W × n dimensional feature map consist-
ing of n-dimensional feature vectors with different combi-
nations of depth, color, and SIFT features at each 2D loca-
tion. Except for the number of input/output channels in the
 nD
Input Tensor
=
z RGB SIFT descriptor encoder decoder conv. layers

nD Input VisibNet Visibility CoarseNet RefineNet

RGB image RGB image
Map (output)

Figure 2: N ETWORK A RCHITECTURE : Our network has three sub-networks – V ISIB N ET, C OARSE N ET and R EFINE N ET.
The upper left shows that the input to our network is a multi-dimensional nD array. The paper explores network variants where
the inputs are different subsets of depth, color and SIFT descriptors. The three sub-networks have similar architectures. They
are U-Nets with encoder and decoder layers with symmetric skip connections. The extra layers at the end of the decoder
layers (marked in orange) are there to help with high-dimensional inputs. See the text and supplementary material for details.

first/final layers, each sub-network has the same architec-
ture consisting of U-Nets with a series of encoder-decoder
layers with skip connections. Compared to conventional U-
Nets, our network has a few extra convolutional layers at
the end of the decoder layers. These extra layers facilitate
propagation of information from the low-level features, par-
ticularly the information extracted from SIFT descriptors,
via the skip connections to a larger pixel area in the out-
put, while also helping to attenuate visual artifacts resulting
from the highly sparse and irregular distribution of these
features. We use nearest neighbor upsampling followed by
standard convolutions instead of transposed convolutions as
the latter are known to produce artifacts [32].
3.3. Optimization
where V : RH×W ×N → RW ×H×1 denotes a differen-
tiable function representing V ISIB N ET, with learnable pa-
rameters, Ux ∈ RH×W ×1 denotes the ground-truth visibil-
ity map for feature map Fx , and the summation is carried
out over the set of M non-zero spatial locations in Fx .
C OARSE N ET was trained next, using a combination of
an L1 pixel loss and an L2 perceptual loss (as in [22, 8])
over the outputs of layers relu1 1, relu2 2, and relu3 3 of
VGG16 [40] pre-trained for image classification on the Im-
ageNet [6] dataset. The weights of V ISIB N ET remained
fixed while C OARSE N ET was being trained using the loss
3
X
LC = ||C(Fx ) − x||1 + α ||φi (C(Fx )) − φi (x)||22 , (2)
i=1

We separately train the sub-networks in our architecture,
V ISIB N ET, C OARSE N ET, and R EFINE N ET. Batch normal-
ization was used in every layer, except the final one in each
network. We applied Xavier initialization and projections
were generated on-the-fly to facilitate data augmentation
during training and novel view generation after training.
V ISIB N ET was trained first to classify feature map points
as either visible or occluded, using ground-truth visibility
masks generated automatically by running V ISIB D ENSE for
all train, test, and validation samples. Given training pairs
of input feature maps Fx ∈ RH×W ×N and target source
images x ∈ RH×W ×3 , V ISIB N ET’s objective is
M
X 
images used to generate the SfM models and images syn-
LV (x) = − Ux log (V (Fx ) + 1)/2 +
i=1
(1)


(1 − Ux )log (1 − V (Fx ))/2 i ,
where C : RH×W ×N → RH×W ×3 denotes a differentiable
function representing C OARSE N ET, with learnable param-
H W
eters, and φ1 : RH×W ×3 → R 2 × 2 ×64 , φ2 : RH×W ×3 →
H W H W
R4 4× ×128
, and φ3 : R H×W ×3
→ R 8 × 8 ×256 denote
the layers relu1 1, relu2 2, and relu2 2, respectively, of the
pre-trained VGG16 network.
R EFINE N ET was trained last using a combination of an
L1 pixel loss, the same L2 perceptual loss as C OARSE N ET,
and an adversarial loss. While training R EFINE N ET, the
weights of V ISIB N ET and C OARSE N ET remained fixed.
For adversarial training, we used a conditional discrimi-
nator whose goal was to distinguish between real source
thesized by R EFINE N ET. The discriminator trained using
cross-entropy loss similar to Eq. (1). Additionally, to sta-
bilize adversarial training, φ1 (R(Fx ))1 , φ2 (R(Fx ))1 , and
 Desc. Inp. Feat. MAE SSIM Inp. Feat. Accuracy
Src. D O S 20% 60% 100% 20% 60% 100%
Data
z D C 20% 60% 100%
Si X X X .126 .105 .101 .539 .605 .631 X × × .948 .948 .946
Si X X × .133 .111 .105 .499 .568 .597 X × X .938 .943 .941
Si X × X .129 .107 .102 .507 .574 .599 MD
X X × .949 .951 .948
Si X × × .131 .113 .109 .477 .550 .578 X X X .952 .952 .950
M X × × .147 .128 .123 .443 .499 .524 X × × .892 .907 .908
X × X .897 .908 .910
Table 1: I NVERTING S INGLE I MAGE S IFT F EATURES : NYU
X X × .895 .907 .909
The top four rows compare networks designed for differ- X X X .906 .916 .917
ent subsets of single image (Si) inputs: descriptor (D), key-
point orientation (O) and scale (S). Test error (MAE) and
Table 2: E VALUATION OF V ISIB N ET: We trained four ver-
accuracy (SSIM) obtained when 20%, 60% and all the SIFT
sion of V ISIB N ET, each with a different set of input at-
features are used. Lower MAE and higher SSIM values are
tributes, namely, z (depth), D (SIFT) and C (color) to eval-
better. The last row is for when the descriptors originate
uate their relative importance. Ground truth labels were ob-
from multiple (M) different and unknown source images.
tained with VisibDense. The table reports mean classifica-
tion accuracy on the test set for the NYU and MD datasets.
φ3 (R(Fx ))1 were concatenated before the first, second, and The results show that V ISIB N ET achieves accuracy greater
third convolutional layers of the discriminator as done in than 93.8% and 89.2% on MD and NYU respectively and is
[42]. R EFINE N ET denoted as R() has the following loss. not very sensitive to sparsity levels and input attributes.

3
X partitioned the scenes into training, validation, and testing
LR =||R(Fx ) − x||1 + α ||φi (R(Fx )) − φi (x)||22
i=1
(3)
+ β[log(D(x)) + log(1 − D(R(Fx )))].
Here, the two functions, R : RH×W ×N +3 → RH×W ×3
and D : RH×W ×N +3 → R denote differentiable functions
representing R EFINE N ET and the discriminator, respec-
tively, with learnable parameters. We trained R EFINE N ET
to minimize LR by applying alternating gradient updates
to R EFINE N ET and the discriminator. The gradients were
computed on mini-batches of training data, with different
batches used to update R EFINE N ET and the discriminator.
4. Experimental Results
We now report a systematic evaluation of our method.
Some of our results are qualitatively summarized in Fig.
3, demonstrating robustness to various challenges, namely,
missing information in the point clouds, effectiveness of our
visibility estimation, and the sparse and irregular distribu-
tion of input samples over a large variety of scenes.
Dataset. We use the MegaDepth [24] and NYU [39]
datasets in our experiments. MegaDepth (MD) is an In-
ternet image dataset with ∼150k images of 196 landmark
scenes obtained from Flickr. NYU contains ∼400k images
of 464 indoor scenes captured with the Kinect (we only used
the RGB images). These datasets cover very different scene
content, image resolution, and generate very different dis-
tribution of SfM points and camera poses. Generally, NYU
scenes produce far fewer SfM points than the MD scenes.
Preprocessing. We processed the 660 scenes in MD and
NYU using the SfM implementation in COLMAP [38]. We
sets with 441, 80, and 139 scenes respectively. All images
of one scene were included only in one of the three groups.
We report results using both the average mean absolute error
(MAE), where color values are scaled to the range [0,1].
and average structured similarity (SSIM). Note that lower
MAE and higher SSIM values indicate better results.
Inverting Single Image SIFT Features. Consider the sin-
gle image scenario, with trivial visibility estimation and
identical input to [9]. We performed an ablation study in
this scenario, measuring the effect of inverting features with
unknown keypoint scale, orientation, and multiple unknown
image sources. Four variants of C OARSE N ET were trained,
then tested at three sparsity levels. The results are shown
in Table 1 and Figure 4. Table 1 reports MAE and SSIM
across a combined MD and NYU dataset. The sparsity per-
centage refers to how many randomly selected features were
retained in the input, and our method handles a wide range
of sparsity reasonably well. From the examples in Figure 4,
we observe that the networks are surprisingly robust at in-
verting features with unknown orientation and scale; while
the accuracy drops a bit as expected, the reconstructed im-
ages are still recognizable. Finally, we quantify the effect
of unknown and different image sources for the SIFT fea-
tures. The last row of Table 1 shows that indeed the feature
inversion problem becomes harder but the results are still re-
markably good. Having demonstrated that our work solves
a harder problem than previously tackled, we now report
results on inverting SfM points and their features.
4.1. Visibility Estimation
We first independently evaluate the performance of the
proposed V ISIB N ET model and compare it to the geomet-
Figure 3: Q UALITATIVE R ESULTS : Each result is a 3 × 1 set of square images, showing point clouds (with occluded points
in red), image reconstruction and original. The first four columns (top and bottom) show results from the MegaDepth dataset
(internet scenes) and the last four columns (top and bottom) show results from indoor NYU scenes. Sparsity: Our network
handles a large variety in input sparsity (density decreases from left to right). In addition, perspective projection accentuates
the spatially-varying density differences, and the MegaDepth outdoor scenes have concentrated points in the input whereas
NYU indoor scenes have far samples. Further, the input points are non-homogeneous, with large holes which our method
gracefully fills in. Visual effects: For the first four columns (MD scenes) our results give the pleasing effect of uniform
illumination (see top of first column). Since our method relies on SfM, moving objects are not recovered. Scene diversity:
The fourth column is an aerial photograph, an unusual category that is still recovered well. For the last four columns (NYU
scenes), despite lower sparsity, we can recover textures in common household scenes such as bathrooms, classrooms and
bedrooms. The variety shows that our method does not learn object categories and works on any scene. Visibility: All scenes
benefit from visibility prediction using V ISIB N ET which for example was crucial for the bell example (lower 2nd column).

ric methods V ISIB S PARSE and V ISIB D ENSE. We trained
four variants of V ISIB N ET designed for different subsets of
input attributes to classify points in the input feature map
as “visible” or “occluded”. We report classification accu-
racy separately on the MD and NYU test sets even though
the network was trained on the combined training set (see
Table 2). We observe that V ISIB N ET is largely insensitive
to scene type, sparsity levels, and choice of input attributes
such as depth, color, and descriptors. The V ISIB N ET vari-
ant designed for depth only has 94.8% and 89.2% mean
classification accuracy on MD and NYU test sets, respec-
tively, even when only 20% of the input samples were used
to simulate sparse inputs. Table 3 shows that when points
predicted as occluded by V ISIB N ET are removed from the
input to C OARSE N ET, we observe a consistent improve-
ment when compared to C OARSE N ET carrying both the
burdens of visibility and image synthesis (denoted as Im-
plicit in the table). While the improvement may not seem
numerically large, in Figure 5 we show insets where visual
artifacts (bookshelf above, building below) are removed.
 (a) Input (b) SIFT (c) SIFT + s (d) SIFT + o (e) SIFT + s + o (f) Original

Figure 4: I NVERTING S IFT F EATURES IN A S INGLE I MAGE : (a) 2D keypoint locations. Results obtained with (b) only
descriptor, (c) descriptor and keypoint scale, (d) descriptor and keypoint orientation, (e) descriptor, scale and orientation. (f)
Original image. Results from using only descriptors (2nd column) are only slightly worse than the baseline (5th column).

(a) Input (b) Pred. (VisibNet) (c) Implicit (d) VisibNet (e) VisibDense (f) Original

Figure 5: I MPORTANCE OF VISIBILITY ESTIMATION : Examples showing (a) input 2D point projections (in blue), (b) pre-
dicted visibility from V ISIB N ET – occluded (red) and visible (blue) points, (c–e) results from I MPLICIT (no explicit visibility
estimation), V ISIB N ET (uses a CNN) and V ISIB D ENSE (uses z-buffering and dense models), and (f) the original image.

4.2. Relative Significance of Point Attributes
We trained four variants of C OARSE N ET, each with a
different set of the available SfM point attributes. The goal
here is to measure the relative importance of each of the at-
tributes. This information could be used to decide which
optional attributes should be removed when storing SfM
model to enhance privacy. We report reconstruction error on
the test set for both indoor (NYU) and outdoor scenes (MD)
for various sparsity levels in Table 4 and show qualitative
evaluation on the test set in Figure 6. The results indicate
that our approach is largely invariant to sparsity and capable
of capturing very fine details even when the input feature
map contains just depth, although, not surprisingly, color
and SIFT descriptors significantly improves visual quality.
4.3. Significance of RefineNet
In Figure 7 we qualitatively compare two scenes where
the feature maps had only depth and descriptors (left) and
when it had all the attributes (right). For privacy preser-
vation, these results are sobering. While Table 4 showed
that C OARSE N ET struggles when color is dropped (sug-
gesting an easy solution of removing color for privacy),
Figure 7 (left) unfortunately shows that R EFINE N ET recov-
ers plausible colors and improves results a lot. Of course,
R EFINE N ET trained on all features also does better than
C OARSE N ET although less dramatically (Figure 7 (right)).
 Visibility MAE SSIM
Data
Est. 20% 60% 100% 20% 60% 100%
Implicit .201 .197 .195 .412 .436 .445
VisibSparse .202 .197 .196 .408 .432 .440
MD
VisibNet .201 .196 .195 .415 .440 .448
VisibDense .201 .196 .195 .417 .442 .451
Implicit .121 .100 .094 .541 .580 .592
VisibSparse .122 .100 .094 .539 .579 .592
NYU
VisibNet .120 .098 .092 .543 .583 .595
VisibDense .120 .097 .090 .545 .587 .600

Table 3: I MPORTANCE OF V ISIBILITY E STIMATION : Both z+D z+D+C

sub-tables show results obtained using I MPLICIT i.e. no
explicit occlusion reasoning where of burden of visibility
estimation implicitly falls on C OARSE N ET, VisibNet and
the geometric methods V ISIB S PARSE and V ISIB D ENSE.
Lower MAE and higher SSIM values are better.
Figure 7: I MPORTANCE OF R EFINE N ET: (Top row)
C OARSE N ET results. (Bottom Row) R EFINE N ET results.
(Left) Networks use depth and descriptors (z + D). (Right)
Networks use depth, descriptor and color (z + D + C).

Inp. Feat. MAE SSIM

Data
z D C 20% 60% 100% 20% 60% 100%
X × × .258 .254 .253 .264 .254 .250
X × X .210 .204 .202 .378 .394 .403
MD
X X × .228 .223 .221 .410 .430 .438
X X X .201 .196 .195 .414 .439 .448
X × × .295 .290 .289 .244 .209 .197
X × X .148 .121 .111 .491 .528 .546
Figure 8: N OVEL V IEW S YNTHESIS : Synthesized images
NYU from virtual viewpoints in two NYU scenes [39] helps to
X X × .207 .179 .171 .493 .528 .539
X X X .121 .099 .093 .542 .582 .594 interpret the cluttered scenes (see supplementary video).

Table 4: E FFECT OF P OINT ATTRIBUTES : Performance of
four networks designed for different sets of input attributes
– z (depth), D (SIFT) and C (color), on MD and NYU. Input
sparsity is simulated by applying random dropout to input
samples during training and testing.
such results is more difficult (in contrast to our experiments
where aligned real camera images are available), we show
qualitative results in Figure 8 and generate virtual tours
based on the synthesized novel views1 . Such novel view
based virtual tours can make scene interpretation easier for
an attacker even when the images contain some artifacts.

z z+D z+C z+D+C orig
Figure 6: E FFECT OF P OINT ATTRIBUTES : Results ob-
tained with different attributes. Left to right: depth [z],
depth + SIFT [z + D], depth + color [z + C], depth + SIFT
+ color [z + D + C] and the original image. (see Table 4).
4.4. Novel View Synthesis
Our technique can be used to easily generate realistic
novel views of the scene. While quantitatively evaluating
5. Conclusion
In this paper, we introduced a new problem, that of in-
verting a sparse SfM point cloud and reconstructing color
images of the underlying scene. We demonstrated that sur-
prisingly high quality images can be reconstructed from the
limited amount of information stored along with sparse 3D
point cloud models. Our work highlights the privacy and
security risks associated with storing 3D point clouds and
the necessity for developing privacy preserving point cloud
representations and camera localization techniques, where
the persistent scene model data cannot easily be inverted to
reveal the appearance of the underlying scene. This was
also the primary goal in concurrent work on privacy pre-
serving camera pose estimation [41] which proposes a de-
fense against the type of attacks investigated in our paper.
Another interesting avenue of future work would be to ex-
plore privacy preserving features for recovering correspon-
dences between images and 3D models.
1 see the video in the supplementary material.
References
[1] 6D.AI. http://6d.ai/, 2018.
[2] ARCore. developers.google.com/ar/,
2018.
[3] ARKit. developer.apple.com/arkit/,
2018.
[4] Z. Chen, V. Badrinarayanan, G. Drozdov, and A. Rabi-
novich. Estimating depth from RGB and sparse sens-
ing. In ECCV, pages 167–182, 2018.
[5] T. Dekel, C. Gan, D. Krishnan, C. Liu, and W. T. Free-
man. Smart, sparse contours to represent and edit im-
ages. In CVPR, 2018.
[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and
L. Fei-Fei. ImageNet: A large-scale hierarchical im-
age database. In CVPR, pages 248–255, 2009.
[7] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial
feature learning. In ICLR, 2017.
[8] A. Dosovitskiy and T. Brox. Generating images with
perceptual similarity metrics based on deep networks.
In Advances in Neural Information Processing Sys-
tems, pages 658–666, 2016.
[9] A. Dosovitskiy and T. Brox. Inverting visual represen-
tations with convolutional networks. In CVPR, pages
4829–4837, 2016.
[10] H. Edwards and A. Storkey. Censoring representations
with an adversary. In ICLR, 2016.
[11] D. Eigen, C. Puhrsch, and R. Fergus. Depth map pre-
diction from a single image using a multi-scale deep
network. In Advances in neural information process-
ing systems, pages 2366–2374, 2014.
[12] J. Flynn, I. Neulander, J. Philbin, and N. Snavely.
Deepstereo: Learning to predict new views from the
world’s imagery. In CVPR, pages 5515–5524, 2016.
[13] J. Hamm. Minimax filter: learning to preserve pri-
vacy from inference attacks. The Journal of Machine
Learning Research, 18(1):4704–4734, 2017.
[14] P. Hedman, J. Philip, T. Price, J.-M. Frahm, G. Dret-
takis, and G. Brostow. Deep blending for free-
viewpoint image-based rendering. ACM Transactions
on Graphics (SIGGRAPH Asia Conference Proceed-
ings), 37(6), November 2018.
[15] Hololens. https://www.microsoft.com/
en-us/hololens, 2016.
[16] J. Hong. Considering privacy issues in the context of
google glass. Commun. ACM, 56(11):10–11, 2013.
[17] T.-W. Hui, C. C. Loy, and X. Tang. Depth map super-
resolution by deep multi-scale guidance. In ECCV,
2016.
[18] A. Irschara, C. Zach, J.-M. Frahm, and H. Bischof.
From structure-from-motion point clouds to fast loca-
tion recognition. In CVPR, pages 2599–2606, 2009.
[19] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-
to-image translation with conditional adversarial net-
works. In CVPR, pages 1125–1134, 2017.
[20] H. Kato and T. Harada. Image reconstruction from
bag-of-visual-words. In CVPR, pages 955–962, 2014.
[21] P. Labatut, J.-P. Pons, and R. Keriven. Efficient multi-
view reconstruction of large-scale scenes using inter-
est points, Delaunay triangulation and graph cuts. In
ICCV, pages 1–8, 2007.
[22] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cun-
ningham, A. Acosta, A. Aitken, A. Tejani, J. Totz,
Z. Wang, et al. Photo-realistic single image super-
resolution using a generative adversarial network. In
CVPR, pages 4681–4690, 2017.
[23] Y. Li, N. Snavely, D. Huttenlocher, and P. Fua. World-
wide pose estimation using 3d point clouds. In ECCV,
pages 15–29. Springer, 2012.
[24] Z. Li and N. Snavely. Megadepth: Learning single-
view depth prediction from internet photos. In Com-
puter Vision and Pattern Recognition (CVPR), 2018.
[25] H. Lim, S. N. Sinha, M. F. Cohen, M. Uyttendaele, and
H. J. Kim. Real-time monocular image-based 6-dof
localization. The International Journal of Robotics
Research, 34(4-5):476–492, 2015.
[26] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised
image-to-image translation networks. In Advances in
Neural Information Processing Systems, pages 700–
708, 2017.
[27] M.-Y. Liu and O. Tuzel. Coupled generative adver-
sarial networks. In Advances in neural information
processing systems, pages 469–477, 2016.
[28] J. Lu and D. Forsyth. Sparse depth super resolution.
In CVPR, pages 2245–2253, 2015.
[29] A. Mahendran and A. Vedaldi. Understanding deep
image representations by inverting them. In CVPR,
pages 5188–5196, 2015.
[30] R. McPherson, R. Shokri, and V. Shmatikov. De-
feating image obfuscation with deep learning. arXiv
preprint arXiv:1609.00408, 2016.
[31] M. Moukari, S. Picard, L. Simoni, and F. Jurie. Deep
multi-scale architectures for monocular depth estima-
tion. In ICIP, pages 2940–2944, 2018.
[32] A. Odena, V. Dumoulin, and C. Olah. Deconvolution
and checkerboard artifacts. Distill, 2016.
[33] F. Pittaluga, S. Koppal, and A. Chakrabarti. Learn-
ing privacy preserving encodings through adversarial
 training. In 2019 IEEE Winter Conference on Appli- [49] M. D. Zeiler and R. Fergus. Visualizing and under-
cations of Computer Vision (WACV), pages 791–799. standing convolutional networks. In ECCV, pages
IEEE, 2019. 818–833. Springer, 2014.
[34] X. Qi, Q. Chen, J. Jia, and V. Koltun. Semi-parametric [50] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired
image synthesis. In CVPR, pages 8808–8816, 2018. image-to-image translation using cycle-consistent ad-
[35] N. Raval, A. Machanavajjhala, and L. P. Cox. Protect- versarial networks. In CVPR, pages 2223–2232,
ing visual secrets using adversarial nets. In CV-COPS 2017.
2017, CVPR Workshop, pages 1329–1332, 2017.
[36] G. Riegler, M. Rüther, and H. Bischof. ATGV-Net:
Accurate depth super-resolution. In ECCV, pages
268–284, 2016.
[37] T. Sattler, B. Leibe, and L. Kobbelt. Fast image-based
localization using direct 2d-to-3d matching. In ICCV,
pages 667–674. IEEE, 2011.
[38] J. L. Schönberger and J.-M. Frahm. Structure-from-
motion revisited. In CVPR, pages 4104–4113, 2016.
[39] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. In-
door segmentation and support inference from rgbd
images. In ECCV, 2012.
[40] K. Simonyan and A. Zisserman. Very deep convolu-
tional networks for large-scale image recognition. In
ICLR, 2015.
[41] P. Speciale, J. L. Schönberger, S. B. Kang, S. N. Sinha,
and M. Pollefeys. Privacy preserving image-based lo-
calization. arXiv preprint arXiv:1903.05572, 2019.
[42] D. Sungatullina, E. Zakharov, D. Ulyanov, and
V. Lempitsky. Image manipulation with perceptual
discriminators. In ECCV, pages 579–595, 2018.
[43] J. Uhrig, N. Schneider, L. Schneider, U. Franke,
T. Brox, and A. Geiger. Sparsity invariant CNNs. In
International Conference on 3D Vision (3DV), pages
11–20, 2017.
[44] I. Vasiljevic, A. Chakrabarti, and G. Shakhnarovich.
Examining the impact of blur on recognition
by convolutional networks. arXiv preprint
arXiv:1611.05760, 2016.
[45] C. Vondrick, A. Khosla, T. Malisiewicz, and A. Tor-
ralba. Hoggles: Visualizing object detection features.
In CVPR, pages 1–8, 2013.
[46] P. Weinzaepfel, H. Jégou, and P. Pérez. Reconstructing
an image from its local descriptors. In CVPR, pages
337–344, 2011.
[47] L. Xu, J. S. Ren, C. Liu, and J. Jia. Deep convo-
lutional neural network for image deconvolution. In
Advances in Neural Information Processing Systems,
pages 1790–1798, 2014.
[48] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lip-
son. Understanding neural networks through deep vi-
sualization. In ICML Workshop on Deep Learning,
2015.
 Supplementary Material:
Revealing Scenes by Inverting Structure from Motion Reconstructions

Francesco Pittaluga1 Sanjeev J. Koppal1 Sing Bing Kang2 Sudipta N. Sinha2

1 2
University of Florida Microsoft Research

A. Implementation Details of size 3 × 3 and stride equal to 1 followed by a 2 × 2

max pooling operation followed by an addition of a bias,
In the supplementary material we describe our network
batch normalization, and a leaky ReLU operation. FCN
architecture and the training procedure in more details.
denotes a fully connected layer with N nodes followed
A.1. Architecture by an addition of a bias, and a leaky ReLU operation. In
the final layer, the leaky ReLU is replaced by a softmax
Our network architecture consists of three sub-networks function.
– V ISIB N ET, C OARSE N ET and R EFINE N ET. The input to
our network is an H × W × n dimensional feature map A.2. Optimization
where at each 2D location in the feature map where there is
We used the Adam optimizer with β1 = 0.9, β2 = 0.999,
a valid sample, we have one n-dimensional feature vector.
 = 1e−8 and a learning rate of 0.0001 for training all
This feature vector is obtained by concatenating different
networks. Images with resolution 256 × 256 pixels were
subsets of depth, color, and SIFT features which are associ-
used as input to the network during training. However, the
ated with each 3D point in the SfM point cloud. Except for
trained network was used to process images at a resolution
the number of input/output channels in the first/final layers,
of 512 × 512 pixels. During training, we resized each im-
each sub-network has the same architecture, that of a U-Net
age such that the smaller dimension of the resized image
with an encoder and a decoder and with skip connections
was randomly assigned to either 296, 394, or 512 pixels, af-
between the layers in the encoder and decoder networks at
ter which we applied a random 2D cropping to the resized
identical depths. In contrast to conventional U-Nets, where
image to obtain a 256 × 256 image which was the actual
the decoder directly generates the output, in our network,
input to our network. We used Xavier initialization for all
the output of the decoder is passed through three convolu-
the parameters of our network.
tional layers in sequence to obtain the final output.
More, specifically, the architecture of the encoder is
CE256 - CE256 - CE256 - CE512 - CE512 - CE512, where CEN de-
B. Additional Results
notes a convolutional layer with N kernels of size 4 × 4 and We now present qualitative results to show that our net-
stride equal to 2 followed by an addition of a bias, batch work is robust to 2D input which is very sparse. Figure A1
normalization, and a ReLU operation. shows two example results. Three images are synthesized
The architecture of the decoder is CD512 - CD512 - CD512 - on randomly selected 20%, 60% and 100% of all the pro-
CD256 - CD256 - CD256 - C128 - C64 - C32- C3, where CDN de- jected 3D points for the scenes. Despite the high simulated
notes nearest neighbor upsampling by a factor of 2 followed 2D sparsity in the input, the output images are quite inter-
by a convolutional layer with N kernels of size 3 × 3 and pretable. Figure A2 shows some failure examples.
a stride equal to 1, followed by an addition of a bias, batch
Supplementary Video. Finally, we encourage the reader
normalization, and a ReLU operation. CN layers are the
to view the supplementary video which makes it easier to
same as CDN layers but without the upsampling operation.
visualize the qualitative results shown in the main paper.
In the final layer of the decoder, the ReLU is replaced with
For two scenes, where the SfM camera poses are available,
a tanh non-linearity. In R EFINE N ET, all ReLU operations
we show that we can reconstruct the source video by run-
are replaced by leaky ReLU operations in all the layers of
ning our method on a frame by frame basis with the camera
the decoder network. In V ISIB N ET, the final layer of the
poses for the source images. Finally, we show results of
decoder has 1 kernel instead of 3.
synthesizing images from novel camera viewpoints. Such
Our discriminator used for adversarial training of
results can be used to create virtual tours of the scene, thus
R EFINE N ET has the following architecture – CA256 - CA256 -
making it easier to reveal and the appearance, layout and
CA256 - CA512 - CA512 - CA512 - FC1024 - FC1024 - FC1024 - FC2,
geometry of the scene.
where CAN denotes a convolutional layer with N kernels
 (a) Sparse Input (20%) (b) Using 20% (c) Using 60% (d) Using all (e) Original Images

Figure A1: E VALUATING ROBUSTNESS TO S PARSITY: Two sets of images synthesized using our complete pipeline, by
running V ISIB N ET, C OARSE N ET and R EFINE N ET. From left to right: (a) Simulated sparse inputs to our networks. Here,
only 20% of the 3D points in the respective SfM models were used. Image synthesized using our method using (b) 20% of
the points, (c) 60% of the points, (d) all the points and (e) the original source images. Even when the inputs are extremely
sparse, most of the contents of the synthesized images can be easily recognized.

(a) (b) (c) (d) (e) (f) (g) (h)

Figure A2: FAILURE E XAMPLES : (a) Dense points on the building in the background overwhelms a few sparse points in the
foreground on the base of the statue. V ISIB N ET in this case incorrectly predicts that the building is visible and this causes the
base of the statue to disappear completely in the synthesized image. (b) A similar artifact for a different scene. (c) Parallel
straight lines are sometimes poorly handled, such as the lines on the vertical pillars of the monument. (d) The complex
occlusions in the architectural structure produce artifacts where the occluded surfaces and the occluders are fused into each
other. (e) Straight lines are often reconstructed as curved or bent (f–g) Low sample density in the input common in indoor
scenes results in blurry and wavy edges. (h) Finally, spurious 3D points may cause our method to hallucinate structures such
as the dark line on the wall which is not actually there.
 Semantic Image Synthesis with Spatially-Adaptive Normalization

Taesung Park1,2∗ Ming-Yu Liu2 Ting-Chun Wang2 Jun-Yan Zhu2,3

1 2 2,3
UC Berkeley NVIDIA MIT CSAIL

cloud sky
arXiv:1903.07291v2 [cs.CV] 5 Nov 2019

tree mountain

sea grass

Figure 1: Our model allows user control over both semantic and style as synthesizing an image. The semantic (e.g., the
existence of a tree) is controlled via a label map (the top row), while the style is controlled via the reference style image (the
leftmost column). Please visit our website for interactive image synthesis demos.

Abstract https://github.com/NVlabs/SPADE.

We propose spatially-adaptive normalization, a simple 1. Introduction

but effective layer for synthesizing photorealistic images
given an input semantic layout. Previous methods directly
feed the semantic layout as input to the deep network, which
is then processed through stacks of convolution, normaliza-
tion, and nonlinearity layers. We show that this is subop-
timal as the normalization layers tend to “wash away” se-
mantic information. To address the issue, we propose using
the input layout for modulating the activations in normal-
ization layers through a spatially-adaptive, learned trans-
formation. Experiments on several challenging datasets
demonstrate the advantage of the proposed method over ex-
isting approaches, regarding both visual fidelity and align-
ment with input layouts. Finally, our model allows user
control over both semantic and style. Code is available at
∗Taesung Park contributed to the work during his NVIDIA internship.
Conditional image synthesis refers to the task of gen-
erating photorealistic images conditioning on certain in-
put data. Seminal work computes the output image by
stitching pieces from a single image (e.g., Image Analo-
gies [16]) or using an image collection [7, 14, 23, 30, 35].
Recent methods directly learn the mapping using neural net-
works [3, 6, 22, 47, 48, 54, 55, 56]. The latter methods are
faster and require no external database of images.
We are interested in a specific form of conditional im-
age synthesis, which is converting a semantic segmentation
mask to a photorealistic image. This form has a wide range
of applications such as content generation and image edit-
ing [6, 22, 48]. We refer to this form as semantic image
synthesis. In this paper, we show that the conventional net-
work architecture [22, 48], which is built by stacking con-
volutional, normalization, and nonlinearity layers, is at best

1
suboptimal because their normalization layers tend to “wash
away” information contained in the input semantic masks.
To address the issue, we propose spatially-adaptive normal-
ization, a conditional normalization layer that modulates the
activations using input semantic layouts through a spatially-
adaptive, learned transformation and can effectively propa-
gate the semantic information throughout the network.
We conduct experiments on several challenging datasets
including the COCO-Stuff [4, 32], the ADE20K [58], and
the Cityscapes [9]. We show that with the help of our
spatially-adaptive normalization layer, a compact network
can synthesize significantly better results compared to sev-
eral state-of-the-art methods. Additionally, an extensive ab-
lation study demonstrates the effectiveness of the proposed
normalization layer against several variants for the semantic
image synthesis task. Finally, our method supports multi-
modal and style-guided image synthesis, enabling control-
lable, diverse outputs, as shown in Figure 1. Also, please
see our SIGGRAPH 2019 Real-Time Live demo and try our
online demo by yourself.
2. Related Work
Deep generative models can learn to synthesize images.
Recent methods include generative adversarial networks
(GANs) [13] and variational autoencoder (VAE) [28]. Our
work is built on GANs but aims for the conditional image
synthesis task. The GANs consist of a generator and a dis-
criminator where the goal of the generator is to produce re-
alistic images so that the discriminator cannot tell the syn-
thesized images apart from the real ones.
Conditional image synthesis exists in many forms that dif-
fer in the type of input data. For example, class-conditional
models [3, 36, 37, 39, 41] learn to synthesize images given
category labels. Researchers have explored various models
for generating images based on text [18,44,52,55]. Another
widely-used form is image-to-image translation based on a
type of conditional GANs [20, 22, 24, 25, 33, 57, 59, 60],
where both input and output are images. Compared to
earlier non-parametric methods [7, 16, 23], learning-based
methods typically run faster during test time and produce
more realistic results. In this work, we focus on converting
segmentation masks to photorealistic images. We assume
the training dataset contains registered segmentation masks
and images. With the proposed spatially-adaptive normal-
ization, our compact network achieves better results com-
pared to leading methods.
Unconditional normalization layers have been an impor-
tant component in modern deep networks and can be found
in various classifiers, including the Local Response Nor-
malization in the AlexNet [29] and the Batch Normaliza-
tion (BatchNorm) in the Inception-v2 network [21]. Other
popular normalization layers include the Instance Normal-
2
ization (InstanceNorm) [46], the Layer Normalization [2],
the Group Normalization [50], and the Weight Normaliza-
tion [45]. We label these normalization layers as uncondi-
tional as they do not depend on external data in contrast to
the conditional normalization layers discussed below.
Conditional normalization layers include the Conditional
Batch Normalization (Conditional BatchNorm) [11] and
Adaptive Instance Normalization (AdaIN) [19]. Both were
first used in the style transfer task and later adopted in var-
ious vision tasks [3, 8, 10, 20, 26, 36, 39, 42, 49, 54]. Dif-
ferent from the earlier normalization techniques, condi-
tional normalization layers require external data and gen-
erally operate as follows. First, layer activations are nor-
malized to zero mean and unit deviation. Then the nor-
malized activations are denormalized by modulating the
activation using a learned affine transformation whose pa-
rameters are inferred from external data. For style trans-
fer tasks [11, 19], the affine parameters are used to control
the global style of the output, and hence are uniform across
spatial coordinates. In contrast, our proposed normalization
layer applies a spatially-varying affine transformation, mak-
ing it suitable for image synthesis from semantic masks.
Wang et al. proposed a closely related method for image
super-resolution [49]. Both methods are built on spatially-
adaptive modulation layers that condition on semantic in-
puts. While they aim to incorporate semantic information
into super-resolution, our goal is to design a generator for
style and semantics disentanglement. We focus on provid-
ing the semantic information in the context of modulating
normalized activations. We use semantic maps in different
scales, which enables coarse-to-fine generation. The reader
is encouraged to review their work for more details.
3. Semantic Image Synthesis
Let m ∈ LH×W be a semantic segmentation mask
where L is a set of integers denoting the semantic labels,
and H and W are the image height and width. Each entry
in m denotes the semantic label of a pixel. We aim to learn
a mapping function that can convert an input segmentation
mask m to a photorealistic image.
Spatially-adaptive denormalization. Let hi denote the ac-
tivations of the i-th layer of a deep convolutional network
for a batch of N samples. Let C i be the number of chan-
nels in the layer. Let H i and W i be the height and width
of the activation map in the layer. We propose a new condi-
tional normalization method called the SPatially-Adaptive
(DE)normalization1 (SPADE). Similar to the Batch Nor-
malization [21], the activation is normalized in the channel-
wise manner and then modulated with learned scale and
bias. Figure 2 illustrates the SPADE design. The activation
1 Conditional normalization [11, 19] uses external data to denormalize
the normalized activations; i.e., the denormalization part is conditional.
 conv
conv
𝛽

Batch
element-wise
Norm

Figure 2: In the SPADE, the mask is first projected onto an Figure 3: Comparing results given uniform segmentation
embedding space and then convolved to produce the modu- maps: while the SPADE generator produces plausible tex-
lation parameters γ and β. Unlike prior conditional normal- tures, the pix2pixHD generator [48] produces two identical
ization methods, γ and β are not vectors, but tensors with outputs due to the loss of the semantic information after the
spatial dimensions. The produced γ and β are multiplied normalization layer.
and added to the normalized activation element-wise.
SPADE generator. With the SPADE, there is no need to
value at site (n ∈ N, c ∈ C i , y ∈ H i , x ∈ W i ) is feed the segmentation map to the first layer of the genera-
tor, since the learned modulation parameters have encoded
i
hin,c,y,x − µic i enough information about the label layout. Therefore, we
γc,y,x (m) + βc,y,x (m) (1)
σci discard encoder part of the generator, which is commonly
used in recent architectures [22, 48]. This simplification re-
where hin,c,y,x is the activation at the site before normaliza-
sults in a more lightweight network. Furthermore, similarly
tion and µic and σci are the mean and standard deviation of
to existing class-conditional generators [36,39,54], the new
the activations in channel c:
generator can take a random vector as input, enabling a sim-
1 X
ple and natural way for multi-modal synthesis [20, 60].
µic = hi (2)
N H W n,y,x n,c,y,x
i i
Figure 4 illustrates our generator architecture, which em-
s ploys several ResNet blocks [15] with upsampling layers.
i 1 X
i

2 − (µi )2 .
σc = (h n,c,y,x ) c (3) The modulation parameters of all the normalization layers
N H i W i n,y,x are learned using the SPADE. Since each residual block
i i
operates at a different scale, we downsample the semantic
The variables γc,y,x (m) and βc,y,x (m) in (1) are the mask to match the spatial resolution.
learned modulation parameters of the normalization layer.
We train the generator with the same multi-scale discrim-
In contrast to the BatchNorm [21], they depend on the in-
inator and loss function used in pix2pixHD [48] except that
put segmentation mask and vary with respect to the location
i i we replace the least squared loss term [34] with the hinge
(y, x). We use the symbol γc,y,x and βc,y,x to denote the
loss term [31,38,54]. We test several ResNet-based discrim-
functions that convert m to the scaling and bias values at
inators used in recent unconditional GANs [1, 36, 39] but
the site (c, y, x) in the i-th activation map. We implement
i i observe similar results at the cost of a higher GPU mem-
the functions γc,y,x and βc,y,x using a simple two-layer con-
ory requirement. Adding the SPADE to the discriminator
volutional network, whose design is in the appendix.
also yields a similar performance. For the loss function, we
In fact, SPADE is related to, and is a generalization
observe that removing any loss term in the pix2pixHD loss
of several existing normalization layers. First, replacing
function lead to degraded generation results.
the segmentation mask m with the image class label and
making the modulation parameters spatially-invariant (i.e., Why does the SPADE work better? A short answer is that
i i i i it can better preserve semantic information against common
γc,y 1 ,x1
≡ γc,y 2 ,x2
and βc,y 1 ,x1
≡ βc,y 2 ,x2
for any y1 , y2 ∈
{1, 2, ..., H } and x1 , x2 ∈ {1, 2, ..., W i }), we arrive at the
i normalization layers. Specifically, while normalization lay-
form of the Conditional BatchNorm [11]. Indeed, for any ers such as the InstanceNorm [46] are essential pieces in
spatially-invariant conditional data, our method reduces to almost all the state-of-the-art conditional image synthesis
the Conditional BatchNorm. Similarly, we can arrive at models [48], they tend to wash away semantic information
the AdaIN [19] by replacing m with a real image, mak- when applied to uniform or flat segmentation masks.
ing the modulation parameters spatially-invariant, and set- Let us consider a simple module that first applies con-
ting N = 1. As the modulation parameters are adaptive to volution to a segmentation mask and then normalization.
the input segmentation mask, the proposed SPADE is better Furthermore, let us assume that a segmentation mask with
suited for semantic image synthesis. a single label is given as input to the module (e.g., all the

3
 pix2pixHD
3x3 Conv
3x3 Conv
SPADE
SPADE
ReLU
ReLU
SPADE
~ ResBlk
SPADE ResBlk
Figure 4: In the SPADE generator, each normalization layer uses the segmentation mask to modulate the layer activations.
(left) Structure of one residual block with the SPADE. (right) The generator contains a series of the SPADE residual blocks
with upsampling layers. Our architecture achieves better performance with a smaller number of parameters by removing the
downsampling layers of leading image-to-image translation networks such as the pix2pixHD model [48].
pixels have the same label such as sky or grass). Under this
setting, the convolution outputs are again uniform, with dif-
ferent labels having different uniform values. Now, after we
apply InstanceNorm to the output, the normalized activation
will become all zeros no matter what the input semantic la-
bel is given. Therefore, semantic information is totally lost.
This limitation applies to a wide range of generator archi-
tectures, including pix2pixHD and its variant that concate-
nates the semantic mask at all intermediate layers, as long
as a network applies convolution and then normalization to
the semantic mask. In Figure 3, we empirically show this is
precisely the case for pix2pixHD. Because a segmentation
mask consists of a few uniform regions in general, the issue
of information loss emerges when applying normalization.
In contrast, the segmentation mask in the SPADE Gen-
erator is fed through spatially adaptive modulation without
normalization. Only activations from the previous layer are
normalized. Hence, the SPADE generator can better pre-
serve semantic information. It enjoys the benefit of normal-
ization without losing the semantic input information.
Multi-modal synthesis. By using a random vector as the
input of the generator, our architecture provides a simple
way for multi-modal synthesis [20, 60]. Namely, one can
attach an encoder that processes a real image into a random
vector, which will be then fed to the generator. The encoder
and generator form a VAE [28], in which the encoder tries
to capture the style of the image, while the generator com-
bines the encoded style and the segmentation mask informa-
tion via the SPADEs to reconstruct the original image. The
encoder also serves as a style guidance network at test time
to capture the style of target images, as used in Figure 1.
For training, we add a KL-Divergence loss term [28].
4. Experiments
Implementation details. We apply the Spectral Norm [38]
to all the layers in both generator and discriminator. The
4
SPADE SPADE SPADE
ResBlk ResBlk ResBlk
learning rates for the generator and discriminator are
0.0001 and 0.0004, respectively [17]. We use the ADAM
solver [27] with β1 = 0 and β2 = 0.999. All the exper-
iments are conducted on an NVIDIA DGX1 with 8 32GB
V100 GPUs. We use synchronized BatchNorm, i.e., these
statistics are collected from all the GPUs.
Datasets. We conduct experiments on several datasets.
• COCO-Stuff [4] is derived from the COCO dataset [32].
It has 118, 000 training images and 5, 000 validation im-
ages captured from diverse scenes. It has 182 semantic
classes. Due to its vast diversity, existing image synthe-
sis models perform poorly on this dataset.
• ADE20K [58] consists of 20, 210 training and 2, 000 val-
idation images. Similarly to the COCO, the dataset con-
tains challenging scenes with 150 semantic classes.
• ADE20K-outdoor is a subset of the ADE20K dataset that
only contains outdoor scenes, used in Qi et al. [43].
• Cityscapes dataset [9] contains street scene images in
German cities. The training and validation set sizes are
3, 000 and 500, respectively. Recent work has achieved
photorealistic semantic image synthesis results [43, 47]
on the Cityscapes dataset.
• Flickr Landscapes. We collect 41, 000 photos from
Flickr and use 1, 000 samples for the validation set. To
avoid expensive manual annotation, we use a well-trained
DeepLabV2 [5] to compute input segmentation masks.
We train the competing semantic image synthesis methods
on the same training set and report their results on the same
validation set for each dataset.
Performance metrics. We adopt the evaluation protocol
from previous work [6, 48]. Specifically, we run a seman-
tic segmentation model on the synthesized images and com-
pare how well the predicted segmentation mask matches the
ground truth input. Intuitively, if the output images are re-
alistic, a well-trained semantic segmentation model should
be able to predict the ground truth label. For measuring the
segmentation accuracy, we use both the mean Intersection-
 Label Ground Truth CRN [6] pix2pixHD [48] Ours

Figure 5: Visual comparison of semantic image synthesis results on the COCO-Stuff dataset. Our method successfully
synthesizes realistic details from semantic labels.
Label Ground Truth CRN [6] SIMS [43] pix2pixHD [48] Ours

Figure 6: Visual comparison of semantic image synthesis results on the ADE20K outdoor and Cityscapes datasets. Our
method produces realistic images while respecting the spatial semantic layout at the same time.
COCO-Stuff ADE20K ADE20K-outdoor Cityscapes
Method mIoU accu FID mIoU accu FID mIoU accu FID mIoU accu FID
CRN [6] 23.7 40.4 70.4 22.4 68.8 73.3 16.5 68.6 99.0 52.4 77.1 104.7
SIMS [43] N/A N/A N/A N/A N/A N/A 13.1 74.7 67.7 47.2 75.5 49.7
pix2pixHD [48] 14.6 45.8 111.5 20.3 69.2 81.8 17.4 71.6 97.8 58.3 81.4 95.0
Ours 37.4 67.9 22.6 38.5 79.9 33.9 30.8 82.9 63.3 62.3 81.9 71.8
Table 1: Our method outperforms the current leading methods in semantic segmentation (mIoU and accu) and FID [17]
scores on all the benchmark datasets. For the mIoU and accu, higher is better. For the FID, lower is better.

over-Union (mIoU) and the pixel accuracy (accu). We use
the state-of-the-art segmentation networks for each dataset:
DeepLabV2 [5, 40] for COCO-Stuff, UperNet101 [51] for
ADE20K, and DRN-D-105 [53] for Cityscapes. In addi-
tion to the mIoU and the accu segmentation performance
metrics, we use the Fréchet Inception Distance (FID) [17]
to measure the distance between the distribution of synthe-
sized results and the distribution of real images.
Baselines. We compare our method with 3 leading seman-
tic image synthesis models: the pix2pixHD model [48],
the cascaded refinement network (CRN) [6], and the semi-
parametric image synthesis method (SIMS) [43]. The
pix2pixHD is the current state-of-the-art GAN-based con-
ditional image synthesis framework. The CRN uses a deep
network that repeatedly refines the output from low to high
resolution, while the SIMS takes a semi-parametric ap-
proach that composites real segments from a training set and
refines the boundaries. Both the CRN and SIMS are mainly
trained using image reconstruction loss. For a fair compar-
ison, we train the CRN and pix2pixHD models using the
implementations provided by the authors. As image syn-
thesis using the SIMS requires many queries to the training

5
Figure 7: Semantic image synthesis results on the Flickr Landscapes dataset. The images were generated from semantic
layout of photographs on the Flickr website.

dataset, it is computationally prohibitive for a large dataset Ours vs. Ours vs. Ours vs.
Dataset
such as the COCO-stuff and the full ADE20K. Therefore, CRN pix2pixHD SIMS
we use the results provided by the authors when available. COCO-Stuff 79.76 86.64 N/A
ADE20K 76.66 83.74 N/A
Quantitative comparisons. As shown in Table 1, our ADE20K-outdoor 66.04 79.34 85.70
method outperforms the current state-of-the-art methods by Cityscapes 63.60 53.64 51.52
a large margin in all the datasets. For the COCO-Stuff, our Table 2: User preference study. The numbers indicate the
method achieves an mIoU score of 35.2, which is about 1.5 percentage of users who favor the results of the proposed
times better than the previous leading method. Our FID method over those of the competing method.
is also 2.2 times better than the previous leading method.
We note that the SIMS model produces a lower FID score
but has poor segmentation performances on the Cityscapes In Figures 7 and 8, we show more example results from
dataset. This is because the SIMS synthesizes an image by the Flickr Landscape and COCO-Stuff datasets. The pro-
first stitching image patches from the training dataset. As posed method can generate diverse scenes with high image
using the real image patches, the resulting image distribu- fidelity. More results are included in the appendix.
tion can better match the distribution of real images. How- Human evaluation. We use the Amazon Mechanical Turk
ever, because there is no guarantee that a perfect query (e.g., (AMT) to compare the perceived visual fidelity of our
a person in a particular pose) exists in the dataset, it tends method against existing approaches. Specifically, we give
to copy objects that do not match the input segments. the AMT workers an input segmentation mask and two
Qualitative results. In Figures 5 and 6, we provide quali- synthesis outputs from different methods and ask them to
tative comparisons of the competing methods. We find that choose the output image that looks more like a correspond-
our method produces results with much better visual quality ing image of the segmentation mask. The workers are given
and fewer visible artifacts, especially for diverse scenes in unlimited time to make the selection. For each comparison,
the COCO-Stuff and ADE20K dataset. When the training we randomly generate 500 questions for each dataset, and
dataset size is small, the SIMS model also renders images each question is answered by 5 different workers. For qual-
with good visual quality. However, the depicted content ity control, only workers with a lifetime task approval rate
often deviates from the input segmentation mask (e.g., the greater than 98% can participate in our study.
shape of the swimming pool in the second row of Figure 6). Table 2 shows the evaluation results. We find that users

6
Figure 8: Semantic image synthesis results on COCO-Stuff. Our method successfully generates realistic images in diverse
scenes ranging from animals to sports activities.
Method #param COCO. ADE. City. Method COCO ADE20K Cityscapes
decoder w/ SPADE (Ours) 96M 35.2 38.5 62.3 segmap input 35.2 38.5 62.3
compact decoder w/ SPADE 61M 35.2 38.0 62.5 random input 35.3 38.3 61.6
decoder w/ Concat 79M 31.9 33.6 61.1 kernelsize 5x5 35.0 39.3 61.8
pix2pixHD++ w/ SPADE 237M 34.4 39.0 62.2 kernelsize 3x3 35.2 38.5 62.3
pix2pixHD++ w/ Concat 195M 32.9 38.9 57.1 kernelsize 1x1 32.7 35.9 59.9
pix2pixHD++ 183M 32.7 38.3 58.8 #params 141M 35.3 38.3 62.5
compact pix2pixHD++ 103M 31.6 37.3 57.6 #params 96M 35.2 38.5 62.3
pix2pixHD [48] 183M 14.6 20.3 58.3 #params 61M 35.2 38.0 62.5
Table 3: The mIoU scores are boosted when the SPADE Sync BatchNorm 35.0 39.3 61.8
BatchNorm 33.7 37.9 61.8
is used, for both the decoder architecture (Figure 4) and
InstanceNorm 33.9 37.4 58.7
encoder-decoder architecture of pix2pixHD++ (our im-
proved baseline over pix2pixHD [48]). On the other hand, Table 4: The SPADE generator works with different con-
simply concatenating semantic input at every layer fails to figurations. We change the input of the generator, the con-
do so. Moreover, our compact model with smaller depth at volutional kernel size acting on the segmentation map, the
all layers outperforms all the baselines. capacity of the network, and the parameter-free normaliza-
tion method. The settings used in the paper are boldfaced.

strongly favor our results on all the datasets, especially on

the challenging COCO-Stuff and ADE20K datasets. For the
Cityscapes, even when all the competing methods achieve
high image fidelity, users still prefer our results.
Effectiveness of the SPADE. For quantifying importance
of the SPADE, we introduce a strong baseline called
pix2pixHD++, which combines all the techniques we find
useful for enhancing the performance of pix2pixHD except
the SPADE. We also train models that receive the segmen-
tation mask input at all the intermediate layers via feature
concatenation in the channel direction, which is termed as
pix2pixHD++ w/ Concat. Finally, the model that com-
bines the strong baseline with the SPADE is denoted as
pix2pixHD++ w/ SPADE.
As shown in Table 3, the architectures with the proposed
SPADE consistently outperforms its counterparts, in both
the decoder-style architecture described in Figure 4 and
more traditional encoder-decoder architecture used in the
pix2pixHD. We also find that concatenating segmentation
masks at all intermediate layers, a reasonable alternative
to the SPADE, does not achieve the same performance as
SPADE. Furthermore, the decoder-style SPADE generator
works better than the strong baselines even with a smaller
number of parameters.

7
Figure 9: Our model attains multimodal synthesis capability when trained with the image encoder. During deployment,
by using different random noise, our model synthesizes outputs with diverse appearances but all having the same semantic
layouts depicted in the input mask. For reference, the ground truth image is shown inside the input segmentation mask.

Variations of SPADE generator. Table 4 reports the per-
formance of several variations of our generator. First, we
compare two types of input to the generator where one is the
random noise while the other is the downsampled segmen-
tation map. We find that both of the variants render similar
performance and conclude that the modulation by SPADE
alone provides sufficient signal about the input mask. Sec-
ond, we vary the type of parameter-free normalization lay-
ers before applying the modulation parameters. We observe
that the SPADE works reliably across different normaliza-
tion methods. Next, we vary the convolutional kernel size
acting on the label map, and find that kernel size of 1x1
hurts performance, likely because it prohibits utilizing the
context of the label. Lastly, we modify the capacity of the
generator by changing the number of convolutional filters.
We present more variations and ablations in the appendix.
Multi-modal synthesis. In Figure 9, we show the mul-
timodal image synthesis results on the Flickr Landscape
dataset. For the same input segmentation mask, we sam-
ple different noise inputs to achieve different outputs. More
results are included in the appendix.
Semantic manipulation and guided image synthesis. In
Figure 1, we show an application where a user draws dif-
ferent segmentation masks, and our model renders the cor-
responding landscape images. Moreover, our model allows
users to choose an external style image to control the global
appearances of the output image. We achieve it by replac-
ing the input noise with the embedding vector of the style
image computed by the image encoder.
5. Conclusion
We have proposed the spatially-adaptive normalization,
which utilizes the input semantic layout while performing
the affine transformation in the normalization layers. The
proposed normalization leads to the first semantic image
synthesis model that can produce photorealistic outputs for
diverse scenes including indoor, outdoor, landscape, and
street scenes. We further demonstrate its application for
multi-modal synthesis and guided image synthesis.
Acknowledgments. We thank Alexei A. Efros, Bryan
Catanzaro, Andrew Tao, and Jan Kautz for insightful ad-
vice. We thank Chris Hebert, Gavriil Klimov, and Brad
Nemire for their help in constructing the demo apps. Tae-
sung Park contributed to the work during his internship at
NVIDIA. His Ph.D. is supported by a Samsung Scholarship.

8
References
[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gen-
erative adversarial networks. In International Conference on
Machine Learning (ICML), 2017. 3
[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization.
arXiv preprint arXiv:1607.06450, 2016. 2
[3] A. Brock, J. Donahue, and K. Simonyan. Large scale gan
training for high fidelity natural image synthesis. In Inter-
national Conference on Learning Representations (ICLR),
2019. 1, 2
[4] H. Caesar, J. Uijlings, and V. Ferrari. Coco-stuff: Thing and
stuff classes in context. In IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2018. 2, 4
[5] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Deeplab: Semantic image segmentation with
deep convolutional nets, atrous convolution, and fully con-
nected crfs. IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence (TPAMI), 40(4):834–848, 2018. 4, 5
[6] Q. Chen and V. Koltun. Photographic image synthesis with
cascaded refinement networks. In IEEE International Con-
ference on Computer Vision (ICCV), 2017. 1, 4, 5, 13, 14,
15, 16, 17, 18
[7] T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu.
Sketch2photo: internet image montage. ACM Transactions
on Graphics (TOG), 28(5):124, 2009. 1, 2
[8] T. Chen, M. Lucic, N. Houlsby, and S. Gelly. On self mod-
ulation for generative adversarial networks. In International
Conference on Learning Representations, 2019. 2
[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
cityscapes dataset for semantic urban scene understanding.
In IEEE Conference on Computer Vision and Pattern Recog-
nition (CVPR), 2016. 2, 4
[10] H. De Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin,
and A. C. Courville. Modulating early visual processing
by language. In Advances in Neural Information Process-
ing Systems, 2017. 2
[11] V. Dumoulin, J. Shlens, and M. Kudlur. A learned repre-
sentation for artistic style. In International Conference on
Learning Representations (ICLR), 2016. 2, 3
[12] X. Glorot and Y. Bengio. Understanding the difficulty of
training deep feedforward neural networks. In Proceedings
of the thirteenth international conference on artificial intel-
ligence and statistics, pages 249–256, 2010. 12, 13
[13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Gen-
erative adversarial nets. In Advances in Neural Information
Processing Systems, 2014. 2
[14] J. Hays and A. A. Efros. Scene completion using millions of
photographs. In ACM SIGGRAPH, 2007. 1
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2016. 3
[16] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H.
Salesin. Image analogies. 2001. 1, 2
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and
S. Hochreiter. GANs trained by a two time-scale update rule
9
converge to a local Nash equilibrium. In Advances in Neural
Information Processing Systems, 2017. 4, 5, 13
[18] S. Hong, D. Yang, J. Choi, and H. Lee. Inferring seman-
tic layout for hierarchical text-to-image synthesis. In IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), 2018. 2
[19] X. Huang and S. Belongie. Arbitrary style transfer in real-
time with adaptive instance normalization. In IEEE Inter-
national Conference on Computer Vision (ICCV), 2017. 2,
3
[20] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal
unsupervised image-to-image translation. European Confer-
ence on Computer Vision (ECCV), 2018. 2, 3, 4
[21] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
In International Conference on Machine Learning (ICML),
2015. 2, 3
[22] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-
image translation with conditional adversarial networks. In
IEEE Conference on Computer Vision and Pattern Recogni-
tion (CVPR), 2017. 1, 2, 3, 11, 12
[23] M. Johnson, G. J. Brostow, J. Shotton, O. Arandjelovic,
V. Kwatra, and R. Cipolla. Semantic photo synthesis. In
Computer Graphics Forum, volume 25, pages 407–413,
2006. 1, 2
[24] L. Karacan, Z. Akata, A. Erdem, and E. Erdem. Learning
to generate images of outdoor scenes from attributes and se-
mantic layouts. arXiv preprint arXiv:1612.00215, 2016. 2
[25] L. Karacan, Z. Akata, A. Erdem, and E. Erdem. Manipu-
lating attributes of natural scenes via hallucination. arXiv
preprint arXiv:1808.07413, 2018. 2
[26] T. Karras, S. Laine, and T. Aila. A style-based generator
architecture for generative adversarial networks. In IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), 2019. 2
[27] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. In International Conference on Learning Rep-
resentations (ICLR), 2015. 4
[28] D. P. Kingma and M. Welling. Auto-encoding variational
bayes. In International Conference on Learning Representa-
tions (ICLR), 2014. 2, 4, 11, 12
[29] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in Neural Information Processing Systems, 2012.
2
[30] J.-F. Lalonde, D. Hoiem, A. A. Efros, C. Rother, J. Winn,
and A. Criminisi. Photo clip art. In ACM transactions on
graphics (TOG), volume 26, page 3. ACM, 2007. 1
[31] J. H. Lim and J. C. Ye. Geometric gan. arXiv preprint
arXiv:1705.02894, 2017. 3, 11
[32] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-
manan, P. Dollár, and C. L. Zitnick. Microsoft coco: Com-
mon objects in context. In European Conference on Com-
puter Vision (ECCV), 2014. 2, 4
[33] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-
image translation networks. In Advances in Neural Informa-
tion Processing Systems, 2017. 2
[34] X. Mao, Q. Li, H. Xie, Y. R. Lau, Z. Wang, and S. P. Smol-
ley. Least squares generative adversarial networks. In IEEE
International Conference on Computer Vision (ICCV), 2017.
3, 11
[35] T. B. Mathias Eitz, Kristian Hildebrand and M. Alexa. Pho-
tosketch: A sketch based image query and compositing sys-
tem. In ACM SIGGRAPH 2009 Talk Program, 2009. 1
[36] L. Mescheder, A. Geiger, and S. Nowozin. Which training
methods for gans do actually converge? In International
Conference on Machine Learning (ICML), 2018. 2, 3, 11
[37] M. Mirza and S. Osindero. Conditional generative adversar-
ial nets. arXiv preprint arXiv:1411.1784, 2014. 2
[38] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spec-
tral normalization for generative adversarial networks. In In-
ternational Conference on Learning Representations (ICLR),
2018. 3, 4, 11
[39] T. Miyato and M. Koyama. cGANs with projection discrim-
inator. In International Conference on Learning Representa-
tions (ICLR), 2018. 2, 3, 11
[40] K. Nakashima. Deeplab-pytorch. https://github.
com/kazuto1011/deeplab-pytorch, 2018. 5
[41] A. Odena, C. Olah, and J. Shlens. Conditional image synthe-
sis with auxiliary classifier GANs. In International Confer-
ence on Machine Learning (ICML), 2017. 2
[42] E. Perez, H. De Vries, F. Strub, V. Dumoulin, and
A. Courville. Learning visual reasoning without strong
priors. In International Conference on Machine Learning
(ICML), 2017. 2
[43] X. Qi, Q. Chen, J. Jia, and V. Koltun. Semi-parametric im-
age synthesis. In IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2018. 4, 5, 13, 17, 18
[44] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and
H. Lee. Generative adversarial text to image synthesis. In In-
ternational Conference on Machine Learning (ICML), 2016.
2 from layout. In IEEE Conference on Computer Vision and
[45] T. Salimans and D. P. Kingma. Weight normalization: A
simple reparameterization to accelerate training of deep neu-
ral networks. In Advances in Neural Information Processing
Systems, 2016. 2
[46] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance nor-
malization: The missing ingredient for fast stylization. arxiv
2016. arXiv preprint arXiv:1607.08022, 2016. 2, 3
[47] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz,
and B. Catanzaro. Video-to-video synthesis. In Advances in
Neural Information Processing Systems, 2018. 1, 4
[48] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and
B. Catanzaro. High-resolution image synthesis and semantic
manipulation with conditional gans. In IEEE Conference on
10
Computer Vision and Pattern Recognition (CVPR), 2018. 1,
3, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18
[49] X. Wang, K. Yu, C. Dong, and C. Change Loy. Recover-
ing realistic texture in image super-resolution by deep spatial
feature transform. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages 606–615,
2018. 2
[50] Y. Wu and K. He. Group normalization. In European Con-
ference on Computer Vision (ECCV), 2018. 2
[51] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified per-
ceptual parsing for scene understanding. In European Con-
ference on Computer Vision (ECCV), 2018. 5
[52] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and
X. He. Attngan: Fine-grained text to image generation with
attentional generative adversarial networks. In IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
2018. 2
[53] F. Yu, V. Koltun, and T. Funkhouser. Dilated residual net-
works. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2017. 5
[54] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-
attention generative adversarial networks. In International
Conference on Machine Learning (ICML), 2019. 1, 2, 3, 11
[55] H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and
D. Metaxas. Stackgan: Text to photo-realistic image synthe-
sis with stacked generative adversarial networks. In IEEE
International Conference on Computer Vision (ICCV), 2017.
1, 2
[56] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang,
and D. Metaxas. Stackgan++: Realistic image synthesis
with stacked generative adversarial networks. IEEE Transac-
tions on Pattern Analysis and Machine Intelligence (TPAMI),
2018. 1
[57] B. Zhao, L. Meng, W. Yin, and L. Sigal. Image generation
Pattern Recognition (CVPR), 2019. 2
[58] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and
A. Torralba. Scene parsing through ade20k dataset. In
IEEE Conference on Computer Vision and Pattern Recog-
nition (CVPR), 2017. 2, 4
[59] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-
to-image translation using cycle-consistent adversarial net-
works. In IEEE International Conference on Computer Vi-
sion (ICCV), 2017. 2
[60] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros,
O. Wang, and E. Shechtman. Toward multimodal image-to-
image translation. In Advances in Neural Information Pro-
cessing Systems, 2017. 2, 3, 4
A. Additional Implementation Details
Generator. The architecture of the generator consists of a
series of the proposed SPADE ResBlks with nearest neigh-
bor upsampling. We train our network using 8 GPUs simul-
taneously and use the synchronized version of the Batch-
Norm. We apply the Spectral Norm [38] to all the convolu-
tional layers in the generator. The architectures of the pro-
posed SPADE and SPADE ResBlk are given in Figure 10
and Figure 11, respectively. The architecture of the genera-
tor is shown in Figure 12.
Discriminator. The architecture of the discriminator fol-
lows the one used in the pix2pixHD method [48], which
uses a multi-scale design with the InstanceNorm (IN). The
only difference is that we apply the Spectral Norm to all the
SPADE
Sync Batch Norm
3x3-Conv-k
3x3-Conv-128, ReLU
Resize (order=0)
3x3-Conv-k
Figure 10: SPADE Design. The term 3x3-Conv-k denotes a
3-by-3 convolutional layer with k convolutional filters. The
segmentation map is resized to match the resolution of the
corresponding feature map using nearest-neighbor down-
sampling.
SPADE ResBlk(k)
SPADE
ReLU
SPADE
3x3-Conv-k
ReLU
SPADE
3x3-Conv-k
ReLU
3x3-Conv-k
Figure 11: SPADE ResBlk. The residual block design
largely follows that in Mescheder et al. [36] and Miyato et
al. [39]. We note that for the case that the number of chan-
nels before and after the residual block is different, the skip
connection is also learned (dashed box in the figure).
convolutional layers of the discriminator. The details of the
discriminator architecture is shown in Figure 13.
11
Linear(256, 16384)
Reshape(1024, 4, 4)
SPADE ResBlk(1024), Upsample(2)
SPADE ResBlk(1024), Upsample(2)
SPADE ResBlk(1024), Upsample(2)
SPADE ResBlk(512), Upsample(2)
SPADE ResBlk(256), Upsample(2)
SPADE ResBlk(128), Upsample(2)
SPADE ResBlk(64), Upsample(2)
3x3Conv-3, Tanh
Figure 12: SPADE Generator. Different from prior im-
age generators [22, 48], the semantic segmentation mask is
passed to the generator through the proposed SPADE Res-
Blks in Figure 11.
Image Encoder. The image encoder consists of 6 stride-2
convolutional layers followed by two linear layers to pro-
duce the mean and variance of the output distribution as
shown in Figure 14.
Learning objective. We use the learning objective function
in the pix2pixHD work [48] except that we replace its LS-
GAN loss [34] term with the Hinge loss term [31, 38, 54].
We use the same weighting among the loss terms in the ob-
jective function as that in the pix2pixHD work.
When training the proposed framework with the image
encoder for multi-modal synthesis and style-guided image
synthesis, we include a KL Divergence loss:
LKLD = DKL (q(z|x)||p(z))
where the prior distribution p(z) is a standard Gaussian dis-
tribution and the variational distribution q is fully deter-
mined by a mean vector and a variance vector [28]. We
use the reparamterization trick [28] for back-propagating
the gradient from the generator to the image encoder. The
weight for the KL Divergence loss is 0.05.
In Figure 15, we overview the training data flow. The
image encoder encodes a real image to a mean vector and
a variance vector. They are used to compute the noise in-
put to the generator via the reparameterization trick [28].
The generator also takes the segmentation mask of the in-
put image as input with the proposed SPADE ResBlks. The
 Concat and the output image from the generator as input and aims
to classify that as fake.
4x4-↓2-Conv-64, LReLU Training details. We perform 200 epochs of training on the
4x4-↓2-Conv-128, IN, LReLU
Cityscapes and ADE20K datasets, 100 epochs of training
on the COCO-Stuff dataset, and 50 epochs of training on the
4x4-↓2-Conv-256, IN, LReLU Flickr Landscapes dataset. The image sizes are 256 × 256,
4x4-Conv-512, IN, LReLU except the Cityscapes at 512 × 256. We linearly decay the
learning rate to 0 from epoch 100 to 200 for the Cityscapes
4x4-Conv-1
and ADE20K datasets. The batch size is 32. We initialize
the network weights using thes Glorot initialization [12].
Figure 13: Our discriminator design largely follows that in
the pix2pixHD [48]. It takes the concatenation the segmen-
tation map and the image as input. It is based on the Patch-
GAN [22]. Hence, the last layer of the discriminator is a
convolutional layer.
Image
Encoder

3x3-↓2-Conv-64, IN, LReLU

3x3-↓2-Conv-128, IN, LReLU Generator

3x3-↓2-Conv-256, IN, LReLU

3x3-↓2-Conv-512, IN, LReLU

Concat
3x3-↓2-Conv-512, IN, LReLU

3x3-↓2-Conv-512, IN, LReLU

Reshape(8192, 1, 1)
Linear(256) Linear(256)
𝜇 𝜎𝟐
Figure 14: The image encoder consists a series of convolu-
tional layers with stride 2 followed by two linear layers that
output a mean vector µ and a variance vector σ.
discriminator takes concatenation of the segmentation mask
Discriminator
Figure 15: The image encoder encodes a real image to a la-
tent representation for generating a mean vector and a vari-
ance vector. They are used to compute the noise input to the
generator via the reparameterization trick [28]. The gener-
ator also takes the segmentation mask of the input image as
input via the proposed SPADE ResBlks. The discriminator
takes concatenation of the segmentation mask and the out-
put image from the generator as input and aims to classify
that as fake.

12
B. Additional Ablation Study In Table 5, we also analyze the effectiveness of each
component used in our strong baseline, the pix2pixHD++
Method COCO. ADE. City.
method, derived from the pix2pixHD method. We
found that the Spectral Norm, synchronized BatchNorm,
Ours 35.2 38.5 62.3
TTUR [17], and the hinge loss objective all contribute to
Ours w/o Perceptual loss 24.7 30.1 57.4
the performance boost. Adding the SPADE to the strong
Ours w/o GAN feature matching loss 33.2 38.0 62.2
baseline further improves the performance. Note that the
Ours w/ a deeper discriminator 34.9 38.3 60.9
pix2pixHD++ w/o Sync BatchNorm and w/o Spectral Norm
pix2pixHD++ w/ SPADE 34.4 39.0 62.2
still differs from the pix2pixHD in that it uses the hinge loss
pix2pixHD++ 32.7 38.3 58.8
objective, TTUR, a large batch size, and the Glorot initial-
pix2pixHD++ w/o Sync BatchNorm 27.4 31.8 51.1
ization [12].
pix2pixHD++ w/o Sync BatchNorm, 26.0 31.9 52.3
and w/o Spectral Norm
C. Additional Results
pix2pixHD [48] 14.6 20.3 58.3
In Figure 16, 17, and 18, we show additional synthe-
Table 5: Additional ablation study results using the mIoU sis results from the proposed method on the COCO-Stuff
metric: the table shows that both the perceptual loss and and ADE20K datasets with comparisons to those from the
GAN feature matching loss terms are important. Mak- CRN [6] and pix2pixHD [48] methods.
ing the discriminator deeper does not lead to a perfor- In Figure 19 and 20, we show additional synthesis re-
mance boost. The table also shows that all the compo- sults from the proposed method on the ADE20K-outdoor
nents (Synchronized BatchNorm, Spectral Norm, TTUR, and Cityscapes datasets with comparison to those from the
the Hinge loss objective, and the SPADE) used in the pro- CRN [6], SIMS [43], and pix2pixHD [48] methods.
posed method helps our strong baseline, pix2pixHD++. In Figure 21, we show additional multi-modal synthesis
results from the proposed method. As sampling different z
Table 5 provides additional ablation study results ana- from a standard multivariate Gaussian distribution, we syn-
lyzing the contribution of individual components in the pro- thesize images of diverse appearances.
posed method. We first find that both of the perceptual loss In the accompanying video, we demonstrate our seman-
and GAN feature matching loss inherited from the learn- tic image synthesis interface. We show how a user can cre-
ing objective function of the pix2pixHD [48] are impor- ate photorealistic landscape images by painting semantic
tant. Removing any of them leads to a performance drop. labels on a canvas. We also show how a user can synthe-
We also find that increasing the depth of the discriminator size images of diverse appearances for the same semantic
by inserting one more convolutional layer to the top of the segmentation mask as well as transfer the appearance of a
pix2pixHD discriminator does not improve the results. provided style image to the synthesized one.

13
 Label Ground Truth CRN pix2pixHD Ours

Figure 16: Additional results with comparison to those from the CRN [6] and pix2pixHD [48] methods on the COCO-Stuff
dataset.
14
 Label Ground Truth CRN pix2pixHD Ours

Figure 17: Additional results with comparison to those from the CRN [6] and pix2pixHD [48] methods on the COCO-Stuff
dataset.
15
 Label Ground Truth CRN pix2pixHD Ours

Figure 18: Additional results with comparison to those from the CRN [6] and pix2pixHD [48] methods on the ADE20K
dataset.

16
 Label Ground Truth CRN SIMS pix2pixHD Ours

Figure 19: Additional results with comparison to those from the CRN [6], SIMS [43], and pix2pixHD [48] methods on the
ADE20K-outdoor dataset. 17
 Label Ground Truth Ours

CRN SIMS pix2pixHD

Label Ground Truth Ours

CRN SIMS pix2pixHD

Label Ground Truth Ours

CRN SIMS pix2pixHD

Figure 20: Additional results with comparison to those from the CRN [6], SIMS [43], and pix2pixHD [48] methods on the
Cityscapes dataset.

18
 Label Ground Truth Multi-modal results

Figure 21: Additional multi-modal synthesis results on the Flickr Landscapes Dataset. By sampling latent vectors from a
standard Gaussian distribution, we synthesize images of diverse appearances.
19