IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO.
1, JANUARY 2018
Tensor Rank Preserving Discriminant Analysis
for Facial Recognition
Dapeng Tao , Yanan Guo, Yaotang Li, and Xinbo Gao , Senior Member, IEEE
Abstract— Facial recognition, one of the basic topics in com-
puter vision and pattern recognition, has received substantial
attention in recent years. However, for those traditional facial
recognition algorithms, the facial images are reshaped to a
long vector, thereby losing part of the original spatial con-
straints of each pixel. In this paper, a new tensor-based feature
extraction algorithm termed tensor rank preserving discriminant
analysis (TRPDA) for facial image recognition is proposed;
the proposed method involves two stages: in the first stage,
the low-dimensional tensor subspace of the original input tensor
samples was obtained; in the second stage, discriminative locality
alignment was utilized to obtain the ultimate vector feature
representation for subsequent facial recognition. On the one
hand, the proposed TRPDA algorithm fully utilizes the natural
structure of the input samples, and it applies an optimization cri-
terion that can directly handle the tensor spectral analysis prob-
lem, thereby decreasing the computation cost compared those
traditional tensor-based feature selection algorithms. On the
other hand, the proposed TRPDA algorithm extracts feature
by finding a tensor subspace that preserves most of the rank
order information of the intra-class input samples. Experiments
on the three facial databases are performed here to determine
the effectiveness of the proposed TRPDA algorithm.
Index Terms— Tensor representation, rank preserving, face
recognition, discriminant analysis.
I. I NTRODUCTION
F ACEIAL recognition, one of the popular topics in com-
puter vision that has been studied and used for several
decades, can be applied to a mass of applications such as
security, entertainment, and forensics. The development of
Manuscript received March 18, 2016; revised January 26, 2017; accepted
October 5, 2017. Date of publication October 12, 2017; date of current version
November 3, 2017. This work was supported in part by the National Natural
Science Foundation of China under Grant 61562053, Grant 61572486,
Grant 61432014, Grant 61772402, Grant 61772455, and Grant 61402458,
in part by the Yunnan Natural Science Funds under Grant 2016FB105, in part
by the Guangdong Natural Science Funds under Grant 2014A030310252,
in part by the Program for Excellent Young Talents of Yunnan Univer-
sity under Grant WX069051, in part by the National Key Research and
Development Program of China under Grant 2016QY01W0200, in part by
the National High-Level Talents Special Support Program of China under
Grant CS31117200001, and in part by the Project of Innovative Research
Team of Yunnan province. The associate editor coordinating the review of
this manuscript and approving it for publication was Prof. Shiguang Shan.
(Corresponding author: Dapeng Tao.)
D. Tao is with the School of Information Science and Engineering, Yunnan
University, Kumming 650091, China (e-mail: dapeng.tao@gmail.com).
Y. Guo and Y. Li are with the School of Mathematics and Statistics,
Yunnan University, Kumming 650091, China (e-mail: yananguo.ynu@qq.com;
liyaotang@ynu.edu.cn).
X. Gao is with the State Key Laboratory of Integrated Services Networks,
School of Electronic Engineering, Xidian University, Xi’an 710071, China
(e-mail: xbgao@mail.xidian.edu.cn).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2017.2762588
1057-7149 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
325
intelligent algorithms for face recognition results in efficiency
improvements, innovations, and cost savings in several areas.
In general, the basic steps for facial recognition
involve facial detection, handcrafted-based feature extrac-
tion, subspace-based feature extraction and classification [14].
At the facial detection and handcrafted-based feature extrac-
tion stage [27], local features encode features on interest
regions [31]. Biswas et al. [2] used Scale Invariant Feature
Transform (SIFT) features [6] to describe each landmark and
combined the SIFT features of total landmarks to represent
a face. The SIFT feature was popularized in the computer
vision field and was initially designed for recognizing the
identical object under different conditions. Because of its high
discriminative power, the SIFT feature is widely adopted for
facial recognition [5]. Chen et al. [3] obtained multi-scale
Local Binary Pattern (LBP) [15] features from 27 landmarks,
where the 27 landmarks come from a patch. For all patches,
LBP features are connected to become a long feature vector
as the pose feature. LBP features are originally proposed for
texture classification; for an image, the values of LBP features
are determined by its local geometric structure, based on a
non-parametric method. The LBP features have been used in
image description widely [33].
At the subspace-based feature extraction stage
[24]–[26], [34], one of the most important problems is
to find a projection that transforms the original samples from
a high-dimensional space to a low-dimensional subspace for
the subsequent classification. This approach aims to reveal
an effective representation of the distribution of the samples
in the original high-dimensional space. From the perspective
of the input sample is a vector or tensor, we can group these
algorithms into two categories: vector-based algorithms and
tensor-based algorithms.
Traditional vector-based algorithms, which represent each
input sample as a vector, have been developed for sev-
eral decades [1], [10], [11], [16], [19], [20], [22], [23],
[28], [29], [40]. Representative conventional unsupervised
feature selection algorithms include locality preserving projec-
tions (LPP) [12], principal component analysis (PCA) [13] and
ISOMAP [35]. LPP, a popular manifold learning based unsu-
pervised linear feature selection algorithm, builds a weighted
graph to preserve the distance-based sample relationship infor-
mation. In the training set, the undirected weighted graph
combines the neighborhood information of the pair samples.
The disadvantage of LPP is that the class label information is
not utilized; hence, it is not optimal for the classification tasks.
PCA is the traditional globally unsupervised linear feature
326
selection algorithm that maximizes the mutual information
between primitive high-dimensional Gaussian distributed sam-
ples and projected low-dimensional samples. However, as for
LPP, PCA does not consider the label information; thus, we do
not use it to classify directly in general. As a variant of
MDS [7], ISOMAP preserves the global geodesic distances
of all pair-wise samples. Representative conventional super-
vised feature selection algorithms include linear discriminant
analysis (LDA) [8], Marginal Fisher’s Analysis (MFA) [37]
and discriminative locality alignment (DLA) [42]. LDA is
one of the widely used globally supervised linear feature
selection algorithms that not only maximizes the determi-
nant of the between-class scatter matrix but also minimizes
the determinant of the within-class scatter matrix with low-
dimensional projected samples. Although LDA has extensive
applications in pattern classification tasks [4], it often meets a
circumstance that requires a mass of training samples to obtain
a good model approximation, which can be seen as the small
sample size (SSS) problem [32]. MFA is a popular supervised
manifold learning based linear feature selection algorithm,
which builds a penalty graph with the inter-class marginal
samples to keep the divisibility of the inter-class. However,
MFA ignores the discriminative information of non-marginal
samples and faces the ill-posed problem. DLA is a popular
supervised manifold learning based linear feature selection
algorithm. DLA utilizes the classification optimization criteria:
the distance between the intra-class output samples is as small
as possible and the distance between the inter-class output
samples is as large as possible to preserve the discriminative
information in a local patch. In addition, DLA combines all
the optimal weighted parts to form a global subspace structure.
However, DLA does not preserve the rank order information
of the intra-class samples.
For vector feature representation, the key shortcoming
is that this scheme loses part of the original spatial con-
straints of each pixel in the face images, which hinders the
subsequent algorithm to construct the optimal model and
classification model. To solve the aforementioned difficulty,
some researchers propose to use tensor representation rather
than vector representation as the input sample [41], [36].
Two-dimensional PCA (2DPCA) [38], an unsupervised algo-
rithm, projects an image matrix to a low-dimensional matrix
by a linear transformation in the 2-mode while maximizing
the mutual information. Thus, the linear transformation in the
1-mode is ignored, resulting in poor performance. Multi-linear
PCA (MPCA) [21] is an unsupervised algorithm where the
input sample can be vectors, matrices, or higher-order tensors;
MPCA captures most of the original structure of the input
sample. The disadvantage of MPCA is that the class label
information is not utilized; hence, it is not optimal for the
classification tasks. Two-dimensional LDA (2DLDA) [39] is
a supervised algorithm that projects an image matrices to
a low-dimensional matrix by a linear transformation in the
1-mode and 2-mode simultaneously, the advantage of 2DLDA
is that it maximizes the determinant of the between-class
scatter matrix and minimizes the determinant of the within-
class scatter matrix; moreover, 2DLDA preserves the original
matrix structure of data. However, 2DLDA does not consider
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 1, JANUARY 2018
the manifold structure of the data. Tensor discriminative
locality alignment (TDLA) [41], a supervised algorithm, is a
tensor generalization for DLA where the optimal solution is
obtained by optimizing each mode of the input samples. The
disadvantage of TDLA is that its computation is expensive
and it does not consider the rank order information of the
intra-class samples in a patch.
Lim [17] proposed a theory of singular values and singular
vectors for tensors based on a constrained variational approach
quite similar to the Rayleigh quotient for symmetric matrix
eigenvalues. These notions are particularly useful in general-
izing certain areas where the spectral analysis of matrices has
traditionally played an important role. Thus, the traditional
spectral analysis based feature selection algorithm can be
generalized to tensor spectral analysis.
In this paper, we present a new feature selection algo-
rithm called Tensor Rank Preserving Discriminant Analy-
sis (TRPDA) that aims to directly approach the optimization
criterion. TRPDA differs from the aforementioned tensor-
based feature selection algorithms, which iteratively approach
the optimization criterion and finally return to the traditional
spectral analysis problem. TRPDA applies an optimization
criterion that can directly solve the tensor spectral analysis
problem. The main contributions of the proposed TRPDA
algorithm are summarized as follows:
1) We represent the facial image as a 2-order-tensor, and
its data structure is preserved. Based on the 2-order
tensor representation, the first step of the proposed
TRPDA algorithm is to extract tensor feature by finding
a tensor subspace that preserves most of the rank order
information of the intra-class input samples.
2) Followed by the first step of the TRPDA algorithm,
we vectorized the redefined tensor features. Next,
discriminative locality alignment is adopted to obtain
the final vector feature representation, by which the
recognition rate is improved.
The rest of the paper is summarized as follows. Section II
introduces a brief description of related tensor algebra. Next,
the proposed TRPDA algorithm is presented in detail in
section III. Finally, section IV reports the experiments details
and discussions, followed by the conclusion in section V.
II. T ENSOR A LGEBRA
Let R denote the set of all real numbers. Tensors [18]
are multidimensional arrays of numbers that transform lin-
early
 under  coordinate transformations. We call X =
X n1 ,n2 ,...,n M ∈ R N1 ×N2 ×...×N M a real tensor of order M if
X n1 ,n2 ,...,n M ∈ R, where 1 ≤ n i ≤ Ni , and 1 ≤ i ≤ M.
We briefly introduce the following relevant definitions in
multi-linear algebra.
Definition 1 (Tensor Product): Let A ∈ R N1 ×N2 ×...×N M and
  
B ∈ R N1 ×N2 ×...×N M  be tensors of order M and M  . In this
case, the tensor product of A and B is a tensor of order
  
M + M  denoted A ⊗ B ∈ R N1 ×N2 ×...×N M ×N1 ×N2 ×...×N M  ,
with its (n 1 , . . . , n M , n 1 , . . . , n M  )-entry given by
A ⊗ Bn1 ×n2 ×...×n M ×n1 ×n2 ...,n = An1 ×n2 ×...×n M Bn1 ×n2 ...,n  ,
M M
for all index values.
TAO et al.: TRPDA FOR FACIAL RECOGNITION 327

Definition 2 (Tensor Contraction): Let A ∈ R1N × N2 ×. . .× A. Rank Preserving Discriminant Analysis

N M × N1 × N2 × . . . × N M and B ∈ R N1 × N × . . . × N ×
 2 M For each sample Xi , we use the Euclidean distance to find its
N 1 × N 2 × . . . × N M  be tensors of order M + M  and
  
k1 nearest within-class neighbors Xi 1 , · · · , Xi k1 and k2 nearest
M + M  . In this case, the contraction on the tensor product between-class neighbors Xi1 , · · · , Xik2 . Thus, we can build a
A ⊗ B is local patch
[A ⊗ B; (1 : M) (1 : M)]

P(Xi ) = Xi , Xi 1 , · · · , Xi k1 , Xi1 , · · · , Xik2
N1 
NM
= ··· (A)n1 ×...×n M ×n1 ×...×n (B)n1 ×...×n M ×n 1 ×...×n M ,
M
∈ R D1 ×D2 ···×D M−1 ×N1+k1 +k2
n 1 =1 n M =1
with the corresponding representation denoted by
which is of order M  + M  .

Definition 3 (Mode-d Product): Let A ∈ R N1 ×N2 ×...×N M be P(yi ) = yi , yi 1 , · · · , yi k1 , yi1 , · · · , yik2 ∈ R 1×(1+k1 +k2 ) .

a tensor and U ∈ R Nd ×Nd be a matrix of order M and 2.
The mode-d product of A and U is a tensor of order M As our goal is to preserve the discriminability of classes for

denoted A ×d U ∈ R N1 ×N2 ×...×Nd−1 ×Nd ×Nd+1 ×...×N M with its classification, the between-class distances will be as large as
(n 1 , . . . , n d−1 , j, n d+1, . . . , n M )-entry given by possible and the within-class rank order information will be
preserved as much as possible. Based on this point, we have
(A ×d U )n1 ×...×nd−1 × j ×nd+1 ×...×n M the following optimizations on patch Xi :

Nd
= An1 ×...×nd−1 ×nd ×nd+1 ×...×n M U j ×nd , 
k1
 2
arg min yi − yi j , (2)
n d =1 yi
j =1
for all index values. The mode-dproduct is a type of contrac- 
k2
 2
tion. arg max yi − yi j . (3)
yi
To simplify the notation in this paper, we denote j =1


M We unify the above by proposing a penalty factor as ωi and
A ×1 U1 ×2 U2 × · · · × M U M  A ×iUi . a combination factor α
i=1 ⎛ ⎞
k1
 2 
k2
 2
Definition 4 (Frobenius Norm): The Frobenius norm of a arg min ⎝ yi − yi j (ωi ) j − α yi − yi j ⎠ , (4)
tensor A ∈ R N1 ×N2 ×...×N M is given by yi
j =1 j =1

 A = ([A ⊗ A; (1 : M) (1 : M)])1/2 where ωi denoted by

⎛ ⎞1/2 ⎧ 
N1 
NM
⎪
=⎝ ··· A2n1 ,...,n M ⎠ . ⎨ Xi ⊗Xij ;(1:M−1)(1:M−1)
  , i f Xi j ∈ Nk1 (Xi )
  
(ωi ) j =  Xi · Xi j 
n 1 =1 n M =1 ⎪
⎩
0, other wi se
Definition 5 (Euclidean Distance): The Euclidean distance
between two tensors A,B ∈ R N1 ×N2 ×...×N M is given by and α ∈ [0, 1], The tradeoff factor (ωi ) j uses different
weighting value to emphasize different distances in the original
D (A,B) = A-B  . sample space. In this way, majority of the rank order infor-
mation of within-class samples is preserved, and the small
III. T ENSOR R ANK P RESERVING distances in raw sample space will lead to more heavily
D ISCRIMINANT A NALYSIS penalization in obtained subspace.
To simplify the following derivation, we set
Given a training set {X i } , (i = 1 : N ) in high-dimensional
space R D1 ×D2 ···×D M−1 , for any sample X i , the corresponding
k1 k
    
2

class label is Ci ∈ Z n . First, we consider the problem βi = [(ωi )1 , · · · , (ωi )k1 , −α, · · · , −α].
of representing samples {Xi } (i = 1 : N ) by a single vector
y = {y1 , y2 , · · · , y N } such that yi represents Xi . Specifically, Thus, (4) reduces to
the objective is to find M − 1 two order transformation vector 
k1
 2 
k2
 2
denoted by u j ∈ R 1×D j , ( j = 1, · · · , M − 1). That is, arg min (βi ) j . yi − yi j + (βi ) j +k1 . yi − yi j
yi
j =1 j =1

M−1
1 +k2
k
y i = Xi ×ju j, (1)  2
j =1
= arg min (βi ) j . y Pi (1) − y Pi ( j +1)
yi
j =1
where i ∈ {1, · · · , N }. Here, we consider the tensor-based = arg min P(yi )L i (P(yi ))T , (5)
Patch Alignment framework (PAF) [42]. yi
328 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 1, JANUARY 2018

where 
N 
N

⎡ k1 +k2 ⎤ = arg min

 u 1 ,u 2 ,···u M−1
j1 =1 j2 =1
⎢ (βi ) j −βiT ⎥ ⎛ ⎞
Li = ⎣ ⎦. (6)
j =1 
D1 
D M−1
−βi di ag (βi ) ×⎝ ··· Xi1 ···i M−1 j1 (u 1 )i1 · · · (u M−1 )i M−1⎠ L j1 j2
i1 =1 i M−1 =1
⎛ ⎞
B. Whole Alignment
⎜ 
D1 
D M−1
⎟
Each P(yi ) can be selected from the global coordinates ×⎝ ··· Xi1 ···i M−1
 j2 (u 1 )i1 · · · (u M−1 )i M−1
 ⎠
Y = [yi , y2 , · · · , y N ] ∈ R 1×N by using a selection matrix, i1 =1 
i M−1 =1
i.e. = arg min
u 1 ,u 2 ,···u M−1
P(yi ) = YSi , (7)
 ! !  2(M−1)

where Si ∈ R N×(1+k1 +k2 ) is the selection matrix whose entry × X × M L T ⊗ X ; (M) (M) ×i u P(i) , (11)
is defined by: i=1
 where P = {1, 2, · · · , M − 1, 1, 2, · · · , M − 1} is the
1 i f p = Fi (q)
(Si ) pq = (8) index set.
0 else, To avoid trivial solutions, we impose the following con-
  straints to uniquely determine the projection matrices:
for all index values, where Fi = i, i 1 , · · · , i k1 , i 1 , · · · , i k2
denotes the set of indices for i t h patch that is built by the  u 1 2 = · · · = u M−1 2 = 1. (12)
patch P(Xi ). For simplicity, we denote
Thus, (5) can be rewritten as:  ! ! 
! X × M L T ⊗ X ; (M) (M)  A. (13)
arg min YSi L i SiT YT . (9)
Y Using (12), (11) can be transformed to
We have the whole alignment by summing over all the part 
2(M−1)
optimizations defined in (8) to achieve arg min A ×i u P(i)
u 1 ,u 2 ,···u M−1
i=1

N !
s.t.  u 1 2 = · · · = u M−1 2 = 1. (14)
arg min YSi L i SiT YT = arg min YLYT , (10)
Y Y
i=1 Similar to Lagrange’s multiplier method used in matrix
N  eigenvalue problem, by taking a constrained variation
" 
where L = Si L i SiT ∈ R N×N is the alignment matrix. approach [15], we can transform (14) to a tensor singular value
i=1 problem. The projection matrices Ui (i = 1, · · · , M − 1) are
By using (1), (10) can be written as
di singular vectors related to di smallest mode-i singular
⎛ ⎞ ⎛ ⎞T values of A.

M−1 
M−1
arg min ⎝X × j U j ⎠ L ⎝X ×jUj⎠
In this part, to the best knowledge of the authors, most
u 1 ,u 2 ,···u M−1 tensor-based algorithms aim to approach an optimization
j =1 j =1
criterion iteratively. The objective function of tensor-based
= arg min algorithm is often defined as:
u 1 ,u 2 ,···u M−1
!
⎛ ⎞T arg min tr UkT F (k) Uk s.t.UkT Uk = I, (15)

D1 
D M−1
Uk
⎜ ··· Xi1 ···i M−1 1 (u 1 )i1 · · · (u M−1 )i M−1 ⎟
⎜ ⎟ where k = 1, 2, · · · , M−1, M−1 is the order of the input sam-
⎜ i1 =1 i M−1 =1 ⎟
⎜. ⎟ ple, Uk (k = 1, · · · , M − 1) are the projection matrices, and
⎜. ⎟
×⎜. ⎟ L F (k) refers to the alignment matrix varies by different tensor-
⎜ ⎟
⎜ D1 ⎟ based methods. Next, when selecting any k ∈ {1, · · · , M − 1},
⎜ 
D M−1 ⎟
⎝ ··· Xi1 ···i M−1 2 (u 1 )i1 · · · (u M−1 )i M−1 ⎠ they update the projection matrix Uk by fixing the other
i1 =1 i M−1 =1 M − 2 projection matrix U j ( j = k). Thus, these tensor-based
⎛ ⎞ algorithms not only will increase computation complexity but

D1 
D M−1
also result in a convergence problem.
⎜ ··· Xi1 ···i M−1 1 (u 1 )i1 · · · (u M−1 )i M−1 ⎟
⎜ ⎟
⎜ i1 =1 i M−1 =1 ⎟
⎜. ⎟ C. Discriminative Locality Alignment
⎜. ⎟
×⎜. ⎟
⎜ ⎟ The output from the above process is a tensor that could not
⎜ D1 ⎟
⎜ 
D M−1 ⎟ be processed by a conventional classifier directly. A tensor to
⎝ ··· Xi1 ···i M−1 2 (u 1 )i1 · · · (u M−1 )i M−1 ⎠ vector feature selection scheme is introduced here to select the
i1 =1 i M−1 =1 vector feature from the above output tensor data.
TAO et al.: TRPDA FOR FACIAL RECOGNITION
TABLE I
A G ENERAL F RAMEWORK OF TRPDA
Fig. 1. UMIST database.
Let Bn ∈ R d1 ×d2 ···×d M−1 be the output tensor. The ele-
ments in Bn are rearranged into a vector feature x n by the
vectorization of Bn . To promote the discriminability of the
vector feature representation further, in this paper, we utilize
the DLA [42] algorithm to transform the facial image set to a
more discriminative feature representation.
As a summary of this section, we provide the procedure of
TRPDA algorithm in Table 1.
IV. E XPERIMENTAL R ESULTS
This section describes experiments conducted on two public
face image databases to highlight the capability of the pro-
posed TRPDA algorithm. Because facial image databases are
naturally a two-order tensor with a row and a column, the input
is a tensor of order two, with its row space and column space
accounting for its two modes.
A. Datasets
The first experiment was performed on the UMIST data-
base [3], which contains 575 facial images from twenty people.
Each of the individuals covers a range of poses from profile
to frontal views, and the subjects cover a range of races,
sexes and appearances. Fig. 1 shows several images of one
individual.
The ORL database [30] was the second dataset to perform
on the experiment, which contains 400 facial images collected
from forty persons. Ten images for each person vary in pose
and facial expression, such as closed or open eyes, smiling or
not, and so on. Fig. 2 shows one individual image set.
The third experiment was performed on the CAS-PEAL-R1
expression and distance database [9], the CAS-PEAL-R1
expression and distance database contains one training set,
two probe sets, and one gallery set. The training set contains
1200 images collected from 300 individuals, four images for
329
Fig. 2. ORL database.
Fig. 3. CAS-PEAL-R1 expression and distance database.
each individual vary in accessory, expression, background, dis-
tance, lighting, and time. The two probe sets (PE and PS) cor-
responding to variations in expression and distance of frontal
faces respectively, the PE probe set contains 1570 images
collected from 377 individuals while the PS probe set contains
275 images collected from 247 individuals. The gallery set
contains 1040 images collected from the 1040 individuals
under a normal condition, with each individual having one
image. Fig. 3 shows several typical images.
B. Experimental Setup and Results
In this section, six vector-based feature extraction algo-
rithms (LDA, PCA, ISOMAP, DLA, MFA, and LPP), and five
tensor-based feature extraction algorithms (2DPCA, 2DLDA,
MPCA, TDLA, TLDA) are compared to evaluate the proposed
TRPDA algorithm on three facial databases. For six vector-
based methods, they reshape the facial image to a long vector
by fixing its order to arrange its pixel values. Regarding
whether or not to consider the class label information, LDA,
DLA, ISOMAP, MFA, 2DLAD, TDLA, and TLDA are super-
vised algorithms that do consider the class label information,
whereas PCA, LPP, 2DPCA, and MPCA are unsupervised
algorithms that do not. When applying the aforementioned
algorithms on three databases, all facial images were normal-
ized to the 40 × 40 pixel with 256 gray levels for each pixel.
For the LDA and LPP algorithms, because the number of
the training samples is substantially smaller than the original
features’ dimension, PCA projection is the first stage to ensure
the within-scatter matrix is nonsingular.
For UMIST and ORL database, they were divided into two
separate sets randomly: training set and testing set. The role of
the training set was to obtain the projection matrix or the pro-
jection matrices to learn the low-dimensional subspace, while
the role of testing set was to report the recognition accuracy
by using the Nearest Neighbor (NN) rule, which is convenient
compared to the Support Vector Machine (SVM) [14]. Specifi-
cally, for the UMIST database, different numbers (3, 5, and 7)
of facial images per individual were randomly selected for
training, with the remaining facial images used for test-
ing; for the ORL database, different numbers (2, 4, and 6)
of facial images per individual were randomly selected for
training, with the remaining facial images used for testing.
Each experiment was performed ten times independently to
330 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 1, JANUARY 2018

Fig. 4. The average recognition rate versus the subspace dimension on the sets of UMIST for six vector-based methods and TRPDA method: (a) three
samples for training; (b) five samples for training; (c) seven samples for training.

Fig. 5. The average recognition rate versus the subspace dimension on the sets of UMIST for six tensor-based methods: (a) three samples for training;
(b) five samples for training; (c) seven samples for training.

Fig. 6. The boxplots on UMIST for seven methods from left to right of LDA, PCA, ISOMAP, DLA, MFA, LPP and TRPDA: (a) three samples for training;
(b) five samples for training; (c) seven samples for training.

obtain the average recognition. For CAS-PEAL-R1 expression
and distance database, the role of the training set was to
obtain the projection matrix or the projection matrices to
learn the low-dimensional subspace, while the role of gallery
set and probe sets was to report the recognition accuracy.
Fig. 4, Fig. 7, and Fig. 9 shows the average recognition
rate versus the subspace dimensions for the UMIST, ORL,
and CAS-PEAL-R1 expression and distance database for six
vector-based methods and TRPDA method. Fig. 5 show the
average recognition rate versus the subspace dimensions for
the UMIST database for six tensor-based methods. Fig. 6 and
Fig. 8 show the boxplots of the experimental results of the
six vector-based methods and TRPDA method on the UMIST
and ORL datasets. For all boxplots, the number of dimensions
of all the algorithms on the test was set from 11 to 15.
Because training time (> 4 hour) of TDLA on CAS-PEAL-R1
expression and distance database was too long, fig. 10 show
the shows the average recognition rate versus the subspace
TAO et al.: TRPDA FOR FACIAL RECOGNITION
Fig. 7. The average recognition rate versus the subspace dimension on the sets of ORL for six vector-based methods and TRPDA method: (a) two samples
for training; (b) four samples for training; (c) six samples for training.
Fig. 8. The boxplots on ORL for seven methods from left to right of LDA, PCA, ISOMAP, DLA, MFA, LPP and TRPDA: (a) two samples for training;
(b) four samples for training; (c) six samples for training.
TABLE II
B EST AVERAGE R ECOGNITION R ATE OF T WELVE
A LGORITHMS ON THE UMIST D ATABASE
dimensions for the CAS-PEAL-R1 expression and distance
database for five tensor-based methods (2DPCA, 2DLDA,
MPCA, TLDA, and TRPDA) combine with DLA. Table 2,
Table 3, and Table 4 report the best average recognition rates
of the corresponding dimensionalities for the UMIST database,
the ORL database, and the CAS-PEAL-R1 expression and
distance database, respectively.
We selected CAS-PEAL-R1 expression and distance dataset
and applied TRPDA, MPCA [1] and LTDA to update the
331
Fig. 9. The average recognition rate versus the subspace dimension on
the sets of CAS-PEAL-R1 expression and distance database for six vector-
based methods and TRPDA method, and the probe sets are: (a) expression;
(b) distance.
projection matrices, respectively. Table 5 reports training time
of each method. We conduct all experiments on an i5-2500K
3.30GHz computer with an 8-GB memory.
C. Discussion
Based on the above experiments, we made the following
observations:
1. 1. Figs. 4-5, Fig. 7, and Figs.9-10 show the recognition
rate of the eleven feature extraction algorithms on the three
facial image datasets. TRPDA was found to have the most
robust performance and produce the highest recognition rate
of the algorithms considered. Specifically, when the number
332
Fig. 10. The average recognition rate versus the subspace dimension on the
sets of CAS-PEAL-R1 expression and distance database for five tensor-based
methods, and the probe sets are: (a) expression; (b) distance.
TABLE III
B EST AVERAGE R ECOGNITION R ATE OF S EVEN
A LGORITHMS ON THE ORL D ATABASE
TABLE IV
B EST AVERAGE R ECOGNITION R ATE OF E LEVEN A LGORITHMS ON
THE CAS-PEAL-R1 E XPRESSION AND D ISTANCE D ATABASE
TABLE V
T RAINING T IME
of dimensions of the selected subspace is low and the training
set is small, our method still presents robust performance
because it simultaneously considers the natural structure of
the input samples and the rank order information of intra-class
samples.
2. Fig. 6 and Fig. 8 show the boxplots of the experimental
results of the six vector-based algorithms and TRPDA on the
two UMIST and ORL datasets. Each boxplot contains a box
and whisker, the box has lines at the lower, median, and high
quartile values, and the whisker extend from end of the box
to the adjoin values in the data-by default. According to these
boxplots, TRPDA extract features with the rank information
and eliminates the most unstable ones, manifold learning
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 1, JANUARY 2018
methods including MFA, DLA and ISOMAP are more stable
than LDA because they consider the manifold structure of
sample.
3. According to the average recognition rate of these
vector-based methods and TRPDA, PCA and LPP do not
exhibit promising performance because they ignore the class
label information. Because the class label information is
considered, ISOMAP, MFA and LDA perform better than
PCA and LPP, and their performance is superior to those
unsupervised algorithms. TRPDA and DLA perform the best
because they consider both the discriminative information and
local geometric information. Because TRPDA takes advantage
of the natural structure of the input samples and preserves
the rank order information in a local patch, it outperforms
DLA. According to the average recognition rate of these
tensor-based methods, TDLA’s performance inferior to other
tensor-based method due to its convergence problem.
Typical tensor-based methods, such as 2DPCA, 2DLDA,
MPCA, achieve comparable performance, because they take
advantage of the natural structure of the input samples.
TRPDA perform the best because it considers the rank infor-
mation and directly approach the optimization criterion.
V. C ONCLUSION
During the past decade, a large number of subspace-based
feature extraction algorithms for facial recognition have been
proposed. However, most traditional extraction algorithms are
vector-based feature selection algorithms, which lose part of
the original spatial constraints of each pixel in the facial
images. Thus, it will be more effective to propose a tensor-
based feature extraction algorithm, by which the natural struc-
ture of the input samples is fully utilized.
In this paper, by considering the facial image as a two order
tensor, the low-dimensional tensor subspace of the original
input tensor samples was obtained; moreover, discriminative
locality alignment was captured to transform the refined ten-
sor samples to the ultimate vector feature representation for
subsequent facial recognition. In addition, numerous exper-
iments on the three facial image databases were conducted
to demonstrate the effective performance of the proposed
algorithm.
R EFERENCES
[1] M. Belkin and P. Niyogi, “Laplacian eigenmaps and spectral techniques
for embedding and clustering,” in Proc. Adv. Neural Inf. Process. Syst.,
vol. 14. Dec. 2002, pp. 585–591.
[2] S. Biswas, G. Aggarwal, P. J. Flynn, and K. W. Bowyer, “Pose-robust
recognition of low-resolution face images,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 35, no. 12, pp. 3037–3049, Dec. 2013.
[3] D. Chen, X. Cao, F. Wen, and J. Sun, “Blessing of dimensionality:
High-dimensional feature and its efficient compression for face verifi-
cation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013,
pp. 3025–3032.
[4] M. Devanne, H. Wannous, S. Berretti, P. Pala, M. Daoudi, and
A. Del Bimbo, “3-D human action recognition by shape analysis of
motion trajectories on Riemannian manifold,” IEEE Trans. Cybern.,
vol. 45, no. 7, pp. 1340–1352, Jul. 2015.
[5] D. B. Graham and N. M. Allinson, “Characterizing virtual eigensigna-
tures for general purpose face recognition,” in Face Recognition:
From Theory to Applications (NATO ASI Series F, Computer and
Systems Sciences), vol. 163, H. Wechsler, P. J. Pil-Lips, V. Bruce,
F. Fogelman-Soulie, and T. S. Huang, Eds. Berlin, Germany: Springer,
1998, pp. 446–456.
TAO et al.: TRPDA FOR FACIAL RECOGNITION
[6] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”
Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.
[7] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. Hoboken,
NJ, USA: Wiley, 2012.
[8] R. A. Fisher, “The use of multiple measurements in taxonomic prob-
lems,” Ann. Eugenics, vol. 7, no. 2, pp. 179–188, 1936.
[9] W. Gao et al., “The CAS-PEAL large-scale Chinese face database and
baseline evaluations,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans,
vol. 38, no. 1, pp. 149–161, Jan. 2008.
[10] X. Gao, X. Wang, D. Tao, and X. Li, “Supervised Gaussian process
latent variable model for dimensionality reduction,” IEEE Trans. Syst.,
Man, Cybern. B, Cybern., vol. 41, no. 2, pp. 425–434, Apr. 2011.
[11] N. Guan, D. Tao, Z. Luo, and B. Yuan, “Non-negative patch alignment
framework,” IEEE Trans. Neural Netw., vol. 22, no. 8, pp. 1218–1230,
Aug. 2011.
[12] X. He and P. Niyogi, “Locality preserving projections,” in Proc. Adv.
Neural Inf. Process. Syst., vol. 45. Dec. 2005, pp. 186–197.
[13] H. Hotelling, “Analysis of a complex of statistical variables into principal
components,” J. Edu. Psychol., vol. 24, no. 6, pp. 417–441, Sep. 1933.
[14] C. Hou, F. Nie, C. Zhang, D. Yi, and Y. Wu, “Multiple rank multi-linear
SVM for matrix data classification,” Pattern Recognit., vol. 47, no. 1,
pp. 454–469, 2014.
[15] D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen, “Local binary
patterns and its application to facial image analysis: A survey,” IEEE
Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 41, no. 6, pp. 765–781,
Nov. 2011.
[16] Y. Jia, F. Nie, and C. Zhang, “Trace ratio problem revisited,” IEEE
Trans. Neural Netw., vol. 20, no. 4, pp. 729–735, Apr. 2009.
[17] L.-H. Lim, “Singular values and eigenvalues of tensors: A variational
approach,” in Proc. IEEE Int. Workshop Comput. Adv. Multi-Sensor
Adapt. Process., Dec. 2005, pp. 129–132.
[18] L. D. Lathauwer, “Signal processing based on multilinear algebra,”
Ph.D. dissertation, Dept. Elektrotechniek, Katholike Universiteit Leuven,
Leuven, Belgium, 1997.
[19] W.-Y. Liu, K. Yue, and M.-H. Gao, “Constructing probabilistic graphical
model from predicate formulas for fusing logical and probabilistic
knowledge,” Inf. Sci., vol. 181, no. 18, pp. 3825–3845, May 2011.
[20] W. Liu, H. Zhang, D. Tao, Y. Wang, and K. Lu, “Large-scale paralleled
sparse principal component analysis,” Multimedia Tools Appl., vol. 75,
no. 3, pp. 1481–1493, 2014, doi: 10.1007/s11042-014-2004-4.
[21] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “MPCA: Multilin-
ear principal component analysis of tensor objects,” IEEE Trans. Neural
Netw., vol. 19, no. 1, pp. 18–39, Jan. 2008.
[22] X. Lu, X. Zheng, and X. Li, “Latent semantic minimal hashing
for image retrieval,” IEEE Trans. Image Process., vol. 26, no. 1,
pp. 355–368, Jan. 2017.
[23] X. Lu, Y. Yuan, and X. Zheng, “Joint dictionary learning for multispec-
tral change detection,” IEEE Trans. Cybern., vol. 47, no. 4, pp. 884–897,
Apr. 2017.
[24] F. Nie, S. Xiang, Y. Song, and C. Zhang, “Extracting the optimal
dimensionality for local tensor discriminant analysis,” Pattern Recognit.,
vol. 42, no. 1, pp. 105–114, 2009.
[25] F. Nie, S. Xiang, Y. Song, and C. Zhang, “Orthogonal
locality minimizing globality maximizing projections for feature
extraction,” Opt. Eng., vol. 48, no. 1, pp. 017202-1–017202-5,
2009.
[26] F. Nie, D. Xu, I. W. Tsang, and C. Zhang, “Flexible manifold embed-
ding: A framework for semi-supervised and unsupervised dimension
reduction,” IEEE Trans. Image Process., vol. 19, no. 7, pp. 1921–1932,
Jul. 2010.
[27] F. Nie, J. Yuan, and H. Huang, “Optimal mean robust principal
component analysis,” in Proc. Int. Conf. Mach. Learn., Jun. 2007,
pp. 1062–1070.
[28] F. Nie, S. Xiang, and C. Zhang, “Neighborhood MinMax projections,”
in Proc. Int. Joint Conf. Artif. Intell., 2007, pp. 993–998.
[29] S. T. Roweis and L. K. Saul, “Nonlinear dimensionality reduction by
locally linear embedding,” Science, vol. 290, no. 5500, pp. 2323–2326,
2000.
[30] F. S. Samaria and A. C. Harter, “Parameterisation of a stochastic model
for human face identification,” in Proc. IEEE Workshop Appl. Comput.
Vis., Dec. 1994, pp. 138–142.
[31] F. M. Sukno, J. L. Waddington, and P. F. Whelan, “3-D facial landmark
localization with asymmetry patterns and shape regression from incom-
plete local features,” IEEE Trans. Cybern., vol. 45, no. 9, pp. 1717–1730,
Sep. 2015.
333
[32] D. Tao, X. Li, X. Wu, and S. J. Maybank, “Geometric mean for subspace
selection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2,
pp. 260–274, Feb. 2009.
[33] D. Tao, X. Tang, X. Li, and X. Wu, “Asymmetric bagging and random
subspace for support vector machines-based relevance feedback in image
retrieval,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 7,
pp. 1088–1099, Jul. 2006.
[34] D. Tao, X. Li, X. Wu, and S. J. Maybank, “General tensor discriminant
analysis and Gabor features for gait recognition,” IEEE Trans. Pattern
Anal. Mach. Intell., vol. 29, no. 10, pp. 1700–1715, Oct. 2007.
[35] J. B. Tenenbaum, V. de Silva, and J. C. Langford, “A global geometric
framework for nonlinear dimensionality reduction,” Science, vol. 290,
no. 5500, pp. 2319–2323, Dec. 2000.
[36] W. K. Wong, Z. Lai, Y. Xu, J. Wen, and C. P. Ho, “Joint tensor feature
analysis for visual object recognition,” IEEE Trans. Cybern., vol. 45,
no. 11, pp. 2425–2436, Nov. 2015.
[37] S. Yan, D. Xu, B. Zhang, H.-J. Zhang, Q. Yang, and S. Lin, “Graph
embedding and extensions: A general framework for dimensionality
reduction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 1,
pp. 40–51, Jan. 2007.
[38] J. Yang, D. Zhang, A. F. Frangi, and J.-Y. Yang, “Two-dimensional
PCA: A new approach to appearance-based face representation and
recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 1,
pp. 131–137, Jan. 2004.
[39] J. Ye, R. Janardan, and Q. Li, “Two-dimensional linear discriminant
analysis,” in Proc. Adv. Neural Inf. Process. Syst., 2004, pp. 1569–1576.
[40] J. Yu, D. Tao, J. Li, and J. Cheng, “Semantic preserving distance metric
learning and applications,” Inf. Sci., vol. 281, pp. 674–686, Oct. 2014.
[41] L. Zhang, L. Zhang, D. Tao, and X. Huang, “Tensor discrimina-
tive locality alignment for hyperspectral image spectral–spatial fea-
ture extraction,” IEEE Trans. Geosci. Remote Sens., vol. 51, no. 1,
pp. 242–256, Jan. 2013.
[42] T. Zhang, D. Tao, X. Li, and J. Yang, “Patch alignment for dimen-
sionality reduction,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 9,
pp. 1299–1313, Sep. 2009.
Dapeng Tao received the B.E degree from
Northwestern Polytechnical University and the Ph.D.
degree from the South China University of Tech-
nology. He is currently a Professor with the School
of Information Science and Engineering, Yunnan
University, Kunming, China. He has authored and
co-authored over 50 scientific articles. His research
interests include machine learning, computer vision,
and robotics. He has served over 10 international
journals including the IEEE TNNLS, the IEEE
TCYB, the IEEE TMM, the IEEE CSVT, the IEEE
TBME, and the Information Sciences.
Yanan Guo received the B.Eng. degree from Hubei
Polytechnic University. She is currently pursuing
the M.Sc. degree with Yunnan University, Kunming,
China. Her research interests include machine learn-
ing and computer vision.
334 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 1, JANUARY 2018

Yaotang Li is currently a Professor with the Xinbo Gao (M’02–SM’07) received the B.E., M.S.,
School of Mathematics and Statistics, Yunnan Uni- and Ph.D. degrees from Xidian University, Xi’an,
versity, China. He has authored or co-authored over China, in 1994, 1997, and 1999, respectively, all
80 research papers. His main research interests in signal and information processing. He was a
include numerical algebra and special matrices. Research Fellow with the Department of Computer
Science, Shizuoka University, Shizuoka, Japan, from
1997 to 1998. From 2000 to 2001, he was a
Post-Doctoral Research Fellow with the Department
of Information Engineering, The Chinese University
of Hong Kong, Hong Kong. Since 2001, he has been
with the School of Electronic Engineering, Xidian
University, where he is currently a Cheung Kong Professor with the Ministry
of Education, and also a Professor of pattern recognition and intelligent
systems and the Director of the State Key Laboratory of Integrated Services
Networks. He has authored five books and around 200 technical articles
in refereed journals and proceedings. His current research interests include
multimedia analysis, computer vision, pattern recognition, machine learning,
and wireless communications. He is a fellow of the Institution of Engineering
and Technology. He is on the Editorial Boards of several journals, including
Signal Processing (Elsevier) and Neurocomputing (Elsevier). He served as
the General Chair/Co-Chair, the Program Committee (PC) Chair/Co-Chair,
or a PC Member for around 30 major international conferences.