Documente Academic
Documente Profesional
Documente Cultură
fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
1
Abstract—Photo sharing is an attractive feature which popularizes Online Social Networks (OSNs). Unfortunately, it may leak users’
privacy if they are allowed to post, comment, and tag a photo freely. In this paper, we attempt to address this issue and study the
scenario when a user shares a photo containing individuals other than himself/herself (termed co-photo for short). To prevent possible
privacy leakage of a photo, we design a mechanism to enable each individual in a photo be aware of the posting activity and participate
in the decision making on the photo posting. For this purpose, we need an efficient facial recognition (FR) system that can recognize
everyone in the photo. However, more demanding privacy setting may limit the number of the photos publicly available to train the
FR system. To deal with this dilemma, our mechanism attempts to utilize users’ private photos to design a personalized FR system
specifically trained to differentiate possible photo co-owners without leaking their privacy. We also develop a distributed consensus-
based method to reduce the computational complexity and protect the private training set. We show that our system is superior to
other possible approaches in terms of recognition ratio and efficiency. Our mechanism is implemented as a proof of concept Android
application on Facebook’s platform.
Index Terms—Social network, photo privacy, secure multi-party computation, support vector machine, collaborative learning
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
2
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
3
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
4
n classes. The goal of each binary classifier is to them, we only need to find a binary decision function
distinguish one class from the rest with a deci- f (·). When a probing sample x comes, if f (x) > 0,
sion function. Hence, the ith decision function fi x belongs to user1 and vice versa. In this paper, the
is trained by taking records from user i as positive decision function is determined by the support vector
samples and the records from all the other users machine as f (x) = K(w, x) + b, where K(·, ·) is the
as the negative samples. When a testing record x kernel function and we use linear kernel for the ease of
comes, if fi concludes that it belongs to class i, x is presentation. For the training samples xi of size Ni × p,
labeled as class i. where Ni is the number of training samples, and p is the
• One-against-one method uses max-voting-win number of features in each training sample. Denote u as
strategy. It constructs n(n − 1)/2 binary classifiers, u = [w, b] of size (p + 1) × 1, Xi as Xi = [xi , 1] of size
in which each classifier is aimed to distinguish two Ni ×(p+1) and Yi is a Ni ×Ni diagonal matrix indicating
classes. The idea is that if we can distinguish any class labels of samples in Xi on its diagonal elements.
two classes, then we can identify any of them. Let X1 denote the positive sample set, X2 the negative
Hence, classifier uij is constructed by taking records sample set and a diagonal matrix Π is constructed as
from i as positive samples and records from j as a (p + 1) × (p + 1) diagonal matrix with Π(i, i) = 1 for
negative ones. Later on when we are trying to i = 1, 2, . . . , p and Π(p + 1, p + 1) = 0. Then, the decision
identify a test record x, if uij concludes that x is in function f (·) can be obtained by solving the following
class i, then the vote of class i is added by one. After problem:
testing all the n(n − 1)/2 classifiers, x is assigned to
the class with the largest voting value. 1 T
min u Πu + Ckξ1 k + Ckξ2 k
However, no matter which method we use, it requires u,ξ1 ≥0,ξ2 ≥0 2
s.t. Y1 X1 u ≥ 1 − ξ1 , (2)
a centralized node to access all the training samples
from each class, which is conflicting with our promise Y2 X2 u ≥ 1 − ξ2 .
that the private training samples will not be disclosed
during the whole process. In the rest of this paper we In problem (2), by minimizing 21 uT Πu, we find u that
will focus on how to build the personal FR engines maximizes the margin between the positive and negative
without disclosing the private photo sets. Notice that training set. The constraints are used to ensure that the
the identification criterion could be asymmetric between decision function satisfies the training set. ξi is a set
different personal FR engines, which means that the way of slack variables in case the training samples are not
how David finds out Bob and how Bob finds out David separable. If a certain positive sample X1k cannot make
are not the same as shown in Fig. 1. The reason is that, X1k u > 1, a positive slack variable ξ1k is assigned so
for Bob, his personal FR engine only knows how to find that X1p u > 1 − ξ1p . Meanwhile, a penalty of Cξ1p is
out David from the candidate set (”suspects” for short) assigned to the objective function, where C is the user-
of {Bob, David, Eve, Tom}, while for David, his personal chosen penalty parameter and vice versa for the negative
FR only knows how to find out Bob from the suspects of samples. Notice that the constraints are private training
{Alice, Bob, David, Tom}. In other words, with different data which are not available for a centralized SVM
friend sets (friendship graph) at each node, the personal solver. Our approach is to split (2) into two subproblems
FR engines are trained with different negative training with their own constraints and an additional constraint
samples. u1 = u2 as:
4 S YSTEM OVERVIEW 1 T
min u Πu1 + Ckξ1 k
In this section, we present the detailed description of our u1 ,ξ1 ≥0 4 1
system. Generally speaking, the consensus result could s.t. Y1 X1 u1 ≥ 1 − ξ1 ,
be achieve by iteratively refining the local training re- u1 = u2 , (3a)
sult: firstly, each user performs local supervised learning
1 T
only with its own training set, then the local results min u2 Πu2 + Ckξ2 k
u2 ,ξ2 ≥0 4
are exchanged among collaborators to form a global
knowledge. In the next round, the global knowledge is s.t. Y2 X2 u2 ≥ 1 − ξ2 ,
used to regularize the local training until convergence. In u1 = u2 . (3b)
this section, firstly, we use a toy system with two users
to demonstrate the principle of our design. Then, we We can easily show that problem (3) is an identical
discuss how to build a general personal FR with more transformation of problem (2) by substituting u = u1 =
than two users. Finally, we discuss the scalability of our u2 and putting together the constraints[8]. Problem (3a)
design at the large scale of OSNs. and (3b) could be assigned to user1 and user2 accord-
ingly and be solved by alternatively optimize u1 and
4.1 A toy system u2 . ut1 and ut2 might be very different at the first few
Suppose there are only two users user1 and user2 with iterations, however, they will slowly reach the consensus
private training data x1 and x2 . In order to distinguish as t grows.
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
5
To solve this problem, firstly, we need to find the Algorithm 1: Iterative Method to Compute uij
augmented Lagrange function with the Language mul- Input: Positive samples Xi , Negative samples Xj
tipliers of {λi } and {αi } as: Output: The classifier uij (·)
1 X T X Initial λ, u0i , u0j , αi0 , αj0 as vectors of all zeros;
L({ui }, {λi }, {αi }) = ui Πui + αiT (ui − uj ) A = 2Xi (Π + 4ρI)−1 XiT ;
4 i=1,2 i,j=1,2 for t = 0, 1, 2... do
X ρ
B = 1 + 2Xi (Π + 4ρI)−1 (αit − αjt − 2ρutj );
X
T
− λi (Yi Xi ui − 1 + ξi ) + kui − uj k2 .
i=1,2 i,j=1,2
2 λt+1 = qd(A, B);
(4) ut+1
i = 2(Π + 4ρI)−1 [XiT λt+1 − (αit − αjt ) + 2ρutj ];
t+1
In Eq. (4), we omit the Language multipliers of the if |ui − uti | <threshold then
slack variables, which can be canceled out in the Wolfe break;
dual problem. Here, ρ2 kui − uj k2 is the regularization else
term, which has two roles: (1) It eliminates the condition αit+1 = αit + ρ(ut+1 i − ut+1
j );
t+1 t+1
that L is differentiable such that the solution converges send ui and αi to user j;
under far more general conditions. (2) By adjusting the request ut+1 j and αjt+1 from user j;
parameter of ρ, we can trade off the speed of conver- end
gence for better steady-state approximation[8]. end
L could then be minimized in a cyclic fashion: at each return ut+1
i ;
iteration, L is minimized with respect to one variable
while keeping all other variables fixed. According to
Alternating Direction Method of Multipliers (ADMM)[3],
Proof: For the toy system, problem (2) could be
update of the variables at each iteration t + 1 could be
written in a general form as follows:
summarized as follows,
min F1 (u1 ) + F2 (u2 )
ut+1
i = argmin L(ui , {utj6=i }, {λti }, {αit }); v,u
ui (7)
(5) s.t. Au1 = u2 ,
αit+1 =αit + ρ(ut+1
i − ut+1
j ). u1 ∈ S1 , u2 ∈ S2 .
In (5), ui is calculated through the Wolfe dual problem. In problem (7), F1 (·) is the local problem for user1 and
User i is could compute ut+1 i locally, because it is only F2 (·) is the local problem for user2 . A is an identity
related to Xi , Yi , λti and utj but have nothing to do with matrix to ensure that u1 = u2 . It is proved in [8] and
Xj and Yj . This data isolation property is the essence [3] that the convergence of problem (7) is guaranteed as
of our secure collaborative learning model and the de- long as one of the following two conditions is true: S1
tailed security analysis will be presented in Section 5). is bounded; or AT A is nonsingular. In our scheme, A is
With KKT conditions and Wolfe dual, detailed iterative an identity matrix, hence, u1 and u2 will converge to the
updates are listed in Eq. (6). same optimal value.
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
6
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
7
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
8
To protect the training photos, a privacy-preserving properties of polynomials. We can directly adopt their
SVM training method [25] is used. In this approach, scheme to find the access policy S.
Yu et al. use secure dot product protocols [9] to Friend list: Basically, in our proposed one-against-one
evaluate kernel matrix of SVM. The computational strategy a user needs to establish classifiers between
cost for the secure dot product protocol based on {self, friend} and {friend, friend} also known as the two
homomorphic encryption is O p log m , where m is loops in Algorithm. 2. During the first loop, there is no
the value of the exponent. SVM kernel matrix is privacy concerns of Alice’s friend list because friendship
2
composed of (N n) dot products, hence the total graph is undirected. However, in the second loop, Alice
+1
cost is O N n + O N 3 n2 p log m , where is a need to coordinate all her friends to build classifiers
factor between 2 and 3. between them. According to our protocol, her friends
• One-against-all approach decompose the friend- only communicate with her and they have no idea of
ship graph and use our proposed consensus-based what they are computing for.
training method to perform collaborative training. Friend list could also be revealed during the classifier
As we discussed in the previous section, at each reuse stage. For example, suppose Alice want to find
iteration, local SVM problem only deals with a ubt between Bob and Tom, which has already been
training set of sizen × p. Hence,
computational cost computed by Bob. Alice will first query user k to see
is O Ta (n + n2 p) ≈ O nTa . There are D̄2 local if ukj has already been computed. If this query is made
training problems to find D̄ classifiers for one neigh- in plaintext, Bob immediately knows Alice and Bob are
borhood. Hence the total cost for N neighborhoods friends. To address this problem, Alice will first make
is O N D̄2 Ta n . a list for desired classifiers use private set operations in
• One-against-one: The analysis of this approach is [10] to query against her neighbors’ classifiers lists one
similar to the one-against-all approach, except that by one. Classifiers in the intersection part will be reused.
the average rounds in one training process should Notice that even with this protection, mutual friends
be much less, due to the fact that there are only two between Alice and Bob are still revealed to Bob, this
participants instead of D̄ + 1 ones. If we consider is the trade-off we made for classifiers reuse. Actually,
the complete subgraph in the friendship graph, the OSNs like Facebook shows mutual friends anyway and
expected cost should be less than O N D̄2 To n . there is no such privacy setting as “hide mutual friends”.
A theoretical comparison of the three approaches are Private training sets: We assume that Alice and Bob in
listed in Table. 1. We can see that the distributed so- a toy system are semi-honest. They will follow the pro-
lutions with context information can greatly reduce the tocol but are so curious that they store all the exchanged
computation. Meanwhile, among the two distributed data and try to trace back others’ private training sets.
approaches, the proposed approach should be much The analysis is done on behalf of Alice (Alice stores all
more efficient than using the one-against-all approach. the data and tries to find the private photo set of Bob
In Section 6, we will further demonstrate one-against- Xb ) and the analysis for Bob is similar. To show the
one strategy is much more efficient than one-against-all private training sets are secure, we only need to show
strategy with numerical results. that during the To rounds of parameter exchanges, an
adversary cannot reverse engineer X of the other user.
complexity Privacy- Stranger After To rounds of parameter exchange, information
preserving detection
available to Alice is {utb , αbt }, for t = 1...To . Her goal is
O N +1 n
Centralized X ×
to find an Nb × (p + 1) matrix Xb with Nb × p unknowns.
N D̄2 T n
OVA O a X × Alice is familiar with the training mechanism and she
O N D̄2 To n knows that the parameters at hand have the relationship
Our approach X X
as follows:
TABLE 1: Theoretical comparison of the three
approaches A = 2Xb c−1 XbT , (9)
XbT λ = cutb + d, (10)
−1
5.2 Security analysis B = 1 + Xb c d, (11)
In this paper, private information of a user is considered 1 T
λ = arg min λ Aλ + B T λ (12)
as his/hers privacy and exposure policies; friend list 0≤λ≤ C
2
2
and the private training data set Xa . In the rest of this
subsection, we show how these private information are where c = 2(Π + 4ρI)−1 , d = αbt − αat − 2ρuta could
protected from a semi-honest adversary. be computed accordingly for each iteration. Notice that
Privacy and exposure policies: In 1, access policy of the value of λ comes from the quadratic optimization
x is determined by the intersection of owner’s privacy problem (12), in which A is a fixed matrix determined
policy and co-owners’ exposure policy. In [10], Kissner by Xb , B is changing by iterations. We need to show
and Song proposed privacy-preserving set operations in- that, with multiple {B, ub } tuples, Alice cannot get any
cluding set intersection by employing the mathematical information of Xb . To solve the quadratic optimization
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
9
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
10
5
x 10
request permissions. If they all agree to post x, x will be 5 10
ova
shared on the owner’s page like a normal photo. In this ovo
sense, users could specify their privacy policy but their ova
ovo
8
Efficiency gain
6
Training time(s)
Iterations
users. The database contains photos for 395 individuals 10 20
and 20 images per individual with varying poses and
facial expressions. Users are assigned with photos from
the same individual randomly. 5 10
4 0.41
is highly related to the geodesic distance (the average
shortest distance between any two vertices). We want 3 0.39
shortest distance
to show that in a small-world network, there exist a reuse probability
lot of complete subgraphs, which greatly reduces the 2 0.37
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
11
6.3 Facial recognition performance that a false positive recognition will reveal the test image
to the wrong person. Thus, a low false positive rate is
In this subsection, we study the recognition ratio against
desirable. If there are no strangers, the false positive rate
the number of friends and the number of strangers. Stan-
is only determined by the recognition accuracy. If there
dard face detection in [23] is used for face detection and
are strangers, the false positive is also determined by
eigenface [22] is used to extract features and vectorize the
misclassification of the strangers. Fig. 9 illustrates both
training image. However, the standard eigenface method
false positive rate and false negative rate of our scheme
is a centralized approach, it may not be applicable to our
and the DAG scheme. We observe that false positive rate
distributed case. To address this, we assume principle
of our scheme is 10% lower than original DAG scheme
components have already been extract to form a vector
on average. Notice that false negative recognitions could
space S. User’s facial photos are projected into this
also be introduced by our stranger detection scheme,
space as feature vectors. Based on our simulation results,
according to Fig. 9, the more users, the higher chance
we find that this modification is reasonable due to the
a user is recognized as a stranger.
fact that the important features on human face lie on
only a few directions. Facial feature extraction is beyond
the scope of this paper. Better facial feature extraction
method can be applied to our system to obtain a better 7 C ONCLUSION AND D ISCUSSION
recognition ratio.
In Fig.8, we show the recognition ratios of our pro- Photo sharing is one of the most popular features in
posed scheme and the scheme with DAG decision tree. online social networks such as Facebook. Unfortunately,
As in Fig.8(a), when there are no strangers, both our careless photo posting may reveal privacy of individuals
proposed scheme and the DAG scheme could achieve in a posted photo. To curb the privacy leakage, we
very high recognition ratio of more than 80% when the proposed to enable individuals potentially in a photo
number of users is fewer than 30. While in Fig.8(b), to give the permissions before posting a co-photo. We
among the users, 10% of them are strangers, we can see designed a privacy-preserving FR system to identify
that the recognition ratio of our scheme has a higher individuals in a co-photo. The proposed system is fea-
recognition ratio than the DAG scheme by 5%. The tured with low computation cost and confidentiality of
reason is that our scheme is able to reject strangers. the training set. Theoretical analysis and experiments
The solid line on each figure represents recognition ratio were conducted to show effectiveness and efficiency
of strangers ps , which is increasing with number of of the proposed scheme. We expect that our proposed
users. Intuitively, if there are more users, there will scheme be very useful in protecting users’ privacy in
be more classifiers and the chance that a stranger gets photo/image sharing over online social networks. How-
contradictory decisions will be higher. Fig.8(c) shows a ever, there always exist trade-off between privacy and
similar case where there are 30% strangers. In this case, utility. For example, in our current Android application,
our scheme outperforms the DAG scheme by 10% in the co-photo could only be post with permission of all
terms of recognition ratio. This is achieved by the ability the co-owners. Latency introduced in this process will
of identifying strangers. With 30 users, the probability of greatly impact user experience of OSNs. More over, local
identifying a stranger is around 35%. FR training will drain battery quickly. Our future work
Another criterion to measure the performance is the could be how to move the proposed training schemes to
false positive rate. In the previous section we argued personal clouds like Dropbox and/or icloud.
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
12
1 1 0.4 1 0.4
Recognition ratio
Recognition ratio
0.6 0.6 0.32 0.6 0.32
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TDSC.2015.2443795, IEEE Transactions on Dependable and Secure Computing
13
Yuanxiong Guo received his B.E. degree in Xiaolin Li is Associate Professor in Department
electronic information science and technology of Electrical and Computer Engineering at Uni-
from Huazhong University of Science and Tech- versity of Florida. His research interests include
nology, Wuhan, China, in 2009. He received Parallel and Distributed Systems, CyberCPhysi-
M.S. and Ph.D. degree in Electrical and Com- cal Systems, and Network Security. His research
puter Engineering from the University of Florida has been sponsored by National Science Foun-
in 2012 and 2014, respectively. He has been dation (NSF), Department of Homeland Security
an assistant professor in the School of Electri- (DHS), and other funding agencies. He is an
cal and Computer Engineering, Oklahoma State associate editor of several international journals
University from August 2014. His research in- and a program chair or co-chair for over 10
terests include Smart Grids, Power and Energy international conferences and workshops. He is
Systems, Cyber-Physical Systems, and Sustainable Computing and on the executive committee of IEEE Technical Committee on Scalable
Networking Systems. Computing (TCSC) and served as a panelist for NSF. He received a
Ph.D. degree in Computer Engineering from Rutgers University, USA.
He is the founding director of the Scalable Software Systems Laboratory
(http://www.s3lab.ece.ufl.edu). He received the National Science Foun-
dation CAREER Award in 2010. He is a member of IEEE and ACM.
1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.