770 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 22, NO.

6, JUNE 2010

Bridging Domains Using World Wide

Knowledge for Transfer Learning
Evan Wei Xiang, Bin Cao, Derek Hao Hu, and Qiang Yang, Fellow, IEEE

Abstract—A major problem of classification learning is the lack of ground-truth labeled data. It is usually expensive to label new data
instances for training a model. To solve this problem, domain adaptation in transfer learning has been proposed to classify target
domain data by using some other source domain data, even when the data may have different distributions. However, domain
adaptation may not work well when the differences between the source and target domains are large. In this paper, we design a novel
transfer learning approach, called BIG (Bridging Information Gap), to effectively extract useful knowledge in a worldwide knowledge
base, which is then used to link the source and target domains for improving the classification performance. BIG works when the
source and target domains share the same feature space but different underlying data distributions. Using the auxiliary source data, we
can extract a “bridge” that allows cross-domain text classification problems to be solved using standard semisupervised learning
algorithms. A major contribution of our work is that with BIG, a large amount of worldwide knowledge can be easily adapted and used
for learning in the target domain. We conduct experiments on several real-world cross-domain text classification tasks and
demonstrate that our proposed approach can outperform several existing domain adaptation approaches significantly.

Index Terms—Data mining, transfer learning, cross-domain, text classification, Wikipedia.

1 INTRODUCTION

T EXT classification, which aims to assign a document to one

or more categories based on its content, is a fundamental
task for Web and document data mining applications,
ranging from information retrieval, spam detection, to online
advertisement and Web search. Traditional supervised
learning approaches for text classification require sufficient
labeled instances in a problem domain in order to train a high-
quality model. However, it is not always easy or feasible to
obtain new labeled data in a domain of interest (hereafter,
referred to as the target domain). The lack of labeled data
problem can seriously hurt classification performance in
many real world applications.
To solve this problem, transfer learning techniques, in
particular domain adaptation techniques in transfer learning,
are introduced by capturing the shared knowledge from
some related domains where labeled data are available, and
use the knowledge to improve the performance of data
mining tasks in a target domain. In transfer learning
terminologies, one or more auxiliary domains are identified
as the source of knowledge transfer, and the domain of
interest is known as the target domain. Much effort has been
dedicated to this problem in recent years in machine learning,
data mining, and information retrieval [1], [2], [3], [4], [5].
However, transfer learning may not work well when the
difference between the source and target domains is large. In
. The authors are with the Department of Computer Science and
Engineering, Hong Kong University of Science and Technology, Clear-
water Bay, Kowloon, Hong Kong.
E-mail: {wxiang, caobin, derekhh, qyang}@cse.ust.hk.
Manuscript received 31 Mar. 2009; revised 22 Sept. 2009; accepted 30 Oct.
2009; published online 4 Feb. 2010.
Recommended for acceptance by C. Zhang, P.S. Yu, and D. Bell.
For information on obtaining reprints of this article, please send e-mail to:
tkde@computer.org, and reference IEEECS Log Number
TKDESI-2009-03-0256.
Digital Object Identifier no. 10.1109/TKDE.2010.31.
1041-4347/10/$26.00 ß 2010 IEEE
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
particular, when the distribution gap between the source and
target domains is large, transfer learning can hardly be used
to benefit learning in the target domain [6]. For example,
when we use some financial documents as the source
domain and information technology documents as the target
domain, the differences are so large that the performance in
the target domain may decrease. Another problem is when
the source and target domains have a large divergence in
feature space; for example, the source data might be written
for one audience and the target data for another. In these
situations, traditional transfer learning might not work well.
We note that the above difficulty is caused by a so-called
information gap in domain adaptation tasks, which essen-
tially is a combination of feature space and data distribution
differences between the training and test data. Since
previous domain adaptation methods only focused on the
data on the source and target domains, they may fail to
connect the related but indirectly shared parts of the
domains. Our observation is that such a gap can potentially
be found and bridged using knowledge from other domains.
To solve this problem, we introduce a bridge between the two
different domains by leveraging additional knowledge
sources that are readily available and have wide coverage
in scope. This knowledge source can be a third domain, such
as Wikipedia or the Open Directory Project (ODP). For
example, the connection between commutative algebra and
geometry can be found through a large knowledge base on
algebraic geometry topics. Once we find such a knowledge
bridge, we can use the auxiliary data and semisupervised
learning methods to fill in the information gap.
In this paper, we apply semisupervised learning (SSL) to
domain adaption problems based on the use of the auxiliary
data. We take the labeled data from the source domain and
the unlabeled data from the target domain, as well as an
auxiliary data source such as the Wikipedia. We then apply
Published by the IEEE Computer Society
XIANG ET AL.: BRIDGING DOMAINS USING WORLD WIDE KNOWLEDGE FOR TRANSFER LEARNING
SSL to utilize the information contained in the unlabeled
data to help the classification task on the target data.
Although domain adaptation (DA) and SSL share similar
problem settings, directly using SSL to solve DA problems
may result in poor performance, as validated in [3], due to
the fact that SSL assumes that the distribution of the
unlabeled data is similar to that of the labeled data.
However, the existence of the information gap can make
the assumption invalid. Using our approach, we show that
with the extracted bridge for filling in the information gap,
SSL-based algorithms can be applied successfully to the
classification problems.
This paper introduces a novel domain adaptation
algorithm called BIG (Bridging Information Gap). Our
BIG algorithm requires that the source domain and the
target domain share the same feature space, but the
distributions between domains can be highly different. A
major contribution of our work is that we make use of a
large amount of worldwide knowledge to build a bridge for
linking the source and target domains, even when their
distribution differences are large. We conduct thorough
experiments on several real-world cross-domain text
classification tasks, and demonstrate that our proposed
approach can outperform a number of existing approaches
for classification, including nontransfer learning ap-
proaches, traditional domain adaptation approaches that
do not use the auxiliary data, and approaches that use the
auxiliary data in a naive manner. We show that, especially
in situations when the source and target domains are far
away from each other, our approach can outperform the
baseline methods significantly.
The remainder of the paper is organized as follows: In
Section 2, we first survey related works in domain
adaptation. In Section 3, we define the concept of informa-
tion gap and propose our algorithm BIG for filling in the
information gap. In Section 4, we demonstrate its effective-
ness through several cross-domain text classification tasks.
In Section 5, we conclude the paper and discuss some
directions for future research.
2 RELATED WORK
In this section, we briefly review some previously proposed
methods for solving the task of domain adaptation. Since
the Wikipedia is used as our auxiliary data for building the
information bridge, we also review some methods that
extract useful knowledge from Wikipedia and other similar
knowledge bases such as ODP. Finally, since our approach
to domain adaptation problem is highly related to semi-
supervised learning, we also briefly review this topic.
2.1 Domain Adaptation
Domain adaptation has attracted more and more attention in
the recent years. In general, previous domain adaptation
approaches can be classified into two categories [7]: instance-
based approaches [1], [2] or feature-based approaches [4],
[5], [8], [9].
Instance-based methods try to seek some reweighting
strategies on the source data, such that the source distribu-
tion can match the target distribution. Feature-based
methods try to discover a shared feature space on which
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
771
the distributions of different domains are pulled closer. Both
types are trying to discover the relation between source and
target domains within the scope of two domains. For
example, instance-based transfer learning models assume
that there is a subset of instances sharing similar distribu-
tions in different domains, and then they emphasize the
impact of these data in the models since they are more
“similar.” For the feature-based domain adaptation models,
they assume that different domains may share some features,
for instance, a subset of explicit features or implicit features.
Here, we consider some well-known instance-based
domain adaptation methods. Jiang et al. in [2] use the
instance weighting method for natural language processing,
where the method used is a type of importance sampling
method for solving sample selection bias problems [10]. Dai
et al. in [1] propose a boosting-style reweighting method
and provide different weighting schemes for data in
different domains. Other feature-based methods have been
developed and are compared to instance-based methods.
Daume et al. [11] propose a simple feature augmentation
method for NLP tasks. Blitzer et al. [12] use the Structural
Correspondence Learning model (SCL) to identify corre-
spondences among features from different domains by
modeling their correlations with pivot features that behave
in the same way for discriminative learning. They choose
the pivot features that are used to bridge two domains. Lee
et al. [13] use transfer learning on an ensemble of related
tasks to construct an informative prior on feature relevance.
They assume that features themselves have metafeatures
that are predictive of their relevance to the prediction task,
and modeled their relevance as a function of the metafea-
tures. Raina et al. [5] describe an approach to self-taught
learning that uses sparse coding to construct high-level
features using the unlabeled data. They first express the
unlabeled data instance with a sparse weighted linear
combination of the basis and emphasize the L1 norm. They
then use these features as input to standard supervised
classification algorithms. In [9], a co-clustering-based
classification algorithm called CoCC is proposed to classify
out-of-domain documents. The class structure is passed
through word clusters from the in-domain data to the out-
of-domain data. Additional class-label information given by
the in-domain data is extracted and used for labeling the
word clusters for out-of-domain documents. However, one
drawback of many previous works is that for the source and
target domains, the shared knowledge sometimes may be
quite limited and the relation between the two domains
cannot be fully exploited.
2.2 Data Mining with Online Knowledge Repository
A major component of our approach is to use online
knowledge repositories as auxiliary information sources to
help bridge the gap between the source domain and the
target domain. Therefore, we review some latest approaches
of data mining with online knowledge repositories.
In recent years, understanding and using online knowl-
edge repositories to aid real world data mining tasks has
become a hot research topic. There are more and more
works trying to use the Wikipedia for feature enrichment.
Gabrilovich and Markovitch [14], [15] try to use the Open
Directory Project (ODP) for feature enrichment in the text
772 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,
classification problem. They also show that using Wikipedia
as the external Web knowledge resource for feature
enrichment performs better than using ODP [16]. Gabrilo-
vich and Markovitch [18] try to explicitly represent the
meaning of texts in terms of a weighted vector of
Wikipedia-based concepts. Their semantic analysis is
explicit in the sense that they manipulate manifest concepts
grounded in human cognition, rather than latent concepts
used by LSA. In [19], a general framework for building
classifiers with hidden topics discovered from large-scale
data collections is proposed. The framework is mainly
based on latent topic analysis models like PLSA [20] and
LDA [21] and machine learning methods like maximum
entropy and SVMs. The underlying idea of such a frame-
work is that for each classification task, a very large external
data collection is collected and called a “universal data set,”
then a classification model on both a small set of labeled
training data and a rich set of hidden topics discovered
from that data collection is built. Currently, few previous
approaches have used auxiliary knowledge such as online
knowledge database for transfer learning or domain
adaptation. Wang et al. make an extension [22], [23] to the
feature-based transfer learning models via incorporating a
semantic kernel [17], [24] learned from Wikipedia. How-
ever, building a semantic kernel from the whole knowledge
base is costly. Huge cost could be saved by considering only
the “most useful” concepts to bridge the information gap.
Moreover, instance-based transfer, especially our method,
adds more interpretability into the transfer scheme and it is
easy to study “what kind” of instances are useful for
bridging the gap instead of the more compact and abstract
semantic kernel. In this paper, we propose to incorporate
the background knowledge efficiently from the instance-
based transfer perspective, which could also establish a
connection between the transfer learning problems and
traditional semisupervised learning problems.
Another collection of works are done by Zelikovitz and
Hirsh [25], [26] that are related to our research. These works
use some unlabeled data from a background knowledge
document collection to enhance document similarities [10]
or dimensionality reduction [26]. However, their target is to
reduce the data sparsity, while we are focusing on
connecting the different domains apart in the problem of
transfer learning.
2.3 Semisupervised Learning
Domain adaptation could also be viewed as transductive
transfer learning if the source domain and the target
domain had no information gap. In this case, the problem
can be reduced to a semisupervised learning problem.
However, when there is information gap, how to exploit
semisupervised learning is not clear. In this section, we first
review some semisupervised learning research works.
Semisupervised learning addresses the problem when the
labeled data are too few to build a good classifier and makes
use of a large amount of unlabeled data, together with a
small amount of labeled data to enhance the classifiers.
Many semisupervised learning algorithms have been
developed in the past 10 years. Some notable models
developed include co-training, transductive SVM, and some
other graph-based regularization methods like mincut [27].
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
VOL. 22, NO. 6, JUNE 2010
Cotraining [28] assumes that the features can be split into
two sets and each subset is sufficient to train a good
classifier. Unlabeled data in co-training help to reduce the
size of the version space. Transductive SVM [29] builds the
connection between class distribution and decision bound-
ary by putting the boundary in low density regions. The
goal is to find a labeling of the unlabeled data such that a
linear boundary has the maximum margin on both the
original labeled data and the unlabeled data. It can be
viewed as an SVM with additional regularization term on
unlabeled data. Graph-based semisupervised methods
define a graph where the nodes are labeled and unlabeled
examples and edges reflect the similarities between exam-
ples. Blum and Chawla [27] view semisupervised learning
as a graph mincut problem, where positive labels act as
sources and negative labels act as sinks and the mincut
problem aims to find a minimum set of edges whose
removal would block all flow from sources to sinks. Nodes
connected to the sources would be labeled as positive and
those connected to the sinks would be labeled as negative.
In this work, we will exploit the ability of semisupervised
learning to aid the problem of domain adaptation.
3 METHODOLOGY
3.1 Preliminaries
In this section, we provide several definitions to clarify our
terminology and present our analysis on the problem of
domain adaptation.
Definition 1 (Feature Space). A feature space is an abstract
space where each data instance is represented as a point in an
n-dimensional space. Its dimension is determined by the
number of features used to describe the instances.
In the problem of text classification, the most commonly
used feature space is the term space used in the vector space
model [30]. The term space is usually of very high
dimension. For example, in the 20Newsgroup data set,1 the
dimension of the vector space model is over 40,000. To solve
the high dimensionality issue, topic models are introduced
for text modeling [20]. Then documents can be represented
in a low dimensional topic space. However, one disadvan-
tage of topic space is that the topics are hidden and it is not
easy to obtain their semantic meanings. Therefore, recently
the space of Wikipedia concept is introduced as the feature
space for text modeling [18]. Due to the existence of rich
background knowledge in the Wikipedia concept space,
even short texts can be accurately modeled [18].
Definition 2 (Domain). A domain D is a probabilistic
distribution PD over the data instances in a feature space.
Let x represent the feature vector of one data instance in
the feature space, then when we say a data instance x is in
the domain x 2 D, it means that it is sampled from the
distribution PD . A data set in the domain D is a set of data
instances sampled from PD . We slightly overuse the
notation D to represent both the domain and the data set
in the domain.
1. http://people.csail.mit.edu/jrennie/20Newsgroups/.
XIANG ET AL.: BRIDGING DOMAINS USING WORLD WIDE KNOWLEDGE FOR TRANSFER LEARNING
In traditional supervised learning problems, we are
given a set of labeled instances from a specific domain as
a training set. A learning machine is trained on the training
set and will be applied on newly incoming instances from
the same domain to obtain their labels. The condition that
training and test sets are drawn from the same domain
guarantees the consistence and generalization ability of the
learning machine [31]. However, in practice we may not be
able to ensure that the training and test sets are from the
same domain.
Definition 3 (Domain Adaptation). In this paper, we refer to
domain adaptation as the following problem: We are given a
set of labeled instances Dsl ¼ fðxi ; yi ÞgN s
i¼1 2 D  f1g from
a source domain. However, we need to make prediction for
some unlabeled data Dtu ¼ fxj gM t
j¼1 2 D from a target domain.
The source domain and target domain are different domains in
the same feature space. Such a problem setting for domain
adaptation is quite common in a variety of applications, such
as text categorization [1], image classification [5], sentiment
analysis [12], localization [32], [33], activity recognition [34]
and so on. Since all data instances in the source domain are
labeled and data instances in the target domain are unlabeled,
we can remove the subscript in the notations Dsl and Dtu and
use Ds and Dt without introducing any ambiguity.
The problem of domain adaption is likely to be encoun-
tered in many real-world applications. For example, we may
have trained a sentiment classifier for reviews on movies but
we want to use it to classify reviews from other domains
such as books or music [12]. Another example is that we may
have trained a classifier to classify news into topical
categories but we want to use it also on blogs. In these
cases, we do not want to relabel the data in the new domains
but hope to borrow the knowledge from the old domains.
When the differences between the source and target
domains are large, the model trained on the source domains
cannot generalize well for the target domain data [2], [9]. A
natural approach to follow is to consider transductive (or
semisupervised) learning, since unlabeled data from the
target domain is available. However, some previous works
have found that after introducing some unlabeled data in the
target domain, transductive learning is still not sufficient in
improving the performance [1]. The reason may be that
transductive or semisupervised learning generally assumes
that the decision boundary lies in the low-density region of
the feature space.2 When the distributions of source and
target domains are different, there may exist a low density
region between different domains which is a gap that
disconnects the same-class data in different domains. We
refer to this gap as the information gap in domain adaptation.
For example, Fig. 1 illustrates an example where the feature
space is the Wikipedia concept space. From Fig. 1, we could
see that there exist some information gaps between the
source domain and target domains.
To solve the problem of domain adaptation under large
information gaps, an intuitive idea is to find the shared part
of different knowledge between the domains, and ignore
the differences. One instantiation of this idea is to make use
2. Some semisupervised learning algorithms have slightly different
assumptions.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
773
Fig. 1. Measuring similarities between documents based on the
knowledge base.
of the abundant and potentially useful information sources
that are around, and use them to connect the information
separated by the gap. Such an intuition motivates us to
think of a different way for solving the domain adaption
problem, i.e., through finding an information bridge.
3.2 Margin as Information Gap
An intuitive way to understand the concept of information
gap is to consider separability of the source and target
domains. Consider the simplest case when we want to
transfer knowledge from a single source domain to a target
domain. Intuitively, the difficulty in separating these
domains shows how large the information gap is between
them. If the two domains can be easily separated, then there
exists a large information gap between them, which may
prevent our adapting the original learned model from the
source to the target domain. On the contrary, if the two
domains cannot be separated from each other easily, then
the information gap is small, in which case we can treat the
two domains as essentially data that are sampled from a
single underlying distribution. In other words, the original
“domain adaptation problem” is transformed into a
classification problem under the supervised setting or a
semisupervised (transductive) setting. A similar idea is
used in [6] where a classifier is trained to distinguish the
source and target domains and the classification error is
used as an empirical estimation for domain distance.
Although this idea is useful, it did not consider the
existence of auxiliary information sources that can be used
to bridge two domains.
Following the above intuitive idea, we use GðDs ; Dt ; KÞ
to denote the information gap between source domain Ds
and target domain Dt when knowledge base K is available.
In the case where no other knowledge is obtainable besides
the two domains, we use the notation GðDs ; Dt Þ. Previous
works on domain adaptation only focused on modeling
GðDs ; Dt Þ and ignored K. In contrast, we are modeling
GðDs ; Dt ; KÞ directly in this work.
We first consider the case where K is not available.
Since the concept of margin in SVM can be used to
measure the separability between two classes in classifica-
tion problems, therefore, we can define one form of
information gap of two domains GðDs ; Dt Þ to be the margin
between the source domain and target domain, when
treating them as two classes.
774 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 22, NO. 6, JUNE 2010

Definition 4 (Information Gap with No Background consists of some unlabeled data that are chosen so that we
Knowledge Available). The definition of information gap can apply semisupervised learning algorithms for training a
between Ds and Dt without background knowledge is given by classifier. These unlabeled data carry important information
the following equation: about the distribution between the two domains.

2 Algorithm 1. BIG: A Greedy Algorithm for Min-Margin

GðDs ; Dt Þ ¼ ð1Þ Domain Adaptation
kwk
Input: Source domain data set Ds , target domain data set
Dt , a knowledge base K, terminating threshold
w ¼ arg min kwk2 s:t: yi ðwT xi  bÞ  1 for all i; ð2Þ
t, relevance threshold 
where xi 2 Ds [ Dt , yi ¼ 1 if xi 2 Ds and yi ¼ 1 if xi 2 Dt . Output: A subset D from K

Given the form of knowledge bases as set of auxiliary data Initialize: Set D ¼ ;.
K ¼ fx0i g from other domains, the margin between two Preprocess: Set D0 ¼ CandSelectðDs ; Dt ; K; Þ.
domains not only depends on the data from source and Train an SVM model using ðxi ; yi Þ, where xi 2 Ds [ Dt ,
target domains which are treated as labeled data, but also
yi ¼ 1 if xi 2 Ds and yi ¼ 1 if xi 2 Dt .
depends on the auxiliary data K ¼ fx0i g, which are treated as
while D0 6¼ ; do
unlabeled data. The information gap can be defined as the
maximum margin under the transductive learning setting. for j = 1 to k do
Select xi from C satisfying jwT xi  bj < 1 by:
Definition 5 (Information Gap with Background Knowl- xi ¼ arg minxi jwT xi  bj
edge). The definition of information gap between Ds and Dt Let D ¼ D [ fxi g, D0 ¼ D0 =fxi g.
with background knowledge K is given by the following
end for
equation:
wold ¼ w
2 Train a TSVM model using ðxi ; yi Þ and D.
GðDs ; Dt ; KÞ ¼ ; ð3Þ 2
kwk G ¼ ðkwk  kw2old kÞ= kw2old k
if G  t then
where Output D and Exit.
w ¼ arg min kwk2 end if
  ð4Þ end while
s:t: yi wT xi  b  1; jwT x0i  bj  1 for all i:
A comprehensive example is illustrated in Fig. 2, where
Given the above definition, reducing the information gap three domains are shown. Domain A is a source domain,
domain B is the target domain, and domain C is an auxiliary
can be expressed as selecting the set of unlabeled data fxi g
domain used to bridge domains A and B. Without any data
from K to minimize the margin. Therefore, it can be
from the domain C, the problem cannot be solved directly
formulated as,
using a transductive SVM method, since the information
max ½½minkwk2  gap is large. In applying the algorithm, some data instances
fx0i gK from the auxiliary domain C are selected into the unlabeled
data sets which reduce the information gap between the
    two domains. Finally, when the algorithm converges, some
s:t: yi wT xi  b  1; j wT x0i  b j  1 for all i: ð5Þ
data are selected from the domain C. These data can be
We can also extend the problem by adding slack variables used to fill the gap between domains A and B.
for inseparable problems, where the optimal boundary is
obtained by 3.4 Algorithm Convergence
"" ! In this section, we show that the BIG algorithm is guaranteed
2
Xl to converge. We first prove several properties the margin-
w ¼ arg min kwk þ C 1  yi fðxi Þ based information gap has. Given the Definitions 4 and 5, we
f
i¼1 þ
!## ð6Þ show the following properties for information gap.
X
n
þ ð1  jfðx0i ÞjÞþ ; Lemma 1 (Nonincreasing Margin Lemma). Given the
lþ1 definition of information gap, we have
 
where ðzÞþ ¼ maxðz; 0Þ. However, the above max-min GðDs ; Dt Þ  G Ds ; Dt ; K :
problem is difficult to solve directly. Finding the optimal
solution is an NP-hard problem and is impractical. In the Furthermore, if we have K1  K2 , then
next section, we propose an algorithm that can optimize (5)  
in a greedy manner. GðDs ; Dt ; K1 Þ  G Ds ; Dt ; K2 :

3.3 BIG: A Min-Margin Algorithm to Reduce Proof. We can show that if the conclusion does not hold,
Information Gap which means GðDs ; Dt Þ < GðDs ; Dt ; KÞ, then GðDs ; Dt Þ is
Algorithm 1 shows our proposed solution. Its inputs include not the maximum margin. This conflicts with the
the source domain data set Ds , target domain data set Dt and definition of GðDs ; Dt Þ. u
t
auxiliary domain data K. The output of the algorithm

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
XIANG ET AL.: BRIDGING DOMAINS USING WORLD WIDE KNOWLEDGE FOR TRANSFER LEARNING
Fig. 2. An illustration of the BIG algorithm. Domain A is the source domain; Domain B is the target domain. After adding the new auxiliary data during
each iteration, the information gap between domain A and B is reduced. After reducing the information gap, TSVM can successfully solve the domain
adaptation problem.
The above lemma indicates that the information gap
between two domains is nonincreasing when more and
more knowledge is available. This property is consistent
with our intuition. We can further obtain a stronger
conclusion as shown in the following lemma.
Lemma 2 (Decreasing Margin Lemma). We have
GðDs ; Dt Þ > GðDs ; Dt ; KÞ;
if 9xi satisfies that jwT xi  bj < 1. Furthermore, if we have
K1  K2 , then
GðDs ; Dt ; K1 Þ > GðDs ; Dt ; K2 Þ;
if 9xi 2 K2 and xi 62 K1 satisfies that jwT xi  bj < 1.
Proof. We can prove the converse-negative proposition of
this proposition. Suppose we have
GðDs ; Dt ; K1 Þ  GðDs ; Dt ; K2 Þ:
Let w1 and w2 be the separating planes for GðDs ; Dt ; K1 Þ
and GðDs ; Dt ; K2 Þ, respectively. It is easy to validate that
w2 is also a separating plane with margin 2=GðDs ; Dt ; K2 Þ
for K1 . Therefore, such xi does not exist, which satisfies
xi 62 K1 and jwT xi  bj < 1 but xi 2 K2 . u
t
This theorem shows that selecting the data that fall in the
margin can reduce the size of the margin. Algorithm BIG is
motivated by this lemma. We can further guarantee its
convergence as shown in the following theorem:
Theorem 1 (Convergence Theorem). The iteration process in
Algorithm 1 is always increasing for the objective in (5) until
convergence.
The proof of the above theorem can be easily obtained
from Lemma 2.
3.5 Selecting Candidates from Auxiliary Knowledge
Base
In this section, we make our algorithm more concrete by
considering the data as text documents.
Based on our proposed algorithm in the previous
section, theoretically we are able to directly retrieve some
unlabeled documents from the knowledge base to bridge
domains. However, in practice, in order to fill in the
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
775
information gap, we need to involve a large auxiliary
knowledge base for better coverage. However, it is very
costly to perform the TSVM algorithm for searching the
bridging documents over a large knowledge base. There-
fore, it is necessary for us to identify some possible related
subsets as candidates first. For example, in Fig. 2, we are
only interested in retrieving those relevant unlabeled
documents which are “helpful” to connect a domain A
with another domain B. An intuitive idea to estimate the
helpfulness is to check whether one unlabeled document
stays “close” to both of the domains. This can be done by
calculating the similarity between these documents. The
most intuitive idea to identify the related domain data is to
measure the similarity between the domain documents and
each article in the knowledge base [18].
However, such a method is computationally expensive,
since it requires that we pair up these documents with each
article in the knowledge base from over one million articles.
To address this problem, we introduce a novel method to
solve this problem by unifying the idea of topic models [21]
and the knowledge-base document space projection [18]. To
this end, we apply Latent Dirichlet Allocation (LDA) [21] to
construct a topic space over the knowledge base docu-
ments. LDA is a generative graphical model which can be
used to model and discover underlying topic structures of
any discrete data such as texts. In LDA, a document dm ¼
fwm;n gNn¼1 is generated by first picking a distribution over
m
topics m from a Dirichlet distribution DirðÞ, which
determines the topic assignment of the document dm . Then,
the topic assignment for each word placeholder ½m; n is
performed by sampling a particular topic zm;n from multi-
nomial distribution Multðm Þ. Finally, a particular word
wm;n is generated for the word place holder ½m; n by
sampling from multinomial distribution Multðzm;n Þ. In
order to estimate the parameters  and  for LDA, we
need to maximize the likelihood of the whole data
collection—i.e., the entire knowledge base K:
Y
pðKj; Þ ¼ pðdm j; Þ
dm 2K
Z Z Y
Nm
¼ pðjÞ pðwm;n jzm;n Þpðzm;n jm Þpðm jÞddm :
n¼1
ð7Þ
776 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,
However, the two integrations in (7) are intractable. We
use Gibbs Sampling for approximation [35] here (due to space
limitation, the details of the Gibbs Sampling algorithm for
LDA are omitted.) When learning parameters  and , we can
obtain the topic association of each word in the documents in
our knowledge base. Thus, we are able to obtain the
document-topic association and the word-topic association,
represented as pðzj jdi Þ and pðwjzj Þ, respectively. Actually,
LDA summarizes the documents fdi gN
i¼1 in the whole
knowledge base to a few latent topics fzj gkj¼1 . It can be
viewed as a soft clustering over the documents in the
knowledge base where each hidden topic zj can be viewed
as a cluster of documents. We can also obtain the conditional
probabilities of generating document di from a topic zj with
Bayes rule:
pðzj jdi Þpðdi Þ
pðdi jzj Þ ¼ PN ; ð8Þ
i¼1 pðzj jdi Þpðdi Þ
where pðzj jdi Þ is the topic proportion provided by the topic
model. We assume a uniform prior distribution for pðdi Þ.
Using the word-topic association pðwjzj Þ, we can also
infer the hidden topic for the newly incoming document d0 :
Q end for
pðd0 jzj Þpðzj Þ pðzj Þ w2d0 pðwjzj Þ
pðzj jd0 Þ ¼ Pk ¼ ; ð9Þ
0
j¼1 pðd jzj Þpðzj Þ
Z
where pðzj Þ is the prior of hidden topic zj , and Z is the
normalization factor. Then we are able to obtain the
similarity between a new document d0 and the document
di in the knowledge base:
X
k illustrate that we can successfully discover the information
pðdi jd0 Þ ¼ pðdi jzj Þpðzj jd0 Þ:
j¼1
Consider an example to illustrate how to calculate the
similarity between a Wikipedia article and a newly incoming
document. We assume that the latent topics might refer to the
following four aspects: <“Wearing”, “Outdoor Sports”,
“Game Hardware”, “Maths”> . Consider one article with
the title “Xbox NBA Live 2008,” which refers to a video game.
During the learning process of the topic model, each word in
this paper is assigned to one of the four latent topics. After
normalization, the topic distribution of this document can be
represented by a vector <0:3; 0:5; 0:7; 0:2>, which denotes
the strength of the soft association between this paper and
the four latent topics. Similarly, when a newly incoming
document talking about “Jordan Shoes” comes, using (9), it
can also be represented with the latent topic vectors as
<0:8; 0:6; 0:1; 0:1>. Finally, we can calculate the similarity
between “Xbox NBA Live 2008” and “Jordan Shoes” on the
latent topic space using (10), that is 0:3 0:8 þ 0:5
0:6 þ 0:7 0:1 þ 0:2 0:1 ¼ 0:63.
Our above method differs from the method used in [18]
in that the inner product is calculated at the topic space
instead of the term space. Therefore, the computational cost
of document projection can be greatly reduced.
Finally, in order to evaluate the relatedness of a document
with a domain, we aggregate the conditional distribution of
all the data in the same domain Dt or Ds :
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
VOL. 22, NO. 6, JUNE 2010
1 X  
pðdi jDs Þ ¼ p di jxsj ð11Þ
Z xs 2Ds
j
and we form the related domain data set C by collecting the
top M ranked candidates which are related to both Dt and
Ds according to their relevance score:
Scoreðdi jDs ; Dt Þ ¼ pðdi jDs Þpðdi jDt Þ: ð12Þ
We can now select the most relevant documents based on
their scores. Algorithm 2 shows how to preprocess the entire
knowledge base to obtain a set of candidate documents.
Algorithm 2. CandSelect: Algorithm for Candidate
Selection
Input: Source domain data set Ds , target domain data set
Dt , entire knowledge base K, relevance threshold 
Output: A candidate set D0 from K
Initialize: Set D0 ¼ ;.
for each di in K do
if Scoreðdi jDs ; Dt Þ   then
Let D0 ¼ D0 [ fdi g.
end if
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of our
semisupervised learning approach on several real-world
domain adaptation tasks. To examine our approach in more
detail, we analyze the basic step of our algorithm to
ð10Þ gap. Furthermore, we demonstrate that our proposed
algorithm can reduce the information gap successfully
through a small number of iterations and converge to a local
optima efficiently.
4.1 Tasks and Data Sets
We conduct our experiments with three different domain
adaptation tasks. The first task is cross-domain text
classification on the 20 Newsgroups data set. The second
task is a domain adaptation problem in sentiment classifica-
tion and the third task is a cross-domain short text
classification task.
4.1.1 20 Newsgroups
The 20 Newsgroups dat set3 [36] is a text collection of
approximately 20,000 newsgroup documents partitioned
across 20 different newsgroups nearly evenly. Since we are
comparing our algorithm’s performance with that of [9], we
follow exactly the same experimental settings as given in
[9]. We create six different data sets for evaluating cross-
domain classification algorithms and for each data set, two
top categories are chosen, one as positive and the other as
negative. Data are split based on subcategories and
different subcategories are considered as different domains.
The six different data sets (shown in Table 1) are comp versus
sci, rec versus talk, rec versus sci, sci versus talk, comp versus rec,
and comp versus talk. Detailed descriptions of these data sets
3. http://people.csail.mit.edu/jrennie/20Newsgroups/.
XIANG ET AL.: BRIDGING DOMAINS USING WORLD WIDE KNOWLEDGE FOR TRANSFER LEARNING
TABLE 1
Data Description—20 Newsgroups
can also be found in [9]. For preprocessing details, all the
alphabets are converted into lower cases and stemmed
using the Porter stemmer. Stop words are also removed and
the document frequency (DF) feature is used to cut down
the number of features.
4.1.2 Sentiment Reviews
The data of sentiment domain adaptation [12] consist of
Amazon product reviews for four different product types,
including books, DVDs, electronics, and kitchen appliances.4
Each review consists of a rating with scores ranging from 0
to 5, a reviewer name and location, a product name, a
review title and date and the review text. Reviews with
ratings higher than three are labeled as positive and
reviews with ratings lower than three are labeled as
negative, the rest are discarded since the polarity of these
reviews are ambiguous. The details of the data in different
domains are summarized in Table 2. Such experimental
settings are the same as that of [12]. To study the
performance of our approach in this task, we construct
12 pairs of cross-domain sentiment classification tasks, e.g.,
we use the reviews from domain A as training data and
then predict the sentiment of the reviews in the domain B.
4.1.3 Query Classification
We also construct a set of tasks on cross-domain query
classification for a search engine. We use a set of search
snippets gathered crawled from Google as our training data
and and some incoming unlabeled queries as the test data.
The detailed descriptions of the process can be found in
[19]. We use the labeled queries from AOL provided by [37]
(shown in Table 3) for evaluation.5 We consider queries
from five classes: Business, Computer, Entertainment, Health,
and Sports which are shown in both of the training and test
data sets. We form 10 binary classification tasks for query
classification and queries are enriched according to [38],
[39], where we gather the top 50 search snippets for the
query enrichment.
4.2 Evaluation Metrics
To compare the performance of the classification methods,
we use classification accuracy, which is defined as the
percentage of correct predictions among all test examples. In
order to validate the robustness of our method, for each time
we randomly sampled 90 percent training data for training
the model. Each experiment was repeated ten times, and both
the mean and variance of the accuracies are reported.
4. http://www.seas.upenn.edu/mdredze/data-sets/sentiment/.
5. http://gregsadetsky.com/aol-data/.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
777
TABLE 2
Data Description—Amazon Product Reviews
TABLE 3
Data Description—AOL Queries
For projecting the domain data onto the latent topic
space shared with the documents in the knowledge base,
we train a topic model using the documents in the
knowledge base using the GibbsLDA++6 package. In [19],
they used the similar idea of using the topic model learned
from the knowledge base to reduce the data sparseness
problem, and they also investigated the effect of different
parameter settings for learning a topic model from
Wikipedia. Following their conclusion, we set the para-
meter of topic dimension as 100, and use the default hyper-
parameter settings, that is  ¼ 50=K (K is the number of
topics), and  ¼ 0:1. We follow such empirical parameters
described in [19] for simplifying our parameter settings.
4.3 Experimental Results
In this section, we answer the following two questions,
empirically. 1) Can our min-margin-based semisupervised
learning approach outperform traditional transfer learning
approaches on the domain adaptation tasks? 2) Can our
algorithm automatically identify the most important docu-
ments connecting the source domain to the target domain?
4.3.1 Question 1: Comparison with Traditional Transfer
Learning Methods
To answer the first question, we aim to compare our
algorithm performance against some baseline algorithms
presented in previous transfer learning papers (shown in
Tables 4, 5, and 6). To evaluate the effectiveness of our
approach, we adopt three classification models for domain
adaptation: the first model is Support Vector Machines
(SVM), which is usually used for supervised learning; the
second model is Transductive SVM (TSVM) [29], which is a
semisupervised learning model; and the last model is Co-
Cluster-based Classification (CoCC) [9] which is a transfer
learning model designed for cross domain classification. For
all of the three baseline models, we only use the labeled
documents in the source domain and unlabeled documents
in the target domain for training, and evaluate the model on
the test documents in the target domain. Since our BIG
algorithm is based on TSVM, in order to obtain the optimal
6. http://gibbslda.sourceforge.net/.
778 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 22, NO. 6, JUNE 2010

TABLE 4
Accuracy on 20NG Data

TABLE 5
Accuracy on Sentiment Data

TABLE 6
Accuracy on Web Query Data

parameter settings for SVM, TSVM, and BIG, we first tune
the parameter C to achieve the optimal accuracy via 10-fold
cross validation on each data set individually, and we
report the optimal results of different models on different
data sets in the following tables.
We first compare the experimental results on the
20 Newsgroups data, which are shown in Table 4.7 Each
row in Table 4 represents a different cross-domain text
classification task, including comp versus sci, rec versus talk,
rec versus sci, sci versus talk, comp versus rec, and comp versus
7. For all the tables, the best model for each data set is marked with bold
face. We also measure the statistical significance of the improvement using
Student’s t-test. * and ** mean that the improvement hypothesis are
accepted at different significant levels (p ¼ 0:05, 0.01).
talk, correspondingly. Each column denotes the mean and
variance of different models. We use bold face to mark the
best classifiers for each data set. We also add a column on
the right to denote our improvement to the best result in
baselines. For this and all subsequent experiments, the
Student’s t-test was applied to assess statistical significance
with respect to the best baseline method. We use the
notation ** to mark the rows in the improvement column in
the tables to indicate significance with p < 0:01; and use * to
indicate significance with p < 0:05; unmarked results
indicate no significant differences.
Comparing the results of SVM and TSVM, we find that
TSVM can boost the performance of SVM almost all the
time. When we involve the top 500 important Wikipedia

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
XIANG ET AL.: BRIDGING DOMAINS USING WORLD WIDE KNOWLEDGE FOR TRANSFER LEARNING
articles as the unlabeled data to fill in the information gap,
where the articles are selected by BIG, we are able to boost
the performance of TSVM even further. Furthermore,
amongst all six tasks, our algorithm can outperform the
CoCC algorithm.
Table 5 shows the experimental results on the sentiment
classification tasks. Each row in the table corresponds to a
different cross-domain review sentiment classification task,
where D, B, E, K corresponds to reviews on DVDs, books,
electronics, and kitchen appliances, correspondingly. A ! B
means we use reviews in domain A as training data and then
use reviews in domain B as test data. Polarity classification is
quite a different problem comparing with the traditional
genre categorization. We can find that, different from 20NG,
when the information gap between the domains becomes
larger, TSVM does not help a lot. This phenomenon might be
caused by the fact that simply pulling the data across
different domains together is not reasonable. We can find
that, our BIG algorithm, after filling in the information gap,
boosts the performance of TSVM. Almost for all tasks, our
BIG algorithm can outperform the TSVM method, again
suggesting that we are adding valuable information into the
unlabeled data set.
In Table 6, we show our result on cross-domain query
classification tasks. The results are similar to previous tasks,
where BIG outperforms TSVM most of the time, and SVM,
CoCC perform at the same level as TSVM.
4.3.2 Question 2: Comparison with Random Selection
Methods
To answer the second question, we can compare our BIG
algorithm with some Random Selection baselines. We
adopt two random selection methods for comparison. The
“Random 1” method randomly selects 500 data instances
from the entire knowledge base; The “Random 2” method
first uses the CandSelect algorithm mentioned in Section 3
to generate a candidate set, and then randomly selects
500 instances from the candidate set. Once we have
selected the 500 instances through either kind of Random
methods, the 500 instances are added to the training set
as unlabled data. Then we apply the TSVM model for
semisupervised learning. From Tables 4, 5, and 6, we can
observe that, the performance of our approach is much
better than both of the Random baselines, indicating that
we really found the most important nodes during the
domain drift process through which distribution is drifted
from the source domain to the target domain. We can
also observe that “Random 2” is better than “Random 1,”
since the Topic-model-based CandSelect algorithm can
really filter out a huge amount of irrelevant data from
the knowledge base.
All the experiments in different tasks are consistent and
validate the effectiveness of our proposed algorithm. In the
following we will investigate how our proposed method
can find where the bridging documents are located in the
knowledge base, and verify that our BIG algorithm could
converge and its performance is stable.
4.4 Information Gap
In this section, we propose to further analyze the effectiveness
of our algorithm by showing directly what “related domains”
we have actually found in our experiments. Here, we provide
the related domains, as top-ranked Wikipedia concepts
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
779
TABLE 7
Related Concepts in the Sentiment Reviews Task
which were extracted by BIG, from the sentiment domain
adaptation task. Detailed results are shown in Table 7.
From our results, we can see that the top-ranked
Wikipedia concepts contain rich content that could be used
to fill in the information gap between the two domains. Take
the cross-domain text classification task “D ! B” as an
example, the top-ranked Wikipedia concepts the algorithm
extract include concepts like “James Bond” and “Batman,”
which are good examples of concepts that contain knowl-
edge about both DVD domains and the Book domains.
Therefore, the detailed results of extracted related concepts
could also validate the effectiveness of our algorithm to fill
in the information gap from another perspective.
4.5 Convergence and Stability
We first demonstrate that our algorithm can reduce the
information gap between domains during the process of
including the unlabeled data from the related domains. We
randomly sample three tasks for each of the three data sets
and display the performance (shown in Fig. 4), together
with their corresponding margin sizes (shown in Fig. 3). For
each iteration, we include the top 100 unlabeled data that
are closest to the decision boundary for TSVM. The x-axis is
the iteration count. We find that our algorithm is able to
reduce the information gap and converge quickly. We also
list another group of figures to demonstrate the tradeoff
between CPU time and accuracy (shown in Fig. 5).
Since BIG is based on the semisupervised learning model
TSVM, a very sensitive parameter is C, which is the error
tolerance factor. It is true that the performance of BIG might
be different under different settings of C. Another para-
meter in our algorithm is the margin threshold t. Here, we
will discuss how to set the two parameters together. Since
our margin is closely related to the error tolerance factor C,
we also investigate how stable the algorithm performs
under different settings of C.
A very interesting observation is that our optimal margin
threshold is relatively stable over different data sets. Fig. 6
shows the optimal threshold under different C values. This
makes our method robust and reliable when applying to
different tasks. Thus, for different domain adaptation tasks,
we first use 10-fold cross validation on traditional TSVM to
determine the optimal parameter C. Then, based on the
parameter C, we use 10-fold cross validation again to seek
the optimal parameter of threshold t for halting the BIG
algorithm.
We also investigate the impact of the scope of related
domains. We randomly choose one task from each of the
780 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 22, NO. 6, JUNE 2010

Fig. 3. The change of information gap (margin) during iterations. We can observe that the information gap is reducing during iterations. (a) shows the
results on 20 Newsgroups; (b) shows the result on sentiment; (c) shows the results on AOL query set. The position of the data sets are the same in
the following figures.

Fig. 4. The change of accuracy during iterations. We observe that the performance converges within a few iterations.

Fig. 5. The trade-off between accuracy and CPU time.

Fig. 6. The relation between optimal threshold and parameter C.

Fig. 7. The relation between the number of auxiliary data and the best performance.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
XIANG ET AL.: BRIDGING DOMAINS USING WORLD WIDE KNOWLEDGE FOR TRANSFER LEARNING 781

TABLE 8
Different Knowledge Bases on 20NG Data

TABLE 9
Different Knowledge Bases on Sentiment Data

TABLE 10
Different Knowledge Bases on Web Query Data

three data sets. We vary the size of candidate related
documents from 4,000 to 10,000. We observe that the
performance is quite stable (shown in Fig. 7). This may be
due to the fact that we use the topic model to retrieve the
data from related domain as the candidate. The data
selection for filling in the information gap is achieved by
BIG, which means we can retrieve relevant data from
related domains as much as possible as the computational
cost is low. The BIG algorithm is able to discover the gap
and choose the useful data automatically.
4.6 Choice of Different Auxiliary Knowledge Bases
Since our algorithm does not depend on any specific
structure of the knowledge base, we are free to choose
any common knowledge repository available online, such
as Wikipedia8 or ODP.9 In this paper, we incorporate
Wikipedia as our external data source. Wikipedia is
currently the largest knowledge repository on the Web,
and the quality of its article content is remarkable due to the
open editing strategy. In our experiments, we use the
snapshot of Wikipedia on 30 Nov. 2006. There are totally
1,614,132 articles and 34,172,627 hyperlinks between them.
We preprocess the articles by stemming and stop words
removing. We also filtered out the short articles with length
less than 500, and finally there remain 53,803 articles as our
knowledge base. For ODP, we download its snapshot on
8. http://en.wikipedia.org.
9. http://www.dmoz.org/.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
782 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,
18 Oct. 2006, which covers 1,733,500 pages. We also filtered
out the short articles with length less than 1,000, and finally
there remain 59,986 articles as our knowledge base.
From Tables 8, 10, and 9, we can observe that our BIG
algorithm can always boost the performance of TSVM. We
also compare our BIG algorithm with the Random base-
lines, suggesting that we are selecting useful unlabeled data
in the semisupervised learning process. This further
demonstrates that it is necessary for us to design an
algorithm to carefully “pick out” the bridging documents.
5 CONCLUSIONS AND FUTURE WORK
In this paper, we proposed a novel framework for tackling
the problem of domain adaption under large information
gaps. We model the learning problem as a semisupervised
learning problem aided by a method for filling in the
information gap between the source and target domains with
the help of an auxiliary knowledge base (such as the
Wikipedia). By conducting experiments on different difficult
domain adaptation tasks, we show that our algorithm can
significantly outperform several existing domain adaptation
approaches in situations when the source and target domains
are far from each other. In each case, an auxiliary domain can
be used to fill in the information gap efficiently.
We make three major contributions in this paper.
1) Instead of the traditional instance-based or feature-based
perspective to view the problem of domain adaptation, we
view the problem from a new perspective, i.e., we consider
the problem of transfer learning as one of filling in the
information gap based on a large document corpus. We
show that we can obtain useful information to bridge the
source and the target domains from auxiliary data sources.
2) Instead of devising new models for tackling the domain
adaptation problems, we show that we can successfully
bridge the source and target domains using well developed
semisupervised learning algorithms. 3) We propose a min-
margin algorithm that can effectively identify and reduce
the information gap between two domains.
We plan to continue our research work on this direction in
the future, by pursuing several avenues. First, we plan to
validate the effectiveness of our approach through other
semisupervised learning algorithms and other relational
knowledge bases to more extensively demonstrate the
effectiveness of our approach. Second, in this paper, we
only investigate the case where the source, target and
auxiliary data sources share the same feature space. We plan to
extend our approach to be able to consider heterogeneous
transfer learning [40]. Finally, since our current approach is an
iterative model based on TSVM, which is quite slow for large
learning tasks, we will try to develop online TSVM methods
for incremental cross-domain transductive learning.
ACKNOWLEDGMENTS
The authors thank the support from Hong Kong CERG
grant 621307 and a grant from NEC China Lab.
REFERENCES
[1] W. Dai, Q. Yang, G.-R. Xue, and Y. Yu, “Boosting for Transfer
Learning,” Proc. 24th Ann. Int’l Conf. Machine Learning (ICML ’07),
pp. 193-200, June 2007.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
VOL. 22, NO. 6, JUNE 2010
[2] J. Jiang and C. Zhai, “Instance Weighting for Domain Adaptation
in NLP,” Proc. 45th Ann. Meeting of the Assoc. for Computational
Linguistics (ACL ’07), June 2007.
[3] G.-R. Xue, W. Dai, Q. Yang, and Y. Yu, “Topic-Bridged PLSA for
Cross-Domain Text Classification,” Proc. 31st Ann. Int’l ACM
SIGIR Conf. Research and Development in Information Retrieval
(SIGIR ’08), pp. 627-634, July 2008.
[4] A. Argyriou, C.A. Micchelli, M. Pontil, and Y. Ying, “A
Spectral Regularization Framework for Multi-Task Structure
Learning,” Proc. 21st Ann. Conf. Neural Information Processing
Systems (NIPS ’07), Dec. 2007.
[5] R. Raina, A. Battle, H. Lee, B. Packer, and A.Y. Ng, “Self-Taught
Learning: Transfer Learning from Unlabeled Data,” Proc. 24th
Ann. Int’l Conf. Machine Learning (ICML ’07), pp. 759-766, June
2007.
[6] J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman,
“Learning Bounds for Domain Adaptation,” Proc. 21st Ann. Conf.
Neural Information Processing Systems (NIPS ’07), Dec. 2007.
[7] S.J. Pan and Q. Yang, “A Survey on Transfer Learning,”
IEEE Trans. Knowledge and Data Eng., preprint, 12 Oct. 2009,
doi: 10.1109/TKDE.2009.191.
[8] S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, “Analysis of
Representations for Domain Adaptation,” Proc. 20th Ann. Conf.
Neural Information Processing Systems (NIPS ’06), pp. 137-144, Dec.
2006.
[9] W. Dai, G.-R. Xue, Q. Yang, and Y. Yu, “Co-Clustering Based
Classification for Out-of-Domain Documents,” Proc. 13th ACM
SIGKDD Int’l Conf. Knowledge Discovery and Data Mining (KDD ’07),
pp. 210-219, Aug. 2007.
[10] B. Zadrozny, “Learning and Evaluating Classifiers Under
Sample Selection Bias,” Proc. 21th Ann. Int’l Conf. Machine
Learning (ICML ’04), p. 114, July 2004.
[11] H.D. III, “Frustratingly Easy Domain Adaptation,” Proc. 45th Ann.
Meeting of the Assoc. for Computational Linguistics (ACL ’07), June
2007.
[12] J. Blitzer, M. Dredze, and F. Pereira, “Biographies, Bollywood,
Boom-Boxes and Blenders: Domain Adaptation for Sentiment
Classification,” Proc. 45th Ann. Meeting of the Assoc. for Computa-
tional Linguistics (ACL ’07), pp. 440-447, June 2007.
[13] S.I. Lee, V. Chatalbashev, D. Vickrey, and D. Koller, “Learning a
Meta-Level Prior for Feature Relevance from Multiple Related
Tasks,” Proc. 24th Ann. Int’l Conf. Machine Learning (ICML ’07),
pp. 489-496, June 2007.
[14] E. Gabrilovich and S. Markovitch, “Feature Generation for Text
Categorization Using World Knowledge,” Proc. 19th Int’l Joint
Conf. Artificial Intelligence (IJCAI ’05), pp. 1048-1053, July/Aug.
2005.
[15] E. Gabrilovich and S. Markovitch, “Harnessing the Expertise of
70,000 Human Editors: Knowledge-Based Feature Generation for
Text Categorization,” J. Machine Learning Research, vol. 8, pp. 2297-
2345, 2007.
[16] E. Gabrilovich and S. Markovitch, “Overcoming the Brittleness
Bottleneck Using Wikipedia: Enhancing Text Categorization with
Encyclopedic Knowledge,” Proc. 21th Nat’l Conf. Artificial Intelli-
gence and the 18th Innovative Applications of Artificial Intelligence
Conf. (AAAI ’06), July 2006.
[17] P. Wang, J. Hu, H.-J. Zeng, L. Chen, and Z. Chen, “Improving Text
Classification by Using Encyclopedia Knowledge,” Proc. Seventh
IEEE Int’l Conf. Data Mining (ICDM ’07), Oct. 2007.
[18] E. Gabrilovich and S. Markovitch, “Computing Semantic Related-
ness Using Wikipedia-Based Explicit Semantic Analysis,” Proc.
20th Int’l Joint Conf. Artificial Intelligence (IJCAI ’07), pp. 1606-1611,
Jan. 2007.
[19] X.H. Phan, M.L. Nguyen, and S. Horiguchi, “Learning to Classify
Short and Sparse Text & Web with Hidden Topics from Large-
Scale Data Collections,” Proc. 17th Int’l Conf. World Wide Web
(WWW ’08), pp. 91-100, Apr. 2008.
[20] T. Hofmann, “Probabilistic Latent Semantic Analysis,” Proc. 15th
Conf. Uncertainty in Artificial Intelligence (UAI ’99), pp. 289-296,
July/Aug. 1999.
[21] D.M. Blei, A.Y. Ng, and M.I. Jordan, “Latent Dirichlet Alloca-
tion,” Proc. 14st Ann. Conf. Neural Information Processing Systems
(NIPS ’01), pp. 601-608, Dec. 2001.
[22] P. Wang, C. Domeniconi, and J. Hu, “Using Wikipedia for Co-
Clustering Based Cross-Domain Text Classification,” Proc. Eighth
IEEE Int’l Conf. Data Mining (ICDM ’08), pp. 1085-1090, Dec. 2008.
XIANG ET AL.: BRIDGING DOMAINS USING WORLD WIDE KNOWLEDGE FOR TRANSFER LEARNING
[23] P. Wang, C. Domeniconi, and J. Hu, “Cross-Domain Text
Classification Using Wikipedia,” The IEEE Intelligent Informatics
Bull., vol. 9, pp. 5-17, Nov. 2008.
[24] P. Wang and C. Domeniconi, “Building Semantic Kernels for Text
Classification Using Wikipedia,” Proc. 14th ACM SIGKDD Int’l
Conf. Knowledge Discovery and Data Mining (KDD ’08), pp. 713-721,
Aug. 2008.
[25] S. Zelikovitz and H. Hirsh, “Improving Short-Text Classification
Using Unlabeled Background Knowledge to Assess Document
Similarity,” Proc. 17th Int’l Conf. Machine Learning (ICML ’00),
pp. 1183-1190, 2000.
[26] S. Zelikovitz and H. Hirsh, “Using LSI for Text Classification in
the Presence of Background Text,” Proc. 10th ACM Int’l Conf.
Information and Knowledge Management (CIKM ’01), pp. 113-118,
Nov. 2001.
[27] A. Blum and S. Chawla, “Learning from Labeled and Unlabeled
Data Using Graph Mincuts,” Proc. 18th Int’l Conf. Machine Learning
(ICML ’01), pp. 19-26, June/July 2001.
[28] A. Blum and T. Mitchell, “Combining Labeled and Unlabeled Sata
with Co-Training,” Proc. 11th Ann. Conf. Computational Learning
Theory (COLT ’98), pp. 92-100, 1998.
[29] T. Joachims, “Transductive Inference for Text Classification Using
Support Vector Machines,” Proc. 16th Int’l Conf. Machine Learning
(ICML ’99), pp. 200-209, June 1999.
[30] G. Salton, A. Wong, and C.S. Yang, “A Vector Space Model for
Automatic Indexing,” Comm. ACM, vol. 18, no. 11, pp. 613-620,
1975.
[31] C. Bishop, Pattern Recognition and Machine Learning. Springer-
Verlag, 2006.
[32] V.W. Zheng, S.J. Pan, Q. Yang, and J.J. Pan, “Transferring Multi-
Device Localization Models Using Latent Multi-Task Learning,”
Proc. 23rd Nat’l Conf. Artificial Intelligence (AAAI ’08), pp. 1427-
1432, July 2008.
[33] V.W. Zheng, E.W. Xiang, Q. Yang, and D. Shen, “Transferring
Localization Models Over Time,” Proc. 23rd Nat’l Conf. Artificial
Intelligence (AAAI ’08), pp. 1421-1426, July 2008.
[34] V.W. Zheng, D.H. Hu, and Q. Yang, “Cross-Domain Activity
Recognition,” Proc. 11th Int’l Conf. Ubiquitous Computing (Ubicom
’09), pp. 61-70, 2009.
[35] T.L. Griffiths and M. Steyvers, “Finding Scientific Topics,”
Proc. Nat’l Academy of Sciences USA, vol. 101, suppl. 1,
pp. 5228-5235, http://dx.doi.org/10.1073/pnas.0307752101,
Apr. 2004.
[36] K. Lang, “Newsweeder: Learning to Filter Netnews,” Proc. 12th
Int’l Machine Learning Conf. (ICML ’95), pp. 331-339, 1995.
[37] S.M. Beitzel, E.C. Jensen, O. Frieder, D.D. Lewis, A. Chowdhury,
and A. Kolcz, “Improving Automatic Query Classification via
Semi-Supervised Learning,” Proc. Fifth IEEE Int’l Conf. Data
Mining (ICDM ’05), pp. 42-49, Nov. 2005.
[38] D. Shen, J.-T. Sun, Q. Yang, and Z. Chen, “Building Bridges for
Web Query Classification,” Proc. 29th Ann. Int’l ACM SIGIR Conf.
Research and Development in Information Retrieval (SIGIR ’06),
pp. 131-138, Aug. 2006.
[39] A.Z. Broder, M. Fontoura, E. Gabrilovich, A. Joshi, V. Josifovski,
and T. Zhang, “Robust Classification of Rare Queries Using Web
Knowledge,” Proc. 30th Ann. Int’l ACM SIGIR Conf. Research and
Development in Information Retrieval, pp. 231-238, July 2007.
[40] Q. Yang, Y. Chen, G.-R. Xue, W. Dai, and Y. Yu, “Heterogeneous
Transfer Learning with Real-World Applications,” Proc. 47th Ann.
Meeting of the Assoc. for Computational Linguistics (ACL ’09), Aug.
2009.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on July 29,2010 at 13:16:22 UTC from IEEE Xplore. Restrictions apply.
783
Evan Wei Xiang received the BE degree in
software engineering from Nanjing University in
2006 and is working toward the PhD degree in the
Department of Computer Science and Engineer-
ing, the Hong Kong University of Science and
Technology. His research interests include large
scale data mining, transfer learning, and their
applications in Social Web mining. More details
can be found at http://ihome.ust.hk/~wxiang.
Bin Cao received the BS and MS degrees in
mathematics from Xi’an Jiaotong Univeristy and
Peking University, in 2004 and 2007, respec-
tively, and and is currently working toward the
PhD degree in the Department of Computer
Science and Engineering, the Hong Kong Uni-
versity of Science and Technology. His research
interests include data mining, machine learning,
and information retrieval. More details can be
found at http://ihome.ust.hk/~caobin.
Derek Hao Hu received the BS degree in
computer science from Nanjing University, in
2007 and is working toward the PhD degree in
the Department of Computer Science and
Engineering, the Hong Kong University of
Science and Technology. His research interests
include probabilistic graphical models, and their
applications in sensor-based activity recognition
and Web mining. More details can be found at
http://www.cse.ust.hk/~derekhh.
Qiang Yang received the bachelor’s degree in
astrophysics from Peking University and the
PhD degree in computer science from the
University of Maryland, College Park. He is a
faculty member in the Department of Computer
Science and Engineering, Hong Kong University
of Science and Technology. His research inter-
ests include data mining and machine learning,
AI planning, and sensor-based activity recogni-
tion. He is a fellow of the IEEE, member of AAAI
and ACM, a former associate editor for the IEEE TKDE, and a current
associate editor for IEEE Intelligent Systems. More details can be found
at http://www.cse.ust.hk/~qyang.
. For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/publications/dlib.