Generating Incremental ETL Processes Automatically
Xufeng Zhang, Weiwei Sun, Wei Wang, Yahui Feng, and Baile Shi
Department of Computing and Information Technology, Fudan University, 200433, China
011021380@fudan.edu.cn
Abstract
Incremental ETL processes are used for the
incremental maintenance of data warehouses, which is
generally designed by users with ETL tools. Using existing
methods of incremental maintenance of materialized views
for reference, we put forward an approach to generate an
incremental ETL process automatically from the full ETL
process in this paper. Existing researches are focused on
the incremental maintenance of materialized views in such
circumstances which involve the operators of selection,
projection, join and aggregation but difference operators
excluded. Since difference operators are used frequently in
an ETL process, we first discuss incremental maintenance
of materialized views defined with difference operators in
detail.
1. Introduction
Data warehouses collect data from multiple,
distributed sources and integrate the information for
querying and analysis. The process of collecting data from
data sources to a data warehouse is called ETL (Extraction,
Transformation and Loading) process.
Once a data warehouse is built, all the data are required to
be loaded in. This process is called full ETL process. In
daily maintenance, when changes occur in the data sources,
it is obviously not efficient to reload all the data using full
ETL process. We need only to load those data which are
newly created in the data sources, so incremental ETL
process is more efficient. At present, both full ETL process
and incremental ETL process are designed manually.
In [10], the author proposes that data warehouse can
be considered as the materialized views of data sources.
Correspondingly, full ETL process can be thought of as
view definition and incremental ETL process as the
incremental maintenance of materialized views which has
received much attention. The results of these researches
can be utilized to generate an incremental ETL process
automatically from the full ETL process.
But existing researches on incremental maintenance
of materialized views are focused on data warehouse
environments. Data in the data warehouse are organized
according to themes. In queries, the operators of
aggregation, selection, projection and join are frequently
used while difference operators are seldom used. So
Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06)
0-7695-2581-4/06 $20.00 © 2006 IEEE
existing researches are focused on the incremental
maintenance of materialized views in situations which
involve the operators of selection, projection, join and
aggregation but difference operators are excluded. In
ETL process, difference operators are frequently used to
eliminate useless data, which is different from the
maintenance of materialized views in data warehouse.
This paper research how to automatically generate
an incremental ETL process which involves operator of
aggregation, selection, projection, union, join and
difference. As a precondition, we first discuss the
incremental maintenance of various operators in which
we emphasized the operator of difference. Then we
discuss how to select the optimal auxiliary views to
implement the minimum cost in incremental
maintenance. At last we present the algorithm which
generates an incremental ETL process automatically
from the full ETL process.
The rest of this paper is organized as follows.
Section 2 is an introduction to related work. Section 3
introduces some basic conceptions needed in this paper,
including the canonical form for ETL process. Section 4
introduces the method of incremental maintenance of
AUSPJ segment and D segment, both of which are in
canonical form. Section 5 presents the method of
generating incremental ETL process from full ETL
process. Section 6 is the experiments and section 7 is the
summary and prospect of the research.
2. Related Work
In relational database, various types of operators
have various methods of incremental maintenance.
The incremental maintenance of projection and union
can be implemented with the same method. Since
several input tuples may correspond to one output tuple,
when one input tuple is deleted, we must check to see
whether there are some other input tuples that have the
same output as the deleted one. [2] uses bag (duplicate)
semantics to solve this problem. Another solution is to
attach an additional attribute to the output relation to
take count of the corresponding input tuples[3].
The incremental maintenance of join generally needs
full data of basic relations. But when a modification to
these basic relations only includes deletion and update,
the output relation is self-maintainable [1]. [5] discusses
the incremental maintenance of out-join, in which null
values must be considered.
 Aggregation is the most complicated, the incremental
maintenance of which depends on concrete aggregation
formula. SUM and COUNT are self-maintainable. AVG is
self-maintainable when the output relation adds an attribute
for counting [6] .incremental maintenance of MAX or MIN
needs to access base relation. But we can use some means
to reduce the visiting times of the base relation [7].
Some other literatures discuss the incremental maintenance
of composite operators. [4] discusses how to implement the
incremental maintenance of SPJ (Selection, Projection,
Join) views, [9] discusses how to implement the
incremental maintenance of ASPJ (Aggregation, Selection,
Projection, Join) views. [12] doesn’t consider specific
combinations of relational operators but think about how to
select the optimal materialized views in given storage
space, it is aimed to minimize the total cost of the queries
and incremental maintenance of views.
[8] discusses how to calculate the net increment of the
various basic relational operators(including difference
operator), but doesn’t involve the problem of incremental
maintenance.
[11] presents the canonical form for general view, but
the literature in itself doesn’t consider about incremental
maintenance.
There hasn’t been any literature on automatically
generating incremental ETL process, which is the research
content of this paper. Our research can take advantage of
the existing research results of incremental maintenance of
materialized views. But the existing research results don’t
include the incremental maintenance of difference operator.
In data warehouse environments, difference operator is
seldom used in the definition of materialized views; but in
the definition of ETL process, difference operator is used
as frequently as the other basic relational operators. So as
the precondition of our research, the incremental
maintenance of difference operator is also one of our
works.
3. Basic Conceptions
3.1. The Basic Conceptions of Incremental
Maintenance of Materialized Views
In the discussion of incremental ETL process, we need
some existing conceptions which we will introduce briefly
here, details can be found in [1] [4] [9].
Materialized View [1]: a view who’s tuples are stored in the
database.
Base Relation [1]: existing relation in the database,
which is used for creating views and materialized views.
Incremental view Maintenance [1]: a process to compute
only the changes in the view to update its materialization.
Irrelevant updates[4]: a set of updates which updates base
relations but has no effect on the state of a view.
Self-maintainable views [1]: views that can be maintained
using only the materialized view and incremental data.
Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06)
0-7695-2581-4/06 $20.00 © 2006 IEEE
Net Increment (Minimum Increment) [4]: an increment of
a materialized view containing only data of relevant
updates.
Primary Views [9]: the original materialized views stored
in warehouse.
Auxiliary Views [9]: additional materialized views which
store auxiliary data to reduce overall primary views
maintenance cost.
3.2. Canonical Form for ETL Processes
The ETL process which includes selection,
projection, union, join, difference and aggregation
operators can be canonicalized [11]. The canonical form
is composed of two type segments: AUSPJ-segments
and D-segments. More details can be found in [11], we
only give the results.
A D-segment contains a single difference operator.
An AUSPJ-segment, which is an operator sequence in
the form Į-‰-ʌ-ı-  , may omit any of the operators in
the operator sequence, but it must satisfy one of the
following:
z It has an aggregation operator on top.
z It is directly below a D-segment.
z It is below a join operator, and has a union
operator on the top.
z It is the top segment in the view definition,
which means that no more segments lies above
this segment.
Example 1 (Canonical Form for an ETL Process)
Suppose we have an ETL process to store customers’
total consume into data warehouse, but the target table
dose not contain VIP customers’ data. There are four
tables in data source, and one target table named
Total_Consume in data warehouse. The five tables list
as below.
Customer & VIP: C_ID int PRIMARY KEY,
C_NAME char(20)
Order_A & Order_B:
ORDER_ID int,
C_ID int REFERENCES Customer(C_ID),
PRODUCT_ID int,
P_NUM int,
P_PRICE int,
PRIMARY KEY (ORDER_ID, PRODUCT_ID)
Total_Consume: C_NAME char(20),
TOTAL_CONSUME int
The definition of the ETL process is shown in fig1(a),
and the canonical form is shown in fig1(b).
3.3. Self-maintainable Materialized Views and
Self-maintainable Net Increment
The self-maintainability of materialized views has
been discussed in detail in some literatures. All these
discussions rely on one precondition: the input
 relation(s) of a relation operator (or a set of relation
operators) are base relation(s), and the output relation is
materialized view. In an ETL process, the input and output
relations of a relation operator (or a segment) may be all
views. Using the results of self-maintainability of
materialized views, we can judge whether to materialize
the input relation(s), but can’t judge whether to materialize
the output relation. In order to judge simultaneously
whether to materialize the input and output relation, we put
forward a definition as follows.
Definition 1 self-maintainable net increment: a net
increment of a view or a materialized view can be obtained
only with the increment(s) of the base relation(s), and the
net increment is called self-maintainable net increment.
In an ETL process, if a net increment of a segment’s output
relation is self-maintainable, all the input and output
relations of the segment need not to be materialized. It’s
helpful to lower the executing cost of the incremental ETL
process. Obviously, if the net increment of a view is self-
maintainable, the view must be self-maintainable. But the
contrary is not necessarily true.
ĮC_NAME,SUM(T_PRICE)
ʌC_NAME, T_PRICE
 C_ID
ʌC_ID,P_NUM*P_PRICEas T_PRICE
 ‰
Customer VIP Order_A Order_B
4. Incremental Maintenance of One
Segment
According to section 3.2, the canonical form for
an ETL process is composed of AUSPJ-segments and
D-segments. In order to generate an incremental ETL
process from the ETL process, we should first consider
the implementation of incremental maintenance of one
segment.
4.1. Incremental Maintenance of AUSPJ-
Segment
We can use the existing methods to implement the
incremental maintenance of an AUSPJ-segment which
contains all the five kinds of operators. The methods can
be found in [1,3,4,5,7], here we only give the result.
To implement the incremental maintenance of a
complete AUSPJ segment, we must maintain
materialized views listed below:
All the input relations of join operator; The output
relation of projection and/or union operator: a counting
attribute is added to this relation or duplicate data are
allowed to be saved. But if we can sure that each tuple
in the output relation is unique, we need no any
materialized view;
The output relation of aggregation operator, and if
the formula is AVG, the output relation should add a
counting attribute. If the formula is MAX or MIN, the
input relation also should be materialized.
According to the definition of AUSPJ-segment, an
D AUSPJ-segment may omit some operators, and the
corresponding materialized views of which is omitted.
ĮC_NAME,SUM(
C_NAME,SUM(T_PRICE)
4.2. Incremental Maintenance of D-Segment
ʌC_NAME, T_PRICE
According to the definition, D-segment only contains a
single difference operator. Without any loss of
 C_ID
generality, consider a difference operation AB, and
suppose its output relation is C. According to the
Temp2 formulas in [8], C’s net increments are:
'C=('A(B‰'B))‰((A‰'A)ˆ’B) ’A (1)
Temp1 ’C=(’AB)‰(A‰'B) (2)
In formula (1) (2), relation A, B is used to calculate the
‰
net increment of C, so A, B must be materialized. If a
 view definition contains only one difference operator,
ʌC_ID,P_NUM*P_PRICE
_ID,P_NUM*P_PRICE as T_PRICE
the import relations are bass relations. So the method of
ʌC_ID,P_NUM*P_PRIC
C_ID,P_NUM*P_PRICE as T_PRICE
calculating C’s increment using A and B is called BRA
Customer VIP Order_A (Base Relations Access).
Order_B
Or
Formula C=AB is converted to C=A(AˆB). And for
E an arbitrary relation R, we have Rˆ'R=‡, Rˆ’R=’R.
Figure 1. Standardization of an ETL process (a) A common Let D=AˆB, E=B(AˆB), formula(11) (12) are
ETL process and (b) The canonical form for the ETL converted to
process
'C=('A(E‰'B))‰((D‰'A)ˆ’B) ’A (3)

Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06)
0-7695-2581-4/06 $20.00 © 2006 IEEE
 ’C=(’AD)‰(Cˆ'B) (4)
So when relation C, D and E are materialized, the
incremental maintenance of C can also be implemented.
And the method is named SRA (Split Relations Access).
SRA method is more efficient than BRA method. But in
SRA method, in order to maintain relation D and E, we
need to calculate the increment of them. Using 'C and ’C,
the formulas for the increment of D are:
'D=('A'C)‰(’C’A) (5)
’D=('C'A)‰(’A’C) (6)
Using 'D and ’D, the formulas for the increment of E are:
'E=('B'D)‰(’D’B) (7)
’E=('D'B)‰(’B’D) (8)
In a data warehouse environment, the increment is very
small, so the cost of maintaining D and E can be neglected.
And thus we should choice SRA method to implement the
increment maintenance of D-segment.
The Analysis of Self-maintainability of D-segment.
According to the formulas (1) (2), we get that difference
operation’s output relation is not self-maintainable. But
under some conditions, it can be self-maintainable, as the
following theorem states:
Theorem 1 Given a materialized view C and view
definition C=AB, when one of the three conditions is
satisfied, C is self-maintainable.
i) A’s increment only has ’A and B’s increment only has
'B;
ii) A’s increment only has ’A and B’s increment is null;
iii) A’s increment is null and B’s increment only has 'B.
Theorem 1 can be easily proved using formulas (1) (2).
Now we discuss another situation: given a difference
operation AB, and A and B satisfy B  A. The situation
frequently occurs in ETL process. And we have the
following theorem.
Theorem 2 Given a materialized view C and view
definition C=AB, when A and B satisfy B  A, the net
increments of C are self-maintainable.
Proof: Using B  A, and formulas (3) (4), we can get
'C=('A'B)‰(’B’A) (9)
’C=(’A’B)‰('B'A) (10)
Formulas (19) (20) contain only increments, so C’s
increments are self-maintainable.
According to Theorem 2, we get if in a D-segment, the
input relations A and B satisfy B  A, the input and output
relations need not to be materialized.
5. Automatic Generation of Incremental ETL
Process
The implementation of incremental ETL process
depends on the auxiliary views created when processing
full ETL process.
In section 4 we only consider the selection of auxiliary
views in single AUSPJ-segment and D-segment. But in
Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06)
0-7695-2581-4/06 $20.00 © 2006 IEEE
practice, an ETL process contains many AUSPJ-
segments and D-segments, in which the selection of
auxiliary views is more complicated. We know that
adding auxiliary views can lower the cost of calculating
increment. But it will increase the maintenance cost of
auxiliary views. Moreover, the storage space of a
database is generally finite, auxiliary views can’t be
increased infinitely.
Similar to the method in [12], we abstract the problem
of selecting auxiliary views into math’s models. Given a
full ETL process G, storage space S and auxiliary views
M, the execution cost of the incremental ETL process is:
k m
W (G, M ) ¦ D( D , M )  ¦U (V , M )  C
i 1
i
j 1
j
(11)
given auxiliary views M, Function D(Di,M) represents
the cost of calculating increment Di, function U(Vj,M)
represents the maintenance cost of auxiliary view Vj,
and C represents the maintenance cost of primary views,
it is a constant.
Meanwhile, the storage space S should satisfy:
m
¦V
j 1
j dS
(12)
The original problem now converts to: Under the
precondition of formula (12), select the auxiliary views
M to minimize the value of formula (11). In fact, this is
a NP-hard problem [12].
But there are differences between the incremental
ETL process and the incremental maintenance of
materialized views. First of all, the storage space of a
data warehouse is vast, so we can eliminate the
constraint in formula (12).
The queries in the data warehouse will only visit the
primary views. In order to further lower the lock time of
a data warehouse when executing incremental ETL
process, we partition it into two steps: First, calculate
net increments of the output relation of each operator
which is contained in the ETL process, and only updated
primary view; second, update all the auxiliary views.
We call the above method TSU(Two Step Update).
Using TSU method, the data warehouse need be locked
only in the first step. So we need only consider how to
lower the execution cost of the first step. Formula (12) is
simplified as follows:
k
W (G, M ) ¦ D( D , M )  C
i 1
i
(13)
Now we need only select the auxiliary views M which
can minimize the value of formula (25). To obtain this
aim, we need only assure that the cost of calculating the
net increment of the output relation of each operator is
minimum. Then we can solve the problem by selecting
appropriate auxiliary views for each operator according
to the method discussed in the last section.
 Using the method in [11], a canonicalized ETL process can
be described by a tree named ETL tree. Using the ETL tree,
we give the algorithm named MCCI (minimal cost of
calculating increment) to generate incremental ETL.
MCCI Algorithm CreateIncrementalETL(t0,V0)
Input a canonicalized full ETL process tree t0, a set of
view definition V0(each one associated with a
segment in t0)
Output a set of changed view definition V with materialized
views, a set of SQL statements S for incremental
ETL process
begin
VŇ , SŇ ;
do{get a top most unprocessed segment si from t0;
Get vi associated with si from V0;
if (vi is an AUSPJ segment)
system running on the PC with Pentium IV 2.0G CPU
and 512M memory, and kingbase database as the
DBMS. In the four base tables, each of
Customer,Order_A and Order_B contains 100000 tuples
and VIP contains 10000 tuples.
In the experiment, we also consider the optimization of
the incremental ETL process. Because the attributes
(ORDER_ID, PRODUCT_ID) are the primary key of
Order_A union Order_B, we keep the attributes in
relation TEMP_2, and we don’t need to attach an
additional attribute to the relation TEMP_2 for counting.
Because VIP is the subset of Customer, we use formula
(9) (10) to incremental maintain the relation TEMP_1.
The experiment’s result is shown in fig2.


W L PH RI  H[HFXW L RQ PV
for each operation in vi, add materialized views into 
vi 1 ; 
else for the difference operation in vi, add materialized 
views into vi 2 ; 
Generate SQL statements Si for incremental I XO O
 L QFU HPHQW
maintenance of vi; RSW L PL ]HG

VÅV‰vi;       
SÅS‰Si;
}
until (there is no unprocessed segment in t0);
return (V, S);
end;
The algorithm only scans the ETL tree once when
generating the incremental ETL process. If there are N
segments in the ETL tree, its running time is O(N). Let’s
suppose that when MCCI algorithm is selecting the method
of incremental maintenance, it assures that the calculating
cost of the increment of each segment is minimum. Then
the incremental ETL process generated by MCCI algorithm
is executed according to the first step of TSU method, the
execution cost is minimum. But in fact whether the
incremental maintenance cost is minimum depends on the
instance of the increment[1], so we can only assure that the
incremental ETL process generated by MCCI algorithm
has a good execution efficiency in general cases.
6. Experiments
We implement MCCI algorithm in an ETL tool and
compare the running time of the generated incremental
ETL processes with that of full ETL processes.
We choose example 1 shown in section 3 to do the
experiment, because the example includes join, projection,
union, difference and aggregation operators except
selection, it can show all the methods of incremental
maintenance. We choose WIN2003 server as the operating
1
The method to select materialized views AUSPJ-segment has been
discussed in section 4.1
2 We choose SRA method to maintain D-segment incrementally.
Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06)
0-7695-2581-4/06 $20.00 © 2006 IEEE
U DW L R RI  L QFU HPHQW  W R I XO O  
Figure 2. Time of executing full or incremental ETL
processes
In the result, we can get that when the increment is very
little, the running time of the incremental ETL processes
generated by MCCI algorithm is lower than that of the
full ETL process. And the optimized incremental ETL
processes take more little time.
7. Conclusion
This paper mainly discusses how to automatically
generate incremental ETL process according to the
existing full ETL process. As ETL process is similar to
the view definitions, we use the existing method of
incremental maintenance of materialized views for
reference to implement the automatic creation of
incremental ETL process.
First we use the method in [11] to canonicalize the ETL
process. And the canonicalized ETL process only
contains AUSPJ-segments and D-segments. Using
existing results of incremental maintenance of
materialized views, we give the method to incremental
maintain AUSPJ-segments.
We also discuss the incremental maintenance of D-
segment in detail. First we put forward the plain method
BRA which uses full information of the basic relations.
Secondly according to the hypothesis that “in a data
warehouse environment, the scale of general increment
is much smaller than that of basic relation”, we put
forward the improved method SRA. In SRA method,
 two auxiliary views are introduced to lower the total cost of
incremental maintenance. Then we discuss the self-
maintainability of the materialized views of difference
operator. The materialized views defined by difference
operator are not always self-maintainable. Through
analysis, we get three conditions in which the materialized
views are self-maintainable. We also discuss a common
difference operator in an ETL process, in which one input
relation is the subset of the other input relation. In this
situation, we prove that the net increment of the output
relation is self-maintainable.
At last we discuss how to automatically generate
incremental ETL process. In order to implement
incremental ETL process, we must select necessary
auxiliary views. And the selection of optimal auxiliary
views in given storage space is NP-hard. Since the storage
space of data warehouse is vast, we eliminate the
corresponding constraint and put forward the TSU method
which postpones the execution of updating on auxiliary
views. Finally, we simplify the formula which is used to
select auxiliary views. According to the simplified formula
we give the algorithm of automatically generating ETL
process.
The incremental ETL process used to be designed
manually. It is our work that enables it to be generated
automatically. And the output incremental ETL process has
a good performance in most situations.
[7] K. Yi et al., “Efficient maintenance of materialized top-k
views,” in Proc. ICDE 2003, Bangalore, India, 2003, pp. 189-
200.
[8] T. Griffin, L. Libkin, and H. Trickey, “An improved
algorithm for the incremental recomputation of active
relational expressions,” IEEE Trans. Knowl. Data Eng., 9(3),
1997, pp. 508-511.
[9] Y. Cui and J. Widom, “Storing auxiliary data for efficient
maintenance and lineage tracing of complex views,” in Proc.
DMDW 2000, Stockholm, Sweden, 2000.
[10] D. Agrawal, et al. “Efficient view maintenance at data
warehouses,” SIGMOD Conference 1997, Tucson, USA, 1997,
pp. 417-427.
[11] Y. Cui, J. Widom, and J.L. Wiener, “Tracing the lineage
of view data in a warehousing environment,” ACM
Transactions on Database Systems, 25(2), 2000, pp. 179-227.
[12] H. Gupta, “Selection of views to materialize in a data
warehouse,” in Proc. ICDT 1997, Delphi, Greece, 1997, pp.
98-112.

Acknowledgements

This work was supported by National Basic Research

Program of China (973 Program) 2005CB321905 and
National Science Foundation of China (NSFC) 60503035.
We thank all members (Yuan Hong, Yongrui Qin, et al. ) of
ETL TOOL research team for their contributions.

References
[1] A. Gupta and I. Mumick, “Maintenance of materialized views:
problems, techniques, and applications,” IEEE Data Eng.
Bulletin, Vol. 18, No. 2, 1995, pp. 3-18.
[2] A. E. Abbadi et al., “Performance issues in incremental
warehouse maintenance,” in Proc. VLDB 2000, Cairo, Egypt,
2000, pp. 461-472.
[3] I. S. Mumick and O. Shmueli, “Finiteness properties of
database queries,”,Advances in Database Research: Proc. of the
4th Australian Database Conference, Brisbane, Australia, 1993,
pp. 274-288.
[4] J. Blakeley, P. N. Larson, and F. Tompa, “Efficiently updating
materialized views,” ACM SIGMOD International Conference on
Management of Data, Washington, USA, 1986, pp. 269-400.
[5] A. Gupta, H. Jagadish, and I. Mumick, “Data integration using
self-maintainable views,” The 5th International Conference on
Extending Database Technology(EDBT), Avignon, France, 1996,
pp. 140–144.
[6] T. Palpanas, R. Sidle, R. Cochrane, and H. Pirahesh,
“Incremental maintenance for non-distributive aggregate
functions,” in Proc. VLDB 2002, Hong Kong, 2002, pp. 802-813.

Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06)
0-7695-2581-4/06 $20.00 © 2006 IEEE