Documente Academic
Documente Profesional
Documente Cultură
By Ram Reddy
Confidential
Contents
Confidential
28. Handson 25: Version Control 105
29. Handson 26: Modifying Knowledge Modules 108
30. Handson 27: Change Data Capture (CDC) 119
31. Handson 28: Migration (Exporting and Importing) 128
32. Handson 29: Security 138
33. Handson 30: CKM 141
34. Handson 31: Slowly Changing Dimension (SCD2) 153
Confidential
1. Data Warehouse Concepts
Data
Transactional Data
Analytical Data
Confidential
DW Architecture
DW Tools
1. DW tools are divided into two types. some of those tools are
ETL Reporting
Informatica/ODI OBIEE
DataStage Cognos
SSIS SSRS,SSAS
BODI SAP BO
OBIA
Confidential
5. Pre build Semantic layer RPD
6. Pre build Reports & Pre build Dashboards Web Catalog
OLTP Vs OLAP
OLTP OLAP
1. Is useful to store Transactional data 1. Is useful to store Analyatical data
3. The nature of data is current and 3. The nature of data is historical and
Detail summarized
6. It is volatile 6. It is nonvolatile
7. In OLTP data storage time is fixed 7. In OLAP data storage time is variant
10. In OLTP we will use normalizes 10. In OLAP we will use Denormalized
schema Schema
Confidential
RR ITEC #209,Nilagiri Block,Adithya Enclave,Ameerpet @8801408841,8790998182 7
Confidential
RR ITEC #209,Nilagiri Block,Adithya Enclave,Ameerpet @8801408841,8790998182 8
Confidential
Schemas
1. Star
2. Snow Flake
3. Constellation or mixed
1. Star Schema
1. Organizes data into a central fact table with surrounding dimension tables
2. Each dimension row has many associated fact rows
3. Dimension tables do not directly relate to each other
4. All Dimension Tables are de normalized
5. Optimized to read data
6. User friendly ,easy to understand
7. In OBIEE BMM layer only Star schemas are used
Confidential
Star schema fact
Confidential
Star Schema with Sample Data
Confidential
Snow Flake Schema
Confidential
Mixed Schema
1. It contains more than one fact with some common dimensions (Conformed Dimensions)
2. It is combination of some stars or some snows or both
Conformed Dimensions
Confidential
2. Introduction to Oracle Data Integrator
1. A widely used data integration software product, Oracle Data Integrator provides a new
Declarative design approach to defining data transformation and integration processes,
Resulting in faster and simpler development and maintenance.
2. Based on a unique E-LT architecture (Extract - Load Transform), Oracle Data
Integrator not only guarantees the highest level of performance possible for the
execution of data transformation and validation processes but is also the most cost-
effective solution available today.
3. Oracle Data Integrator provides a unified infrastructure to streamline data and
application integration projects.
A Unique Solution
Confidential
1. Faster and simpler development and maintenance: The declarative rules
driven Approach to data integration greatly reduces the learning curve of the
product and Increases developer productivity while facilitating ongoing
maintenance. This approach separates the definition of the processes from their
actual implementation, and separates the declarative rules (the "what") from the
data flows (the "how").
2. Data quality firewall: Oracle Data Integrator ensures that faulty data is
automatically detected and recycled before insertion in the target application.
This is performed without the need for programming, following the data integrity
rules and constraints defined both on the target application and in Oracle Data
Integrator.
3. Better execution performance: traditional data integration software (ETL) is
based on proprietary engines that perform data transformation row by row thus
limiting performance. By implementing an E-LT architecture, based on your
existing RDBMS Engines and SQL, you are capable of executing data
transformations on the target server at a set-based level, giving you much higher
performance
4. Simpler and more efficient architecture: The E-LT architecture removes the
need for an ETL Server sitting between the sources and the target server. It
utilizes the source and target servers to perform complex transformations,
most of when happen in batch mode when the server is not busy processing end-
user queries
5. Platform Independence: Oracle Data Integrator supports many platforms,
hardware and OSs with the same software.
6. Data Connectivity: Oracle Data Integrator supports many RDBMSs including
leading Data Warehousing platforms such as Oracle, Exadate, Teradata, IBM
DB2, Netezza, Sybase IQ and numerous other technologies such as flat files,
ERPs, LDAP, XML.
7. Cost-savings: The elimination of the ETL Server and ETL engine reduces both
the initial hardware and software acquisition and maintenance costs. The
reduced learning curve and increased developer productivity significantly reduce
the overall labor costs of the project, as well as the cost of ongoing
enhancements.
Confidential
3. ODI 12C Installation And Architecture
Architecture
Repositories:
1. Oracle Data Integrator Repository is composed of one Master Repository and several
Work Repositories.
2. ODI objects developed or configured through the user interfaces (ODI Studio) are stored
in one of these repository types.
3. Master repository stores the following information
Confidential
1. Security information including users, profiles and rights for the ODI platform
2. Topology information including technologies, server definitions, schemas,
contexts, languages and so forth.
3. Versioned and archived objects.
4. Work repository stores the following information
1. The work repository is the one that contains actual developed objects. Several
work repositories may coexist in the same ODI installation (example Dev work
repository ,prod work repository …etc )
2. Models, including schema definition, datastores structures and metadata, fields
and columns definitions, data quality constraints, cross references, data lineage
and so forth.
3. Projects, including business rules, packages, procedures, folders, Knowledge
Modules, variables and so forth.
4. Scenario execution, including scenarios, scheduling information and logs.
5. When the Work Repository contains only the execution information (typically for
production purposes), it is then called an Execution Repository.
Confidential
2. Operator Navigator: is the production management and monitoring tool. It is
designed for IT production operators. Through Operator Navigator, you can
manage your mapping executions in the sessions, as well as the scenarios in
production.
Confidential
4. Security Navigator: is the tool for managing the security information in Oracle
Data Integrator. Through Security Navigator you can create users and profiles
and assign user rights for methods (edit, delete, etc) on generic objects (data
server, data types, etc.), and fine tune these rights on the object instances
(Server 1, Server 2, etc).
Confidential
4. Hands on 1: Creating Master and Work Repository
Confidential
3. Select the Master Repository Creation Wizard click on ok
Confidential
5. Select the Master Repository Creation Wizard click on ok
6. Click on test connection click on ok Click on next
Confidential
9. Once Process completed
10. Go back to sql plus and execute select count(*) from tab and notice that 67
tables are created
Confidential
Create the Work Repository
Confidential
4. Click on test click on ok click on ok
5. Click on Topology expand repositories section
6. Right click on work repositories click on new work repository
Confidential
8. In conformation window click on yes
Confidential
11. In ODIWR schema observe that 153 tables created
Exercise: please complete \02 ODI\04 ORACLE REFERENCE\ Creating Master and
Work Repository by RCU.docx (Please do not run drop RCU Section)
Confidential
5. Hands On 2: Creating master and work repositories using RCU
1. Naviogate to C:\Oracle\Middleware\Oracle_Home\oracle_common\bin
2. Double click on rcu.bat file Click on RUN
3. In welcome screen click on next
4. Select create click on next
Confidential
7. Click on next click on ok
Confidential
6. Hands On 3: Configuring RRITEC database
6.1 Configuring source database
Step 1: Creating user TDBU and load tables into TDBU schema
Confidential
7. Hands On 4: Creating and Managing Topology
The Oracle Data Integrator Topology is the physical and logical representation of
1. The physical architecture defines the different elements of the information system, as
well as their characteristics taken into account by Oracle Data Integrator.
2. Each type of database (Oracle, DB2, etc.) or file format (XML, Flat File), or
application software is represented in Oracle Data Integrator by a technology.
3. The physical components that store and expose structured data are defined as
Data servers.
Process:
Confidential
3. Go to Right click on oracle Click on New Data Server
Confidential
4. In Definition tab provide below information
7. Close
Confidential
2. Provide below information
Confidential
7.2 Creating Logical Architecture:
1. The logical architecture allows a user to identify as a single Logical Schema a group
of similar physical schemas - that is containing data stores that are structurally
identical - but located in different physical locations.
2. Logical Schemas, like their physical counterpart, are attached to a technology.
3. All the components developed in Oracle Data Integrator are designed on top of the
logical architecture. For example, a data model is always attached to logical Schema
Process:
4. Similarly create one more new logical schema with the name of TARGET_TDBU
Confidential
7.3 Creating Contexts
With
Confidential
Process (optional):
1. By default one context created with installation of ODI with the name of Global
2. We already mapped logical and physical schemas while we were creating logical
schemas For just conformation right click on global context open observe mapping
Agents:
1. Oracle Data Integrator run-time Agents orchestrate the execution of jobs. The agent
executes jobs on demand and to start the execution of scenarios according to a
schedule defined in Oracle Data Integrator.
Languages:
1. Languages defines the languages and language elements available when editing
expressions at design-time.
Confidential
8. Hands on 5: Creating Model
1. Models are the objects that will store the metadata in ODI.
2. They contain a description of a relational data model. It is a group of DataStores
(Known as Tables) stored in a given schema on a given technology.
3. A model typically contains metadata reverse-engineered from the “real” data model
(Database, flat file, XML file, Cobol Copybook, LDAP structure ...etc)
4. Database models can be designed in ODI. The appropriate DDLs can then be generated
by ODI for all necessary environments (development, QA, production)
5. Reverse engineering is an automated process to retrieve metadata to create or update
a model in ODI.
6. Reverse- Engineering also known as RKM(Reverse engineering Knowledge Module)
7. Reverse engineering two types
1. Standard reverse-engineering
i. Uses JDBC connectivity features to retrieve metadata, then writes it to the
ODI repository.
ii. Requires a suitable driver
2. Customized reverse-engineering
i. Read metadata from the application/database system repository, then
writes these metadata in the ODI repository
ii. Uses a technology-specific strategy, implemented in a Reverse-
engineering Knowledge Module (RKM)
Confidential
8. Some other methods of reverse engineering are
1. Delimited format reverse-engineering
i. File parsing built into ODI.
2. Fixed format reverse-engineering
i. Graphical wizard, or through COBOL copybook for Mainframe files.
3. XML file reverse-engineering (Standard)
i. Uses JDBC driver for XML
9. Reverse engineering is incremental.
10. New metadata is added and Deleted metadata is removed.
Process:
Confidential
3. Provide below information Click on save Click on reverse engineer
Confidential
5. Similarly create one more data model with name of RRITEC_TARGET_MODEL
Confidential
9. Hands on 6: Creating Project
2. A folder is a hierarchical grouping beneath a project and can contain other folders and
objects.
3. Every package, Mapping, Reusable Mapping and procedures must belong to a folder.
4. Objects cannot be shared between projects. except (Global variables, sequences, and
user functions)
5. Objects within a project can be used in all folders.
6. A knowledge module is a code template containing the sequence of commands
necessary to carry out a data integration task.
7. There are different knowledge modules for loading, integration, checking, reverse
engineering, Service and journalizing.
8. All knowledge modules code will be executed at run time.
9. There are six types of knowledge modules:
Confidential
MAPPINGS
ings
P
Confidential
2. LKM Oracle to Oracle (DBLINK)
i. Loads from Oracle data server to Oracle data server
ii. Uses Oracle DB Link
3. CKM Oracle
i. Enforces logical constraints during data load
ii. Automatically captures error records
Confidential
10. Hands on 7: Markers
1. A marker is a tag that you can attach to any ODI object, to help organize your project.
2. Markers can be used to indicate progress, review status, or the life cycle of an object.
3. Graphical markers attach an icon to the object, whereas non graphical markers attach
numbers, strings, or dates.
4. Markers can be crucial for large teams, allowing communication among developers from
within the tool.
1. Review priorities.
2. Review completion progress.
3. Add memos to provide details on what has been done or has to be done.
4. This can be particularly helpful for geographically dispersed teams.
5. Project markers:
1. Are created in the Markers folder under a project
2. Can be attached only to objects in the same project
6. Global markers:
1. Are created in the Global Markers folder in the Others view
2. Can be attached to models and global objects
Process:
1. Under Designer Navigator under Project Right click on markers New Marker
Group name it as RRITEC order as 0 Add two markers as shown
2. Click on save
Confidential
11. Hands On 8: Create a Mapping using expression
In this exercise, the student will create a mapping that represents the data flow between the
EMPLOYEE source and the ODS_EMPLOYEE target.
A mapping represents the dataflow between sources and targets. The instructions defined in
the mapping tell the ODI Server how to read, Load and transform the data
1. Under RRITEC folder Right click on mappings Click on new mapping name it
as m_ODS_EMPLOYEE Click on ok
2. In Logical tab of the mapping drag and drop employee table from
RRITEC_SOURCE_MODEL
3. Drag and drop ods_employee table from RRITEC_TARGET_MODEL
Expression transformation is useful to do all calculations except aggregate calculations (ex sum,
avg ...etc)
Confidential
6. Develop below expression
Confidential
Step3: Run the mapping
Confidential
12. Hands On 9:Create a Mapping using Flat File and Filter
2. Create a notepad file with below data and save it as emp.txt Close file
Empno, sal,deptno
101,1000,10
102,2000,20
103,3000,30
Confidential
2. In Definition and JDBC tab provide below information Click on save
Confidential
Step4: Create Model based on File Logical Schema
Confidential
Step5: Create Target Table and adding to model
Confidential
4. Drag and drop Filter Transformation from Components pane
5. Drag and drop Deptno from emp to filter
6. Connect empno,sal,deptno from emp to temp
7. Select Filte object In filter Properties pane develop filter condition to get 10 and 20
Exercise: please complete \02 ODI\04 ORACLE REFERENCE\ Flat File to a Table.docx
Confidential
13. Hands On 10: Create a Mapping using Split
1. A Split is a Selector component that divides a flow into two or more flows based on if-
then-else conditions.
2. It is a new feature in ODI 12C
3. It is equivalent to ROUTER transformation in Informatica
4. It can be used in place of multiple conditions. The same row may be passed to multiple
flows
5. We can capture rejected records in to a table by selecting Remainder option
The exercises in this lab are designed to walk the student through the process of using a Split
transformation.
Objectives
Summary
In this lab, the student will develop a mapping where a row can be sent to one of three different
target tables. The mapping will use a Split transformation to group employee records based
on various criteria, then route each record to the appropriate target table.
SOURCE: ODS_EMPLOYEE
TARGETS: ODS_EMPLOYEE_SALES
ODS_EMPLOYEE_NON_SALES
ODS_EMPLOYEE_ROOKIE
Confidential
ODS_EMPLOYEE_SALES should contain records for employees where the TYPE_CODE
column contains the value ‘SALES’.
Process:
7. Drag and Drop all the columns one by one from ODS_EMPLOYEE table to
ODS_EMPLOYEE_SALES
8. Drag and Drop all the columns one by one from ODS_EMPLOYEE table to
ODS_EMPLOYEE_NON_SALES
9. Drag and Drop all the columns one by one from ODS_EMPLOYEE table to
ODS_EMPLOYEE_ROOKIE
Confidential
10. Click on Validate Save Run
Exercise: From emp table load deptno 10, 20 and 30 data into different tables .Please do
as shown below
Confidential
14. Hands On 11: Create a Mapping using Joiner
1. A join is a selector component that creates a join between multiple flows. The attributes
of all flows are combined as the attributes of the join component.
2. The following join types can be created.
3. A NATURAL JOIN is a JOIN operation that creates an implicit join clause for you based
on the common columns in the two tables being joined. Common columns are columns
that have the same name in both tables.
EMPNO NUMBER,
Confidential
JOB VARCHAR2(20),
SAL NUMBER,
DEPTNO NUMBER,
DNAME VARCHAR2(20)
);
Confidential
8. Click on Join Component Select required type of join by default it is Inner Join
Confidential
Exercise 2: Join EMP and DEPT tables using Right outer Join
Confidential
15. Hands On 12: Create a Mapping using Lookup
1. Use a lookup component using a source as the driving table and lookup another Data
Store.
2. Lookups appear as compact graphical objects in the interface sources diagram.
3. Choose how the each lookup is generated
a. SQL left outer join in the FROM clause: The lookup source will be left-outer-
joined with the driving source in the FROM clause of the generated SQL.
b. SQL expression in the SELECT clause: A SELECT-FROM-WHERE statement
for the lookup source will be embedded into the SELECT clause for the driving
source.Left Outer Join in the FROM clause
4. Lookup Works similar to Join
5. Benefits of using lookups
a. Simplifies the design and readability of interfaces using lookups
b. Enables optimized code for execution
);
Confidential
3. Reverse Engineer LKP_EMP_DNAME_LOC table into RRITEC_MODEL_TARGET
Confidential
16. Hands On 13: Create a Mapping using Sort
1. A sort is a selector component (see Selector Components) that sorts the incoming flow
based on a list of attributes.
2. The functionality is equivalent to the SQL ORDER BY clause.
3. Sort condition fields can be added by dragging source attributes onto the sort
component.
Confidential
7. Click on Physical tab Select target table EMP_SORT Select
CREATE_TARGET_TABLE value as True
Confidential
17. Hands On 14: Create a Mapping using Aggregate
1. Use to group and aggregate attributes using aggregate functions, such as average,
count, maximum, sum, and so on.
2. ODI automatically selects attributes without aggregation functions to be used as group-
by attributess.
3. You can override this by using the Is Group By Column and Manual Group By Clause
properties.
Confidential
5. Drag and drop Deptno from EMP table to SORT
6. Drag and drop Deptno,sal from EMP table to Aggregate
7. Select Aggregate and change expression as SUM(EMP.SAL)
Confidential
18. Hands On 15: Create a Mapping using Distinct
2. Create a notepad file with below data and save it as Region_country.txt Close file
REGION_ID,REGION,COUNTRY_ID,COUNTRY
20,South,1,USA
21,West,1,USA
22,East Coast,1,USA
23,Mid West,1,USA
24,South India,2,India
25,North India,2,India
Confidential
3. Right Click on Model RRITEC_FLATFILES Click on New Data Store
4. In Definition tab provide Name ,Alias ,Resource Name Click on save In Files Tab
Provide File Format ,Heading(No of Rows) ,Field Separator
Confidential
Step2: Create Target Table
Confidential
19. Hands On 16: Create a Mapping using Set Component
1. A set component is a projector component that combines multiple input flows into one
output flow by using set operation such as UNION, INTERSECT, EXCEPT, MINUS, and
others.
Confidential
1. Create a mapping with the name of m_ EMP_UNION102030
2. Drag and drop emp1020 and emp2030 from RRITEC_SOURCE_MODEL
3. Drag and drop EMP_UNION102030 from RRITEC_TARGET_MODEL
4. Drag and drop SET Components into work area
5. Select SET and change properties as shown
Confidential
20. Hands on 17: User Functions
Confidential
2. Provide Name ,Group and Syntax
DECODE($(DNAME),'ACCOUNTING','ACC','RESEARCH','R and
D','OPERATIONS','OPT','SALES')
Confidential
Step 3: Utilizing User Function in a mapping
5. Click on physical tab select target table under integration knowledge module
select create table property as true
6. Click on validate save run and observe output
Step1: Creating Flat File and reverse engineer into Source Flat File Model
Confidential
2. Reverse engineer into source flat file model
Step2: Creating Target table and reverse engineer into Target Model
Confidential
3. Click on implementations tab Click on provide below formula and select
technology
SUBSTR($(PHONE),7,6)
Confidential
5. Connect Ename of source to ENAME of target
6. Select target phone column and call user function
Confidential
21. Hands on 18: Variables
1. Variable – An ODI object which stores a typed value, such as a number, string or date.
2. Variables are used to customize transformations as well as to implement control
structures, such as if-then statements and loops, into packages.
3. Variables are two types
a. Global
b. Project Level
4. To refer to a variable, prefix its name according to its scope:
a. Global variables: GLOBAL.<variable_name>
b. Project variables: <project_code>.<variable_name>
5. Variables are used either by string substitution or by parameter binding.
a. Substitution: #<project_code>.<variable_name>
b. Binding: :<project_code>.<variable_name>
6. Variables are four types
a. Declare
b. Set
c. Evaluate
d. Refresh
7. Variables can be used in packages in several different ways, as follows:
a. Declaration: When a variable is used in a package (or in certain elements of the
topology which are used in the package), it is strongly recommended that you
insert a Declare Variable step in the package. This step explicitly declares the
variable in the package.
b. Refreshing: A Refresh Variable step allows you to re-execute the command or
query that computes the variable value.
c. Assigning: A Set Variable step of type Assign sets the current value of a
variable.
d. Incrementing: A Set Variable step of type Increment increases or decreases a
numeric value by the specified amount.
e. Conditional evaluation: An Evaluate Variable step tests the current value of a
variable and branches depending on the result of the comparison.
8. Variables also can be used in expressions of mappings, procedures and so forth.
Confidential
Exercise 1: Filter DEPTNO10 using default value/static assignment
Confidential
Exercise 2: Filter MIN DEPTNO using SQL Query result
3. Save close
Confidential
2. Truncate target table
3. Click on validate Click on save click on Run
4. Notice that no data loaded into target as we have no value in the variable .From this we
come to understand variable corresponding query will not execute with mapping run
Confidential
22. Handson 19: Sequences
Oracle Data Integrator reads the value, locks the row (for concurrent updates)
and updates the row after the last increment.
Confidential
Step2: Create Target Table
Confidential
6. Validate Save RunObserve output
Confidential
2. Provide Name: NAT_SEQ_EMP_ROW_WID
3. Select Native sequence select Schema target_tdbu Select native sequence name
as EMPID_SEQUENCE
4. Click on save
Confidential
3. Click on save Click on Run observe output
Confidential
23. Handson 20: Procedures
Procedure Examples:
Confidential
Commands or ODI objects that can be used in ODI procedures:
1. SQL Statements
2. OS Commands
3. ODI Tools
4. JYTHON Programs
5. Variables
6. Sequences
7. User Functions
8. …etc
Confidential
Let us see about OdiFileCopy
OdiFileCopy Syntax
Examples
1. Copy the file "host" from the directory /etc to the directory /home:
a. OdiFileCopy -FILE=/etc/hosts -TOFILE=/home/hosts
2. Copy all *.csv files from the directory /etc to the directory /home and overwrite:
a. OdiFileCopy -FILE=/etc/*.csv -TODIR=/home -OVERWRITE=yes
3. Copy all *.csv files from the directory /etc to the directory /home while changing their
extension to .txt:
a. OdiFileCopy -FILE=/etc/*.csv -TOFILE=/home/*.txt -OVERWRITE=yes
4. Copy the directory C:\odi and its sub-directories into the directory C:\Program Files\odi
a. OdiFileCopy -DIR=C:\odi "-TODIR=C:\Program Files\odi" -
RECURSE=yes
Confidential
1. Name : Delete EMP10
2. LogCounter :Delete
3. Technology :Oracle
4. Context : Global
5. Schema : Target_TDBU
6. Command : delete from <%=snpRef.getObjectName("L","EMP10","D")%>
Note :
"L" use the local object mask to build the complete path of the object.
"D"Returns the complete name of the object in the physical catalog and the data physical
schema that corresponds to the specified tuple (context, logical schema.)
4. Similarly Create two more tasks to delete from EMP20 and EMP30
Confidential
1. Open SQL Developer create EMP_PROC table in TDBU schema
AS
BEGIN
END;
Confidential
Step3: Creating ODI Procedure
Confidential
6. Click on save Run observe that the record is inserted in table EMP_PROC
Exercise : Create a target table with the name emp_totalsal_tax with columns empno
ename sal comm. ,totalsal,tax and populate totalsal and tax columns using stored
procedure
Tax=totalsal*0.1
Exercise 3: Copy files from one folder to another folder using OdiFileCopy
Confidential
Step2: Create a task to copy files
OdiFileCopy -FILE=/Oracle/Middleware/Oracle_Home/odi/demo/RRITEC/*.txt -
TODIR=/Oracle/Middleware/Oracle_Home/odi/demo -OVERWRITE=yes
Confidential
24. Handson 21: Packages
Confidential
5. Select OdiFileMove and change Filename and Target Directory as shown below
Confidential
25. Handson 22: Scenarios
of the components which contributed to creating it will not change it in any way.
6. Once generated, the scenario is stored inside the work repository. The scenario can be
Exported then imported to another repository (remote or not) and used in different
contexts. A scenario can only be created from a development work repository, but can
be imported into both development and execution work repositories.
Note: Regenerate: It is useful to update the scenario with new code of the odi object
Confidential
1 - Synchronous (default). Execution of the session waits until the scenario has terminated;
2 - Asynchronous. Execution of the session continues without waiting for the called session.
Confidential
26. Handson 23: Load Plan
1. A Load Plan is the largest executable object in Oracle Data Integrator. It uses Scenarios in its
steps. When an executable object is used in a Load Plan, it is automatically converted into a
scenario. For example, a package is used in the form of a scenario in Load Plans. Note that Load
Plans cannot be added to a Load Plan. However, it is possible to add a scenario in form of a Run
Scenario step that starts another Load Plan using the OdiStartLoadPlan tool.
2. Load plans are not substitutes for packages but are used to organize at a higher level the
execution of packages.
3. Unlike packages, Load Plans provide native support for parallelism, restartability and exception
handling. Load plans are moved to production as is, whereas packages are moved in the form of
scenarios. Load Plans can be created in Production environments.
4. Unlike Load Plan, Packages are useful to handle loops.
Process
Creating Load Plan
1. In designer navigator Click on Load Plans and Scenerios Click on new Load plan Name it
as LP Click on Steps Go to new step add parallel step drag and drop m_totalsal_tax
and m_flatfile_to_table interfaces
2. To understand serial mode select root_step add new step select serial step drag and
drop any two interfaces/packages..etc
Confidential
3.
4. Save click on run (note: without agent we can not run load plan )
5. We can monitor load plans from operator tab.
6.
Confidential
1. In designer navigator Click on Load Plans and Scenerios Click on new
Load plan Name it as LP1 Click on Steps drag and drop the package calling
load plan on to root_step save and run it
Agents
3.
Step 2 : Creating Logical Agent and Mapping Physical and logical agents in context
1. Go to topology tab → in LogicalArchitecture right click on Agents → Click on new
agent name it as OracleDIAgent → under physical agent select OracleDIAgent
opposite to development context → click on save → Click on close
Confidential
2.
5.
6. Change odi params.bat file repository section as shown
7.
8. Change User credentials for agent startup program
9. Let us say ODI username : SUPERVISOR and password : RRitec123
10. As already we encoded RRItec123 password we can use same
11. Change odi params.bat file user credentials section as shown
Confidential
12.
13. Change work repository section as shown
14.
15. save and close
1. Go to location C:\OBIEE_HOME\Oracle_ODI1\oracledi\agent\bin
2. double click on agent.bat file
3. Observe cmd prompt
4.
5. On go if agent is not running then debug using log file
6.
Confidential
RR ITEC #209,Nilagiri Block,Adithya Enclave,Ameerpet @8801408841,8790998182 104
Confidential
27. Handson 25: Version Control
Note: Version management is supported for master repositories installed on database engines
such as Oracle, Hypersonic SQL, and Microsoft SQL Server.
1. Projects, Folders
2. Packages, Scenarios
3. Mappings (including Reusable Mappings), Procedures, Knowledge Modules
4. Sequences, User Functions, Variables
5. Models, Model Folders
6. Solutions
7. Load Plans
Confidential
3. Save close
4. Right click on package pkg_MAPPINGS_ARCHIVE_FILES Click on version click on
Create version click on ok
5. Now if you want to restore the initial version (1.0.0.0) Right click on package
pkg_MAPPINGS_ARCHIVE_FILES Click on Restore Select 1.0.0.0 version click
on ok
6. Open package observe the initial code
Comparing versions
3. Save close
4. Right click on ODS_EMPLOYEE version compare with version click on ok
5. Remove deleted ,New and unchanged check boxes and observe the difference
Confidential
RR ITEC #209,Nilagiri Block,Adithya Enclave,Ameerpet @8801408841,8790998182 107
Confidential
28. Handson 26: Modifying Knowledge Modules
1. Knowledge Modules are templates of code that define integration patterns and their
implementation
2. They are usually written to follow Data Integration best practices, but can be modified for
project specific requirements
3. These are 6 types
Confidential
8. CKM (Check Knowledge Modules) are used to check that constraints on the sources
and targets are not violated. These KMs are used in data model’s static check and
interfaces flow checks.
9. SKM (Service Knowledge Modules) are used to generate the code required for creating
data services. These KMs are used in data models.
1. When processing happens between two data servers, a data transfer KM is required.
a. Before integration (Source Staging Area)
2. When processing happens within a data server, it is entirely performed by the server.
a. A single-technology IKM is required.
3. LKM and IKMs can use in four possible ways
Confidential
4. Normally we ever create new KMs .But sometimes we may need to modify the existing
KMs
5. While modifying KMs , Duplicate existing steps and modify them. This prevents typos in
the syntax of the odiRef methods.
ENAME VARCHAR2(30),
SAL NUMBER
Confidential
2. Reverse Engineer and Import table EMP_INCR_MKT into RRITEC_TARGET_MODEL
3. Click on ok
Confidential
6. Click on physical tab select target table EMP_INCR_MKTSelect integration
Knowledge Module as IKM oracle Incremental Update Select
Flow_control=FALSE
Confidential
2. Right-click on the IKM Oracle Incremental Update Knowledge ModuleRight click
select Duplicate Selection.
3. Open the copy of IKM Oracle Incremental Update and rename it to IKM Oracle
Incremental Update Audit
AUDIT_DATE DATE,
AUDIT_INDICATOR VARCHAR2(1)
Confidential
7. Similarly create one more task to Insert records Click on add name it as Insert into
Audit table select ignore errors
8. Select target technology as oracle provide target command as shown below ok
AUDIT_DATE,
AUDIT_INDICATOR
sysdate,
Confidential
IND_UPDATE
from <%=odiRef.getTable("L","INT_NAME","W")%>
Note: To reduce typing, you can copy the code from a similar step and modify as
needed.
GetTable:
a. “L”: Local naming convention. For example, in Oracle that would be schema.table
(versus “R” for remote: schema.table@server).
b. “A”: Automatic. It enables ODI to determine which physical schema to use (the
Data schema [“D”] or the Staging schema [“W”])
getColList:
Confidential
1. Notice the “PK” parameter. If it is used, only the columns that are part of the
primary key are included.
9. Save it Verify that your new knowledge module IKM Oracle Incremental Update
Audit appears in the Knowledge Modules tree.
10. Select Create Audit Table task and move to on top of commit transactions task
11. Select Insert into Audit Table task and move to on top of commit transactions task
Created knowledge module. Change IKM entry to use IKM Oracle Incremental Update
Audit
Confidential
Exercise 3: Add an Option to your KM
1. To make your KM more user friendly, you can add an option that will let the end user
Confidential
Step2: Link the option to your tasks
1. In above audit KM click two tasks one by one (create audit table and insert records
into audit table) Unselect Always Execute Select AUDIT_CHANGES
2. Save and close the IKM. The execution of these steps can now be set by the end-user.
3. In the Physical tab of your mapping, click on the Target table
4. verify that the AUDIT_CHANGES option is set to True.
5. Run the mapping ,check the behavior in the Operator mapping.
6. Change the value to False, run the mapping again, and compare the generated code in
the operator mapping.
Confidential
29. Handson 27: Change Data Capture (CDC)
1. The purpose of Changed Data Capture is to allow applications to process changed data
only
2. Loads will only process changes since the last load
3. The volume of data to be processed is dramatically reduced
4. CDC is extremely useful for near real time implementations, synchronization, Master
Data Management
5. In general CDC Techniques are Four types
1. Trigger based – ODI will create and maintain triggers to keep track of the
changes
2. Logs based – for some technologies, ODI can retrieve changes from the
database logs. (Oracle, AS/400)
3. Timestamp based – If the data is time stamped, processes written with ODI can
filter the data comparing the time stamp value with the last load time. This
approach is limited as it cannot process deletes. The data model must have
been designed properly.
4. Sequence number – if the records are numbered in sequence, ODI can filter the
data based on the last value loaded. This approach is limited as it cannot
process updates and deletes. The data model must have been designed
properly.
6. CDC in ODI is implemented through a family of KMs: the Journalization KMs
7. These KMs are chosen and set in the model
8. Once the journals are in place, the developer can choose from the interface whether he
will use the full data set or only the changed data
9. Changed Data Capture (CDC), also referred to as Journalizing
Journalizing Components
Confidential
2. Capture processes: Captures the changes in the source datastores either by creating
triggers on the data tables, or by using database-specific programs to retrieve log data
from data server log files
3. Subscribers (applications, integration processes, and so on): That use the changes
tracked on a datastore or on a consistent set
Using CDC
Confidential
This approach has a limitation, illustrated in the following example: You want to process
changes in the ORDER and ORDER_LINE datastores (with a referential integrity
constraint based on the fact that an ORDER_LINE record should have an associated
ORDER record). If you have captured insertions into ORDER_LINE, you have no
guarantee that the associated new records in ORDERS have also been captured.
Processing ORDER_LINE records with no associated ORDER records may cause
referential constraint violations in the integration process.
Consistent Set Journalizing provides the guarantee that when you have an
ORDER_LINE change captured, the associated ORDER change has been also
captured, and vice versa. Note that consistent set journalizing guarantees the
consistency of the captured changes. The set of available changes for which
consistency is guaranteed is called the Consistency Window. Changes in this window
should be processed in the correct sequence (ORDER followed by ORDER_LINE) by
designing and sequencing integration interfaces into packages.
Although consistent set journalizing is more powerful, it is also more difficult to set up. It
should be used when referential integrity constraints need to be ensured when capturing
the data changes. For performance reasons, consistent set journalizing is also
recommended when a large number of subscribers are required.
It is not possible to journalize a model (or datastores within a model) using both
consistent set and simple journalizing.
6. For consistent CDC, arrange the datastores in the appropriate order (parent/child
relationship) : in the model definition, select the Journalized tables tab and click the
Reorganize button
7. Add the subscriber (The default subscriber is SUNOPSIS) Right-click on a table and
select Changed Data Capture / Add subscribers
8. Start the journals: Right-click on a table and select Changed Data Capture / Start
Journal
Confidential
Exercise 1: Create a simple Journalizing
C:\Oracle\Middleware\Oracle_Home\odi\sdk\xml-reference
4. Click on OK
Confidential
1. Right click on RRITEC_MODEL_SCOTT click on open select Journal select
Journalizing mode as simple
2. Select KM as JKM oracle Simple
2. Make sure Emp table is appearing under journalized tables of the model
Confidential
Step 4: Start Journal
1. Right Click on EMP table Changed data capture Click on start Journal
Confidential
4. By observing this you come to notice below objects are creating as part of CDC
1. One J$<TABLE> created
2. One Subscriber table created SNP_SUBSCRIBERS
3. Two Views created JV$<TABLE>,JV$D<TABLE>
4. One trigger created T$<TABLE>
ENAME VARCHAR2(30),
JOB VARCHAR2(30),
SAL NUMBER(10),
Deptno NUMBER(2)
Confidential
5. Click on Physical tab select target table select IKM as IKM oracle incremental
update
Note: First time we are loading all data into target table
1. Go to EMP table and update any one record and notice that this change registered in
J$emp table
2. In next run we need to load only the records available in J$emp table for that go to
mapping select source table and select Journalized data only
Confidential
\
4. Save run
Confidential
30. Handson 28: Migration (Exporting and Importing)
2. Smart export
i. We can click on connect navigator and export required objects by using
simple drag and drop method
ii. We can export in two formats XML and ZIP file
iii. Even dependencies also exported
iv. It is recommended method
Confidential
4. Import:
1. Duplication
i. This mode creates a new object (with a new GUID and internal ID) in the
target Repository, and inserts all the elements of the export file.
ii. Note that this mode is designed to insert only 'new' elements.
2. Synonym
i. Synonym Mode INSERT
1. Tries to insert the same object (with the same GUID) into the
target repository. The original object GUID is preserved.
2. If an object of the same type with the same internal ID already
repository.
Confidential
2. This import type updates the objects already existing in the target
will be inserted, and child objects existing in the repository but not
3. This import type is not recommended when the export was done
3. Smart
i. This method is always given first priority
Confidential
Exercise 1: Migrating from development to test work repository
Confidential
3. Select desktop and create a folder and save it.
Step 3: Creating user TDBU_TEST and load tables into TDBU _TEST schema
Confidential
Step5: Creating a testing context and mapping the source and target schemas
Confidential
Step7: Running and loading data into test data base
Confidential
2. Right click on scenario m_ODS_EMPLOYEE Version 001 click on export
3. Select export path provide export file name click on export close
Similar to on page 24Create the Work Repository (refer page no 24 ) (however please take
care below step)
Step 3: Creating user TDBU_PROD and load tables into TDBU _PROD schema
Confidential
c. Conn TDBU_ PROD @ORCL
d. Password RRitec123
e. Select count (*) from tab;
6. Go to RRITEC labcopy labdata folder and take full path of target.sql file and execute
Step5: Creating a Production context and mapping the source and target schemas
Confidential
1. Connect to RRITEC_EWR work repository Go to operator tab Under load Plans
and Scenarios click on import scenario
Confidential
31. Handson 29: Security
1. All the security information for Oracle Data Integrator is stored in the master repository
2. Security consist of three objects
a. Object
b. Profiles
c. Users
3. An Object is a representation of a design-time or run-time component
4. For example, agents, models, data stores, scenarios, interfaces/Mappings and even
repositories are objects.
5. A Profile contains a set of privileges for working with Oracle Data Integrator. One or
more profiles can be assigned to a user to grant the sum of these privileges to this user.
6. A profile method is an authorization granted to a profile on a method of an object type.
Each granted method allows a user with this profile to perform an action on an instance
of an object type.
Creating User
Confidential
1. Generic profiles have the generic privilege option selected for all object methods. This
implies that a user, or role, with such a profile is by default authorized for all methods of
all instances of an object to which the profile is authorized.
2. Nongeneric profiles are not by default authorized for all methods on the instances
because the generic privilege option is not selected for all object methods. You must
grant the user, or role, the rights on the methods for each instance.
3. If you want a user, or role, to have the rights on no instance by default, but want to grant
the rights on a per instance basis, the user or role must be given a nongeneric profile.
4. If you want a user or role to have the rights on all instances of an object type by default,
you must give the user or role a generic profile.
5. Predefined Profiles are
Confidential
RR ITEC #209,Nilagiri Block,Adithya Enclave,Ameerpet @8801408841,8790998182 140
Confidential
32. Handson 30: CKM
1. Data Quality control is essential in ensuring the overall consistency of the data in
your information System's applications.
2. Application data is not always valid for the constraints and declarative rules imposed
by the information system. You may, for instance, find orders with no customer, or
order lines with no product, etc. In addition, such incorrect data may propagate via
integration flows
3. The CKM can be used in 2 ways:
a. Check the consistency of existing data.This can be done on any data store or
within interfaces, by setting the STATIC_CONTROL option to "Yes".
b. Check consistency of the incoming data before loading the records to a target data
store. This is done by using the FLOW_CONTROL option. In this case, the CKM simulates
the constraints of the target data store on the resulting flow prior to writing to the target.
4. In summary: the CKM can check either an existing table or the temporary "I$" table
created by an IKM.
5. The CKM accepts a set of constraints and the name of the table to check. It creates
an "E$" error table which it writes all the rejected records to. The CKM can also
remove the erroneous records from the checked result set.
6. The following figures show how a CKM operates in both STATIC_CONTROL and
FLOW_CONTROL modes.
Check Knowledge Module (STATIC_CONTROL)
In STATIC_CONTROL mode, the CKM reads the constraints of the table and checks them
against the data of the table. Records that don't match the constraints are written to the
"E$" error table in the staging area.
Confidential
In FLOW_CONTROL mode, the CKM reads the constraints of the target table of the
Interface. It checks these constraints against the data contained in the "I$" flow table of the
staging area. Records that violate these constraints are written to the "E$" table of the
staging area.
Confidential
2. Select CKM as CKM ORACLE → Click on ok
Confidential
2. save → close
STEP 3: Checking COMM column for not null values
4. save → close
RR ITEC #209,Nilagiri Block,Adithya Enclave,Ameerpet @8801408841,8790998182 144
Confidential
5. right click on this table → click on control → click on check select context properly
→ click on ok → again ok
6. go to database notice that two tables created
a. SNP_CHECK_TAB
b. E$_B107EMP
7. Observe that all comm null records are copied into E$B107EMP table
Confidential
5. save → run
6. observe output
Confidential
Exercise 3: Flow Control Understanding
Confidential
INSERT INTO "SCOTT"."B108EMP" (EMPNO, ENAME, JOB, MGR, HIREDATE, SAL,
COMM, DEPTNO) VALUES ('5555', 'RRITEC', 'Training', '7698', TO_DATE('08-SEP-81',
'DD-MON-RR'), '1500', '100', '20');
INSERT INTO "SCOTT"."B108EMP" (EMPNO, ENAME, JOB, MGR, HIREDATE, SAL,
COMM, DEPTNO) VALUES ('5555', 'RRITEC', 'Training', '7698', TO_DATE('08-SEP-81',
'DD-MON-RR'), '1500', '100', '20');
INSERT INTO "SCOTT"."B108EMP" (EMPNO, ENAME, JOB, MGR, HIREDATE, SAL,
COMM, DEPTNO) VALUES ('12', 'John', 'Manager', '7782', TO_DATE('08-SEP-81', 'DD-
MON-RR'), '1500', '100', '20');
INSERT INTO "SCOTT"."B108EMP" (EMPNO, ENAME, JOB, MGR, HIREDATE, SAL,
COMM, DEPTNO) VALUES ('6666', 'Nancy', 'Manager', '7782', TO_DATE('08-SEP-81',
'DD-MON-RR'), '1500', '100', '60');
3.
2.
Confidential
3. Reverse engineer into target model
Step 3: Creating Constraints
1. open the target model navigate to target table TGT_B107EMP
2. right click on constraints → click on newkey
Confidential
5. save → close
6. Right click on constraints → Click on New Reference
9. Save → close
10. Again right click on constraints → select New Condition
Confidential
11. Provide below information
12.
13. Save→ close
Confidential
5. Select Control tab and observe conditions
Confidential
33. Handson 31: Slowly Changing Dimension (SCD2)
1. Although dimension tables are typically static lists, most dimension tables do change
over time.
2. Since these changes are smaller in magnitude compared to changes in fact tables,
these dimensions are known as slowly growing or slowly changing dimensions.
3. If we load first Time
Confidential
5. Third time load
Confidential
3. Reverse engineer into TARGET MODEL
2. Expand table → Expand columns → right click on one by one column and set as
shown below
Confidential
RR ITEC #209,Nilagiri Block,Adithya Enclave,Ameerpet @8801408841,8790998182 156
Confidential
3. Save and close one by one
2 EMPNO EMP.EMPNO
3 ENAME EMP.ENAME
4 JOB EMP.JOB
Confidential
5 SAL EMP.SAL
6 DEPTNO EMP.DEPTNO
7 CURR_FLG 0
8 EFF_FROM_DT
9 EFF_TO_DT
Confidential
7. Click on SAVE
8. Click on RUN
9. Change source data sal and run again
10. Again change source data sal run (in my case i changed Ram Reddy sal from
10000 to 30000 and 30000 to 35000)
Confidential