Documente Academic
Documente Profesional
Documente Cultură
Release 8.0
Revision A
Initiate, InitiateSM and Initiate Identity Hub are trademarks and/or service marks of Initiate Systems, Inc., which may be registered in some jurisdictions. All rights reserved. All other marks are owned by their respective owners. The information in this document is protected under the applicable federal law as an unpublished work, and is confidential and proprietary to Initiate Systems, Inc. Its use, disclosure, reproduction, or publication, in whole or in part, without the express prior written consent of Initiate Systems, Inc. is prohibited.
Table of Contents
ABOUT THIS MANUAL ........................................................................................................ 4 Audience and purpose .............................................................................................. 4 Organization ............................................................................................................ 4 Additional reference documentation ........................................................................ 4 How to get help........................................................................................................ 4 ATSC........................................................................................................................ 4 Support Center Knowledge Base .................................................................................. 5 Acknowledgements .................................................................................................. 5 CHAPTER 1: MASTER DATA EXTRACT OVERVIEW .................................................................... 6 Clover.ETL basics ..................................................................................................... 6 The Master Data Extract Sample Graphs .................................................................. 7 CHAPTER 2: USING THE MASTER DATA EXTRACT SAMPLE GRAPHS ............................................. 10 Importing the sample graphs ................................................................................. 10 Configuring Readers............................................................................................... 15 Creating a database connection.................................................................................. 15 Specifying a database connection for each Reader ........................................................ 16 Configuring the extract_full_all.grf sample graph.................................................. 17 Configuring the extract_incremental_db.grf sample graph .................................... 19 Parameters for incremental extraction......................................................................... 19 Configuring the extract_incremental_file.grf sample graph ................................... 23 Parameters for incremental extraction......................................................................... 23 Running a graph..................................................................................................... 25 Troubleshooting graphs ......................................................................................... 25 Debugging a graph Edge ........................................................................................... 25 Viewing logs and error messages ............................................................................... 26 Automatic graph execution .................................................................................... 26 Using the madconfig utility to create a properties file for a scheduled job ......................... 26 Using madconfig to launch a graph using a specified properties file ................................. 26 Recording responses to the madconfig utility ............................................................... 27 Using extract.ddl to create target database schema .............................................. 27
iii
Organization
The information presented includes: Contents of Manual In Chapter 1 2 You will find Overview of the Master Data Extract application Detailed information about using the Master Data Extract sample graphs
Acknowledgements
Third party software code files are shipped along with the Initiate 8.0 (the Third Party Code) software. Third Party Code files are the property of their respective owners and not Initiate Systems and Initiate Systems claims no rights in or to the Third Party Code. Your use and access to the Third Party Code is governed by the specific restrictions and limitations set forth in the applicable licenses provided by the Third Party Code owners. The Third Party Code is provided to you by Initiate solely for use with the Initiate software product and Initiate Systems does not authorize or promote any other use of the Third Party Code by you. The full text of the applicable Third Party Code licenses is provided in the Third Party License.zip file included along with the Initiate Release 8.0, located on the Initiate Systems product CD or downloaded CD image.
Clover.ETL basics
For basic information on using the Clover.ETL application, refer to the Initiate Workbench User Guide and to the Clover documentation.
Master Data Extract overview extract_incremental_db.grf: This graph does an incremental extract of entity-level attribute data from the Master Data Engine database, filters it based on audit-record parameters supplied by the user as a configuration parameter, removes duplicates, and writes the output to a specified database. The graph consists of several subgraphs or series of connected Reader, Transformer, and Writers, which operate in parallel; each subgraph reads data from a specific database table in the Master Data Engine database.
Master Data Extract overview extract_incremental_file.grf: Like the extract_incremental_db.grf, this graph does an incremental extract of entity-level attribute data and filters it on user-supplied audit record numbers, and removes duplicates. The output of this graph is written to a series of delimited files. The graph consists of several subgraphs or series of connected Reader, Transformer, and Writers, which operate in parallel; each subgraph reads data from a specific database table in the Master Data Engine database.
10
Using the Master Data Extract sample graphs 2. In the Import - Select dialog, navigate to and select Import graphs version conversion (in the Clover ETL node).
3. Click Next.
11
Using the Master Data Extract sample graphs 4. In the Import Clover ETL Graphs dialog, click the Browse button beside the From directory field.
12
Using the Master Data Extract sample graphs 5. Navigate to and select the <ROOTDIR>\Workbench x.x.x\samples\graphs directory (where <ROOTDIR> is your Initiate program files installation directory and x.x.x is your application version number).
6. Click OK. 7. The Into folder field lists the folder into which the graphs will be imported; the field you right-clicked on in Step 1 is displayed here by default. If you wish to specify a different folder, click the Browse button beside the Into folder field to browse to and select another folder.
13
Using the Master Data Extract sample graphs 8. The Import Clover ETL Graphs window is now populated with all available sample graphs. Check the boxes for the graphs you want to import: extract_full_all.grf extract_incremental_db.grf extract_incremental_file.grf
9. Click Finish.
14
Configuring Readers
The Readers in the sample graphs query database tables in parallel, ordering results by entity record number and modified audit record number. Before executing the graph, each of the Reader elements must be configured with the appropriate database connection information. Before you can specify a database connection for your Reader(s), you must create a database connection. Once the database connection is created, it can be used for all your Reader(s) in the sample graph.
15
2. On the Properties tab, under Clover.ETL properties basic, click in the Value field for DB connection. A down arrow appears. 3. Click the down arrow and select the database connection you created for this database. 4. Click OK to save your changes and close the Edit component dialog. Note: You must specify a database connection for each of the Readers in the sample graph.
16
2. Edit the Reader to provide database connection parameters. Detailed information on how to configure a Reader to connect to a database is given in the Configuring Readers section above.
17
Using the Master Data Extract sample graphs 3. Using the Select tool from the Palette, select and delete each of the Writers you do not wish to use. Each subgraph includes the following Writer types; delete all but the type you wish to use:
Sample graph Writer types
Name format oracle_data type db2_data type mssql_data type delimited_data type db2_ssn
Note: You can also disable a Writer by right-clicking the Writer and choosing Disable. 4. Using the Select tool, delete the Edge linking the Copy Transformer to your remaining Writer. 5. Use the Select tool to drag the Edge linking the Dedup Transformer to the Copy Transformer so that it connects the Dedup Transformer to the input port of your remaining Writer instead.
6. Delete the Copy Transformer. 7. If you are using a database Writer, connect the Writer to a database: A. Double-click the Writer to open the Edit component dialog. B. On the Properties tab, enter the relevant required properties according to the tables below. Required properties with missing values are flagged with a yellow exclamation-point icon.
Required Oracle properties
Value The path to Oracles SQL Loader (sqlldr) utility. Click in this field to display an ellipsis, then click on the ellipsis to browse to the utility. The user name for connecting to the database The password for connecting to the database 18
Value The database to which this data will be written The user name for connecting to the database The password for connecting to the database The name of the database table where this data will be written
Value Path to the utility that copies data between Microsoft SQL Server and a data file. Click in this field to display an ellipsis, then click on the ellipsis to browse to the utility. The database to which this data will be written
Database
C. Click OK to save your changes and close the Edit component dialog. 8. Repeat the steps above as needed to edit each of the subgraphs in the sample graph. Refer to the Running a graph section below for information on how to run your graph once it is configured.
Using the Master Data Extract sample graphs Note: Each sample graph consists of several subgraphs or connected series of Readers, Transformers, and Writers, which operate in parallel when the graph is executed. The section below describes how to edit one of these subgraphs. You will need to repeat the steps below for each subgraph in your sample graph. To configure the extract_incremental_db.grf sample graph: 1. In the Navigator view, double-click the extract_incremental_db.grf sample graph to open it in the Graph editor.
2. Edit the Reader to provide database connection parameters. Detailed information on how to configure a Reader to connect to a database is given in the Configuring Readers section above.
20
Using the Master Data Extract sample graphs 3. Verify that the parameters for attributes and auditor record numbers are correct. Parameters are listed in the Outline view, in the Parameters node.
Note that parameters listed here apply to the graph as a whole, and are not edited for individual components. Note: In a typical use case, you will edit the attribute record parameters on a one-time basis as part of general graph configuration, but update the auditor record numbers each time you run the graph. You can use the madconfig utility to populate the auditor record number parameters via a scheduled job. See the Automatic graph execution section below for more information. 4. Using the Select tool from the Palette, select and delete each of the Writers you do not wish to use. Each subgraph includes the following Writer types; delete all but the type you wish to use:
Sample graph Writer types
Note: You can also disable a Writer by right-clicking the Writer and choosing Disable. 5. Using the Select tool, delete the Edge linking the Copy Transformer to your remaining Writer.
21
Using the Master Data Extract sample graphs 6. Use the Select tool to drag the Edge linking the Reformat Transformer to the Copy Transformer so that it connects the Reformat Transformer to the input port of your remaining Writer instead.
7. Delete the Copy Transformer. 8. Edit the Writer to connect to a database: A. Double-click the Writer to open the Edit component dialog. B. On the Properties tab, enter the relevant required properties according to the tables below. Required properties with missing values are flagged with a yellow exclamation-point icon.
Required Oracle Properties
Value The path to Oracles SQL Loader (sqlldr) utility. The user name for connecting to the database The password for connecting to the database The transparent network substrate (TNS) name identifier
Value The database to which this data will be written The user name for connecting to the database The password for connecting to the database The name of the database table where this data will be written
Value Path to the utility that copies data between Microsoft SQL Server and a data file. Click in this field to display an ellipsis, then click on the ellipsis to browse to the utility. The database to which this data will be written 22
Database
C. Click OK to save your changes and close the Edit component dialog. 9. Repeat the steps above as needed to edit each of the subgraphs in the sample graph.
23
Using the Master Data Extract sample graphs To configure the extract_incremental_file.grf sample graph: 1. In the Navigator view, double-click the extract_incremental_file.grf sample graph to open it in the Graph editor.
2. Edit the Reader to provide database connection parameters. Detailed information on how to configure a Reader to connect to a database is given in the Configuring Readers section above.
24
Using the Master Data Extract sample graphs 3. Verify that the parameters for attributes and auditor record numbers are correct. Parameters are listed in the Outline view, in the Parameters node.
Note that parameters listed here apply to the graph as a whole, and are not edited for individual components. Note: In a typical use case, you will edit the attribute record parameters on a one-time basis as part of general graph configuration, but update the auditor record numbers each time you run the graph. You can use the madconfig utility to populate the auditor record number parameters via a scheduled job. See the Automatic graph execution section below for more information. 4. Repeat the steps above as needed to edit each of the subgraphs in the sample graph.
Running a graph
To run a graph, click the Run icon in the toolbar, or choose Run > Run from the menu. When a graph is run, the number of records processed along each Edge is displayed. For detailed information about graph runtime options, refer to the Clover documentation.
Troubleshooting graphs
Use the following processes and tools to troubleshoot your graphs.
Using the Master Data Extract sample graphs You can view debug data after the graph is run by right-clicking the edge and choosing Debug > View Data.
Using the madconfig utility to create a properties file for a scheduled job
Incremental extracts typically select data based on a range of audit record numbers which change each time the graph is run. Although you may manually set the range of record numbers to extract manually in the graph, it may be more practical to generate a properties file automatically via a scheduled job. The properties file then supplies the graph with the appropriate values for the record number range. This section describes how to use the madconfig utility to launch a graph using a designated, external properties file. You can set up a scheduled job to launch the madconfig utility on a regular basis. Note: It is outside the scope of this document to describe how to set up a scheduled job which generates the properties file. You can use a standard utility such as the Windows Task Scheduler or a Unix chron utility (or other methods) to set up a scheduled job.
26
27