Defining components

Composition view of data

Double-click group object in Type tree view to define and view group components in the
order they appear in data in group view

Defining components
Identify the data types that make up a group
Describe the order that data appears in a group




Component lists
Component name in group view refers to the type in the type tree using the full name
Always drag and drop types into component lists





Component lists

Component ranges
Component range indicates the number of consecutive occurrences of component allowed
Specified as minimum_occurrences:maximum_occurrences
The default is one and only one, designated by 1:1
The range of 1:1 is assumed if no value is displayed
If the component is optional,
minimum occurrences is
zero, designated by 0:1
If component can appear an
unknown number of times;
the maximum_occurrences is S




Displaying all range values

Setting the range

Type tree analyzer
Logical analysis verifies integrity of defined relationships
Detects undefined components
Identifies distinguishability problems
Structure analysis verifies integrity of physical type tree file
Does not compare definitions to actual data
Type tree must analyze error-free before it can be used in map
All errors must be resolved
Warnings indicate an inconsistency that should be resolved
Type tree analysis results
Messages include:
Error number
L for error in logical analysis
S for error in structural analysis
Text message describing error
(error) or (warning)

Interpreting type tree analysis error messages
1. Identify type in error
2. Read error message relative to type in error
3. Double click error message in Analyzer Results view to find and select type in type tree
4. Identify component in error by component number


Displaying component numbers

















Validating data definitions
Using Map Designer
Create maps to specify logic necessary to transform data:
Identify data sources and targets
Identify type trees that describe format and structure of data sources and targets
Provide map rules that describe transformation requirements and business rules
Map Designer user interface

Map Designer basics
From window
Data sources (inputs)
Shows format of input data
To window
Data targets (outputs)
Shows format of output data
Cards represent data objects
One input card for each data source
One output card for each data target
Rule bar used to create and view complete map rules
Rule column displays portion of map rules
Map preferences

Maps and map source files
Map is a definition of how to generate outputs from given inputs
Map source file (.mms file) stores one or more map definitions

Composition view
Shows map source file and map definitions they contain
Shows map relationships
Arranged in compositional hierarchy
Functional maps listed beneath executable map that calls it
New maps only appear when at least one card is defined

Outline view
Shows map definitions in alphabetical
order
In expanded view
Shows input cards and output
cards for the map definition
Provides access to map
Organizer views
New maps appear immediately in
Outline view



Navigator map icons
Color and type of icon in the map
navigator provides general information
regarding the map purpose and content
Main, executable map definition

Functional map definition



Map Solution Diagram
Provides graphical view of map
relationships
1. Right-click map in map Outline
or Composition view
2. Select Map Solution Diagram


Defining an input card
At a minimum:
1. Enter CardName that describes input
2. Select TypeTree that
contains input data definition
3. Select Type in type tree
that defines input data
4. Specify Source resource adapter
5. Enter adapter specific
settings



How card definition relates to card view


CardName appears in card header and a first object in tree structure
Type appears in parentheses next to card name in card header and determines top level
object for mapping





Defining an output card
At a minimum:
1. Enter a descriptive CardName
2. Select the type tree
that defines the output
data format
3. Select Type from
the type tree that
defines output data
4. Select Target adapter
5. Specify adapter
specific settings


Map Designer card hierarchy

Executable map
Map that is built and run
Top level map that operates on entire files, tables and messages
Minimally must include data description (type tree) and physical location of data (source or
target settings) on cards

Creating an executable map
1. Open the Map Designer
2. Create a new map source file
or
Open an existing map source file
3. Create a new map definition (Map > New)
4. Add an input card for each data source (Card > New)
5. Add an output card for each data target (Card > New)
6. Enter map rules
7. Build (compile) the map
How the Transformation Extender engine works
. Validates input data against type tree
starting with Input card #1

. Applies map rules starting with Output card #1





Data validation
Parses data from deepest level out:
1. Validates each field (level 2) in a
record against type properties,
restrictions and component rules
2. Validates record (level 1) against
group properties and
component rules
3. Repeat steps 1 and 2 for each
record until end of file (level 0)
4. Validates file (level 0) against
group properties and
component rules



Validation map: method #1
Map contains input and output cards
Same TypeTree and Type in both cards
Map creates an output file that can be compared to input file
If input file exactly matches output file, type tree correctly defines data


Validation map: method #1

1. Contract tree in input card and output card to only show top level object
2. Map top level object in input to top level object in output to force validation of input data
against type tree definitions and create output file
Validation map: method #2
Map contains input card only
No output file created
Eliminates rules processing
Preferred method when validating large data files

Modifying the type tree
Always open the type tree from the map to ensure that that the type tree is the same one
referenced in the map


After making changes to trees
Map uses the saved version of type tree
When the type tree is changed, follow this procedure:

Option to always save after analysis in Transformation Extender Preferences
Troubleshooting aids
Command Server confirmation window
Run Results views
Trace file
Map debugger
Audit logs
Data audit log
Execution audit log
Command Server confirmation window
Displays information that may be of value for debugging
Return message
Input and output object counts
Run time


Run Results views
1. Select Map > View Run Results or click to verify that map:
Read correct input
Generated correct output
2. Select inputs and outputs
to display

Trace file contents
Summary trace to determine card that failed
Trace on input (InputContentTrace)
Data objects found
Why data is found to be not valid
Sizes and counts of data objects
Position of data objects in data stream
Trace on output (RulesTrace)
Data objects built
Data objects that evaluated to NONE
When to use trace
Generate an input trace when map returns a data related error during validation
Generate an output trace when a step-by-step account of data object generation is required

Generating the trace file
1. Select Map
Settings from
Map menu
2. Expand
MapTrace
options
3. Set Switch to ON
4. Specify trace
options
5. Build map
6. Run map



Viewing the trace file

Reading the trace file
Trace file is text file in the format mapname.mtr where mapname is the name of the
executable map
Use Edit > Find/Replace to search within trace information to locate error
Search on words such as Not, Failed, Wrong, or Invalid
Look immediately above or below error in trace file for additional information

Summary trace

1. Generate a summary trace to identify the card in error
2. If an error occurred on input during validation, run an input content trace on the card
If an error occurred on output, run map debugger


Interpreting input content trace results

Interpreting trace results

Reasons for validation failure
Input data did not match a type property in type tree
Data object has wrong Initiator or Terminator
Group is missing a required component
Delimiter does not match definition
Data object failed a component rule
Data object not in restriction list


Example: Input data did not match type property

Missing required component





Input Valid, Unknown Data Found
Transformation Extender found enough data to meet definition but found additional data
Usual causes
Source object defined as having a single occurrence
Source object defined as having zero or more objects and Transformation Extender
has found at least one, but it is followed by additional data
Functional map being built and run
Undefined data at end of file
Undefined data at end of file

Incorrectly defined data

Additional validation
During basic validation, data is isolated and compared to type definitions
Additional validation can be added to prevent processing of invalid data

Additional validation can be added to check if data, once isolated, matches specific data
value or set of values
Use a restriction list to limit item to particular value or set of values
Use a component rule to specify condition that must be met for a particular
component to be valid
Restriction lists
Limit valid values of item to a specific set
After changing restriction list, type tree must be analyzed and map re-built
Restriction list is case sensitive unless the Ignore case item property is set to Yes

Disabling restriction validation
Restrictions can be ignored when running a map
Set Validation Map Setting to Custom
Set RestrictionError to ignore

Restriction settings
Include or exclude
Value restrictions
Range restrictions
Character restrictions

Include or exclude value restrictions
Included values are considered to be valid
Excluded values are considered to be not valid
Include or exclude values supported for:
Characters
Numbers
Dates

Identifying restriction values

Use when object data is limited to a set of static values
To add restrictions:
Double-click item type in Type tree view or
right-click and select Open
Enter values in restriction columns
Restriction view column content varies based on restriction property settings
Include character restrictions
Include First
Character list of valid first character
Include After
Character list of valid characters that
may follow first character
To specify common range of characters,
enter reserved word in restriction list:
<LETTER/>
<LETTERUPPER/>
<LETTERLOWER/>
<DIGIT/>


Exclude character restrictions
Exclude is list of character
substrings to be excluded
Reference String is character string that replaces excluded character substring on output
On input, any data that contains any substring in the Exclude list would be marked as invalid
On output, a character text item would be built by using the Reference String if the content
contained any of the Exclude character substrings

Include or exclude value ranges


Trace file with restriction error

Component rules
Used for data validation
Statement about component that determines if data is valid
Evaluate business rule
Evaluate to TRUE or FALSE


Rule can refer to:
Component it applies to
Any component earlier in same component list


Shorthand notation for rules
Use $ as shorthand to represent current component in rule


Creating component rules
Created in rule bar
Must include full component name in rule
Press and hold Ctrl to drag and drop component name into rule bar
Always press Enter to save changes

Component rule references
Rule can refer to:
Component it applies to
Nested components of any component that can be referred to


Using functions in component rules
Component rules can use most Transformation Extender prebuilt functions

What is a function?


Expression that produces a result based on a certain operation
Can be nested with other functions
Can be entered in component rules and map rules
Not all Transformation Extender functions are available for use in component rules
Example: COUNT function
Returns number of occurrences of an object
Syntax: COUNT (series-object-expression)
Meaning: COUNT (objects_to_count)
Returns: Result is an integer equal to the number of objects_to_count. If input
argument evaluates to NONE, COUNT returns zero.


Functions view
Lists all Transformation Extender functions
Includes description of function syntax
Drag-and-drop
function from
Functions view
into component rule


Component rules
Component rules always evaluated even if an optional component is not present in data
Use WHEN(PRESENT ) if component rule should only be evaluated when data is present

Sample trace message
failed COMPONENT RULE test appears in trace message when component rule evaluates to
False


Comments in component rules
Comment can be added to component and map rules
- Begin with /* and end with */
- Appear anywhere in a rule
- Cannot be placed within object name

When to use what
For data validation use:
Restriction list
When an item is limited to a particular set of valid values
Component rule
To specify a condition that must be met for a particular component to be valid
For assigning a value to an output group or item, use a mapping rule










Map Designer methodology

Map rules
Describe how to build an output data object
Can be any combination of the following:
Hard-coded string enclosed in double quotes
Hard-coded numeric values
Map functions
Object names
Mathematical functions
Concatenation of multiple objects using a plus (+) character
Functional map references
Always start with an equal sign (=) character
Are required for every output object
Enter =None for objects that do not require output

Map rule examples
















Functions
Text
BCDTOTEXT
COUNTSTRING
DATETOTEXT
FIND
FILLLEFT
FILLRIGHT
HEXTEXTTOSTREAM
ISALPHA
ISLOWER
ISNUMBER
ISUPPER
LEAVEALPHA
LEAVEALPHANUM

LEAVENUM
LEAVEPRINT
LEFT
LOWERCASE
MAX
MID
MIN
NUMBERTOTEXT
OFFSET
PACKAGE
PROPERCASE
REVERSEBYTE
RIGHT
SERIESTOTEXT
SIZE
SQUEEZE
SUBSTITUTE
TEXT
TEXTTOBCD
TEXTTODATE
TEXTTONUMBER
TEXTTOTIME
TIMETOTEXT
TODATETIME
TONUMBER
TRIMLEFT
TRIMRIGHT
UPPERCASE
WORD

Conversion
BCDTOHEX
BCDTOINT
BCDTOTEXT
CONVERT
DATETONUMBER
DATETOTEXT
FROMBASETEN
FROMDATETIME
FROMNUMBER
HEXTEXTTOSTREAM
INT
NUMBERTODATE
NUMBERTOTEXT
PACK
PACKAGE
REFORMAT
SERIESTOTEXT
SYMBOL
TEXTTOBCD
TEXTTODATE
TEXTTONUMBER
TEXTTOTIME
TIMETOTEXT
TOBASETEN
TODATETIME
TONUMBER
UNPACK
UNZONE
ZONE



External Interface
DDEQUERY
DBQUERY
DBLOOKUP
ECHOIN
EXIT
HANDLEIN
GET
PUT
RUN
Lookup and Reference
CHOOSE
DBLOOKUP
DBQUERY
DDEQUERY
EXTRACT
GETANDSET
GETDIRECTORY
GETFILENAME
GETPARTIONNAME
GETRESOURCENAME
INDEX
INDEXABS
LASTERRORCODE
LASTERRORMSG
LOOKUP
MEMBER
SEARCHDOWN
SEARCHUP
SORTDOWN
SORTUP
UNIQUE




Bit Manipulation and
Testing
OFFSET
SETOFF
SETON
TESTOFF
TESTON



Error Handling
CONTAINSERRORS
FAIL
ISERROR
LASTERRORCODE
LASTERRORMSG
ONERROR
REJECT
VALID

Date and Time
ADDDAYS
CURRENTDATE
CURRENTTIME
CURRENTDATETIME
DATETONUMBER
DATETOTEXT
FROMDATETIME
MAX
MIN
NUMBERTODATE
TEXTTODATE
TEXTTOTIME
TIMETOTEXT
TODATETIME



Math and Statistics
ABS
COUNT
FROMBASETEN
INT
MAX
MIN
MOD
ROUND
SQRT
SUM
TOBASETEN
TRUNCATE

Logical
ALL
OR
NOT
IF
EITHER
WHEN



What can map rules do?
Map input object to output object
Extract input records based on specific criteria
Count a conditional number of records
Map input records based on specific qualifier to specific output record
Convert an object in input data from one value to another value
Perform conditional logic
Run another map to produce an output
Entering map rules

Map rules entered in rule bar
Colors distinguish functions from arguments, object names and other map rule components
Key words can be configured with different colors
Invalid rule can be highlighted with background color
Always press ENTER to accept changes
Always drag-and-drop or copy and paste object names from cards

Getting ready for map rules

New card default is collapsed view
Expand group objects to lowest level
Grayed rule cells or objects occurring multiple times indicate possible functional
map use
Must enter a rule in every rule cell that is not gray
Mapping items
If output object is an item type, enter map rule at item level

Mapping groups occurring once
If output object is a group that occurs exactly once, enter map rule at group or item level
(not both)

Mapping groups in a series
If a group or an item can occur more than once, decide how many occurrences are to be
built
If there is a known number of outputs, use indexing
If the number of outputs depend upon a variable number of inputs, use a functional
map
Indexing an output
Used when you want to:
Map a specific, known number of outputs one object at a time
Create test data


Indexing an output

Managing indexes
Reference a specific index by enclosing instance number in square brackets
Example: Record [2] to reference the second record number
Add an index
1. Right-click object
2. Select Add Index
Delete an index
1. Right-click object
2. Select Delete Index
Rearrange indexes
1. Select index object
2. Press Ctrl+Shift and drag to desired location
3. Moves rules associated with selected object to new location














Functional maps
What is a functional map?
Like a subroutine
Maps portion of data at a time
Number of output objects are created produced based on some variable number of
corresponding input objects
Is not built and run separately from main map
Does not contain references to data sources and targets
Defining a functional map
1. Determine need for a functional map
2. Determine input arguments to functional map
3. Create functional map call
4. Create the functional map
Manually by creating a new map and input and output cards
Automatically using the Functional Map Wizard
5. Enter map rules in functional map output card
Determining the need
Create a functional map when:
Output group object can occur more than once (component range is greater than 1)
and
Some unknown number of objects are created in output (component range is s)
and
Output group type is different from input group type
Determining the arguments

Functional map syntax is used to:
Name the functional map
Specify data to pass to the functional map
Functional maps only have access to data passed to them through function arguments
Identify which objects are required to produce output
Data objects must be passed from calling map as an argument in the functional map
call
One argument is thought of as a trigger
I want one output object for each ...
Creating the functional map call
Created on a component that occurs an unknown number of times (range of s)
At least one argument should be component that occurs an unknown number of times
Recommended to precede functional map name with F_ to easily distinguish functional
maps
Creating the functional map call example

Functional map example

One input card created for each argument in functional map call
Order of input cards matches order of arguments
One output card always created
Creating functional map manually
1. Create a new map (Map > New)
2. Enter new map name exactly as in functional map reference
3. Add one input card for each argument in the functional map call (Card > New)
4. Add one output card (Card > New)
5. Enter map rules in the functional map
Creating a functional map using the Functional Map Wizard
Automatically creates functional maps based on map rules
Creates input and output cards
Creates multiple maps when a rule contains calls to more than one functional map


Functional Map Wizard results
Functional map is created in the current map source file (.mms) using the parameters
specified in the Functional Map Wizard dialog

Functional Map Wizard icons
Meaning Action to Take
Card is undefined (card name, type
tree file, or type name is not valid)
In the Functional Map Wizard dialog, specify
a valid card name, type tree file, or type
name.
Invalid input argument in a
map rule.
Exit from the Functional Map Wizard and
change the invalid input argument in the
rule.
Everything has been properly
specified for given card or map
No action necessary. Functional map can be
created.
Modifying card attributes

Functional Map Wizard restrictions
Does not automatically create input cards for arguments defined as expressions (such as
literals) that evaluate to text or number items
Does not run if an argument is invalid
Does not dynamically update if changes are made to the functional map call
Main and functional map input cards

Functional map output cards

From arguments to cards

Functional map input cards
One input card must be created for each argument passed from calling map
Input cards must be created in same order as arguments

Functional map output card
Only one output card per functional map
Type matches type of output in calling rule
Uses same type tree as output card in calling map

Argument must match input card
Card must match output type

Evaluating functional maps
Evaluated once for each combination of occurrences of each argument passed
To conditionally control input to a functional map or output of an object, use functions and
expressions within a functional map call
Example
=F_MakeNewOrderRecord (EXTRACT(LegacyRecord:LegacyFile, PARTITION ( ShipToCode
Field:.:LegacyFile, West)), XrefFile )
If any argument evaluates to NONE, the functional map will not be called for that
combination of arguments
Functional maps build output; functions do not
Tracking rules
Rules are saved with output name, not position
If output objects change sequence, rules are maintained
If object disappears or the name changes, rules become unresolved

What causes unresolved rules
- Name of output type changes in type tree
- Output card is changed to point to a different object
- An intermediate level is added between object in output

Displaying Unresolved Rules view
Automatically displayed when unresolved rules are present
Can be selected from Show View menu

Resolving unresolved rules
Drag rules from unresolved rules into place
Use Modify to find and replace changed type names
Must delete all unresolved rules before building map


Map Debug option
Debug complex rules step by step in Map Designer
Set breakpoints on any rule, in executable or functional maps
Visually step into every rule in every map
Inspect data while it is changing
Step In, Step Out, Step Over, and Run commands
Debug maps remotely by referencing map source locally
Map Debugger features
Enable the Debug Map icon
to switch Map Designer into debug mode
When enabled, allows setting of break points on output objects with rules
Building maps with debug mode enabled produces separate map debug file
No changes to compiled map (.mmc) structure, so maps built with debug are same as other
compiled maps
Map Debugger usage

Map Debugger operation
Enable debug and build map
Set breakpoints
Run map
Debug mode causes Debug view window to appear
When breakpoint is reached, engine pauses and the rule and its composite functions
and objects appear in Debug view window
Use buttons in Debug view window to continue processing
Adding breakpoints (1 of 2)


Debugger view window

Map Debug options
Step In to see what data is held by objects referenced in rule and data returned by functions
and functional maps
Steps into local, non-referenced functional maps and RUN maps contained in same
map source file (.mms)
Literals and operators are not shown
Step Thru goes to end of current rule
Does not show resultant object values or function returns
Step Over ignores current iteration of a rule and goes to beginning of the next rule to be run
Step Thru/Over processes rules in functional maps or RUN maps called from the current
rule, unless a breakpoint is set