VImp Professional User Manual PDF

Visual Importer ETL
Professional & Enterprise

User manual

DB Software Laboratory 2008-2014

Visual Importer ETL Professional & Enterprise User Manual

Copyright 2014 DB Software Laboratory

Page 2 of 291

www.etl-tools.com
Contents

Contents ....................................................................................................................................... 2
1. Introduction ............................................................................................................................. 8
2. Requirements ........................................................................................................................... 8
3. Key features ........................................................................................................................... 10
4. Visual Importer ETL Architecture ......................................................................................... 12
5. Demo Data ............................................................................................................................. 13
6. Options .................................................................................................................................. 15
6.1 Global Variables .............................................................................................................. 20
7. User Interface ........................................................................................................................ 21
7.1 System Menu ................................................................................................................... 21
7.2 Visual Importer ETL Graphical User Interface: Main window ....................................... 21
7.2.1 Quick Access Tabs ................................................................................................... 21
7.2.2 Objects tree toolbar ................................................................................................... 22
7.2.3 Objects tree description ............................................................................................ 22
7.2.4 Objects tree list ......................................................................................................... 23
7.2.5 Objects Groups ......................................................................................................... 25
8. Directories and Connections .................................................................................................. 28
8.1 Directories ....................................................................................................................... 28
8.2 Database Connections ..................................................................................................... 29
8.2.1 Oracle Connection .................................................................................................... 30
8.2.2 JDBC Connection ..................................................................................................... 31
8.2.3 Microsoft SQL Server Connection ........................................................................... 32
8.2.4 ODBC Connections .................................................................................................. 33
8.2.5 ODBC Connection Strings ....................................................................................... 35
8.2.6 MySQL Connections ................................................................................................ 36
8.2.7 PostgreSQL Connection ........................................................................................... 38
8.2.8 Interbase/Firebird Connection .................................................................................. 40
8.2.9 Ole DB Connection .................................................................................................. 42
8.2.10 SQLite Connection ................................................................................................. 43
8.2.11 Microsoft SQL Server CE Connection ................................................................... 44
8.2.12 BDE Connection ..................................................................................................... 45
8.3 POP3 Connection ............................................................................................................ 48
8.4 IMAP4 Connection .......................................................................................................... 49
8.5 SMTP Connection ........................................................................................................... 50
8.6 IP Socket connection ....................................................................................................... 51
8.7 FTP Connection ............................................................................................................... 52
8.8 Google Spread sheet connection...................................................................................... 55
8.9 SalesForce connection ..................................................................................................... 56
9. Importing Data ....................................................................................................................... 60
9.1 Mapping editor screen overview ..................................................................................... 62
9.1.1 Main tool bar ............................................................................................................ 63
9.1.2 Source tool bar .......................................................................................................... 63
9.1.3 Mapping panel .......................................................................................................... 64
9.1.4 SQL Statements ........................................................................................................ 65


Page 3 of 291

9.1.5 Template tab ............................................................................................................. 66
9.1.6 Log File tab ............................................................................................................... 67
9.1.7 Rejected Records Tab ............................................................................................... 68
9.1.8 Import Process .......................................................................................................... 69
9.2 Data Mapping for a Flat File Data Source ....................................................................... 70
9.3 How to clear mapping ..................................................................................................... 76
9.4 Auto mapping .................................................................................................................. 77
9.5 Data Mapping for an ODBC Data Source ....................................................................... 80
9.7 Loading data from the pivot tables ................................................................................. 84
9.8 Performing Calculations .................................................................................................. 86
9.9 Filtering Records ............................................................................................................. 87
9.10 Working with Date fields .............................................................................................. 88
9.11 How to Update/Delete Records ..................................................................................... 89
9.12 Using Database Specific functions. ............................................................................... 91
9.13 Error Handling ............................................................................................................... 92
9.14. Date formats ................................................................................................................. 93
10. Data sources ......................................................................................................................... 94
10.1 Data source is a XML file .............................................................................................. 94
10.2 Data source is Excel file ................................................................................................ 96
10.3 Data source is Qvx file .................................................................................................. 98
10.4 Data source is Windows Event Log .............................................................................. 99
10.5 Data source is MS Access ........................................................................................... 100
10.6 Data source is a DBF File ............................................................................................ 105
10.7 Data source is a ODBC Connection ............................................................................ 106
10.8 Data source is a JDBC Connection ............................................................................. 108
10.9 Data source is an Ole DB Connection ......................................................................... 109
10.10 Data source is a MS SQL Server ............................................................................... 110
10.11 Data source is MS SQL Server CE ............................................................................ 111
10.12 Data source is an Oracle Database ............................................................................ 112
10.13 Data source is an MySql ............................................................................................ 113
10.14 Data source is an PostgreSQL Database .................................................................... 114
10.15 Data source is an Interbase or Firebird Database ...................................................... 115
10.16 Data source is SQLIte ................................................................................................ 116
10.17 Data source is Google Spread Sheets ........................................................................ 117
10.18 Data source is RSS feed ............................................................................................ 118
10.19 Data source is SalesForce .......................................................................................... 119
10.20 Data source is BrightPearl ......................................................................................... 120
10.21 Data source is BDE ................................................................................................... 123
11. Data targets ........................................................................................................................ 124
11.1 Target type is ODBC connection ................................................................................. 124
11.2 Target type is Ole DB .................................................................................................. 126
11.3 Target type is SQLIte ................................................................................................... 127
11.4 Target type is MS SQL Server ..................................................................................... 128
11.5 Target type is MS SQL Server CE .............................................................................. 130
11.6 Target type is Oracle .................................................................................................... 131
11.7 Target type is MySql ................................................................................................... 133
11.8 Target type is PostgreSQL ........................................................................................... 134
11.9 Target type is Interbase/Firebird .................................................................................. 135
11.10 Target type is BDE .................................................................................................... 136


Page 4 of 291

11.11 Target type is a stored procedure ............................................................................... 137
12. SQL Scripts ....................................................................................................................... 138
12.1 SQL Scripts Tool Bar .................................................................................................. 140
12. Reports ............................................................................................................................... 141
12.1 Report Designer ........................................................................................................... 141
12.1.1 Report Title ........................................................................................................... 142
12.1.2 Page Header .......................................................................................................... 142
12.1.3 Group Header ....................................................................................................... 142
12.1.4 Code ...................................................................................................................... 142
12.1.5 Data....................................................................................................................... 142
12.1.6 Report tool bar ...................................................................................................... 143
12.1.7 Report Properties .................................................................................................. 143
12.1.8 Save Report to the Repository .............................................................................. 143
12.1.9 Report Connection ................................................................................................ 143
12.1.10 Show Only Connections for Current Project ...................................................... 143
12.1.11 Manage Versions ................................................................................................ 144
12.1.12 Add Versions ...................................................................................................... 144
12.1.13 Creating Basic Report ......................................................................................... 144
12.1.14 Report in Report Designer .................................................................................. 150
13. Email Templates ................................................................................................................ 152
14. Packages ............................................................................................................................ 154
14.1 Package screen overview ............................................................................................. 156
14.2 Joining Actions ............................................................................................................ 157
14.3 Package Tool bar ......................................................................................................... 158
14.4 Debugging Package ..................................................................................................... 160
14.6 Working with filenames and directories ...................................................................... 166
14.7 Import Action .............................................................................................................. 167
14.8 SQL Data Check Action .............................................................................................. 168
14.9 SQL Script Action ....................................................................................................... 169
14.10 Package Action .......................................................................................................... 170
14.11 FTP Action ................................................................................................................ 171
14.12 HTTP Download Action............................................................................................ 174
14.13 Archive Action .......................................................................................................... 175
14.14 Receive Email POP3 Action ..................................................................................... 177
14.15 Receive Email IMAP4 Action ................................................................................... 180
14.17 External Application Action ...................................................................................... 185
14.18 Check file action ........................................................................................................ 186
14.19 File Operation Action ................................................................................................ 187
14.20 Compare Files Action ................................................................................................ 189
14.21 Check Disk Space Action .......................................................................................... 190
14.22 Backup Repository Action ......................................................................................... 191
14.23 Pause Action .............................................................................................................. 192
14.24 Check date action ...................................................................................................... 193
14.25 Report Action ............................................................................................................ 194
14.26 Export Data Action .................................................................................................... 196
14.27 Show Message and Ask Question Action ................................................................. 200
14.28 Set Variable Action ................................................................................................... 202
14.29 Script Action .............................................................................................................. 204
14.30 Check Connection Action.......................................................................................... 207


Page 5 of 291

14.31 PING Server Action.................................................................................................. 208
14.32 Case Action ............................................................................................................... 209
14.33 GPG4WIN Action ..................................................................................................... 211
14.34 HTTP Form Action.................................................................................................... 213
15. Lookups ............................................................................................................................. 215
15.1 Tool bar ....................................................................................................................... 216
16. Scheduler ........................................................................................................................... 217
16.1 Tool bar ....................................................................................................................... 217
17. Execution Monitor ............................................................................................................. 222
17.1 Tool bar ....................................................................................................................... 224
17.2 Stopping execution ...................................................................................................... 224
18. SQL .................................................................................................................................... 225
18.1 SQL Toolbar ................................................................................................................ 225
18.2 Side toolbar .................................................................................................................. 226
19. Version Control ................................................................................................................. 227
20. Using Command Line interface ......................................................................................... 228
21. Scripting Language ............................................................................................................ 231
21.1 The Basic Structure ..................................................................................................... 231
21.2 Variables ...................................................................................................................... 232
21.3 Logical Operations ...................................................................................................... 237
21.4 Repeating sets of commands ....................................................................................... 241
21.5 Functions ..................................................................................................................... 244
21.6 Script Examples ........................................................................................................... 245
21.7 Passing variables between objects ............................................................................... 246
21.8 Supported Functions List ............................................................................................. 247
21.8.1 String Functions.................................................................................................... 247
21.8.2 Conversion Functions ........................................................................................... 252
21.8.3 File System Functions .......................................................................................... 254
21.8.4 Date and Time Functions ...................................................................................... 258
21.8.5 Numeric Functions ............................................................................................... 262
21.8.6 Miscellaneous Functions ...................................................................................... 264
21.8.7 Procedures ............................................................................................................ 266
21.8.8 Math functions ...................................................................................................... 267
22. Execution Agent ................................................................................................................ 272
Parallel Execution ................................................................................................................ 273
Checking Agent Status ........................................................................................................ 274
23. Repository database ........................................................................................................... 275
Transferring data between repositories ................................................................................ 275
Repository objects synchronization ..................................................................................... 276
Creating new repository ....................................................................................................... 278
ODBC Settings for PostgreSQL repository ......................................................................... 279
ODBC Settings for MySQL repository ............................................................................... 281
ODBC Settings for Interbase/Firebird repository ................................................................ 285
24. Management Console ........................................................................................................ 286
25. Support Procedure ............................................................................................................. 287
26. Dynamic file names ........................................................................................................... 288
27. License Agreement ............................................................................................................ 289


Page 6 of 291

Copyright 2008-2014 DB Software Laboratory Limited. All rights reserved.

No portion of this document may be reproduced, transmitted, transcribed, stored in
a retrieval system, or translated into any language, in any form or by any means,
without prior written consent of DB Software Laboratory Limited.

Note to U.S. Government users:
Documentation and programs related to restricted rights - use, duplication or
disclosure is subject to restrictions set forth in:
GSA FMSS Schedule Contract No. GS00K92AFS2505-PS05.

License Information

You have access to Visual Importer Professional and Enterprise software and
documentation pursuant to the terms of a Software License Agreement granted by
DB Software Laboratory Limited. As a user of this software and documentation, you
are bound by the terms of the Software License Agreement. All rights, title, and
interest to this software remain with DB Software Laboratory Limited.
Requests for copies of this publication and for technical information about DB
Software Laboratory products should be made directly to DB Software Laboratory
Limited.

Disclaimer

All information in this manual is subject to periodic change and revision without
notice. While every effort has been made to ensure that this manual is accurate, DB
Software Laboratory Limited excludes its liability for errors or inaccuracies (if any)
contained herein.

Registered Marks

Any products or services mentioned or depicted in this document are identified by
the trademarks or service marks of their respective companies or organisations.

Edition Information

This document refers to Visual Importer ETL Professional and Enterprise version
8.3.8.3


Page 7 of 291

Copyright

Copyright 2008-2014 DB Software Laboratory Limited. All rights reserved.

No portion of this document may be reproduced, transmitted, transcribed, stored in
a retrieval system, or translated into any language, in any form or by any means,
without prior written consent of DB Software Laboratory Limited.

Note to U.S. Government users:
Documentation and programs related to restricted rights - use, duplication or
disclosure is subject to restrictions set forth in:
GSA FMSS Schedule Contract No. GS00K92AFS2505-PS05.

License Information

You have access to Visual Importer ETL Professional and Enterprise software
and documentation pursuant to the terms of a Software License Agreement granted
by DB Software Laboratory Limited. As a user of this software and documentation,
you are bound by the terms of the Software License Agreement. All rights, title, and
interest to this software remain with DB Software Laboratory Limited.
Requests for copies of this publication and for technical information about DB
Software Laboratory products should be made directly to DB Software Laboratory
Limited.

Disclaimer

All information in this manual is subject to periodic change and revision without
notice. While every effort has been made to ensure that this manual is accurate, DB
Software Laboratory Limited excludes its liability for errors or inaccuracies (if any)
contained herein.

Registered Marks

Any products or services mentioned or depicted in this document are identified by
the trademarks or service marks of their respective companies or organisations.

Edition Information

This document refers to Visual Importer ETL Professional and Enterprise version
8.3.1.4


Page 8 of 291

1. Introduction

Visual Importer is a business intelligence tool that provides an integrated solution for
designing and deploying data warehouses and data marts. It solves complex problem of
integration between different data sources and targets. It combines components of both ETL
and designs tools into one tool.

2. Requirements

Below is the list of Software that must be installed before installation of Visual Importer:

Software Version Notes
Microsoft Windows 98 or higher
MDAC 2.6 or higher Part of OS on W2K,
XP, Vista,Win7,Win8.

MS Access ODBC
driver
4.00.6364.00 or
higher
Part of OS on W2K,
XP, Vista.
Only to work with MS
Access Databases
MS Access 2007
ODBC driver
12.00.4518.1014 or
higher
Separate download Only to work with MS
Access 2007/2010
Databases
FoxPro ODBC
driver
6.1.8629.1 or
higher
Separate download Only to work with
DBF/FoxPro Files
SQL ODBC driver 2000.81.9041.40 Part of OS on W2K,
XP, Vista.
Only to work with MS
SQL Server 7/2008
Interbase client GDS32.DLL Only to work with
Interbase or Firebird
Databases
SQLite Sqlite3.dll Only to work with
SQLite databases
Oracle Client 7.3.4 or higher Provided by Oracle Only to work with
Oracle Databases
BDE Drivers Latest Version Only to work with
Paradox or other BDE
databases
Jet 4.0 Service pack 5 or
higher
Part of OS on W2K,
XP, Vista,Win7,Win8.
To connect to MS
Access repository

Separate Downloads:

FoxPro ODBC driver
http://msdn.microsoft.com/en-us/vfoxpro/bb190233.aspx

Office 2007 Data Access Components
http://www.microsoft.com/downloads/details.aspx?FamilyID=7554F536-8C28-4598-9B72-
EF94E038C891&displaylang=en



Page 9 of 291

BDE
www.etl-tools.com/dmdocuments/BDE.zip

Working with Oracle:

Oracle client 8.1.7 to load data into/from Oracle
Or
Oracle client 9 to load data into/from Oracle
Or
Or

Note:
Depending on the Requirements you may or may not need to have all components installed
There is no need to install clients for MySql and PostgreSQL as they are integrated into the
software itself.

Repository Scripts

Repository Creation scripts are provided for following databases:

MS Access
Oracle
MS Sql Server
Interbase/Firebird
MySQL
PostgreSQL

Note:
Once installation is completed user may find all the scripts in
C:\Users\Public\Documents\DBSL\Repository Scripts\Repository


Page 10 of 291

3. Key features

Data import

Data targets:
Oracle 7-11g database (using OCI API)
SQL server 7- 2008 (using BCP API)
ODBC source (using ODBC API)
Interbase/Firebird
MySQL
PostgreSQL
SQLite
Ole DB
BDE

Data sources:

Multiple Delimited or Fixed width Text files
Multiple Excel files + Multiple Excel Spreadsheets
Multiple MS Access Databases + Multiple Tables
Multiple DBF Files
QlikView QVX files
Multiple Tables
Oracle 7-11g database
MS SQL Server
ODBC source
Interbase/Firebird
MySQL
PostgreSQL
SQLite
Ole DB
XML
Windows Event log
Google Spreadsheets
SalesForce
BrightPearl
BDE

Repository:
MS Access
Oracle
MS Sql Server
Interbase
MySQL
PostgreSQL



Page 11 of 291

This product features:
Great performance - thousands of records per second
Comprehensive logging
Rejected records file
Integrated Expressions builder
Filtering Data

Allows the user to perform calculation during the loading process such as splitting fields,
concatenations, data formatting, and loading cross/pivot tables.

Oracle

There are two ways of loading available:
Oracle direct path loading
Conventional path

MS SQL server

This software uses the same API as Microsoft DTS/SSIS service.

ODBC

Any ODBC compliant database.

Other features are:

SQL Scripts
Email Templates
Report Designer
Integrated Scheduler
Logging
Email Notifications
POP3 Email Receiver
IMAP4 Email Receiver
File Operations
Check Files
Check Disk Space
Compare Files
Full range of FTP operations
ZIP
Applications
SQL Data Check
Automatic Backups
Mass Data Export
Http Download
Ping Server
Check Database Connection



Page 12 of 291

4. Visual Importer ETL Architecture



Page 13 of 291

5. Demo Data

In order to provide a flavour of the types of data the Visual Importer ETL can
handle, a number of different file types have been provided for demonstration
purposes.

A range of different types of data files have been provided, in addition to SQL scripts
to create database tables in Oracle, and MS SQL Server.

Text files

Examples provided are using text files in

C:\Users\Public\Documents\DBSL\Demo\Buffer and
C:\Users\Public\Documents\DBSL\Demo\Text Files

Access Databases

C:\Users\Public\Documents\DBSL\Demo\MS Access

DBF Files

C:\Users\Public\Documents\DBSL\Demo\DBF

Excel Files

C:\Users\Public\Documents\DBSL\Demo\Excel

Note:
If user wants to use other locations please amend Directories properties.

Demo tables

Use SQL Scripts provided to create demo tables for Oracle and MS SQL Server.
Please adjust connection details before executing these scripts.

Most of the SQL Server Imports use DEMO database. User have to create demo
tables within this database first before executing import scripts.

ODBC connections

During the setup process, the following ODBC DSN is created during the installation:

ODBC_FOXPRO
ODBC_ACCESS_SOURCE
ODBC_ACCESS_TARGET
ODBC_MS
ODBC_ORACLE
ODBC_EXCEL


Page 14 of 291

These ODBC connections provide the basis for the Visual Importer ETL to perform
its tasks and process different types of data, while ensuring that the automation of
these processes is completely transparent.


Page 15 of 291

6. Options

The Visual Importer ETL allows user to tweak key settings. Using this provision
user can change the repository type, control the execution of packages and where
the results will be logged, set interface actions such as the prevention of accidental
deletion and set email options to report the outcome of the automation to individuals
or groups of people.

To change Visual Importer ETL settings click System menu-> File-> Options.

The Dialog box will appear.



Page 16 of 291

The Repository tab defines the repository connection. Once the logon parameters
have been verified via the Test Connection dialogue this information is saved so it is
not necessary to provide this information again.

The Repository type can be:

MS Access
MS SQL Server
Oracle
Interbase (Firebird)
MySql
PostgreSQL

Note:
Use Repository creation wizard to create new repository.

Execution tab defines settings related to the logging and Packages execution.


Page 17 of 291

Note:
To enable parallel execution select following options
Execute packages in separate thread
Use Agent to execute Packages



Page 18 of 291

The Interface tab defines key settings. These operate globally during the activities of
the Visual Importer ETL. For instance, in order to save time when creating packages
to automate tasks, the editor can be set so that it automatically opens. In order to
ensure that any changes made to scripts and packages upon exit, user can
safeguard this by checking the Save all scripts and packages on exit box.
Accidental deletion is prevented by requesting that the user is asked before deleting
any objects. Another aspect of the interface actions is the updating of the log. This
can be set so that once a package has completed its actions the log is updated with
all the actions which took place during execution. Finally, if user wishes to
automatically refresh table list when creating/opening transformation you simply
check the Update Tables List to ensure that user this is accurate, and up to date.



Page 19 of 291

The Email tab defines settings required to send Emails. As the Visual Importer ETL
is designed to work automatically and in batch mode, it will be necessary to specify
where emails will be sent from. Emails messages are used to convey when
processes and activities have been completed. It is also possible to specify the
default recipient. The default recipient can be set to a group email address, or it can
be left blank.



Page 20 of 291

6.1 Global Variables

Global Variables are used to replace Variable with Value, for example before SQL is executed
<CustomerId> is be replaced with 1

EG

Select * from Customers where CustomerId =<CustomerId>

Is changed to

Select * from Customers where CustomerId =1

Predifined variables:

<username> OS user name
<computername>
<datetime>



Page 21 of 291

7. User Interface
Upon starting the Visual Importer ETL software, user will be presented with the
main window. This is where the main tasks are controlled such as processing cycles,
and other processing tasks taking place in the processor.
7.1 System Menu

The system menu allows user access to the essential setting and tools user will
need to use the processor.
7.2 Visual Importer ETL Graphical User Interface: Main window

System Menu
Objects tree
Status bar
Quick Access Tabs
Objects tree toolbar


Page 22 of 291

7.2.1 Quick Access Tabs

The "Quick Access Tabs" gives the user easy access to various functions of
application:

The titles of the tabs are self-explanatory, default tab is design.

7.2.2 Objects tree toolbar

1 2 3 4 5 6 7 8 9 10 11

1. Selected object properties
2. Refresh object tree
3. Create new object
4. Delete selected object
5. Cut
6. Copy
7. Paste
8. Find Object
9. Find Next Object
10. Expand selected tree node
11. Collapse selected tree node
7.2.3 Objects tree description

The "Objects" tree lists the main categories for the provision of objects. For instance,
the "Directories" category contains icons representing the location of files and related
information belonging to an individual item. For example, by clicking on the "Excel
Files" icon you will access the directory area where all the Excel files are stored for
processing.

You may also find icons representing individual database connection services. For
instance, you can see connections available for both MS SQL Server and Oracle.

Other object types include the ODBC connections. You can see from the illustration
that the DEMO system has been configured, and as a result the ODBC connections
created are listed in the objects tree. The Visual Importer ETLprovides the Objects
tree so that users have easy access to the main objects. The full list of objects is
described below:


Page 23 of 291

7.2.4 Objects tree list

Below is a list of possible objects with the short descriptions:

Projects Provides a way to group objects together

Directories Defines path to flat files, Excel files, MS Access databases or
DBF files

JDBC Connection Defines JDBC connection properties

SQL Server CE
Connection
Defines MS SQL Server CE connection properties

SQL Server Connection Defines MS SQL Server connection properties

ODBC Connection Defines ODBC connection properties

Oracle Connection Defines Oracle connection properties

Ole DB Defines Ole DB connection properties

MySQL Defines MySql connection properties

PostgreSQL Defines PostgreSQL connection properties

SQLite Defines SQLite connection properties

Interbase/Firebird Defines SQLite connection properties

FTP Connection Defines FTP Server connection properties

SMTP Connection Defines SMTP connection properties

POP3 Connection Defines POP3 server connection properties

IMAP4 Connection Defines IMAP4 server connection properties

SalesForce Connection Defines SalesForce connection properties

BrightPearl Connection Defines BrightPearl connection properties

Sugar CRM
Connection
Defines Sugar CRM connection properties

Google analytics
Connection
Defines Google analytics connection properties

Google Spreadsheet
Connection
Defines Google Spreadsheet connection properties

Youtube Connection Defines Youtube connection properties


Page 24 of 291

Twitter Connection Defines Twitter connection properties

IP Socket Connection Defines IP Socket connection properties

Lookups and Lookups
Groups
Provides an easy access to user defined data entry screens

Imports and Imports
Groups
Defines mapping between Data source and Data target

Reports and Reports
Groups
Reports design

SQL Scripts and
Scripts Groups
Defines SQL statements to perform against target Databases

Email templates and
groups
Creates general email templates.

Packages and packages
groups
Combines complex Actions together like Ftp downloads File
operations, emails, Check files, SQL scripts and
Transformations.



Page 25 of 291

7.2.5 Objects Groups

The Visual Importer ETL also has the ability to group objects together into groups. The
purpose of doing this is so that objects of a specific type can be viewed as a branch of the
object tree. Whenever a new object is added it will be placed under the appropriate branch for
its type.

Groups of Objects


Page 26 of 291

To create a new object group select appropriate object type for example
transformation and click new.

Depending on the object type, the appropriate dialogue will be presented. In this
example the Transformation Group has been selected. The dialogue in this case
requests a description and comment about the object to be created. Some of the
fields in a dialogue of this type will be mandatory and will therefore have to be
entered. However, comments will be optional, and just provide the user with the
ability to specify more detail about an object and what it does.



Page 27 of 291



Page 28 of 291

8. Directories and Connections
8.1 Directories

If a new directory is required, selecting New on the new directory icon brings up the
following dialogue:

The following steps are needed to allow creation of a directory, as follows:

In the Name Text Box type in a new name for the directory you are about to create
Fill in Directory path you wish to load data from
Click OK to close the directory Properties Window

Note:
The user may change connection or directory properties at any time by double
clicking on it.
It is also possible to use sub directory in existing directory
<DirectoryName>\Subdirectory.

For example <Buffer>\tmp will expand to:

C:\Program Files\DB Software Laboratory\Demo\Buffer\tmp



Page 29 of 291

8.2 Database Connections

The database connection functionality provides a way of connecting to database
servers.
Once a connection is set up, it is always available until deleted or amended. In order
to set up a new connection it will be necessary to have database credentials to hand.

When setting up database connections, you should ensure that the account you
wish to use has access to all the available functionality you may wish to use. For
instance, it is not much use, setting up a connection to an Oracle account to use
certain stored procedures, if the underlying Oracle permissions are not set for this
facility. The account used in the Visual Importer ETL for the database should allow
the user to perform underlying database actions, which may not be under control of
the processor directly.

To create a new connection select appropriate connection type and click New.

Once this step has been completed, you will be provided with an appropriate
dialogue which will request information about the new connection you wish to create,
and the type of connection you are creating.



Page 30 of 291

8.2.1 Oracle Connection

For an Oracle connection you will need to specify the TNS name required, and
provide the user name and password for the connection.

Note:
It is also possible to use Oracle instant client. (servername:1521/ORCL)

The procedure to set up an Oracle Connection is as follows:

In the Name Text Box type in a new name for the connection you are about to create
Select a TNS Name from the Server Drop Down List
Fill in Username/Password for the database you wish to connect to

a. If you are unsure of these parameters, please contact your Database
Administrator for the correct settings.

Click Test to ensure the details you have provided are correct
Click OK to close the connection properties window
Browse connection
tables and data


Page 31 of 291

8.2.2 JDBC Connection

JDBC is a Java-based data access technology (Java Standard Edition platform)
from Sun Microsystems, Inc.. It is an acronym as it is unofficially referred to as Java
Database Connectivity, with DB being universally recognized as the abbreviation for
database. This technology is an API for the Java programming language that defines
how a client may access a database. It provides methods for querying and updating
data in a database. JDBC is oriented towards relational databases. A JDBC-to-ODBC
bridge enables connections to any ODBC-accessible data source in the JVM host
environment.

Source: Wikipedia.

Note:
SQL builder does not work with JDBC connections. List of JDBC drivers can be found
here http://developers.sun.com/product/jdbc/drivers



Page 32 of 291

8.2.3 Microsoft SQL Server Connection

The procedure for many database connections is very similar. In the case of MS
SQL Server it is necessary to specify the server, database name and user
name/password combination.

An SQL Server connection setup requires the following steps:

Select a Server Name from Server Drop Down List
Select a Database Name form the Drop Down List

a. If you are unsure of these parameters, please contact your Database
Administrator for the correct settings.


Note:
Leave user name and password blank to use Windows Authentication.
Double click to update
list of databases


Page 33 of 291

8.2.4 ODBC Connections

ODBC history

ODBC, standing for Open Data Base Connectivity, represents a standard database
access method working as SQL API software. It was developed back in 1992 with the
sole purpose of facilitating the work with database management systems (DBMS). It
was created by the Microsoft corporation and is since then included in every copy of
Microsoft Windows. In 1995 the ODBC was included in the SQL Standard, which
played a great role in making ODBC more popular. Today, there are versions for
almost any operating system currently used.

How ODBC works

ODBC was intended to enable developers' access to any data through any
application, regardless of the DBMS used for managing that data. ODBC boasts
platform independence since it has been purposefully designed in a way that makes it
distinct from database systems, programming languages and operating systems.

Facilitating the data access from an application to a database management system
through ODBC is done through a specific mechanism. A common ODBC
implementation contains one or more applications, a core ODBC 'Driver Manager'
library, and one or more database drivers. The Driver Manager's role is to interpret
the data queries coming from an application by using the DBMS-specific details
contained in database drivers. The latter represent a middle layer inserted between
an application and the DBMS in use. This way, the application's data queries are
translated into commands that can easily be read by the DBMS.

A basic requirement for an ODBC implementation to be run is that both the
application and the DBMS be ODBC-compliant. In other words, the application must
be able to issue ODBC commands and the DBMS must be capable of responding to
them.

Thanks to its modular nature ODBC gives developers a great freedom in creating
the separate components of an ODBC implementation. Thus, a programmer can
write applications that use standard features without needing to worry about the type
of DBMS used for administering the database that the application tries to access.
Likewise, the only thing the database driver programmers need to keep in mind
during the development process is how to attach their database driver to the 'Driver
Manager' library. The result is that currently there are hundreds of ODBC drivers
created for a large variety of data sources.

Thanks to the long period of existence and the fruitful efforts of its team of
developers, ODBC now offers access to a much wider range of data sources than
any other database access method available today. It has turned into a universal data
access standard, working with a great variety of operating systems and providing
access to even non-relational data, including text and XML files.



Page 34 of 291

The ODBC Connection properties dialogue allows the administrator to create
appropriate connections to various databases. It is be necessary to create
connections to databases, whenever they are to be processed.

In order to create an ODBC connection, the following steps should be followed:

Select a ODBC DSN Name from Server Drop Down List

a. You may use ODBC admin to create/modify ODBC data sources.

Build ODBC connection
string


Page 35 of 291

8.2.5 ODBC Connection Strings

It also possible to use ODBC connection strings for both Reader and Writer
connections.
For example for MS SQL Server connection string is:

Driver={SQL Native Client};
Server=myServerAddress;Database=myDataBase;Uid=myUsername;Pwd=myPassword;

One of the major benefits of using connection strings that it makes it no longer necessary to
create ODBC Dsns manually on every single computer where Visual Importer ETL is
installed. It also gives a greater control over the connection parameters.

Note:
Leave user name and password blank and provide it within connection string

More information about connection strings can be found at:
http://www.connectionstrings.com

The simplest way to create ODBC connection string is to use ODBC Connection builder
dialog. Double click on ODBC driver name to create a connection string



Page 36 of 291

8.2.6 MySQL Connections

About MySQL

The MySQL database has become the world's most popular open source
database because of its consistent fast performance, high reliability and ease of use.
It's used on every continent -- Yes, even Antarctica! -- by individual Web developers
as well as many of the world's largest and fastest-growing organizations to save time
and money powering their high-volume Web sites, business-critical systems and
packaged software -- including industry leaders such as Yahoo!, Alcatel-Lucent,
Google, Nokia, YouTube, and Zappos.com.

Creating MySQL connection

In order to create a MySQL connection, the following steps should be followed:

Type in the host name
Select a Database name from the drop down List
Select appropriate connection port default is 3306


Page 37 of 291

Click Test Connection to ensure the details you have provided are correct


Page 38 of 291

8.2.7 PostgreSQL Connection

About PostgreSQL

PostgreSQL is a powerful, open source object-relational database system. It has
more than 15 years of active development and a proven architecture that has earned
it a strong reputation for reliability, data integrity, and correctness. It runs on all major
operating systems, including Linux, UNIX (AIX, BSD, HP-UX, SGI IRIX, Mac OS X,
Solaris, Tru64), and Windows. It is fully ACID compliant, has full support for foreign
keys, joins, views, triggers, and stored procedures (in multiple languages). It includes
most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR,
VARCHAR, DATE, INTERVAL, and TIMESTAMP. It also supports storage of binary
large objects, including pictures, sounds, or video.

Creating PostgreSQL connection

In order to create a PostgreSQL connection, the following steps should be followed:



Page 39 of 291

Type in the host name
Select a Database name from the drop down List
Select appropriate connection port default is 5432


Page 40 of 291

8.2.8 Interbase/Firebird Connection

About Interbase

InterBase is a relational database management system (RDBMS) currently
developed and marketed by Embarcadero Technologies. InterBase is distinguished
from other DBMSs by its small footprint, close to zero administration requirements,
and multi-generational architecture. InterBase runs on the Linux, Microsoft Windows,
Mac OS X and Solaris operating systems.

About Firebird

Firebird is a relational database offering many ANSI SQL standard features that
runs on Linux, Windows, and a variety of Unix platforms. Firebird offers excellent
concurrency, high performance, and powerful language support for stored procedures
and triggers. It has been used in production systems, under a variety of names, since
1981.

Creating Interbase/Firebird connection



Page 41 of 291

In order to create an Interbase/Firebird connection, the following steps should be
followed:

Type in the database name


Page 42 of 291

8.2.9 Ole DB Connection

About Ole DB

OLE DB (Object Linking and Embedding, Database, sometimes written as OLEDB
or OLE-DB) is an API designed by Microsoft for accessing different types of data
stored in a uniform manner. It is a set of interfaces implemented using the
Component Object Model (COM); it is otherwise unrelated to OLE. It was designed as
a higher-level replacement for, and successor to, ODBC, extending its feature set to
support a wider variety of non-relational databases, such as object databases and
spreadsheets that do not necessarily implement SQL.

Creating Ole DB connection

In order to create an Ole DB connection, the following steps should be followed:

Type in the connection string or use Build Connection String button to create one.
Note:
Ole DB is one of the slowest ways of accessing the data and it uses a lot of memory. It is
not recommended for very large datasets.
Build Ole Db Connection string
connection string


Page 43 of 291

8.2.10 SQLite Connection

SQLite is a software library that implements a self-contained, serverless, zero-
configuration, transactional SQL database engine. SQLite is the most widely deployed SQL
database engine in the world. The source code for SQLite is in the public domain.

To create a SQLite connection, follow these steps:

In the Name Text Box type in a new name for the SQLite connection you are about to
create
Select Database name



Page 44 of 291

8.2.11 Microsoft SQL Server CE Connection

SQL Server Compact Edition contains the runtime bits that can be used to develop
applications and to deploy applications on desktop computers and on Tablet PCs. SQL Server
Compact Edition is a powerful yet lightweight relational database engine. SQL Server
Compact Edition supports familiar SQL syntax. Additionally, SQL Server Compact Edition
provides a development model and an API that are consistent with SQL Server. Therefore, you
can more easily develop desktop applications by using the RTW version of SQL Server
Compact Edition than by using SQL Server Mobile. SQL Server Compact Edition includes
new features such as |DataDirectory| substitution string support and ClickOnce support so that
deployment is easy.

To create a SQL Server Compact Edition connection, follow these steps:

In the Name Text Box type in a new name for the SQL Server Compact Edition connection
you are about to create
Select Database name



Page 45 of 291

8.2.12 BDE Connection

Borland Database Engine (BDE) is the Windows-based core database engine and
connectivity software behind Borland Delphi, C++Builder, IntraBuilder, Paradox for
Windows, and Visual dBASE for Windows.

History

Borlands Turbo Pascal included a "database" Toolbox, it was the beginning of the
Borland compiler add-ons that facilitated database connectivity. Then came the
Paradox Engine for Windows PXENGWIN which could be compiled into a
program to facilitate connectivity to Paradox tables.

The first DLL-based connectivity engine was ODAPI (Open Database API). It
represented Borlands attempt to centralise connectivity in its suite of applications
which included the brand-new Paradox for Windows 4 and Quattro. With version 4.5 /
5.0 of Paradox for Windows, this database engine was crystallised as IDAPI.

The included set of database drivers enables consistent access to standard data
sources: Paradox, dBASE, FoxPro, Access, and text databases. You can add
Microsoft ODBC drivers as needed to the built-in ODBC socket. Optionally, Borland's
SQL Links product provides access to a range of database management systems
(DBMS), including Informix, DB2, InterBase, Oracle, and Sybase.

BDE is object-oriented in design. At runtime, application developers interact with
BDE by creating various BDE objects. These runtime objects are then used to
manipulate database entities, such as tables and queries. BDE's application program
interface (API) provides direct C and C++ optimized access to the database engine,
as well as BDE's built-in drivers for dBASE, Paradox, FoxPro, Access, and text
databases.

The core database engine files consist of a set of DLLs that are fully re-entrant and
thread-safe. Included with BDE are a set of supplemental tools and examples with
sample code.

BDE system is configured using the BDE Administrator (BDEADMIN.EXE).

Included with BDE is Borland's Local SQL, a subset of ANSI-92 SQL enhanced to
support Paradox and dBASE (standard) naming conventions for tables and fields
(called "columns" in SQL). Local SQL lets you use SQL to query "local" standard
database tables that do not reside on a database server as well as "remote" DBMS
servers. Local SQL is also essential to make multi-table queries across both local
standard tables and those on remote SQL servers.

The older name for the BDE API is the "Integrated Database Application Program
Interface" or "IDAPI".

Source: Wikipedia
Recommended settings for BDE:


Page 46 of 291

Note: Although it is possible to work with industrial data bases like Oracle or SQL
server using BDE it is not recommended. BDE is no longer supported by
Embacadero.



Page 47 of 291

To create a BDE connection, follow these steps:

In the Name Text Box type in a new name for the BDE connection you are about to create
Select appropriate driver
Fill in connection parameters

BDE Administrator


Page 48 of 291

8.3 POP3 Connection

POP3 (Post Office Protocol 3) is the most recent version of a standard protocol for
receiving e-mail. POP3 is a client/server protocol in which e-mail is received and held
for you by your Internet server. Periodically, you (or your client e-mail receiver) check
your mail-box on the server and download any mail, probably using POP3. This
standard protocol is built into most popular e-mail products, such as Eudora and
Outlook Express. It's also built into the Netscape and Microsoft Internet Explorer
browsers.

POP3 is designed to delete mail on the server as soon as the user has downloaded
it. However, some implementations allow users or an administrator to specify that
mail be saved for some period of time. POP can be thought of as a "store-and-
forward" service.

To setup email access using POP as the protocol follow these steps:

In the Name Text Box type in a new name for the POP3 connection you are about to create
Fill in host name
Select TCP/IP port (default is 110)
Type in user name and password
Click OK to close the POP3 connection properties window


Page 49 of 291

8.4 IMAP4 Connection

Internet message access protocol (IMAP) is one of the two most prevalent Internet
standard protocols for e-mail retrieval, the other being the Post Office Protocol (POP).
Virtually all modern e-mail clients and mail servers support both protocols as a means
of transferring e-mail messages from a server.

To setup email access using IMAP as the protocol follow these steps:

In the Name Text Box type in a new name for the IMAP4 connection you are about to
create
Fill in host name
Select folder
Click OK to close the IMAP4 connection properties window


Page 50 of 291

8.5 SMTP Connection

SMTP stands for Simple Mail Transfer Protocol, a protocol for sending e-mail
messages between servers. Most e-mail systems that send mail over the Internet use
SMTP to send messages from one server to another; the messages can then be
retrieved with an e-mail client using either POP or IMAP. In addition, SMTP is
generally used to send messages from a mail client to a mail server.

To create an SMTP connection, follow these steps:

In the Name Text Box type in a new name for the SMTP connection you are about to
create
Fill in host name
Click OK to close the SMTP connection properties window


Page 51 of 291

8.6 IP Socket connection

To setup IP Socket Connection follow these steps:

In the Name Text Box type in a new name for the Google Spread sheet connection you are
about to create
Type in host name
Select relevant port number
Test the connection
Click OK to close the IP Socket connection properties window


Page 52 of 291

8.7 FTP Connection

FTP stands for File Transfer Protocol, the protocol for exchanging files over the
Internet. FTP works in the same way as HTTP for transferring Web pages from a
server to a user's browser and SMTP for transferring electronic mail across the
Internet in that, like these technologies, FTP uses the Internet's TCP/IP protocols to
enable data transfer.

FTP is most commonly used to download a file from a server using the Internet or to
upload a file to a server (e.g., uploading a Web page file to a server).

Visual Importer ETL supports full range of FTP operations.

To create an FTP connection, follow these steps:

In the Name Text Box type in a new name for the FTP Connection you are about to create
Fill in the host name
Type the user name and password
Test connection
Click OK to close the FTP connection properties window



Page 53 of 291

About FTP protocol

FTP is a TCP based service exclusively. There is no UDP component to FTP. FTP
is an unusual service in that it utilizes two ports, a 'data' port and a 'command' port
(also known as the control port). Traditionally these are port 21 for the command port
and port 20 for the data port. The confusion begins however, when we find that
depending on the mode, the data port is not always on port 20.

Active FTP

In active mode FTP the client connects from a random unprivileged port (N > 1023)
to the FTP server's command port, port 21. Then, the client starts listening to port
N+1 and sends the FTP command PORT N+1 to the FTP server. The server will then
connect back to the client's specified data port from its local data port, which is port
20.

From the server-side firewall's standpoint, to support active mode FTP the following
communication channels need to be opened:

FTP server's port 21 from anywhere (Client initiates connection)
FTP server's port 21 to ports > 1023 (Server responds to client's control port)
FTP server's port 20 to ports > 1023 (Server initiates data connection to
client's data port)
FTP server's port 20 from ports > 1023 (Client sends ACKs to server's data
port)

The main problem with active mode FTP actually falls on the client side. The FTP
client doesn't make the actual connection to the data port of the server--it simply tells
the server what port it is listening on and the server connects back to the specified
port on the client. From the client side firewall this appears to be an outside system
initiating a connection to an internal client--something that is usually blocked.

Passive FTP

In order to resolve the issue of the server initiating the connection to the client a
different method for FTP connections was developed. This was known as passive
mode, or PASV, after the command used by the client to tell the server it is in passive
mode.

In passive mode FTP the client initiates both connections to the server, solving the
problem of firewalls filtering the incoming data port connection to the client from the
server. When opening an FTP connection, the client opens two random unprivileged
ports locally (N > 1023 and N+1). The first port contacts the server on port 21, but
instead of then issuing a PORT command and allowing the server to connect back to
its data port, the client will issue the PASV command. The result of this is that the
server then opens a random unprivileged port (P > 1023) and sends the PORT P
command back to the client. The client then initiates the connection from port N+1 to
port P on the server to transfer data.



Page 54 of 291

From the server-side firewall's standpoint, to support passive mode FTP the
following communication channels need to be opened:

FTP server's port 21 from anywhere (Client initiates connection)
FTP server's port 21 to ports > 1023 (Server responds to client's control port)
FTP server's ports > 1023 from anywhere (Client initiates data connection to
random port specified by server)
FTP server's ports > 1023 to remote ports > 1023 (Server sends ACKs (and
data) to client's data port)

While passive mode FTP solves many of the problems from the client side, it opens
up a whole range of problems on the server side. The biggest issue is the need to
allow any remote connection to high numbered ports on the server. Fortunately, many
FTP daemons, including the popular WU-FTPD allow the administrator to specify a
range of ports which the FTP server will use. See Appendix 1 for more information.

A quick summary of the pros and cons of active vs. passive FTP is also in order:

Active FTP is beneficial to the FTP server admin, but detrimental to the client side
admin. The FTP server attempts to make connections to random high ports on the
client, which would almost certainly be blocked by a firewall on the client side.
Passive FTP is beneficial to the client, but detrimental to the FTP server admin. The
client will make both connections to the server, but one of them will be to a random
high port, which would almost certainly be blocked by a firewall on the server side.


Page 55 of 291

8.8 Google Spread sheet connection

To setup Google Spread sheet access follow these steps:

In the Name Text Box type in a new name for the Google Spread sheet connection you are
about to create
Test the connection
Click OK to close the Google Spread sheet connection properties window


Page 56 of 291

8.9 SalesForce connection

To setup SalesForce Connection follow these steps:

In the Name Text Box type in a new name for the SalesForce connection you are about to
create
Type in Customer Key
Type in Customer Secret
Type in Security Token
Type in User name
Type in Password
Test the connection
Click OK to close the SalesForce connection properties window

Note:
1. To get Consumer Key and Consumer Secret login into SalesForce, click develop,
click Remote access authentication. To get security token click my personal
information, reset security token.
2. SalesForce support subset of SQL more information can be found here:
http://www.salesforce.com/us/developer/docs/api/index_Left.htm#CSHID=sforce_api_
calls_soql.htm|StartTopic=Content%2Fsforce_api_calls_soql.htm|SkinName=webhelp


Page 57 of 291

Getting Consumer key and Consumer secret



Page 58 of 291

Getting Security Token


Page 59 of 291

8.10 BrightPearl connection

Brightpearl is a fully integrated, web-based business management software
system. It allows you to run all your business processes through one piece of
software, so you no longer have to suffer the pain of transferring data between
multiple different business systems. All your business's information is visible in real-
time across every department whether it is entered via your website, inventory
control, accounts or CRM.

To setup BrightPearl Connection follow these steps:

In the Name Text Box type in a new name for the BrightPearl connection you are about to
create
Select datacentre
Type in email address
Type in account name
Type in password
Test the connection
Click OK to close the BrightPearl connection properties window

Note:
More information about datacentres:
http://www.brightpearl.com/developer/latest/concept/uri-syntax.html


Page 60 of 291

9. Importing Data

In order to load data from the data source into the data target user must define data mapping
between target table and data source.

Possible data sources are:

Multiple Delimited or Fixed width Text files
Multiple Excel files + Multiple Excel Spread sheets
Multiple MS Access Databases + Multiple Tables
XML Files
Qvx QlikView files
Multiple DBF Files
Any ODBC compliant database
Any JDBC compliant database
Full support for OLE DB
Direct connection to Oracle, SQL Server, MySql, PostgreSQL, SQLite, SQL Server
Compact and Interbase/Firebird
Multiple Tables
Windows Event Log
RSS
Google Spread Sheets
SalesForce
BrightPearl

Import screen is designed to allow user to create, delete, modify, and test data mapping to the
target databases.

At the beginning the simplest way to create Import sctipt is to use import script wizard.

To create a new Import click System menu-> New-> Import.
Fill in Description edit box with the name of an Import you are about to create.
Follow wizard Instructions



Page 61 of 291



Page 62 of 291

9.1 Mapping editor screen overview

Double click on any demo Import.
Import editor will appear.

Main Tool bar
Source tool bar
Mapping Panel Target table fields list
Target Fields
Source fields
Source data
Source file/Table
Field No
Field Width
Starting Position


Page 63 of 291

9.1.1 Main tool bar

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1. Data Target options
2. Loads Import Script From the file
3. Saves Import Script to the file
4. Saves as
5. Saves Import to the Repository
6. Refreshes fields list fro the database
7. Checks Import for mapping errors
8. Data preview
9. Allows user to clear field mapping
10. Hides mapping panel
11. Data Import
12. Manage Versions
13. Add Version
14. Revert to previous version

9.1.2 Source tool bar

1 2 3 4 5 6

7 8

1. Data source options
2. Refreshes Source data
3. Adds new column
4. Deletes last column
5. Auto map the source fields to the target fields
6. Filter
7. Records to Show
8. Sources file name/ table name


Page 64 of 291

9.1.3 Mapping panel

Mapping panel is designed to provide user with all information related to the mapping of one
particular field. There are two ways of mapping: direct and through calculations.

Alternately you may hide Mapping panel and use grid to perform mapping.
See the picture below:

Date format
Calculation
Default value
Mapping type
Source Field
Errors Handling


Page 65 of 291

9.1.4 SQL Statements

Visual Importer ETL provides functionality to execute SQL statements before and after data
import.

Note:

In order to Execute several SQL statements user must specify SQL statements
delimiter.
No select statements allowed.



Page 66 of 291

9.1.5 Template tab



Page 67 of 291

9.1.6 Log File tab



Page 68 of 291

9.1.7 Rejected Records Tab



Page 69 of 291

9.1.8 Import Process
Cal cul ati on?
No Errors?
Resul t Is Nul l or Bl ank
Cal cual te Fi el d Val ue
Appl y Date Format
Rej ect If Nul l ?
Set to defaul t If Nul l ?
Ignore If Nul l ?
Date Format Defi ned?
Set To Defaul t i f Error
Set To Nul l If Error
Rej ect If Error
Rej ect Record
Rej ect If Error
Set To Nul l If Error
Set To Defaul t i f Error
Set to defaul t If Nul l ?
Assi gn Source Fi el d to Target Fi el d
Conti nue to the next Fi el d

Note:
Records can be also rejected by the Server.


Page 70 of 291

9.2 Data Mapping for a Flat File Data Source

Visual Importer ETL can load data from ASCII and Unicode files in UTf8, UTF16BE and
UTF16LE formats with BOM marker and without

Unicode is a computing industry standard allowing computers to consistently represent and
manipulate text expressed in most of the world's writing systems.

To perform data mapping:

Click Data Target Options button.

Dialog box will appear.
Select relevant data target type
Select appropriate connection
Click ODBC tab
Click Get tables list.
Select Table you would like to import data into from the Drop down List.
Click OK.



Page 71 of 291



Page 72 of 291

The following list of fields should appear:

Click Data Source Option Button.

Dialog box will Appear.
Select Text file(s) as Data Source Type.
Click Text tab
Set Delimiter and Quota to appropriate values.



Page 73 of 291



Page 74 of 291

Click OK.


Page 75 of 291

Click and select the file you want to import.

Select First field in the Data Target fields list and drag and drop it above [CUSTOMERID]
field.

You may change field mapping by using mapping panel at any time.



Page 76 of 291

9.3 How to clear mapping

Click .
Click on the field you would like to clear



Page 77 of 291

9.4 Auto mapping

If the Data Source and Data Target have got the same fields names you may use Auto map
feature.

Click , Fill in all necessary data and click map.



Page 78 of 291

Now we are ready to import data.
Lets check script first.
Click to check script.

We have to correct the error first.

Click to load data into the database

Date format is missing


Page 79 of 291

Once loading is finished you may check Log file or Rejected records file.


Page 80 of 291

9.5 Data Mapping for an ODBC Data Source

Data mapping for ODBC is very similar to the flat file mapping.

Click Data Source Option Button.

Dialog box will appear.
Set Data Source Type to ODBC.

Select relevant conenction
Click Update Tables List.
Specify tables you want to load data from or altenatevely type mask to find tables
automatically every time you import data.
Click OK.


Page 81 of 291



Page 82 of 291

Select a table you want to see from the drop down box.



Page 83 of 291

9.6 ODBC Manager



Page 84 of 291

9.7 Loading data from the pivot tables

Let us say we have table like the following in the database:

No Field Data Type
1 CUSTOMERID CHAR
2 YEAR DECIMAL
3 MONTH DECIMAL
4 PRODUCTID DECIMAL
5 AMOUNT DECIMAL

And a text files like the one below:

Year
CustomerID
ProductID
Month_1
Qty_1

Month_12
Qty_12



Page 85 of 291

Click Data Source Button and check Source file is a Cross table check box and set First Field
to 4, Blocks to 12 and Block length to 2

Finally we are ready to import data
Click to load data into the database



Page 86 of 291

9.8 Performing Calculations

Visual Importer ETL is capable of performing calculations during the import.

To perform a simple calculation set mapping type to calculation and type constant or formula
into calculation edit box.

Multiplying fields
StrToFloat([INTEGER_F])* StrToFloat( [FLOAT_F])

Concatenation
[INTEGER_F]+ kilos"

More complicated examples
Iif(StrToFloat([FLOAT_F])> StrToFloat([INTEGER_F]),1,2)
Trim('[CHAR_F]')

You may also use an Expression Editor.



Page 87 of 291

9.9 Filtering Records

User may filter records using the following example:
RECORDTYPE=1 Customer information
RECORDTYPE=2 Invoices
Press and type

If you want to use multiple criteria use following example

([RECORDTYPE]=1) or ([RECORDTYPE]=56)

Note:
Add brackets for complex expressions


Page 88 of 291

9.10 Working with Date fields

In order to load data into date or time fields date format must be provided for source field
Visual Importer ETL automatically converts data into format required for the target database
Full list of date formats can be found in chapter 17

Date Format

Source Date Field



Page 89 of 291

9.11 How to Update/Delete Records

In order to Update/Delete records you must specify update key.

For the example provided below, Visual Importer will execute the following SQL
(Update key is CustomerId,OrderNo)

Add New And Update Old Records

Select count(*)
from [DEMO].[dbo].[orders]
where CustomerId=? And OrderNo=?

If any records found Visual Importer will update them by executing

Update [DEMO].[dbo].[orders]
set orderdate=?,
amount=?
where customerid=? And OrderNo=?



Page 90 of 291

If no records is found Visual Importer will add new records
Update Records

Update [DEMO].[dbo].[orders]
set OrderDate=?,
Amount=?
where CustomerId=? And OrderNo=?

Delete Records

Delete from [DEMO].[dbo].[orders]
Where CustomerId=? And OrderNo=?

Note:
It is also possible to manually edit Insert/Update/Delete/Count SQL. Tick user defined
SQL than edit every SQL statement individually. To regenerate the SQL statements
clear it and open it again
Update Key


Page 91 of 291

9.12 Using Database Specific functions.

Mapping type SQL function allows using Database specific functions like sequences during
import.

Note:
This option may not work for all connection types


Page 92 of 291

9.13 Error Handling

Error handling tab defines format of rejected records file and allows user to abort import on
error.

Note:
When Abort Import is selected import is immediately aborted after an error and SQL
After script is not executed



Page 93 of 291

9.14. Date formats

Date/Time format strings control the conversion of strings into date time type.

Date/Time format strings are composed from specifiers which describe values to be
converted into the date time value.

In the following table, specifiers are given in lower cases. Case is ignored in formats,
except for the "am/pm" and "a/p" specifiers.

Specifier Description

d Day as a number without a leading zero (1-31).
dd Day as a number with a leading zero (01-31).

m Month as a number without a leading zero (1-12).
mm Month as a number with a leading zero (01-12).
mmm Month as an abbreviation (Jan-Dec).
mmmm Month as a full name (January-December).

yy Year as a two-digit number (00-99).
yyyy Year as a four-digit number (0000-9999).

h Hour without a leading zero (0-23).
hh Hour with a leading zero (00-23).

n Minute without a leading zero (0-59).
nn Minute with a leading zero (00-59).

s Second without a leading zero (0-59).
ss Second with a leading zero (00-59).

fff Fraction of Second with a leading zero (000-999).
(Works only for oracle time stamp fileds)

tt Uses the 12-hour clock for the preceding h or hh specifier, 'am' for any hour
before noon, and 'pm' for any hour after noon.

Important thing is to understand that this format has nothing to do with your target database.
This is the format of the source data. It is there to help to covert string into date time type
inside of the software, so it can be loaded later into date or timestamp field

So if source data is:
16/08/2009 than the format is DD/MM/YYYY
1/31/2009 than the format is M/D/YYYY
2006-05-23 22:34:42.096 than the format is YYYY-MM-DD HH:NN:SS.FFF
1992/mar/12 00:00 than the format is YYYY/MMM/DD HH:NN


Page 94 of 291

10. Data sources
10.1 Data source is a XML file

Xml file example:

XML Editor


Page 95 of 291

Note:
Visual Importer ETL will also bring in all data between table and row tag f Ignore tags
are not checked.
For complex XML files it is also possible to apply XSLT transformations

XML Editor



Page 96 of 291

10.2 Data source is Excel file

Data can be extracted from entire Sheet or just from the data range.
Visual Importer ETL assumes that "Sheet name" and "Range name" are delimited
by dot.

Users can still call Sheets as a "Great.Victory"

The logic as follows:

Visual Importer ETL checks if Great.Victory sheet exits than it will read it
If it does not exists it will check if Great sheet exits
If it does exists it will check if Victory range exists if it does It will read data
from just Victory range
If it does not it will read the data from Sheet "Great"

Note:
Please make sure that the range names are not duplicated with in the file


Page 97 of 291

Sheet
Range


Page 98 of 291

10.3 Data source is Qvx file

QVX (QlikView data eXchange) is a new file/stream format for high performance
data input into QlikView. A QVX formatted file contains metadata describing a table of
data and the actual data. In contrast to the QVD format, which is proprietary and
optimized for minimum transformations inside QlikView, the QVX format is public and
requires a few transformations when exporting data from traditional data base
formats.

Note:
More about QVX format can be found on QlikView web site www.qlikview.com


Page 99 of 291

10.4 Data source is Windows Event Log

Note:
If you are having any problems please contact support and we will do our best to help
you


Page 100 of 291

10.5 Data source is MS Access



Page 101 of 291



Page 102 of 291

Note:
It is also possible to use Query builder to design queries
Or user can use table browser
Query Builder Table Browser


Page 103 of 291

Table browser

Allows editing data in tables run scripts or create/drop database objects



Page 104 of 291

SQL Query Builder

Allows designing complex SQL queries



Page 105 of 291

10.6 Data source is a DBF File

Source Directory


Page 106 of 291

10.7 Data source is a ODBC Connection

Note:
Some ODBC drivers do not support bulk fetch.
If table has blob fields use SQL as source and put blob fields last
Select field1,blob1,blob2 from table
Table Browser
ODBC String builder


Page 107 of 291

ODBC Connection Strings

It also possible to use ODBC connection strings for both Reader and Writer
connections.
For example for MS SQL Server connection string is:

Driver={SQL Native Client};
Server=myServerAddress;Database=myDataBase;Uid=myUsername;Pwd=myPassword;

One of the major benefits of using connection strings that it makes it no longer necessary to
create ODBC Dsns manually on every single computer where Visual Importer ETL is
installed. It also gives a greater control over the connection parameters.

Note:
Leave user name and password blank and provide it within connection string

More information about connection strings can be found at:
http://www.connectionstrings.com

The simplest way to create ODBC connection string is to use ODBC Connection builder
dialog. Double click on ODBC driver name to create a connection string



Page 108 of 291

10.8 Data source is a JDBC Connection

JDBC is a Java-based data access technology (Java Standard Edition platform)
from Sun Microsystems, Inc.. It is an acronym as it is unofficially referred to as Java
Database Connectivity, with DB being universally recognized as the abbreviation for
database. This technology is an API for the Java programming language that defines
how a client may access a database. It provides methods for querying and updating
data in a database. JDBC is oriented towards relational databases. A JDBC-to-ODBC
bridge enables connections to any ODBC-accessible data source in the JVM host
environment.

Source: Wikipedia.

Note:
SQL builder does not work with JDBC connections. List of JDBC drivers can be found
here http://developers.sun.com/product/jdbc/drivers


Page 109 of 291

10.9 Data source is an Ole DB Connection

Note:
Ole DB is one of the slowest ways to extract the data and it uses a lot of memory. It is not
recommended for very large datasets.


Page 110 of 291

10.10 Data source is a MS SQL Server

Note:
All properties are very similar to ODBC connection



Page 111 of 291

10.11 Data source is MS SQL Server CE



Page 112 of 291

10.12 Data source is an Oracle Database

Note:
All properties are very similar to ODBC connection



Page 113 of 291

10.13 Data source is an MySql

Note:
All properties are very similar to MS SQL Server connection


Page 114 of 291

10.14 Data source is an PostgreSQL Database



Page 115 of 291

10.15 Data source is an Interbase or Firebird Database

Note:
All properties are very similar to ODBC/MS Access connection


Page 116 of 291

10.16 Data source is SQLIte



Page 117 of 291

10.17 Data source is Google Spread Sheets

Note:
If you delete the sheet the sheet number will stay the same.
If you are having any problems please contact support and we will do our best to help
you



Page 118 of 291

10.18 Data source is RSS feed

RSS (originally RDF Site Summary, often dubbed Really Simple Syndication) is a
family of web feed formats used to publish frequently updated workssuch as blog
entries, news headlines, audio, and videoin a standardized format. An RSS
document (which is called a "feed", "web feed", or "channel") includes full or
summarized text, plus metadata such as publishing dates and authorship.

RSS feeds benefit publishers by letting them syndicate content automatically. A
standardized XML file format allows the information to be published once and viewed
by many different programs. They benefit readers who want to subscribe to timely
updates from favorite websites or to aggregate feeds from many sites into one place.

Source: Wikipedia

Note:
Some web sites use slightly different format, if you are having any problems please
contact our support team and we will do our best to help you


Page 119 of 291

10.19 Data source is SalesForce

Note:
SalesForce support subset of SQL more information can be found here:
http://www.salesforce.com/us/developer/docs/api/index_Left.htm#CSHID=sforce_api_
calls_soql.htm|StartTopic=Content%2Fsforce_api_calls_soql.htm|SkinName=webhelp


Page 120 of 291

10.20 Data source is BrightPearl

Brightpearl is a fully integrated, web-based business management software
system. It allows you to run all your business processes through one piece of
software, so you no longer have to suffer the pain of transferring data between
multiple different business systems. All your business's information is visible in real-
time across every department whether it is entered via your website, inventory
control, accounts or CRM.

Until now the only way to prepare data for Data-warehouse was to export data
manually. But today with help from Visual Importer ETL it is possible to automate
loading data from Brightpearl.



Page 121 of 291

Following data can be extracted

brand
http://www.brightpearl.com/developer/latest/product/brand/search.html

brightpearl-category
http://www.brightpearl.com/developer/latest/product/brightpearl-category/search.html

contact
http://www.brightpearl.com/developer/latest/contact/contact/search.html

order
Search
http://www.brightpearl.com/developer/latest/order/order/search.html
List of fields
http://www.brightpearl.com/developer/latest/order/order/get.html

option
http://www.brightpearl.com/developer/latest/product/option/search.html

option-value
http://www.brightpearl.com/developer/latest/product/option%20value/search.html

product
http://www.brightpearl.com/developer/latest/product/product/search.html

product-type
http://www.brightpearl.com/developer/latest/product/product-type/search.html

goods-in-note
http://www.brightpearl.com/developer/latest/warehouse/goods-in-note/search.html

goods-out-note
http://www.brightpearl.com/developer/latest/warehouse/goods-out-note/search.html

Additional parameters can be passed using search string

As described here

http://www.brightpearl.com/developer/latest/tutorial/working-with-resource-search.html

Columns

Each resource search supports a set of fixed set of columns - data that will be included in your
results like contact names and email addresses. By default, every result will include a value for
each column (although that value may be null) it is possible to change the set and order of the
columns returned in the results.


Page 122 of 291

It is also possible to constrain the search by specifying one or more column names as query
parameters in your resource search URI.

For example:

primaryEmail=brightpearl.com&firstName=Ben

will only include results where the contact's email address contains brightpearl.com and their
first name contains Ben.



Page 123 of 291

10.21 Data source is BDE




Page 124 of 291

11. Data targets
11.1 Target type is ODBC connection

About ODBC

ODBC provides a standard software API method for accessing both relational and non-
relational DBMS. It was developed by the SQL Access Group in 1992 in order to facilitate
easier communication between applications and databases across computing platforms. Prior
to its creation, if an application needed the ability to communicate with more than a single
database, it would have to support and maintain an interface for each. ODBC provides a
universal middleware layer between the application and DBMS, allowing the application
developers to only have to learn a single interface, nor do they have to update their software if
changes are made to the DBMS specification, only the driver needs updating. An ODBC driver
can thus be thought of as analogous to a printer or other driver, providing a standard set of
calls for the application to use, which then translates those commands into the correct
commands at the DBMS end.



Page 125 of 291

When Commit is set to every statement import works as follows:

Execute SQL before statement
Commit
Insert one record
Commit
Insert one record
Commit
Execute SQL after statement
Commit

Note:
Most databases support this way of loading data including files

When Commit is set to once import is completed import is executed inside one big
transaction:

Start transaction
Insert one record
Insert one record
More inserts
Commit transaction

Note:
Not all databases support this way of loading data.



Page 126 of 291

11.2 Target type is Ole DB

Note:
Ole DB is one of the slowest ways to load the data.

About OLE DB

OLE DB (Object Linking and Embedding, Database, sometimes written as OLEDB or
OLE-DB) is an API designed by Microsoft for accessing data from a variety of sources in a
uniform manner. It is a set of interfaces implemented using the Component Object Model
(COM); it is otherwise unrelated to OLE. It was designed as a higher-level replacement for,
and successor to, ODBC, extending its feature set to support a wider variety of non-relational
databases, such as object databases and spreadsheets that do not necessarily implement SQL.

OLE DB separates the data store from the application that needs access to it through a set of
abstractions that include the datasource, session, command, and rowsets. This was done
because different applications need access to different types and sources of data, and do not
necessarily want to know how to access functionality with technology-specific methods. OLE
DB is conceptually divided into consumers and providers. The consumers are the applications


Page 127 of 291

that need access to the data, and the providers are the software components that implement the
interface and thereby provides the data to the consumer.
11.3 Target type is SQLIte



Page 128 of 291

11.4 Target type is MS SQL Server

Check constraints
Ensure that any constraints on the destination table are checked during the bulk copy
operation. By default, constraints are ignored.
Keep identity
Specify that there are values in the data file for an identity column.
Keep NULLS
Specify that any columns containing a null value should be retained as null values, even if a
default value was specified for that column in the destination table.
Batch size
Specify the number of rows in a batch. The default is the entire data file.
The following values for the Batch size property have these effects:


Page 129 of 291

If you set Batch size to zero, the data is loaded in a single batch. The first row that fails will
cause the entire load to be cancelled, and the step fails.
If you set Batch size to one, the data is loaded a row at a time. Each row that fails is counted
as one row failure. Previously loaded rows are committed.
If you set Batch size to a value greater than one, the data is loaded one batch at a time. Any
row that fails in a batch fails that entire batch; loading stops and the step fails. Rows in
previously loaded batches are either committed or, if the step has joined the package
transaction, provisionally retained in the transaction, subject to later commitment or rollback.


Page 130 of 291

11.5 Target type is MS SQL Server CE



Page 131 of 291

11.6 Target type is Oracle

Visual importer ETL supports three methods of loading data into oracle Direct path load,
Conventional path load, Ole DB and ODBC

Direct-path load in Oracle is used when a session is reading buffers from disk directly into
the PGA(opposed to the buffer cache in SGA). During direct-path INSERT operations, the
database appends the inserted data after existing data in the table. Data is written directly into
data files, bypassing the buffer cache. Free space in the existing data is not reused, and
referential integrity constraints are ignored. You may prefer to use a direct path load when you
have a large amount of data to load quickly and you want to load data in parallel for maximum
performance, but there are alternative costs to be aware of.

With the conventional path load method, arrays of rows are inserted with standard SQL
INSERT statements; integrity constraints and insert triggers are automatically applied. But
when you load data with the direct path, Visual Importer ETL disables some integrity
constraints and all database triggers.



Page 132 of 291

Note:
Option Commit every Array works only for Oracle conventional path loading
The constraints that remain in force are:
NOT NULL
UNIQUE
PRIMARY KEY (unique-constraints on not-null columns)
The following constraints are automatically disabled by default:
CHECK constraints
Referential constraints (FOREIGN KEY)

For conventional path loading when Commit every Array is checked import works as
follows:

Commit
Insert Array of records
Commit
Commit
Commit

When Commit every Array is not checked import is executed inside one big transaction:

Start transaction
More inserts
Commit transaction

Note:
Loading Unicode data via direct path is not supported. If your database was not created
with Unicode support but you have some NCHAR or NVARCHAR2 fields you may
need to set array size to 1.


Page 133 of 291

11.7 Target type is MySql



Page 134 of 291

11.8 Target type is PostgreSQL



Page 135 of 291

11.9 Target type is Interbase/Firebird



Page 136 of 291

11.10 Target type is BDE


Note:

Although it is possible to work with industrial data bases like Oracle or SQL server
using BDE, it is not recommended. BDE is no longer supported by Embacadero.



Page 137 of 291

11.11 Target type is a stored procedure

Visual importer ETL supports stored procedures as data targets. Stored procedures can be
very useful for complex calculations, lookups and validations.



Page 138 of 291

12. SQL Scripts

Once data is loaded into the database user may want to perform various tasks like
transformations, validations or calculations. User may perform any SQL command allowed by
the target database. SQL commands must be separated by a separator. SQL Scripts Screen is
designed to provide such functionality for the end user.

To create a new SQL Script select SQL scripts group from objects tree and click New
Dialog box will appear
Fill in Description edit box with the name of SQL Script you are about to create
Fill in comment if required
Click OK to finish creation of SQL Script

To edit script double click on any previously created SQL Scripts.



Page 139 of 291



Page 140 of 291

12.1 SQL Scripts Tool Bar

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

17 18 19 20 21 22 23 24 25

1. SQL Script Properties
2. Load the SQL script from the disk
3. Save the SQL script to the disk
4. Save the SQL Script to the repository
5. Print the SQL script
6. Preview
7. Cut
8. Copy
9. Paste
10. Undo
11. Search
12. Repeat Search
13. Find Previous
14. Replace
15. Execute SQL Script
16. Stop execution when any error happens
17. SQL script separator
18. Connection
19. Show connections only for current project
20. Schedule for execution
21. View Execution Log (When executed within the package or by agent/scheduler)
22. Show/hide log
23. Manage Versions
24. Add Version



Page 141 of 291

12. Reports

This chapter illustrates the process involved in creating Visual Importer ETL
reports.

A report provides the end user with the ability to view data as a neatly formatted
presentation of data. A typical report will be specific to a set of parameters or
business concepts, such as a "Latest Orders" list, or in the case of accounts, an
Overdue Payments" report.

The type of report created will be dependent on the nature of the tasks and
reporting procedures required.

Visual Importer ETL reports can be designed specifically, for individual business
cases or can be generated from the system. The following "Report Designer" section
describes the process in more detail:

12.1 Report Designer

Upon opening Report Designer you will be presented with a screen similar to the
following:



Page 142 of 291

The units of measurement in the report can be determined from the horizontal and
vertical rulers shown in the display. The units of measurement are usually based on
centimetres, but this can be changed if required. This enables the user to design the
exact layout of the data and titles on the report on paper before the report is
designed. Therefore, it is possible to sue X and Y measurements.

Reports have three main aspects to them, as you can see on this illustration. These
are described as follows:

12.1.1 Report Title

As the name suggests, this is the name the report will be identified by, such as
"Overdue Orders Report". This title is global to the whole report and will not change.
12.1.2 Page Header

This header will appear on every page of the report. The headings for each page
should be relevant for the data you intend to print on the report. For instance, the
heading on this report for each page will contain the "Product Name", "Quantity Per
Unit" etc.
12.1.3 Group Header

Each report can be grouped by a specific category e.g. "group all prices by product
type". The groupings usually take place in the SQL driving the report. Grouping can
also take place at a number of levels.

Hence, in this example there are two levels of grouping. The "CategoryName" and
the "CompanyName" reflect the order of the groupings, i.e. group the data by
"Category Name", then by "Company Name". The order of grouping can change,
depending on the SQL or the groupings physically defined in the report.

You will also note the other two tabs in the report. These are described as follows:
12.1.4 Code

This reflects the actual SQL code used in the report. Therefore, if you wish to
change the data in the report, or the way it is grouped, for example, you can amend
the SQL to reflect the changes you wish to implement in the report.
12.1.5 Data

This feature allows you to see a list of all the data used in the compilation of the
report. The information presented here will probably change as a result of any
changes made to the "code" section.


Page 143 of 291

12.1.6 Report tool bar

1 2 3 4 5 6 7

1. Report Properties
2. Saves Report to the Repository
3. Report Connection
4. Show only connections for current project
5. Manage Versions
6. Add Version

The above aspects are all described below:

12.1.7 Report Properties

The Report Properties hold the basic information about the report, such as the
report description and any comments describing the report in more detail.

12.1.8 Save Report to the Repository

All reports are stored internally in the Visual Importer ETL. This facility allows
provides the report writer with an easy way to save the report for later retrieval or
amendment.

12.1.9 Report Connection

In order to retrieve information for the report, it is necessary to connect to a relevant
database. The report connection tab shows which database the user is connected to
at time of generating the report.

12.1.10 Show Only Connections for Current Project

It is possible to have a number of different projects stored on the ETL configuration. If you
wish only to show only connections for the current project you would click this icon.



Page 144 of 291

12.1.11 Manage Versions

Every report can have a number of versions associated with it. For instance, one
version may show all the prices for a list of products, but a different version of the
same report may only show a subset of these prices. Hence, there may be a
requirement for different versions of the same report. The "Manage Versions" utility
allows you to organize different versions of the report according to your preferences.
These may include subsets of your report which can be "branched" e.g. you can have
a version 1.0 or a version 1.1. Version 1.1 may have features which are slightly
different from version 1.0.

12.1.12 Add Versions

This allows user to add a new version of the original report.

12.1.13 Creating Basic Report

In order to demonstrate how the report process works, lets create very simple report
which returns a list of customers.

First, the user should select an appropriate report group from the objects tree and
click new.

This will bring up the Report Properties tab as shown. Fill in description and comment edit
box if necessary:



Page 145 of 291

Provide a report description, and any comments in the Comment box. Once you are
happy with this, click OK. The report Wizard Form will then appear. Once the report
wizard commences, the user can enter the details necessary for production of the
report. The first dialogue box requests the data source. The data source can be
based on an SQL statement or it can be based on data contained within a table.

The example below will create a report based on a query. Therefore, click next and
new query. At this point we have a choice. We can type in SQL directly or we can
use the query builder.



Page 146 of 291

If you are not very confident of your SQL skills, we recommend that the query
builder it used in the process. However, in this example we will generate the report
from an SQL query:



Page 147 of 291

Query Builder

The example illustrated shows that we have chosen the "Customers" table and
chosen from the list of fields within that table for the report.

Once you have the data you wish to include in the report the next step is to choose
the report layout. The process of choosing the fields from the table does not actually
mean they will be displayed in the report.

The objective here is to include appropriate data, perhaps to include in the query.
However, not all fields will be needed in the report. This takes us on to the next step,
i.e. deciding from the sub-set of fields which of these will be printed on the final
report:



Page 148 of 291

The "Fields" dialogue allows you to select the fields you would like to be printed in the
report, from the query you have just created. In the fields dialogue shown, you can see that all
fields in the query have been chosen for the report in this case.
We do not wish to group by anything in this report, therefore leave the "Groups" section
blank as shown:

Here user can define the preferred layout of the report, including the orientation, and
if you prefer a tabular (the standard report format), or if you prefer you can have the
information printed in a columnar format. In this case we have chosen a portrait
layout, a tabular format and so that the data in the report fits neatly on to the page,
we have clicked the "Fit fields to page width" checkbox.

The other information about the report is shown, such as the company information
and the address.

Next, we come the to "Style" dialogue. This allows us to specify how the report will
"look and feel", by the use of colours and fonts. In our example there are four report
styles to choose from but it is possible to create your own styles. For our report we
chose "Corporate".



Page 149 of 291



Page 150 of 291

12.1.14 Report in Report Designer

Finally, we click "Finish" to create the new report. Once we have done this we see
the original Report Designer screen. From here we can choose the "Preview" button
as shown:

We are then presented with the report as it will look when it is printed like so:

Preview Button


Page 151 of 291

Using the preview button, the report can be exported to PDF, Html or into an Excel
file as a CSV. Reports can also be emailed, either by saving them into one of the
formats stated or directly from the preview facility.



Page 152 of 291

13. Email Templates

Email templates is a library of the most often used email texts. These can be used
when automating processes such as when a set of transformations have taken place.
Individual emails can then be set for various departments or management. For
example most of the packages will have notification email to administrator.

For instance, if a package fails it will be necessary to inform about the fact. An email
template allows specific variables and information to be provided relating to the
package failure. In addition email templates can be used to attach the actual
execution log relating to the specific problem occurrence. For instance, a typical
message about this sent in an email might be:

Package have failed please find attached execution log.

One of the advantages of the approach of using email templates is that they save a
lot of time designing and sending emails individually. Another factor is that the email
system reports incidents as they occur, and are not sent on after the event has
finished.

In the example email template below you can see that certain variables have been
defined. These variables are replaced by actual values at run time, or when the
incident occurs.



Page 153 of 291

Email templates can be used to indicate incidents, end of processes,
commencement of processes, the results of processes and a variety of other
business uses. They fit in nicely with the Visual Importer ETLs overall functionality.


Page 154 of 291

14. Packages

This chapter describes Packages - a great way of combining different Actions
together. Most of the actions are self-explanatory therefore only screenshots
provided.

Below is list of actions supported by Visual Importer ETL:

Import Script
SQL Data check
SQL Script
Package
FTP Operation
HTTP Download
ZIP Operation
Receive Email POP3
Receive Email IMAP4
Send Email
External Application
Check File
File operation
Compare Files
Check Disk Space
Backup Repository
Pause
Check Date
Report
Export Data
Show Message
Ask Question
Set Variable
Script
Check Connection
Ping Server
Case
Gpg4win

We are constantly working on improving our software if some of the functionality is
missing please let us know and we will be more than happy to add it.



Page 155 of 291

Creating New Package

To create a new Package select a Package group from objects tree and click New
Dialog box will appear
Fill in Description edit box with the name of the Package you are about to create
Fill in comment if required
Click OK to finish creation of the Package



Page 156 of 291

14.1 Package screen overview

The package screen overview provides the user with the ability to create objects in
the Visual Importer ETL. The objects are numerous and can be combined with any
number of other objects. For instance, you can combine an SQL script to output
information to an email and vice versa, to store information from an email into a
database. There is a great deal of flexibility in the system.

Objects Tree
Drag and Drop Actions from
the left to the right
What to do if error
happened
What to do next
after success
Available Actions
Double click to edit
an Action
Drag and Drop Actions
from the list


Page 157 of 291

Every package must have a starting point (Action on the left on the picture).

14.2 Joining Actions

Objects can be joined to other objects as shown here. This is achieved by simple
drag and drop of the items from one to another. It is a very simple process.

Note:
To remove the link repeat same procedure again.
Drag and Drop


Page 158 of 291

14.3 Package Tool bar

The package tool bar provides all the tools you need in order to create and
manipulate objects. It is also possible to change how objects are viewed in the
interface.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

1. Packages Properties
2. Open Package from file
3. Save Package to file
4. Save to the Repository
5. Print
6. Print Preview
7. Cut
8. Copy
9. Paste
10. Delete selected Action(s)
11. Clear all
12. Undo
13. Redo
14. Search
15. Align Left
16. Arrange Vertically
17. Align Right
18. Align Bottom
19. Arrange Horizontally
20. Align Top
21. Space Horizontally
22. Space Vertically
23. Snap to grid/show grid

24 25 26 27 28 29 30 31 32 33 34 35 36
24. Manage Versions
25. Add Version
27. Show/Hide execution log
28. Open execution log directory
29. Schedule Package for Execution
30. View Execution Log (When executed by agent/scheduler)
31. Debug package
32. Continue execution
1. Execute Package using agent/Integrated Scheduler
2. Zoom In
3. Zoom Out


Page 159 of 291

4. Zoom back to 100%


Page 160 of 291

14.4 Debugging Package

It is possible to run individual actions within the package or debug entire package. It is also
possible to set breakpoints.

Breakpoints are highlighted in orange.

To start debugging click green arrow.

Action in blue is being currently executed.

Read arrow contines execution of package after the breakpoint



Page 161 of 291

Click to show execution log

Execution log provides addional information about the package actions

More detailed information



Page 162 of 291

Another important point is current state of variables



Page 163 of 291

14.5 Using Variables

Variables are used to provide the user with additional information. For example SQL script
has failed and you would like to email the SQL to the developer. Insert <Sql Script>
Into email message and it will be replaced with actual SQL

Note:
There are several ways to create variables
1 Using script action
2 Using transformation action
3 Using calculation object within transformation action
4 Using Set Variable Action



Page 164 of 291

Predefined Variables List:

<username>
<computername>
<datetime>

<Package Name>
<Current Package Item Name>
<Actual Execution Status>
<Execution Status>

<Main Package Log Name>'
<Package Item Log Name>'
<Rejected Records File Name>'

<LAST EMAIL>
<SMTP Log>
<SMTP Email>
<SMTP From>
<SMTP Sender>
<SMTP CC Email>
<SMTP BCC Email>
<SMTP Priority><SMTP Subject>
<SMTP Message>

<Check Sql>
<Check Sql Value>

<Source File>
<Source Directory>
<Source Database>
<Source Server>
<Source UserName>
<Source Password>
<Transformation Processed>
<Transformation Errors>
<Transformation Rejected>
<Transformation Records/Sec>
<Transformation Log>'

<Sql Script>
<Sql Log>

<Application File Name>

<Check File Name>

<File To Compare 1>
<File To Compare 2>


Page 165 of 291

<Zip File>
<File To Zip>
<Directory To Un Zip>

<File Operation Source File>
<File Operation Target Dir>
<File Operation Directory To Create>
<File Operation File To Delete>
<File Operation Target File>
<File Operation File To Change>

<Ftp Remote Dir>
<Ftp Files>
<Ftp File 1>
<Ftp File 2>
<Ftp Target Dir>
<Ftp Local Dir>
<Ftp Log>

<POP3 Email>
<POP3 Subject>
<POP3 Message>
<POP3 Directory>
<POP3 Log>

<Report Export File>
<Backup File Name>

<Export File>

<Disk Space>

<Question>

<Message>

<HTTP Download Url>
<HTTP Download File>

<HTTP Form Url>
<HTTP Form Log>
<HTTP Form Result>

<Script>


Page 166 of 291

14.6 Working with filenames and directories

Consider the following scenarios.

Every day we export data from our database and would like to save it into different file
using current date as a part of the file.
We would like to load yesterdays data from the folder which has date part in the name

We can use functions in filenames to do it as follows

GETSYSTEMVARIABLE('SYSTEM_DATE') will return current date in
YYYYMMDDHHNNSS format

If we want only part of date we can use LeftString(String,Count):String for example

LeftString(GETSYSTEMVARIABLE('SYSTEM_DATE'),8) will return only date part.

Now more complicated example for loading yesterdays data;

LeftString(DecDateS(GETSYSTEMVARIABLE('SYSTEM_DATE'),'YYYYMMDDHH
NNSS', 'DAY',1),8)

Another example:

C:\Data{GetSystemDate('YYYYMMDD')}.csv

Note:
For complete reference of available functions consult chapter 24.
Only one {} pair is allowed.


Page 167 of 291

14.7 Import Action

To create a new Import Action, Drag and Drop it from the Object tree. You will then be
presented with the following properties dialogue for transformations:



Page 168 of 291

14.8 SQL Data Check Action

The SQL Check Action object can be used to check data queries in the database, to
ensure that data integrity and ensure that results are returned as expected.

Example

SQL Data Check uses SQL scripts to check data in the database.
For example user may write following SQL:

Select count(*) from table

Execution will be successful if the value of first field of lookup query is more than 0
Execution will fail if the value of first field of lookup query is equals to 0 or less than 0
or field type is not numeric.


Page 169 of 291

14.9 SQL Script Action

The SQL Script Action object allows the user to define and identify any SQL scripts to be run
(in this case Drop Demo Objects), the connection to be used:



Page 170 of 291

14.10 Package Action

A Package Action object allows the user to specify what if conditions based on
the results of package executions. For instance, in the example below an email
notification will be sent when the package completes its processing.

Note:
Recursion is not allowed



Page 171 of 291

14.11 FTP Action

The Ftp Action performs various operations on the ftp server. For example, using the
Visual Importer ETL you will be able to download data directly from the FTP server.



Page 172 of 291



Page 173 of 291

Multiple files rename:



Page 174 of 291

14.12 HTTP Download Action

Use HTTP download package action to download files from the internet



Page 175 of 291

14.13 Archive Action

The Archive object allows an automatic compression of data in the processor. This
can be used to backup log files, as in the following example, although compression
can be used when sending attachments via email, when saving space on servers,
and so on.

With the compression facility you can compress items and uncompress them as
required. The location of the zip file and the files to compress are specified, as
illustrated below:



Page 176 of 291



Page 177 of 291

14.14 Receive Email POP3 Action

The Receive email object sets up the configuration you need in order to receive
emails. The setting on here will be based on your particular server. A variety of
connections are available including POP, IMAP, HTTP, Exchange and others.

The dialogue below allows the user to specify a directory where attachments will be
downloaded to. This is useful in the automation process as attachments can be
gathered in batch mode and saved to a central area.



Page 178 of 291

Next tab allows the user to define additional filters based on sender and subject. It
is also possible to use masks for example *dbsoftlab.com. or ????.org



Page 179 of 291



Page 180 of 291

14.15 Receive Email IMAP4 Action

The Receive email object sets up the configuration you need in order to receive
emails. The setting on here will be based on your particular server.

The dialogue below allows the user to specify a directory where attachments will be
downloaded to. This is useful in the automation process as attachments can be
gathered in batch mode and saved to a central area.



Page 181 of 291

Next tab allows the user to define additional filters based on sender and subject. It
is also possible to use masks for example *dbsoftlab.com. or ????.org



Page 182 of 291



Page 183 of 291

14.16 Send Email Action

The Send email action object allows you to configure the settings required for
sending email, and is the converse objects to Receive email. The workflow settings
allow the user to specify email templates to use for sending emails. The configuration
can be set to the users preferred settings. In this case, the Simple Network
Management Protocol (SMTP) has been defined by the user as the preferred method
of transmitting messages.



Page 184 of 291

Note:
Type <last email> in TO box to reply to the last sender



Page 185 of 291

14.17 External Application Action

Another useful feature is the ability in the Visual Importer ETL to call and use
external scripts and programs. This enhances the existing functionality already built
into the system, and provides an even greater degree of flexibility.



Page 186 of 291

14.18 Check file action

The Check File Action object is provided to check on the status of files, for things
like their existence, and attributes. The radio box dialogue below allows the user to
specify which option is required to check for the file specified in the Files To Check
text box.



Page 187 of 291

14.19 File Operation Action

In addition to being provided to with the ability to check the properties of files, it is
also possible to decide the actions to take in relation to files and folders. For instance,
the following dialogue is set up to create a directory, in c:\temp as specified in the
filename text box:

Note:

Move file only works locally within one disk, to move files between computers copy than
delete files.



Page 188 of 291

Rename multiple files



Page 189 of 291

14.20 Compare Files Action

This Compare files object allows the user to perform a validation check against two
sets of files by comparing their properties, via a check sum. The comparison can
be done via the creation date as the comparison factor, or it can check to ensure that
the MD5 checksums match. The files are successfully compared when the
comparisons of both files are the same.

The files to be compared are specified in the dialogue boxes as shown:



Page 190 of 291

14.21 Check Disk Space Action



Page 191 of 291

14.22 Backup Repository Action

From time to time, it will be necessary to perform backup procedures on the
repository. This object allows the user to set up a backup scheme. This returns the
results of the backup to the user, in terms of success or fail status.



Page 192 of 291

14.23 Pause Action



Page 193 of 291

14.24 Check date action



Page 194 of 291

14.25 Report Action

Report package action is a convenient way of generation reports in various formats
including PDF, CSV and RTF.



Page 195 of 291



Page 196 of 291

14.26 Export Data Action

Using export object you can export all tables form the database into the text files.



Page 197 of 291



Page 198 of 291

Note:
One of the major benefits of Data Export objects that the user can specify the mask and
export all tables from the database.



Page 199 of 291



Page 200 of 291

14.27 Show Message and Ask Question Action

Show Message and Ask Question do not work when agent is used for execution or in separate
thread. They are the most useful for debugging.



Page 201 of 291



Page 202 of 291

14.28 Set Variable Action

Set Variable package action is a convenient way of changing package object
properties dynamically. Once variable is set, it will be replaced with actual value for
the objects executed after. Package variable name can be any string. It is not
recommended to use very short names for package variables, for example user can a
variable I and set value to Test. This might create problems later because letter I
will be replaced with Test so C:\Program file will become C:\Program ftestle

Example:



Page 203 of 291



Page 204 of 291

14.29 Script Action

Script package action allows writing additional complicated checks and validations.
It supports all function used by Calculation object + variables functions

Example:

Var I : Integer;
begin
I:=1;
SetVariable('I',I);
I:=GetVariable('I');
WriteToLog('Error',I);
Result:=True;
end;

Result:=True; => Succes
Result:=False; => Failure



Page 205 of 291

Expression Builder



Page 206 of 291



Page 207 of 291

14.30 Check Connection Action

Check Connection package action is a convenient way of checking if database
server is up and running



Page 208 of 291

14.31 PING Server Action

PING Server package action is a convenient way of checking if network server is up
and running



Page 209 of 291

14.32 Case Action

Case package action is a convenient way of executing different actions depending in
variable value.



Page 210 of 291

Note:
It is also possible to use calculations in Variable names and Values
{GetSystemDate('YYYYMMDD')}.


Page 211 of 291

14.33 GPG4WIN Action

GPG4WIN package action allows the end users to encrypt / decrypt files using
OpenPGP.

What is Gpg4win?

Gpg4win enables users to securely transport emails and files with the help of encryption
and digital signatures. Encryption protects the contents against an unwanted party reading it.
Digital signatures make sure that it was not modified and comes from a specific sender.

Gpg4win supports both relevant cryptography standards, OpenPGP and S/MIME (X.509),
and is the official GnuPG distribution for Windows. It is maintained by the developers of
GnuPG. Gpg4win and the software included with Gpg4win are Free Software (Open Source;
among other things free of charge for all commercial and non-commercial purposes).

Source:

http://www.gpg4win.org

Note:

Gpg4Win must be installed before using Gpg4Win action.



Page 212 of 291



Page 213 of 291

14.34 HTTP Form Action

HTTP Form package action allows the end users to post data using HTTP. Files can
be posted as well

<form name="input" action="http://localhost:8000/html_form_submit.html"
method="post" enctype="multipart/form-data">
<p>Simple Post Form Example</p>
Name: <input type="text" name="user">
<br>
<input type="checkbox" name="vehicle" value="Bike">I have a bike<br>
<input type="checkbox" name="vehicle" value="Car">I have a car
<br>
<textarea name="area" rows="10" cols="30">
The cat was playing in the garden.
</textarea>
<br>
<input type="submit" value="Submit">
</form>

Note:

The form above is just for information and it is not required for this action. The
action below will submit exactly same information using URL provided



Page 214 of 291



Page 215 of 291

15. Lookups

Lookups allow the users to edit data manually in any database.
.



Page 216 of 291

15.1 Tool bar

The lookup tool bar provides all the tools you need in order to create and
manipulate records in any database

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1. Lookup Properties
2. Refresh Data Grid
3. Create new record
4. Post changes to the database
5. Delete Record
6. Clone Record
7. Find
8. Print
9. Print Preview
10. Export Data
11. Import Data
12. Export Data into Excel
13. Show grouping panel
14. Auto format grid
15. Read only switch



Page 217 of 291

16. Scheduler

Once a Package, Transformation or SQL Script is created the Scheduler allows user
to execute it on a regular basis. It may be executed once, daily, weekly, or monthly.
The user may also specify a day of the week or month when to execute it.
.

16.1 Tool bar

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1. Schedule Properties
2. Refresh Data Grid
3. Create new record
4. Post changes to the database
5. Delete Record
6. Clone Record
7. Print
8. Print Preview
9. Find
10. Export Data
12. Show/Hide Fields
15. Edit selected object
16. Run select object right now



Page 218 of 291

Advanced Schedule Options allows you to define execution boundaries.



Page 219 of 291



Page 220 of 291

The user may specify day of the week when to execute the batch. User must specify
at least one day of the week.



Page 221 of 291

The user may specify month when to execute the batch. User must specify at least
one month.



Page 222 of 291

17. Execution Monitor

Once Package, Transformation or SQL Script is running or completed the Execution Monitor
screen allows checking status or troubleshooting if any error happens.

Package may have four different statuses:
Executing
Submitted
Failed
Finished

The login screen consists of two panels.

The Top panel shows overall status of the execution.

Double click on the top panel to see the log.



Page 223 of 291

The Bottom panel shows status of individual items within the Package.
Double click on Bottom panel to check item log.



Page 224 of 291

17.1 Tool bar

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

1. Shows Log dialog
2. Refreshes the screen
3. Post changes
4. Deletes the record
5. Prints
6. Previews Grid
7. Find Record
8. Export Grid
10. Show/Hide Fields
13. Delete all records from the log (except the ones being executed right now)
14. Stops Execution
15. Show/Hides bottom panel
16. Refresh log every minute
17. Show first 10 records only
18. Open directory containing log files
19. Edit selected object
17.2 Stopping execution
The user can stop execution at any time by pressing .

Note for stopping SQL scripts
SQL script stops once execution of current SQL statement is finished. It could take some
time to do.



Page 225 of 291

18. SQL

The SQL Screen is a tool to run a free hand SQL or SQL scripts against the target databases.

18.1 SQL Toolbar

1 2 3 4 5

1. Show/hide SQL Statement
2. Show/hide connection panel
3. Show/hide log panel
4. Show System Tables
5. Creates new SQL tab



Page 226 of 291

18.2 Side toolbar

1. Previous SQL Statement
2. Next SQL Statement
3. Select SQL Statement to execute

Note
Most of toolbars buttons are self-explanatory or were covered in previous chapters.


Page 227 of 291

19. Version Control

Of the major benefits of using Visual Importer ETL is integrated version control
system. User can have multiple versions of transformations, SQL scripts, Reports,
Email templates and Packages.

Version control toolbar

1 2 3

1. Manage versions
2. Add version
3. Revert to the old version



Page 228 of 291

20. Using Command Line interface

To run a Package, Import or SQL Script from the command line type vpr.exe ID.

ID


Page 229 of 291



Page 230 of 291

Parallel Execution

Another benefit of using vpr.exe that the combination of Application Action and vpr
makes it possible to execute several packages in parallel. See the pictures below.



Page 231 of 291

21. Scripting Language

21.1 The Basic Structure

Below is the basic structure that every Script/Calculation transformation must follow:

Var
VariableName : VariableType;
VariableName : VariableType;
...
// Some single line Comment if necessary

{
Multi line
Comment
}

Procedure ProcedureName;
variables here if necessary
Begin
Some Code;
End;

Function FunctionName(variableList): VariableType;
variables here if necessary
Begin
Some Code if necessary;
Result := some expression
More Code if necessary;
End;

... some more functions and procedures if necessary ...

Begin
the main program block.
Result:= some calculation
End.

Note:

The functions and procedures can appear in any order. The only requirement is that if one
procedure or function uses another one, that latter one must have been defined already.



Page 232 of 291

21.2 Variables
Declaring Variables

var // This starts a section of variables
LineTotal : Integer; // This defines an Integer variable called LineTotal
First,Second : String; // This defines two variables to hold strings of text

Assigning Values to Variables

Variables are simply a name for a block of memory cells in main memory. If a value is
assigned to a variable, that value must be of the same type as the variable, and will be stored in
the memory address designated by the variable name. The assignment statement is the
semicolon-equal :=.

Variables must be declared at the beginning of the program, a procedure, or a function
Variables must be initialized before they can be used.
Variables can be reused as often as necessary. Their old value is simply overwritten by
a new assignment.

Example:

Var
i : Integer: { variable name is i, type is integer)
Begin
i := 10; { valid integer number assigned to variable i }
End.



Page 233 of 291

Variable Types

Numeric Data Types

Type Storage size Range
Byte 1 0 to 255
ShortInt 1 -127 to 127
Word 2 0 to 65,535
SmallInt 2 -32,768 to 32,767
LongWord 4 0 to 4,294,967,295
Cardinal 4* 0 to 4,294,967,295
LongInt 4 -2,147,483,648 to 2,147,483,647
Integer 4* -2,147,483,648 to 2,147,483,647
Int64 8 -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
Single 4 7 significant digits, exponent -38 to +38
Currency 8 50+ significant digits, fixed 4 decimal places
Double 8 15 significant digits, exponent -308 to +308
Extended 10 19 significant digits, exponent -4932 to +4932

Assigning to and from number variables

Number variables can be assigned from other numeric variables, and expressions:

var
Age : Byte; // Smallest positive integer type
Books : SmallInt; // Bigger signed integer
Salary : Currency; // Decimal used to hold financial amounts
Expenses : Currency;
TakeHome : Currency;

begin
Expenses := 12345.67; // Assign from a literal constant
TakeHome := Salary; // Assign from another variable
TakeHome := TakeHome - Expenses; // Assign from an expression
end;

Numerical operators

Number calculations, or expressions, have a number of primitive operators available:

+ Add one number to another
- Subtract one number from another
* Multiply two numbers
/ Divide one decimal number by another
div Divide one integer number by another
mod Remainder from dividing one integer by another



Page 234 of 291

When using these multiple operators in one expression, you should use round brackets to
wrap around sub-expressions to ensure that the result is obtained. This is illustrated in the
examples below:

var
myInt : Integer; // Define integer and decimal variables
myDec : Single;
begin
myInt := 20; // myInt is now 20
myInt := myInt + 10; // myInt is now 30
myInt := myInt - 5; // myInt is now 25
myInt := myInt * 4; // myInt is now 100
myInt := 14 div 3; // myInt is now 4 (14 / 3 = 4 remainder 2)
myInt := 14 mod 3; // myInt is now 2 (14 / 3 = 4 remainder 2)
myInt := 12 * 3 - 4; // myInt is now 32 (* comes before -)
myInt := 12 * (3 - 4); // myInt is now -12 (brackets come before *)
myDec := 2.222 / 2.0; // myDec is now 1.111
end;

Character Types

var
Str1 : Char; // Holds a single character, small alphabet
Str2 : WideChar; // Holds a single character, International alphabet
Str3 : AnsiChar; // Holds a single character, small alphabet
Str4 : ShortString; // Holds a string of up to 255 Char's
Str5 : String; // Holds strings of Char's of any size desired
Str6 : AnsiString; // Holds strings of AnsiChar's any size desired
Str7 : WideString; // Holds strings of WideChar's of any size desired

Some simple text variable usage examples are given below:

+ Concatenates two strings together
= Compares for string equality
< Is one string lower in sequence than another
<= Is one string lower or equal in sequence with another
> Is one string greater in sequence than another
>= Is one string greater or equal in sequence with another
<> Compares for string inequality



Page 235 of 291

Variants

The Variant data type provides a flexible general purpose data type.

It can hold anything but structured data and pointers.

Variants are useful in very specific circumstances, where data types and their content are
determined at run time rather than at compile time.

Example:

var
myVar : Variant;

Date Variables

TDateTime

Description

The TDateTime type holds a date and time value.

It is stored as a Double variable, with the date as the integral part, and time as fractional
part. The date is stored as the number of days since 30 Dec 1899. Quite why it is not 31 Dec is
not clear. 01 Jan 1900 has a days value of 2.

Because TDateTime is actually a double, you can perform calculations on it as if it were a
number. This is useful for calculations such as the difference between two dates.

Note:
No local time information is held with TDateTime - just the day and time values.

Example:
Finding the difference between two dates

var
day1, day2 : TDateTime;
diff : Double;
begin
day1 := StrToDate('12/06/2002');
day2 := StrToDate('12/07/2002');
diff := day2 - day1;
Result:='day2 - day1 = '+FloatToStr(diff)+' days';
end;

day2 - day1 = 30 days



Page 236 of 291

Logical data types

These are used in conjunction with programming logic. They are very simple:

Var
Log1 : Boolean; // Can be 'True' or 'False'

Boolean variables are a form of enumerated type. This means that they can hold one of a
fixed number of values, designated by name. Here, the values can be True or False.



Page 237 of 291

21.3 Logical Operations

Simple if then else

Here is an example of how the if statement works:

var
number : Integer;
text : String;
begin
number := Sqr(17); // Calculate the square of 17
if number > 400
then text := '17 squared > 400' // Action when if condition is true
else text := '17 squared <= 400'; // Action when if condition is false
result:=text;
end;

text is set to : '17 squared <= 400'

There are a number of things to note about the if statement. First that it spans a few lines -
remember that statements can span lines - this is why it insists on a terminating ;
Second, that the then statement does not have a terminating ; -this is because it is part of the if
statement, which is finished at the end of the else clause.

Third, that we have set the value of a text string when the If condition is successful - the
Then clause - and when unsuccessful - the Else clause. We could have just done a then
assignment:

if number > 400
then text := '17 squared > 400';

Note that here, the then condition is not executed (because 17 squared is not > 400), but
there is no else clause. This means that the if statement simply finishes without doing anything.

Note also that the then clause now has a terminating ; to signify the end of the if statement.

Compound if conditions, and multiple statements
We can have multiple conditions for the if condition. And we can have more than one
statement for the then and else clauses. Here are some examples:
If (condition1) And (condition2) // Both conditions must be satisfied
then
begin
statement1;
statement2;
...
end // Notice no terminating ';' - still part of 'if'
else
begin


Page 238 of 291

statement3;
statement4;
...
end;

We used And to join the if conditions together - both must be satisfied for the then clause to
execute. Otherwise, the else clause will execute. We could have used a number of different
logical primitives, of which And is one, covered under logical primitives below.

Nested if statements:

There is nothing to stop you using if statements as the statement of an if statement. Nesting
can be useful, and is often used like this:

if condition1
then statement1
else if condition2
then statement2
else statement3;

However, too many nested if statements can make the code confusing. The Case statement,
discussed below, can be used to overcome a lot of these problems.

Logical primitives

Before we introduce these, it is appropriate to introduce the Boolean data type. It is an
enumerated type that can have one of only two values: True or False. We will use it in place of
a condition in the if clauses below to clarify how they work:

begin
if false And false
then Result:='false and false = true';

if true And false
then Result:= 'true and false = true';

if false And true
then Result:= 'false and true = true';

if true And true
then Result:= 'true and true = true';

if false Or false
then Result:= 'false or false = true';

if true Or false
then Result:= 'true or false = true';


Page 239 of 291

if false Or true
then Result:= 'false or true = true';

if true Or true
then Result:= 'true or true = true';

if false Xor false
then Result:= 'false xor false = true';

if true Xor false
then Result:= 'true xor false = true';

if false Xor true
then Result:= 'false xor true = true';

if true Xor true
then Result:= 'true xor true = true';

if Not false
then Result:= 'not false = true';

if Not true
then Result:= 'not true = true';
end;

true and true = true
false or true = true
true or false = true
true or true = true
false xor true = true
true xor false = true
not false = true

Note that the Xor primitive returns true when one, but not both of the conditions are true.

Case statements

The If statement is useful when you have a simple two way decision. Ether you go one way
or another way. Case statements are used when you have a set of 3 or more alternatives.

A simple numerical case statement:

var
i : Integer;
begin
i := [F1];
Case i of


Page 240 of 291

15 : Resut := ('Number was fifteen');
16 : Resut := ('Number was sixteen');
17 : Resut := ('Number was seventeen');
18 : Resut := ('Number was eighteen');
19 : Resut := ('Number was nineteen');
20 : Resut := ('Number was twenty');
end;
end;

Number was fifteen

The case statement above routes the processing to just one of the statements. OK, the code
is a bit silly, but it is used to illustrate the point.

Using the otherwise clause

Supposing we were not entirely sure what value our case statement was processing? Or we
wanted to cover a known set of values in one fell swoop? The Else clause allows us to do that:

var
i : Integer;
begin
i := [F1];
Case i of
15 : Resut := Number was fifteen';
16 : Resut := 'Number was sixteen';
17 : Resut := 'Number was seventeen';
18 : Resut := 'Number was eighteen';
19 : Resut := 'Number was nineteen';
20 : Resut := 'Nuumber was twenty';
else
Resut := 'Unexpected number;
end;
end;

Unexpected number : 10



Page 241 of 291

21.4 Repeating sets of commands

Why loops are used in programming

One of the main reasons for using computers is to save the tedium of many repetitive tasks.
One of the main uses of loops in programs is to carrry out such repetitive tasks. A loop will
execute one or more lines of code (statements) as many times as you want.

Your choice of loop type depends on how you want to control and terminate the looping.

The For loop

This is the most common loop type. For loops are executed a fixed number of times,
determined by a count. They terminate when the count is exhausted. The count (loop) is held
in a variable that can be used in the loop. The count can proceed upwards or downwards, but
always does so by a value of 1 unit. This count variable can be a number or even an
enumeration.

Counting up

Here is a simple example counting up using numeric values:

var
count : Integer;
begin
For count := 1 to 5 do
Result:= 'Count is now '+IntToStr(count);
end;

Counting down

Here is a simple example counting up using numeric values:

var
count : Integer;
begin
For count := 5 downto 1 do
Result:= 'Count is now '+IntToStr(count);
end;

The For statements in the examples above have all executed one statement. If you want to
execute more than one, you must enclose these in a Begin and End pair.



Page 242 of 291

The Repeat loop

The Repeat loop type is used for loops where we do not know in advance how many times
we will execute. For example, when we keep asking a user for a value until one is provided, or
the user aborts. Here, we are more concerned with the loop termination condition.

Repeat loops always execute at least once. At the end, the Until condition is checked, and
the loop aborts of condition works out as true.

A simple example

var
stop : Boolean; // Our exit condition flag
i : Integer;
begin
i := 1;
exit := False; // do not exit until we are ready
repeat
i := i+1; // Increment a count
if Sqr(i) > 99
then stop:= true; // Exit if the square of our number exceeds 99
until stop; // Shorthand for 'until exit := true'
result:=I;
end;

Upon exit, i will be 10 (since Sqr(10) > 99)

Here we exit the repeat loop when a Boolean variable is true. Notice that we use a
shorthand - just specifying the variable as the condition is sufficient since the variable value is
either true or false.

Using a compound condition

var
i : Integer;
begin
i := 1;
repeat
until (Sqr(i) > 99) or (Sqrt(i) > 2.5);
result:=i;
end;

Upon exit, i will be 7 (since Sqrt(7) > 2.5)

Notice that compound statements require separating brackets. Notice also that Repeat
statements can accommodate multiple statements without the need for a begin/end pair. The
repeat and until clauses form a natural pairing.


Page 243 of 291

While loops

While loops are very similar to Repeat loops except that they have the exit condition at the
start. This means that we use them when we wish to avoid loop execution altogether if the
condition for exit is satisfied at the start.

Var
i : Integer;
begin
i := 1;
while (Sqr(i) <= 99) and (Sqrt(i) <= 2.5) do
result:=i;
end;

Upon exit, i will be 7 (since Sqrt(7) > 2.5)

Notice that our original Repeat Until condition used Or as the compound condition joiner -
we continued until either condition was met. With our While condition, we use And as the
joiner - we continue whilst neither condition is met. Have a closer look to see why we do this.
The difference is that we repeat an action until something or something else happens. Whereas
we keep doing an action while neither something nor something else have happened.



Page 244 of 291

21.5 Functions

Functions provide a flexible method to apply one formula many times to possibly different
values. They are comparable to procedures but

functions are of always of a certain type
functions usually have one or more input variable(s)
the function name must appear at least once inside the definition

The general form of the function statement looks like this:

Function FunctionName(VariableName: VariableType): VariableType;
Begin
some code, if necessary;
Result := some computation;
more code if necessary;
End;



Page 245 of 291

21.6 Script Examples

1. Strings Concatenation

Begin
Result:= [F001] + [F002];
End;

2. Adding one value to another

Begin
Result:= StrToFloat([F001]) + StrToFloat([F002]);
End;

2. If Statement

Begin
If [F002]='' then
Result:= 0
else If [F002]=0 then
Result:= 0
Else
Result:= StrToFloat([F001]) mod StrToFloat([F002]);
End;

3. Variables

Var MyVariable : integer;
Begin
MyVariable:=10;
Result :=StrToFloat([F001]) mod MyVariable;
end;



Page 246 of 291

21.7 Passing variables between objects

There are two calculation functions which can be used to pass data from one object into
another:
1. SetVariable
2. GetVariable

Example:

begin
SetVariable('VariableName',[F0001]);
end;

begin
Result:= GetVariable('VariableName');
end;

Sequence:

Var I : Integer;
s: string;
begin
I:=1;
S:=GetVariable('I');
if s='' then
SetVariable('I',I)
else
begin
I:=GetVariable('I');
I:=I+1;
SetVariable('I',I);
end;
Result:=I;
end;



Page 247 of 291

21.8 Supported Functions List
21.8.1 String Functions

AddCharLeft(Char,S,Count)

AddCharLeft returns a string left-padded to Length with characters Char

AddCharRight(Char,S,Count)

AddCharRight returns a string right-padded to Length with characters Char

RightString(S,Count)

RightString returns the trailing characters of String up to a length of Count characters

Replace(S,OldPattern,NewPattern)

Replace replaces all occurrences of the OldPattern by NewPattern within the String

SubString(S,Index,Count)

SubString returns a substring containing Count Characters or elements starting from Index.

LeftString(S,Count)

LeftString returns the leading characters of String up to a length of Count characters

MakeString(Char,Count)

MakeString returns a string of Count filled with character Char.

DelSpaces(S)

DelSpaces returns string with all spaces deleted except one.
"two spaces"->"two spaces

Delete(S,Index,Count)

Delete removes a substring of Count characters from string S starting with S[Index]. S is a
string-type variable. Index and Count are integer-type expressions. If index is larger than the
length of the string or less than 1, no characters are deleted. If count specifies more characters
than remain starting at the index, Delete removes the rest of the string. If count is less than or
equal to 0, no characters are deleted

DeleteCharacters(String,CharactersToDelete):String

Deletes specific characters from the string


Page 248 of 291

KeepCharacters(String,CharactersToKeep):String

Deletes all characters from the string except Characters To Keep

Insert(Substr,Dest,Index)

Insert merges Source into S at the position S[index]. Source is a string-type expression. S is
a string-type variable of any length. Index is an integer-type expression. It is a character index
and not a byte index. If Index is less than 1, it is mapped to a 1. If it is past the end of the
string, it is set to the length of the string, turning the operation into an append. If the Source
parameter is an empty string, Insert does nothing

ProperCase(S)

ProperCase returns string, with the first letter of each word in uppercase and all other letters
in lowercase "proper case"->"Proper Case"

UpperCase(S)

UpperCase returns a string with the same text as the string passed in S, but with all letters
converted to lowercase. The conversion affects only 7-bit ASCII characters between 'A' and 'Z'.
To convert 8-bit international characters, use AnsiUpperCase.

LowerCase(S)

LowerCase returns a string with the same text as the string passed in S, but with all letters
converted to lowercase. The conversion affects only 7-bit ASCII characters between 'A' and 'Z'.
To convert 8-bit international characters, use AnsiLowerCase.

AnsiUpperCase(S)

AnsiUpperCase returns a string that is a copy of S, converted to upper case.

AnsiLowerCase(S)

AnsiLowerCase returns a string that is a copy of the given string converted to lower case.

AnsiCompareStr(S1,S2)

AnsiCompareStr compares S1 to S2, with case sensitivity.
The return value is:

Conditio
n Return Value
S1 > S2 > 0
S1 < S2 < 0
S1 = S2 = 0


Page 249 of 291

AnsiCompareText(S1,S2)

AnsiCompareText compares S1 to S2, without case sensitivity. AnsiCompareText returns a
value less than 0 if S1 < S2, a value greater than 0 if S1 > S2, and returns 0 if S1 = S2.

AnsiStrLIComp (S1,S2,MaxLen)

AnsiStrLIComp compares S1 to S2, without case sensitivity. If S1 or S2 is longer than
MaxLen characters, AnsiStrLIComp only compares up to the first MaxLen characters.
The return value is:

Condition Return Value
S1 > S2 > 0
S1 < S2 < 0
S1 = S2 (up to MaxLen characters) = 0

AnsiLastChar(S)

Call AnsiLastChar to obtain the last character in a string.

Trim(S)

Trim removes leading and trailing spaces and control characters from the given string S.

TrimLeft(S)

TrimLeft returns a copy of the string S with leading spaces and control characters removed.

TrimRight(S)

TrimRight returns a copy of the string S with trailing spaces and control characters
removed.

QuotedStr(S)

Use QuotedStr to convert the string S to a quoted string. A single quote character (') is
inserted at the beginning and end of S, and each single quote character in the string is repeated.

AnsiQuotedStr(S,Quote)

Use AnsiQuotedStr to convert a string (S) to a quoted string, using the provided Quote
character. A Quote character is inserted at the beginning and end of S, and each Quote
character in the string is doubled.



Page 250 of 291

AnsiExtractQuotedStr(S,Quote)

AnsiExtractQuotedStr removes the quote characters from the beginning and end of a quoted
string, and reduces pairs of quote characters within the string to a single quote character. The
Quote parameter defines what character to use as a quote character. If the first character in S is
not the value of the Quote parameter, AnsiExtractQuotedStr returns an empty string.

Copy(S,Index,Count)

S is an expression of a string or dynamic-array type. Index and Count are integer-type
expressions. Copy returns a substring or sub array containing Count characters or elements
starting at S[Index]. The substring or sub array is a unique copy (that is, it does not share
memory with S, although if the elements of the array are pointers or objects, these are not
copied as well.) If Index is larger than the length of S, Copy returns an empty string or array. If
Count specifies more characters or array elements than are available, only the characters or
elements from S[Index] to the end of S are returned.

CRC16(String,Format)

Returns CRC16 checksum of a String, Format 0=HEX ,1=Base64')





Adler32(String,Format)

Returns Adler32 checksum of a String, Format 0=HEX ,1=Base64')



eDonkey(String,Format)

Returns eDonkey checksum of a String, Format 0=HEX ,1=Base64')

eMule(String,Format)

Returns eMule checksum of a String, Format 0=HEX ,1=Base64')

MD4(String,Format)

Returns MD4 checksum of a String, Format 0=HEX ,1=Base64')



Page 251 of 291

MD5(String,Format)

Returns MD5 checksum of a String, Format 0=HEX ,1=Base64')

RIPEMD160(String,Format)

Returns RIPEMD160 checksum of a String, Format 0=HEX ,1=Base64')
SHA1(String,Format)

Returns SHA1 checksum of a String, Format 0=HEX ,1=Base64')

SHA224(String,Format)








Whirlpool(String,Format)

Returns Whirlpool checksum of a String, Format 0=HEX ,1=Base64')

StrGetLine(String,FieldNo,Delimiter,Quote):String

Returns part of delimited string


Page 252 of 291

21.8.2 Conversion Functions

IntToStr(S)

IntToStr converts an integer into a string containing the decimal representation of that
number.

IntToHex(I,Digits)

IntToHex converts a number into a string containing the number's hexadecimal (base 16)
representation. Value is the number to convert. Digits indicates the minimum number of
hexadecimal digits to return.

StrToInt(S)

StrToInt converts the string S, which represents an integer-type number in either decimal or
hexadecimal notation, into a number. If S does not represent a valid number, StrToInt raises an
exception.

StrToIntDef(S,Default)

StrToIntDef converts the string S, which represents an integer-type number in either
decimal or hexadecimal notation, into a number. If S does not represent a valid number,
StrToIntDef returns Default.

FloatToStr(F)

FloatToStr converts the floating-point value given by Value to its string representation. The
conversion uses general number format with 15 significant digits.

StrToFloat(S)

Use StrToFloat to convert a string, S, to a floating-point value. S must consist of an
optional sign (+ or -), a string of digits with an optional decimal point, and an optional
mantissa. The mantissa consists of 'E' or 'e' followed by an optional sign (+ or -) and a whole
number. Leading and trailing blanks are ignored.

IntegerToString(Integer)

IntegerToString converts integer value to a string value.

NumberToString(Float)

NumberToString converts float value to an string value.

StringToInteger(S)

StringToInteger converts string value to an integer value.


Page 253 of 291

StringToNumber(S)

StringToNumber converts string value to a float value.


Page 254 of 291

21.8.3 File System Functions

FileAge(FileName)

Call FileAge to obtain the OS timestamp of the file specified by FileName. The return value
can be converted to a TDateTime object using the FileDateToDateTime function. The return
value is -1 if the file does not exist.

FileExists(FileName)

FileExists returns true if the file specified by FileName exists. If the file does not exist,
FileExists returns false.

DeleteFile(FileName)

DeleteFile deletes the file named by FileName from the disk. If the file cannot be deleted or
does not exist, the function returns false.

RenameFile(OldFile,NewFile)

RenameFile attempts to change the name of the file specified by OldFile to NewFile. If the
operation succeeds, RenameFile returns true. If RenameFile cannot rename the file (for
example, if the application does not have permission to modify the file), it returns false.

ChangeFileExt(FileName,EXT)

ChangeFileExt takes the file name passed in FileName and changes the extension of the file
name to the extension passed in Extension. Extension specifies the new extension, including
the initial dot character.

ChangeFileExt does not rename the actual file, it just creates a new file name string.

ExtractFilePath(FileName)

The resulting string is the leftmost characters of FileName, up to and including the colon or
backslash that separates the path information from the name and extension. The resulting
string is empty if FileName contains no drive and directory parts.

ExtractFileDir(FileName)

Extracts Directory part from the File Name provided

ExtractFileDrive(FileName)

ExtractFileDrive returns a string containing the drive portion of a fully qualified path name
for the file passed in the FileName. For file names with drive letters, the result is in the form
"drive". For file names with a UNC path the result is in the form "\servername\sharename". If
the given path contains neither style of path prefix, the result is an empty string.


Page 255 of 291

ExtractFileName(FileName)

The resulting string is the rightmost characters of FileName, starting with the first character
after the colon or backslash that separates the path information from the name and extension.
The resulting string is equal to FileName if FileName contains no drive and directory parts.

ExtractFileExt(FileName)

Use ExtractFileExt to obtain the extension from a file name.

ExpandFileName(FileName)

ExpandFileName converts the relative file name into a fully qualified path name.
ExpandFileName does not verify that the resulting fully qualified path name refers to an
existing file, or even that the resulting path exists.

ExpandUNCFileName(FileName)

ExpandUNCFileName returns the fully-qualified file name for a specified file name.

ExtractRelativePath(FileName)

Call ExtractRelativePath to convert a fully qualified path name into a relative path name.
The DestName parameter specifies file name (including path) to be converted. BaseName is
the fully qualified name of the base directory to which the returned path name should be
relative. BaseName may or may not include a file name, but it must include the final path
delimiter.

DiskFree(Drive)

DiskFree returns the number of free bytes on the specified drive, where 0 = Current, 1 = A,
2 = B, and so on.

DiskSize(Drive)

DiskSize returns the size in bytes of the specified drive, where 0 = Current, 1 = A, 2 = B,
etc. DiskSize returns -1 if the drive number is invalid.

GetCurrentDir(Directory)

GetCurrentDir returns the fully qualified name of the current directory.

SetCurrentDir(Directory)

The SetCurrentDir function sets the current directory. The return value is true if the current
directory was successfully changed, or false if an error occurred.



Page 256 of 291

CreateDir(Directory)

CreateDir creates a new directory. The return value is true if a new directory was
successfully created, or false if an error occurred.

RemoveDir(Directory)

Call RemoveDir to remove the directory specified by the Dir parameter. The return value is
true if a new directory was successfully deleted, false if an error occurred. The directory must
be empty before it can be successfully deleted.

FileCRC16(String,Format)

Returns CRC16 checksum of a File, Format 0=HEX ,1=Base64')





FileAdler32(String,Format)

Returns Adler32 checksum of a File, Format 0=HEX ,1=Base64')



FileeDonkey(String,Format)

Returns eDonkey checksum of a File, Format 0=HEX ,1=Base64')

FileeMule(String,Format)

Returns eMule checksum of a File, Format 0=HEX ,1=Base64')

FileMD4(String,Format)

Returns MD4 checksum of a File, Format 0=HEX ,1=Base64')

FileMD5(String,Format)

Returns MD5 checksum of a File, Format 0=HEX ,1=Base64')


Page 257 of 291

FileRIPEMD160(String,Format)

Returns RIPEMD160 checksum of a File, Format 0=HEX ,1=Base64')

FileSHA1(String,Format)

Returns SHA1 checksum of a File, Format 0=HEX ,1=Base64')









FileWhirlpool(String,Format)

Returns Whirlpool checksum of a File, Format 0=HEX ,1=Base64')


Page 258 of 291

21.8.4 Date and Time Functions
Day(Date,Format)

Use Day to get the day part of a date value.

Day('01012003','DDMMYYYY')

Hour(Date,Format)

Use Hour to get the hour part of a date value.

Hour('01012003','DDMMYYYY')

Minute(Date,Format)

Use Minute to get the minute part of a date value.

Minute('01012003','DDMMYYYY')

Month(Date,Format)

Use Month to get the month part of a date value.

Month('01012003','DDMMYYYY')

Second(Date,Format)

Use Second to get the second part of a date value.

Second('01012003','DDMMYYYY')

Year(Date,Format)

Use Year to get the year part of a date value.

Year('01012003','DDMMYYYY')

DayS(Date,Format)

Use DayS to get the day part of a date value as string.

DayS('01012003','DDMMYYYY')

HourS(Date,Format)

Use HourS to get the hour part of a date value as string.

HourS('01012003','DDMMYYYY')


Page 259 of 291

MinuteS(Date,Format)

Use MinuteS to get the minute part of a date value as string.

MinuteS('01012003','DDMMYYYY')

MonthS(Date,Format)

Use MonthS to get the month part of a date value as string.

MonthS('01012003','DDMMYYYY')

SecondS(Date,Format)

Use SecondS to get the second part of a date value as string.

SecondS('01012003','DDMMYYYY')

YearS(Date,Format)

Use YearS to get the year part of a date value as string.

YearS('01012003','DDMMYYYY')

IncDateS (Date,Format,ChangeType,Increment)

ChangeType: YEAR,MONTH,WEEK,DAY,HOUR,MINUTE,SECOND

Use IncDateS to Increase ChangeType part of a date value by an Increment.

IncDateS ('01012003','DDMMYYYY', 'YEAR',1)

DecDateS(Date,Format,ChangeType,Decrement)

ChangeType: YEAR,MONTH,WEEK,DAY,HOUR,MINUTE,SECOND

Use DecDateS to Decrease ChangeType part of a date value by an Decrement.

DecDateS ('01012003','DDMMYYYY', 'YEAR',1)

EncodeDate(Year,Month,Day)

EncodeDate returns a TDateTime value from the values specified as the Year, Month, and
Day parameters. The year must be between 1 and 9999. Valid Month values are 1 through 12.
Valid Day values are 1 through 28, 29, 30, or 31, depending on the Month value. For example,
the possible Day values for month 2 (February) are 1 through 28 or 1 through 29, depending on
whether or not the Year value specifies a leap year.



Page 260 of 291

EncodeTime(Hour,Min,Sec,MSec)

EncodeTime encodes the given hour, minute, second, and millisecond into a TDateTime
value. Valid Hour values are 0 through 23. Valid Min and Sec values are 0 through 59. Valid
MSec values are 0 through 999. If the specified values are not within range, EncodeTime
raises an EConvertError exception. The resulting value is a number between 0 and 1
(inclusive) that indicates the fractional part of a day given by the specified time or (if 1.0)
midnight on the following day. The value 0 corresponds to midnight, 0.5 corresponds to noon,
0.75 corresponds to 6:00 pm, and so on.

DayOfWeek(D)

DayOfWeek returns the day of the week of the specified date as an integer between 1 and 7,
where Sunday is the first day of the week and Saturday is the seventh.

Date

Use Date to obtain the current local date as a TDateTime value. The time portion of the
value is 0 (midnight).

Time

Use Time to return the current time as a TDateTime value. The two functions are
completely equivalent.

Now

Returns the current date and time.

IncMonth(D)

IncMonth returns the value of the Date parameter, incremented by NumberOfMonths
months. NumberOfMonths can be negative, to return a date N months previous. If the input
day of month is greater than the last day of the resulting month, the day is set to the last day of
the resulting month. The time of day specified by the Date parameter is copied to the result.

IsLeapYear(D)

Call IsLeapYear to determine whether the year specified by the Year parameter is a leap
year. Year specifies the calendar year. Use YearOf to obtain the value of Year for IsLeapYear
from a TDateTime value.

DateToStr(D)

Use DateToStr to obtain a string representation of a date value that can be used for display
purposes.



Page 261 of 291

TimeToStr(D)

TimeToStr converts the Time parameter, a TDateTime value, to a string.

DateTimeToStr(D)

Converts a TDateTime value to a string.

StrToDate(S)

Call StrToDate to parse a string that specifies a date. If S does not contain a valid date,
StrToDate raises an exception.

StrToTime(S)

Call StrToTime to parse a string that specifies a time value. If S does not contain a valid
time, StrToTime raises an exception.

StrToDateTime(S)

Call StrToDateTime to parse a string that specifies a date and time value. If S does not
contain a valid date, StrToDateTime raises an exception.

FormatDateTime(Format,DateTime)

FormatDateTime formats the TDateTime value given by DateTime using the format given
by Format. See the table below for information about the supported format strings.

FormatStrToDate(String,Format)

FormatStrToDate(String,Format):Date Converts String into date

DateToEpoch(String,Format)

DateToEpoch(String,Format):Integer Converts String into Epoch (Unix) Date

EpochToDate(Integer)

EpochToDate(Integer):Date Converts Epoch (Unix) Date into Date


Page 262 of 291

21.8.5 Numeric Functions

Round(Float,Integer)

Use Round to round Value to a specified power of ten.
The following examples illustrate the use of Round:

Expression Value
Round(1234567, 3) 1234000
Round(1.234, -2) 1.23
Round(1.235, -2) 1.24
Round(1.245, -2) 1.24

Sign(I)

Use Sign to test the sign of a numeric value.
Sign returns
0 if AValue is zero.
1 if AValue is greater than zero.
-1 if AValue is less than zero.

Abs(X)

Returns an absolute value.

Trunc(X)

Truncates a real number to an integer.

Ceil(X)

Call Ceil to obtain the lowest integer greater than or equal to X. The absolute value of X
must be less than MaxInt. For example: Ceil(-2.8) = -2 Ceil(2.8) = 3 Ceil(-1.0) = -1

Floor(X)

Call Floor to obtain the highest integer less than or equal to X. For example: Floor(-2.8) = -
3 Floor(2.8) = 2 Floor(-1.0) = -1

RandomRange(AFrom,ATo)

RandomRange returns a random integer from the range that extends between AFrom and
ATo (non-inclusive). RandomRange can handle negative ranges (where AFrom is greater than
ATo). To initialize the random number generator, add a single call Randomize or assign a
value to the RandSeed variable before making any calls to RandomRange.



Page 263 of 291

Max(A,B)

Call Max to compare two numeric values. Max returns the greater value of the two.

Min(A,B)

Call Min to compare two numeric values in Delphi. Min returns the smaller value of the
two.



Page 264 of 291

21.8.6 Miscellaneous Functions

Iif(expr1=expr2;expr3;expr4)

Iif function returns expr3 or expr4 depending on expr1=expr2

GetSystemVariable(VariableName)

GetSystemVariable returns value of 'VARIABLENAME'.

Possible values for 'VARIABLENAME' are:
COMPUTERNAME
OSUSERNAME
DBUSERNAME
BLOCKNUMBER
LINENUMBER
RECORDNUMBER
SYSTEM_DATE
SOURCE_FILE_NAME
SOURCE_TABLE_NAME

GetSystemDate(Format)

Returns Current system date/time in format specified
GetSystemDate('MMDDYYYY')

Pos(Substr,S)

Pos searches for Substr within String and returns an integer value that is the index of the
first character of Substr within String. Pos is case-sensitive. If Substr is not found, Pos returns
zero.

AnsiPos(Substr,S)

Call AnsiPos to obtain the byte offset of the Substr parameter, as it appears in the string S.
For example, if Substr is the string "AB", and S is the string "ABCDE", AnsiPos returns 1. If
Substr does not appear in S, AnsiPos returns 0.

Chr(X)

Returns the character for a specified ASCII value.

Length(X)

Returns the number of characters in a string or elements in an array.



Page 265 of 291

Pos(Substr,Str)

Pos searches for Substr within S and returns an integer value that is the index of the first
character of Substr within S. Pos is case-sensitive. If Substr is not found, Pos returns zero

SetLength(S,Length)

Set Length of dynamic array or string

High(X)

Call High to obtain the upper limit of an Array

Low(X)

Call Low to obtain the lowest value or first element of an Array.

SourceChanged

Call SourceChanged to force Data writer to create new file. File is created only it is
different comparing to current one

Usage example:

begin
if GetVariable('<file_name>')<>[F001] then // record changed
begin
SetVariable('<file_name>',[F001]);
SourceChanged; // creating new file
end;
Result:='test';
end;

ExecuteObject(ObjectId)

Call ExecuteObject to execute Import, Sql Script or Package.
Returns 0 on success and 1 on error

begin
result:=executeobject(4196)=0;
end;

ObjectId



Page 266 of 291

21.8.7 Procedures

Abort

Use Abort to escape from an execution path without reporting an error.

Beep

Beep generates a conventional message beep.

Break

The Break procedure forces a jump out of the set of statements within a loop

Continue

The Continue procedure forces a jump past the remaining statements within a loop, back to
the next loop iteration

Exit

The Exit procedure abruptly terminates the current function or procedure.



Page 267 of 291

21.8.8 Math functions

Sqr(X)

the Sqr function returns the square of the argument. X is a floating-point expression. The
result, of the same type as X, is the square of X, or X*X.

Sqrt(X)

The result is the square root of X.

Exp(X)

Exp returns the value of e raised to the power of X, where e is the base of the natural
logarithms

Ln(X)

Ln returns the natural logarithm (Ln(e) = 1) of the real-type expression X.

Sin(X)

Sin returns the sine of the angle X in radians.

Cos(X)

Cos returns the cosine of the angle X. X expression that represents an angle in radians

Tan(X)

Tan returns the tangent of X. Tan(X) = Sin(X) / Cos(X).
ArcTan(X)

ArcTan returns the arctangent of X. X is a real-type expression that gives an angle in
radians

PI

Represents the mathematical value pi, the ratio of a circle's circumference to its diameter. Pi
is approximated as 3.1415926535897932385.

ArcCos(X)

ArcCos returns the inverse cosine of X. X must be between -1 and 1. The return value is in
the range [0..Pi], in radians.



Page 268 of 291

ArcCosh(X)

ArcCosh returns the inverse hyperbolic cosine of X. The value of X must be greater than or
equal to 1.

ArcCot(X)

ArcCot returns the inverse cotangent of X.

ArcCotH(X)

ArcCot returns the inverse hyperbolic cotangent of X.

ArcCsc(X)

ArcCsc returns the inverse cosecant of X.

ArcCscH(X)

ArcCsc returns the inverse hyperbolic cosecant of X.

ArcSec(X)

ArcSec returns the inverse secant of X.

ArcSecH(X)

ArcSec returns the inverse hyperbolic secant of X.

ArcSin(X)

ArcSin returns the inverse sine of X. X must be between -1 and 1. The return value will be
in the range [-Pi/2..Pi/2], in radians.

ArcSinh(X)

ArcSinh returns the inverse hyperbolic sine of X.

ArcTan(X)

ArcTan returns the arctangent of X. X is a real-type expression that gives an angle in
radians.

ArcTanh(X)

ArcTanh returns the inverse hyperbolic tangent of X. The value of X must be between -1
and 1 (inclusive).



Page 269 of 291

Cosecant(X)

Use the Cosecant to calculate the cosecant of X, where X is an angle in radians. The
cosecant is calculated as 1/ Sin(X).

Cosh(X)

Use the Cosh to calculate the hyperbolic cosine of X.

Cot(X)

Call Cot to obtain the cotangent of X. The cotangent is calculated using the formula 1 / Tan
(X).

Cotan(X)

Call Cotan to obtain the cotangent of X. The cotangent is calculated using the formula 1 /
Tan (X)
Do not call Cotan with X = 0

CotH(X)

Call CotH to obtain the hyperbolic cotangent of X, where X is an angle in Radians.

Csc(X)

Use the Csc to calculate the cosecant of X, where X is an angle in radians.

CscH(X)

Use the CscH to calculate the hyperbolic cosecant of X, where X is an angle in radians.

CycleToDeg(X)

CycleToDeg converts angles measured in cycles into degrees, where degrees = cycles * 360.

CycleToGrad(X)

CycleToGrad converts angles measured in cycles into grads.

CycleToRad(X)

CycleToRad converts angles measured in cycles into radians, where radians = 2pi * cycles.

DegToCycle(X)

Use DegToCycle to convert angles expressed in degrees to the corresponding value in
cycles.



Page 270 of 291

DegToGrad(X)

Use DegToGrad to convert angles expressed in degrees to the corresponding value in grads.

DegToRad(X)

Use DegToRad to convert angles expressed in degrees to the corresponding value in
radians, where radians = degrees(pi/180).

GradToCycle(X)

GradToCycle converts angles measured in grads into cycles.

GradToDeg(X)

GradToDeg converts angles measured in grads into degrees.

GradToRad(X)

GradToRad converts angles measured in grads into radians, where radians = grads(pi/200).

Hypot(X,Y)

Hypot returns the length of the hypotenuse of a right triangle. Specify the lengths of the
sides adjacent to the right angle in X and Y. Hypot uses the formula Sqrt(X**2 + Y**2)

IntPower(Base,Exponent)

IntPower raises Base to the power specified by Exponent.

Ldexp(X)

Ldexp returns X times (2 to the power of P).

LnXP1(X)

LnXP1 returns the natural logarithm of (X+1). Use LnXP1 when X is a value near 0.

Log10(X)

Log10 returns the log base 10 of X.

Log2(X)

Log2 returns the log base 2 of X.

LogN(Base,X)

LogN returns the log base Base of X.


Page 271 of 291

Power(Base,Exponent)

Power raises Base to any power. For fractional exponents or exponents greater than MaxInt,
Base must be greater than 0.

RadToCycle(X)

Use RadToCycle to convert angles measured in radians into cycles, where cycles =
radians/(2pi).

RadToDeg(X)

Use RadToDeg to convert angles measured in radians to degrees, where degrees =
radians(180/pi).

RadToGrad(X)

Use RadToGrad to convert angles measured in radians to grads, where grads =
radians(200/pi).

RandG(Mean,StdDev)

RandG produces random numbers with Gaussian distribution about the Mean. This is useful
for simulating data with sampling errors and expected deviations from the Mean.

Sec(X)

Call Sec to obtain the secant of X, where X is an angle in radians. The secant is calculated
using the formula 1 / Cos(X).

SecH(X)

Call SecH to obtain the hyperbolic secant of X, where X is an angle in Radians.

Sinh(X)

Sinh calculates the hyperbolic sine of X.

Tan(X)

Tan returns the tangent of X. Tan(X) = Sin(X) / Cos(X).

Tanh(X)

Tanh calculates the hyperbolic tangent of X.



Page 272 of 291

22. Execution Agent

The Execution Agent is Microsoft Windows service which allows user to schedule
and execute Visual Importer ETL packages.

Note:
Execution Agent is only available with Visual I mporter ETL Enterprise Version.

The Execution Agent can be installed using command line or via Visual I mporter ETL
Enterprise user interface.

To install the Visual Importer ETL Enterprise Execution Agent as a service run following
command
VImpAgent.exe /INSTALL

To uninstall the Visual Importer ETL Enterprise Execution Agent as a Windows
service you must run the Visual Importer ETL Enterprise Execution Agent with the
/UNINSTALL switch as follows

VImpAgent.exe /UNINSTALL

Note:
It is recommended run Execution Agent using same user you use to design
packages/transformations/sql script.



Page 273 of 291

Parallel Execution
To enable parallel execution in the Execution Agent

Click Options
Click Execution
Check Execute actions in separate thread
Check Use Agent to execute Actions
Start Execution Agent



Page 274 of 291

Checking Agent Status



Page 275 of 291

23. Repository database

Visual Importer ETL uses repository database to store all its objects. Default
repository database is MS Access. It is not recommended to use MS Access as
repository in production environment.

Supported repository types are:

MS Access
SQL server
Oracle

PostgreSQL
MySQL
Interbase/Firebird

Transferring data between repositories

Connect to source repository
Backup repository
Connect to target repository
Restore repository


Page 276 of 291

Repository objects synchronization

Connect to source repository
Run Repository Synchronization Wizard
Select target repository
Drug and drop source repository objects on top of target repository objects to
update target object
Drug and drop source repository objects on the target category to add new
object
Connect to target repository and update connections if necessary



Page 277 of 291

Update

Add



Page 278 of 291

Creating new repository

Click create new
Follow Repository creation wizard steps
Connect to newly created repository


Page 279 of 291

ODBC Settings for PostgreSQL repository



Page 280 of 291



Page 281 of 291

ODBC Settings for MySQL repository



Page 282 of 291



Page 283 of 291



Page 284 of 291

Note:
It is recommended to increase max_allowed_packet. The server's default
max_allowed_packet value is 1MB. You can increase this if the server needs to handle big
queries (for example, if you are working with big BLOB columns). For example, to set the
variable to 16MB, start the server like this:
shell> mysqld --max_allowed_packet=16M
You can also use an option file to set max_allowed_packet. For example, to set the size for
the server to 16MB, add the following lines in an option file:
[mysqld]
max_allowed_packet=16M


Page 285 of 291

ODBC Settings for Interbase/Firebird repository



Page 286 of 291

24. Management Console

The Visual Importer ETL Management Console is Microsoft Windows service which
allows user to monitor execution Visual Importer ETL objects.

Note:
Management Console is only available with Visual I mporter ETL EnterpriseVersion.

The Management Console can be installed using command line.

amc.exe /INSTALL

To uninstall the Management Console as a Windows service you must run the Management
Console with the /UNINSTALL switch as follows

amc.exe /UNINSTALL

Once service is installed and running open internet explorer and run
http://localhost:8080/index.html

Note:
It is recommended run Management Console using same user you use to design
packages/transformations/sql script.

Default Username and password are Admin/Admin.

Default port is 8080, to change port edit port.ini file in settings folder


Page 287 of 291

25. Support Procedure

Solving problems today

We respect your time and hope you respect ours.

People ask us same questions again and again,

Please do not be selfish use our support forum, save your time and help the other users.
We are here to help you.

A lot of issues can be resolved in minutes provided that you supply all necessary information:

Version Number
Repository type
Database type you are working with.
Useful screenshots so we can see your entire screen not just error message
Steps to reproduce the problem eg click this click that => got error message
Description of the problem: Want to do this.. Do not know how...
Suggestions or ideas. (We do implement most of them)

If you do not provide this information, we will ask for it anyway and you will waste time.

We intend to resolve 90 percent of the reported problems within 48 hours.
Most difficult issues or functionality extension are resolved within one week

For General Sales and License Queries

Email to: sales@dbsoftlab.com

For Technical Help

Support Forum: http://www.etl-tools.com/etl-forum.html
Email To: support@etl-tools.com

Online Contact Form:

http://www.etl-tools.com/contact-db-software-laboratory.html



Page 288 of 291

26. Dynamic file names

It is possible to use calculated filenames by placing calculation inside {}

Note:
Only one pair of {} is allowed

For example:

1. C:\Data{GetSystemDate('YYYYMMDD')}.vis

2. <CommonDocumentsDir>\DBSL\{GetSystemVariable('SOURCE_FILE_NAME')}.txt

Note:

<CommonDocumentsDir> is a variable and will be replaced with actual value before
execution



Page 289 of 291

27. License Agreement
Visual Importer ETL Professional by
DB Software Laboratory
www.dbsoftlab.com
info@dbsoftlab.com

END-USER LICENSE AGREEMENT FOR THIS SOFTWARE
IMPORTANT - READ CAREFULLY:

This End-User License Agreement ("EULA") is a legal agreement between you (either
an individual or a single entity) and DB Software Laboratory for the SOFTWARE
PRODUCT identified above, which includes computer software and may include
associated media, printed materials, and "online" or electronic documentation. By
installing, copying, or otherwise using the SOFTWARE PRODUCT, you agree to be
bound by the terms of this EULA. If you do not agree to the terms of this EULA, you
may be subject to civil liability if you install and use this SOFTWARE PRODUCT.

SOFTWARE PRODUCT LICENSE

The SOFTWARE PRODUCT is protected by copyright laws and international
copyright treaties, as well as other intellectual property laws and treaties.

The SOFTWARE PRODUCT is licensed, not sold.

INSTALLATION AND USAGE

Once SOFTWARE PRODUCT is installed you may use it for 30 days. After
evaluation period ends, you have to purchase a license or stop using the
SOFTWARE PRODUCT.

If this is an EVALUATION VERSION of the SOFTWARE PRODUCT, you may copy
and distribute an unlimited number of copies of the SOFTWARE PRODUCT;
provided that each copy shall be a true and complete copy, including all copyright and
trademark notices, and shall be accompanied by a copy of this EULA.

If this is a REGISTERED VERSION of the SOFTWARE PRODUCT, you may install
and use it for your personal use only. You may not reproduce or distribute the
SOFTWARE PRODUCT for use by others.



Page 290 of 291

LICENSING

There are two types of licenses available

1. A single computer usage license. The user purchases one license to use the
SOFTWARE PRODUCT on one computer.

2. A SITE usage license. The user purchases a single usage license, authorising the
use of SOFTWARE PRODUCT, by the purchaser, the purchaser's
employees or accredited agents, on an unlimited number of computers at the same
physical site location. This site location would normally be defined as a single
building, but could be considered to be a number of buildings within the same,
general, geographical location, such as an industrial estate or small town.

OTHER RIGHTS AND LIMITATIONS

You may not reverse engineer, decompile, or disassemble the SOFTWARE
PRODUCT, except and only to the extent that such activity is expressly permitted by
applicable law notwithstanding this limitation. Any such unauthorized use shall result
in immediate and automatic termination of this license and may result in criminal
and/or civil prosecution. All rights not expressly granted here are reserved by DB
Software Laboratory.

The SOFTWARE PRODUCT is licensed as a single product. Its component parts
may not be separated for use on more than one computer.

You may permanently transfer all of your rights under this EULA, provided the
recipient agrees to the terms of this EULA.

SEVERABILITY

In the event of invalidity of any provision of this license, the parties agree that such
invalidity shall not affect the validity of the remaining portions of this license.

COPYRIGHT

The SOFTWARE PRODUCT is protected by copyright laws and international treaty
provisions. All title and copyrights related to the SOFTWARE PRODUCT (including
but not limited to any images, photographs, animations, video, audio, music, text, and
"applets" incorporated into the SOFTWARE PRODUCT), the accompanying printed
materials, and any copies of the SOFTWARE PRODUCT are owned by DB Software
Laboratory.

MISCELLANEOUS

Should you have any questions concerning this EULA, or if you desire to contact the
author of this Software for any reason, please contact DB Software Laboratory (see
contact information at the top of this EULA).
LIMITED WARRANTY


Page 291 of 291

DB Software Laboratory expressly disclaims any warranty for the SOFTWARE
PRODUCT. The SOFTWARE PRODUCT and any related documentation is provided
"as is" without warranty of any kind, either express or implied, including, without
limitation, the implied warranties or merchantability, fitness for a particular purpose, or
no infringement. The entire risk arising out of use or performance of the SOFTWARE
PRODUCT remains with you.

In no event shall DB Software Laboratory be liable for any damages whatsoever or
refund any money (including, without limitation, damages for loss of business profits,
business interruption, loss of business information, or any other pecuniary loss)
arising out of the use of or inability to use this product, even if DB Software
Laboratory has been advised of the possibility of such damages. Because some
states/jurisdictions do not allow the exclusion or limitation of liability for consequential
or incidental damages, the above limitation may not apply to you.

SUPPORT AND MAINTENACE

The annual maintenance fee is 20 percent of initial software license cost.

Users with a fully paid annual maintenance fee get the following benefits:

Priority Support
Free software enhancements, updates and upgrades during the maintenance period
Advanced and exclusive notification of software promotions
"Maintenance Owner ONLY" product promotions

ENTIRE AGREEMENT

This is the entire agreement between you and DB Software Laboratory which
supersedes any prior agreement or understanding, whether written or oral, relating to
the subject matter of this license.

Thank you for using the Visual Importer ETL.
DB Software Laboratory

VImp Professional User Manual PDF

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

VImp Professional User Manual PDF

Încărcat de

Drepturi de autor:

Formate disponibile

Visual Importer ETL

Professional & Enterprise

S-ar putea să vă placă și