Documente Academic
Documente Profesional
Documente Cultură
11/11/2015
Starfish Technical Guide Technology Advisors, Inc.
Table of Contents
Installation ............................................................................................................ 4
Overview ............................................................................................................ 4
System Architecture .............................................................................................. 4
Starfish Engine ................................................................................................... 5
Engine Web.Config Settings ................................................................................... 7
Starfish Admin .................................................................................................... 8
Starfish Scheduler ............................................................................................ 10
Scheduler Config Settings .................................................................................... 12
Admin Usage ........................................................................................................ 13
Server-wide Settings ........................................................................................ 13
Email Settings .................................................................................................... 13
License Key ....................................................................................................... 14
Job-specific Settings ......................................................................................... 15
Notification Settings ............................................................................................ 15
Job Chaining ...................................................................................................... 15
Jobs .................................................................................................................. 16
Creating a New Job ............................................................................................. 16
Copying a Job .................................................................................................... 16
Configuring the Origin ...................................................................................... 17
Configuring the Destination .............................................................................. 18
Using Transactions .............................................................................................. 18
Mapping ............................................................................................................ 20
Creating a Stage ................................................................................................ 20
Options ............................................................................................................. 21
Mapping Screen Overview .................................................................................... 22
Straight Field-to-Field Mapping ............................................................................. 24
Function Field Mapping ........................................................................................ 25
SQL Lookup..................................................................................................... 27
Scripted Value ................................................................................................. 28
Exec Before/After Operations ............................................................................... 29
Action Types ................................................................................................... 29
Exec When ...................................................................................................... 30
VBScript Starfish Class ........................................................................................ 31
Cross-reference (Xref) ......................................................................................... 36
Xref List Initialization ....................................................................................... 36
Xref Writes ...................................................................................................... 37
Xref Reads ...................................................................................................... 38
Inserting Variables .............................................................................................. 39
Scheduling Jobs ................................................................................................ 41
Previewing Jobs ................................................................................................ 42
Running Jobs .................................................................................................... 43
Run Options ....................................................................................................... 43
Using Multi-threading .......................................................................................... 44
Saving/Deploying Jobs ..................................................................................... 46
Starfish Updater ................................................................................................... 48
System Backup Wizard ......................................................................................... 50
Connector-specific Guidelines .............................................................................. 56
SData ............................................................................................................... 56
Salesforce .......................................................................................................... 56
Installation
Overview
The Starfish ETL suite is, in a nutshell, an import tool with a lot of flexibility and power.
There are 3 essential components to that make up the ETL suite: The Engine, the Admin
tool, and the Scheduler. The Starfish Engine runs as a web service, and requires IIS to run.
The Scheduler runs as a Windows Service. The Admin tool runs as a Windows GUI
application, which allows the end user to configure jobs, mappings & schedules.
System Architecture
Admin Tool
Scheduler 3rd Party
- Windows Application
- Configure Jobs - Windows Service - SOAP Calls
- Set Schedule - Event Logging - Pass Params
- Set Notifications - Notifications
- Run Jobs in Real Time
Engine
- Web Service
Origin - Multiple Jobs Destination
- Multiple Stages per Job
SQL - XML formatted SQL
OLEDB OLEDB
ODBC Methods ODBC
SOAP - GetConfig () SOAP
XML - DeployConfig (XML) XML
CSV - StartJob (JobID) CSV
- GetJobStatus ()
Starfish Engine
The server where the Engine will be installed must have IIS and ASP.NET installed.
Microsoft .NET Framework 4.0 is required. This may be a server on your network; it does
not have to be a computer where the end users have physical access to it. The only other
requirement is that this server will need direct access to the databases and files that the
jobs will be operating on.
Installation Steps
1. Run the StarfishEngineSetup.exe file in the StarfishEngine folder on the desired
server.
2. No configuration is needed. The installation will automatically place the files in a new
Virtual Directory (StarfishEngine) your IIS Inetpub directory (typically
C:\Inetpub\wwwroot). It will also create a new Application Pool for use with the
Virtual Directory.
3. Click Next and complete the installation. After the install, you can test to make sure
it worked correctly by opening a browser to
http://localhost/StarfishEngine/StarfishService.asmx
(http://<server_name>/<virtual_directory>), and you should get a window that
looks like the one below.
Setting Description
LogPath Path to save log files in when LogToFile is enabled.
LogToFile Boolean, Default False. If enabled, a log file (at the “High”
level) with be created within the specified LogPath location.
CommandTimeout Length in seconds for SQL commands to execute
ProxyEnabled Boolean, Default False. If enabled, Starfish will use the
specified Proxy server when validating it’s license.
ProxyURL URL for proxy server
ProxyPort Port proxy server is running on
ProxyUseDefaultCredentials Boolean, Default False. If enabled, default network
credentials will be used.
ProxyUseSuppliedCredentials Boolean, Default False. If enabled, the credentials specified
below will be used.
ProxyCredentialsUsername Proxy username
ProxyCredentialsPassword Proxy password
ProxyCredentialsDomain Proxy user’s domain
RowThreadCount Default: 1. The default number of threads to use when
executing a Starfish job.
UseParamQueries Boolean, Default False. If enabled (default), Starfish will
build parameterized SQL queries. If disabled, queries will
instead be sent as normal text. Useful for performance
tweaking.
UseNoLock Boolean, Default False. If enabled, Starfish will append the
SQL “WITH(NOLOCK)” command to generated lookup
statements. Useful for performance tweaking.
CountStatement Default “COUNT(*)”. When doing lookups, Starfish will
perform a statement such as “SELECT COUNT(*) FROM
TABLE WHERE …” to determine if a record exists or not.
Changing this value to something like “COUNT(1)” may help
performance for some database systems.
Starfish Admin
You will install the Admin tool on the workstation where you plan to do your job
development. The installation requires the .NET Framework 4.0, and will need the ability to
access the web service set up in the previous step. An easy way to test this is open the web
service in a browser, substituting “localhost” for the actual server name.
Installation Steps
1. Run the Setup.exe file in the StarfishAdmin folder on the desired Workstation.
Starfish Scheduler
The Scheduler is a run as Windows Service. It may installed on any server which has access
to the web service, but for the sake of simplicity it is recommended to be installed the same
server as the Starfish Engine. The Scheduler reads in the jobs’ configured schedules and
launches them at the appropriate times.
Installation Steps
1. Run the Setup.exe file in the StarfishScheduler folder on the desired server.
Setting Description
SchedulePollMS Interval in milliseconds on how often the Service should cycle to
check if it needs to execute any Jobs based on the schedules.
Default is 5 seconds.
Commit True/False, whether Jobs executed through the Scheduler should
be committed to the destination database. Default is True.
ChainJobs True/False, whether Jobs executed through the Scheduler should
automatically call the next Job configured in Job Changing.
Default is True.
LoggingLevel Level of logging to return through notification.
0=None, 1=Low, 2=Medium, 3=High
FromAddress The email address notifications should come from.
FromDisplay The display name of the notification email address.
SMTPPort Port of your SMTP server. Default is port 25.
Admin Usage
For new installations, when you first start Admin you will have to create a New “Bindings
File” from the File->New menu. This will allow you to continue and create new jobs. Only
one Bindings File may be active on the Starfish Engine server at a time, but each file may
contain any number of Jobs and Stages.
Server-wide Settings
Email Settings
When a job is run through the Scheduler, logs can be automatically emailed to an
administrator. Use these fields to set the SMTP email server used transporting these
messages.
License Key
To run a job using Starfish, you must supply your license key. If you do not have a license
key, please contact Technology Advisors to obtain one. To apply your key, from the General
tab, click the “Update License” button.
Job-specific Settings
Notification Settings
When a job is run through the Scheduler, logs can be automatically emailed to an
administrator. Use these fields to set the email address(es) for the logs to be mailed to, and
the level of logging that you want reported back. These settings can be changed for each
job.
Job Chaining
You can configure a Job to automatically call another Job in order when its parent is started.
Any number of jobs can be “daisy chained” together in this manner.
Jobs
Jobs are a series of tasks that can be performed against a singe data source. Each Jobs may
contain different Stages, which allow you to move the data to different destinations. You
may have an unlimited number of Stages per Job, and an unlimited number of Jobs per
Bindings File.
Copying a Job
On the General tab select the Job you want to copy, then click the Copy Job button. Enter a
meaningful name for the new Job. This will copy all database connection settings, mappings,
and schedules. The only setting that is (intentionally) not copied over is the Next Job for Job
Chaining.
ODBC Note: Use the Count Rows option if your driver/database platform does not support
the TOP function.
Using Transactions
If you wish to use commit/rollback transactions, select the Check Box “Use Transactions” on
the Destination tab for OLE DB or ODBC connections. When enabled, all database operations
to the destination will be performed, but not committed until the end of the Job. If an error
occurs while processing the Job, all actions are rolled back.
Mapping
The Mapping tab is where Stage, Exec Operations and Data field mapping is configured.
Creating a Stage
1. On the Mapping tab, click the Stages… button
4. The New Stage should now be displayed in the list on the left.
Options
Option Name Description
Stage Type - Update Will perform an update-only operation on matched records for each
Origin row. Unmatched records will be skipped.
Stage Type – Insert Will perform an insert-only operation for each Origin row. Use this
operation when performing a “dump” of the data.
Stage Type – Delete Will perform a delete operation on the Destination database for each
Origin row, based on the match.
Stage Type – Will first perform a lookup, based on matches. If a match is found,
Update/Insert and update operation will be performed. If no match is found, an
insert operation will be performed.
Skip Blank Writes If checked, the insert/operation will ignore NULL/blank fields while
building the SQL statement – resulting in NULL values on the
destination.
Only Update on For update operations, this setting will first check the matched
Difference record for values that are different and will only actually update the
record if a mismatch of data values occurs and it’s actually needed.
2. OR click the ellipsis inside the Operation column (or double-click this field).
Select from the following types of Function Fields, more information is provided below.
Option Name Description
Hardcoded Value Allows the user to specify a static value, although you can also insert
a Variable here as well. It is not necessary to include quotes for
text/string values.
SQL Lookup Performs a database lookup on the Destination database. Supply a
SQL SELECT statement, will return the first column from the first
row. More on Server-Side Lookups
Smart Lookup Performs a lookup on the destination, returning a single field’s value.
Allows caching of results so the multiple runs of the same lookup
don’t have to make round-trips to the destination. This cache is
stored in memory, so it’s a good idea to only use it when there is a
SQL Lookup
Server-Side Lookup Function Fields can be used to obtain data from other tables within the
Destination database. Below is an example of a lookup returning an AccountID based on an
Account-name lookup. The Account name is provided by the Origin field “Account”. Note
how variables can be used within the SQL statement itself. The Variable code will be
replaced with the actual value at run-time.
Server-Side lookups are also useful for generating ID’s. The example below calls a
SalesLogix stored procedure to generate an ID for the Account table.
Scripted Value
Scripted Value fields allow for extreme flexibility when assigned values to a Destination
field. Basically, any function that can be performed with VB Script can be performed within a
Scripted Value field. Variables can also be referenced from within the script.
Below are a few examples of how Scripted Value fields can be used.
Samples
Return the Current Date/Time Return the left-most 5 characters from
“ZIPCODE” Origin field
Function ScriptedField Function ScriptedField
Dim var
ScriptedField = Now() var = "@@ORG:ZIPCODE@@"
End Function ScriptedField = Left(var,5)
End Function
Press the button to test the results of your script. Keep in mind; this will not
replace variable values.
Enter a meaningful name, select the Action Type, assign a variable name where necessary,
set the Exec When option, and configure the Action.
Action Types
The Action Type defines what type of operation will be performed. Variable-type actions
create a User-defined variable for use through the Job.
Exec When
An Exec operation can be performed at different times throughout a job run. The table
below defines what each of these settings mean.
Class Properties
Property Name Description
CurrentStageName Read-only. Returns the name of the current stage. Useful during
“Repeat Each Stage” Exec Operations to determine path of
execution/next steps.
TotalRows Returns the total number of Origin Rows in the job.
CurrentRow Returns the current row number.
ReturnValue Sets the return value to the calling application. Useful for passing an
output back when the Job was called with an argument.
Origin ( String ) Returns the current Origin row’s value for the field specified.
Class Methods
Method Name Description
GotoStage ( String ) Directs Starfish Engine to go directly to a Stage
(skipping any others before it). String Parameter is
the Name of the stage to go to. Once GotoStage has
been called within a Job, it is considered in a “manual
operation mode”. No more stages will be called in
order and it will be up to the user to implement a
After Each Stage Exec Operation to control stage flow.
GotoNextRow () Skips all Stages and moves to the next Origin row.
EndJob ( String ) Cancels the Job execution immediately, ending with
an error message as passed into the String param.
LogMessage ( String, MsgBoxStyle ) Appends a custom message to the Log which will
always be displayed (even if Logging Level is set to
None). Message is the passed in String parameter.
Second parameter is a valid MsgBoxStyle
(vbInformation, vbQuestion, vbExclamation,
vbCritical)
GetFields ( String ) Executes a SQL SELECT statement returns all field
values from the first row, comma-delimited. VBScript
Split() function may be used to convert to an array
within your function. Can improve performance when
needing multiple fields from a single table.
ExecScalar ( String ) Executes a SQL SELECT statement on the Destination
database. Returns the first column of the first row
returned by the query. Compatible with OLE DB
destinations.
ExecScalarCache ( String ) Same as ExecScalar, except that results are cached
and before a query is executed, a search is performed
on a locally-stored set of results to prevent execution
of identical queries, thus boosting performance.
Should only be used for queries where multiple
identical queries may be run (such as looking up user
id’s or picklist values) – otherwise could consume
resources needlessly.
ExecSQL Executes a non-query statement, such as UPDATE or
DELETE against the destination database. Returns
number of rows affected. Compatible with OLE DB
destinations.
SendEmail ( String, String, String ) Sends an email directly from within the VBScript
function/procedure.
Parameters:
To_Addr – Email Recipient Address
Subject – Subject of the Email
Body – Body of the Email
ParseName ( String, String ) Parses as string containing a person’s name, and
returns the part of the name requested.
Parameters:
String – Input String containing entire name
Part – Part of the name to return. Valid parts are
“Title”,”First”,”Last”,”Middle”,”Pedigree”,”Degree”.
GetStageValue ( Integer, String) Returns the destination value used on a previous
stage.
Integer – Index number of the stage, 0-based.
String – Field name of the value from this stage to
return.
SugarCRM-Specific
SugarSetRelationship (String, Sets a relationship between two modules.
String, String, String ) String – Module 1 Name
String – Module 1 ID
String – Module 2 Name
String – Module 2 ID
SugarSetRelationshipv4 (String, Sets a relationship between two entities, compatible
String, String, String ) also with custom modules.
String – Module 1 Name
String – Module 1 ID
String – Relationship Name
String – Module 2 ID
SugarGetRelationships ( String, Gets comma-separated a list of ID’s related to an
String, String ) entity
String – Module 1 Name
String – Module 1 ID
String – Module 2 Name
Email (IMAP)-Specific
MoveIMAPMessage ( String ) Moves the current email message to the specified
folder on the server.
String – Name of IMAP folder to move message to
Samples
Jump to a Specific Origin Row
(first create a Global Include and dim a variable)
Dim Counter
Sub VBScriptProcedure
Dim sql
sql = "SELECT Count(*) FROM Table WHERE UniqueID=@@ORG:UNIQUEID@@"
If Starfish.ExecScalar(sql) > 0 Then
Starfish.ExecSQL("DELETE FROM Table WHERE UniqueID=@@ORG:UNIQUEID@@")
Else
Starfish.GotoNextRow
End If
End Sub
Send an email notification
Sub VBScriptProcedure
Starfish.SendEmail("person@domain.com","Email Subject","Email Body")
End Sub
Perform a SmartQuery Lookup, loop through results
Dim mdarray
mdarray = SmartQuery("Table", "Column='sample'", "Column1,Column2,Column3")
for i = 0 to UBound(mdarray) 'Loop through all rows
for j = 0 to UBound(mdarray, 2) 'Loop through all columns in the row
LogMessage "(" & i & "," & j & "): " & mdarray(i, j)
next
next
Cross-reference (Xref)
Starfish ETL has built-in methods for maintaining relationships between different systems.
Put simply, you will be storing an ID from your Origin, and the new related ID in your
destination for the record.
For an example, if you are migrating Company records from one system to another, and
later in the process you’ll also be importing Contacts that will be related to these companies
you’ll need a method to select which company the contact belongs to. One method would
simply be to store the Old (Origin) primary company ID somewhere inside of the new
company table on your destination. Then you could use a SQL/SmartLookup and pull down
the ID by a filter clause. This is perfectly acceptable; however you’ll take a performance hit
because for each row you’ll have to perform another lookup operation. Using Xref functions,
these ID relationships are stored in memory, thus making the lookup operation
instantaneous – speeding up your process. The idea will be to write out the ID relationships
for accounts while that job is running. Then, when the Contacts job runs it can initialize and
read from this list to quickly & easily relate to the correct Company.
Internally, Xref Lists are stored in your StarfishEngine\Xref folder as simple text files with
an .ids file extension. You may modify these files by hand if necessary.
Write to a new list to store a relationship between an Origin Primary Key and the
Destination Primary Key. The checkbox indicated whether the list should be appended to.
When it is written to, by default, the entire list will be blanked and written to as new when
the job is run. Check to box to instead append to the end of the existing list. (for instance if
you need to run multiple jobs which write to the same list, or need to run the job multiple
times for some reason).
Reading from an existing list will allow you to retrieve the “New ID” for a system, given an
“Old ID”. The drop-down on the Read side gets a list of Xref Lists that are available on the
server.
Rather than using the built-in screens for Xref List Initializations, you may also use the
XrefInitRead() and XrefInitWrite() VBScript functions.
Xref Writes
To write to an initialized Xref list, you may use the Exec Operation “Xref Write”. Select the
list from the Xref List drop-down. Then supply a value or variable to use as the Origin/Old
ID, and one for the Destination/New ID.
Alternatively, you can use the XrefWrite() VBScript function to write to an Xref list.
Xref Reads
To read from an initialized Xref list, you may use the “Xref Read” function field in your
mapping.
Alternatively, you can use the XrefRead() VBScript function to read from an Xref List.
Inserting Variables
Variables that have been created during an Exec Before/After Operation can be inserted as
the end-result value in any kind of Function Field. Place your cursor inside the text box, and
click the “Insert Variable” link.
From this list, you can choose a User-Defined variable, any Origin field, any Destination field
(that comes before it in the sort order, and that has been assigned a value), or a Special
Character. At run-time, the Starfish Engine will “replace” the variable code with the actual
value for the current Origin row.
Scheduling Jobs
Each Job can be set up to have its own Schedule. When a schedule has been configured and
enabled for a Job, it will be automatically executed by the Starfish Scheduler Windows
Service.
There are four different types of schedules that can be configured. (As of version 1.0, only
Time Interval and Daily are supported)
Be sure the “Starfish Scheduler” service is running on the desired server. Check the Event
Viewer for error and informational logs from the Scheduler service. To make sure everything
is being read correctly by the service, after the service starts, check the Event Viewer. An
Informational event should be created for each Job, showing the next time the Job is
scheduled to run.
Previewing Jobs
Before running a job for the first time, you can check to make sure all your formulas and
translations are working correctly by stepping through each row on the Preview tab. No data
will be modified in your destination system, and you may click Log to review the high-level
output logging from the Starfish Engine.
Running Jobs
If you wish to run a Job directly from the Starfish Admin, follow the steps below.
1. Deploy the current configuration to the Web Service.
2. On the General tab, select the Job that you want to run from the drop-down.
3. Click the Run Job tab.
4. Configure the options for the run.
5. Click the Start button. The status box should show “Running” and will display in real-
time, the # of Origin Rows processed.
The Message box will display any error messages encountered during the run. The Log box
will display the entire log for the Job run (when it’s finished), depending on the Logging
Level specified at the beginning of the run.
Run Options
Option Name Description
Commit Valid for OLE DB and ODBC destination-type jobs, where “Use
Transaction” has been enabled. If checked, all transactions
against the destination database will be committed at the end of
the Job. If unchecked, no actual changes will be made to the
database (all transactions will be Rolled Back). This is very
Using Multi-threading
A new feature in Starfish 2.1 is the ability to use multithreading. This ability has been built-
in natively to the Starfish Engine, so there is no need set up any additional message queue
systems. The default number of threads is set to 1 in the Engine Web.config file. To override
this, set a value in the Thread Count box in the Starfish Admin Run tab, or modify the value
in the Config file.
How many threads to use depends on a number of factors and finding the “sweet spot” may
involve some trial and error. Multi-threading is most beneficial in scenarios where there may
be network latency, such as integrations between backend systems and cloud-based or
hosted solutions. In cases where data is being moved between two systems on the same
database server or even within the network, using more than one thread may not produce
better results (and may degrade performance). However, in ground-to-cloud migrations
very significant performance gains will likely be seen.
For these cases, we recommend starting with a value of 8 threads. This number can be
increased if performance allows it. Starfish does not block an upper-limit of the number of
threads that can be used. You’re only limited by the memory and speed of your server and,
realistically, the limitations of your network bandwidth and the server you’re communicating
with. For this reason, we recommend not exceeding 24 threads.
NOTE: Multi-threading hands off the next available data row to the next available thread as
it because available. This means, rows being processed can be in any state at any given
time. Rows will not complete in the sequential order they are fed into the queue. So, for
example if you’re using 8 threads, Row #6 may finish before Row #1, then Row #3 may be
next, and so on. The results will basically be random. For this reason, you cannot use
multiple threads if you require data to be handled in a sequential order. For instance, if your
job performs an Insert/Update operation and relies on this lookup to work properly based on
data from previous rows.
Saving/Deploying Jobs
The Project Document, which is an encrypted XML file, contains the definition for the Jobs.
This file can be saved to the local Admin workstation as a “.SPD” file.
However, before a Job can be executed, it must be “deployed” to the Starfish Engine. This is
done through the Set Web Service URL menu option.
Change the Web Service URL to the server/address location of your Starfish Engine server.
After you change it and deploy the first time, Admin will remember the location. You should
receive the message below if the file deployed correctly.
Starfish Updater
Starfish ETL has a built-in updater. To start, click the Starfish button and select “Check for
Updates” or launch StarfishUpdater.exe directly from within the Starfish Admin installation
path.
The first time the Updater is run, you’ll have to supply the paths to the Starfish ETL
components. If your Starfish Engine is installed on another server, UNC paths are
supported. If you don’t have a certain component installed (such as the Scheduler), you
may leave the path empty.
Now you may click the “Look for Updates” button. If the updater finds out of date or missing
files, the “Update Now” button will become available. (Files are compared by MD5 hash)
Before proceeding, it is recommended that you close out Starfish Admin and shut down the
Starfish Scheduler service if running. Now you may press the “Update Now” button. This will
download and replace the necessary files from Starfish ETL servers.
You will first have to verify your Origin and Destination connections for compatibility. The
connections for the currently selected Job will be used. The wizard works with any Origin
connection type. The Destination must be of a type which support SQL statements (typically
any current RDBMS).
It may be necessary for you to manually alter the generated table definitions and maps to
fit your needs.
1. Create a new job. Set up the Origin & Destination. Your origin query/table selected
does not matter in the wizard, but will be necessary to populate to get a valid
connection.
2. Start the wizard by clicking the “System Backup Wizard” button on the General tab.
Option Description
Generate matching Uncheck this option to skip this step during the wizard (if you
schema already have created the necessary tables)
Table Prefix Will prefix each table name with the value supplied (ie: prefix “bak_”
for table “account” will become “bak_account”.
Default Type If a data type is not given by the connector or cannot be resolved
through the data_type.map translation file, this value will be used.
Default Length If a type is specified, but not a length this value will be used.
Maximum Length If a value is supplied, and the length specified by the Origin is
greater than this number, the length will be changed to this value.
Change if Maximum If the length specified by the Origin is greater than the Maximum
Length is exceeded Length, the field data type is changed to this.
Drop Existing Tables Before a table is created by the Wizard a Drop Table command will
be issued.
Option Description
Generate Jobs Uncheck this option to skip this step during the wizard.
Job Name This value will be used when creating each job’s name. The
“%tablename” variable will be replaced with the actual table name
when the wizard runs.
Stage Type One stage will be created for each job, with each field mapped
directly across. This specifies that Stage Type: Insert, Update,
Delete, Update/Insert.
Guess Job’s Primary Valid only for Update, Delete, Update/Insert Stage Types. Attempts
Key for Match. to place the Match checkbox on the table’s Primary Key.
Here is the logic used for each connector:
does not exist then Unique Indexes are used. If there are no Unique
Indexes, then regular indexed fields are used.
SugarCRM: The "id" field will be used for each table.
Chain all Jobs All jobs will be chained together to run after another.
together
Option Description
Only pull data Check this option to process this step during the wizard.
changed from last run
Last Run Table Name of the table inside your Destination where the Last Run Dates
for each job will be stored.
Default Filter For SQL-based connectors, will be appended as a WHERE clause to
Origin query; otherwise will be populated as the Origin criteria box.
Default Format Date format to use in in filter/criteria.
Initial Date Date to initially populate in the Last Run table for each job. The first
time the job is run, it will pull data starting from this date based on
the filter clause.
Connector-specific Guidelines
SData
1. Filters (where clauses) must use SData format. Such as ‘eq’ instead of ‘=’. Details for
the query language are provided at:
http://interop.sage.com/daisy/sdata/AnatomyOfAnSDataURL/QueryLanguage.html
2. To insert data into related tables you must select it as a Resource when setting up
your stage [All Fields].
3. To supply an ID to related entity you must select the table as a Resource when
setting up your stage [Key Only].
Salesforce
1. Passwords must contain the security token generated by Salesforce.
2. Destination: Updates and Deletes MUST match on the ID field only.
3. Filters containing quotes or single quotes must be escaped. IE: “John’s Pizza” must
be escaped to “John\’s Pizza”. If referencing an origin field such as
“@@ORG:AccountName@@”, change ORG to SFO to allow Starfish to automatically
escape these characters, example “@@SFO:AccountName@@”. This is not necessary
for standard field mappings, but only when part of a filter clause.
Microsoft CRM
1. Filters (where clauses) should use fully qualified field name (tablename dot field
name). ie: account.name, contact.firstname, etc
2. Filter statements may use the following operands: =, <>, >, <, >=, <=.
3. Strings in filter must be surrounded by single or double quotes. ie: name = ‘John
Smith’
4. Filter statements may use multiple logical operands (AND/OR), but of only one type
per statement. You may not use both an AND and OR in the same statement but
may use multiple ANDs or ORs in a statement.
5. Parenthesized filters are not possible due to limitations in the CRM API.
6. It is not necessary to generate an ID first for an insert, the API generates one for
you. If you need to get the newly generated ID within the same job on a future
stage, use the GetStageValue() function. If you need to get the ID on a different job,
then use a Smart Lookup.
7. For Lookup-type fields, provide data in following format:
MSCRMLOOKUP,table_name,guid
8. For Owner (Team/User) lookup type fields provide data in following format:
USER: MSCRMOWNER,USER,user_guid TEAM: MSCRMOWNER,TEAM,team_guid
9. To force a NULL/blank in an update operation into a particular field, write out
“MSCRMNULL”.
10. You may also use MSCRMNULL in filter statements (ie: name <> MSCRMNULL).
11. You may set the createdby/modifiedby field to a valid systemuserid guid.
SugarCRM
1. For many types of relationships, you cannot just supply the ID – you’ll need to make
use of the SugarGetRelationship and SugarSetRelationship functions.
NetSuite
2. Related fields (such as Addresses) are loaded for the Destination only and allow 2 of
each type.
3. For RecordRef type fields, supply the Internal ID of the entity being linked to.
Email
1. Protocol uses IMAP, not POP.
2. Default server port is 143, Secure IMAP is 585.
3. When connection to a MS Exchange Server IMAP account, use port 993 with SSL
checked.
SageCRM
1. In SageCRM, the following options should be configured as specified below.
(Administration > System > Web Services)
a. Enable Web Services: Yes
b. Make WSDL Available to all: Yes
c. Dropdown fields as strings in WSDL: Yes
d. Force web service log on: Yes
2. Ensure the user you are using has “Allow Web Service Access” enabled in their user
profile.
3. Sub-entities are only enabled for Company and Person parent entities. This is a
limitation of the SageCRM web service API.
Oracle
1. In order for the Oracle connector to work properly, you must first install the 32-bit
version of the Oracle Data Provider for .NET (ODP.NET). Download the latest version
from here:
http://www.oracle.com/technetwork/database/windows/downloads/index-
101290.html
Troubleshooting
Problem: Starfish Engine installation fails with message “Installation was
interrupted”.
Solution: On some versions of Windows, in addition to IIS needing to be installed – “IIS 6
Management Compatibility” will also need to be installed. This is added through the “Turn
Windows features on or off” function in Control Panel and is located in the “Internet
Information Services\Web Management Tools” Program group. Be sure to check all items
under the “IIS 6 Management Compatibility” group.
Problem: ODBC DSN entries do not show up in the drop-down list, or DSN entry not found
error when running.
Solution: On 64-bit systems, the DSN must be configured using the 32-bit version of the
ODBC manager, it is typically located in: C:\Windows\SysWow64\odbcad32.exe.
Problem: When deploying, you get an error message “The HTTP request is
unauthorized with the client scheme ‘Anonymous’. The authentication header
received from the server was ‘Negotiate,NTLM’”.
Solution: In IIS Manager, expand Sites, Default Web Site, and select StarfishEngine. Under
IIS, select the “Authentication” item, and make sure “Anonymous Authentication” is set to
“Enabled”.
Problem: When testing the Starfish Engine website, you get the following (or similar) error
message: Could not load file or assembly 'Microsoft.Crm.Sdk' or one of its
dependencies. An attempt was made to load a program with an incorrect format.
Solution: On 64-bit systems, this can be caused if 32-bit applications have been disabled
for IIS.
For IIS6, go to a command prompt. Change to the “C:\Inetpub\AdminScripts” folder and
execute the following command:
cscript.exe adsutil.vbs set W3SVC/AppPools/Enable32BitAppOnWin64 true
For IIS7+, go to IIS Manager, select the Application Pool that Starifh is running under, select Advanced
Settings, and set "Enable 32-Bit Applications" to "Enabled".
Problem: Errors on Insert & Update stages when using SData Destination with Saleslogix
7.5.3, such as “Object not set to an instance of an object”.
Solution: This is caused by a bug in Saleslogix 7.5 Service Pack 3. If you are running 7.5.3,
be sure to apply 7.5.3 Hot Fix 1, which will fix this issue. SP, Hot Fix 3 may also be
necessary to get more detailed messages from SData (such as ValidationException
reasons).
Criteria: After installing the Engine, Admin tool, and Scheduler; use a provided
bindings file - training.oct to import a provided .csv file of new customers who
need to be added into SalesLogix as Accounts. We will be importing the data
into the SalesLogix Eval database.
NOTE: We will not address setting up the Eval database in this exercise.
You may wish to do the import on a development environment of your own.
This is perfectly fine.
3. Give Read access to the IIS_WPG user in the directory where you extracted the
Starfish_ETL_v1.1.zip. This is where the Customers.csv file we will be importing
is located.
4. Create a Text DSN to CSV file. This will allow you to set your .csv as an ODBC
data source Origin.
- Select Microsoft Text Driver (*.txt; *.csv) option from the list and click
Finish.
- .Give the Data Source a Name and Description similar to what you see below,
and click OK.
5. Log into the Starfish Admin and choose the File>Load Bindings File option.
Navigate to the folder where you unzipped the Starfish_ETL_v1.1.zip file and
choose to open the training.oct file.
NOTE: We will be using an existing Bindings file that has a Job set up, as well
as the Mappings for an import to Accounts in SalesLogix. We will need to
tweak the Origin and Destination, however.
6. The General Tab should appear. Within the Jobs dropdown menu, select
Import Customers. The rest of the tabs will then appear.
NOTE: An error will appear after selecting. This is because the Origin that was
originally mapped has not been configured yet. Click OK to get through the
error. A few seconds will go by and another related error will appear. Click OK
on that, as well. We will configure the Origin in a few minutes.
- Click on the ellipses next to the DSN. Choose the option for .csv Files. This is
available only after you have set up a DSN for .csv files, as we did earlier.
- Click on the ellipses next to Available Tables. Choose your Customers.csv file
that should appear here.
- Click the Test Connection button. You should see a Test Result of
‘Successful.’
- The existing connection information is left behind on the form. Use your
SalesLogix information to populate this form – enter your server on top, then
choose the database you wish to import the data into. Be sure to type in
your username and password in the User Settings area, and hit the Test
Connection button to be sure you will be properly connected.
- Click the Test Connection button on the right to be sure your Destination
connection succeeds.
- Click the dropdown next to Stages, and you will see three stages already
mapped: Create Account, Create Account Summary, and Create Address.
- Before we look at a few of the actual mappings in the stages, let’s first look at
the Exec Before section, where there are already two Exec Operations set up.
These particular Execs allow a SalesLogix stored procedure to be called that
creates IDs for new Accounts and Addresses, respectively. You will need to
map these IDs several times in the import, and the great thing about this
function is that it will allow you to set the Account and Address IDs as
variables that you can call each time you need to map the field.
o Get Account ID
o Get Address ID
- In the Stages dropdown again, select the Create Account Stage. This shows
all of the Bindings between our .csv and SalesLogix fields (as mentioned,
these have all been set up already).
As noted in the manual, some fields are mapped straight to each other.
Other fields, like the AccountID and AddressID fields use a Function Field.
This allows you to Edit the Function, and enter in a Hardcoded Value.
If you choose from the User-Defined Variable Class, you can select the ID’s
that were created by the Exec Before statements we just looked at.
- Feel free to look at the other fields in the various stages that have been
mapped out here. They may help to give you some ideas for your own
imports.
10. Now that we have our Origin and Destination set up, as well as our Mappings, we
are ready to run our import (we are not going to configure the Scheduler for this
import, as it is a one-time import). Before we run the import, we need to
Deploy to Web Service. As noted in the manual, this is how we get all of the
information we have just configured to the Starfish Engine.
- Depending on how and where you installed the Starfish Engine, use a path
similar to the following:
- After a minute, you should get word that the Import Customers job is
Complete. The amount of information you see in the log correlates with the
Logging Level you specified on the right hand side.
- If you were to check the SalesLogix Eval database right now, you would not
see your data. This is because you need to Commit the data. Now that we
know the import worked from our test run, check the Commit box, and hit the
Start button again. You should get the same result.
- From here, you may login to the SalesLogix Eval database and make sure the
data is there.
NOTE: The data in the Customers.csv file is originally from the SalesLogix Eval
database, so you should see two Abbott Ltd. Accounts, amongst others if your
import was successful.