Oracle Goldengate – Middleware

Sabalesh Mahajan
August, 2014 351229

1
 Introduction to Oracle Goldengate

Oracle Goldengate Evolution

Goldengate Software Inc was founded in 1995 by Eric Fish and Todd
Davidson.

Originating in San Francisco, the company was named after the famous
Goldengate Bridge by its founders.

Originally designed for the fault tolerant Tandem computers, the resilient and
fast data replication solution.

Became very popular within financial industry. The banks initially used
Goldengate software in their ATM networks for sending transactional data from
high street machines to mainframe central computers.

The data integrity and guaranteed zero data loss is obviously paramount and
plays a key factor.

Oracle Corporation acquired Goldengate Software in September 2009.

2
Oracle Goldengate – General Overview

Oracle Goldengate provides solutions that enable mission-critical systems

to have continuous availability and access to real-time data.

It captures, routes, transforms, and delivers transactional data in real time

and it works across heterogeneous environments with very low impact and
preserved transaction integrity.

It offer a robust yet easy platform for moving real-time transactional data
between operational and analytical systems to enable both:
* High Availability solutions
* Real Time Integration solutions

3
Oracle Goldengate - Advantages
• Real-Time Access - Critical data is accessible and available 24x7.

• Real-Time Information - The data available is as current as possible – not 24 hours

old, not even 4 hours old.

• Works in heterogeneous environments - Across different database and hardware

types.

• Transactional - Goldengate is “transaction aware” and apply readconsistent

changed data to maintain its referential integrity between source and target systems.

• High performance with low impact – Moves large volumes of data very efficiently
while maintaining very low lag times/latency.

• Flexibility – Due to open and modular architecture, it suits wide range of customer
solution and integration needs.

• Reliability – Extremely resilient architecture against potential interruptions, no single

point of failure or dependencies and easy to recover.

4
Oracle Goldengate Supported Databases and OS

Databases O/S and Platforms

Capture: Windows 2000, 2003, XP
Oracle Linux
DB2 Sun Solaris
Microsoft SQL Server HP NonStop
Sybase ASE HP-UX
Teradata HP TRU64
Enscribe HP OpenVMS
SQL/MP IBM AIX
SQL/MX IBM z/OS
Delivery:
All listed above, plus:
HP Neoview, Netezza, Greenplum, and
any ODBC compatible databases
ETL products
JMS message queues
MySQL
TimesTen

5
Oracle Goldengate Architecture and Components

6
Oracle Goldengate Architecture and Components

7
 Oracle Goldengate Components

Goldengate's fundamental building blocks are:

(the processes listed are depicts the sequence of events for Goldengate data
replication across distributed systems):

• Manager
• Capture process (Extract)
• Trail files
• Data pump
• Server collector
• Apply processes (Replicat)

• A Manager process runs on both the source and the target systems that
"oversee" the processing and transmission of data.

• All the individual processes are modular and can be easily decoupled or
combined to provide the best solution to meet the business requirements.

• It is normal practice to configure multiple Capture and Apply processes to

balance the load and enhance.

8
 Oracle Goldengate Components

Manager Process

• The Manager process runs on both source and target systems.

• Its job is to control activities such as starting, monitoring and restarting all
configured Oracle Goldengate processes on that system, allocating data storage
and reporting errors and events.

• The Manager process must exist in any Goldengate implementation.

• There can be only one Manager process per Changed Data Capture
configuration on the source and target.

• The Manager process can have either of the following statuses:

• STOPPED
• RUNNING

9
 Oracle Goldengate Components

Capture Process (Extract)

• Oracle Goldengate's capture process, known as Extract, obtains the

necessary data from the databases' transaction logs.
• For Oracle, these are the online redo logs that contain all the data changes
made in the database.
• Goldengate does not require access to the source database and only extracts
the committed transactions from the online redo logs.
• It can also read archived redo logs to extract the data from long running
transactions.
• The Extract process will regularly checkpoint its read and write position,
typically to a file. The checkpoint data insures Goldengate can recover its
processes without data loss in the case of failure.

• The Extract process can have one the following statuses:

• STOPPED
• STARTING
• RUNNING
• ABENDED (Short form of Abnormal end, usually because of some error)
10
 Oracle Goldengate Components

Trail Files

• To replicate transactional data efficiently from one database to another, Oracle

Goldengate converts the captured data into a Canonical Format which is written to files
known as Trail Files, both on the source and the target system.

• The provision of source and target trail files in the Goldengate's architecture eliminates
any single point of failure and ensures data integrity is maintained.

• A dedicated checkpoint process keeps track of the data being written to the trails on both
the source and target for fault tolerance.

• It is possible to configure Goldengate not to use trail files on the source system and write
data directly from the database's redo logs to the target server data collector. In this case,
the Extract process sends data in large blocks across a TCP/IP network to the target
system. However, this configuration is not recommended due to the possibility of data loss
occurring during unplanned system or network outages.

• Best practice states, the use of local trail files would provide a history of transactions and
support the recovery of data for retransmission via a Data Pump.

11
 Oracle Goldengate Components

Dump Files

• When using trail files on the source system, known as a local trail, Goldengate requires an
additional Extract process called Data Pump that sends data in large blocks
across a TCP/IP network to the target system.

• This is best practice and should be adopted for all Extract configurations.

Server Collector

• The Server Collector process runs on the target system and accepts data from the source
(Extract/Data Pump).

• Its job is to reassemble the data and write it to a Goldengate trail file, known as a remote
trail.

12
 Oracle Goldengate Components

Apply Process (Replicat)

• The Apply process, known in Goldengate as Replicat, runs on the target system and it is
the final step in the data delivery.

• It reads the target trail file and applies it to the target database in the form of DML
(deletes, updates and inserts) or DDL(database structural changes). This can be
concurrent with the data capture or performed later.

• DDL is only supported in unidirectional configurations and non-heterogeneous (Oracle

to Oracle) environments.

• The Replicat process will regularly checkpoint its read and write position, typically to a
file.

• The checkpoint data ensures that Goldengate can recover its processes without data
loss in the case of failure.

• The Replicat process can have one of the following statuses:

• STOPPED
• STARTING
• RUNNING
• ABENDED
13
 Oracle Goldengate Components

Capture Process (Extract Process)

• Oracle Goldengate's capture process, known as Extract, obtains the necessary data
from the databases' transaction logs.
• For Oracle, these are the online redo logs that contain all the data changes made in
the database.
• Goldengate does not require access to the source database and only extracts the
committed transactions from the online redo logs.
• It can also read archived redo logs to extract the data from long running transactions.
• The Extract process will regularly checkpoint its read and write position, typically to a
file. The checkpoint data insures Goldengate can recover its processes without data
loss in the case of failure.
• The Extract process can have one the following statuses:
• STOPPED
• STARTING
• RUNNING
• ABENDED (Short form of Abnormal end, usually because of some error)

14
 Oracle Goldengate Components
Checkpointing

• Checkpoint are used during online change synchronization to store the current read &
write position of a process.

• It ensures that data changes marked for synchronization are extracted and they prevent
redundant extractions.

• Checkpoints provides fault tolerance by preventing the loss of data should the
filesystem, the network or a Oracle Goldengate process needs to be restarted.

• Extract, Data Pump and Replicat process maintains their checkpoint information.

• Best practice is to create a checkpoint table in the database.

• Checkpoints data are maintained in both the checkpoint table (if it exists) and a
checkpoint file.

• Extract process maintains 3 input checkpoints and 1 output checkpoint for each trail file
it writes to. Replicat maintains 2 input checkpoints.

15
Oracle Goldengate Components

16
 Oracle Goldengate Components

Parameter File, Process Groups and Commands

• Goldengate processes are configured by ASCII parameter files known as parameter files
or prm files.

• In Oracle Goldengate, a process group consists of:

• An Extract or Replicat process
• Associated parameter file
• Associated checkpoint file
• Any other files associated with that process

• Each process group on a system must have a unique group name.

• Oracle Goldengate has it's own command line interface known as Goldengate Software
Command Interface (GGSCI).

• This tool provides the administrator with a comprehensive set of commands to create,
configure and monitor all Goldengate processes.

• The GGSCI commands are not case sensitive, but do support wildcards (*) where
appropriate.
17
 Oracle Goldengate Components

Oracle Goldengate Process

Data Flow

• Diagram on the right illustrates

the Goldengate processes and
their dependencies.
• The arrows largely depict
replicated data flow (committed
transactions), apart from
checkpoint data and configuration
data.
• The Extract and Replicat
processes periodically checkpoint
to a file for Persistence.
• The parameter file provides the
configuration data for the process.
• As described earlier, two options
exist for
sending data from source to target.
These are shown as broken
arrows.
18
Oracle Goldengate Topology

19
Oracle Goldengate Topology

20
Oracle Goldengate Installation on Linux

21
 Oracle Goldengate Software / Hardware Requirements

Software Requirements

• Download the software from Oracle Site http://edelivery.oracle.com /

http://www.oracle.com depending upon the Database Version, OS Bit and Oracle
Goldengate version.

• Oracle has placed Goldengate in the Oracle Fusion Middleware family section, under
Business Intelligence.

• For Linux installations, there are no specific requirements for kernel parameter settings
or RPMs. Typically, Goldengate is installed on a database server which has the
necessary kernel parameter settings and OS RPMs for Oracle.

22
Hardware Requirements

• Disk - Oracle Golengate require 300MB disk space for it's binaries and working
directories. Storage requirements for trail files is dependent upon the kind of source
database environment ( high/medium/low transaction volume), no. of days for preserving
trail file. Usually 1-5 GB disk space should be sufficient for any kind of Oracle Golengate
installation.

• Memory – Each Oracle Golengate process consumes 50MB of memory and each
instance of Oracle Golengate supports up-to 300 processes (Extract & replicat). If
maximum supported processes are used then memory requirement for Oracle Golengate
would be 16GB (300*50MB). Add OS and Database memory requirement accordingly.

• Network - Golengate requires a number of TCP/IP ports in which to operate. Any

network firewall must allow to pass traffic on these ports. One port is used solely for
communication between the Manager process and other Golengate processes. This is
normally set to port 7809 but can be changed. A range of other ports are for local
Golengate communications. These can be the default range starting at port 7840 or a
predefined range of up to 256 other ports. There should be sufficient network bandwidth
available between source and target to achieve almost real time data replication.

23
 Oracle Goldengate Installation

Installing Oracle Goldengate Software (Source and Target)

• Extract the Oracle Goldengate Mediapack zipped file downloaded from Oracle. Transfer the
binaries to source and target system. Unzipping the file produces a .tar file.

• Log on using telnet of ssh client to source database system as the owner of the installation
(It could also be non-database owner but ensure Oracle Goldengate installation owner is part
of the DBA OS group).

24
 Configuring & Preparing the Environment

Ensure Oracle Environment Variable is set correctly.

• ORACLE_SID=oltp
• ORACLE_BASE=/opt/oracle
• ORACLE_HOME=$ORACLE_BASE/app/product/11.1.0/db_1
• LD_LIBRARY_PATH=$ORACLE_HOME/lib
• PATH=$PATH:$ORACLE_HOME/bin:.
• export ORACLE_SID ORACLE_BASE ORACLE_HOME PATH LD_LIBRARY_PATH

• Create a directory ggs under the user home directory and extract the .tar file in this directory.

• $ mkdir ggs

• $ cd ggs

• $ tar -xvf <filename.tar>

• From the ggs directory, run the Goldengate Software Command line Interpreter (GGSCI) program.

$ ggsci – Run this to create Goldengate directories

GGSCI (ITIS.tcs.com) 1> create subdirs

25
 Goldengate Directories

dirchk
• The default location for checkpoint files created by the Extract and Replicat processes
that provide data persistence of read and write operations.
• The file name format is <group name><sequence number>.<file extension>.
• The file extension is cpe for Extract checkpoint files or cpr for Replicat checkpoint files.

dirdat
• The default location for Goldengate trail files and extract files created by the Extract
processes. These files are subsequently processed by either a Replicat process or
another application or utility.
• The file name format for trail files is < prefix> <sequence number>
• The prefix must be two alphanumeric characters specified during Extract or Replicat
creation. Typical prefix names are as follows:
• sa, sb, sc etc for the 1st, 2nd and 3rd trail files on the source
• ta, tb, tc etc for the 1st, 2nd and 3rd trail files on the target
• A 6-digit sequential number is automatically appended to each file prefix for each new
trail file created. The filename for extract files is user-defined name and has no
sequence number.

26
 Goldengate Directories

dirrpt
• The default location for report files created by Extract, Replicat, and Manager
processes.
• These ASCII files report statistical information relating to a processing run. When a
process abends the file is updated automatically.
• To obtain process statistics "on the fly", the REPORT command must be invoked from
within the GGSCI tool.
• The file name format is <group name><sequence number>.rpt.

dirtmp
• The default location for Goldengate process temporary files that "swap out" data related
to large transactions that exceed the allocated memory size.
• It is recommended that this subdirectory be created on its own disk to reduce I/O
contention.

Dirsql - The default location for SQL scripts.

Dirver - Stores Goldengate Veridata files

UserExitExamples - Stores Goldengate User Exit example files

28
Executable installed with Oracle Goldengate

29
Oracle Goldengate - Upgrade and De-install

30
Goldengate Upgrade

• Goldengate upgrade processes are very simple.

• For example, We need to follow the following short steps to achieve a successful
upgrade from Oracle Goldengate for Linux x86-64 bit version 10.4.0.19 Build 002 to
Version 11.1.1.0.0 build 78.
1. Log on to the database server as Oracle Goldengate Installation owner.
2. start a GGSCI session and stop all Extract and Replicat processes.
GGSCI (ITIS.tcs.com) 1> stop *
3. Now stop the Manager process.
GGSCI (ITIS.tcs.com) 2> stop mgr
Manager process is required by other GGS processes.
Are you sure you want to stop it (y/n)? y
4. Exit GGSCI and copy the patch file to the Goldengate home directory and unzip it.
$ unzip p10146318_11110_Linux-x86-64.zip
5. Now extract the archive from the resultant tar file.
$ tar -xvf ggs_Linux_x64_ora11g_64bit_v11_1_1_0_5_003.tar
6. Log in to GGSCI and start the Manager process.
GGSCI (ITIS.tcs.com) 1> start mgr
7. Start the Extract and Replicat processes if not already started.
GGSCI (ITIS.tcs.com) 2> start *
8. Ensure all processes are running.
31
Goldengate De-install

• Uninstalling the Goldengate software is the reverse of installing it.

• The de-installation is as simple as the installation and can be done by following the example
below:
1. Log on to the database server as the owner of oracle Goldengate Software.
2. Change directory to the Goldengate home:
cd /home/oradb03/ggs
3. Start GGSCI:
ggsci
4. Stop all Goldengate processes:
GGSCI (ITIS.tcs.com) 1> stop EXTRACT *
GGSCI (ITIS.tcs.com) 1> stop REPLICAT *
GGSCI (ITIS.tcs.com) 2> stop MGR
Manager process is required by other GGS processes.
Are you sure you want to stop it (y/n)? y
Sending STOP request to MANAGER ...
Request processed.
Manager stopped.
GGSCI (ITIS.tcs.com) 3> exit

32
5. Change directory to the installation directory:
cd /home/oradb03
6. Remove the GoldenGate files:
rm -rf ggs
7. Logon to the Oracle database as SYSDBA and drop the GoldenGate Admin user.
Include the CASCADE keyword:
$ sqlplus / as sysdba
SQL> drop user ggs_admin cascade;
User dropped.

33
Thank You !

34