Documente Academic
Documente Profesional
Documente Cultură
Abstract
This white paper provides an in-depth review of the capabilities of the EMC Avamar Oracle plug-in. The
primary focus of this paper is two-fold:
1. Provide the reader with a complete understanding of Avamar Oracle plug-in capabilities including
Oracle backup and recovery features and advanced concepts such as performance tuning
2. Describe how to protect large Oracle databases with the Avamar Oracle plug-in via multiple streams.
September 2009
Table of Contents
Executive summary ............................................................................................ 5
Introduction ......................................................................................................... 5
Audience ...................................................................................................................................... 5
Physical bottleneck................................................................................................................. 20
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
Executive summary
This white paper details procedures to enhance the performance and scalability of the EMC Avamar
Oracle plug-in for larger Oracle databases. In most cases the Oracle database size was greater than 1
terabyte.
The main goal of the testing was to demonstrate that the Avamar Oracle plug-in can address the data
protection needs of these larger databases. The test sample database was approximately 5 terabytes in size.
Multiple streams were used to achieve the recovery time objectives that were required.
Introduction
This white paper describes backup and recovery options for Oracle via Avamar and Oracle Recovery
Manager (RMAN) using the SBT_TAPE interface supplied in Avamar.
The primary focus of this paper is two-fold:
1. Provide the reader with a complete understanding of the Avamar Oracle plug-in capabilities including
Oracle backup and recovery concepts and advanced concepts such as performance tuning
2. Describe how to protect large Oracle databases with the Avamar Oracle plug-in via multiple streams.
It should be noted that Avamar complies with the standards that are set forth by Oracle. A shared library is
installed into the Oracle environment. This library allows Oracle RMAN to invoke the SBT interface. In
addition, there are several field-tested samples of scripts and configurations.
Audience
This white paper is intended for the technicians and database administrators (DBAs) who want to
understand how to utilize Avamar for their Oracle data protection needs. This group includes customers,
backup administrators, DBAs, technical consultants, delivery engineers, technical support engineers, and
EMC internal professionals.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
Was the Oracle database installed on the storage system using Oracles best practices?
By allocating multiple channels, and also having the proper environmental factors in place, Avamar is a
perfect fit for larger Oracle environments.
Several rounds of testing have been done on 4 TB and 5 TB Oracle databases. Avamar was used to back up
the database within a 10-12 hour nightly backup window.
Following are three real-world customer examples.
Example No. 1
Customer environment: 5 TB database hosted on a Sun Solaris machine with 128 GB of RAM (64 GB
of free RAM) and 12 core processors. The back-end disk is EMC Symmetrix DMX. Customer has a
recovery time objective of 24 hours and the backup window is 12 hours.
Can the Avamar Oracle plug-in be leveraged to protect this configuration?
Answer: Avamar would be a good fit here. You can allocate multiple RMAN channels to get the
backups and restores done within their windows.
Example No. 2
Customer environment: Sun Solaris machine with 128 GB of RAM running 25 instances of Oracle.
The system has 12 KB of RAM free.
Can the Avamar Oracle plug-in be leveraged to protect this configuration?
Answer: In this case, the machine is already memory constrained with its current load. You can try
Avamar for backup and recovery, but the RAM bottleneck will still be there.
Example No. 3
Customer environment: 30 TB data warehouse hosted on a large AIX server with plenty of RAM and
processors, and a very fast disk subsystem.
Can the Avamar Oracle plug-in be leveraged to protect this configuration?
Answer: It would be very difficult to complete the backup within the standard 10-12 hour backup
window. Avamar is likely not the best fit.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
Restore capabilities
Full system restore
Datafile restore
Tablespace level restore
Point-in-time restore
Archived redo log restores
In recent versions of Oracle, RMAN provides some advanced functionality. Please refer to Oracle
documentation for a complete listing.
Oracle datafile
This is a physical file that typically has tables of data inside it. It is where the data is essentially stored. This
object can be put online or offline. A single datafile can be backed up or restored.
An example is F:\oradata\payroll.dbf.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
Oracle tablespace
A tablespace is a logical collection of datafiles. It is a logical object, not a physical object. It can be
confused with an Oracle data table. It is useful to recover an entire tablespace depending on where the
database corruption occurred and how far it spread. The entire tablespace can be taken offline and online.
Examples are /u01/oradata/payroll.dbf, /u01/oradata/currency.dbf, and /u01/oradata/taxes.dbf.
Table: Where you store data. It is comprised of rows and columns. Various datatypes.
Views: Logical views of specific data. For example, V$Database allows you to see statistics on the
database. Sometimes referred to the system schema.
Instance: This is the actual target database you are working with. There may be a production instance,
a test instance, and also a development instance. This is typically referenced by a System Identifier
(SID) name. A good example of this may be Payroll_Prod.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
Recovery catalog
The recovery catalog database is an Oracle database that is created and used to keep track of the RMAN
objects and allows you to recover them if you lose your target database control file. The target database is
the database that you are backing up. This catalog also allows you to perform advanced reporting on
backups and lists the target database schema.
If you choose not to use the recovery catalog database, your RMAN information is then stored in the target
databases control file.
Later versions of Oracle 9i have a featured called control file Autobackup, which makes a snapshot copy of
the control file and parameter file, and stores them in a location that you can take with the normal Avamar
file system backup. You can then recover the control file directly from the Autobackup location. This is a
configurable location.
It is important to understand that Oracle recommends that all production RMAN environments use a
recovery catalog. However Test and Dev environments running Oracle 9i and later can use the option. Be
careful on what the DBAs perception is. Most will push to use the recovery catalog either way.
The recovery catalog database should be running on a separate machine from the target database, which
will allows you to get your environment up and running quicker. Imagine the difficulty in having to recover
the catalog and then the target database.
To use RMAN to perform hot backups, the database must be running in Archive Log Mode.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
Ensure that the Avamar File System client is installed and the client is registered to the Avamar Utility
Node. Please refer to the EMC Avamar Oracle Client Installation Guide and Reference Manual.
2.
Install the Avamar SBT_TAPE library into your target database. This is outlined well in the EMC
Avamar Oracle Client Installation Guide and Reference Manual.
3.
Create a user within the target database and grant that user privileges. Also create the same user in
Avamar. If you are using a recovery catalog database, create that user in the recovery catalog as well.
4.
Put your target database into Archive Log Mode. This is required for hot backups.
5.
/ as sysdba
shutdown immediate
startup
exit
This step may be necessary to bind the SBT library with the Oracle kernel.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
10
/ as sysdba
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 Production
You may need to identify the users default tablespace. Here is an example.
SQL> alter user backup identified by backup
DEFAULT TABLESPACE users;
User altered.
11
Universal
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 - Production
/ as sysdba
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 Production
SQL> alter user backup identified by backup; Oracle 9
User altered.
SQL> alter user backup account unlock; Oracle 9
User altered.
You may have to identify the users default tablespace. Here is an example.
SQL> alter user backup identified by backup
DEFAULT TABLESPACE users;
User altered.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
12
Universal
If you are running Oracle 9i with an SPFILE do the following. You will not need to do this with Oracle 10g
and later.
oradba@oratest1# sqlplus /nolog
SQL*Plus: Release 9.2.0.1.0 - Production on Thu Apr 24 17:38:56 2003
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected
to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the
Partitioning, OLAP and Oracle Data Mining options
SQL> connect / as sysdba
SQL> alter system set log_archive_start=true scope=spfile;
System altered.
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options JServer Release
9.2.0.1.0 Production.
Now we can actually put the database into Archive Log Mode.
oracle@oratest1# setenv ORACLE_SID targetdb
oradba@oratest1# sqlplus /nolog
SQL*Plus: Release 9.2.0.1.0 - Production on Thu Apr 24 17:38:56 2003
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected
to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the
Partitioning, OLAP and Oracle Data Mining options
SQL> connect /
as sysdba
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
13
Database dismounted.
ORACLE instance shut down
SQL> startup mount
ORACLE instance started.
Where PASSWORD is the sys account password on the database that is pointed to by ORACLE_SID.
Verify that the Oracle password file was created by entering:
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
14
ls -la orapw
Now we can have Oracle read the passwd file when Oracle starts up.
This next step is required for the password file to be read during database
initilization.
oracle@oratest1# vi /oracle/product/10.1.0/Db_1/dbs/initdbname.ora
(or the equivalent initSID.ora file for your database)
un-comment the following entryin the init.ora file
REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE
/ as sysdba
Invoke RMAN from the Avamar Administrator. Please review Appendix C: Avamar Administrator
Oracle GUI backups on page 35.
Invoke RMAN manually from the command interpreter (good for testing)
Invoke RMAN via the OS scheduler (cron) using RMAN cmdfiles. The Oracle 9i syntax is:
o oradba@targ1# rman cmdfile=fullbackup.rman
o oradba@targ1# rman @`fullbackup.rman` (the most popular option)
15
{
allocate channel T1 type 'SBT_TAPE'
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,
ENV=(PATH=/bin:/usr/bin)";
send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';
backup database;
release channel T1;
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
16
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
17
}
Then invoke the script like this.
oracle@test1# rman cmdfile fullbkavamar.rman
My-Avtar-Flags.txt explained
The following is a sample copy of the My-Avtar-Flags file with flag descriptions and values.
--debug
--pidname=Oracle
--pidnum=1002
--logfile=/usr/local/avamar/var/avtar.log
--vardir=/usr/local/avamar/var
--id=backup
--ap=backup
--path=/clients/vaerp-db4.va.company.org
18
RMAN Restore Command This command initiates the actual physical rebuild of the Oracle object
that you need. It is rebuilding the data from the Avamar Storage Node. This is the physical restore.
RMAN Recovery Command This command allows you to bring the database to a consistent point
in time. Depending on the options, you may need to apply the archived redo logs. This is the logical
recovery.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
19
You will find various permutations of this exercise depending on what needs to be recovered and rolled
back. You can set the SCN to find a point in time to recover the database from.
It is also important to note that Oracle 9i and later have the auto control file backup option. If this is being
used the syntax to restore the control file is slightly different.
Physical bottleneck
The system in question here would be the Avamar backup client system that hosts the Oracle database. It
could be a physical or virtual machine with various hardware components.
Every system built will have a slowest component that translates into a bottleneck. The goal is to identify
the bottleneck to improve the efficiency of the backup and recovery jobs. Often, once you have addressed
one bottleneck, another bottleneck will quickly surface.
The process is outlined in the following sections. Table 1 shows some examples of physical system
bottlenecks. Your backup and recoveries will go only as fast as the system bottleneck. Every system will
have a physical bottleneck.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
20
Physical component
Memory
CPU
I/O Subsystem
Network (First Backup)
Network
I/O Subsystem
Avamar will process Oracle backups as fast as the memory and CPU will allow. Be aware that Avamars
initial first backup will appear to utilize more network and server resources as the effect of the
deduplication will not be fully realized. Subsequent backups will process faster and utilize less resources
as the local cache files and data store are populated.
During backups, the I/O subsystem read performance may be the bottleneck and will need to provide
exceptional read and seek performance.
EMCs experience at customer sites is that database backup performance ranges from approximately 60 to
100 GB/hour per backup stream. Performance may even be in excess of 200 GB/hour per stream. EMC
has, in numerous cases, observed a linear increase in backup performance as multiple parallel backup
streams are launched to back up the data.
Allocating multiple RMAN channels to boost the overall backup performance is definitely worth exploring
under the following conditions:
If the I/O subsystem is well architected and can support multiple concurrent backup streams that are
each capable of streaming backup data in excess of 100 GB per hour.
NOTE: "Well architected" in this case means that backing up the data through multiple streams should not create
contention in the I/O subsystem and therefore, slow down, rather than improve, the overall backup performance.
If the host has multiple processors available during the backup window. By default, the Avamar client
will distribute the SHA-1 hashing and compression out to all the processors on the client. However,
the main avtar thread is mapped to a single processor, and on some clients with slower individual
processors, this could be the bottleneck. In this case, allocating multiple RMAN channels creates
multiple avtar processes, each of which will have a main thread that is mapped to a different processor,
thus boosting overall performance.
As described in the next section "Avamar cache sizing for databases, the maximum amount of
memory that will be consumed by the Avamar process to optimally perform the Avamar backups of
the database data will be approximately 0.001 (1/1,000) of the total size of the database.
For example, to back up a 5 TB Oracle database, the maximum amount of memory consumed will be
approximately 5 GB total for all concurrent streams.
Although the examples later are based on using four RMAN channels to back up the Oracle database data,
the optimum number of required RMAN channels needs to be determined based on the following factors:
The optimum number of separate backup streams that can be supported by the I/O subsystem before
excessive contention slows down performance.
The CPU utilization will tend to continue to improve as more channels are added, until the number of
channels equals the number of processors.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
21
As seen in the earlier formula, the amount of memory that needs to be allocated for the backup process
is independent of the number of channels that are used to perform the backup.
For reasons that will become apparent in the next section, enough RMAN channels will need to be
allocated so that the average amount of data per channel does not exceed 1.5 TB.
Based on current lab testing, the Avamar/Oracle recovery should go as fast as the network will allow.
This pertains to the client-side network interface, which in the testing to date has been a single GigE
interface. A second client-side network interface should help reduce this bottleneck.
22
MBs. Therefore, for example, setting "--hashcachemax=500" will allow the hash cache file to grow to
384 MB (because 384 MB is the largest hash cache file less than 500 MB).
From rules (3), (4), (5), and (6), we can derive the following rule for setting the "hashcachemax" option:
--hashcachemax = 2 x MB <Size of database in GB>
So if the database is 700 GB the formula should look like this.
Hash Cache Max should be less than 2 x 700 MB (Converted GB to MB) = 1400 MB
Hashcachemax
10 GB DB
Stream
--hashcachemax=20
24 MB
100 GB DB
Stream
--hashcachemax= 200
192 MB
500 GB DB
Stream
--hashcachemax= 1000
768 MB
700 GB DB
Stream
--hashcachemax=1400
768 MB
1000 GB DB
Stream
--hashcachemax=2000
1536 MB
1536 GB DB
Stream
--hashcachemax=3072
1536 MB
So you can see as we get to the larger dataset size the maximum hash cache will be likely 1,536, which is
the largest it can possibly get today. A dataset over 2 TB that uses a single hash cache will likely slow
down because it does not have a large enough hash cache. This is where the multiple stream discussion
becomes relevant.
You can specify the "--hashcachemax=<>" parameter in the my-avtar-flags file.
You will have to rename the original p_cache.dat file and your next backup will build a new one.
When you allocate multiple RMAN channels to back up the Oracle database, each channel will process
roughly the same amount of data. This means that you can specify the same "--hashcachemax=<2 x
amount of data per RMAN channel (in GB)>" in the my-avtar-flags.txt file.
When backing up large databases (greater than 2 TB databases), it might be important to minimize the
amount of memory consumed on the host. In this case, the amount of data backed up per RMAN channel
should be ~750 GB or ~1.5 TB. This ensures that we can effectively utilize a 768 MB or 1,536 MB hash
cache file with very little wasted space in the hash cache file.
It is also important to understand that if your cache files are close to these values and not exactly 100
percent the same, you should be safe. You should be aware of major deviations from the rules.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
23
This leads to the case for allocating multiple channels, which is equivalent to generating multiple streams
or multiple Avtar processes. This is definitely needed when it comes to a restore of a larger database.
The blue text outlines the unique parts of this script. This is essentially how to address the multiple
channels and map them to the multiple cache files. The cacheprefix=c(n) creates a unique set of cache
files for that channel. There are four in this example.
The logfile= creates a unique set of log files for each channel.
All of these files are stored in the standard /usr/local/avamar/var directory.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
24
This script is used for the restore and looks very much like the backup script. Always make sure the DBA is
involved with the restore and recovery of an Oracle database. There are many restore scenarios and the
DBA will need to be aware of the state of the database before you begin.
The main concept to gather from this example is that the recovery will come back in multiple streams. This
is just how the backup was performed.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
25
Sample outputs
Figure 1. Prebackup
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
26
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
27
Conclusion
As can be seen by examples in this white paper, Avamar can be used as a viable solution for larger database
environments. By using multiple RMAN channels you can easily achieve respectable backup and recovery
time objectives.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
28
SunOS 5.8
login: oracle
Password:
Last login: Wed Jan 21 08:06:31 from legato1
Sun Microsystems Inc. SunOS 5.8 Generic February 2000
Test1%
2.
3.
4.
5.
Select General Purpose on the Database Templates screen and click Next.
6.
7.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
29
8.
Enter password information on the Oracle Database Configuration Assistant screen and click Exit.
Example:
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
30
{
allocate channel T1 type 'SBT_TAPE'
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/b
in)";
send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';
backup datafile '/d1/dev3data/usr01.dbf' ;
release channel T1;
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
31
run {
allocate channel T1 type 'SBT_TAPE'
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:
/usr/bin)";
send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';
backup tablespace DEMO;
release channel T1;
}
run {
allocate channel T1 type 'SBT_TAPE'
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:
/usr/bin)";
send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';
restore tablespace DEMO;
recover tablespace DEMO;
sql "alter tablespace DEMO online";
release channel T1;
}
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
32
run {
allocate channel T1 type 'SBT_TAPE'
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:
/usr/bin)";
send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';
#restore spfile;
restore spfile from autobackup;
release channel T1;
}
shutdown immediate;
startup nomount;
#restore controlfile;
restore controlfile from autobackup;
alter database mount;
restore database;
recover database;
release channel T1;
}
run {
shutdown immediate;
startup mount;
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
33
run {
configure controlfile autobackup on;
allocate channel c1 type sbt
PARMS="SBT_LIBRARY=C:\PROGRA~1\avs\bin\orasbt.dll,ENV=(PATH=C:\PROGRA~1\avs\bin
)" format '%d_%U';
set controlfile autobackup format for device type sbt to
"CONTROLFILE.DUBLIN.%F";
send '"--flagfile=C:\TEMP\FlagFile.txt" --debug';
backup database plus archivelog delete input;
}
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
34
Oracle Instance Name specifies the Oracle server name. This field can normally be left blank because
the Oracle instance name is automatically stored when the target client is browsed during on-demand
snapup or when a dataset is applied to a member of a group during a scheduled snapup.
User name specifies the username used to authenticate with the Oracle database. If this field is left
blank, RMAN tries to log in with the same username and password that the Avamar client agent is
running under, and attempts to assume SYSDBA privileges. Typically, this field should contain the
special account name (backupuser) manually created as part of database preparation. Username and
Password comprise a connection string to Oracle. The connection string must specify a user who has
backup privileges for the database.
The Delete Archive Logs after backup checkbox, if selected, automatically deletes Oracle archive
logs immediately following a successful database backup.
Enhanced Commonality Factoring enables or disables enhanced data deduplication. During snapups,
enhanced data deduplication typically reduces the amount of client data that must be sent to the server
but requires additional client CPU resources. Choices are:
Disabled
Default (use the global enhanced data deduplication setting already set on the server)
Enabled
The Enable debugging messages checkbox, if selected, adds all available debugging output to both
avtar and avoracle. This is useful when submitting a problem to EMC Technical Support so that an
issue can be more easily diagnosed.
The Use Recovery Catalog checkbox is followed by three recovery catalog fields. The Recovery
Catalog Server Name, Recovery Catalog Username and Recovery Catalog Password settings are used
to specify a Recovery Catalog Server connection string for RMAN. By using a Recovery Catalog
Server, many specialized features of RMAN may be used. A thorough discussion of these features is
beyond the scope of this paper.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
35
Recovery Catalog Server Name specifies the recovery catalog server name.
Restore options
The options in Figure 7 can be specified during the restore operations. They are explained below.
Oracle Instance Name specifies the Oracle server name. This field can normally be left blank because
the Oracle instance name is automatically stored when the target client is browsed during restore
operations
User name specifies the username used to authenticate with the Oracle database. If the field is left
blank, RMAN tries to log in with the same username and password the Avamar client agent is running
under, and attempts to assume SYSDBA privileges. Typically, this field should contain the special
account name (backupuser) manually created as part of database preparation. Username and Password
comprise a connection string to Oracle. The connection string must specify a user that has backup
privileges for the database.
Recovery Until SCN is for the recovery portion, so you can play back archive log files until the
transaction with this SCN is reached. If this field is left blank, avoracle will scan through the known
backups in the backup catalog (either in the system control file or the recovery catalog) and find the
highest SCN associated with a backup file.
The Enable debugging messages checkbox, if selected, adds all available debugging output to both
avtar and avoracle. This is useful when submitting a problem to EMC Technical Support so that an
issue can be more easily diagnosed.
The Only restore control file checkbox, if selected, restores only the control file, and known backups
in the backup catalog (either in the system control file or the recovery catalog) are listed. This is useful
when the user wants to find an SCN for a database or archive log without actually performing the
restore.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
36
Restore Command Options is for additional command line options that will be passed to the RMAN
restore command.
Recover Command Options is for additional command line options that will be passed to the RMAN
recover command.
For descriptions of Use Recovery Catalog, Recovery Catalog Server Name, Recovery Catalog
Username, and Recovery Catalog Password, see the Snapup options section.
2.
--debug
--logfile=/opt/AVMRclnt/var/avoracle_browse.log
3.
4.
Look in the avoracle_browse.log file and you will be able to identify the error you have and take
the necessary steps to correct it.
Browse
Backups
Restores
While the functionality of Backup and Restore should be fairly obvious, Browse is an intermediate and not
so visible operation.
When an Avamar Oracle operation is selected in Avamar Administrator, it sends a message to the
concerned client node. This message is called a workorder, and it contains all the information a client needs
to perform an operation.
Communication between the Avamar Administrator and the client node is always initiated through
Avagent. Avagent runs as a daemon (service) and listens for messages from the Administrator.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
37
Avagent inspects the workorder received, and uses the information to decide which of the Avamar clients
needs to be invoked. If the operation is meant for the Oracle plug-in, Avagent starts Avoracle and passes it
the workorder.
Avoracle operations
Browse
The first step in starting a backup of an Oracle database is the Browse operation.
When the Oracle plug-in is selected in the Backup panel of Avamar Administrator, it sends a Browse
request to Avoracle (through Avagent). This triggers a database discovery process in Avoracle.
Avoracle reads the /etc/oratab to get a list of databases on the system and their home directories. Using the
Oracle home directories it then looks for the tnsnames.ora file.
The special fields of interest in tnsnames.ora are HOST and SERVICE. Using this, Avoracle generates a
list of databases that are local to the system. This list is then returned to Avamar Administrator in a
message. Avamar Administrator then uses this list to populate the Backup panel.
Backup
When one or more databases are selected for backup, a new message is sent to Avoracle. This message
contains the list of databases to be backed up, together with any options that the user specifies. The
message also contains credentials that Avoracle can use to communicate with the Avamar Server.
In the current implementation Avoracle can only do backups and restores sequentially. So, even if multiple
databases have been specified, they are done one by one. The following description is for one database. For
multiple databases the sequence is just repeated.
Avoracle uses RMAN and Oracles SBT feature for taking backups. For this Avoracle specifies a loadable
library that can communicate with the Avamar Server. This library implements certain entry points that
have been written to Oracles specification. The library path and some other parameters are communicated
to RMAN through a script that Avoracle creates.
In Oracle 10g and later, Oracle has a restriction that prohibits RMAN from running as root. Avoracle
scripts are run as the default user oracle unless an alternate user id has been specified.
The RMAN script, credentials, and logfiles are created and stored in the Avamar var directory. Then
avoracle creates another process and starts RMAN, telling it what script to use. Avoracle then goes into a
loop waiting for a connection from avtar (more on this later) or for RMAN to complete its operations and
exit.
Once RMAN starts, it communicates with Oracle and tells it which library needs to be loaded. Oracle loads
this library (which is actually an Avoracle component) and uses it for backup. After this point the backup
process is controlled by RMAN. RMAN, by default, backs up archive logs, the database file, and the
control file. Based on certain factors RMAN splits up the entire backup into one of more backup pieces.
For each backup piece (which is actually opaque to Avoracle), RMAN begins by communicating the piece
name to libobk. It then performs the write using multiple calls to libobk, ending the operation with a close
command.
When libobk receives the backup command it creates an avtar process. The name of the backup piece is
communicated to avtar together with some other information.
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
38
Avtar is also told how to connect with the already running Avoracle. This last step is required so that avtar
can get the workorder from Avoracle and be able to send it status updates.
Avtar establishes connection with the Avamar Server and then waits for data from libobk. Every time the
write entry point of libobk is called by Oracle, libobk takes the data and send it to avtar.
Avtar will process the data (chunking and hashing) and store it on the server. The backup piece is stored on
the server with the same name that RMAN had generated. This makes it easier to find the right file during
restores. Once all the data for a backup piece has been received avtar terminates and libobk returns control
to RMAN.
After the entire database has been backed, RMAN unloads libobk and exits, returning control to Avoracle.
Restores
The Restore process is similar to the Backup process except for the direction of data flow. The Restore
browse panel is populated by the Avamar Administrator and it lets the user select one or more backups to
be restored.
Avoracle receives this list and does preprocessing on it to extract the control file names. For every control
file that has been selected for restore the following happens.
Avoracle first creates an RMAN script for restoring just the control file. It starts RMAN with the script
(everything else is the same as the backup operation). RMAN loads libobk and restores the control file. If
the user has not specified the SCN for restore, Avoracle examines the RMAN logfile and extracts the latest
SCN corresponding to the control file selected.
Avoracle then creates a new script for recovery of the database files. At this point RMAN has mounted the
database using the control files. Once again RMAN is started up with the proper script, and libobk is
loaded. Based on the SCN specified and the backup history in the control file, RMAN decides which
backup pieces need to be restored. These piece names are communicated to libobk, which in turn talks to
avtar. Avtar gets these files from the Avamar Server and the data is sent back to RMAN through the
corresponding entry point in libobk. Once all the required files have been restored, RMAN does a recover
operation, opens the database, and exits.
Following is a graphical representation of the entire process.
39
For 64-bit Oracle vs. a 32-bit library, verify the "bitness" is the same.
Must pass the correct PATH, such that RMAN can initiate "avtar" and "uname".
Must set the right environment variable such that the Avamar library can be located and loaded.
Different platforms (Solaris, HP-UX, Linux, AIX, and Windows) sometimes use different environment
variable names for the library.
Different shells (cmd, bash, sh, csh) use different syntax for setting and exporting environment
variables.
Avamar software, config, and log location (--vardir, Library, logs) vary by platform.
The PATH and ENVIRONMENT are important because of the complex process structure involved:
1.
2.
3.
4.
5.
The first step can be short-cut, by invoking "rman" directly from a command line. However, the PATH and
ENVIRONMENT are still critical.
cat
/opt/AVMRclnt/bin/my_avtar_flags.txt
--debug
--pidname=Oracle
--pidnum=2002
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
40
--logfile=/usr/local/avamar/var/avtar.log
--vardir=/usr/local/avamar/var
--id=backup
--ap=backup
--path=/clients/ux210.fairisaac.com
total 5684
drwxr-xr-x
2 root
root
drwxr-xr-x
5 root
root
-rwxr-xr-x
1 root
root
-rwxr-xr-x
1 root
root
-rwxr-xr-x
1 root
root
-rwxr-xr-x
1 root
root
file /opt/AVMRclnt/lib/libobk_avamar64.so
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
41
Try loading the library using the oracle "SBTTEST" (again for Solaris64, use bash). The library path and
command to set the appropriate variable have the following variations:
Table 3. Operating system and environment variable name
OS
SHLIB_PATH
AIX
LIBPATH
The following are the commands to set the variable changes, depending on the shell:
bash, sh
LD_LIBRARY_PATH="$LD_LIBRARY_PATH":/opt/AVMRclnt/lib
export LD_LIBRARY_PATH
csh
echo $LD_LIBRARY_PATH
/usr/oracle/product/9.2.0.8.64bit/lib:/usr/lib:/usr/dt/lib:/usr/openv/netbackup
/bin
echo $LD_LIBRARY_PATH
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
42
/usr/oracle/product/9.2.0.8.64bit/lib:/usr/lib:/usr/dt/lib:/usr/openv/netbackup
/bin
Check that it is installed properly, and that the LD_LIBRARY_PATH environment variable (or its
equivalent on your platform) includes the directory where this file can be found. Additional error
information is listed below:
export LD_LIBRARY_PATH=/opt/AVMRclnt/lib
echo $LD_LIBRARY_PATH
/opt/AVMRclnt/lib
Avamar
: sbtinit V1
-- sbtinit succeeded
Avamar
: sbtinit V1
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
43
Avamar
: sbtinit2
-- sbtinit2 succeeded
Avamar
: sbtbackup
: file=testfile
: block_size=16384
Avamar
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
44
--sysdir=/opt/AVMRclnt/etc --label=testfile -c
testfile --from-stdin --check-stdin-
path=false
address.
Avamar
: avtar exited
MMAPI error from sbtbackup: 7501, sbtbackup: avtar exited, trying to create testfile
-- sbtbackup failed
rm testfile
Set LD_LIBRARY_PATH (Or equivalent on HPUX/AIX)
export LD_LIBRARY_PATH=/opt/AVMRclnt/lib
Then re-running the RMAN command.
rman
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
45
RMAN> run
PARMS="SBT_LIBRARY=/opt/AVMRclnt/lib/libobk_avamar64.so,ENV=(PATH=/bin:/usr/bin:/opt/AVMR
clnt/bin)";
send '"--flagfile=/opt/AVMRclnt/bin/my_avtar_flags.txt"';
Avamar
: sbtinit V1
trace_file=/usr/oracle/admin/ivvint0/udump/sbtio.log
Avamar
: sbtinit2
allocated channel: T1
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
46
Avamar
: sbtcommand = "--flagfile=/opt/AVMRclnt/bin/my_avtar_flags.txt"
--Snip--
Avamar
: sbtbackup
: file=83k5u2c2_1_1
: block_size=262144
Avamar
--Snip--
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
47
2009/01/28-13:36:35.85013 [avtar]
socktype=sock_tcp
-- Snip -Recovery Manager complete
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Best Practices Planning
48