Roll Forward Physical Standby Database using

RMAN incremental backup

2 Votes

There would be scenarios where the standby database lags far behind from the primary database leading
to Archive Gap. It could be due to one of the following reasons

1. Might be due to the network outage between the primary and the standby database leading to the
archive gaps. Data guard would be able to detect the archive gaps automatically and can fetch the
missing logs as soon as the connection is re-established.

2. It could also be due to archive logs getting missed out on the primary database or the archives getting
corrupted and there would be no valid backups.

In such cases where the standby lags far behind from the primary database, incremental backups can be
used as one of the methods to roll forward the physical standby database to have it in sync with the
primary database.

Oracle Database version : 11.2.0.1.0 My Oracle Database is using ASM.

Primary database : sspm Standby database : sssb

Primary Host : dev Standby Host : uat
The maximum archivelog sequence generated on the Primary Database is 1005.

1 SQL> select status,instance_name,database_role from v$database,v$instance;

2
3 STATUS INSTANCE_NAME DATABASE_ROLE
4 ------------ ---------------- ----------------
5 OPEN sspm PRIMARY
6
7 SQL> select thread#,max(sequence#) from v$archived_log group bythread#;
8
9 THREAD# MAX(SEQUENCE#)
------- ----------------
10
1 1005
11
On the standby database, the maximum archivelog sequence that is applied is sequence 865.
1 SQL> select status,instance_name,database_role from v$database,v$instance;
2
3 STATUS INSTANCE_NAME DATABASE_ROLE
4 ------------ ---------------- ----------------
MOUNTED sssb PHYSICAL STANDBY
5
6 SQL> select thread#,max(sequence#) from v$archived_log where applied='YES' group by
7 thread#;
8
9 THREAD# MAX(SEQUENCE#)
10 ------- ----------------
11 1 865

The standby database is lagging behind the primary database by around 140 archives (1005 – 865).

When I investigated the alert log file of the Primary database to find out the reason for the logs not
getting applied on the standby database, I got to see the below error message.
Sun Mar 25 15:40:23 2012
1 Errors in file /u01/app/oracle/diag/rdbms/sspm/sspm/trace/sspm_arc2_18816.trc:
2 ORA-00308: cannot open archived log
'+FRA/sspm/archivelog/2012_03_25/thread_1_seq_866.1117.778865785'
3 ORA-17503: ksfdopn:2 Failed to open file
4 +FRA/sspm/archivelog/2012_03_25/thread_1_seq_866.1117.778865785
5 ORA-15012: ASM file '+FRA/sspm/archivelog/2012_03_25/thread_1_seq_866.1117.778865785' do
exist
So the problem was here. The archivelog sequence 866 was missing and was unavailable at the FRA
site. There were few more archives missing on the FRA and nor did I had the backup to restore them on
the standby database. My option was to go with Roll Forwarding the Standby Database using Incremental
Backups. Below are the steps on how to roll forward the physical standby database.

Step 1: Take a note of the Current SCN of the Physical Standby Database.
Standby Database:
1 SQL> select current_scn from v$database;
2
3 CURRENT_SCN
4 -----------
5 991247

Note down the CURRENT_SCN value of the standby database (991247) to proceed further.

Step 2 : Cancel the Managed Recovery Process on the Standby database.

Standby Database:
1 SQL>alter database recover managed standby database cancel;
Step 3: On the Primary database, take the incremental SCN backup from the SCN that is currently
recorded on the standby database (991247)
Connect to the primary database and take the incremental SCN backup.

Primary Database:
1 [oracle@dev ~]$ rman target sys/oracle@sspm
2
3 Recovery Manager: Release 11.2.0.1.0 - Production on Sun Mar 2515:44:45 2012
4
5 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
6
connected to target database: SSPM (DBID=1624493265)
7
8 RMAN> backup incremental from scn 991247 database format
9 '/u02/bkp/stnd_backp_%U.bak';
10
11 Starting backup at 25-MAR-12
12
13 using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
14 channel ORA_DISK_1: SID=39 device type=DISK
15 backup will be obsolete on date 01-APR-12
16 archived logs will not be kept or backed up
17 channel ORA_DISK_1: starting full datafile backup set
18 channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001
19 name=+DATA_NEW/sspm/datafile/system.256.778803539
20 input datafile file number=00002
21 name=+DATA_NEW/sspm/datafile/sysaux.257.778803541
22 input datafile file number=00003
name=+DATA_NEW/sspm/datafile/undotbs1.258.778803541
23 input datafile file number=00004 name=+DATA_NEW/sspm/datafile/users.259.778803543
24 channel ORA_DISK_1: starting piece 1 at 25-MAR-12
25 channel ORA_DISK_1: finished piece 1 at 25-MAR-12
26 piece handle=/u02/bkp/stnd_backp_10n6p3nl_1_1.bak tag=TAG20120325T154639
27 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:45
28
29 using channel ORA_DISK_1
30 backup will be obsolete on date 01-APR-12
31 archived logs will not be kept or backed up
32 channel ORA_DISK_1: starting full datafile backup set
33 channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
34 channel ORA_DISK_1: starting piece 1 at 25-MAR-12
35 channel ORA_DISK_1: finished piece 1 at 25-MAR-12
36 piece handle=/u02/bkp/stnd_backp_11n6p3p4_1_1.bak tag=TAG20120325T154639
37 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
38 Finished backup at 25-MAR-12
39
Step 4: Take the standby controlfile backup of the Primary database controlfile.
Connect to the Primary database and create the standby controlfile backup.

Primary Database :
1 RMAN> backup current controlfile for standby format '/u02/stnd_%U.ctl';
2
Starting backup at 25-MAR-12
3 using channel ORA_DISK_1
4 channel ORA_DISK_1: starting full datafile backup set
5 channel ORA_DISK_1: specifying datafile(s) in backup set
6 including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 25-MAR-12
7 channel ORA_DISK_1: finished piece 1 at 25-MAR-12
8 piece handle=/u02/stnd_12n6p3qt_1_1.ctl tag=TAG20120325T154845
9 comment=NONE
10 channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 25-MAR-12
11
12
Step 5: Transfer the backups from the Primary Server to the Standby Server.
Primary Database :
1 [oracle@dev bkp]$ pwd
2 /u02/bkp
3 [oracle@dev bkp]$ ls -lrt
4 total 13576
-rw-r----- 1 oracle oinstall 540672 Mar 25 15:47
5 stnd_backp_10n6p3nl_1_1.bak
6 -rw-r----- 1 oracle oinstall 13336576 Mar 25 15:47
7 stnd_backp_11n6p3p4_1_1.bak
8 [oracle@dev bkp]$ scp stnd* uat:/u02/bkp
oracle@uat's password:
9
stnd_backp_10n6p3nl_1_1.bak 100% 528KB 528.0KB/s 00:00
10
11 stnd_backp_11n6p3p4_1_1.bak 100% 13MB 6.4MB/s 00:02
[oracle@dev bkp]$ cd /u02
12 [oracle@dev u02]$ ls -lrt stnd*
13 -rw-r----- 1 oracle oinstall 13336576 Mar 25 15:48 stnd_12n6p3qt_1_1.ctl
14 [oracle@dev u02]$ scp stnd* uat:/u02
15 oracle@uat's password:
16 stnd_12n6p3qt_1_1.ctl 100% 13MB 12.7MB/s 00:01
[oracle@dev u02]$
17
Step 6: On the standby server, connect the Standby Database through RMAN and catalog the copied
incremental backups so that the Controlfile of the Standby Database would be aware of these incremental
backups.
I had the incremental backuppieces copied to the location ‘/u02/bkp‘ on the standby server.
Standby Database:
1 [oracle@uat ~]$ rman target sys/mydbpwd@sssb
2
3 Recovery Manager: Release 11.2.0.1.0 - Production on Sun Mar 2515:51:02
2012
4
5 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights
6 reserved.
7
8 connected to target database: SSPM (DBID=1624493265, not open)
9
10 RMAN> catalog start with '/u02/bkp';
Starting implicit crosscheck backup at 25-MAR-12
11 using target database control file instead of recovery catalog
12 allocated channel: ORA_DISK_1
13 channel ORA_DISK_1: SID=37 device type=DISK
14 Crosschecked 15 objects
15 Finished implicit crosscheck backup at 25-MAR-12
16
Starting implicit crosscheck copy at 25-MAR-12
17 using channel ORA_DISK_1
18 Crosschecked 2 objects
19 Finished implicit crosscheck copy at 25-MAR-12
20
21 searching for all files in the recovery area
cataloging files...
22 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_200.453.778846881
23 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_201.454.778846881
24 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_202.455.778846881
25 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_203.456.778846881
26 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_137.457.778846881
File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_204.458.778846881
27 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_205.459.778846883
28 .
29 .
30 .
.
31
32 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_856.947.778861691
33 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_858.949.778861709
34 File Name: +arch/SSSB/ARCHIVELOG/2012_03_25/thread_1_seq_857.950.778861719
35
36 searching for all files that match the pattern /u02/bkp
37
38 List of Files Unknown to the Database
=====================================
39 File Name: /u02/bkp/stnd_backp_10n6p3nl_1_1.bak
40 File Name: /u02/bkp/stnd_backp_11n6p3p4_1_1.bak
41
42 Do you really want to catalog the above files (enter YES or NO)? YES
43 cataloging files...
44 cataloging done
45
List of Cataloged Files
46 =======================
47 File Name: /u02/bkp/stnd_backp_10n6p3nl_1_1.bak
48 File Name: /u02/bkp/stnd_backp_11n6p3p4_1_1.bak
49
50
51
52
53
54
Step 7: Recover the standby database with the cataloged incremental backup pieces.
1 RMAN> recover database noredo;
2
Starting recover at 25-MAR-12
3 allocated channel: ORA_DISK_1
4 channel ORA_DISK_1: SID=32 device type=DISK
5 channel ORA_DISK_1: starting incremental datafile backup set restore
6 channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: +DATA/sssb/datafile/system.274.778865099
7 destination for restore of datafile 00002: +DATA/sssb/datafile/sysaux.275.778865193
8 destination for restore of datafile 00003: +DATA/sssb/datafile/undotbs1.276.778865259
9 destination for restore of datafile 00004: +DATA/sssb/datafile/users.277.778865273
10 channel ORA_DISK_1: reading from backup piece /u02/bkp/stnd_backp_10n6p3nl_1_1.bak
channel ORA_DISK_1: piece handle=/u02/bkp/stnd_backp_10n6p3nl_1_1.bak
11 tag=TAG20120325T154639
12 channel ORA_DISK_1: restored backup piece 1
13 channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
14
15 Finished recover at 25-MAR-12
16
17
Step 8 : Shutdown the physical standby database, start it in nomount stage and restore the standby
controlfile backup that we had taken from the primary database.
Standby Database:
1
2
RMAN> shutdown immediate
3
4 database dismounted
5 Oracle instance shut down
6
7 RMAN> startup nomount
8
9 connected to target database (not started)
10 Oracle instance started
11
Total System Global Area 659730432 bytes
12
13 Fixed Size 2216264 bytes
14 Variable Size 398462648 bytes
15 Database Buffers 255852544 bytes
16 Redo Buffers 3198976 bytes
17
RMAN> restore standby controlfile from '/u02/stnd_12n6p3qt_1_1.ctl';
18
19 Starting restore at 25-MAR-12
20 allocated channel: ORA_DISK_1
21 channel ORA_DISK_1: SID=21 device type=DISK
22
23 channel ORA_DISK_1: restoring control file
24 channel ORA_DISK_1: restore complete, elapsed time: 00:00:04
output file name=+DATA/sssb/controlfile/current.273.778864875
25 Finished restore at 25-MAR-12
26
27
Step 9: Shutdown the standby database and mount the standby database, so that the standby database
would be mounted with the new controlfile that was restored in the previous step.
Standby Database:
1
2 RMAN> <strong>shutdown immediate
3
4 Oracle instance shut down
5
RMAN> startup mount
6
7 connected to target database (not started)
8 Oracle instance started
9 database mounted
10
11 Total System Global Area 659730432 bytes
12
Fixed Size 2216264 bytes
13 Variable Size 398462648 bytes
14 Database Buffers 255852544 bytes
15 Redo Buffers 3198976 bytes
16
Step 10: If the datafile location of the primary and standby databases are different, then you need to
follow this step. If not, then proceed with Step 11.
The datafiles of my primary database are residing on the Diskgroup +DATA_NEW on the primary server
and the datafiles on the standby database are residing on the Diskgroup +DATA on the standby server,
the datafiles location are different.

Since, I have restored the standby controlfile backuppiece of my primary database on the standby
database (Step 7) and mounted the standby database, the standby database controlfile would now have
the locations of the datafiles recorded as available in the Primary database. So, we need to make the
standby controlfile understand that the datafiles location of the standby database are different from that of
the Primary database. For this, you need to catalog the datafile location of the standby database to its
controlfile as shown below.

Connect the standby database through RMAN and catalog the location of its datafiles and later switch
them.

Standby Database:
1 RMAN> catalog start with '+DATA/SSSB/DATAFILE';
2
searching for all files that match the pattern +DATA/SSSB/DATAFILE
3
4 List of Files Unknown to the Database
5 =====================================
6 File Name: +data/SSSB/DATAFILE/SYSTEM.274.778865099
7 File Name: +data/SSSB/DATAFILE/SYSAUX.275.778865193
8 File Name: +data/SSSB/DATAFILE/UNDOTBS1.276.778865259
File Name: +data/SSSB/DATAFILE/USERS.277.778865273
9
10 Do you really want to catalog the above files (enter YES or NO)? YES
11 cataloging files...
12 cataloging done
13
14 List of Cataloged Files
=======================
15 File Name: +data/SSSB/DATAFILE/SYSTEM.274.778865099
16 File Name: +data/SSSB/DATAFILE/SYSAUX.275.778865193
17 File Name: +data/SSSB/DATAFILE/UNDOTBS1.276.778865259
18 File Name: +data/SSSB/DATAFILE/USERS.277.778865273
19
RMAN> switch database to copy;
20
21 datafile 1 switched to datafile copy
22 "+DATA/sssb/datafile/system.274.778865099"
23 datafile 2 switched to datafile copy
24 "+DATA/sssb/datafile/sysaux.275.778865193"
25 datafile 3 switched to datafile copy
"+DATA/sssb/datafile/undotbs1.276.778865259"
26 datafile 4 switched to datafile copy
27 "+DATA/sssb/datafile/users.277.778865273"
28
29 RMAN>
30
Step 11: If the datafile locations of the primary and the standby databases are same, then there is no
necessity to perform the catalogging operation as done in the previous step.
On the standby database, start the Managed Recovery Process.

Standby Database:
1
2 SQL> alter database recover managed standby database disconnectfrom
session;
3
4 Database altered.
5
6 SQL> select process,status,sequence# from v$managed_standby;
7
8 PROCESS STATUS SEQUENCE#
9 --------- ------------ ----------
10 ARCH CONNECTED 0
ARCH CONNECTED 0
11 ARCH CONNECTED 0
12 ARCH CONNECTED 0
13 RFS IDLE 0
14 RFS IDLE 0
RFS IDLE 1010
15
RFS IDLE 0
16 MRP0 WAIT_FOR_LOG 0
17
18 9 rows selected.
19
Step 12: On the Primary database, check the Maximum Archivelog Sequence generated.
Primary Database:
1 SQL> select thread#,max(sequence#) from v$archived_log group bythread#;
2
3 THREAD# MAX(SEQUENCE#)
4 ------- ----------------
5 1 1009

Step 13: Check the maximum archivelog sequence that is applied on the Physical standby
database.
1 SQL> select thread#,max(sequence#) from v$archived_log where applied='YES' group by
thread#;
2
3 THREAD# MAX(SEQUENCE#)
4 ------- ----------------
5 1 1009
So, here we can see from Steps 12 and 13 that the maximum archivelog sequence generated on the
Primary database is sequence# 1009 and that applied on the Physical Standby Database is also 1009
which means that the Standby database is in sync with the Primary Database. You can check it out by
generating an archive sequence on the Primary database and check if its shipped and applied on the
standby database.

Primary Database:
1 SQL> alter system switch logfile;
2
3 System altered.
4
5 SQL> select thread#,max(sequence#) from v$archived_log group bythread#;
6
7 THREAD# MAX(SEQUENCE#)
------- ----------------
8
1 1010
9
Standby Database:
1 SQL> select thread#,max(sequence#) from v$archived_log where applied='YES' group by
thread#;
2
3 THREAD# MAX(SEQUENCE#)
4 ------- ----------------
5 1 1010
Now standby database is in sync with the Primary Database.

Here you go !