Configure NetBackup For Oracle RAC..Or..”Back Me Up!”

When configuring Netbackup to backup a RAC database, we ran into the following error:

1/29/2016 3:28:02 PM - Info bphdb(pid=2717) INF - Database status is: SHUTDOWN
1/29/2016 3:28:02 PM - Info bphdb(pid=2717) INF - Database open mode is: NOT MOUNTED
1/29/2016 3:28:02 PM - Info bphdb(pid=2717) INF - ARCHIVE LOG mode is: UNKNOWN
1/29/2016 3:28:12 PM - Error bpbrm(pid=13636) from client <servername>: ERR - Script exited with status = -5407 <Database is in the wrong state (must be OPEN) for the requested action>
1/29/2016 3:28:12 PM - Error bpbrm(pid=13636) from client <servername>: ERR - bphdb exit status = 5407: Database is in the wrong state (must be OPEN) for the requested action
1/29/2016 3:28:14 PM - Info bphdb(pid=2717) done. status: 5407: Database is in the wrong state (must be OPEN) for the requested action
1/29/2016 3:28:14 PM - end writing
Database is in the wrong state (must be OPEN) for the requested action(5407)

In Netbackup the node was automagically picked up by the Netbackup client, which reads the /etc/oratab. In the oratab this database was known by it’s SCAN name. However when trying to connect, the connection to the node is random due to the nature of RAC. When the backup starts, the check is failing, resulting in a somewhat wrong error message of the client.

Now the question is: how to fix this?

Delete all the references in Netbackup of this database (to avoid confusion, since the listing of the /etc/oratab will be propagated again by the client to the server in due time).

Make sure the node/instance name is NOT in the /etc/oratab.

+ASM:/oracle/11.2.0.4/grid:N # line added by Agent
dummydb0:/oracle/11.2.0.4/base/db/dbhome_1:N # line added by Agent

In this case: the instance is dummydb01, the SCAN name is dummydb0.

In Netbackup:

sshot01.png

Add the instance MANUALLY by providing the information above. Make sure to provide the credentials.

sshot02.png

sshot03.png

On the oracle database server:

Make sure the added INSTANCE is reachable by adding this to the listener.ora.

$ vi listener.ora

SID_LIST_LISTENER=
(SID_LIST=
(SID_DESC=
(GLOBAL_DBNAME=dummy01)
(SID_NAME=dummydb01)
(ORACLE_HOME=/oracle/11.2.0.4/base/db/dbhome_1)
)
)

$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 29-FEB-2016 12:11:38

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 12-OCT-2015 02:51:52
Uptime 140 days 10 hr. 19 min. 45 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/11.2.0.4/grid/network/admin/listener.ora
Listener Log File /oracle/base/diag/tnslsnr/<servername>/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.xx)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.xx)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM", status READY, has 1 handler(s) for this service...
Instance "dummydb01", status READY, has 1 handler(s) for this service...
The command completed successfully

Configure the RMAN settings:

RMAN> show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name DUMMYDB0 are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE 'SBT_TAPE' TO '%F';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';

Make sure the settings in bold are set.

The PARALLELISM is set to 1, further testing should show if the backups are faster by a higher number. For now we just want to get this proces to work.

For testing purposes, create a manual backup in netbackup. When this is finished, this backup is visible in RMAN:

RMAN> list backup summary;
<..>
6545 B 0 A SBT_TAPE 29-FEB-16 1 1 NO TAG20160229T115522
6546 B F A SBT_TAPE 29-FEB-16 1 1 NO TAG20160229T141956
6547 B A A SBT_TAPE 29-FEB-16 1 1 NO TAG20160229T142121
6548 B F A SBT_TAPE 29-FEB-16 1 1 NO TAG20160229T142157
6549 B F A SBT_TAPE 29-FEB-16 1 1 NO TAG20160229T142314
6550 B F A SBT_TAPE 29-FEB-16 1 1 NO TAG20160229T142351

RMAN>

Validate the backup (from the “tape”):

RMAN> run {
allocate channel t1 type 'SBT_TAPE' parms 'ENV=(NB_ORA_CLIENT=<nodeName),SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
send 'NB_ORA_SERV=<netbackup_server>,NB_ORA_CLIENT=<nodeName>';
restore database validate;
release channel t1;
}

using target database control file instead of recovery catalog
allocated channel: t1
channel t1: SID=606 instance=dummbydb01 device type=SBT_TAPE
channel t1: Veritas NetBackup for Oracle - Release 7.6 (2013111313)

sent command to channel: t1

Starting restore at 29-FEB-16

channel t1: starting validation of datafile backup set
channel t1: reading from backup piece bk_dDUMMYDB01_u9sqv4u1b_s8508_p1_t905082923

For more information about restoring, see the other post on my blog!

Advertisements

About GemsOfProgramming

Beeing a previously enthusiastic Java programmer, I rolled into the Oracle Database Administration world. It turned out I got a knack for this, and since approx. 2000 I'm a full time DBA. My experiences touches lot of Oracle products like Forms and Reports 9/10, JDAPI, Application Server, Weblogic Fusion and of course: Oracle Enterprise Databases, JavaFX, Swing and other Java components.
This entry was posted in Databases and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s