When Adding A Node To RAC fails..Or.. “Try It Again, Sammy!”

Normally when adding a new node to a RAC Cluster is fairly simple, provided one keeps the OS the same, and specifically the Linux kernel EXACTLY the same..But as always, it’s different in the real world where reality is creating havoc..

So, as it happened in our case: the addNode failed..to be more specific: it failed multiple times on various points.

This is our story… (ok, watched to many episodes of Law&Order)..

The first issue was: the voting disk was not found..

Error:
[ CSSD][1686370048]clssnmvDiskVerify: discovered a potential voting file
[ CSSD][2286941952]clssnmvDiskVerify: TOC format mismatch expected(0x634c7373 0x546f636b), found(0x0000 0x0000)

ASM was able to find the disks, however not able to determine the voting disk. Strange issue, but after some searching it had to do with the newer kmod-oracleasm (kmod-oracleasm-2.0.8-13.el6_8.x86_64) we had installed, where we had a lower version on the other nodes. For details that lead to the solution: Doc ID 1994371.1

In short: add to the oracleasm config file (/etc/sysconfig/oracleasm) the following:

ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true 

(in the Doc ID it states to set this to ‘false’ but this was not logically for us..)

The above solved the  issue with the voting disk.

The second issue we ran into was: ASM instance refused to start..

Error:
GIMH: GIM-00104: Health check failed to connect to instance.
GIM-00090: OS-dependent operation:open failed with status: 2
GIM-00091: OS failure message: No such file or directory
GIM-00092: OS failure occurred at: sskgmsmr_7

When running the root.sh multiple times due to troubleshooting the first error, the file hc_* in $GRID_HOME/dbs/hc was created, but due to the voting disk error, never completed correctly.

Removing this file, and cleaning the whole oracle home of traces of this DB solved the issue, and enabled us to re-run the root.sh installation step to succes.

Advertisements

About GemsOfProgramming

Beeing a previously enthusiastic Java programmer, I rolled into the Oracle Database Administration world. It turned out I got a knack for this, and since approx. 2000 I'm a full time DBA. My experiences touches lot of Oracle products like Forms and Reports 9/10, JDAPI, Application Server, Weblogic Fusion and of course: Oracle Enterprise Databases, JavaFX, Swing and other Java components.
This entry was posted in ASM, Databases, RAC and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s