“Get here ASM!Intermediate!” … huh?…

This post is more  solution based, since the issue I had, was resolved faster than I had the time to analyze the cause. This is most of the times the case when a production database is involved, and yes, I know, first analyze the cause, then attack the cause so you know the solution is solid..well…try that when the IT Manager is screaming at you while you “carefully” analyze the situation..

but I digress…The symptoms where:

as user “system”:

SQL> select tablespace_name from dba_tablespaces;
select tablespace_name from dba_tablespaces
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kkopmCheckSmbUpdate:2], [], [], [], [], [], [], [], [], [], [], []

This…is bad. Also, ora-600 errors where thrown, those where the baseline is too old, and Oracle is flipping the finger..

I was in the lucky situation where I could bounce this database, but this did not help: One node (node02) would bounce perfectly, but Node01 kept complaining the ora.DATADG.dg was already online.

Trying to start the instance on Node01 resulted in the instance trying to start from a PFILE instead of the SPFILE, since the SPFILE was not available.

It took me a minute to run all of my default status checks, and lo-and-behold: we had the suspicious situation in our sights.

The  crsctl stat res -t command gave me this overview:

ora.DATADG.dg
ONLINE INTERMEDIATE Node01 CHECK TIMED OUT
ONLINE ONLINE                Node02

ora.FRADG.dg
ONLINE INTERMEDIATE Node01 CHECK TIMED OUT
ONLINE ONLINE                Node02

So. When googling this, it seems there is a bug 11807012 which seems to be causing this. Great. Now the solving part. a shutdown of this database was not a big issue at this time , since my experimental command of

crsctl stop resource ora.FRADG.dg

took down the database instance..not in a bad way, it was a neat shutdown..but still: a shutdown. We’re running a RAC for a reason, so this needs to be worked on..

But since it was down on both instances now, and the rest of the databases using this instance kept running (also with everything on Node01 in INTERMEDIATE state) I considered this a win.

Quickly fired the command

crsctl start resource ora.FRADG.dg

and the resource came back up, on BOTH nodes.

Now I was able to start the database correctly with:

srvctl start database -d RACDB

The solution for preventing the timeout in the future should be:

crsctl modify resource "resource_name" -attr "CHECK_TIMEOUT=132"

I need to figure out why the 132, and not another value, and if this also works for the situation above. I’m going to see if I can shutdown the specific resource on one node only, and keep the database online.

UPDATE 01:

Issue the command:

crsctl stop resource ora.FRADG.dg -n node01

to stop the resource on just one node. It may produce an error stating the resource is  not stopping due to it’s state but this can be ignored, since we don’t actually want to stop it, but it is not really doing anything anyway.

crstctl stat res -t gives us now:
ora.FRADG.dg
OFFLINE ONLINE       node01
ONLINE  ONLINE       node02

Now issue the command (omit the node parameter):

crsctl start resource ora.FRADG.dg

The message below will appear:

CRS-5702: Resource 'ora.FRADG.dg' is already running on 'node01'

The time-out state should now be solved, without downtime.

crstctl stat res -t gives us now:
ora.FRADG.dg
ONLINE  ONLINE       node01
ONLINE  ONLINE       node02

Advertisements

About GemsOfProgramming

Beeing a previously enthusiastic Java programmer, I rolled into the Oracle Database Administration world. It turned out I got a knack for this, and since approx. 2000 I'm a full time DBA. My experiences touches lot of Oracle products like Forms and Reports 9/10, JDAPI, Application Server, Weblogic Fusion and of course: Oracle Enterprise Databases, JavaFX, Swing and other Java components.
This entry was posted in Databases, Technical Stuff and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s