After some semi-successful patching of the Oracle Grid, and everything regarding the ASM instance was working ok, the database just plain refused to start and hook up to the ASM instance.
The database alert log showed us this:
Errors in file /oracle/11_4/base/diag/rdbms/vmrac/RAC1/trace/RAC1_asmb_8040.trc: ORA-15055: unable to connect to ASM instance ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Operation not permitted ORA-27302: failure occurred at: skgpwinit6
After a lot of looking, probing and frustration, it turned out the permissions on the oracle binary were wrong.
To solve this error, the following steps are needed:
Change to the GRID_HOME:
$ chmod 7521 oracle $ ls -l oracle -r-s-wS--t 1 oracle dba 239627031 Nov 11 11:35 oracle
Next change to the ORACLE_HOME/bin, and execute:
chmod 7521 oracle $ ls -l oracle -r-s-wS--t 1 oracle dba 239627031 Nov 11 11:35 oracle
Next, back to the GRID_HOME and execute:
(Could take a couple of minutes)
When the prompt returns, start the database:
SQL> startup ORACLE instance started. Total System Global Area 1653518336 bytes Fixed Size 2253784 bytes Variable Size 469765160 bytes Database Buffers 1174405120 bytes Redo Buffers 7094272 bytes Database mounted. Database opened. SQL>