CRS-4124: Oracle High Availability Services startup failed

While  installing Oracle11g(11.2.0.1) clusterware on RedHat Linux 6, i got the below error at time of running root.sh file. It is two node RAC and node names are RACNODE1 and RACNODE2.

CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

Here is the complete log.

[root@racnode1 grid]# sh root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 11:17:19: Parsing the host name
2015-09-28 11:17:19: Checking for super user privileges
2015-09-28 11:17:19: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_           params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

After googling, i found that 11.2.0.1 does not support RHL 6.0.

Here is workaround to fix the issue.

Step 1

Login to racnode1 and open the file  $GRID_HOME/crs/install/s_crsconfig_lib.pm
Add the following command before # Start OHASD

my $UPSTART_OHASD_SERVICE = "oracle-ohasd";
my $INITCTL = "/sbin/initctl";

($status, @output) = system_cmd_capture ("$INITCTL start $UPSTART_OHASD_SERVICE");
if (0 != $status)
{
error ("Failed to start $UPSTART_OHASD_SERVICE, error: $!");
return $FAILED;
}

Please see the screenshot.

Step 2

Create a file /etc/init/oracle-ohasd.conf with below content.

# Oracle OHASD startup
start on runlevel [35]
stop on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1

Step 3

Rollback the root.sh changes on racnode1. 

cd $GRID_HOME/crs/install

[root@racnode1 install]# ./roothas.pl -deconfig -force -verbose
2015-09-28 13:03:50: Checking for super user privileges
2015-09-28 13:03:50: User has super user privileges
2015-09-28 13:03:50: Parsing the host name
Using configuration parameter file: ./crsconfig_params
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
/grid/app/11.2.0/grid/bin/acfsdriverstate: line 51: /lib/acfstoolsdriver.sh: No such file or directory
/grid/app/11.2.0/grid/bin/acfsdriverstate: line 51: exec: /lib/acfstoolsdriver.sh: cannot execute: No such file or directory

Successfully deconfigured Oracle Restart stack

Step 4

Rerun the root.sh on racnode1

[root@racnode1 grid]# sh root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 13:04:35: Parsing the host name
2015-09-28 13:04:35: Checking for super user privileges
2015-09-28 13:04:35: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded

ASM created and started successfully.

DiskGroup DATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 2c2253ff0c544fdabf043b4752630d9e.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c2253ff0c544fdabf043b4752630d9e (ORCL:DISK1) [DATA]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode1'
CRS-2677: Stop of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode1'
CRS-2677: Stop of 'ora.gipcd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode1'
CRS-2677: Stop of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'racnode1'
CRS-2676: Start of 'ora.evmd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'racnode1'
CRS-2676: Start of 'ora.DATA.dg' on 'racnode1' succeeded

racnode1     2015/09/28 13:08:57     /grid/app/11.2.0/grid/cdata/racnode1/backup_20150928_130857.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /grid/app/oraInventory
'UpdateNodeList' was successful.

[root@racnode1 grid]#

Now racnode1 is successfully started OHASD service. Let us move on to racnode2 and run the root.sh

Step 5

[root@racnode2 oraInventory]#  sh orainstRoot.sh

Creating the Oracle inventory pointer file (/etc/oraInst.loc)

Changing permissions of /grid/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /grid/app/oraInventory to oinstall.

The execution of the script is complete.

[root@racnode2 oraInventory]#  cd /grid/app/11.2.0/grid

[root@racnode2 grid]# sh root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

   Copying dbhome to /usr/local/bin ...

   Copying oraenv to /usr/local/bin ...

   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

2015-09-28 11:30:55: Parsing the host name

2015-09-28 11:30:55: Checking for super user privileges

2015-09-28 11:30:55: User has super user privileges

Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_                                   params

Creating trace directory

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

  root wallet

  root wallet cert

  root cert export

  peer wallet

  profile reader wallet

  pa wallet

  peer wallet keys

  pa wallet keys

  peer cert request

  pa cert request

  peer cert

  pa cert

  peer root cert TP

  profile reader root cert TP

  pa root cert TP

  peer pa cert TP

  pa peer cert TP

  profile reader pa cert TP

  profile reader peer cert TP

  peer user cert

  pa user cert

Adding daemon to inittab

CRS-4124: Oracle High Availability Services startup failed.

CRS-4000: Command Start failed, or completed with errors.

ohasd failed to start: Inappropriate ioctl for device

ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

[root@racnode2 grid]#

[root@racnode2 grid]#

Opps.. RACNODE2 also has similar issue as we had in RACNODE1.

Step 6

Repeat the step1, step2 & step3 on RACNODE2. 

Run root.sh on racnode2.

The OHASD service again failed  and found the below info on log

2015-09-28 18:23:38.934: [    CSSD][1588156160]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 338904345, wrtcnt, 4437, LATS 10349194, lastSeqNo 4437, uniqueness 1443489200, timestamp 1443490040/4294892050

2015-09-28 18:23:38.934: [    CSSD][1588156160]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 338904344, wrtcnt, 733, LATS 10349194, lastSeqNo 733, uniqueness 0, timestamp 1443478983/10313614

2015-09-28 18:23:38.954: [    CLSF][1588156160]Closing handle:0x1a6ed70

2

Realized that firewall issue on private network which is blocking between racnode1 and racnode. 

Found the above info in $GRID_HOME/log/racnode2/cssd/ocssd.log

Execute the below command on  both node to fix the firewall issue on private network

service iptables stop

service ip6tables stop

chkconfig iptables off

chkconfig ip6tables off

Step 7

Run root.sh on racnode2 as below

[root@racnode2 ~]# pwd

/root

[root@racnode2 ~]# cd /grid/app/11.2.0/grid

[root@racnode2 grid]# pwd

/grid/app/11.2.0/grid

[root@racnode2 grid]# cd crs

[root@racnode2 crs]# cd install

[root@racnode2 install]# ./roothas.pl -deconfig -force -verbose

2015-09-28 18:41:44: Checking for super user privileges

2015-09-28 18:41:44: User has super user privileges

2015-09-28 18:41:44: Parsing the host name

Using configuration parameter file: ./crsconfig_params

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Delete failed, or completed with errors.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode2'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode2'

CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode2'

CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode2'

CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode2' succeeded

cd ..

cd CRS-2677: Stop of 'ora.ctssd' on 'racnode2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'racnode2'

..

CRS-2677: Stop of 'ora.mdnsd' on 'racnode2' succeeded

CRS-2677: Stop of 'ora.cssd' on 'racnode2' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode2'

CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode2'

pwd

CRS-2677: Stop of 'ora.gpnpd' on 'racnode2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode2'

CRS-2677: Stop of 'ora.gipcd' on 'racnode2' succeeded

CRS-2677: Stop of 'ora.diskmon' on 'racnode2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

ACFS-9201: Not Supported

Successfully deconfigured Oracle Restart stack

[root@racnode2 install]# cd ..

[root@racnode2 crs]# cd ..

[root@racnode2 grid]# pwd

/grid/app/11.2.0/grid

[root@racnode2 grid]# sh root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

2015-09-28 18:42:17: Parsing the host name

2015-09-28 18:42:17: Checking for super user privileges

2015-09-28 18:42:17: User has super user privileges

Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_params

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

Adding daemon to inittab

CRS-4123: Oracle High Availability Services has been started.

ohasd is starting

ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node racnode1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode2'

CRS-2676: Start of 'ora.mdnsd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'racnode2'

CRS-2676: Start of 'ora.gipcd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode2'

CRS-2676: Start of 'ora.gpnpd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode2'

CRS-2676: Start of 'ora.cssdmonitor' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'racnode2'

CRS-2672: Attempting to start 'ora.diskmon' on 'racnode2'

CRS-2676: Start of 'ora.diskmon' on 'racnode2' succeeded

CRS-2676: Start of 'ora.cssd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'racnode2'

CRS-2676: Start of 'ora.ctssd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'racnode2'

CRS-2676: Start of 'ora.asm' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'racnode2'

CRS-2676: Start of 'ora.crsd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.evmd' on 'racnode2'

CRS-2676: Start of 'ora.evmd' on 'racnode2' succeeded

racnode2     2015/09/28 18:44:22     /grid/app/11.2.0/grid/cdata/racnode2/backup_20150928_184422.olr

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Updating inventory properties for clusterware

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /grid/app/oraInventory

[root@racnode2 grid]# ps -ef|grep pmon

oracle   10130     1  0 18:43 ?        00:00:00 asm_pmon_+ASM2

root     10689  8438  0 18:44 pts/0    00:00:00 grep pmon

[root@racnode2 grid]#

Step 8

Validate both node and make sure all looks good.