Thursday, October 1, 2015

CRS-4124: Oracle High Availability Services startup failed

While  installing Oracle11g(11.2.0.1) clusterware on RedHat Linux 6, i got the below error at time of running root.sh file. It is two node RAC and node names are RACNODE1 and RACNODE2.

CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

Here is the complete log.
[root@racnode1 grid]# sh root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 11:17:19: Parsing the host name
2015-09-28 11:17:19: Checking for super user privileges
2015-09-28 11:17:19: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_           params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.
After googling, i found that 11.2.0.1 does not support RHL 6.0.

Here is workaround to fix the issue.

Step 1

Login to racnode1 and open the file  $GRID_HOME/crs/install/s_crsconfig_lib.pm
Add the following command before # Start OHASD

my $UPSTART_OHASD_SERVICE = "oracle-ohasd";
my $INITCTL = "/sbin/initctl";

($status, @output) = system_cmd_capture ("$INITCTL start $UPSTART_OHASD_SERVICE");
if (0 != $status)
{
error ("Failed to start $UPSTART_OHASD_SERVICE, error: $!");
return $FAILED;
}

Please see the screenshot.





Step 2

Create a file /etc/init/oracle-ohasd.conf with below content.

# Oracle OHASD startup
start on runlevel [35]
stop on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1













Step 3

Rollback the root.sh changes on racnode1. 

cd $GRID_HOME/crs/install


[root@racnode1 install]# ./roothas.pl -deconfig -force -verbose
2015-09-28 13:03:50: Checking for super user privileges
2015-09-28 13:03:50: User has super user privileges
2015-09-28 13:03:50: Parsing the host name
Using configuration parameter file: ./crsconfig_params
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
/grid/app/11.2.0/grid/bin/acfsdriverstate: line 51: /lib/acfstoolsdriver.sh: No such file or directory
/grid/app/11.2.0/grid/bin/acfsdriverstate: line 51: exec: /lib/acfstoolsdriver.sh: cannot execute: No such file or directory

Successfully deconfigured Oracle Restart stack

Step 4

Rerun the root.sh on racnode1

[root@racnode1 grid]# sh root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 13:04:35: Parsing the host name
2015-09-28 13:04:35: Checking for super user privileges
2015-09-28 13:04:35: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded

ASM created and started successfully.

DiskGroup DATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 2c2253ff0c544fdabf043b4752630d9e.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c2253ff0c544fdabf043b4752630d9e (ORCL:DISK1) [DATA]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode1'
CRS-2677: Stop of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode1'
CRS-2677: Stop of 'ora.gipcd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode1'
CRS-2677: Stop of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'racnode1'
CRS-2676: Start of 'ora.evmd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'racnode1'
CRS-2676: Start of 'ora.DATA.dg' on 'racnode1' succeeded

racnode1     2015/09/28 13:08:57     /grid/app/11.2.0/grid/cdata/racnode1/backup_20150928_130857.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /grid/app/oraInventory
'UpdateNodeList' was successful.

[root@racnode1 grid]#

Now racnode1 is successfully started OHASD service. Let us move on to racnode2 and run the root.sh

Step 5

[root@racnode2 oraInventory]#  sh orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /grid/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /grid/app/oraInventory to oinstall.
The execution of the script is complete.
[root@racnode2 oraInventory]#  cd /grid/app/11.2.0/grid
[root@racnode2 grid]# sh root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 11:30:55: Parsing the host name
2015-09-28 11:30:55: Checking for super user privileges
2015-09-28 11:30:55: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_                                   params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.
[root@racnode2 grid]#
[root@racnode2 grid]#

Opps.. RACNODE2 also has similar issue as we had in RACNODE1.

Step 6

Repeat the step1, step2 & step3 on RACNODE2. 
Run root.sh on racnode2.

The OHASD service again failed  and found the below info on log

2015-09-28 18:23:38.934: [    CSSD][1588156160]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 338904345, wrtcnt, 4437, LATS 10349194, lastSeqNo 4437, uniqueness 1443489200, timestamp 1443490040/4294892050
2015-09-28 18:23:38.934: [    CSSD][1588156160]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 338904344, wrtcnt, 733, LATS 10349194, lastSeqNo 733, uniqueness 0, timestamp 1443478983/10313614
2015-09-28 18:23:38.954: [    CLSF][1588156160]Closing handle:0x1a6ed70
2

Realized that firewall issue on private network which is blocking between racnode1 and racnode. 

Found the above info in $GRID_HOME/log/racnode2/cssd/ocssd.log

Execute the below command on  both node to fix the firewall issue on private network

service iptables stop
service ip6tables stop
chkconfig iptables off
chkconfig ip6tables off

Step 7

Run root.sh on racnode2 as below

[root@racnode2 ~]# pwd
/root
[root@racnode2 ~]# cd /grid/app/11.2.0/grid
[root@racnode2 grid]# pwd
/grid/app/11.2.0/grid
[root@racnode2 grid]# cd crs
[root@racnode2 crs]# cd install
[root@racnode2 install]# ./roothas.pl -deconfig -force -verbose
2015-09-28 18:41:44: Checking for super user privileges
2015-09-28 18:41:44: User has super user privileges
2015-09-28 18:41:44: Parsing the host name
Using configuration parameter file: ./crsconfig_params
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode2' succeeded
cd ..
cd CRS-2677: Stop of 'ora.ctssd' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode2'
..
CRS-2677: Stop of 'ora.mdnsd' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode2'
pwd
CRS-2677: Stop of 'ora.gpnpd' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode2'
CRS-2677: Stop of 'ora.gipcd' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'racnode2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

ACFS-9201: Not Supported
Successfully deconfigured Oracle Restart stack
[root@racnode2 install]# cd ..
[root@racnode2 crs]# cd ..
[root@racnode2 grid]# pwd
/grid/app/11.2.0/grid
[root@racnode2 grid]# sh root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 18:42:17: Parsing the host name
2015-09-28 18:42:17: Checking for super user privileges
2015-09-28 18:42:17: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node racnode1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode2'
CRS-2676: Start of 'ora.mdnsd' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode2'
CRS-2676: Start of 'ora.gipcd' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode2'
CRS-2676: Start of 'ora.gpnpd' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode2'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode2'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode2'
CRS-2676: Start of 'ora.diskmon' on 'racnode2' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode2'
CRS-2676: Start of 'ora.ctssd' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode2'
CRS-2676: Start of 'ora.asm' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racnode2'
CRS-2676: Start of 'ora.crsd' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'racnode2'
CRS-2676: Start of 'ora.evmd' on 'racnode2' succeeded

racnode2     2015/09/28 18:44:22     /grid/app/11.2.0/grid/cdata/racnode2/backup_20150928_184422.olr
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /grid/app/oraInventory
[root@racnode2 grid]# ps -ef|grep pmon
oracle   10130     1  0 18:43 ?        00:00:00 asm_pmon_+ASM2
root     10689  8438  0 18:44 pts/0    00:00:00 grep pmon
[root@racnode2 grid]#

Step 8

Validate both node and make sure all looks good.


4 comments:

Rakesh Sayini said...

Got the same Issue due to using

Oracle 11.2.0.1 on Oracle Linux 6.7 Two Instances
Oracle 11.2.0.1 on Oracle Linux 6.3 Single Instance

Applied the above steps and it worked fine. But create separate user for GRID & ORACLE. Then only it will help.





Govind said...

Thanks Rakesh for the update! Good to know!

I am glad my blog helped.

Firas Tahir said...

Thank you so much Sir , after struggling for six hours I came across your blog and the issue is solved in 10 minutes.

Govind said...

Good to know Firas Tahir!! I am glad, my post is helping...