CRS is not coming up as CTSSD is in OFFLINE status (Doc To

ID 2051836.1) Bottom

In this Document

Symptoms
Cause
Solution
References

Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.
The following entries/logs are for two node cluster, node004, the bad node and node007 where
everything is running.

Symptoms
1. The "crsctl stat" on node004, the bad node shows that ctssd and rest of the higher stack, such
as evmd,crsd,ASM etc are in OFFLINE status :
------------------
# /u01/app/11.2.0.3/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE
ora.crf
1 ONLINE OFFLINE
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE node004
ora.cssdmonitor
1 ONLINE ONLINE node004
ora.ctssd
 1 ONLINE OFFLINE
ora.diskmon
1 ONLINE ONLINE node004
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE ONLINE node004
ora.gpnpd
1 ONLINE ONLINE node004
ora.mdnsd
1 ONLINE ONLINE node004
root@node004:~#
-----------------

2. alertnode004.log (node where CRS is not coming up) :

------------
2015-08-31 15:54:22.595:
[cssd(5694)]CRS-1713:CSSD daemon is started in clustered mode
2015-08-31 15:54:27.827:
[ohasd(5349)]CRS-2765:Resource 'ora.crf' has failed on server 'node004'.
.
2015-08-31 15:55:02.343:
[cssd(5694)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node004 node007 .
2015-08-31 15:55:04.934:
[ctssd(6094)]CRS-2407:The new Cluster Time Synchronization Service reference node is host
node007. <<=============== Reference or master node of CTSSD
2015-08-31 15:55:04.935:
[ctssd(6094)]CRS-2401:The Cluster Time Synchronization Service started on host node004.
2015-08-31 15:55:08.975:
[ctssd(6094)]CRS-2402:The Cluster Time Synchronization Service aborted on host node004.
Details at in /u01/app/11.2.0.3/grid/log/node004/ctssd/octssd.log.
---------------

3. NTPD is being used to sync up the clock on the cluster, and its running on the other(good)
node, node007(see above) which is the master of CTSSD. So CTSSD is running in OBSERVER
mode on node007.
The octssd.log of the bad node, node004 shows that it can't detect the vendor time sync software
as "ntp.conf" doesn't exist :
-----------------
2015-08-31 15:55:09.688: [ CTSS][1]Oracle Database CTSS Release 11.2.0.3.0 Production
Copyright 2006, 2011 Oracle. All rights reserved.

2015-08-31 15:55:10.653: [ CTSS][1]sclsctss_gvss8: Return [0] and NTP status [1].

2015-08-31 15:55:10.653: [ CTSS][1]ctss_check_vendor_sw: Vendor time sync software is not
detected. status [1]. <<=============== NTPD or vendor time sync software
was not detected.
-----------------

Cause
The ntp.conf was removed. The vendor time sync software, such as NTP should not be running
on Solaris zones or Linux/Exadata VMs, it should only be running on the global/control
domains. But the file, "ntp.conf" must exist on the zones/VMs otherwise CTSSD will try to run
in ACTIVE(time-sync) mode and the start up (of CTSSD and CRS) will fail due to the conflict
with the other node.

Solution
1. Please shutdown CRS using the force option as user root :
# /u01/app/11.2.0.3/grid/bin/crsctl stop crs -f

2. Remove all the sockets from /var/tmp/.oracle

3. Create "ntp.conf" and set the owner,group and permission correctly. The "ntp.conf" should be
located in "/etc" on Linux and under "/etc/inet" on Solaris, the "ls -l" output should show :
# ls -l /etc/ntp.conf
-rw-r----- 1 root root 2267 Jun 3 13:07 /etc/ntp.conf

4. Start CRS as root :

1. Please shutdown CRS using the force option as user root :
# /u01/app/11.2.0.3/grid/bin/crsctl stop crs -f

5. Entries similar to the following should be posted in the octssd.log :

-------------------
2015-08-31 18:06:06.584: [ CTSS][1]sclsctss_gvss1: NTP default config file
[/etc/inet/ntp.conf] found. <<==================== Detected the ntp.conf
2015-08-31 18:06:06.584: [ CTSS][1]sclsctss_gvss8: Return [0] and NTP status [2].
2015-08-31 18:06:06.584: [ CTSS][1]ctss_check_vendor_sw: Vendor time sync software is
detected. status [2].
2015-08-31 18:06:06.584: [ CTSS][1]ctss_check_vendor_sw: Ctssd is switching to observer
role <<==================== CTSSD is starting in OBSERVER
mode
------------------

The ntp.conf file exist, so CTSSD starting in OBSERVER mode, which is consistent with
node007, the other node.
Rest of the stack including ASM, CRSD etc and other resources should also come up, run
"/u01/app/11.2.0.3/grid/bin/crsctl stat crs -t" to confirm.