Question: I am getting the CRS-4535 error:

CRS-4535: Cannot communicate with Cluster

Ready Services
CRS-4529: Cluster Synchronization Services is
online
How do I fix this CRSerror?

Answer: The CRS-4535 Cannot communicate with Cluster Ready Services is a

typical issue that does happen for one of the following two reasons:

1) Networking issue between the RAC nodes (CRS not running)

2) Permission issue at the storage level

Check the crsd log under $GRID_HOME/log/hostname/crsd and check the alert log
for any additional details on the CRS-4535 error. Also, check to see if the CRS is
running on the node:

ps -ef |grep crsd.bin

Next, check the status of ora.crsd, if offline try starting the CRS with ora.crsd:

crsctl stat res -t -init

crsctl start res ora.crsd -init

1|Page
CRS-4535: Cannot communicate with Cluster Ready Services
We have SCOM configured for Oracle Servers to know if some service crashes, today I got the following
alert for one of our RAC node.
The OracleASMService+ASM1 service terminated unexpectedly.
I started investigations and tried to get the status by crsctl as below

D:\app\11.2.0.3\grid\BIN>crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
then I tried the same with -init option
D:\app\11.2.0.3\grid\BIN>crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE Instance Shutdown
ora.crf
1 ONLINE OFFLINE
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE or1
ora.cssdmonitor
1 ONLINE ONLINE or1
ora.ctssd
1 ONLINE INTERMEDIATE or1 CHECK TIMED OUT
ora.drivers.acfs

2|Page
 1 ONLINE ONLINE or1
ora.evmd
1 ONLINE ONLINE or1
ora.gipcd
1 ONLINE ONLINE or1
ora.gpnpd
1 ONLINE ONLINE or1
ora.mdnsd
1 ONLINE ONLINE or1

the result showed the ASM instance shutdown same as the SCOM alert but it could not be online
automatically.
Using the ASMCMD I started the ASM instance

and then tried to start the ora.crsd but it could not started and gave the following error.

D:\app\11.2.0.3\grid\BIN>crsctl start res ora.crsd -init

CRS-2800: Cannot start resource 'ora.ctssd' as it is already in the INTERMEDIATE state on server 'or1'
CRS-4000: Command Start failed, or completed with errors.

I tried to stop it but it could not be stopped even and gave the following errors.

D:\app\11.2.0.3\grid\BIN>crsctl stop res ora.crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.

I used the -init option and found the ora.ctssd is in INTERMEDIATE state thats why ora.crsd is not being
started.

3|Page
D:\app\11.2.0.3\grid\BIN>crsctl start res ora.crsd -init
CRS-2800: Cannot start resource 'ora.ctssd' as it is already in the INTERMEDIATE state on server 'or1'
CRS-4000: Command Start failed, or completed with errors.

So I stopped ora.ctssd first and then started it to come out of INTERMEDIATE state

D:\app\11.2.0.3\grid\BIN>crsctl stop res ora.ctssd -init

CRS-2673: Attempting to stop 'ora.ctssd' on 'or1'
CRS-2677: Stop of 'ora.ctssd' on 'or1' succeeded

D:\app\11.2.0.3\grid\BIN>crsctl start res ora.ctssd -init

CRS-2672: Attempting to start 'ora.ctssd' on 'or1'
CRS-2676: Start of 'ora.ctssd' on 'or1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'or1'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-01012: not logged on
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"D:\app\11.2.0.3\grid\log\or1\agent\ohasd\oraagent\oraagent.log".
CRS-2676: Start of 'ora.asm' on 'or1' succeeded

After this cluster services began to start normally and I could check the status as below.

D:\app\11.2.0.3\grid\BIN>crsctl stat res -t

--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DBDATA.dg
ONLINE ONLINE or1

4|Page
 ONLINE ONLINE or2
ora.DBFLASH.dg
ONLINE ONLINE or1
ONLINE ONLINE or2
ora.DGDUP.dg
ONLINE ONLINE or1
ONLINE ONLINE or2
ora.LISTENER.lsnr
ONLINE ONLINE or1
ONLINE ONLINE or2
ora.asm
ONLINE ONLINE or1 Started
ONLINE ONLINE or2 Started
ora.gsd
OFFLINE OFFLINE or1
OFFLINE OFFLINE or2
ora.net1.network
ONLINE ONLINE or1
ONLINE ONLINE or2
ora.ons
ONLINE ONLINE or1
ONLINE ONLINE or2
ora.registry.acfs
ONLINE ONLINE or1
ONLINE ONLINE or2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE or1
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE or2
ora.LISTENER_SCAN3.lsnr

5|Page
 1 ONLINE ONLINE or2
ora.cvu
1 ONLINE ONLINE or2
ora.rac.db
1 ONLINE OFFLINE Instance Shutdown,S
TARTING
2 ONLINE ONLINE or2 Open
ora.oc4j
1 ONLINE ONLINE or2
ora.or1.vip
1 ONLINE ONLINE or1
ora.or2.vip
1 ONLINE ONLINE or2
ora.scan1.vip
1 ONLINE ONLINE or1
ora.scan2.vip
1 ONLINE ONLINE or2
ora.scan3.vip
1 ONLINE ONLINE or2
ora.testrac.db
1 ONLINE OFFLINE Instance Shutdown,S
TARTING
2 ONLINE ONLINE or2 Open

Now everything was fine on the node but what caused all this.

From the crsd.log I found the below which is self explanatory

6|Page