8/24/2015

Document220970.1

saktivel(Available)

PowerViewisOff
Dashboard

Knowledge

ServiceRequests

Patches&Updates

Community

Certifications

Systems

(0)

ContactUs

Help

Collector
GiveFeedback...

RAC:FrequentlyAskedQuestions(DocID220970.1)

APPLIESTO:

ToBottom

DocumentDetails
Type:
Status:
LastMajorUpdate:
LastUpdate:

FAQ
PUBLISHED
20Jun2014
15Jun2015

RelatedProducts

QUESTIONSANDANSWERS

OracleDatabaseEnterpriseEdition
InformationCenters

IndexofOracleDatabaseInformationCenters
[1568043.2]

GeneralRAC

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

FrequentlyAskedQuestionsforRealApplicationClustersandGridInfrastructure.

WhatkindofHWcomponentsdoyourecommendfortheinterconnect?
Isrcpand/orrshrequiredfornormalOracleRACoperation?
WhatisCacheFusionandhowdoesthisaffectapplications?
Isitdifficulttotransition(migrate)fromSingleInstancetoOracleRAC?
WhatarethedependenciesbetweenOCFSandASMinOracleDatabase10g?
DowehavetohaveOracleDatabaseonallnodes?
WhatsoftwareisnecessaryforOracleRAC?DoesithaveaseparateinstallationCDtoorder?
WhatApplicationDesignconsiderationsshouldIbeawareofwhenmovingtoOracleRAC?
ShouldtheSCSI3reservationbitbesetforourOracleClusterwareonlyinstallation?
DoesWeblogic(WLS)supportServices,FAN/FCF,andtheLoadBalancingAdvisory(LBA)withOracleRAC?
AreblockdevicessupportedforOCR,VotingDisks,andASMdevices?
DoIneedtorelinktheOracleClusterware/GridInfrastructurehomeafteranOSupgrade?
ArethereanyissuesfortheinterconnectwhensharingthesameswitchasthepublicnetworkbyusingVLANtoseparatethenetwork?
HowtouseVLANsforOracleRACandtheOracleClusterwareInterconnect?
IsitagoodideatoaddantivirussoftwaretomyRACcluster?
IsthereaneedtoreniceLMSprocessesinOracleRAC10gRelease2?
HowcanaNASstoragevendorcertifytheirstoragesolutionforOracleRAC?
MycustomerhasanXAApplicationwithaOracleRACDatabase,canIdoLoadBalancingacrosstheOracleRACinstances?
HowdoIdeterminewhichnodeintheclusteristhe"Master"node?
IsInfinibandsupportedfortheOracleRACinterconnect?
WherecanIfindalistofsupportedsolutionstoensureNICavailability/redundancy(fortheinterconnect)perplatform?
DoesOraclesupportOracleRACinSolariscontainers(a.k.a.SolarisZones)?
WhatisSCAN?
IgetthefollowingerrorstartingmyOracleRACdatabase,whatdoIdo?
WARNING:Noclusterinterconnecthasbeenspecified.
IfmyOCRandVotingDisksareinASM,canIshutdowntheASMinstance?
Ihavechangedmyspfilewithaltersystemsetparameter_name=....scope=spfile.ThespfileisonASMstorageandthedatabasewillnotstart.
WhatcombinationsofOracleClusterware,OracleRACandASMversionscanIuse?
Ihada3nodeOracleRAC.Oneofthenodeshadtobecompletelyrebuiltasaresultofaproblem.Astherearenobackups,Whatistheproperproceduretoremovethe3rdnodefrom
theclustersoitcanbeaddedbackin?
ArejumboframessupportedfortheRACinterconnect?
AreSunLogicalDomains(ldoms)supportedwithRAC?

Yes
No

OracleDatabaseEnterpriseEditionVersion9.2.0.1to12.1.0.1[Release9.2to12.1]
Informationinthisdocumentappliestoanyplatform.

PURPOSE

Wasthisdocumenthelpful?

InformationCenter:OverviewofDatabase
SecurityProducts[1548952.2]
InformationCenter:OverviewDatabase
Server/ClientInstallationandUpgrade/Migration
[1351022.2]

DocumentReferences

UsingIBMGPFS2.3withOracleRealApplication
Clusters10g[444134.1]
NewPartitionsinWindows2003RAC
EnvironmentsNotVisibleonRemoteNodes
[454607.1]
RACDDTUserGuide[301138.1]
Whataretheeffectsofchangingthesystem
clockonanOracleServerinstance?[77370.1]
Init.oraParameter"CLUSTER_INTERCONNECTS"
ReferenceNote[151051.1]
ShowMore

RecentlyViewed

HowtoresolveMRPstuckissuesonaphysical
standbydatabase?[1221163.1]
ORA16166LGWRtimedoutonNetworkServer
0duetobufferfullcondition[247129.1]
LTOMTheOnBoardMonitorUserGuide
[352363.1]
HowToConnectUsingASqlplusPreliminary
Connection[986640.1]
1/72

8/24/2015

Document220970.1

AreSunLogicalDomains(ldoms)supportedwithRAC?
WhatisStandardEditionOracleRAC?
WheredoIfindOracleClusterwarebinariesandASMbinarieswithOracleDatabase11gRelease2?
Ihavethe11.2GridInfrastructureinstalledandnowIwanttoinstallanearlierversionofOracleDatabase(11.1or10.2),isthissupported?
IgetanerrorwithDBCAfrom10.2or11.1afterIhaveinstalledthe11.2GridInfrastructure?
CanIuseiSCSIstoragewithmyOracleRACcluster?
Whatwouldyourecommendtocustomer,OracleClusterwareorVendorClusterware(I.E.HPServiceGuard,HACMP,SunCluster,Veritasetc.)withOracleRealApplicationClusters?
WhenconfiguringtheNICcardsandswitchforaGigEInterconnectshoulditbesettoFULLorHalfduplexinOracleRAC?
CanIuseOracleRACinadistributedtransactionprocessingenvironment?
CanIrunOracle9iRACandOracleRAC10ginthesamecluster?
WhatstorageissupportedwithStandardEditionOracleRAC?
WhataretherestrictionsontheSIDwithanOracleRACdatabase?Isitlimitedto5characters?
DoesOracleClusterwareorOracleRealApplicationClusterssupportheterogeneousplatforms?
IwanttouserconfigtoconvertasingleinstancetoOracleRACbutIamusingrawdevicesinOracleRAC.DoesrconfigsupportRAW?
HowmanyNICsdoIneedtoimplementOracleRAC?
CanwedesignatetheplaceofarchivelogsonbothASMdiskandregularfilesystem,whenweuseSERAC?
CanmycustomeruseVeritasAgentstomanagetheirOracleRACdatabaseonUnixwithSFRACinstalled?
CanIrunmorethanoneclustereddatabaseonasingleOracleRACcluster?
CanIrunOracleRAC10gwithOracleRAC11g?
CanIhavemultiplepublicnetworksaccessingmyOracleRAC?
IcouldnotgettheuserequivalencechecktoworkonmySolaris10serverwhentryingtoinstall10.2.0.1OracleClusterware.Theinstallranfinewithoutissue.<<Message:Result:
Userequivalencecheckfailedforuser"oracle".>>
IsitsupportedtoinstallOracleClusterwareandOracleRACasdifferentusers?
WhydoestheNOACattributeneedtobesetonNFSmountedRACBinaries?
WeareusingTransparentDataEncryption(TDE).
Wecreateawalletonnode1andcopytonodes2&3.Openthewalletandweareabletoselectencrypteddataonallthreenodes.
Now,wewanttoREKEYtheMASTERKEY.Whatdowehavetodo?
HowdoIcheckfornetworkproblemsonmyinterconect?
TheVeritasinstallationdocumentonpage219asksforsettingLD_LIBRARY_PATH_64.ShouldIremovethis?
WhydoesnetcaalwayscreatesthelistenerwhichlistenstopublicipandnotVIPonly?
DoeschanginguidorgidoftheOracleUseraffectOracleClusterware?
Canweoutputthebackupsetontoregularfilesystemdirectly(notontoflashrecoveryarea)usingRMANcommand,whenweuseSERAC?

HowtoCollectDiagnosticsforDatabase
HangingIssues[452358.1]
ShowMore

RACAssistance
HowdoIuseDBCAinsilentmodetosetupRACandASM?

HighAvailability
CanRMANbackupOracleRealApplicationClusterdatabases?
IamreceivinganORA29740error.WhatshouldIdo?
DoesOraclesupportrollingupgradesinacluster?
IsOracleApplicationServerintegratedwithFANandFCF?
HowdoIconfigureFCFwithBPELsoIcanuseOracleRAC10ginthebackend?
WherecanIfindmoreinformationoncluster_interconnects?
HowdoesOCRmirrorwork?WhathappensifmyOCRislost/corrupt?
IfIchangemyclusterconfiguration,doIneedtoupdatetheONSconfigurationonmymiddletier?
WhydowehaveaVirtualIP(VIP)inOracleRAC10gor11g?Whydoesitjustreturnadeadconnectionwhenitsprimarynodefails?
WhatdotheVIPresourcesdooncetheydetectanodehasfailed/gonedown?AretheVIPsautomaticallyacquired,andpublished,orismanualinterventionrequired?AreVIPs
mandatory?
IfIuseServiceswithOracleRAC,doIstillneedtosetupLoadBalancing?
Howcanacustomermaskthechangeintheirclustereddatabaseconfigurationfromtheirclientorapplication?(I.E.SoIdonothavetochangetheconnectionstringwhenIaddanode
totheOracleRACdatabase)
AfterexecutingDBMS_SERVICE.START_SERVICE,theserviceresourceremainsinanOFFLINEstatuswhenIdisplaytheclusterresourcestatus.Isthatexpectedbehaviour?
WhataremyoptionsforloadbalancingwithOracleRAC?WhydoIgetanunevennumberofconnectionsonmyinstances?
WhatdoIdoifIamgettinghandshakefailedmessagesinmyONS.LOGfileeveryminute?
WhatshouldIdotomakemyOracleRACdeploymenthighlyavailable?
CanourOracleRAC10gVIPfailoverfromNICtoNICaswellasfromnodetonode?
IsthereawaytoprovideorconfigureHAfortheinterconnectusingInfinibandonAIX?
Iamusingsharedserviceswhichthefollowingsetininit.oraSQL>showparametersdispatchers=(protocol=TCP)(listener=listeners_nl01)(con=500)(serv=oltp).Istoppedmyservice
withsrvctlstopservicebutitisstillregisteredwiththelistenerandacceptingconnections.Isthisexpected?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

2/72

8/24/2015

Document220970.1

withsrvctlstopservicebutitisstillregisteredwiththelistenerandacceptingconnections.Isthisexpected?
IsitpossibletouseSVRCTLstartdatabasewithauseraccountotherthanoracle(thatisotherthantheowneroftheoraclesoftware)?
Withthreeprimaryloadbalancingoptions(clientsideconnecttimeLB,serversideconnecttimeLB,andtheruntimeconnectionloadbalancing)IsitfairtosayRuntimeConnectionLoad
BalancingistheonlyoptiontoleverageFANup/downevents?
WhatisServersideTransparentApplicationFailover(TAF)andhowdoIuseit?
WhatisCLB_GOALandhowshouldIsetit?
WhatdoestheVirtualIPservicedo?Iunderstanditisforfailoverbutdoweneedaseparatenetworkcard?Canweusetheexistingprivate/publiccards?Whatwouldhappenifweused
thepublicip?

HighAvailabilityFAN/FCF
IwanttoconfigureasecureenvironmentforONSsohaveaddedaWallethoweverIamseeingerrors(SSLhandshakefailed)afteraddingthewallet?
DoIneedtoinstalltheONSonallmymidtierservesinordertoenableJDBCFastConnectionFailover(FCF)?
WillFAN/FCFworkwiththedefaultdatabaseservice?
WillFANworkwithSQLPlus?
WhyamIseeingthefollowingwarningsinmylistener.logformyRAC10genvironment?
WARNING:Subscriptionfornodedowneventstillpending
CanIusethe10.2JDBCdriverwith10.1databaseforFCF?
WhatclientsprovideintegrationwithFANthroughFCF?
CanIuseTAFandFAN/FCF?
HowdoesthedatasourcepropertiesinitialLimit,minLimit,andmaxLimitaffectFastConnectionFailoverprocessingwithJDBC?
WillFAN/OCIworkwithInstantClient?
WhattypeofcallbacksaresupportedwithOCIwhenusingFAN/FCF?
DoesFCFforOCIreacttoFANHAUPevents?
CanIuseFAN/OCIwithPro*C?
DoIhavetolinkmyOCIapplicationwithathreadlibrary?Why?

Scalability
Iamseeingthewaitevents'gesremotemessage','gcsremotemessage',and/or'gcsforaction'.WhatshouldIdoaboutthese?
WhatarethechangesinmemoryrequirementsfrommovingfromsingleinstancetoRAC?
HowcanIvalidatethescalabilityofmysharedstorage?(TightlyrelatedtoRAC/Applicationscalability)
HowmanynodesaresupportedorcanbeusedinanOracleRACDatabase?
WhataremyoptionsforsettingtheLoadBalancingAdvisoryGOALonaService?
WilladdinganewinstancetomyOracleRACdatabase(newnodetothecluster)allowmetoscaletheworkload?
HowdoIchangemyVeritasSFRACinstallationtouseUDPinsteadofLLT?
CanIhavedifferentserversinmyOracleRAC?Cantheybefromdifferentvendors?Cantheybedifferentsizes?
WhatdoIdoifIseeGCCRBLOCKLOSTinmytop5TimedEventsinmyAWRReport?
AcustomeriscurrentlyusingRACina2nodeenvironment.Howshouldonereviewtheabilitytoscaleoutto4,6,8orevenmorenodes?Whatshouldtherequirementsofascaleout
test?
WhatistheLoadBalancingAdvisory?
HowdoIenabletheloadbalancingadvisory?
HowdoImeasurethebandwidthutilizationofmyNICormyinterconnect?
DoesDatabaseblocksizeortablespaceblocksizeaffecthowthedataispassedacrosstheinterconnect?
WhatisRuntimeConnectionLoadBalancing?

Manageability
HowshouldIdealwithspacemanagement?DoIneedtosetfreelistsandfreelistgroups?
IwasinstallingOracle9iRACandmyOraclefilesdidnotgetcopiedtotheremotenode(s).Whatwentwrong?
HowdoIstoptheGSD?
HowdoIdeterminewhetherornotanOneOffpatchis"rollingupgradeable"?
DoesOracleRACworkwithNTP(NetworkTimeProtocol)?
IfIamusingVendorClusterwaresuchasVeritas,IBM,SunorHP,doIstillneedOracleClusterwaretorunOracleRAC10gorOracleRAC11g?
HowisOracleEnterpriseManagerintegratedwiththeOracleRAC11gRelease2stack?
WhatstorageoptionshouldIuseforOracleRAConLinux?ASM/OCFS/RawDevices/BlockDevices/Ext3?
WhataretheimplicationsofusingsrvctldisableforaninstanceinmyOracleRACcluster?IwanttohaveitavailabletostartifIneeditbutatthistimetonotwanttorunthisextra
instanceforthisdatabase.
Ifusingplsqlnativecode,theplsql_native_library_dirneedstobedefined.InanOracleRACenvironement,mustthedirectorybeinthesharedstorage?
WhatisthepurposeofthegsdserviceinOracle9iRAC?
HowdoIidentifywhichnodewasusedtoinstalltheclustersoftwareand/ordatabasesoftware?

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

3/72

8/24/2015

Document220970.1

HowdoIidentifywhichnodewasusedtoinstalltheclustersoftwareand/ordatabasesoftware?
AretheOracleClusterwarebundlepatchescumulative,dotheyconflictwithoneanother?
Ihaveaddedasecondnetworktomycluster,canIloadbalancemyusersacrossthisnetwork?
Srvctlcannotstartinstance,IgetthefollowingerrorPRKP1001CRS0215,howeversqlpluscanstartitonbothnodes?Whatistheproblem?
WhenIlookatALL_SERVICESviewinmydatabaseIseeservicesIdidnotcreate,whataretheyfor?
Ihave2clustersnamed"crs"(thedefault),howdoIgetGridControltorecognizethemastargets?
Ifoundin10.2thattheEM"ConverttoClusterDatabase"wizardwouldalwaysfalloveronthelaststepwhereitrunsemcaandneedstologintothenewclusterdatabaseasdbsnmpto
createtheclusterdatabasetargetsetc.Ichangedthepasswordforthedbsnmpaccounttobedbsnmp(sameasusername)anditworkedOK.Isthisaknownissue?
WhatistheClusterVerificationUtiltiy(cluvfy)?
WhatversionsofthedatabasecanIusetheclusterverificationutility(cluvfy)with?

PlatformSpecific
HowmanynodescanonehaveinanHPUX/Solaris/AIX/Windows/Linuxcluster?
IscrossovercablesupportedasaninterconnectwithRAConanyplatform?
IsitpossibletorunOracleRAConlogicalpartitions(i.e.LPARs)orvirtualseparateservers.
HowdoIcheckOracleRACcertification?
WhatisOracle'spositionwithrespecttosupportingRAConPolyserveCFS?
CantheOracleDatabaseConfigurationAssistant(DBCA)beusedtocreateadatabasewithVeritasDBE/AC3.5?
IsOracleDatabaseonVMwaresupport?IsOracleRAConVMwaresupported?
IsVeritasStorageFoundationsupportedwithOracleRAC?

PlatformSpecificLinux
Afterinstallingpatchset9013andpatch_2313680onLinux,thestartupwasveryslow
Isthereaclusterfilesystem(CFS)AvailableforLinux?
IsthehangchecktimerstillneededwithOracleRAC10gand11gR1?
OracleClusterwarefailstostartafterarebootduetopermissionsonrawdevicesrevertingtodefaultvalues.HowdoIfixthis?
HowdoIconfigurerawdevicesinordertoinstallOracleClusterware10gonRHEL5orOEL5?
CandifferentreleasesofOracleRACbeinstalledandrunonthesamephysicalLinuxcluster?
Is3rdPartyClusterwaresupportedonLinuxsuchasVeritasorRedhat?
Acustomerinstalled10gRelease2onLinuxRH4Update2,2.6.922.ELsmp#1SMPx86_64GNU/Linux,andgottheerrorErrorininvokingtarget'all_no_orcl'.Customerignoredthe
errorandtheinstallsucceededwithoutanyothererrorsandoracleapparentlyworkedfine.Whatshouldtheydo?
IsOCFS2certifiedwithOracleRAC10g?
HowdoIconfiguremyRACClustertousetheRDSInfiniband?
CustomerdidnotloadthehangchecktimerbeforeinstallingRAC,Canthecustomerjustloadthehangchecktimer?
Howtoreorderorrenamelogicalnetworkinterface(NIC)namesinLinux
AreRedHatGFSandGULMcertifiedforDLM?
Mycustomerisabouttoinstall10202clusterwereonnewLinuxmachinges.Heisgetting"NoORACMrunning"errorwhenrunrootpre.shandexited?Shouldheworryaboutthis
message?
HowtoconfigurebondingonSuseSLES8.
HowtoconfigurebondingonSuseSLES9.

PlatformSpecificSolaris
InSolaris10,doweneedSunClustertoprovideredundancyfortheinterconnectandmultipleswitches?
CanIconfigureIPMPinActie/Activetoincreasebandwidthofmyinterconnect?
DoesSunSolarishaveamultipathingsolution?

PlatformSpecificHPUX
CanIconfigureHP'sAutoportaggregationforNICBondingaftertheinstall?(i.e.notpresentbeforehand)
IsHMPsupportedwithOracleRAC10gorOracleRAC11gonallHPplatforms?

PlatformSpecificWindows
DoestheOracleClusterFileSystem(OCFS)supportnetworkaccessthroughNFSorWindowsNetworkShares?
WhyshouldIuseRACOneNodeinsteadofOracleFailSafeonWindows?
WhenrunningOracleRAConWindows2003,whatistherecommendedOSlevel?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

4/72

8/24/2015

Document220970.1

CanIrunmyOracle9iRACandOracleRAC10gonthesameWindowscluster?
WhenusingMSVSSonWindowswithOracleRAC,doIneedtoruntheVSSoneachnodewhereIhaveanOracleRACinstance?
WhatdoIdowhenIgetanORA01031errorloggingintotheASMinstance?
TheOracleCRServicedoesnotstartwithmywindowsOracleRACimplementation,whatdoIdo?
HowdoIverifythatHostBusAdapterNodeLocalCachinghasbeendisabledforthedisksIwillbeusinginmyRACcluster?
Mycustomerhasafailsafeclusterinstalled,whatarethebenefitsofmovingtheirsystemtoRAC?
MycustomerwantstounderstandwhattypeofdiskcachingtheycanusewiththeirWindowsRACCluster,theinstallguidetellsthemtodisablediskcaching?

PlatformSpecificIBMAIX
DoIneedHACMP/GPFStostoremyOCR/Votingfileonashareddevice.
IsVIOsupportedwithRAConIBMAIX?
IsHACMPneededforRAConAIX5.2usingGPFSfilesystem?

PlatformSpecificIBMz/OS(Mainframe)
CanIrunOracleRAC10gonmyIBMMainframeSysplexenvironment(z/OS)?

OtherApplications&RAC
CanIuseOracleClusterwareforfailoveroftheSAPEnqueueandVIPserviceswhenrunningSAPinaRACenvironment?
AreOracleApplicationscertifiedwithRAC?

Diagnosibility
Whatarethecdmpdirectoriesinthebackground_dump_destusedfor?
HowdoIgatherallrelevantOracleandOSlog/tracefilesinanOracleRACclustertoprovidetoSupport?

EBusinessSuitewithRAC
WhatistheoptimalmigrationpathtobeusedwhilemigratingtheEBusinesssuitetoOracleRAC?
IstheOracleEBusinessSuite(OracleApplications)certifiedagainstRAC?
CanIuseTAFwitheBusinessinaRACenvironment?
HowtoconfigureconcurrentmanagerinaRACenvironment?
ShouldfunctionalpartitioningbeusedwithOracleApplications?
WhicheBusinessversionisprefereable?
CanIuseAutomaticUndoManagementwithOracleApplications?
IsServerSideLoadBalancingsupported/recommended/proventechnologyinOracleEBusinessSuite?

ClusteredFileSystems
WhatarethemaximumnumberofnodesunderOCFSonLinux?
WhatfilescanIputonLinuxOCFS?
WherecanIfinddocumentationonOCFS?
WhataretheBestPracticesforusingaclusteredfilesystemwithOracleRAC?
CanIuseaclusterfilesystemforOCR,VotingDisk,Binariesaswellasdatabasefiles?
CanIuseOCFSwithSEOracleRAC?
IsSunQFSsupportedwithOracleRAC?WhataboutSunGFS?
IsRedHatGFS(GlobalFileSystem)iscertifiedbyOracleforusewithOracleRealApplicationClusters?
IsLinuxOCFS2(OCFSversion2)supportedwithOracleRAC?
WhatisthemaximumnumberofnodesIcanhaveinmyclusterifIamusingOCFS2?

OracleClusterware
Ihavea2nodeOracleRACcluster,ifIpulltheinterconnectonnode1tosimulateafailure,whydoesnode2reboot?
WhichprocessesaccesstheOCR?
WithOracleClusterware10g,howdoyoubackuptheOCR?
Mycustomerhasnoticedtonsoflogfilesgeneratedunder$CRS_HOME/log//client,isthereanywayautomatedwaywecansetupthroughOracleClusterwareto
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

5/72

8/24/2015

Document220970.1

Mycustomerhasnoticedtonsoflogfilesgeneratedunder$CRS_HOME/log//client,isthereanywayautomatedwaywecansetupthroughOracleClusterwareto
prevent/minimize/removethoseaggressivelygeneratedfiles?
WhatdoIdo,IhaveacorruptOCRandnovalidbackup?
WhyisthehomeforOracleClusterware/OracleGridInfrastructurenotrecommendedtobeasubdirectoryoftheOraclebasedirectory?
DoIneedtohaveuserequivalence(ssh,etc...)setupafterGRID/RACisalreadyinstalled?
WithGNS,doALLpublicaddresseshavetobeDHCPmanaged(publicIP,publicVIP,publicSCANVIP)?
HowistheOracleClusterRegistry(OCR)storedwhenIuseASM?
WhendoestheOraclenodeVIPfailovertoanothernodeandsubsequentlyreturntoitshomenode?
HowdoIprotecttheOCRandVotingincaseofmediafailure?
HowdoIusemultiplenetworkinterfacestoprovideHighAvailabilityand/orLoadBalancingformyinterconnectwithOracleClusterware?
CantheNetworkInterfaceCard(NIC)devicenamesbedifferentonthenodesinacluster,forbothpublicandprivate?
CanIruna10.1.0.xdatabasewithOracleClusterware10.2?
Isitsupportedtorerunroot.shfromtheOracleClusterwareinstallation?
Whenctrunthecommand'onsctlstart'receivesthemessage"Unabletoopenlibhasgen10.so".Anyideawhythemessage"unabletoopenlibhasgen10.so"?
VotingFilesstoredinASMHowmanydisksperdiskgroupdoIneed?
OCRstoredinASMWhathappens,ifmyASMinstancefailsonanode?
IsitpossibletouseASMfortheOCRandvotingdisk?
Iamtryingtomovemyvotingdisksfromonediskgrouptoanotherandgettingtheerror"crsctlreplacevotedisknotpermittedbetweenASMDiskGroups."Why?
CanIrunthefixupscriptgeneratedbythe11.2OUIorCVUonarunningsystem?
WhatshouldthepermissionsbesettoforthevotingdiskandocrwhendoinganOracleRACInstall?
HowtomovetheOCRlocation?
IamtryingtoinstallOracleClusterware(10.2)andwhenIruntheOUI,attheSpecifyClusterConfigurationscreen,theAdd,EditandRemovebuttonsaregrayedout.Nothingcomesup
intheclusternodeseither.Why?
WhathappensifIlosemyvotingdisk(s)?
IaminstallingOracleClusterwarewitha3rdpartyvendorclusterwarehoweverinthe"SpecifyClusterConfigurationPage",OracleClusterwareinstallerdoesn'tshowtheexisting
nodes.Why?
ImadeamistakewhenIcreatedtheVIPduringtheinstallofOracleClusterware,canIchangetheVIP?
HowshouldItestthefailureofthepublicnetwork(IEOracleVIPfailover)inmyOracleRACenvironment?
Whatisthevotingdiskusedfor?
CanIconfigureafirewall(iptables)ontheclusterinterconnect?
Doesthehostnamehavetomatchthepublicnameorcanitbeanythingelse?
Ihavea2nodeRACrunning.Inoticethatitisalwaysnode2thatisevictedwhenItestprivatenetworkfailurescenariobydisconnectingtheprivatenetworkcable.Doesn'tmatter
whetheritisnode1'sornode2'sprivatenetworkcablethatisdisconnected,itisalwaysthenode2thatisevicted.Whathappensina3nodesRACclusterifnode1'scableis
disconnected?
CanIuseOracleClusterwaretoprovidecoldfailoverofmysingleinstanceOracleDatabases?
WhatarethelicensingrulesforOracleClusterware?CanIrunitwithoutRAC?
InthecourseoffailuretestinginanextendedRACenvironmentwefindentriesinthecssdlogfilewhichindicateactionslike'diskShortTimeoutsetto(value)'and'diskLongTimeoutsetto
(value)'.
Cananyonepleaseexplainthemeaningofthesetwotimeoutsinadditiontodisktimeout?
DuringOracleClusterwareinstallation,Iamaskedtodefineaprivatenodename,andthenonthenextscreenaskedtodefinewhichinterfacesshouldbeusedasprivateandpublic
interfaces.Whatinformationisrequiredtoanswerthesequestions?
CanIchangethenameofmyclusterafterIhavecreateditwhenIamusingOracleClusterware?
WhydoesOracleClusterwareuseanadditional'heartbeat'viathevotingdisk,whenotherclustersoftwareproductsdonot?
WhydoesOraclestillusethevotingdiskswhenotherclustersofwareispresent?
Customerishittingbug4462367withanerrormessagesayinglowopenfiledescriptor,howdoIworkaroundthisuntilthefixisreleasedwiththeOracleClusterwareBundlefor
10.2.0.3or10.2.0.4isreleased?
HowdoIidentifythevotingfilelocation?
HowmuchI/Oactivityshouldthevotingdiskhave?
DoesOracleClusterwarehavetobethesameorhigherreleasethanallinstancesrunningonthecluster?
CanIuseOracleClusterwaretomonitormyEMAgent?
WhataretheIPrequirementsfortheprivateinterconnect?
CanIsetupfailoveroftheVIPtoanothercardinthesamemachineorwhatdoIdoifIhavedifferentnetworkinterfacesondifferentnodesinmycluster(I.E.eth0onnode1,2andeth1
onnode3,4)?
HowtoRestoreaLostVotingDiskusedbyOracleClusterware10g
HowcanIregisterthelistenerwithOracleClusterwareinRAC10gRelease2?
HowisthevotingdiskusedbyOracleClusterware?
DoesOracleClusterwaresupportapplicationvips?
HowdoIputmyapplicationunderthecontrolofOracleClusterwaretoachievehigheravailability?
Isitsupportedtoallow3rdPartyClusterwaretomanageOracleresources(instances,listeners,etc)andturnoffOracleClusterwaremanagementofthese?
WhatistheHighAvailabilityAPI?
IsitarequirementtohavethepublicinterfacelinkedtoETH0ordoesitonlyneedtobeonaETHlowerthantheprivateinterface?:publiconETH1privateonETH2
HowdoIrestoreOCRfromabackup?OnWindows,canIuseocopy?

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

6/72

8/24/2015

HowdoIrestoreOCRfromabackup?OnWindows,canIuseocopy?

Document220970.1

Streched/ExtendedRACNoSubCategory
Whatarethenetworkrequirementsforthethirdvotingfile(inathirdlocation)inanextendedOracleRACsetup?
CanIuseASMtomirrorOracledatainanextendedRACenvironment?
WhatarethenetworkrequirementsforanextendedRACcluster?
CanacustomeruseSERACtoimplementan"ExtendedRACCluster"?
WhatisthemaximumdistancebetweennodesinanextendedRACenvironment?
CanIuseASMasmechanismtomirrorthedatainanExtendedRACcluster?
Howshouldvotingdisksbeimplementedinanextendedclusterenvironment?CanIusestandardNFSforthethirdsitevotingdisk?

ClusterVerificationUtility(CVU)NoSubCategory
WherecanIfindtheCVUtracefiles?
WhyisvalidateUserEquivfailingduringinstall(orcluvfyrun)?
HowdoIturnontracing?
CanIcheckifthestorageissharedamongthenodes?
WhenIrun10.2CLUVFYonasystemwhereRAC10gRelease1isrunningIgetfollowingoutput:
Packageexistencecheckfailedfor"SUNWscucm:3.1".
Packageexistencecheckfailedfor"SUNWudlmr:3.1".
Packageexistencecheckfailedfor"SUNWudlm:3.1".
Packageexistencecheckfailedfor"ORCLudlm:Dev_Release_06/11/04,_64bit_3.3.4.8_reentrant".
Packageexistencecheckfailedfor"SUNWscr:3.1".
Packageexistencecheckfailedfor"SUNWscu:3.1".
CheckingthisSolarissystemIdon'tseethosepackagesinstalled.CanIcontinuemyinstall?
Whatarethedefaultvaluesforthecommandlinearguments?
HowdoIchecktheOracleClusterwarestackandothersubcomponentsofit?
IsthereawaytoverifythattheOracleClusterwareisworkingproperlybeforeproceedingwithRACinstall?
Atwhatpointcluvfyisusable?CanIusecluvfybeforeinstallingOracleClusterware?
WhatisCVU?Whatareitsobjectivesandfeatures?
Whatisastage?
Whatisacomponent?
Whatisnodelist?
DoIhavetoberoottouseCVU?
Whataboutdiscovery?DoesCVUdiscoverinstalledcomponents?
WhataretherequirementsforCVU?
Whatis'cvuqdisk'rpm?WhyshouldIinstallthisrpm?
HowdoIinstall'cvuqdisk'package?
HowdoIknowaboutcluvfycommands?Theusagetextofcluvfydoesnotshowindividualcommands.
DoIhavetotypethenodelisteverytimefortheCVUcommands?Isthereanyshortcut?
HowdoIgetdetailoutputofacheck?
HowdoIchecknetworkornodeconnectivityrelatedissues?
HowdoIcheckwhetherOCFSisproperlyconfigured?
HowdoIcheckuseraccountsandadministrativepermissionsrelatedissues?
HowdoIcheckminimalsystemrequirementsonthenodes?
Isthereawaytocomparenodes?
Whythepeercomparisonwithrefnodesayspassedwhenthegrouporuserdoesnotexist?
Whycluvfyreports"unknown"onaparticularnode?
Whataretheknownissueswiththisrelease?

RACOneNodeNoSubCategory
HowdoesRACOneNodecomparewithdatabaseDRproductslikeDataGuardorGoldenGate?
IfacurrentcustomerhasanEnterpriseLicenseAgreement(ELA),aretheyentitledtouseOracleRACOneNode?
HowisOracleRACOneNodelicensedandpriced?
IsOracleRACOneNodesupportedwith3rdpartyclusterwareand/or3rdpartyCFS?
HowdoesRACOneNodecomparewithtraditionalcoldfailoversolutionslikeHPServiceguard,IBMHACMP,SunClusterandSymantec,andVeritasClusterServer?
HowdoesRACOneNodecomparewithasingleinstanceOracleDatabaseprotectedwithOracleClusterware?
WhatisOracleRealApplicationClustersOneNode(RACOneNode)?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

7/72

8/24/2015

WhatisOracleRealApplicationClustersOneNode(RACOneNode)?
IfIaddorremovenodesfromthecluster,howdoIinformRACOneNode?
IsRACOneNodesupportedwithdatabaseversionspriorto11.2?
HowdoIgetOracleRealApplicationClustersOneNode(OracleRACOneNode)?
WheredoIfindthedocumentationforRACOneNode?
DoesEnterpriseManagerSupportRACOneNode?
HowdoIinstallthecommandlinetoolsforRACOneNode?
ArewecertifyingapplicationsspecificallyforRACOneNode?
DoesRacOneNodemakesenseinastretchclusterenvironment?
HowdoesRACOneNodecomparewithvirtualizationsolutionslikeVMware?
CanIuseOracleRACOneNodeforStandardEditionOracleRAC?
WhatisRACOneNodeOmotion?

Document220970.1

GridInfrastructureNoSubCategory
HowtochangetheOracleGridInfrastructuresoftwareowner?
IsitrecommendedthatweputtheOCR/VotingDisksinOracleASMand,ifso,isitpreferabletocreateaseparatediskgroupforthem?
WhatdoIneedtodowhenIneedtochangethehostnameforaclusternoderunningOracleClusterware?
HowtoefficientlyrecoverfromalossofanOracleASMdiskgroupcontainingtheOracleClusterwarefiles?
CanIremoveOracleresourcesfromthecluster?HowcanIhideunusedresourceswhenlistingallresourcesinthecluster?
HowtouseSCANandnodelistenerswithdifferentports?
HowtochangetheSCANconfigurationaftertheOracleGridInfrastructure11gRelease2installationiscomplete?
WhyamIonlyusing1outof3SCANIPaddresses?
HowtoinstallOracleGridInfrastructureusingSCANwithoutusingDNS?
HowcanIaddmoreSCANVIPsorlistenersnotusingDNS?
HowdoIexplainthefollowingphraseinthe"OracleClusterwareAdministrationandDeploymentGuide11gRelease2(11.2)"toacustomer?
Page227:"IfOracleASMfails,thenOCRisnotaccessibleonthenodeonwhichOracleASMfailed,buttheclusterremainsoperational.TheentireclusteronlyfailsiftheOracleASM
instanceontheOCRmasternodefails,ifthemajorityoftheOCRlocationsareinOracleASM,andifthereisanOCRreadorwriteaccess,thenthecrsdstopsandthenodebecomes
inoperative."
Iftheroot.shscriptfailsonanodeduringtheinstalloftheGridInfrastructurewithOracleDatabase11gRelease2,canIrerunit?
IstheGNSrecommendedformostOracleRACinstallations?

ClusterHealthMonitor(IPD/OS)NoSubCategory
WhatisClusterHealthMonitor(IPD/OS)?
WhatOSdoesClusterHealthMonitor(IPD/OS)support?

QoSManagementNoSubCategory
WhatisOraclesgoalindevelopingQoSManagement?
WhattypeofapplicationsdoesOracleQoSManagementmanage?
WhatdoesQoSManagementmanage?
WhattypesofresourcesdoesQoSManagementmanage?
WhattypeofuserinterfacesdoesQoSManagementsupport?
WhatQoSManagementfunctionalityisinOracleEnterpriseManager?
Whattypesofperformanceobjectivescanbeset?
DoesQoSManagementrequireanyspecificdatabasedeployment?
WhatareServerPools?
WhatmethodsdoesQoSManagementsupportforclassifyingapplicationsandworkloads?
WhatistheoverheadofusingQoSManagement?
DoesQoSManagementnegativelyaffectanapplicationsavailability?
WhathappensshouldtheQoSManagementServerfail?
WhatisMemoryGuardandhowdoesitwork?
HowdoesQoSManagementenablethePrivateDatabaseCloud?
WhichversionsofOracledatabasesdoesQoSManagementsupport?
IsthisaproducttobeusedbyanITadministratororDBA?
WherecanIfinddocumentationforQoSManagement?

Answers
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

8/72

Answers

8/24/2015

Document220970.1

WhatkindofHWcomponentsdoyourecommendfortheinterconnect?
Thegeneralrecommendationfortheinterconnectistoprovidethehighestbandwidthinterconnect,togetherwiththelowestlatencyprotocolthatisavailableforagivenplatform.Inpractice,
GigabitEthernetwithUDPhasprovensufficientineverycaseithasbeenimplemented,andtendstobethelowestcommondenominatoracrossplatforms.

Isrcpand/orrshrequiredfornormalOracleRACoperation?
rcp""and""rsh""arenotrequiredfornormalOracleRACoperation.Howeverinolderversions""rsh""and""rcp""shouldtobeenabledforOracleRACandpatchsetinstallation.Inlater
releases,sshisusedfortheseoperations.
NoteOracleEnterpriseManagerusesrsh.

WhatisCacheFusionandhowdoesthisaffectapplications?
CacheFusionisanewparalleldatabasearchitectureforexploitingclusteredcomputerstoachievescalabilityofalltypesofapplications.CacheFusionisasharedcachearchitecturethatuses
highspeedlowlatencyinterconnectsavailabletodayonclusteredsystemstomaintaindatabasecachecoherency.Databaseblocksareshippedacrosstheinterconnecttothenodewhere
accesstothedataisneeded.Thisisaccomplishedtransparentlytotheapplicationandusersofthesystem.AsCacheFusionusesatmosta3pointprotocol,thismeansthatiteasilyscalesto
clusterswithalargenumbersofnodes.Formoreinformationaboutcachefusionseethefollowinglinks:
AdditionalInformationcanbefoundat:
Note:139436.1Understanding9iRealApplicationClustersCacheFusion

Isitdifficulttotransition(migrate)fromSingleInstancetoOracleRAC?
Iftheclusterandtheclustersoftwarearenotpresent,thesecomponentsmustbeinstalledandconfigured.TheOracleRACoptionmustbeaddedusingtheOracleUniversalInstaller,which
necessitatestheexistingDBinstancemustbeshutdown.Therearenochangesnecessaryontheuserdatawithinthedatabase.However,ashortageoffreelistsandfreelistgroupscan
causecontentionwithheaderblocksoftablesandindexesasmultipleinstancesvieforthesameblock.Thismaycauseaperformanceproblemandrequiredatapartitioning.However,the
needforthesechangesshouldberare.
Recommendation:applyautomaticspacesegmentmanagementtoperformthesechangesautomatically.Thefreespacemanagementwillreplacethefreelistsandfreelistgroupsandis
better.ThedatabaserequiresoneRedothreadandoneUndotablespaceforeachinstance,whichareeasilyaddedwithSQLcommandsorwithEnterpriseManagertools.NOTE:With
ORacleRAC11gRelease2,youdonotneettoprecreateredothreadsorundotablespacesifyouareusingOracleManagedFiles(EGASM).
Datafileswillneedtobemovedtoeitheraclusteredfilesystem(CFS)sothatallnodescanaccessthem.OraclerecommendstheuseofAutomaticStorageManagement(ASM)Also,the
MAXINSTANCESparameterinthecontrolfilemustbegreaterthanorequaltonumberofinstancesyouwillstartinthecluster.
Formoredetailedinformation,pleaseseeMigratingfromsingleinstancetoRACintheOracleDocumentation.
WithOracleDatabase10gRelease2,$ORACLE_HOME/bin/rconfigtoolcanbeusedtoconvertSingleinstancedatabasetoRAC.ThistooltakesinaxmlinputfileandconverttheSingleInstance
databasewhoseinformationisprovidedinthexml.Youcanrunthistoolin"verifyonly"modepriortoperformingactualconversion.ThisisdocumentedintheOracleRACAdminbookanda
samplexmlcanbefound$ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC.xml.ThistoolonlysupportsdatabasesusingaclusteredfilesystemorASM.Youcannotuseitwith
rawdevices.GridControl10gRelease2providesaeasytousewizardtoperformthisfunction.
@Note:Pleasebeawarethatyoumayhitbug:4456047(shutdownimmediatehangs)asyouconvert
@thedatabase.Thebugisupdatedwithworkaroundandthew/ashouldisreleasenotedaswell.
OracleEnterpriseManagerincludesworkflowstoassistewithmigrations.(I.E.MigratingtoASM,CreatingStandby,ConvertingStandbytoRACetc)ThemigrationisautomatedinEnterprise
ManagerGridControl10.2.0.5.

WhatarethedependenciesbetweenOCFSandASMinOracleDatabase10g?
InanOracleRAC10genvironment,thereisnodependencybetweenAutomaticStorageManagement(ASM)andOracleClusterFileSystem(OCFS).
OCFSisnotrequiredifyouareusingAutomaticStorageManagement(ASM)fordatabasefiles.YoucanuseOCFSonWindows(Version2onLinux)forfilesthatASMdoesnothandlebinaries
(sharedoraclehome),tracefiles,etc.Alternatively,youcouldplacethesefilesonlocalfilesystemseventhoughit'snotasconvenientgiventhemultiplelocations.
IfyoudonotwanttouseASMforyourdatabasefiles,youcanstilluseOCFSfordatabasefilesinOracleDatabase10g.
@PleaserefertoASMandOCFSPositioning

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

9/72

8/24/2015

@PleaserefertoASMandOCFSPositioning

Document220970.1

DowehavetohaveOracleDatabaseonallnodes?
EachnodeofaclusterthatisbeingusedforaclustereddatabasewilltypicallyhavethedatabaseandOracleRACsoftwareloadedonit,butnotactualdatafiles(theseneedtobeavailablevia
shareddisk).Forexample,ifyouwishtorunOracleRACon2nodesofa4nodecluster,youwouldneedtoinstalltheclusterwareonallnodes,OracleRACon2nodesanditwouldonlyneed
tobelicensedonthetwonodesrunningtheOracleRACdatabase.Notethatusingaclusteredfilesystem,orNASstoragecanprovideaconfigurationthatdoesnotnecessarilyrequirethe
Oraclebinariestobeinstalledonallnodes.
WithOracleRAC11gRelease2,ifyouareusingpolicymanageddatabases,thenyoushouldhavetheOracleRACbinariesaccessibleonallnodesinthecluster.

WhatsoftwareisnecessaryforOracleRAC?DoesithaveaseparateinstallationCDtoorder?
OracleRealApplicationClustersisanoptionofOracleDatabaseandthereforepartoftheOracleDatabaseCD.WithOracle9i,Oracle9iRACispartofOracle9iEnterpriseEdition.Ifyouinstall
9iEEontoacluster,andtheOracleUniversalInstaller(OUI)recognizesthecluster,youwillbeprovidedtheoptionofinstallingRAC.MostUNIXplatformsrequireanOSDinstallationforthe
necessaryclusterware.ForIntelplatforms(LinuxandWindows),OracleprovidestheOSDsoftwarewithintheOracle9iEnterpriseEditionrelease.
WithOracleDatabase10g,OracleRACisanoptionofEEandavailableaspartofSE.OracleprovidesOracleClusterwareonitsownCDincludedinthedatabaseCDpack.
Pleasecheckthecertificationmatrix(Note184875.1)orwiththeappropriateplatformvendorformoreinformation.
WithOracleDatabase11gRelease2,OracleClusterwareandAutomaticStorageManagementareinstalledasasinglesetofbinariescalledthegridinfrastructure.Themediaforthegrid
infrastructureisonaseparateCDorunderthegriddirectory.Forstandaloneservers,AutomaticStorageManagementandOracleRestartareinstalledasthegridinfrastructurefora
standaloneserverwhichisinstalledfromthesamemedia.

WhatApplicationDesignconsiderationsshouldIbeawareofwhenmovingtoOracleRAC?
ThegeneralprincipalsarethatfundamentallynodifferentdesignandcodingpracticesarerequiredforRAChoweverapplicationflawsinexecutionordesignhaveahigherimpactinRAC.The
performanceandscalabilityinRACwillbemoresensitivetobadplansorbadschemadesign.Serializingcontentionmakesapplicationslessscalable.IfyourcustomerusesstandardSQLand
schematuning,itsolves>80%ofperformanceproblems
Someofthescaleabilitypitfallstheyshouldlookforare:
*Serializingcontentiononasmallsetofdata/indexblocks
>monotonicallyincreasingkey
>frequentupdatesofsmallcachedtables
>segmentwithoutautomaticsegmentspacemanagement(ASSM)orFreeListGroup(FLG)
*Fulltablescans
>Optimizationforfullscansin11gcansaveCPUandlatency
*Frequentinvalidationandparsingofcursors
>Requiresdatadictionarylookupsandsynchronizations
*ConcurrentDDL(e.g.truncate/drop)
Lookfor:
*Indexeswithrightgrowingcharacteristics
>Usereversekeyindexes
>Eliminateindexeswhicharenotneeded
*Frequentupdatedandreadsofsmalltables
>small=fitsintoasinglebuffercache
>UseSparseblocks(PCTFREE99)toreduceserialization
*SQLwhichscanslargeamountofdata
>Perhapsmoreefficientwhenparallelized
>Directreadsdonotneedtobegloballysynchronized(hencelessCPUforglobalcache)

ShouldtheSCSI3reservationbitbesetforourOracleClusterwareonlyinstallation?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

10/72

8/24/2015

Document220970.1

ShouldtheSCSI3reservationbitbesetforourOracleClusterwareonlyinstallation?
OracleClusterwareandOracleRACdonotrequireneitheruseSCSI3PersistentGroupReservation(PGR).InanativeOracleRACStack(nothirdpartyorvendorcluster,
neitherOracleSolarisCluster)SCSI3PGRisnotrequiredbyOracleandshouldbedisabledonthestorage(fordisks/LUNsusedinthestack).
WhenusingathirdpartyorvendorclustersolutionsuchasSymantecVeritasSFRAC,thethirdpartclustersolutionmayrequirethatSCSI3PGRisenabledonthestorage,asthosesolutions
willuseSCSI3PGRaspartoftheirIOfencingprocedures.Ingeneral,SCSI3PGRisenabledatthearraylevelforexampleonEMChypervolumelevel.
Additionalinformation:
EnablingSCSI3PGRonstoragelevel(fordisks/LUNsusedintheclusterstackonlyenablesSCSI3PGRcapabilities.Ifset,aclusterorapplicationusingthispieceofstoragemaymake
useofSCSI3PGR.OracleSolarisClusterandVeritasClusterbydefaultuseSCSI3PGRundercertaincircumstances,OracleClusterwaredoesnot.
AsOracleClusterwaredoesnotuseSCSI3PGRandifyoudonotuseanddonotplanonusinganythirdpartysoftwarethatrequiresit,itisrecommendedtodisableSCSI3PGR
reservationforthedisks/LUNsusedfortheOracleRACStack.Reason:ifyouhavePRenabledonadevice,thedevicewouldhaveadefaultbehaviorregardingPRexpecting
theclientthatmakesuseofPRtocalltherightcommandsasrequired.However,thedefaultbehaviorofadeviceforwhichPRisenableddependsontheplatform,onthedevicedriver
andthePRsettingandmaynotworkforOracleClusterwareandtheRACstack,sincewedonotcallthesecommandstooperatethePRbehavior.
Example:OnAIXonecansetthereserve_policyforadiskonthediskdriverleveltoenablePR.AIXsupports4valuesforthereserve_policy(atthetimeofauthoringthisentry):
"no_reserve","single_path","PR_exclusive",and"PR_shared"Oraclerecommendsusingthefirstone"no_reserve"toavoidproblems.IfweusedPRcommandcalls"PR_shared"could
beused.However,ifyouset"PR_shared"onadeviceandyoudonotmaketherightPRcommandcallsonthisdevice,thedefaultmightpreventconcurrentaccesstothedevice,which
wouldbebadforanydeviceusedintheRACstack(ASMdisks/Voting/OCR).ThismaybeveryparticulartoAIXandotherplatformsmaybedifferent,butingeneralthiscouldbean
issue.

DoesWeblogic(WLS)supportServices,FAN/FCF,andtheLoadBalancingAdvisory(LBA)withOracleRAC?
@FordetailsonhowtoimplementWeblogicwithOracleRACseethefollowing:
@UsingWebLogicServerwithOracleRACchapterintheOracleFusion
@OracleWebLogicServerActiveGridLinkforOracleRealApplicationClusters(RAC)
TherecommendationfromOracleFusionMiddlewareistouseWLSActiveGridlinkwithOracleRAC.

AreblockdevicessupportedforOCR,VotingDisks,andASMdevices?
BlockDevicesareonlysupportedonLinux.OnotherUnixplatforms,thedirectIOsymanticsarenotapplicable(orrathernotimplemented)forblockdevices.
Note:ThedesupportfordirectuseofraworblockdevicesisscheduledforOracleDatabase12c.TheOracleDatabase10gOracleUniversalInstallerdoesnotsupportblockdevices
OracleClusterwareandASMdo.
WithOracleRAC11gRelease2,theOracleUniversalInstallerandtheConfigurationAssistantsdonotsupportraworblockdevicesanymore.TheCommandLineInterfacesstillsupport
rawandblockdevicesandhencetheOracleClusterwarefilescanbemovedaftertheinitialinstallation.
Note:Directuseofraworblockdevices(fordatabasefilesorClusterwarefiles)willbedesupportedwithOracleDatabase12c.UsingraworblockdevicesunderOracleASMwill
remainsupported.

DoIneedtorelinktheOracleClusterware/GridInfrastructurehomeafteranOSupgrade?
UsingOracleClusterware10gand11.1,OracleClusterwarebinariescannotberelinked.However,theclientsharedlibraries,whicharepartofthehomecanberelinked,inmostcases
thereshouldnotbeaneedtorelinkthem.SeeNote:743649.1formoreinformation.
UsingOracleGridInfrastructure11.2andhigher,therearesomeexecutablesintheGridhomethatcanandshouldberelinkedafteranOSupgrade.Thefollowingstepsdescribehowto
relinkanOracleGridInfrastructureforClustershome:
Asroot:
#cdGrid_home/crs/install
#perlrootcrs.plunlock
Asthegridinfrastructureforaclusterowner:
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

11/72

8/24/2015

Document220970.1

$exportORACLE_HOME=Grid_home
$Grid_home/bin/relink
Asrootagain:
#cdGrid_home/crs/install
#perlrootcrs.plpatch
Note:IfusingOracleGridInfrastructureforStandaloneEnvironments(OracleRestart),seetheOracleDocumentationformoreinformation:
http://docs.oracle.com/cd/E11882_01/install.112/e24321/oraclerestart.htm#BABJCHIJ

ArethereanyissuesfortheinterconnectwhensharingthesameswitchasthepublicnetworkbyusingVLANtoseparatethenetwork?
OracleRACandOracleClusterwaredeploymentbestpracticesrecommendthattheinterconnectbedeployedonastandalone,physicallyseparate,dedicatedswitch.
Manycustomers,however,haveconsolidatedthesestandaloneswitchesintolargermanagedswitches.AconsequenceofthisconsolidationisamergingofIPnetworksonasingle
sharedswitch,segmentedbyVLANs.Therearecaveatsassociatedwithsuchdeployments.
TheOracleRACcachefusionprotocolexercisestheIPnetworkmorerigorouslythannonRACOracledatabases.Thelatencyandbandwidthrequirementsaswellasavailability
requirementsoftheOracleRAC/OracleClusterwareinterconnectIPnetworkaremoreinlinewithhighperformancecomputing.
DeployingtheOracleRAC/OracleClusterwareinterconnectonasharedswitch,segmentedbyaVLANmayexposetheinterconnectlinkstocongestionandinstabilityinthelargerIP
networktopology.
IfdeployingtheinterconnectonaVLAN,thereshouldbea1:1mappingoftheVLANtoanonroutablesubnetandtheVLANshouldnotspanmultipleVLANs
(tagged)ormultipleswitches.
DeploymentconcernsinthisenvironmentincludeSpanningTreeloopswhenthelargerIPnetworktopologychanges,Assymetricalroutingthatmaycausepacketflooding,andlackof
finegrainedmonitoringoftheVLAN/port.

HowtouseVLANsforOracleRACandtheOracleClusterwareInterconnect?
ItisOracle'sstandingrequirementtoseparatethevarioustypesofcommunicationinanOracleRACcluster.Thisrequirementaddressesthefollowingseparationofcommunication:
EachnodeinanOracleRACclustermusthaveatleastonepublicnetwork.
EachnodeinanOracleRACclustermusthaveatleastoneprivatenetwork,alsoreferredtoas"clusterinterconnect".
EachnodeinanOracleRACclustermusthaveatleastanadditionalnetworkinterface,ifthesharedstorageisaccessedusinganetworkbasedconnection.
Clusterinterconnectnetworkseparationcanbesatisfiedeitherbyusingstandalone,dedicatedswitches,whichprovidethehighestdegreeofnetworkisolation,orVirtualLocalArea
NetworksdefinedontheEthernetswitch,whichprovidebroadcastdomainisolationbetweenIPnetworks.VLANsarefullysupportedforOracleClusterwareinterconnectdeployments.
PartitioningtheEthernetswitchwithVLANsallowsfor:

Sharingthesameswitchforprivateandpubliccommunication.
Sharingthesameswitchfortheprivatecommunicationofmorethanonecluster.
Sharingthesameswitchforprivatecommunicationandsharedstorageaccess.
Thefollowingbestpracticesshouldbefollowed:
TheClusterInterconnectVLANmustbeonanonroutedIPsubnet.
AllClusterInterconnectnetworksmustbeconfiguredwithnonroutedIPs.TheserverservercommunicationshouldbesinglehopthroughtheswitchviatheinterconnectVLAN.Thereis
noVLANVLANcommunication.
Oraclerecommendsmaintaininga1:1mappingofsubnettoVLAN.
ThemostcommonVLANdeploymentsmaintaina1:1mappingofsubnettoVLAN.ItisstronglyrecommendedtoavoidmultisubnetmappingtoasingleVLAN.Bestpractice

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

12/72

8/24/2015

Document220970.1

ThemostcommonVLANdeploymentsmaintaina1:1mappingofsubnettoVLAN.ItisstronglyrecommendedtoavoidmultisubnetmappingtoasingleVLAN.Bestpractice
recommendsasingleaccessVLANportconfiguredontheswitchfortheclusterinterconnectVLAN.TheserversidenetworkinterfaceshouldhaveaccesstoasingleVLAN.
ThesharedswitchshouldbeconfiguredtomitigatethecostofSpanningTree
TheswitchvendorsbestpracticesshouldbefollowedtoeitherdisableorlimitthecostofSpanningTreeconvergencefortheclusterinterconnectVLAN.
Sharingthesameswitchforprivatecommunicationandsharedstorageaccess
Thisconfigurationissupported,iftheunderlyingnetworkinfrastructuresupportsDataCenterBridging(DCB),zeropacketlossandcansatisfythelatencyandthroughputdefinedforthe
application.ThismayrequireimposingaQualityofService(QoS)onthesharedswitchtoprioritizenetworkbasedcommunicationtothestorage.FiberChanneloverEthernet(FCoE)
convergednetworksaresupportedforcertifiedconfigurations.

IsitagoodideatoaddantivirussoftwaretomyRACcluster?
Forcustomerswhochoosetorunantivirus(AV)softwareontheirdatabaseservers,theyshouldbeawarethatthenatureofAVsoftwareisthatdiskIObandwidthisreducedslightlyas
mostAVsoftwarechecksdiskwrites/reads.Also,astheAVsoftwareruns,itwilluseCPUcyclesthatwouldnormallybeconsumedbyotherserverprocesses(e.gyourdatabase
instance).Assuch,databaseswillhavefasterperformancewhennotusingAVsoftware.AssomeAVsoftwareisknowntolockthefileswhilstitscansthenitisagoodideatoexclude
theOracleDatafiles/controlfiles/logfilesfromaregularAVscan

IsthereaneedtoreniceLMSprocessesinOracleRAC10gRelease2?
LMSprocessesshouldberunninginRTbydefaultsince10.2,sothere'sNOneedtorenicethem,orotherwisemesswiththem.
Checkwithpsefl:
0Soracle311911075027085710:01?00:00:00ora_lmon_appsu01
0Soracle311931575027140310:01?00:00:07ora_lmd0_appsu01
0Soracle31195105827139610:01?00:00:00ora_lms0_appsu01
0Soracle31199105827139610:01?00:00:00ora_lms1_appsu01
7thcolumn,ifitis75or76thenthisisTimeShare,58isRealTime.
Youcanalsousechrttocheck:
LMS(RealTime):
$chrtp31199
pid31199'scurrentschedulingpolicy:SCHED_RR
pid31199'scurrentschedulingpriority:1
LMD(TimeShare)
$chrtp31193
pid31193'scurrentschedulingpolicy:SCHED_OTHER
pid31193'scurrentschedulingpriority:0

HowcanaNASstoragevendorcertifytheirstoragesolutionforOracleRAC?
AsofJanuary2007theOSCPhasbeendiscontinued!!
PleaserefertothislinkonOTNfordetailsonOracleRACTechnologiesMatrix(storagebeingpartofit).
OldAnswertext:
TheyshouldobtainanOCEtestkitandcompletetherequiredOracleRACtests.TheycansubmittherequestforanOCEkittoocesup_ie@oracle.com.

MycustomerhasanXAApplicationwithaOracleRACDatabase,canIdoLoadBalancingacrosstheOracleRACinstances?
No,notinthetraditionalOracleNetServicesLoadBalancing.Wehavewrittenadocumentthatexplainsthe**bestpracticesfor9i,10gRelease1and10gRelease2**.With
theOracleDatabase10gServices,lifegetseasier.Tounderstandservices,readtheOracleRACAdminandDeploymentGuidefor10gRelease2Chapter6.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

13/72

8/24/2015

Document220970.1

theOracleDatabase10gServices,lifegetseasier.Tounderstandservices,readtheOracleRACAdminandDeploymentGuidefor10gRelease2Chapter6.
WithOracleRAC11g,OracleprovidestransparentsupportforXAglobaltransactionsinanOracleRACenvironmentwhichsupportsloadbalancingwithOracleNetServicesacrossOracle
RACinstances.

HowdoIdeterminewhichnodeintheclusteristhe"Master"node?
Fortheclustersynchronizationservice(CSS),themastercanbefoundbysearchingORACLE_HOME/log/nodename/cssd/ocssd.logwhereitiseithertheOracleHOMEfortheOracle
Clusterware(thisistheGridInfrastructurehomeinOracleDatabase11gRelease2).
FormasterofaenqueueresourcewithOracleRAC,youcanselectfromv$ges_resource.Thereshouldbeamaster_nodecolumn.

IsInfinibandsupportedfortheOracleRACinterconnect?
IPoverIBissupported.RDSonLinuxissupportedwith10.2.0.3forward.Qlogic(formerlySilverStorm)isthesupportedRDSvendor.
Watchcertifyforupdates.AsotherplatformsadoptRDS,wewillexpandsupport.Latestupdate:RDSv1issupportedonSolariswith10.2.0.4.
TherearenoplanstosupportuDAPLorITAPIprotocols.
SeeNote:751343.1formoredetails.

WherecanIfindalistofsupportedsolutionstoensureNICavailability/redundancy(fortheinterconnect)perplatform?
IBMAIXavailablesolutions:
Etherchannel(OSbased)
HACMPbasednetworkfailoversolution
Moreinformation:Note:296856.1
HPHP/UXavailablesolutions:
APAAutoPortAggregation(OSbased)
MC/Serviceguardbasednetworkfailoversolution
Combinationofbothsolutions
Moreinformation:Note:296874.1andAutoPortAggregation(APA)SupportGuide
SunSolarisavailablesolutions:
SunTrunking(OSbased)
SunIPMP(OSbased)
SunClusterbasednetworkfailoversolution(clprivnet)
MoreinformationforOracleRAC10gandOracleRAC11gRelease1:
MyOracleSupportNote:283107.1ConfigureIPMPfortheOracleVIPandIPMPintroduction
MyOracleSupportNote:368464.1HowtoSetupIPMPasClusterInterconnect
MoreinformationforOracleRAC11gRelease2:
MyOracleSupportNote:1069584.1SolarisIPMPandTrunkingfortheclusterinterconnectinOracleGridInfrastructure
@RelatedRACFAQentries:
@InSolaris10,doweneedSunClusterwaretoprovideredundancyfortheinterconnectandmultipleswitches?
Linuxavailablesolutions:
Bonding
Moreinformation:Note:298891.1
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

14/72

8/24/2015

Moreinformation:Note:298891.1

Document220970.1

@RelatedRACFAQentries:
@HowdoIusemultiplenetworkinterfacestoprovideHighAvailabilityand/orLoadBalancingformyinterconnectwithOracleClusterware?
Windowsavailablesolutions:
Teaming
OnWindowsteamingsolutionstoensureNICavailabilityareusuallypartofthenetworkcarddriver.
Thus,theydependonthenetworkcardused.Please,contacttherespectivehardwarevendorformoreinformation.
OSindependentsolution:
RedundantInterconnectUsageenablesloadbalancingandhighavailabilityacrossmultiple(uptofour)privatenetworks(alsoknownasinterconnects).
OracleRAC11gRelease2,PatchSetOne(11.2.0.2)enablesRedundantInterconnectUsageasafeatureforallplatforms,exceptWindows.
OnsystemsthatuseSolarisCluster,RedundantInterconnectUsagewilluseclprivnet.

DoesOraclesupportOracleRACinSolariscontainers(a.k.a.SolarisZones)?
YESforOracleRAC10gRel.2onwards.WhileGlobalcontainershavebeensupportedforawhile,Oracleaddedsupportforlocalcontainersrecentlyafterthelocalcontainerswere
extendedtoallowdirecthardwaremodification.
LiftingthisrestrictionallowOracleClusterwaretooperateonhardwareresourcessuchasthenetworkfortheOracleVIPdirectly,enablingOracleRACtoruninlocalcontainers.
MoreinformationaboutSolariscontainersupportcanbefoundinOracleCertify.
@ThefollowingSUNwhitepapercanbedownloadedinternallyfromhttp://rac.us.oracle.comandusedexternallytoexplainthisconfigurationSUNTWP:RAConSolarisZoneClusters

WhatisSCAN?
SingleClientAccessName(SCAN)isasinglenamethatallowsclientconnectionstoconnecttoanydatabaseinanOracleclusterindependentlyofwhichnodeintheclusterthedatabase
(orservice)iscurrentlyrunning.TheSCANshouldbeusedinallclientconnectionstringsanddoesnotchangewhenyouadd/removenodesfromthecluster.SCANallowsclientstouse
EZConnectorthethisJDBCURL.
sqlplussystem/manager@sales1scan:1521/oltp
jdbc:oracle:thin:@sales1scan:1521/oltp
TheSCANisdefinedasasinglenameresolvingto3IPaddressesineitherthecluster'sGNSoryourcorporateDNS.
**ClickhereformoredetailsonSCAN.

IgetthefollowingerrorstartingmyOracleRACdatabase,whatdoIdo?
WARNING:Noclusterinterconnecthasbeenspecified.
Thissimplymeansthatyouneitherhaveacluster_interconnectsparametersetforthedatabase,norwasthereanyclusterinterconnectspecificationfoundintheOCR,sothatthe
privateinterconnectispickedatrandombythedatabase,andhencethewarning.
Youcaneithersetthecluster_interconnectsparameterintheinitializationfile(spfile/pfile)ofthedatbasetospecifyaprivateinterconnectIP,ORyoucanuse"oifcfgsetif"(type"oifcfg"
forhelp)toclassifyacertainnetworkforastheclusterinterconnectnetwork.
$oifcfggetif
eth0138.2.236.0globalpublic
eth2138.2.238.0globalcluster_interconnect
Notethatoifcfgenablesyoutospecify"local"aswellas"global"settings.WithOracleClusterware10gRel.1andRel.2aswellasOracleClusterware11gRel.1,itis,however,only
supportedtouseglobalsettings.Ifthehardware(networkinterface)meanttobeusedfortheinterconnectisnotthesameonallnodesinthecluster,theconfigurationneedstobe
changedonthehardware/OSlevelaccordingly.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

15/72

8/24/2015

Document220970.1

IfmyOCRandVotingDisksareinASM,canIshutdowntheASMinstance?
No.YouwillhavetostoptheOracleClusterwarestackonthenodeonwhichyouneedtostoptheOracleASMinstance.Eitheruse"crsctlstopclusternnode_name"or"crsctlstopcrs"
forthispurpose.

Ihavechangedmyspfilewithaltersystemsetparameter_name=....scope=spfile.ThespfileisonASMstorageandthedatabasewillnotstart.
Howtorecover:</p>
In$ORACLE_HOME/dbs
.oraenv&ltinstance_name&gt
sqlplus"/assysdba"
startupnomount
createpfile='recoversp'fromspfile
/
shutdownimmediate
quit

Noweditthenewlycreatedpfiletochangetheparametertosomethingsensible.
Then:

sqlplus"/assysdba"
startuppfile='recoversp'(orwhateveryoucalleditinstepone).
createspfile='+DATA/GASM/spfileGASM.ora'frompfile='recoversp'
/
<b>N.B.Thenameofthespfileisinyouroriginalinit(instance_name).orasoadjusttosuit</b>
shutdownimmediate
startup
quit

WhatcombinationsofOracleClusterware,OracleRACandASMversionscanIuse?
SeeNote:337737.1foradetailedsupportmatrix.BasicallytheClusterwareversionmustbeatleastthehighestreleaseofASMorOracleRAC.ASMmustbeatleast10.1.0.3towork
with10.2database.
Note:WithOracleDatabase11gRelease2,YoumustupgradeOracleClusterwareandASMto11gRelease2atthesametime.

Ihada3nodeOracleRAC.Oneofthenodeshadtobecompletelyrebuiltasaresultofaproblem.Astherearenobackups,Whatistheproperprocedureto
removethe3rdnodefromtheclustersoitcanbeaddedbackin?
Followthedocumentationforremovinganodebutyoucanskipallthestepsinthenoderemovaldocthatneedtoberunonthenodebeingremoved,likesteps4,6and7(SeeChapter
10ofOracleRACAdminandDeploymentGuide).Makesurethatyouremoveanydatabaseinstancesthatwereconfiguredonthefailednodewithsrvctl,andlistenerresourcesalso,
otherwiserootdeltenode.shwillhavetroubleremovingthenodeapps.
Justrunningrootdeletenode.shisn'treallyenough,becauseyouneedtoupdatetheinstallerinventoryaswell,otherwiseyouwon'tbeabletoaddbackthenodeusingaddNode.sh.And
ifyoudon'tremovetheinstancesandlistenersyou'llalsohaveproblemsaddingthenodeandinstancebackagain.
@Analternative(thanthegenericdocumentation,bug:5929611filed)foraremovenodeisNote:269320.1

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

16/72

8/24/2015

Document220970.1

ArejumboframessupportedfortheRACinterconnect?
Yes.FordetailsseeNote:341788.1ClusterInterconnectandJumboFrames

AreSunLogicalDomains(ldoms)supportedwithRAC?
SunLogicalDomains(ldoms)aresupportedwithOracleDatabase(bothsingleinstanceandRAC).Checkcertifyforthelatestinformation.

WhatisStandardEditionOracleRAC?
AsofOracleDatabase10g,acustomerwhohaspurchasedStandardEditionisallowedtousetheOracleRACoptionwithinthelimitationsofStandardEdition(SE).Forlicensing
restrictionsyoushouldreadtheOracleDatabaseLicenseDoc.Atahighlevelthismeansthatyoucanhaveamaxof4socketsinthecluster,youmustuseASMforalldatabasefiles.As
ofOracleDatabase11gRelease2,ASMincludesACFS(aclusterfilesystem).ASMClusterFileSystem(ACFS)oralocalOSfilesystemmustbeusedtostoreallnondatabasefiles
includingOracleHome,Applicationandsystemfiles,andUserfiles
NOTE:3rdpartyclusterwareandclusteredfilesystems(otherthanASM)arenotsupported.ThisincludesOCFSandOCFS2.
HereisthetextfromtheappropriatefootnoteinthePriceList(asofJan2010,pleasecheckpricelistforanychanges):
OracleDatabaseStandardEditioncanonlybelicensedonserversthathaveamaximumcapacityof4sockets.IflicensingbyNamedUserPlus,theminimumis5NamedUserPlus
licenses.OracleDatabaseStandardEdition,whenusedwithOracleRealApplicationClusters,mayonlybelicensedonasingleclusterofserverssupportinguptoatotalmaximum
capacityof4sockets.
NOTE:Thismeansthattheservercapacitymustmeettherestrictionevenifthesocketsareempty,sincetheycounttowardscapacity.

WheredoIfindOracleClusterwarebinariesandASMbinarieswithOracleDatabase11gRelease2?
WithOracleDatabase11gRelease2,thebinariesforOracleClusterwareandAutomaticStorageManagement(ASM)aredistributedinasinglesetofbinariescalledthegrid
infrastructure.Toinstallthegridinfrastructure,gotothegriddirectoryonyour11gRelease2mediaandruntheOracleUniversalInstaller).ChoosetheGridInfrastructureforaCluster.
IfyouareinstallASMforasingleinstanceofOracleDatabaseonaStandaloneServer,choosetheGridInfrastructureforaStandaloneServer.ThisinstallationincludesOracleRestart.

Ihavethe11.2GridInfrastructureinstalledandnowIwanttoinstallanearlierversionofOracleDatabase(11.1or10.2),isthissupported?
Yeshoweveryouneedto"pin"thenodesintheclusterbeforetryingtocreateadatabaseusinganearlierversionofOracleDatabase(IEnot11.2).Thecommandtopinanodeiscrsctl
pincssnnodename.YoushouldalsoapplythepatchforBug8288940tomakeDBCAworkinan11.2cluster.

IgetanerrorwithDBCAfrom10.2or11.1afterIhaveinstalledthe11.2GridInfrastructure?
YouwillneedtoapplythepatchforBug8288940toyourdatabasehomeinorderforittorecognizeASMrunningfromthenewgridinfrastructurehome.Alsomakesureyouhave
"pinned"thenodes.
crsctlpincssnnodename

CanIuseiSCSIstoragewithmyOracleRACcluster?
ForiSCSI,Oraclehasmadethestatementthat,asablockprotocol,thistechnologydoesnotrequirevalidationforsingleinstancedatabase.Therearemanyearlyadoptercustomersof
iSCSIrunningOracle9iandOracleDatabase10g.AsforOracleRAC,OraclehaschosentovalidatetheiSCSItechnology(noteachvendor'stargets)forthe10gplatformsthishasbeen
completedforLinuxandWindows.ForWindowswehavetestedupto4nodesAnyWindowsiSCSIproductsthataresupportedbythehostandstoragedevicearesupportedbyOracle.
Wedon'tsupportNASdevicesforWindows,howeversomeNASdevices(egNetApp)canalsopresentthemselvesasiSCSIdevices.IfthisisthecasethenacustomercanusethisiSCSI
devicewithWindowsaslongastheiSCSIdevicevendorsupportsWindowsasaninitiatorOS.NovendorspecificinformationwillbepostedonCertify.

Whatwouldyourecommendtocustomer,OracleClusterwareorVendorClusterware(I.E.HPServiceGuard,HACMP,SunCluster,Veritasetc.)withOracleReal
ApplicationClusters?
YouwillbeinstallingandusingOracleClusterwarewhetherornotyouusetheVendorClusterware.OracleClusterwareprovidesacompleteclusteringsolutionandisrequiredforOracle
RACorAutomaticStorageManagement(includingACFS).
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

17/72

8/24/2015

Document220970.1

RACorAutomaticStorageManagement(includingACFS).
VendorclusterwareisonlyrequiredwithOracle9iRAC.CheckthecertificationmatrixinMyOracleSupportfordetailsofcertifiedvendorclusterware.

WhenconfiguringtheNICcardsandswitchforaGigEInterconnectshoulditbesettoFULLorHalfduplexinOracleRAC?
YoumustuseFullDuplexforallnetworkcommunication.HalfDuplexmeansyoucanonlyeithersendORreceiveatatime.

CanIuseOracleRACinadistributedtransactionprocessingenvironment?
YES.BestpracticesistohavealltightlycoupledbranchesofadistributedtransactionrunningonanOracleRACdatabasemustrunonthesameinstance.Betweentransactionsand
betweenservices,transactionscanbeloadbalancedacrossallofthedatabaseinstances.
PriortoOracleRAC11g,youmustuseservicestomanageDTPenvironments.BydefiningtheDTPpropertyofaservice,theserviceisguaranteedtorunononeinstanceatatimeinan
OracleRACdatabase.AllglobaldistributedtransactionsperformedthroughtheDTPserviceareensuredtohavetheirtightlycoupledbranchesrunningonasingleOracleRACinstance.
OracleRAC11gprovidestransparentsupportforXAglobaltransactionsinanOracleRACenvironmentandyoudonotneedtouseDTPservices.

CanIrunOracle9iRACandOracleRAC10ginthesamecluster?
YES.HoweverOracleClusterware(CRS)willnotsupportaOracle9iRACdatabasesoyouwillhavetoleavethecurrentconfigurationinplace.YoucaninstallOracleClusterwareand
OracleRAC10gintothesamecluster.OnWindowsandLinux,youmustrunthe9iClusterManagerforthe9iDatabaseandtheOracleClusterwareforthe10gDatabase.Whenyou
installOracleClusterware,your9isrvconfigfilewillbeconvertedtotheOCR.BothOracle9iRACandOracleRAC10gwillusetheOCR.Donotrestartthe9igsdafteryouhaveinstalled
OracleClusterware.WithOracleClusterware11gRelease2,theGSDresourcewillbedisabledbydefault.YouonlyneedtoenablethisresourceifyouarerunningOracle9iRACinthe
clsuter.
RemembertocheckcertifyfordetailsofwhatvendorclusterwarecanberunwithOracleClusterware.
ForexampleonSolaris,yourOracle9iRACwillbeusingSunCluster.YoucaninstallOracleClusterwareandOracleRAC10ginthesameclusterthatisrunningSunClusterandOracle9i
RAC.

WhatstorageissupportedwithStandardEditionOracleRAC?
Asperthelicensingdocumentation,youmustuseASMforalldatabasefileswithSEOracleRAC.ThereisnosupportforCFSorNFS.
FromOracleDatabase10gRelease2LicensingDoc:
OracleStandardEditionandOracleRealApplicationClusters(RAC)WhenusedwithOracleRealApplicationClustersinaclusteredserverenvironment,OracleDatabase
StandardEditionrequirestheuseofOracleClusterware.Thirdpartyclusterwaremanagementsolutionsarenotsupported.Inaddition,AutomaticStorageManagement(ASM)mustbe
usedtomanagealldatabaserelatedfiles,includingdatafiles,onlinelogs,archivelogs,controlfile,spfiles,andtheflashrecoveryarea.Thirdpartyvolumemanagersandfilesystems
arenotsupportedforthispurpose.

WhataretherestrictionsontheSIDwithanOracleRACdatabase?Isitlimitedto5characters?
TheSIDprefixin10gRelease1andpriorversionswasrestrictedtofivecharactersbyinstall/configtoolssothatanORACLE_SIDofuptomaxof5+3=8characterscanbesupportedin
anOracleRACenvironment.TheSIDprefixisrelaxedupto8charactersin10gRelease2,seebug4024251formoreinformation.
WithOracleRAC11gRelease2,SIDsinOracleRACwithPolicyManageddatabasearedynamicallyallocatedbythesystemwhentheinstancestarts.Thissupportsadynamicgrid
infrastructurewhichallowstheinstancetostartonanyserverinthecluster.

DoesOracleClusterwareorOracleRealApplicationClusterssupportheterogeneousplatforms?
OracleClusterwareandOracleRealApplicationClustersdonotsupportheterogeneousplatformsinthesamecluster.EnterpriseManagerGridControlsupportsheterogeneous
platforms.Wedosupportmachinesofdifferentspeedsandsizeinthesamecluster.Allnodesmustrunthesameoperatingsystem(I.E.theymustbebinarycompatible).Inanactive
datasharingenvironment,likeOracleRAC,wedonotsupportmachineshavingdifferentchiparchitectures.

IwanttouserconfigtoconvertasingleinstancetoOracleRACbutIamusingrawdevicesinOracleRAC.DoesrconfigsupportRAW?
No.rconfigsupportsASMandsharedfilesystemonly.

HowmanyNICsdoIneedtoimplementOracleRAC?

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

18/72

8/24/2015

HowmanyNICsdoIneedtoimplementOracleRAC?

Document220970.1

Atminimumyouneed2:external(public),interconnect(private).WhenstorageforOracleRACisprovidedbyEthernetbasednetworks(e.g.NAS/nfsoriSCSI),youwillneedathird
interfaceforI/Osoaminimumof3.Anythingelsewillcauseperformanceandstabilityproblemsunderload.FromanHAperspective,youwantthesetoberedundant,thusneedinga
totalof6.

CanwedesignatetheplaceofarchivelogsonbothASMdiskandregularfilesystem,whenweuseSERAC?
Yes,customersmaywanttocreateastandbydatabasefortheirSERACdatabasesoplacingthearchivelogsadditionallyoutsideASMisOK.

CanmycustomeruseVeritasAgentstomanagetheirOracleRACdatabaseonUnixwithSFRACinstalled?
FordetailsonthesupportofSFRACandVeritasAgentswithRAC10g,pleaseseeNote397460.1Oracle'sPolicyforSupportingOracleRAC10g(appliestoOracleRAC11gtoo)with
SymantecSFRAConUnixandNote332257.1UsingOracleClusterwarewithVendorClusterwareFAQ

CanIrunmorethanoneclustereddatabaseonasingleOracleRACcluster?
YoucanrunmultipledatabasesinaOracleRACcluster,eitheroneinstancepernode(w/differentdatabaseshavingdifferentsubsetsofnodesinacluster),ormultipleinstancesper
node(alldatabasesrunningacrossallnodes)orsomecombinationinbetween.Runningmultipleinstancespernodedoescausememoryandresourcefragmentation,butthisisno
differentfromrunningmultipleinstancesonasinglenodeinasingleinstanceenvironmentwhichisquitecommon.ItdoesprovidetheflexibilityofbeingabletoshareCPUonthenode,
buttheOracleResourceManagerwillnotcurrentlylimitresourcesbetweenmultipleinstancesononenode.YouwillneedtouseanOSlevelresourcemanagertodothis.

CanIrunOracleRAC10gwithOracleRAC11g?
Yes.TheOracleClusterwareshouldalwaysrunatthehighestlevel.WithOracleClusterware11g,youcanrunbothOracleRAC10gandOracleRAC11gdatabases.IfyouareusingASM
forstorage,youcanuseeitherOracleDatabase10gASMorOracleDatabase11gASMhowevertogetthe11gfeatures,youmustberunningOracleDatabase11gASM.Itis
recommendedtouseOracleDatabase11gASM.
Note:Whenyouupgradeto11gRelease2,youmustupgradebothOracleClusterwareandAutomaticStorageManagementto11gRelease2.ThiswillsupportOracleDatabase10gand
OracleDatabase11g(bothRACandsingleinstance).
Yes,youcanrunOracle9iRACintheclusteraswell.9iRACrequirestheclusterwarethatiscertifiedwithOracle9iRACtoberunninginadditiontoOracleClusterware11g.

CanIhavemultiplepublicnetworksaccessingmyOracleRAC?
Yes,youcanhavemultiplenetworkshoweverwithOracleRAC10gandOracleRAC11g,theclustercanonlymanageasinglepublicnetworkwithaVIPandthedatabasecanonlyload
balanceacrossasinglenetwork.FANwillonlyworkonthepublicnetworkwiththeOracleVIPs.
OracleRAC11gRelease2supportsmultiplepublicnetworks.Youmustsetthenewinit.oraparameterLISTENER_NETWORKSsousersareloadbalancedacrosstheirnetwork.Services
aretiedtonetworkssousersconnectingwithnetwork1willuseadifferentservicethannetwork2.EachnetworkwillhaveitsownVIP.

IcouldnotgettheuserequivalencechecktoworkonmySolaris10serverwhentryingtoinstall10.2.0.1OracleClusterware.Theinstallranfinewithoutissue.
<<Message:Result:Userequivalencecheckfailedforuser"oracle".>>
CluvfyandtheOUItriestofindSSHonSolarisat/usr/local/bin.Workaroundistocreateasoftlinkfrom/usr/bin/sshto/usr/local/bin.
Note:Userequivalenceisrequiredforinstallations(IEusingOUI)andpatching.DBCA,NETCA,andDBControlalsorequireuserequivalence.

IsitsupportedtoinstallOracleClusterwareandOracleRACasdifferentusers?
Yes,OracleClusterwareandOracleRACcanbeinstalledasdifferentusers.TheOracleClusterwareuserandtheOracleRACusermustbothhaveOINSTALLastheirprimarygroup.
EveryDatabasehomecanhaveadifferentOSDBAgroupwithadifferentusername.

WhydoestheNOACattributeneedtobesetonNFSmountedRACBinaries?
Thenoacattributeisrequiredbecausetheinstallerdeterminessharednessbycreatingafileandcheckingforthatfilesexistanceonremotenode.Ifthenoacattributeisnotenabled
thenthistestwillincorrectlyfail.Thiswillconfuseinstallerandopatch.Someotherminorissuesissueswithspfileinthedefault$ORACLE_HOME/dbswilldefinitelybeaffected.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

19/72

8/24/2015

Document220970.1

WeareusingTransparentDataEncryption(TDE).
Wecreateawalletonnode1andcopytonodes2&3.Openthewalletandweareabletoselectencrypteddataonallthreenodes.
Now,wewanttoREKEYtheMASTERKEY.Whatdowehavetodo?
Afterarekeyonnodeone,'altersystemsetwalletclose'onallothernodes,copythewalletwiththenewmasterkeytoallothernodes,'altersystemsetwalletopenidentifiedby
"password"onallothernodestoloadthe(obfuscated)masterkeyintonode'sSGA.

HowdoIcheckfornetworkproblemsonmyinterconect?
1.Confirmthatfullduplexissetcorrectlyforallinterconnectlinksonallinterfacesonbothends.Donotrelyonautonegotiation.
2.ifconfigawillgiveyouanindicationofcollisions/errors/overunsanddroppedpackets
3.netstatswillgiveyoualistingofreceivepacketdiscards,fragmentationandreassemblyerrorsforIPandUDP.
4.Settheudpbufferscorrectly
5.Checkyourcabling
Note:IfyouareseeingissueswithRAC,RACusesUDPastheprotocol.OracleClusterwareusesTCP/IP.

TheVeritasinstallationdocumentonpage219asksforsettingLD_LIBRARY_PATH_64.ShouldIremovethis?
YesYoudonotneedtosetLD_LIBRARY_PATHforOracle.

WhydoesnetcaalwayscreatesthelistenerwhichlistenstopublicipandnotVIPonly?
Thisisforbackwardcompatibilitywithexistingclients:considerpre10gto10gserverupgrade.IfwemadeupgradedlistenertoonlylistenonVIP,thenclientsthatdidn'tupgradewill
notbeabletoreachthislisteneranymore.

DoeschanginguidorgidoftheOracleUseraffectOracleClusterware?
TherearealotoffilesintheOracleClusterwarehomeandoutsideoftheOracleClusterwarehomethatarechgrp'edtotheappropriategroupsforsecurityandappropriateaccess.The
filesystemrecordstheuid(nottheusername),andsoifyouexchangethenames,nowthefilesareownedbythewronggroup.

Canweoutputthebackupsetontoregularfilesystemdirectly(notontoflashrecoveryarea)usingRMANcommand,whenweuseSERAC?
Yes,customersmightwanttobackuptheirdatabasetoofflinestoragesothisisalsosupported.

HowdoIuseDBCAinsilentmodetosetupRACandASM?
IfyoualreadyhaveanASMinstance/diskgroupthenthefollowingcreatesaRACdatabaseonthatdiskgroup(runastheOracleuser):
$ORACLE_HOME/bin/dbcasilentcreateDatabasetemplateNameGeneral_Purpose.dbcgdbName$SIDsid$SIDsysPassword$PASSWORDsystemPassword$PASSWORD
sysmanPassword$PASSWORDdbsnmpPassword$PASSWORDemConfigurationLOCALstorageTypeASMdiskGroupName$ASMGROUPNAMEdatafileJarLocation
$ORACLE_HOME/assistants/dbca/templatesnodeinfo$NODE1,$NODE2charactersetWE8ISO8859P1obfuscatedPasswordsfalsesampleSchemafalseoratabLocation/etc/oratab
ThefollowingwillcreateaASMinstance&1diskgroup(runastheASM/Oracleuser)
$ORA_ASM_HOME/bin/dbcasilentconfigureASMgdbNameNOsidNOemConfigurationNONEdiskList$ASM_DISKSdiskGroupName$ASMGROUPNAMEnodeinfo$NODE1,$NODE2
obfuscatedPasswordsfalseoratabLocation/etc/oratabasmSysPassword$PASSWORDredundancy$ASMREDUNDANCY
whereASM_DISKS='/dev/sda1,/dev/sdb1'andASMREDUNDANCY='NORMAL'

CanRMANbackupOracleRealApplicationClusterdatabases?
Absolutely.RMANcanbeconfiguredtoconnecttoallnodeswithintheclustertoparallelizethebackupofthedatabasefilesandarchivelogs.Iffilesneedtoberestored,usingset
AUTOLOCATEONalertsRMANtosearchforbackedupfilesandarchivelogsonallnodes.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

20/72

8/24/2015

AUTOLOCATEONalertsRMANtosearchforbackedupfilesandarchivelogsonallnodes.

Document220970.1

IamreceivinganORA29740error.WhatshouldIdo?
Thiserrorcanoccurwhenproblemsaredetectedonthecluster:
Error:ORA29740(ORA29740)
Text:evictedbymember%s,groupincarnation%s

Cause:Thismemberwasevictedfromthegroupbyanothermemberofthe
clusterdatabaseforoneofseveralreasons,whichmayincludea
communicationserrorinthecluster,failuretoissueaheartbeat
tothecontrolfile,etc.
Action:Checkthetracefilesofotheractiveinstancesinthecluster
groupforindicationsoferrorsthatcausedareconfiguration.
Formoreinformationontroubleshootingthiserror,seethefollowingnote:
Note:219361.1TroubleshootingORA29740inaRACEnvironment

DoesOraclesupportrollingupgradesinacluster?
ThisanswerisforclustersrunningtheOraclestack.If3rdpartyvendorclusterwareinincluded,youneedtocheckwiththevendorabouttheirsupportofarollingupgrade.
Byarollingupgrade,wemeanupgradingsoftware(OracleDatabase,OracleClusterware,ASMortheOSitself)whiletheclusterisoperationalbyshuttingdownanode,upgradingthe
softwareonthatnode,andthenreintegratingitintothecluster,andsoforthonenodeatatimeuntilallthenodesintheclusterareatthenewsoftwarelevel.
FortheOracleDatabasesoftware,itispossibleonlyforcertainsinglepatchesthataremarkedasrollingupgradecompatible.MostBundlepatchesandCriticalPatchUpdates(CPU)
arerollingupgradeable.PatchsetsandDBversion(10gto11g)changesarenotsupportedinarollingfashion,onereasonthatthismaybeimpossibleisthatacrossmajorreleases,
theremaybeincompatibleversionsofthesystemtablespace,forexample.ToupgradetheseinarollingfashiononewillneedtousealogicalstandbywithOracleDatabase10gor11g,
seeNote:300479.1fordetails.
ReadtheMAABestPracticeonRollingDatabaseUpgradesusingDataGuardSQLApplyorwithOracleRAC11g,RollingDatabaseUpgradesforPhysicalStandbyDatabasesusing
TransientLogicalStandby11g
TheOracleClusterwaresoftwarealwaysfullysupportsrollingupgrades,whiletheASMsoftwareisrollingupgradeableatversion11.1.0.6andbeyond.
ForOracleDatabase11gRelease2,OracleClusterwareandASMbinariesarecombinedintoasingleORACLE_HOMEcalledthegridinfrastructurehome.Thishomefullysupports
rollingupgradesforpatches,bundles,patchsetsandreleases.(IfyouareupgradingASMfromOracleDatabase10gto11gRelease2,youwillnotbeabletoupgradeASMinarolling
fashion.)
TheOracleClusterwareandOracleRealApplicationClustersbothsupportrollingupgradesoftheOSsoftwarewhentheversionoftheOracleDatabaseiscertifiedonbothreleasesof
theOS(andtheOSisthesame,noLinuxandWindowsorAIXandSolaris,or32and64bitetc.).Thiscanapplyapatchtotheoperatingsystem,apatchset(suchasEL4u4toEL4u6)or
arelease(EL4toEL5).
Staywithina24hoursofupgradewindowandfullytestthispathasit'snotpossibleforOracletotestallthesedifferentpathsandcombinations.

IsOracleApplicationServerintegratedwithFANandFCF?
Yes,FordetailedinformationontheintegrationwiththevariousreleasesofApplicationServer10g,
http://www.oracle.com/technology/tech/java/newsletter/articles/oc4j_data_sources/oc4j_ds.htm

HowdoIconfigureFCFwithBPELsoIcanuseOracleRAC10ginthebackend?
Note:372456.1describestheproceduretosetupBPELwithaOracleRAC10gRelease1database.
IfyouareusingSSL,ensuretheSSLenableattributeofONSinopmn.xmlfilehassamevalue,eithertrueorfalse,forallOPMNserversintheFarm.TotroubleshootOPMNatthe
applicationserverlevel,lookatappendixAinOracleProcessManagerandNotificationServerAdministrator'sGuide.

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

21/72

8/24/2015

Document220970.1

applicationserverlevel,lookatappendixAinOracleProcessManagerandNotificationServerAdministrator'sGuide.

WherecanIfindmoreinformationoncluster_interconnects?
Oracle9i:Note:183340.1
Oracle10g&11g:Note:787420.1
SolarisIPMPspecific:Note:368464.1
Linkstodocumentation:
Oracle9to11g:Note:151051.1
Aswellas:OracleRealApplicationClustersAdministrationandDeploymentGuide:11gRelease1and10gRelease2

HowdoesOCRmirrorwork?WhathappensifmyOCRislost/corrupt?
OCRistheOracleClusterRegistry,itholdsalltheclusterrelatedinformationsuchasinstances,services.TheOCRfileformatisbinaryandstartingwith10.2itispossibletomirrorit.
Locationoffile(s)islocatedin:/etc/oracle/ocr.locinocrconfig_locandocrmirrorconfig_locvariables.
ObviouslyifyouonlyhaveonecopyoftheOCRanditislostorcorruptthenyoumustrestorearecentbackup,seeocrconfigutilityfordetails,specificallyshowbackupandrestoreflags.
UntilavalidbackupisrestoredtheOracleClusterwarewillnotstartupduetothecorrupt/missingOCRfile.
TheinterestingdiscussioniswhathappensifyouhavetheOCRmirroredandoneofthecopiesgetscorrupt?Youwouldexpectthateverythingwillcontinuetoworkseemlessly.Well..
Almost..Therealanswerdependsonwhenthecorruptiontakesplace.
IfthecorruptionhappenswhiletheOracleClusterwarestackisupandrunning,thenthecorruptionwillbetoleratedandtheOracleClusterwarewillcontinuetofuntionwithout
interruptions.Despitethecorruptcopy.DBAisadvisedtorepairthishardware/softwareproblemthatpreventOCRfromaccessingthedeviceassoonaspossiblealternatively,DBAcan
replacethefaileddevicewithanotherhealthydeviceusingtheocrconfigutilitywithreplaceflag.
IfhoweverthecorruptionhappenswhiletheOracleClusterwarestackisdown,thenitwillnotbepossibletostartitupuntilthefaileddevicebecomesonlineagainorsome
administrativeactionusingocrconfigutilitywithoverwriteflagistaken.WhentheClustewareattemptstostartyouwillseemessagessimilarto:
totalidsets(1),1stset(1669906634,1958222370),2ndset(0,0)myvotes(1),totalvotes(2)
2006071210:53:54.301:[OCRRAW][1210108256]proprioini:disk0(/dev/raw/raw1)doesn'thaveenoughvotes(1,2)
2006071210:53:54.301:[OCRRAW][1210108256]proprseterror:Errorinaccessingphysicalstorage[26]
Thisisbecausethesoftwarecan'tdeterminwhichOCRcopyisthevalidone.IntheaboveexampleoneoftheOCRmirrorswaslostwhiletheOracleClusterwarewasdown.Thereare
3waystofixthisfailure:
a)Fixwhateverproblem(hardware/software?)thatpreventOCRfromaccessingthedevice.
b)Issue"ocrconfigoverwrite"onanyoneofthenodesinthecluster.ThiscommandwilloverwritethevotecheckbuiltintoOCRwhenitstartsup.Basically,ifOCRdeviceisconfigured
withmirror,OCRassigneachdevicewithonevote.Theruleistohavemorethan50%oftotalvote(quorum)inordertosafelymakesuretheavailabledevicescontainthelatestdata.
In2waymirroring,thetotalvotecountis2soitrequires2votestoachievethequorum.Intheexampleabovethereisn'tenoughvotetostartifonlyonedevicewithonevoteis
available.(Intheearlierexample,whileOCRisrunningwhenthedeviceisdown,OCRassign2votetothesurvivingdeviceandthatiswhythissurvivingdevicenowwithtwovotescan
startaftertheclusterisdown).Seewarningbelow
c)Thismethodisnotrecommendtobeperformedbycustomers.Itispossibletomanuallymodifyocr.loctodeletethefaileddeviceandrestartthecluster.OCRwon'tdothevotecheck
ifthemirrorisnotconfigured.Seewarningbelow
EXTREMECAUTIONshouldbeexcersizedifchosingoptionborcabovesincedatalosscanoccurifthewrongfileismanipulated,pleasecontactOracleSupportforassistancebefore
proceeding.
@Bug5055145wasthebasisforthisFAQ,alsothankstoKenLeeforhisvaluablefeedback.

IfIchangemyclusterconfiguration,doIneedtoupdatetheONSconfigurationonmymiddletier?
ForthebestavailabilityandtoensuretheapplicationreceivesallFANevents,yes,youshouldupdatetheconfiguration.Toacertaindegree,ONSwilldiscovernodes.ONSrunsoneach
nodeintheclusterandisawareofallothernodesinthecluster.AslongaswhenONSonthemiddletiercanfindatleastonenodeintheclusterwhenitstarts,itwillfindtherestofthe
nodes.Inthecasewheretheonlynodeupisthenewnodeintheclusterwhenthemiddletierstarts,themiddletierwillnotfindthecluster.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

22/72

8/24/2015

Document220970.1

WhydowehaveaVirtualIP(VIP)inOracleRAC10gor11g?Whydoesitjustreturnadeadconnectionwhenitsprimarynodefails?
Thegoalisapplicationavailability.
Whenanodefails,theVIPassociatedwithitisautomaticallyfailedovertosomeothernode.Whenthisoccurs,thefollowingthingshappen.
(1)VIPdetectspublicnetworkfailurewhichgeneratesaFANevent.
(2)thenewnoderearpstheworldindicatinganewMACaddressfortheIP.
(3)connectedclientssubscribingtoFANimmediatelyreceiveORA3113errororequivalent.ThosenotsubscribingtoFANwilleventuallytimeout.
(4)Newconnectionrequestsrapidlytraversethetnsnames.oraaddresslistskippingoverthedeadnodes,insteadofhavingtowaitonTCPIPtimeouts
WithoutusingVIPsorFAN,clientsconnectedtoanodethatdiedwilloftenwaitforaTCPtimeoutperiod(whichcanbeupto10min)beforegettinganerror.
Asaresult,youdon'treallyhaveagoodHAsolutionwithoutusingVIPsandFAN.TheeasiestwaytouseFANistouseanintegratedclientwithFastConnectionFailover(FCF)suchas
JDBC,OCI,orODP.NET.

WhatdotheVIPresourcesdooncetheydetectanodehasfailed/gonedown?AretheVIPsautomaticallyacquired,andpublished,orismanualintervention
required?AreVIPsmandatory?
WithOracleRAC10gorhigher,eachnoderequiresaVIP.WithOracleRAC11gRelease2,3additionalSCANvipsarerequiredforthecluster.Whenanodefails,theVIPassociated
withthefailednodeisautomaticallyfailedovertooneoftheothernodesinthecluster.Whenthisoccurs,twothingshappen:
1. ThenewnoderearpstheworldindicatinganewMACaddressforthisIPaddress.Fordirectlyconnectedclients,thisusuallycausesthemtoseeerrorsontheirconnectionsto
theoldaddress
2. SubsequentpacketssenttotheVIPgotothenewnode,whichwillsenderrorRSTpacketsbacktotheclients.Thisresultsintheclientsgettingerrorsimmediately.
InthecaseofexistingSQLconenctions,errorswilltypicallybeintheformofORA3113errors,whileanewconnectionusinganaddresslistwillselectthenextentryinthelist.Without
usingVIPs,clientsconnectedtoanodethatdiedwilloftenwaitforaTCP/IPtimeoutperiodbeforegettinganerror.Thiscanbeaslongas10minutesormore.Asaresult,youdon't
reallyhaveagoodHAsolutionwithoutusingVIPs.
WithOracleRAC11gRelease2,youcandelegatethemanagementoftheVIPstothecluster.Ifyoudothis,theGridNamingService(partoftheOracleClusterware)willautomatically
allocatedandmanageallVIPsinthecluster.ThisrequiresaDHCPserviceonthepublicnetwork.

IfIuseServiceswithOracleRAC,doIstillneedtosetupLoadBalancing?
Yes,ServicesallowyougranulardefinitionofworkloadandtheDBAcandynamicallydefinewhichinstancesprovidetheservice.ConnectionLoadBalancing(providedbyOracleNet
Services)stillneedstobesetuptoallowtheuserconnectionstobebalancedacrossallinstancesprovidingaservice.WithOracleRAC10gRelease2orhigher,settheCLB_GOALon
servicetodefinethetypeofloadbalancingyouwant,SHORTforshortlivedconnections(IEconnectionpool)orLONG(default)forapplciationsthathaveconnectionsactiveforlong
periods(IEOracleFormsapplicaiton).

Howcanacustomermaskthechangeintheirclustereddatabaseconfigurationfromtheirclientorapplication?(I.E.SoIdonothavetochangetheconnection
stringwhenIaddanodetotheOracleRACdatabase)
ThecombinationofServerSideloadbalancingandServicesallowsyoutoeasilymaskclusterdatabaseconfigurationchanges.Aslongasallinstancesregisterwithalllisteners(usethe
LOCAL_LISTENERandREMOTE_LISTENERparameters),serversideloadbalancingwillallowclientstoconnecttotheserviceoncurrentlyavailableinstancesatconnecttime.
Theloadbalancingadvisory(settingagoalontheservice)willgiveadviceastohowmanyconnectionstosendtoeachinstancecurrentlyprovidingaservice.Whenaserviceisenabled
onaninstance,aslongastheinstanceregisterswiththelisteners,theclientscanstartgettingconnectionstotheserviceandtheloadbalancingadvisorywillincludethatinstanceisits
advice.
WithOracleRAC11gRelease2,theSingleClientAccessName(SCAN)providesasinglenametobeputintheclientconnectionstring(astheaddress).ClientsusingSCANnever
havetochangeeveniftheclusterconfigurationchangessuchasaddingnodes.

AfterexecutingDBMS_SERVICE.START_SERVICE,theserviceresourceremainsinanOFFLINEstatuswhenIdisplaytheclusterresourcestatus.Isthat
expectedbehaviour?
YESthisisexpectedbehaviour.Unfortunately,theDBMS_SERVICE.START_SERVICEdoesnotupdatetheclusterwareuntil11gRelease2.Youshouldusesrvctlstartserviceddbname
thenyoushouldseeitcomeonline.
Note:WithOracleRAC11gRelease2,theclusterresourceforaService,containsthevaluesforalltheattributesofaservice.OracleClusterwarewillupdatethedatabasewithits
valueswhenitstartsaservice.Inordertosavemodificationsacrossrestarts,allservicemodificationsshouldbemadewithsrvctl(orOracleEnterpriseManager).
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

23/72

8/24/2015

Document220970.1

WhataremyoptionsforloadbalancingwithOracleRAC?WhydoIgetanunevennumberofconnectionsonmyinstances?
Allthetypesofloadbalancingavailablecurrently(9i10g)occuratconnecttime.
Thismeansthatitisveryimportanthowonebalancesconnectionsandwhattheseconnectionsdoonalongtermbasis.
Sinceestablishingconnectionscanbeveryexpensiveforyourapplication,itisgoodprogrammingpracticetoconnectonceandstayconnected.Thismeansoneneedstobecarefulasto
whatoptiononeuses.OracleNetServicesprovidesloadbalancingoryoucanuseexternalmethodssuchashardwarebasedorclusterwaresolutions.
ThefollowingoptionsexistpriortoOracleRAC10gRelease2(for10gRelease2seeLoadBalancingAdvisory):
Random
Eitherclientsideloadbalancingorhardwarebasedmethodswillrandomizetheconnectionstotheinstances.
OnthenegativesidethismethodisunawareofloadontheconnectionsoreveniftheyareupmeaningtheymightcausewaitsonTCP/IPtimeouts.
LoadBased
Serversideloadbalancing(bythelistener)redirectsconnectionsbydefaultdependingontheRunQlengthofeachoftheinstances.Thisisgreatforshortlivedconnections.Terriblefor
persistentconnectionsorloginstorms.Donotusethismethodforconnectionsfromconnectionpoolsorapplicatonservers
SessionBased
Serversideloadbalancingcanalsobeusedtobalancethenumberofconnectionstoeachinstance.Sessioncountbalancingismethodusedwhenyousetalistenerparameter,
prefer_least_loaded_node_listenername=off.Notelistenernameistheactualnameofthelistenerwhichisdifferentoneachnodeinyourclusterandbydefaultislistener_nodename.
Sessionbasedloadbalancingtakesintoaccountthenumberofsessionsconnectedtoeachnodeandthendistributestheconnectionstobalancethenumberofsessionsacrossthe
differentnodes.

WhatdoIdoifIamgettinghandshakefailedmessagesinmyONS.LOGfileeveryminute?
ForExample:TheclientgetsthiserrormessageinProductionintheons.logfileeveryminuteorso:06/11/1010:11:14[2]Connection0,129.86.186.58,6200SSLhandshakefailed
06/11/1010:11:14[2]Handshakefor0,129.86.186.58,6200:nzerror=29049interval=0(180max)Theseannoyingmessagesinons.logaretellingyouthatyouhaveaconfiguration
mismatchforONSsomewhereinthefarm.OracleRAChasitsownONSserverforwhichSSLisdisabledbydefault.YoumusteitherenableSSLforOracleRACONS,ordisableitforOID
ONS(OPMN).YouneedtocreateawalletforeachOracleRACONSserver,orcopyoneofthewalletsfromOPMNontheOIDinstances.
Inons.confyouneedtospecifythewalletfileandpassword:
walletfile=
walletpassword=
ONSonlyusesSSLbetweenservers,andsoONSclientswillnotbeaffected.Youspecifythewalletpasswordwhenyoucreatethewallet.IfyoucopyawalletfromanOPMNinstance,
thenusethesamepasswordconfiguredinopmn.xml.Ifthereisnowalletpasswordconfiguredinopmn.xml,thenyoudon'tneedtospecifyawalletpasswordinons.confeither.

WhatshouldIdotomakemyOracleRACdeploymenthighlyavailable?
CustomersoftendeployOracleRealApplicationClusters(RAC)toprovideahighlyavailableinfrastructurefortheirmissioncriticalapplications.OracleRACremovestheserverasa
singlepointoffailure.Loadbalancingyourworkloadacrossmanyserversalongwithfastrecoveryfromfailuresmeansthatthelossofanyoneservershouldhavelittleornoimpacton
theenduseroftheapplication.Thelevelofimpacttotheenduserdependsonhowwelltheapplicationhasbeenwrittentomaskfailure.IfanoutageoccursonanOracleRACinstance,
theidealsituationwouldbethatthefailovertime+transactionresponsetimetobelessthenthemaximumacceptableresponsetime.OracleRAChasmanyfeaturesthatcustomerscan
takeadvantageoftomaskfailuresfromtheenduserhoweveritrequiresmoreworkthanjustinstallingOracleRAC.Totheapplicationuser,theavailabilitymetricthatmeansthemost
istheresponsetimefortheirtransaction.Thisistheendtoendresponsetimewhichmeansalllayersmustbeavailableandperformingtoadefinedstandardfortheagreedtimes.
IfyouaredeployingOracleRACandrequirehighavailability,youmustmaketheentireinfrastructureoftheapplicationhighlyavailable.Thisrequiresdetailedplanningtoensurethere
arenosinglepointsoffailurethroughouttheinfrastructure.OracleClusterwareisconstantlymonitoringanyprocessthatitunderitscontrol,whichincludesalltheOraclesoftwaresuch
astheOracleinstance,listener,etc.OracleClusterwarehasbeenprogrammedtorecoverfromfailures,whichoccurfortheOracleprocesses.Inordertodoitsmonitoringand
recovery,varioussystemactivitieshappenonaregularbasissuchasuserauthentication,sudo,andhostnameresolution.Inorderfortheclustertobehighlyavailable,itmustbeableto
performtheseactivitiesatalltimes.Forexample,ifyouchoosetousetheLightweightDirectoryAccessProtocol(LDAP)forauthentication,thenyoumustmaketheLDAPserverhighly
availableaswellasthenetworkconnectingtheusers,application,databaseandLDAPserver.IfthedatabaseisupbuttheuserscannotconnecttothedatabasebecausetheLDAP
serverisnotaccessible,thentheentiresystemisdownintheeyesofyourusers.WhenusingexternalauthenticationsuchasLDAPorNIS(NetworkInformationService),apublic
networkfailurewillcausefailureswithinthecluster.Oraclerecommendsthatthehostname,vip,andinterconnectaredefinedinthe/etc/hostsfileonallnodesinthecluster.
DuringthetestingoftheOracleRACimplementation,youshouldincludeadestructivetestingphase.Thisisasystematicsetoftestsofyourconfigurationtoensurethat1)youknow
whattoexpectifthefailureoccursandhowtorecoverfromitand2)thatthesystembehavesasexpectedduringthefailure.Thisisagoodtimetoreviewoperatingproceduresand
documentrecoveryprocedures.Destructivetestingshouldincludetestssuchasnodefailure,instancefailure,publicnetworkfailure,interconnectfailures,storagefailure,storage
networkfailure,votingdiskfailure,lossofanOCR,andlossofASM.
UsingfeaturesofOracleRealApplicationClustersandOracleClientsincludingFastApplicationNotification(FAN),FastConnectionFailover(FCF),OracleNetServiceConnectionLoad
Balancing,andtheLoadBalancingAdvisory,applicationscanmaskmostfailuresandprovideaveryhighlyavailableapplication.Fordetailsonimplementingbestpractices,seetheMAA
documentClientFailoverBestPracticesforHighlyAvailableOracleDatabasesandtheOracleRACAdministrationandDeploymentGuide.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

24/72

8/24/2015

Document220970.1

CanourOracleRAC10gVIPfailoverfromNICtoNICaswellasfromnodetonode?
Yes,theOracleRAC10gVIPimplementationiscapablefromfailingoverwithinanodefromNICtoNICandbackifthefailedNICisbackonlineagain,andalsowefailoverbetween
nodes.TheNICtoNICfailoverisfullyredundantifredundantswitchesareinstalled.

IsthereawaytoprovideorconfigureHAfortheinterconnectusingInfinibandonAIX?
TheHAsupportwillbewithVIPAconfiguredovertwoseparateIBinterfaces.Thetwointerfacescaneitherbetwoportsononeadapter(notidealHA)ortwoportsfromdifferent
adapters.ThisVIPAconfigurationisdifferentfromthe"AIXEtherchannel"configuration."AIXEtherchannel"isnotsupportedwithInfiniband

Iamusingsharedserviceswhichthefollowingsetininit.oraSQL>showparametersdispatchers=(protocol=TCP)(listener=listeners_nl01)(con=500)(serv=oltp).
Istoppedmyservicewithsrvctlstopservicebutitisstillregisteredwiththelistenerandacceptingconnections.Isthisexpected?
YES.ThisisbydesignofdispatcherswhicharepartofOracleNetServices.Ifyouspecifytheserviceattributeofthedispatchersinit.oraparameter,theservicespecifiedcannotbe
managedbythedba.

IsitpossibletouseSVRCTLstartdatabasewithauseraccountotherthanoracle(thatisotherthantheowneroftheoraclesoftware)?
YES.WhenyoucreateaRACdbasauserdifferentthanthehome/softwareowner(oracle)user,thedbcreationassistantwouldsetthecorrectpermissions/ACLsontheCRSresources
thatcontrolthedb/instancesetc,assumingthatyouhadsetupgroupmembershipforthisusertothedbagroupofthehome(finditusingoracle_home/bin/osdbagrp)andalsopartof
thecrshomeownersprimarygroup(usuallyoinstall)andtherewasgroupwritepermissionontheoracle_home.

Withthreeprimaryloadbalancingoptions(clientsideconnecttimeLB,serversideconnecttimeLB,andtheruntimeconnectionloadbalancing)Isitfairtosay
RuntimeConnectionLoadBalancingistheonlyoptiontoleverageFANup/downevents?
No.ThelistenerisasubscribertoallFANevents(bothfromtheloadbalancingadvisoryandtheHAevents).ThereforeserversideconnectionloadbalancingleveragesFANHAevents
aswellaslaodbalancingadvisoryevents.
WiththeOracleJDBCdriver10gRelease2,ifyouenableFastConnectionFailover,youalsoenableRuntimeConnectionLoadBalancing(oneknobforboth).

WhatisServersideTransparentApplicationFailover(TAF)andhowdoIuseit?
OracleDatabase10gRelease2,introducesserversideTAFwhenusingservices.Afteryoucreateaservice,youcanusethedbms_service.modify_servicepl/sqlproceduretodefinethe
TAFpolicyfortheservice.Onlythebasicmethodissupported.NotethisisdifferentthantheTAFpolicy(traditionalclientTAF)thatissupportedbysrvctlandEMServicespage.Ifyour
servicehasaserversideTAFpolicydefined,thenyoudonothavetoencodeTAFontheclientconnectionstring.Iftheinstancewhereaclientisconnected,fails,thentheconnection
willbefailedovertoanotherinstanceintheclusterthatissupportingtheservice.AllrestrictionsofTAFstillapply.
NOTE:boththeclientandservermustbe10.2andaq_ha_notificationsmustbesettotruefortheservice.
Samplecodetomodifyservice:
executedbms_service.modify_service(service_name=>'gl.us.oracle.com'
,aq_ha_notifications=>true
,failover_method=>dbms_service.failover_method_basic
,failover_type=>dbms_service.failover_type_select
,failover_retries=>180
,failover_delay=>5
,clb_goal=>dbms_service.clb_goal_long)

WhatisCLB_GOALandhowshouldIsetit?
CLB_GOAListheconnectionloadbalancinggoalforaservice.Thereare2options,CLB_GOAL_SHORTandCLB_GOAL_LONG(default).
Longisforapplicationsthathavelonglivedconnections.ThisistypicalforconnectionpoolsandSQL*Formssessions.Longisthedefaultconnectionloadbalancinggoal.
Shortisforapplicationsthathaveshortlivedconnections.
TheGOALforaservicecanbesetwithEMorDBMS_SERVICE.
Note:YoumuststillconfigureloadbalancingwithOracleNetServices
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

25/72

8/24/2015

Document220970.1

WhatdoestheVirtualIPservicedo?Iunderstanditisforfailoverbutdoweneedaseparatenetworkcard?Canweusetheexistingprivate/publiccards?What
wouldhappenifweusedthepublicip?
The10gVirtualIPAddress(VIP)existsoneveryRACnodeforpublicnetworkcommunication.AllclientcommunicationshouldusetheVIPsintheirTNSconnectiondescriptions.The
TNSADDRESS_LISTentryshoulddirectcliensttoVIPsratherthanusinghostnames.Duringnormalruntime,thebehaviouristhesameashostnames,howeverwhenthenodegoes
downorisshutdowntheVIPishostedelsewhereonthecluster,anddoesnotacceptconnectionrequests.ThisresultsinasilentTCP/IPerrorandtheclientfailsimmediatelytothenext
TNSaddress.Ifthenetworkinterfacefailswithinthenode,theVIPcanbeconfiguredtousealternateinterfacesinthesamenode.TheVIPmustusethepublicinterfacecards.Thereis
norequirementtopurchaseadditionalpublicinterfacecards(unlessyouwanttotakeadvantageofwithinnodecardfailover.)

IwanttoconfigureasecureenvironmentforONSsohaveaddedaWallethoweverIamseeingerrors(SSLhandshakefailed)afteraddingthewallet?
RememberthatifyouenableSSLforoneinstanceofONS,youmustenableSSLforallinstanceswithONS(includinganyASinstancesrunningOPMN).
TheerrormessageinthiscaseshowedthatSSLisenabledforthelocalONSserver,buttheSSLhandshakeisfailingwhenanotherONSorOPMNserverattemptstoconnecttoit,
indicatingthattheremoteserverdoesnothaveSSLenabled(orhasanincompatiblewalletconfigured).

DoIneedtoinstalltheONSonallmymidtierservesinordertoenableJDBCFastConnectionFailover(FCF)?
With10gRelease1,themiddletiermusthaveONSrunning(startedbysameusersasapplication).ONSisnotincludedontheClientCDhoweverisispartoftheOracleDatabase10g
cd.
With10gRelease2orlater,theydonotneedtoinstalltheonsonthemiddletier.TheJDBCdriverallowstheuseofremoteONS(ieusestheONSrunningintheRACcluster).Justuse
thedatasourceparameterods.setONSConfiguration("nodes=racnode1:4200,racnode2.:4200")

WillFAN/FCFworkwiththedefaultdatabaseservice?
No.IfyouwanttheadvancedfeaturesofRACprovidedbyFANandFCF,thencreateaclustermanagedserviceforyourapplication.UsetheClusteredManagedServicesPagein
EnterpriseManagerDBControltodothis.

WillFANworkwithSQLPlus?
YeswithOracleRAC11g,youcanspecifytheF(FAILOVER)option.ThisenablesSQL*PlustointeractwiththeOCIfailovermodeinaRealApplicationCluster(RAC)environment.Inthis
modeaserviceorinstancefailureistransparentlyhandledwithtransactionstatusmessagesifapplicable.

WhyamIseeingthefollowingwarningsinmylistener.logformyRAC10genvironment?
WARNING:Subscriptionfornodedowneventstillpending
ThismessageindicatesthatthelistenerwasnotabletosubscribetotheONSeventswhichitusestodotheconnectionloadbalancing.Thisismostlikelyduetostartingthelistener
usinglsnrctlfromthedatabasehome.Whenyoustartthelistenerusinglsnrctl,makesureyouhavesettheenvironmentvariableORACLE_CONFIG_HOME={OracleClusterware
HOME},alsosetitinracgwrapinthe$ORACLE_HOME/binforthedatabase.

CanIusethe10.2JDBCdriverwith10.1databaseforFCF?
YeswiththepatchforBug5657975for10.2.0.3,the10.2JDBCdriverwillworkwitha10.1database.Thefixwillbepartofthe10.2.0.4patchset.Ifyoudonothavethepatchthenusing
FCF,usethe10.2JDBCdriverwith10.2database.Ifdatabaseis10.1,use10.1JDBCdriver.

WhatclientsprovideintegrationwithFANthroughFCF?
WithOracleDatabase10gRelease1,JDBCclients(boththickandthindriver)areintegratedwithFANbyprovidingFCF.WithOracleDatabase10gRelease2,wehaveaddedODP.NET
andOCI.OtherapplicationscanintegratewithFANbyusingtheAPItosubscribetotheFANevents.
Note:Ifyouareusinga3rdpartyapplicationserver,thenyoucanonlyuseFCFifyouusetheOracledriverandexceptforOCI,itsconnectionpool.Ifyouareusingtheconnectionpool
ofthe3rdPartyApplicationServer,thenyoudonotgetFCF.YourcustomercansubscribedirectlytoFANeventshoweverthatisadevelopmentprojectforthecustomer.Seethewhite
paperWorkloadManagementwithOracleRAC10gonOTN

CanIuseTAFandFAN/FCF?

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

26/72

8/24/2015

CanIuseTAFandFAN/FCF?

Document220970.1

WithOracleDatabase10gRelease1,NO.WithOracleDatabase10gRelease2,theanswerisYESforOCIandODP.NET,itisrecommended.ForJDBC,youshouldnotuseTAFandFCF
evenwiththeThickJDBCdriver.

HowdoesthedatasourcepropertiesinitialLimit,minLimit,andmaxLimitaffectFastConnectionFailoverprocessingwithJDBC?
TheinitialLimitpropertyontheImplicitConnectionCacheiseffectiveonlywhenthecacheisfirstcreated.Forexample,iftheinitialLimitissetto10,you'llhave10connectionspre
createdandavailablewhentheconncacheisfirstcreated.Plsdon'tbeconfusedbetweenminLimitandinitialLimit.ThecurrentbehavioristhatafteraDOWNeventandtheaffected
connectionsarecleanedup,itispossibleforthenumberofconnectionsinthecachetobelowerthanminLimit.
AnUPeventisprocessedforboth(a)newinstancejoins,aswellas(b)downfollowedbyaninstanceUP.ThishasnorelevancetoinitialLimit,orevenminLimit.WhenaUPevent
comesintoourjdbcImplicitConnectionCache,wewillcreatesomenewconnections.Assumingyouhaveyourlistenerloadbalancingsetupproperly,thenthoseconnectionsshouldgo
totheinstancethatwasjuststarted.Whenyourapplicationdoesagetconnectiontothepool,itwillbegivenanidleconnection,ifyouarerunning10.2andhavetheloadbalancing
advisoryturnedonfortheservice,wewillallocatethesessionbasedonthedefinedgoaltoprovidethebestservicelevel
MaxLimit,whenset,definestheupperboundarylimitfortheconnectioncache.Bydefault,maxLimitisunboundedyourdatabasesetsthelimit.

WillFAN/OCIworkwithInstantClient?
Yes,FAN/OCIwillworkwithInstantClient.BothclientandservermustbeOracleDatabase10gRelease2.

WhattypeofcallbacksaresupportedwithOCIwhenusingFAN/FCF?
Therearetwoseparatecallbackssupported.TheHAEvents(FAN)callbackiscalledwhenaneventoccurs.Whenadowneventoccurs,forexample,youcancleanupacustom
connectionpool.i.e.purgestaleconnections.Whenthefailoveroccurs,theTAFcallbackisinvoked.Atfailovertimeyoucancustomizethenewlycreateddatabasesession.BothFAN
andTAFareclientsidecallbacks.FANalsohasaseparateserversidecalloutthatshouldnotbeconfusedwiththeOCIclientcallback.

DoesFCFforOCIreacttoFANHAUPevents?
OCIdoesnotperformanyimplicitactionsonanupevent,howeverifaHAeventcallbackispresent,itisinvoked.Youcantakeanyrequiredactionatthattime.

CanIuseFAN/OCIwithPro*C?
SincePro*C(sqllib)isbuiltontopofOCI,itshouldsupportHAevents.YouneedtoprecompiletheapplicationwiththeoptionEVENTS=TRUE,makesureyoulinktheapplicationwitha
threadlibrary.ThedatabaseconnectionmustuseaServicethathasbeenenabledforAQevents.Usedbms_service.modify_servicetoenabletheserviceforevents(aq_ha_notifications
=>true)orusetheEMClusterDatabaseServicespage.

DoIhavetolinkmyOCIapplicationwithathreadlibrary?Why?
YES,youmustlinktheapplicationtoathreadslibrary.ThisisrequiredbecausetheAQnotificationsoccurasynchronously,overanimplicitlyspawnedthread.

Iamseeingthewaitevents'gesremotemessage','gcsremotemessage',and/or'gcsforaction'.WhatshouldIdoaboutthese?
Theseareidlewaiteventsandcanbesafetlyignored.The'gesremotemessage'mightshowupina9.0.1statspackreportasoneofthetopwaitevents.Tohavethiswaiteventnot
showupyoucanaddthiseventtothePERFSTAT.STATS$IDLE_EVENTtablesothatitisnotlistedinStatspackreports.

WhatarethechangesinmemoryrequirementsfrommovingfromsingleinstancetoRAC?
Ifyouarekeepingtheworkloadrequirementsperinstancethesame,thenabout10%morebuffercacheand15%moresharedpoolisneeded.Theadditionalmemoryrequirementis
duetodatastructuresforcoherencymanagement.Thevaluesareheuristicandaremostlyupperbounds.Actualesourceusagecanbemonitoredbyqueryingcurrentandmaximum
columnsforthegcsresource/locksandgesresource/locksentriesinV$RESOURCE_LIMIT.
Butingeneral,pleasetakeintoconsiderationthatmemoryrequirementsperinstancearereducedwhenthesameuserpopulationisdistributedovermultiplenodes.Inthiscase:
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

27/72

8/24/2015

Document220970.1

AssumingthesameuserpopulationNnumberofnodesMbuffercacheforasinglesystemthen
(M/N)+((M/N)*0.10)[+extramemorytocompensateforfailedoverusers]
ThusforexamplewithaM=2G&N=2&noextramemoryforfailedoverusers
=(2G/2)+((2G/2))*0.10
=1G+100M

HowcanIvalidatethescalabilityofmysharedstorage?(TightlyrelatedtoRAC/Applicationscalability)
Storagevendorstendtofocustheirsalespitchmainlyonthestorageunit'scapacityinTerabytes(1000GB)orPetabytes(1000TB),howeverforRACscalabilityit'scriticaltoalsolookat
thestorageunit'sabilitytoprocessI/O'spersecond(throughput)inascalablefashion,specificallyfrommultiplesources(nodes).Ifthatcriteriaisnotmet,RAC/Applicationscalability
mostprobablywillsuffer,asitpartiallydependsonstoragescalabilityaswellasasolidandcapableinterconnect(fornetworktrafficebetweennodes).
Storagevendorsmaysometimesdiscouragesuchtesting,boastingabouttheiramazingfrontorbackendbatterybackedmemorycachesthat"eliminate"allI/Obottlenecks.Thisisall
great,andyoushouldtakeadvantageofsuchcachesasmuchaspossible...however,thereisnosubstitutetoarealworldtest,youmayuncoverthattheHBA(HostBusAdapater)
firmwareorthedriverversionsareoutdated(beforeyouclaimpoorRAC/Applicationscalabilityissues).
Itishighlyrecommendedtotestthisstoragescalabilityearlyonsothatexpectationsaresetaccordingly.OnLinuxthereisafreelyavailabletoolreleasedonOTNcalledORION
(OracleI/Otesttool)whichsimulatesOracleI/O.Note:Startingwith11.2theoriontoolisincludedwiththeRDBMS/RACsoftware,seeORACLE_HOME/bin.
OnotherUnixplatforms(aswellasLinux)onecanuseIOzone,ifprebuiltbinarynotavailableyoushouldbuildfromsource,makesuretouseversion3.271orlaterandiftesting
raw/blockdevicesaddthe"I"flag.
InabasicreadtestyouwilltrytodemonstratethatacertainIOthroughputcanbemaintainedasnodesareadded.Trytosimulateyourdatabaseiopatternsasmuchaspossible,i.e.
blocksize,numberofsimultaneousreaders,rates,etc.
Forexample,ona4nodecluster,fromnode1youmeasure20MB/sec,thenyoustartareadstreamonnode2andseeanother20MB/secwhilethefirstnodeshowsnodecrease.You
thenrunanotherstreamonnode3andgetanother20MB/sec,intheendyourun4streamson4nodes,andgetanaggregated80MB/secorclosetothat.Thiswillprovethatthe
sharedstorageisscalable.Obviouslyifyouseepoorscalabilityinthisphase,thatwillbecarriedoverandbeobservedorinterpertedaspoorRAC/Applicationscalability.
InmanycasesRAC/Applicationscalabilityisatblamefornorealreason,thatis,theunderlyingIOsubsystemisnotscalable.

HowmanynodesaresupportedorcanbeusedinanOracleRACDatabase?
TechnicallyandsinceOracleRAC10gRelease2,100nodesaresupportedinonecluster.Thisincludesrunning100databaseinstancesbelongingtothesame(production)
databaseonthiscluster,usingtheOracleDatabaseEnterpriseEdition(EE)withtheOracleRACoptionandOracleClusterwareonly(nothirdparty/vendorclustersolutionunderneath).
Inpreviousreleases,theDBCA(asaresultoffurtherMAXINSTANCESparameterrelatedrestrictions)wouldonlyallowcreating63instancesperdatabase.Theserestrictionshavebeen
liftedwithOracle11gRelease1andlaterversions,infavorofsupporting100nodesasdescribed.Forcompleteness:WithOracleRAC10gRelease1themaximumwas63.InOracle
RAC9ithemaximumisplatformspecificduetothedifferentclustersoftwaresupportbydifferentvendors.
NotethatusingtheOracleDatabaseStandardEdition(SE),whichincludestheOracleRACfunctionality,furtherrestrictionsregardingthenumberofnodesperclusterapply.For
moreinformation,see:SpecialUseLicensing

WhataremyoptionsforsettingtheLoadBalancingAdvisoryGOALonaService?
TheloadbalancingadvisoryisenabledbysettingtheGOALonyourserviceeitherthroughPL/SQLDBMS_SERVICEpackageorEMDBControlClusteredDatabaseServicespage.There
are3optionsforGOAL:
NoneDefaultsetting,turnoffadvisory
THROUGHPUTWorkrequestsaredirectedbasedonthroughput.Thisshouldbeusedwhentheworkinaservicecompletesathomogenousrates.Anexampleisatradingsystem
whereworkrequestsaresimilarlengths.
SERVICE_TIMEWorkrequestsaredirectedbasedonresponsetime.Thisshouldbeusedwhentheworkinaservicecompletesatvariousrates.Anexampleisasinternetshopping
systemwhereworkrequestsarevariouslengths
Note:IfusingGOAL,youshouldsetCLB_GOAL=SHORT

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

28/72

8/24/2015

Document220970.1

WilladdinganewinstancetomyOracleRACdatabase(newnodetothecluster)allowmetoscaletheworkload?
YES!OracleRACallowsyoutodynamicallyscaleoutyourworkloadbyaddinganothernodetothecluster.Youmustrememberthataddingmoreworktothedatabasemeansthatin
additiontotheCPUandMemorythatthenewnodebrings,youwillhavetoensurethatyourI/OsubsystemcansupporttheadditionalI/Orequirements.InanOracleRACenvironment,
youneedtolookatthetotalI/Oacrossallinstancesinthecluster.
@FormoreinformationontestingyourI/Ocapacity,seeFAQ9393

HowdoIchangemyVeritasSFRACinstallationtouseUDPinsteadofLLT?
UsingUDPwithVeritasClusterwareandOracleRAC10gseemstorequireanexceptionfromVeritassothismaybesomethingyoushouldcheckwiththem.
TomakeiteasierforcustomerstoconverttheirLLTenvironmentstoUPD,OraclehascreatedPatch6846006on10.2.0.3whichcontainsthelibrariesthatwereoverwrittenbythe
Veritasinstallation(IEthosementionedabove).ConvertingfromspecializedprotocolstoUDPrequiresarelinkaftertheOraclelibrarieshavebeenrestored.Thisneedsacomplete
clustershutdownandcannotbeaccomplishedinarollingfashion.
NOTE:OracleRAC11gwillnotsupportLLTforinterconnect.

CanIhavedifferentserversinmyOracleRAC?Cantheybefromdifferentvendors?Cantheybedifferentsizes?
OracleRealApplicationClusters(RAC)requiresallthenodestorunthesameOperatingSystembinaryinacluster(IEAllnodesmustbeWindows2008orallnodesmustbeOEL4).All
nodesmustbethesamearchitecture(I.E.Allnodesmustbeeither32bitorallnodesmustbe64bitorallnodesmustbeHPUXPARISCsinceyoucannotmixPARISCwithItanium).
OracleRACdoessupportaclusterwithnodesthathavedifferenthardwareconfigurations.Anexampleisaclusterwith3nodeswith4CPUsandanothernodewith6CPUs.Thiscan
easilyoccurwhenaddinganewnodeaftertheclusterhasbeeninproductionforawhile.Forthistypeofconfiguration,customersmustconsidersomeadditionalfeaturestogetthe
optimalclusterperformance.Theserversusedintheclustercanbefromdifferentvendorsthisisfullysupportedaslongastheyrunthesamebinaries.Sincemanycustomers
implementOracleRACforhighavailability,youmustmakesurethatyourhardwarevendorwillsupporttheconfiguration.Ifyouhaveafailure,willyougetsupportforthehardware
configuration?
TheinstallationofOracleClusterwareexpectsthenetworkinterfacetobethesamenameonallnodesinthecluster.Ifyouareusingdifferenthardware,youmayneedtoworkwith
youroperatingsystemvendortomakesurethenetworkinterfacenamesarethesamenameonallnodes(IEeth0).Customersimplementingunevenclusterconfigurationsneedto
considerhowtheywillbalancetheworkloadacrossthecluster.Somecustomershavechosentomanuallyassigndifferentworkloadstodifferentnodes.Thiscanbedoneusingdatabase
serviceshoweveritisoftendifficulttopredictworkloadsandthesystemcannotdynamicallyreacttochangesinworkload.ChangestoworkloadrequiretheDBAtomodifytheservice.
Youwillalsoneedtoconsiderhowyouwillsurvivefailuresinthecluster.Willtheservicelevelsbemaintainedifthelargernodeintheclusterfails?Especiallyinasmallcluster,the
impactoflosinganodecouldimpacttheabilitytocontinueprocessingtheapplicationworkload.
Theimpactofthedifferentsizednodesdependsonhowmuchdifferencethereisinthesize.IfthereisalargedifferencebetweenthenodesintermsofmemoryandCPUsize,thanthe
"bigger"nodeswillattractmoreload,obviously,andinthecaseoffailurethe"smaller"node(s)willbecomeoverpowered.Insuchacase,staticroutingofworkloadviaservicese.g.
batchandcertainservices,whichcanbesuspended/stoppedifthelargenodefailsandtheclusterhassignificantlyreducedcapacity,maybeadvisable.Thegeneralrecommendationis
thatthenodesshouldbesizedinsuchawaythattheaggregatedpeakloadofthelargenode(s)canbeabsorbedbythesmallernode(s),i.e.smallernodeshouldhavesufficientcapacity
toruntheessentialservicesalone.Anotheroptionistoaddanothersmallnodetotheclusterondemandincasethatthelargeonefails.
Itshouldalsobenotedespeciallyifthereisalargedifferencebetweenthesizesofthenodes,thesmallnodescanslowdownthelargernode.Thiscouldbecriticaloneifthesmaller
nodeisverybusyandmustservedatatothelargenode.
Tohelpbalanceworkloadacrossacluster,OracleRAC10gRelease2andaboveprovidestheLoadBalancingAdvisory(LBA).TheloadbalancingadvisoryrunsinanOracleRAC
databaseandmonitorstheworkexecutedbytheserviceonallinstanceswheretheserviceisactiveinthecluster.TheLBAprovidesrecommendationstothesubscribedclientsaboutthe
stateoftheserviceandwheretheclientshoulddirectconnectionrequests.SettingtheGOALontheserviceactivatestheloadbalancingadvisory.Clientsthatcanutilizetheload
balancingadvisoryareOracleJDBCImplicitConnectionCache,OracleUniversalConnectionPoolforJava,OracleCallInterfaceSessionPool,ODP.NETConnectionPool,andOracleNet
ServicesConnectionManager.TheOracleListeneralsousestheLoadBalancingAdvisoryifCLB_GOALparameterissettoSHORT(recommendedBestPracticeifusinganintegrated
OracleClientmentionedhere).IfCLB_GOALissettoLONG(default),theListenerwillloadbalancethenumberofsessionsfortheserviceacrosstheinstanceswheretheserviceis
available.SeetheOracleRealApplicationClustersAdministrationandDeploymentGuidefordetailsonimplementingservicesandthevariousparametersettings.

WhatdoIdoifIseeGCCRBLOCKLOSTinmytop5TimedEventsinmyAWRReport?
YoushouldneverseethisorBLOCKRETRYevents.Thisismostlikelyduetoafaultinyourinterconnectnetwork.Workwithyoursystemadministratoror/andnetworkadministratorto
findthefault.Checknetstats
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

29/72

8/24/2015

Document220970.1

Ip:
84884742totalpacketsreceived
1201fragmentsdroppedaftertimeout
3384packetreassemblesfailed
Youdonotwanttoseefragmentsdroppedorpacketreassembliesfailed.
ifconfiga:
eth0Linkencap:EthernetHWaddr00:0B:DB:4B:A2:04
inetaddr:130.35.25.110Bcast:130.35.27.255Mask:255.255.252.0
UPBROADCASTRUNNINGMULTICASTMTU:1500Metric:1
RXpackets:21721236errors:135dropped:0overruns:0frame:95
TXpackets:273120errors:0dropped:0overruns:0carrier:0
Youdonotwanttoseehighnumberoferrors.

AcustomeriscurrentlyusingRACina2nodeenvironment.Howshouldonereviewtheabilitytoscaleoutto4,6,8orevenmorenodes?Whatshouldthe
requirementsofascaleouttest?
OnceacustomerisusingRAConatwonodeclusterandwanttoseehowfartheycanactuallyscaleit,thefollowingaresomehandytipstofollow:
1.Ensuretheyareusingarealenoughworkloadthatitdoesnothavefalsebottlenecks.
2.HavetunedtheapplicationsoitisreasonablescalableontheircurrentRACenvironment.
3.Makesureyouaremeasuringavalidscalabilitymeasure.Thisshouldeitherbedoingverylargebatchjobsquicker(viaparallelism)orbeingabletosupportagreaternumberofshort
transactionsinashortertime.
4.Actualscalabilitywillvaryforeachapplicationanditsbottlenecks.Thustherequesttodotheaboveitems.YouwouldseesimilarscalabilityifscalinguponaSMP.
5.Forfailover,youshouldseewhathappensifyouloseanode.Ifyouhave2nodes,youlosehalfyourpowerandreallygetintotroubleorhavelotsofextracapacity.
6.Measuringthatloadbalacingisworkingproperly.MakesureyouareusingRCLBandaFANawareconnectionpool.
7.YourcustomershouldalsotestingusingDBServices.
8.Getfamiliarw/EMGCtomanageaclusterandhelpeliminatealotofthecomplexityofmanyofthenodes.
9.Whystopat6nodes?Amaximumof3waymessagingensureRACcanscalemuch,muchfurther.

WhatistheLoadBalancingAdvisory?
Toassistinthebalancingofapplicationworkloadacrossdesignatedresources,OracleDatabase10gRelease2providestheLoadBalancingAdvisory.ThisAdvisorymonitorsthecurrent
workloadactivityacrosstheclusterandforeachinstancewhereaserviceisactiveitprovidesapercentagevalueofhowmuchofthetotalworkloadshouldbesenttothisinstanceas
wellasservicequalityflag.ThefeedbackisprovidedasanentryintheAutomaticWorkloadRepositoryandaFANeventispublished.Theeasiestwayforanapplicationtotake
advantageoftheloadbalancingadvisory,istoenableRuntimeConnectionLoadBalancingwithanintegratedclient.

HowdoIenabletheloadbalancingadvisory?
TheloadbalancingadvisoryrequirestheuseofservicesandOracleNetconnectionloadbalancing.
Toenableit,ontheserver:setagoal(service_timeorthroughput,andsetCLB_GOAL=SHORT)onyourservice.
Forclient,youmustbeusingtheconnectionpool.
ForJDBC,enablethedatasourceparameterFastConnectionFailoverEnabled.
ForODP.NETenablethedatasourceparameterLoadBalancing=true.

HowdoImeasurethebandwidthutilizationofmyNICormyinterconnect?
Onesimple/quickandnotveryrecommendedwayistolookatoutputof"ifconfigeth0"andcomparevaluesof"RXbytes"and"TXbytes"overtimethiswillshow_average_usageper
periodoftime.
Amorereliable,interactivewayonLinuxistousetheiptrafutilityortheprebuiltrpmsfromredhatorNovell(SuSE),anotheroptiononLinuxisNetperf.OnotherUnixplatforms:
"snoopStrs64dhme0",AIX'stopazcanshowthataswell..Trytolookforthepeak(notaverage)usageandseeifthatisacceptablyfast.
RememberthatNICbandwidthismeasuredinMbpsorGbps(whichisBITSpersecond)andoutputfromaboveutilitiescansometimescomeinBYTESpersecond,soforcomparison,
doproperconversion(dividebpsvalueby8togetbytes/secor,multiplebytesvalueby8togetbpsvalue).
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

30/72

8/24/2015

Document220970.1

Additionally,youcan'texpectanetworkdevicetorunatfullcapacitywith100%efficiency,duetoconcurrency,collisionsandretransmitsthathappensmorefrequentlyastheutilization
getshigher.IfyouarereachinghighlevelsconsiderafasterinterconnectorNICbonding(multipleNICsallservicingthesameIPaddress).
Finally,aboveismeasuringbandwidthutilization(howmuch),notlatency(howfast)oftheinterconnect,youmaystillbesufferingfromhighlatencyconnection(slowlink)eventhough
thereisplentyofbandwidthtospare.Mostexpertsagreethatlowlatencyisbyfarmoreimportantthanahighbandwidthwithrespecttospecificationsoftheprivateinterconnectin
RAC.Latencyisbestmeasuredbytheactualuserofthenetworklink(RACinthiscase),reviewstatspackforstatsonlatency.Also,in10gR2GridControlyoucanviewGlobalCache
BlockAccessLatency,youcanalsodrilldowntotheClusterCacheCoherencypagetoseetheclustercachecoherencymetricsfortheentireclusterdatabase.
KeepinmindthatRACisusingtheprivateinterconnectlikeitwasneverusedbefore,tosynchronizememoryregions(SGAs)ofmultiplenodes(remember,since9i,entiredatablocks
areshippedaccrosstheinterconnect),ifthenetworkisutilizedat50%bandwidth,thismeansthat50%ofthetimeitisbusyandnotavailabletopotentialusers.Inthiscasedelays(due
tocollisionsandconcurrency)willincreasethelatencyeventhoughthebandwidthmightlook"reasonable",it'shidingtherealissue.

DoesDatabaseblocksizeortablespaceblocksizeaffecthowthedataispassedacrosstheinterconnect?
Oracleshipsdatabaseblockbuffers,i.e.blocksinatablespaceconfiguredfor16Kwillresultina16Kdatabuffershipped,blocksresidinginatablespacewithbaseblocksize(8K)willbe
shippedasbaseblocksandsoonthedatabuffersarebrokendowntopacketsofMTUsizes.

WhatisRuntimeConnectionLoadBalancing?
Runtimeconnectionloadbalancingenablestheconnectionpooltorouteincomingworkrequeststotheavailabledatabaseconnectionthatwillprovideitwiththebestservice.Thiswill
providethebestservicetimesglobally,androutingrespondsfasttochangingconditionsinthesystem.OraclehasimplementedruntimeconnectionloadbalancingwithODP.NETand
JDBCconnectionpools.RuntimeConnectionLoadBalancingistightlyintegratedwiththeautomaticworkloadbalancingfeaturesintroducedwithOracleDatabase10gI.E.Services,
AutomaticWorkloadRepository,andthenewLoadBalancingAdvisory.

HowshouldIdealwithspacemanagement?DoIneedtosetfreelistsandfreelistgroups?
Manuallysettingfreelistgroupsisacomplexitythatisnolongerrequired.
WerecommendusingAutomaticSegmentSpaceManagementratherthantryingtomanagespacemanually.UnlessyouaremigratingfromanearlierdatabaseversionwithOPSand
havealreadybuiltandtunedthenecessarystructures,AutomaticSegmentSpaceManagementisthepreferredapproach.
AutomaticSegmentSpaceManagementisNOTthedefault,youneedtosetit.
Formoreinformationsee:
Note:180608.1AutomaticSpaceSegmentManagementinRACEnvironments

IwasinstallingOracle9iRACandmyOraclefilesdidnotgetcopiedtotheremotenode(s).Whatwentwrong?
Firstmakesuretheclusterisrunningandisavailableonallnodes.Youshouldbeabletoseeallnodeswhenrunningan'lsnodesv'command.
Iflsnodesshowsthatallmembersoftheclusterareavailable,thenyoumayhaveanrcp/rshproblemonUnixorshareshavenotbeenconfiguredonWindows.
Youcantestrcp/rshonUnixbyissuingthefollowingfromeachnode:
[node1]/tmp>touchtest.tst
[node1]/tmp>rcptest.tstnode2:/tmp
[node2]/tmp>touchtest.tst
[node2]/tmp>rcptest.tstnode1:/tmp
OnWindows,ensurethateachnodehasadministrativeaccesstoallthesedirectorieswithintheWindowsenvironmentbyrunningthefollowingatthecommandprompt:
NETUSE\\host_name\C$
Clustercheck.exealsochecksforthis.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

31/72

8/24/2015

Document220970.1

HowdoIstoptheGSD?
Ifyouareon9.0onUnixyouwouldissue:
$psef|grepjre
$kill9<gsdprocess>
StoptheOracleGSDServiceonWindows.
Note:MakesurethatthisistheprocessinusebyGSD
Ifyouareon9.2youwouldissue:
$gsdctlstop

HowdoIdeterminewhetherornotanOneOffpatchis"rollingupgradeable"?
Afteryouhavedownloadedapatch,youcangointothedirectorywhereyouunpackedthepatch:
>pwd
/ora/install/4933522
ThenusethefollowingOPatchcommand:
>opatchqueryis_rolling_patch
...
Query...
Pleaseenterthepatchlocation:
/ora/install/4933522
Querystarts
PatchID:4933522
....
RollingPatch:True.
Queryends

DoesOracleRACworkwithNTP(NetworkTimeProtocol)?
YES!NTPandOracleRACarecompatible,asamatteroffact,itisrecommendedtosetupNTPinanOracleRACcluster,forOracle9iDatabase,OracleDatabase10g,andOracle
Database11gRelease1.
WithOracleDatabase11gRelease2,OracleClusterwareincludestheClusterTimeSynchronizationService(CTSS).Onstartup,OracleClusterwarechecksforaNTPconfiguration,if
found,CTSSgoesintoObservermode.ThismeansitwillmonitortheclocksynchronizationandreportintheOracleClusterwarealertlogifitfindsaproblem.IfitdoesnotfindaNTP
configuration,CTSSwillbeactive.Inactivemode,CTSSsynchronizesallthesystemclockstothefirstnodeinthecluster.
FromtheDocumentation:
OracleDatabaseOracleClusterwareandOracleRealApplicationClustersInstallationGuide10gRelease2(10.2)forLinuxB1420305
page221:
"NodeTimeRequirements
Beforestartingtheinstallation,ensurethateachmembernodeoftheclusterissetascloselyaspossibletothesamedateandtime.Oraclestronglyrecommends
usingtheNetworkTimeProtocolfeatureofmostoperatingsystemsforthispurpose,withallnodesusingthesamereferenceNetworkTimeProtocolserver."

Eachmachinehasadifferentclockfrequencyandasaresultaslightlydifferenttimedrift.NTPcomputesthistimedrifteveryabout15minutes,andstoresthisinformationina"drift"
file,itthenadjuststhesystemclockbasedonthisknowndriftaswellascomparesittoagiventimeserverthesysadminssetsup.Thisistherecommendedapproach.
Keepthefollowingpointsinmind:
Minorchangesintime(inthesecondsrange)areharmlessforOracleRACandtheOracleClusterware.Ifyouintendonmakinglargetimechangesitisbesttoshutdowntheinstances
andtheentireOracleClusterwarestackonthatnodetoavoidafalseeviction,especiallyifyouareusingtheOracleRAC10glowbrownoutpatches,whichallowreallylowmisscount
settings.
Backup/recoveryaspectoflargetimechangesaredocumentedinNote:77370.1,basicallyyoucan'tuseRECOVERDATABASEUNTILTIMEtoreachthesecondrecoverypoint,Itis
possibletoovercomewithRECOVERDATABASEUNTILCANCELorUNTILCHANGE.Ifyouaredoingcompleterecovery(mostofthetimes)thenthisisnotanissuesincetheOracle

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

32/72

8/24/2015

Document220970.1

possibletoovercomewithRECOVERDATABASEUNTILCANCELorUNTILCHANGE.Ifyouaredoingcompleterecovery(mostofthetimes)thenthisisnotanissuesincetheOracle
recoverycodeusesSCN(SystemChangeNumbers)toadvanceintheredo/archivelogs.TheSCNnumbersnevergobackintime(unlessaresetlogsoperationisperformed),thereis
alwaysanassociationofanSCNtoahumanreadabletimestamp(whichmaychangeforwardorbackwards),hencetheissuewithrecoveryuntilpointintimevs.untilSCN/Cancel.
IfDBMS_SCHEDULERisinusageitwillbeaffectedbytimechanges,asit'susingactualclockratherthanSCN.
OnplatformswithOPROCDgetfixforbug:5015469"OPROCDREBOOTSNODEWHENTIMEISSETBACKBYXNTPD"
IfNTPisnotconfiguredcorrectly(usingxflag),anddiagwaitnotsetto13Note:559365.110.2/11.1RACsystemscanberebootedduetoOPROCD,duringaleapsecondevent,see
Note:759143.1.
Daylightsavingtimeadjustmentsdonotaffectthesystemclock,onlythedisplayedtime,hencehavenoimpactontheOraclesoftware.
Apartfromtheseissues,theOracleRDBMSserverisimmunedtotimechanges,i.e.willnotaffecttransaction/readconsistencyoperations.
TheOracleClusterwarerequirestheuseof"x"flagtothentpddaemontopreventtheclockfromgoingbackwards(EnterpriseLinux:see/etc/sysconfig/ntpdSolaris:set"slewalways
yes"in/etc/inet/ntp.conf)

IfIamusingVendorClusterwaresuchasVeritas,IBM,SunorHP,doIstillneedOracleClusterwaretorunOracleRAC10gorOracleRAC11g?
Yes.Whencertified,youcanuseVendorclusterwarehoweveryoumuststillinstallanduseOracleClusterwareforOracleRAC.BestPracticeistoleaveOracleClusterwaretomanage
OracleRAC.FordetailsseeNote:332257.1andforVeritasSFRACseeNote:397460.1.

HowisOracleEnterpriseManagerintegratedwiththeOracleRAC11gRelease2stack?
OracleEnterpriseManager(EM)isavailablein2versions:OracleEMGridControlandOracleEMDatabaseControl.OracleEMGridControlunderliesadifferentreleasecyclethan
theOracleDatabase,whilethenewversionofOracleEMDatabaseControlisavailablewitheverynewdatabaserelease.
Atthetimeofwriting,OracleEMGridControlisavailableinversion10.2.0.5.ThisversiondoesnotsupportnewfeaturesoftheOracleDatabase11gRelease2.Oracle11gRel.2
Database,however,canbemanagedwithOracleEMinthecurrentversionwithsomerestrictions(no11.2featuresupport).
WithOracleDatabaseandGridInfrastructure11gRelease2,OracleEMDatabaseControlisnowabletomanagethefullOracleRAC11gRelease2stack.Thisincludes:Oracle
RACDatabases,OracleClusterware,andOracleAutomaticStorageManagement.
ThenewfeaturethatneedstobenotedhereisthefullmanagementofOracleClusterware11gRelease2withOracleEMDatabaseControl11gRelease2.Formoreinformationand
details,seepubliclyavailableTechnicalWhitePaper:TheNewOracleEnterpriseManagerDatabaseControl11gRelease2NowManagingOracleClusterware

WhatstorageoptionshouldIuseforOracleRAConLinux?ASM/OCFS/RawDevices/BlockDevices/Ext3?
TherecommendedwaytomanagelargeamountsofstorageinanOracleRACenvironmentisASM(AutomaticStorageManagement).Ifyoureallyneed/wantaclusteredfilesystem,
thenOracleoffersOCFS(OracleClusteredFileSystem)for2.4kernel(RHEL3/SLES8)useOCFSVersion1andfor2.6kernel(RHEL4/SLES9)useOCFS2.Alltheseoptionsarefreetouse
andcompletelysupported,ASMisbundledintheRDBMSsoftware,andOCFSaswellasASMLibarefreelydownloadablefromOracle'sOSS(OpenSourceSoftware)website.
EXT3isoutofthequestion,sinceit'sdatastructuresarenotclusteraware,thatis,ifyoumountanext3filesystemfrommultiplenodes,itwillquicklygetcorrupted.
AnotheroptionofcourseisNFSandiSCSIbothareoutsidethescopeofthisFAQbutincludedforcompleteness.
Ifforanyreasontheaboveoptions(ASM/OCFS)arenotgoodenoughandyouinsistonusing'rawdevices'or'blockdevices'herearethedetailsonthetwo(Thisinformationisstillvery
usefultoknowinthecontextofASMandOCFS).
OnUnix/Linuxtherearetwotypesofdevices:
blockdevices(/dev/sde9)are**BUFFERED**devices!!unlessyouexplicitlyopentheminO_DIRECTyouwillgetbuffered(linuxbuffercache)IO.
characterdevices(/dev/raw/raw9)are*UNBUFFERRED**devices!!nomatterhowyouopenthem,youalwaysgetunbufferredIO,hencenoneedtospecifyO_DIRECTonthefile
opencall.
Aboveisnotatypo,blockdevicesonUnixdobufferedIObydefault(cachedinlinuxbuffercache),thismeansthatRACcannotoperateonit(unlessopenedwithO_DIRECT),sincethe
IO'swillnotbeimmediatelyvisibletoothernodes.
Youmaycheckifadeviceisblockorcharacterdevicebythefirstletterprintedwiththe"lsl"command:
crwrw1rootdisk162,1Jan2319:53/dev/raw/raw1
brwrw1rootdisk8,112Jan2314:51/dev/sdh
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

33/72

8/24/2015

Document220970.1

Above,"c"standsforcharacterdevice,and"b"forblockdevices.
StartingwithOracle10.1anRDBMSfixaddedtheO_DIRECTflagtotheopencall(O_DIRECTflagtellstheLinuxkerneltobypasstheLinuxbuffercacheandwritedirectlytodisk),inthe
caseofablockdevice,thatmentthatacreatedatafileon'/dev/sde9'wouldsucceed(needtosetfilesystemio_options=directioininit.ora)..Thisenhancementwaswellreceived,and
shortlyafterbug4309443wasfixed(byaddingtheO_DIRECTflagontheOCRfileopencall)meaningthatstartingwith10.2(thereareseveral10.1backportsavailable)theOracleOCR
filecouldalsoaccessblockdevicesdirectly.ForthevotingdisktobeopenedwithO_DIRECTyouneedfixforbug4466428(5021707isaduplicate).Thismeansthatbothvotingdisks
andOCRfilescouldliveonblockdevices.However,duetoOUIbug5005148,thereisstillaneedtoconfigurerawdevicesforthevotingorOCRfilesduringinstallationofRAC,notsuch
abigdeal,sinceit'sjust5filesinmostcases.Itisnotpossibletoaskforabackportofthisbugsinceitmeansafullrereleaseof10g,onealternativeifrawdevicesarenotagood
optionistouse11gClusterware(with10gRACdatabase).
Byusingblockdevicesyounolongerhavetolivewiththelimitationsof255rawdevicespernode.Youcanaccessasmanyblockdevicesasthesystemcansupport.Alsoblockdevices
carrypersistentpermissionsacrossreboots,whilewithrawdevicesonewouldhavetocustomizethatafterinstallationotherwisetheClusterwarestackordatabasewouldfailtostartup
duetopermissionissues.
ASMorASMlibcanbegiventherawdevices(/dev/raw/raw2)aswasdoneintheinitialdeploymentof10gRelease1,orthemorerecommendedway:ASM/ASMLibshouldbegiventhe
blockdevicesdirectly(eg./dev/sde9).
SinceRAWdevicesarebeingphasedoutofLinuxinthelongterm,itisrecommendedeveryoneshouldswitchtousingtheblockdevices(meaning,passtheseblockdevicestoASMor
OCFS/2orOracleClusterware)
Note:WithOracleDatabase11gRelease2,OracleClusterwarefiles(OCRandVotingDiskcanbestoreinASMandthisistheBestPractice).TheOracleUniversalInstallerandthe
configurationassistants(IEDBCA,NETCA)willnotsupportraw/blockdevices.Allcommandlineinterfaceswillsupportraw/blockforthisrelease.Thereforeifyouareusingraw/block
today,youcancontinuetouseitandupgradingto11gRelease2willnotchangethelocationofanyfiles.Howeverduetothedesupportinthenextrelease,youarerecommendedto
planamigrationtoasupportedstorageoption.AllfilessupportednativelyinASM,willnotbesupportedinproductionwiththeASMClusterFileSystem(ACFS)

WhataretheimplicationsofusingsrvctldisableforaninstanceinmyOracleRACcluster?IwanttohaveitavailabletostartifIneeditbutatthistimetonot
wanttorunthisextrainstanceforthisdatabase.
Duringnodereboot,anydisabledresourceswillnotbestartedbytheClusterware,thereforethisinstancewillnotberestarted.Itisrecommendedthatyouleavethevip,ons,gsd
enabledinthatnode.Forexample,VIPaddressforthisnodeispresentinaddresslistofdatabaseservices,soaclientconnectingtotheseserviceswillstillreachsomeotherdatabase
instanceprovidingthatservicevialistenerredirection.JustbeawarethatbydisablinganInstanceonanode,allthatmeansisthattheinstanceitselfisnotstarting.However,ifthe
databasewasoriginallycreatedwith3instances,thatmeansthereare3threadsofredo.So,whiletheinstanceitselfisdisabled,theredothreadisstillenabled,andwilloccasionally
causelogswitches.Thearchivedlogsforthis'disabled'instancewillstillbeneededinanypotentialdatabaserecoveryscenario.So,ifyouaregoingtodisabletheinstancethrough
srvctl,youmayalsowanttoconsiderdisablingtheredothreadforthatinstance.
srvctldisableinstancedorcliorcl2
SQL>alterdatabasedisablepublicthread2
Dothereversetoenabletheinstance.
SQL>alterdatabaseenablepublicthread2
srvctlenableinstancedorcliorcl2

Ifusingplsqlnativecode,theplsql_native_library_dirneedstobedefined.InanOracleRACenvironement,mustthedirectorybeinthesharedstorage?
InOracleRACconfiguration,thisparametermustbesetineachinstance.Theinstancesarenotrequiredtohaveasharedfilesystem.Oneachinstancetheplsql_native_library_dircan
besettopointtoaninstancelocaldirectory.Alternately,iftheOracleRACconfigurationsupportsashared(cluster)filesystem,youcanuseacommondirectory(onthesharedfile
system)forallinstances.YoucanalsocheckoutthePL/SQLNativeCompilationFAQonOTN:www.oracle.com/technology/tech/pl_sql/htdocs/ncomp_faq.htmlWithOracleRAC11g
Release2,useACFS(ASMClusterfilesystem)

WhatisthepurposeofthegsdserviceinOracle9iRAC?
GSDisonlyneededforconfiguration/managementofclusterdatabase.Oncedatabasehasbeenconfiguredandup,itcanbesafelystoppedprovidedyoudon'trunany'srvctlordbcaor
dbua'tools.InOracle9iRAC,theGSDdoesn'twriteanywhereunlesstracingwasturnedon,inwhichcasetracesgotostdout.
Oncethedatabasehasbeenconfiguredandstartedandyoudon'tuse'srvctlorEM'tomanageor'dbcatoextend/remove'or'dbuatoupgrade'thisdatabase,GSDcanbestopped.
Note:WithOracleRAC11gRelease2,thegsdresourceisdisabledbydefault.YouwillonlyneedtoenabletheresourceifyouarerunningOracle9iRACinthesamecluster.

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

34/72

8/24/2015

Document220970.1

Note:WithOracleRAC11gRelease2,thegsdresourceisdisabledbydefault.YouwillonlyneedtoenabletheresourceifyouarerunningOracle9iRACinthesamecluster.

HowdoIidentifywhichnodewasusedtoinstalltheclustersoftwareand/ordatabasesoftware?
Youcanfindoutwhichnodebyrunningolsnodescommand.Thenodewhichisreturnedfirstisthenodefromwhichthesoftwarewasinstalledandpatchesshouldbeinstalled.
Note:Whenapplyingpatchesinarollingfashion,youarerecommendedtoruntherollingscriptsfromthelastnodeaddedtotheclusterfirstandfollowthelistinreverseorder.

AretheOracleClusterwarebundlepatchescumulative,dotheyconflictwithoneanother?
Fixwise,theOracleClusterwarebundlesarecumulative,thatis,CRSbundle#3fixesalltheissuesthatbundle#2did,andsomeadditionalones,seeNote:405820.1forcompletelistof
bugsfixedineachbundle.
However,OPatchdoesnotallowtoapplyANYpatchifthereareanyoverlappinglibsorbinariesbetweenanalreadyexistingpatchandthetobeinstalledpatch.
Iftwopatchestouchaparticularfile,e.g:kcb.o,thentheexistingpatchmustbemanuallyremovedbeforethenewapplied.
So,bundlepatchesarecumulative,howevertheydoconflictwithoneanotherduetothewayOPatchallowspatchapplication,hencepreviousbundlemustbemanuallyremovedbefore
anewoneisapplied.
TocheckifanytwopatchesconflictinvokeOPatchasperNote:458485.1orusing:
$OPatch/opatchprereqCheckConflictAmongPatchesphbasefilepatchlist

wherepatchlistisatextfilecontainingallthepatchnumberstobechecked,separatedbyanewline.

Ihaveaddedasecondnetworktomycluster,canIloadbalancemyusersacrossthisnetwork?
ServersideloadbalancingwillonlyworkonasinglenetworkwhichisconfiguredasthepublicnetworkwiththeOracleVIPS.Ifyouaddasecondnetwork,withasecondlistener,donot
addthisnewlistenertothelocal_listenerandremote_listenerparameter.Youcanuseclientsideloadbalancingandfailoverforusersconnectingtothisnetworkhoweveryouwillbe
unabletouseserversideloadbalancingorreceiveFANeventsforthisnetwork.
OracleRAC11gRelease2addsthesupportformultiplepublicnetworks.Connectionswillbeloadbalancedacrosstheinstances.Eachnetworkwillhaveitsownservice.Toenableload
balancingusetheLISTENER_NETWORKSparameterinsteadofLOCAL_LISTENERandREMOTE_LISTENER.

Srvctlcannotstartinstance,IgetthefollowingerrorPRKP1001CRS0215,howeversqlpluscanstartitonbothnodes?Whatistheproblem?
ThiscouldbemanythingsbutapopularissueiswhenyouhaveaseparateASMHomeandthelistenerisrunningoutofthishome(itwasthefirsthomeinstalled).Srvctlneedsa
TNS_ADMINaliastothenetwork/admindirectoryinthishomeinsteadofusingthedefaultORACLE_HOME/network/adminforthedatabaseyouaretryingtostart.Forsrvctltoworkyou
must
srvctlsetenvnodeappsnnodeTTNS_ADMIN=fullpath
oneachnodeinthecluster.
YoucannotrelyonaTNS_ADMINenvironmentvariable.
SeeNote420977.1
Anothercauseisnonexistentspfile,seeNote732683.1

WhenIlookatALL_SERVICESviewinmydatabaseIseeservicesIdidnotcreate,whataretheyfor?
Youwillalwaysseeadefaultdatabaseservicethatisthesamenameasyourdatabase.Thisserviceisavailableonallinstancesinthecluster.Youwillalsoseetwoservicesusedbythe
databaseSYS$BACKGROUND(forbackgroundprocesses)andSYS$USERS(userswhoconnectviaBEQorwithoutusingaservice_name).YoumayalsoseeservicesthatendwithXDB
whicharecreatedfortheXMLDBfeatureandyouwillnotbeabletomanagetheseservices.

Ihave2clustersnamed"crs"(thedefault),howdoIgetGridControltorecognizethemastargets?
Thereare2options:
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

35/72

8/24/2015

Document220970.1

Thereare2options:
a)ifthegridcontrolagentinstall(whichisaseparateinstall)hasalreadybeendoneandhaspickedupthenameoftheclusterasitwasconfiguredasCRS,onecangototheEM
consoleasis,andforthesecond,manuallydeleteandrediscoverthetarget.Whenyourediscoverthetarget,givewhateverdisplaynameyoulike
b)PriortoperformingtheGridcontrolagentinstall,justsetCLUSTER_NAMEenvironmentvariableandruntheinstall.Thisvariableneedtobesetonlyforthatinstallsession.Noneed
tosetiteverytimeagentstarts.

Ifoundin10.2thattheEM"ConverttoClusterDatabase"wizardwouldalwaysfalloveronthelaststepwhereitrunsemcaandneedstologintothenewcluster
databaseasdbsnmptocreatetheclusterdatabasetargetsetc.Ichangedthepasswordforthedbsnmpaccounttobedbsnmp(sameasusername)andit
workedOK.Isthisaknownissue?
TheconversiontoclusterhappenssuccessfullybuttheEMmonitoringcredentialsfortheconverteddatabasearenotproperlysetduetothisbug.Thisisresolvedinnextpatchset.Inthe
interim,usercansetthemonitoringpasswordfromthe"monitoringconfiguration"screenfortheRACDBfromGCconsoleandproceed.
Thisissuehasbeenfixedin10.2.0.3databaseandtogetthecompletefunctionalityyouwillneed10.2.0.2GridControlpatchalso,asthefixisspreadbetweenthetwopiecesof
software.Fornowyoucanproceedwithsettingpasswordfordbsnmpusersameasthatofsysuser.

WhatistheClusterVerificationUtiltiy(cluvfy)?
TheClusterVerificationUtility(CVU)isavalidationtoolthatyoucanusetocheckalltheimportantcomponentsthatneedtobeverifiedatdifferentstagesofdeploymentinaRAC
environment.ThewidedomainofdeploymentofCVUrangesfrominitialhardwaresetupthroughfullyoperationalclusterforRACdeploymentandcoversalltheintermediatestagesof
installationandconfigurationofvariouscomponents.Cluvfydoesnottakeanycorrectiveactionfollowingthefailureofaverificationtask,doesnotenterintoareasofperformancetuning
ormonitoring,doesnotperformanyclusterorRACoperation,anddoesnotattempttoverifytheinternalsofclusterdatabaseorclusterelements.

WhatversionsofthedatabasecanIusetheclusterverificationutility(cluvfy)with?
TheclusterverificationutilityisreleasewithOracleDatabase10gRelease2butcanalsobeusedwithOracleDatabase10gRelease1.

HowmanynodescanonehaveinanHPUX/Solaris/AIX/Windows/Linuxcluster?
TechnicallyandsinceOracleRAC10gRelease2,100nodesaresupportedinonecluster.Thisincludesrunning100databaseinstancesbelongingtothesame(production)
databaseonthiscluster,usingtheOracleDatabaseEnterpriseEdition(EE)withtheOracleRACoptionandOracleClusterwareonly(nothirdparty/vendorclustersolutionunderneath).
NotethatusingtheOracleDatabaseStandardEdition(SE),whichincludestheOracleRACfunctionality,furtherrestrictionsregardingthenumberofnodesperclusterapply.Also
notethatonecannotuseathirdpartyorvendorclusterforanOracleDatabaseStandardEditionbasedOracleRACcluster.Formoreinformationseethelicensinginformation.
Whenusingathirdparty/vendorclustersoftwarethefollowinglimitsapply(subjecttochangewithoutnotice):
SolarisCluster:8
HPUXServiceGuard:16
HPTru64:8
IBMAIX:
*8nodesforPhysicalShared(CLVM)SSAdisk
*16nodesforPhysicalShared(CLVM)nonSSAdisk
*128nodesforVirtualSharedDisk(VSD)
*128nodesforGPFS
*Subjecttostoragesubsystemlimitations
Veritas:816nodes(checkw/Veritas)
Nodelimitationsshouldalwaysbecheckedwiththeclustersoftwarevendor.

IscrossovercablesupportedasaninterconnectwithRAConanyplatform?
NO.CROSSOVERCABLESARENOTSUPPORTED.Therequirementistouseaswitch:
DetailedReasons:
1)crosscablinglimitstheexpansionofRACtotwonodes
2)crosscablingisunstable:

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

36/72

8/24/2015

Document220970.1

2)crosscablingisunstable:
a)SomeNICcardsdonotworkproperlywithit.TheyarenotabletonegotiatetheDTE/DCEclocking,andwillthusnotfunction.TheseNICSweremadecheaperbyassumingthatthe
switchwasgoingtohavetheclock.UnfortunatelythereisnowaytoknowwhichNICsdonothavethatclock.
b)MediasensebehaviouronvariousOS's(mostnotablyWindows)willbringaNICdownwhenacableisdisconnected.Eitheroftheseissuescanleadtoclusterinstabilityandleadto
ORA29740errors(nodeevictions).
Duetothebenefitsandstabilityprovidedbyaswitch,andtheirafforability($200forasimple16portGigEswitch),andtheexpenseandtimerelatedtodealingwithissueswhenone
doesnotexist,thisistheonlysupportedconfiguration.
FromapurelytechnologypointofviewOracledoesnotcareifthecustomerusescrossovercableorrouterorswitchestodeliveramessage.However,weknowfromexperiencethata
lotofadaptersmisbehavewhenusedinacrossoverconfigurationandcausealotofproblemsforRAC.Hencewehavestatedoncertifythatwedonotsupportcrossovercablestoavoid
falsebugsandfingerpointingamongstthevariousparties:Oracle,Hardwarevendors,Osvendorsetc...

IsitpossibletorunOracleRAConlogicalpartitions(i.e.LPARs)orvirtualseparateservers.
Yes,itispossible.CheckCertifytounderstandthecurrentdetailsforthedifferenthardwaresolutions.
Onhighendserverscanbepartitionedintodomains(partitions)ofsmallersizes,eachdomainwithitsownCPU(s)andoperatingsystem.Eachdomainiseffectivelyavirtualserver.
OracleRACcanberunonclustercomprisesofdomains.Thebenefitsofusingthisissimilartoaregularcluster,anydomainfailurewillhavelittleeffectonotherdomains.Besides,the
managementoftheclustermaybeeasiersincethereisonlyonephysicalserver.Notehowever,sinceoneE10Kisstilljustoneserver.Therearesinglepointsoffailures.Anyfailures,
suchasbackplanefailure,thatcrumbletheentireserverwillshutdownthevirtualcluster.Thatisthetradeoffusershavetomakeinhowbesttobuildaclusterdatabase.

HowdoIcheckOracleRACcertification?
SeethefollowingMetalinknote:Note184875.1PleasenotethatcertificationsforOracleRealApplicationClustersareperformedagainsttheOperatingSystemandClusterwareversions.
ThecorrespondingsystemhardwareisofferedbySystemvendorsandspecializedTechnologyvendors.Somesystemvendorsofferpreinstalled,preconfiguredOracleRACclusters.
TheseareincludedbelowunderthecorrespondingOSplatformselectionwithinthecertificationmatrix.

WhatisOracle'spositionwithrespecttosupportingRAConPolyserveCFS?
PleasecheckthecertificationmatrixavailablethroughMetalinkforyourspecificrelease.

CantheOracleDatabaseConfigurationAssistant(DBCA)beusedtocreateadatabasewithVeritasDBE/AC3.5?
DBCAcanbeusedtocreatedatabasesonrawdevicesin9iRACRelease1and9iRelease2.StandarddatabasecreationscriptsusingSQLcommandswillworkwithfilesystemandraw.
DBCAcannotbeusedtocreatedatabasesonfilesystemsonOracle9iRelease1.Theusercanchoosetosetupadatabaseonrawdevices,andhaveDBCAoutputascript.Thescript
canthenbemodifiedtouseclusterfilesystemsinstead.
WithOracle9iRACRelease2(Oracle9.2),DBCAcanbeusedtocreatedatabasesonaclusterfilesystem.IftheORACLE_HOMEisstoredontheclusterfilesystem,thetoolwillwork
directly.IfORACLE_HOMEisonlocaldrivesoneachsystem,andthecustomerwishestoplacedatabasefilesontoaclusterfilesystem,theymustinvokeDBCAasfollows:dbca
datafileDestination/oradatawhere/oradataisontheCFSfilesystem.See9iR2READMEandbug2300874formoreinfo.

IsOracleDatabaseonVMwaresupport?IsOracleRAConVMwaresupported?
OracleDatabasesupportonVMwareisoutlinedinMetalinkNote249212.1.Effectively,formostcustomers,thismeanstheyarenotwillingtorunproductionOracledatabaseson
VMware.RegardingOracleRACtheexplicitmentionnottorunRAConvmwarewasremovedin11.2.0.2(Novemeber2010)

IsVeritasStorageFoundationsupportedwithOracleRAC?
VeritascertifiesVeritasStorageFoundationforOracleRACwitheachrelease.CheckCeritifyandVeritasSupportMatrixforthelatestdetails.

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

37/72

8/24/2015

Document220970.1

Afterinstallingpatchset9013andpatch_2313680onLinux,thestartupwasveryslow
PleasecarefullyreadthefollowingnewinformationaboutconfiguringOracleClusterManagementonLinux,providedaspartofthepatchREADME:
Threeparametersaffectthestartuptime:
soft_margin(definedatwatchdogmoduleload)
m(watchdogdstartupoption)
WatchdogMarginWait(definedinnmcfg.ora).
WatchdogMarginWaitiscalculatedusingtheformula:
WatchdogMarginWait=soft_margin(msec)+m+5000(msec).
[5000(msec)ishardcoded]
Notethatthesoft_marginismeasuredinseconds,mandWatchMarginWaitaremeasuredinmilliseconds.
Basedonbenchmarking,itisrecommendedtosetsoft_marginbetween10and20seconds.Usethesamevalueform(convertedtomilliseconds)asusedforsoft_margin.Hereisan
example:
soft_margin=10m=10000WatchdogMarginWait=10000+10000+5000=25000
IfCPUutilizationinyoursystemishighandyouexperienceunexpectednodereboots,checkthewdd.logfile.Ifthereareany'pingcametoolate'messages,increasethevalueofthe
aboveparameters.

Isthereaclusterfilesystem(CFS)AvailableforLinux?
Yes,ACFS(ASMClusterFileSystemwithOracleDatabase11gRelease2)andOCFS(OracleClusterFilesystem)areavailableforLinux.ThefollowingMetalinknotehasinformationfor
obtainingthelatestversionofOCFS:
Note238278.1HowtofindthecurrentOCFSversionforLinux

IsthehangchecktimerstillneededwithOracleRAC10gand11gR1?
YES!hangchecktimerisrequiredfor10gand11gR1(11.1.*).ItisnolongerneededinOracleClusterware11gR2.
ThehangchecktimermodulemonitorstheLinuxkernelforextendedoperatingsystemhangsthatcouldaffectthereliabilityoftheRACnode(I/Ofencing)andcausedatabase
corruption.Toverifythehangchecktimermoduleisrunningoneverynode:
asrootuser:
/sbin/lsmod|grephangcheck
Ifthehangchecktimermoduleisnotlistedenterthefollowingcommandastherootuser:
9i:/sbin/insmodhangchecktimerhangcheck_tick=30hangcheck_margin=180hangcheck_reboot=1
10g&11gR1:/sbin/insmodhangchecktimerhangcheck_tick=1hangcheck_margin=10hangcheck_reboot=1

Toensurethemoduleisloadedeverytimethesystemreboots,verifythatthelocalsystemstartupfile(/etc/rc.d/rc.local)containsthecommandabove.
ForadditionalinformationpleasereviewtheOracleRACInstallandConfigurationGuide(541)andnote:726833.1.

OracleClusterwarefailstostartafterarebootduetopermissionsonrawdevicesrevertingtodefaultvalues.HowdoIfixthis?
AfterasuccessfulinstallationofOracleClusterwareasimplerebootandOracleClusterwarefailstostart.ThisisbecausethepermissionsontherawdevicesfortheOCRandvoting
diskse.g./dev/raw/raw{x}reverttotheirdefaultvalues(root:disk)andareinaccessibletoOracle.Thischangeofbehavorstartedwiththe2.6kernelinRHEL4,OEL4,RHEL5,OEL5,
SLES9andSLES10.InRHEL3therawdevicesmaintainedtheirpermissionsacrossrebootssothissymptomwasnotseen.
ThewaytofixthisisonRHEL4,OEL4andSLES9istocreate/etc/udev/permission.d/40udev.permissions(youmustchooseanumberthat'slowerthan50).Youcandothisby
copying/etc/udev/permission.d/50udev.permissions,andremovingthelinesthatarenotneeded(50udev.permissionsgetsreplacedwithupgradessoyoudonotwanttoedititdirectly,

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

38/72

8/24/2015

Document220970.1

copying/etc/udev/permission.d/50udev.permissions,andremovingthelinesthatarenotneeded(50udev.permissionsgetsreplacedwithupgradessoyoudonotwanttoedititdirectly,
alsoatypointhe50udev.permissionscanrenderthesystemnonusable).Examplepermissionsfile:
#rawdevices
raw/raw[12]:root:oinstall:0640
raw/raw[35]:oracle:oinstall:0660

Notethatthisappliedtoallrawdevicefiles,herejustthevotingandOCRdeviceswerespecified.
OnRHEL5,OEL5andSLES10adifferentfileisused/etc/udev/rules.d/99raw.rules,noticethatnowthenumbermustbe(anynumber)higherthan50.Alsothesyntaxoftherulesis
differentthanthepermissionsfile,here'sanexample:
KERNEL=="raw[12]*",GROUP="oinstall",MODE="640"
KERNEL=="raw[35]*",OWNER="oracle",GROUP="oinstall",MODE="660"

ThisisexplainedindetailinNote:414897.1.

HowdoIconfigurerawdevicesinordertoinstallOracleClusterware10gonRHEL5orOEL5?
TherawdevicesOSsupportscriptslike/etc/sysconfig/rawdevicesarenotshippedonRHEL5orOEL5,thisisbecauserawdevicesarebeingdeprecatedonLinux.Thismeansthatin
ordertoinstallOracleClusterware10gyou'dhavetomanuallybindtherawdevicestotheblockdevicesfortheOCRandvotingdiskssothatthe10ginstallerwillproceedwithouterror.
RefertoNote465001.1forexactdetailsonhowtodotheabove.
OracleClusterware11gdoesn'trequirethisconfigurationsincetheinstallercanhandleblockdevicesdirectly.

CandifferentreleasesofOracleRACbeinstalledandrunonthesamephysicalLinuxcluster?
YesHoweverOracleClusterware(CRS)willnotsupportaOracle9iRACdatabasesoyouwillhavetoleavethecurrentconfigurationinplace.YoucaninstallOracleClusterwareand
OracleRAC10gor11gintothesamecluster.OnWindowsandLinux,youmustrunthe9iClusterManagerforthe9iDatabaseandtheOracleClusterwareforthe10gDatabase.When
youinstallOracleClusterware,your9isrvconfigfilewillbeconvertedtotheOCR.Oracle9iRAC,OracleRAC10g,andOracleRAC11gwillusetheOCR.Donotrestartthe9igsdafter
youhaveinstalledOracleClusterware.RemembertocheckcertifyfordetailsofwhatvendorclusterwarecanberunwithOracleClusterware.OracleClusterwaremustbethehighest
level(downtothepatchset).IEOracleClusterware11gRelease2willsupportOracleRAC10gandOracleRAC11gdatabases.OracleClusterware10gcanonlysupportOracleRAC10g
databases.

Is3rdPartyClusterwaresupportedonLinuxsuchasVeritasorRedhat?
No,OracleRAC10gandOracleRAC11gdonotsupport3rdPartyclusterwareonLinux.Thismeansthatifaclusterfilesystemrequiresa3rdpartyclusterware,theclusterfilesystemis
notsupported.

Acustomerinstalled10gRelease2onLinuxRH4Update2,2.6.922.ELsmp#1SMPx86_64GNU/Linux,andgottheerrorErrorininvokingtarget'all_no_orcl'.
Customerignoredtheerrorandtheinstallsucceededwithoutanyothererrorsandoracleapparentlyworkedfine.Whatshouldtheydo?
Becauseofcompatibilitywiththeirstoragearray(EMCDMXwithPowerpath4.5)theymustuseupdate2.OracleinstallguidestatesthatRH464bitsupdate1"orhigher"shouldbeused
for10gR2.
Thebinutilspatchbinutils.15.92.0.213.0.0.0.2.x86_64.rpmisneededtorelinkwithouterror.RedHatisawareofthebug.Customersshouldusethelatestupdate(oratleastupdate3
tofix).

IsOCFS2certifiedwithOracleRAC10g?
Yes.SeeCertifytofindoutwhichplatformsarecurrentlycertified.

HowdoIconfiguremyRACClustertousetheRDSInfiniband?
TheconfigurationtakesplacebelowOracle.YouneedtotalktoyourInfinibandvendor.Checkcertifyforwhatiscurrentlyavailableasthiswillchangeasvendorsadoptthetechnology.
Thedatabasemustbeatleast10.2.0.3.IfyouwanttoswitchadatabaserunningwithIPoverIB,youwillneedtorelinkOracle.
$cd$ORACLE_HOME/rdbms/lib$makefins_rdbms.mkipc_rdsioracle
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

39/72

8/24/2015

Document220970.1

Youcancheckyourinterconnectthroughthealertlogatstartup.CheckforthestringclusterinterconnectIPCversion:OracleRDS/IP(generic)inthealert.logfile.
SeeNote:751343.1formoredetails.

CustomerdidnotloadthehangchecktimerbeforeinstallingRAC,Canthecustomerjustloadthehangchecktimer?
YES.hangchecktimerisakernelmodulethatisshippedwiththeLinuxkernel,allyouhavetodoisloaditasfollows:
9i:/sbin/insmodhangchecktimerhangcheck_tick=30hangcheck_margin=180hangcheck_reboot=1
10g&11g:/sbin/insmodhangchecktimerhangcheck_tick=1hangcheck_margin=10hangcheck_reboot=1
Noneedtorebootthenodes.

Formoredetailsseenote:726833.1

Howtoreorderorrenamelogicalnetworkinterface(NIC)namesinLinux
Althoughthisisrarelyneeded,sincemosthardwarewilldetectthecardsinthecorrectorderonallnodes,ifyoustillneedtochange/controltheordering,seeexternalwebsite,hereis
morehelponwritingUDEVrules.

AreRedHatGFSandGULMcertifiedforDLM?
BotharepartofRedHatRHEL5.ForOracleDatabase10gRelease2onLinuxx86andLinuxx8664,itiscertifiedonOEL5andRHEL5aspercertify.GFSisnotcertifiedyet,certification
inprogressbyRedHat.OCFS2iscertifiedandit'sthepreferredchoiceforOracle.ASMisrecommendedstorageforthedatabase.SinceGFSispartoftheRHEL5distributionandOracle
fullysupportsRHELundertheUnbreakableLinuxProgam,OraclewillsupportGFSaspartofRHEL5forcustomersbuyingtheUnbreakableLinuxSupport.ThisonlyappliestoRHEL5and
nottoRHEL4whereGFSisdistributedwithanadditionalfee

Mycustomerisabouttoinstall10202clusterwereonnewLinuxmachinges.Heisgetting"NoORACMrunning"errorwhenrunrootpre.shandexited?Shouldhe
worryaboutthismessage?
Itisaninformationalmessage.Generallyforsuchscripts,youcanissueecho$?toensurethatitreturnsazerovalue.Themessageisbasicallysaying,itdidnotfindanoracm.If
Customerwereinstalling10gonanexisting9icluster(whichwillhaveoracm)thenthismessagewouldhavebeenserious.Butsincecustomerisinstallingthisonafreshnewbox,They
cancontinuetheinstall.

HowtoconfigurebondingonSuseSLES8.
Pleaseseenote:291958.1

HowtoconfigurebondingonSuseSLES9.
Pleaseseenote:291962.1

InSolaris10,doweneedSunClustertoprovideredundancyfortheinterconnectandmultipleswitches?
LinkAggregation(GLDv3)isbundledintheOSasofSolaris10.IPMPisavailableforSolaris10andSolaris9.NeitherrequireSunClustertobeinstalled.Fortheinterconnectandswitch
redundancy,asabestpractice,avoidVLANtrunkingacrosstheswitches.WecanconfigurestandaloneredundantswitchesthatdonotrequiretheVLANtobetrunkedbetweenthem,
northeneedforaninterswitchlink(ISL).IftheinterconnectVLANistrunkedwithotherVLANSbetweentheredundantswitches,insurethattheinterconnectVLANisprunedfromthe
trunktoavoidunnecessarytrafficpropagationthroughthecorportatenetwork.Foreaseofconfiguration(e.g.fewerIPaddressrequirements),useIPMPwithlinkmodefailuredetection
inprimary/standbyconfiguration.ThiswillgiveyouasinglefailoverIPwhichyouwilldefineincluster_interconnectsinit.oraparameter.Removeanyinterfacesfortheinterconnectfrom
theOCRusing`oifcfgdelif`.ANDTESTTHISRIGOROUSLY.Fornow,asLinkAggregation(GLDv3)cannotspanmultipleswitchesfromasinglehost,youwillneedtoconfigurethe
switchredundancyandthehostNICswithIPMP.WhenconfiguringIPMPfortheinterconnectwithmultipleswitchesavailable,configureIPMPasactive/standbyand*not*active/active.
Thisistoavoidpotentiallatenciesinswitchfailuredetection/failoverwhichmayimpacttheavailabilityoftherdbms.Note,IPMPspreads/loadbalancesoutboundpacketsonthebonded
interfaces,butinboundpacketsarereceivedonasingleinterface.Inanactive/activeconfigurationthismakessend/receiveproblemsdifficulttodiagnose.BothLinkAggregation(GLDv3)
andIPMParecoreOSpackagesSUNWcsu,SUNWcsrrespectivelyanddonotrequireSunClusterware.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

40/72

8/24/2015

andIPMParecoreOSpackagesSUNWcsu,SUNWcsrrespectivelyanddonotrequireSunClusterware.

Document220970.1

CanIconfigureIPMPinActie/Activetoincreasebandwidthofmyinterconnect?
ForIPMPForactive/activeconfigurationspleasefollowthesundocinstructionshttp://docs.sun.com/app/docs/doc/8164554/6maoq027i?a=viewIPMPactive/activeisknowntoload
balanceontransmitbutserializeonasingleinterfaceforreceive.Soyouarelikelynottogetthethroughputyoumighthaveexpected.Unlessyouexperienceexplicitbandwidth
limitationsthatrequireactive/active,itisarecommendedbestpracticetoconfigureformaximumavailability,asdescribedinwebivnote283107.1.
Pleasenotetoothatdebuggingactive/activeinterfacesatthenetworklayeriscumbersomeandtimeconsuming.Inanactive/activeconfigurationandtheswitchsidelinkfails,youare
likelytolosebothinterconnectconnections,whereasactive/standby,youwouldfailover.

DoesSunSolarishaveamultipathingsolution?
SunSolarisincludesaninherentMultipathingtool:MPXIOthisispartofSolaris.YouneedtohavetheSanFoundationKitinstalled(newestversion).Please,beawarethatthe
machinesareinstalledfollowingtheEISstandard.ThisisaqualityassurancestandardintroducedbySunthatmainlytakescarethatyoualwayshavethenewestpatches.
MPXIOisfreeofchargeandcomeswithSolaris8,9,10.BTW,ifyouhaveaSunLVM,itwouldusethisfeatureindirectly.Therefore,SunconfirmedthatMPXIOwillworkwithRAWs.

CanIconfigureHP'sAutoportaggregationforNICBondingaftertheinstall?(i.e.notpresentbeforehand)
YouareabletoaddNICbondingaftertheinstallationalthoughthisismorecomplicatedthantheotherwayround.
Thereareseveralnotesonwebivregardingthis.
Note276434.1ModifyingtheVIPofaClusterNode
Regardingtheprivateinterconnect,pleaseuseoifcfgdelif/setiftomodifythis.
@ForcustomersonLinux,thereismoreinformationonNICbonding,pleaseread
@ConfigureRedundantNetworkCards/SwitchesforOracleDatabase10gRelease1RealApplicationClusteronLinux

IsHMPsupportedwithOracleRAC10gorOracleRAC11gonallHPplatforms?
HPhasdesupportedHMPwithOracleRAC10g.Seehttp://docs.hp.com/en/B625790056/ch01s01.html?jumpid=reg_R1002_USEN

DoestheOracleClusterFileSystem(OCFS)supportnetworkaccessthroughNFSorWindowsNetworkShares?
No,inthecurrentreleasetheOracleClusterFileSystem(OCFS)isnotsupportedforusebynetworkaccessapproacheslikeNFSorWindowsNetworkShares.

WhyshouldIuseRACOneNodeinsteadofOracleFailSafeonWindows?
OracleRACOneNodeprovidesbetterhighavailabilitythanOracleFailSafe.RACOneNode'sabilitytoonlinerelocateadatabaseoffersprotectionfrombothunplannedfailuresand
maintenanceoutages.FailSafeonlyprotectsfromfailuresandcannotonlinerelocateadatabase.RACOneNodesupportsonlinemaintenanceoperationssuchasonlinedatabase
patches,onlineOSpatchesandupgrades,onlinedatabaserelocationforloadbalancing,onlineservermigrations,andonlineupgradetofullRAC.Inanenvironmentwhereitisdifficult
togetwindowsofdowntimeformaintenance,thisisabigadvantage.Also,whereFailSafeisonlyavailableonWindows,RACOneNodeisavailableonallplatforms.Acustomerwitha
mixedplatformenvironmentwouldbenefitfromhavingastandardHAsolutionacrossalltheirplatforms.

WhenrunningOracleRAConWindows2003,whatistherecommendedOSlevel?
ItisstronglyrecommendedtobeatSP2onWindows2003.FordetailsseethefollowingNotes:
Note:464683.1UnexplainedDatabaseSlowdownSeenonWindows2003ServicePack1
Note:454607.1NewPartitionsinWindows2003RACEnvironmentsNotVisibleonRemoteNodes
FordetailsonWindowsBundlepatchessee:
Note:342443.110.2.0.xOracleDatabaseandNetworkingPatchesforMicrosoftPlatforms

CanIrunmyOracle9iRACandOracleRAC10gonthesameWindowscluster?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

41/72

8/24/2015

CanIrunmyOracle9iRACandOracleRAC10gonthesameWindowscluster?

Document220970.1

YesbuttheOracle9iRACdatabasemusthavethe9iClusterManagerandyoumustrunOracleClusterwarefortheOracleDatabase10g.9iClusterManagercancoexsistwithOracle
Clusterware10g.
Besuretousethesame'clustername'intheappropriateOUIfieldforboth9iand10gwhenyouinstallbothtogetherinthesamecluster.
TheOracleCMService9iservicewillremainintactduringtheOracleClusterware10ginstall,asaOracle9iRACdatabasewouldrequirethatthe9iOracleCMService9i,itshouldbeleft
running.Theinformationforthe9idatabasewillgetmigratedtotheOCRduringtheOracleClusterwareinstallation.Then,forfuturedatabasemanagement,youwouldusethe9isrvctl
tomanagethe9idatabase,andthe10gsrvctltomanageanynew10gdatabases.BothsrvctlcommandswillusetheOCR.ThesameappliesforOracleRAC11g

WhenusingMSVSSonWindowswithOracleRAC,doIneedtoruntheVSSoneachnodewhereIhaveanOracleRACinstance?
ThereisnoneedtorunOracleVSSwriterinstanceoneachOracleRACnode(eventhoughitisinstalledandenabledbydefaultonallnodes).AndthedocumentationinWindows
PlatformDocforOracleVSSwriterisapplicabletoOracleRACalso.
TheabilityofclusteredfilesystemtocreateaWindowsShadowcopyisaMUSTtobackupOracleRACdatabaseusingOracleVSSwriter.Theonlyotherrequirementisthat,allthe
archivedlogsgeneratedbydatabasemustbeaccessibleonnodewherebackupisinitiatedusingOracleVSSwriter.
VSScoordinatesstoragesnapshotofdbfilestheVSSwriterplacesthedbinhotbackupmodesothattheVSSprovidercaninitiatethesnapshot.So,RMANisnotbackingupanythingin
thiscase.WhenaVSSrestoreofadbisissued,thewriterautomaticallyinvokesRMANtoperformneededrecoveryactionsafterthesnapshotisrestoredbytheproviderthatisthe
realvalueaddofthewriter.

WhatdoIdowhenIgetanORA01031errorloggingintotheASMinstance?
ThissoundsliketheORA_DBAgrouponNode2isempty,orelsedoesnothavethecorrectusernameinit.DoublecheckwhatuseraccountyouareusingtologontoNode2as(a'set'
commandwillshowyoutheUSERNAMEandUSERDOMAINvalues)andthenmakesurethatthisaccountispartofORA_DBA.
TheotherissuetocheckisthatSQLNET.AUTHENTICATION_SERVICES=(NTS)issetintheSQLNET.ORA

TheOracleCRServicedoesnotstartwithmywindowsOracleRACimplementation,whatdoIdo?
IfOracleCRServicedoesn'tstartthat'squiteadifferentissuethansayOracleCSServicenotstartingbecauseduetodependencies,thisisthelastofthethreeOracleClusterware
servicesthatweexpecttostart.Thiscouldbecausedbyafewdifferentthings.Itcouldbecausedbyachangefromtoautonegotiateinsteadof100/fullontheinterconnect.Onceset
backto100/fullonallNICsaswellasthenetworkswitchassociatedwiththeinterconnecttheproblemisresolved.Thiscouldalsobe:inabilitytoaccesstheshareddiskhousingyour
OCRpermissionsissueORBug:4537790whichintroducedOPMDtobeginwithwhichforreferencesakewasloggedagainst9.2.0.8...andisstillrelevanttodayin10.2.0.3times.
ForOPMD,seeMetalinkNote358156.1

HowdoIverifythatHostBusAdapterNodeLocalCachinghasbeendisabledforthedisksIwillbeusinginmyRACcluster?
Disablingwritecachingisastandardpracticewhileusingthevolumemanagers/filesystemsareshared.GotoMycomputer>Manage>Storage>DiskManagement>DiskProperties
>Policies>anduncheckthe"EnableWriteCachingonDisk".Thiswilldisablethewritecaching.
3rdpartyHBA'smayhavetheirownmanagementtoolstomodifythesesettings.Justrememberthatcentralized,sharedcacheisgenerallyOK.It'sthenodelocalcachethatyouneedto
turnoff.HowexactlyyoudothiswillvaryfromHBAvendortoHBAvendor.

Mycustomerhasafailsafeclusterinstalled,whatarethebenefitsofmovingtheirsystemtoRAC?
FailSafedevelopmentiscontinuing.Mostworkontheproductwillbearoundaccomodatingchangesinthesupportedresources(newreleasesofRDBMS,AS,etc.)andtheunderlying
MicrosoftClusterServicesandWindowsoperatingsystem.
AfailsafeprotectedinstanceisanActive/Passiveinstanceso,assuch,doesnotbenefitthatmuchatallfromaddingmorenodestoacluster.MicrosofthavealimitofnodesinaMSCS
cluster.(typically8nodesbutitdoesvary).RACisactiveactivesoyougetdualbenefitsofincreasedscalabilityandavailabilityeverytimeyouaddanodetoacluster.Wehavealimit
of100nodesinaRACcluster(wedon'tuseMSCS).Yourcustomershouldreallyconsidermorethan2nodes.(becauseofaggregatecomputerpoweronnodefailure).Ifthechoiceis2
of4CPUnodesor4of2CPUnodethenIwouldgofor2CPUnodes.CustomersareusingbothWindowsItaniumRACandWindowsX64RAC.WindowsX64seemsmorepopular.
Keepinmind,though,thatforFailSafe,iftheserveris64Bit,regardlessofflavor,FailSafeManagermustbeinstalledona32Bitclient,whichwillcomplicatethingsjustabit.Thereis
nosuchrestrictionforRAC,asallmanagementforRACcanbedoneviaGridControlorDatabaseControl.
ForEERACyoucanimplementan'extendedcluster'wherethereisadistancebetweenthenodesinthecluster(usuallylessthan20KM).

MycustomerwantstounderstandwhattypeofdiskcachingtheycanusewiththeirWindowsRACCluster,theinstallguidetellsthemtodisablediskcaching?
IfthewritecacheidentifiedislocaltothenodethenthatisbadforRAC.Ifthecacheisvisibletoallnodesasa'singlecache',typicallyinthestoragearray,andisalso'batterybacked'

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

42/72

8/24/2015

Document220970.1

IfthewritecacheidentifiedislocaltothenodethenthatisbadforRAC.Ifthecacheisvisibletoallnodesasa'singlecache',typicallyinthestoragearray,andisalso'batterybacked'
thenthatisOK.

DoIneedHACMP/GPFStostoremyOCR/Votingfileonashareddevice.
TheprerequisitesdocforAIXclearlysays:
"IfyouarenotusingHACMP,youmustuseaGPFSfilesystemtostoretheOracleCRSfiles"==>thisisadocumentationbugandthiswillbefixedwith10.1.0.3
NotealsothatonAIXitisimportanttousethereserve_lock=no/reserve_policy=no_reservepershared,concurrentdeviceinordertoallowAIXtoaccessthedevicesfrommorethan
onenodesimultaneously.Checkthecurrentsettingusing:"/usr/sbin/lsattrElhdiskn|grepreserve".
Dependingonthetypeofstorageused,thecommandshouldreturn"no_reserve"orasimilarvalueforalldisksmeanttobeusedforOracleRAC.Ifrequiredd,usethe/dev/rhdisk
devices(characterspecial)forthecrsandvotingdiskandchangetheattributewiththefollowingcommand
chdevlhdisknareserve_lock=no
(forESS,EMC,HDS,CLARiiON,andMPIOcapabledevicesyouhavetodoanchdevlhdisknareserve_policy=no_reserve)

IsVIOsupportedwithRAConIBMAIX?
VIOissupportedonIBMAIX.Pleasecheck**CertifyUnixRACTechnologyMatrix**forthedetails.

IsHACMPneededforRAConAIX5.2usingGPFSfilesystem?
ThenewestversionofGPFScanbeusedwithoutHACMP,ifitisavailableforAIX5.2thenyoudonotneedHACMP.

CanIrunOracleRAC10gonmyIBMMainframeSysplexenvironment(z/OS)?
YES!ThereisnoseparatedocumentationforRAConz/OS.Whatyouwouldcall"clusterware"isbuiltintotheOSandthenativefilesystemsareglobal.IBMz/OSdocumentation
explainshowtosetupaSysplexClusteroncethecustomerhasdonethatitistrivialtosetupaRACdatabase.ThefewstepsinvolvedarecoveredininChapter14oftheOraclefor
z/OSSystemAdminGuide,whichyoucanreadhere.ThereisalsoanInstallGuideforOracleonz/OS(here)butIdon'tthinkthereareanyRACspecificstepsintheinstallation.Bythe
way,RAConz/OSdoesnotuseOracle'sclusterware(CSS/CRS/OCR).

CanIuseOracleClusterwareforfailoveroftheSAPEnqueueandVIPserviceswhenrunningSAPinaRACenvironment?
Oraclehascreatedsapctltodothisanditisavailableforcertainplatforms.SAPCTLwillbeavailablefordownloadonSAPServicesMarketplaceonAIXandLinux.ForSolaris,itwillnot
beavailablein2007,useVeritasorSunCluster.

AreOracleApplicationscertifiedwithRAC?
ForSiebel,PeopleSoftseehttp://realworld.us.oracle.com/isv/siebel.htmOracle9iRAC(9.2)andOracleRAC10g(10.1)arecertifiedwithOracleApplicationsEBusinessSuute.SeeNote:
285267.1fordetails.

Whatarethecdmpdirectoriesinthebackground_dump_destusedfor?
Thesedirectoriesareproducedbythediagnosibilitydaemonprocess(DIAG).DIAGisadatabaseprocesswhichasoneofitstasks,performscachedumping.TheDIAGprocessdumps
outtracingtofilewhenitdiscoversthedeathofanessentialprocess(foregroundorbackground)inthelocalinstance.Adumpdirectorynamedsomethinglikecdmp_iscreatedinthe
bdumporbackground_dump_destdirectory,andallthetracedumpfilesDIAGcreatesareplacedinthisdirectory.

HowdoIgatherallrelevantOracleandOSlog/tracefilesinanOracleRACclustertoprovidetoSupport?
UseRACDDT(RACDiagnosticDataTool),UserGuideisinNote:301138.1.QuotefromtheUserGuide:
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

43/72

8/24/2015

Document220970.1

RACDDTisadatacollectiontooldesignedandconfiguredspecificallyforgatheringdiagnosticdatarelatedtoOracle'sRealApplicationCluster(RAC)
technology.RACDDTisasetofscriptsandconfigurationfilesthatisrunononeormorenodesofanOracleRACcluster.ThemainscriptiswritteninPerl,
whileanumberofproxyscriptsarewrittenusingKornshell.RACDDTwillrunonallsupportedUnixandLinuxplatforms,butisnotsupportedonanyWindows
platforms.

NewerversionsofRDA(RemoteDiagnosticAgent)havetheRACDDTfunctionality,sogoingforwardRDAisthetoolofchoice.TheRDAUserGuideisinNote:314422.1

WhatistheoptimalmigrationpathtobeusedwhilemigratingtheEBusinesssuitetoOracleRAC?
FollowingistherecommendedandmostoptimalpathtomigrateyouEBusinesssuitetoanOracleRACenvironment:
1.Migratetheexistingapplicationtonewhardware.(Ifapplicable).
2.UseClusteredFileSystem(ASMrecommended)foralldatabasefilesormigratealldatabasefilestorawdevices.(UseddforUnixorocopyforNT)
3.Install/upgradetothelatestavailableeBusinesssuite.
4.EnsurethedatabaseversionissupportedwithOracleRAC
5.Instep4,installOracleRACoptionanduseInstallertoperforminstallforallthenodes.
6.CloneOracleApplicationcodetree.
ReferenceDocuments:
OracleEBusinessSuiteRelease11iwith9iRAC:InstallationandConfiguration:Note:279956.1
EBusinessSuite11ionRAC:ConfiguringDatabaseLoadbalancing&Failover:Note:294652.1
OracleEBusinessSuite11iandDatabaseFAQ:Note:285267.1

IstheOracleEBusinessSuite(OracleApplications)certifiedagainstRAC?
Yes.(ThereisnoseperatecertificationrequiredforRAC.)

""

CanIuseTAFwitheBusinessinaRACenvironment?
TAFitselfdoesnotworkwitheBusinesssuiteduetoForms/TAFlimitations,butyoucanconfigurethetnsfailoverclause.Oninstancefailure,whentheuserlogsbackintothesystem,
theirsessionwillbedirectedtoasurvivinginstance,andtheuserwillbetakentothenavigatortab.Theircommittedworkwillbeavailableanyuncommittedworkmustberestarted.
WealsorecommendyouconfiguretheformserrorURLtoidentifyafallbackmiddletierserverforFormsprocesses,ifnorouterisavailabletoaccomplishswitchingacrossservers.

HowtoconfigureconcurrentmanagerinaRACenvironment?
Largeclientscommonlyputtheconcurrentmanageronaseparateservernow(inthemiddletier)toreducetheloadonthedatabaseserver.Theconcurrentmanagerprogramscanbe
tiedtoaspecificmiddletier(e.g.,youcanhaveCMsrunningonmorethanonemiddletierbox).ItisadvisabletousespecilizeCM.CMmiddletiersaresetuptopointtotheappropriate
databaseinstancebasedonproductmodulebeingused.

ShouldfunctionalpartitioningbeusedwithOracleApplications?
Wedonotrecommendfunctionalpartitioningunlessthroughputonyourserverarchitecturedemandsit.Cachefusionhasbeenoptimizedtoscalewellwithnonpartitionedworkload.
Ifyourprocessingrequirementsareextremeandyourtestingprovesyoumustpartitionyourworkloadinordertoreduceinternodecommunications,youcanuseProfileOptionsto
designatethatsessionsforcertainapplicationsResponsibilitiesarecreatedonaspecificmiddletierserver.Thatmiddletierserverwouldthenbeconfiguredtoconnecttoaspecific
databaseinstance.
Todeterminethecorrectpartitioningforyourinstallationyouwouldneedtoconsiderseveralfactorslikenumberofconcurrentusers,batchusers,modulesused,workload
characteristicsetc.

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

44/72

8/24/2015

characteristicsetc.

Document220970.1

WhicheBusinessversionisprefereable?
Versions11.5.5onwardsarecertifiedwithOracle9iandhencewithOracle9iRAC.Howeverwerecommendthelatestavailableversion.

CanIuseAutomaticUndoManagementwithOracleApplications?
Yes.InaRACenvironmentwehighlyrecommendit.

IsServerSideLoadBalancingsupported/recommended/proventechnologyinOracleEBusinessSuite?
Yes,Customersareusingitsuccessfullytoday.ItisrecommendedtosetupbothClientandServersideloadbalancing.Notethatthepiecescomingfrom8.0.6home(formsandccm),
connectionsaredirectedtoRACinstancebasedonthesequenceitslistedintheTNSentrydescriptionlistandmaynotgetloadbalancedoptimally.ForOracleRAC10.2orhigherdonot
setPREFER_LEAST_LOADED_NODE=OFFinyourlistener.ora,pleasesettheCLB_GOALontheservice.

WhatarethemaximumnumberofnodesunderOCFSonLinux?
Oracle9iRAConLinux,usingOCFSfordatafiles,canscaletoamaximumof32nodes.Accordingtothe**OCFS2UserGuideUserGuide,OCFS2cansupportupto255nodes.

WhatfilescanIputonLinuxOCFS?
Foroptimalperformance,youshouldonlyputthefollowingfilesonLinuxOCFS:
Datafiles
ControlFiles
RedoLogs
ArchiveLogs
SPFILE

OracleClusterwarefilesOCRandVotingDiskcanbeputonOCFS2howeverBestPracticeistoputthemonraworblockdevices.

WherecanIfinddocumentationonOCFS?
ForMainPage>>>http://oss.oracle.com/projects/ocfs/ForUserManual>>>http://oss.oracle.com/projects/ocfs/documentation/ForOCFSFiles>>>
http://oss.oracle.com/projects/ocfs/files/supported/

WhataretheBestPracticesforusingaclusteredfilesystemwithOracleRAC?
CanIuseaclusterfilesystemforOCR,VotingDisk,Binariesaswellasdatabasefiles?
OracleBestPracticeforusingClusterFileSystems(CFS)withOracleRAC
*OracleClusterwarebinariesshouldnotbeplacedonaCFSasthisreducesclusterfunctionalitywhileCFSisrecovering,andalsolimitstheabilitytoperformrollingupgradesofOracle
Clusterware.
*OracleClusterwarevotingdisksandtheOracleClusterRegistry(OCR)shouldnotbeplacedonaCFSastheI/OfreezeduringCFSreconfigurationcanleadtonodeeviction,orcluster
managementactivitiestofail(I.Estart,stop,orcheckofaresource).
*OracleDatabase10gbinariesaresupportedonCFSforOracleRAC10gandforOracleDatabase.ThesystemshouldbeconfiguredtosupportmultipleORACLE_HOMEsinorderto
maintaintheabilitytoperformarollingpatchapplication.
*OracleDatabase10gdatabasefiles(e.g.datafiles,tracefiles,andarchivelogfiles)aresupportedonCFS.
CheckCertifyforcertifiedclusterfilesystems.
RollingUpgradeswithClusterFileSystemsinGeneral
Itisnotrecommendedtouseaclusterfilesystem(CFS)fortheOracleClusterwarebinaries.OracleClusterwaresupportsinplacerollingupgrades.UsingasharedOracleClusterware
homeresultsinaglobaloutageduringpatchapplicationandupgrades.AworkaroundisavailabletoclonetheOracleClusterwarehomeforeachupgrade.Thisisnotcommonpractice.
Ifapatchismarkedforrollingupgrade,thenitcanbeappliedtoaOracleRACdatabaseinarollingfashion.OraclesupportsrollingupgradesfortheOracleDatabaseAutomaticStorage

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

45/72

8/24/2015

Document220970.1

Ifapatchismarkedforrollingupgrade,thenitcanbeappliedtoaOracleRACdatabaseinarollingfashion.OraclesupportsrollingupgradesfortheOracleDatabaseAutomaticStorage
Management(ASM)afteryouhaveupgradedtoOracleDatabase11g.WhenusingaCFSforthedatabaseandASMOraclehomes,theCFSshouldbeconfiguredtouseofcontext
dependentlinks(CDSLs)orequivalentandtheseshouldconfiguredtoworkinconjunctionwithrollingupgradesanddowngrades.ThisincludesupdatingthedatabaseandASMhomesin
theOCRtopointtothecurrenthome.
ThisisincludedinMetalinkNote444134.1

CanIuseOCFSwithSEOracleRAC?
ItisnotsupportedtouseOCFSwithStandardEditionOracleRAC.AlldatabasefilesmustuseASM(redologs,recoveryarea,datafiles,controlfilesetc).Youcannotplacebinarieson
OCFSaspartoftheSEOracleRACterms.Werecommendthatthebinariesandtracefiles(nonASMsupportedfiles)tobereplicatedonallnodes.Thisisdoneautomaticallybyinstall.

IsSunQFSsupportedwithOracleRAC?WhataboutSunGFS?
Fromcertify,checkthereforthelatestdetails.
SunClusterSunStorEdgeQFS(9.2.0.5andhigher,10gand10gR2):
NorestrictionsonplacementoffilesonQFS
SunStorEdgeQFSissupportedforOraclebinaryexecutables,databasedatafiles,databasedatafiles,archivelogs,OracleClusterRegistry(OCR),OracleClusterReadyServicesvoting
diskandrecoveryareacanbeplacedonQFS.
SolarisVolumeManagerforSunClustercanbeusedforhostbasedmirroring
Supportsupto8nodes

IsRedHatGFS(GlobalFileSystem)iscertifiedbyOracleforusewithOracleRealApplicationClusters?
SistinaClusterFilesystemisnotpartofthestandardRedHatkernelandthereforeisnotcertifiedbyOraclebutfallsunderakernelextension.Thishowever,doesnotmeanthatOracle
RACisnotcertifiedwithit.Asafact,OracleRACdoesnotcertifyagainstafilesystemperse,butcertifiesagainstanoperatingsystem.If,asisthecasewithSistinafilesystem,the
filesystemiscertifiedwiththeoperatingsystem,thisonlymeansthattheOracledoesnotprovidedirectsupportandfixthefilesystemincaseofanerror.Customerwillhavetocontact
thefilesystemproviderforsupport.

IsLinuxOCFS2(OCFSversion2)supportedwithOracleRAC?
YesSeeCertifyfordetailsonwhichplatformsaresupported.

WhatisthemaximumnumberofnodesIcanhaveinmyclusterifIamusingOCFS2?
Theroeticallyyoucanhaveupto255howeverithasbeentestedwithupto16nodes.

Ihavea2nodeOracleRACcluster,ifIpulltheinterconnectonnode1tosimulateafailure,whydoesnode2reboot?
Incaseofaprivatenetworkfailureandinordertopreventasplitbrainscenario,OracleClusterwarealwaystriestoletthebiggestsubclustersurvive.
Incaseofa2nodecluster,thedecisionisbasedonthenodenumber.Thenodewiththelowestnodenumberismeanttosurviveanevictiondecision.Thenodewiththelowestnode
numberistypicallythenodethatstartedfirst(betweenthetwonodes).Inthiscontext,itneedstobenotedthatthedecisionisindependentofwhereinthestacktheprivatenetwork
failureoccurred.Giventheexamplephrasedinthequestion:
Ihavea2nodeOracleRACcluster,ifIpulltheinterconnectonnode1tosimulateafailure,whydoesnode2reboot?
Thefirstnodethatstarted(joined/created)theclusterisnode1.Thisnodewillhavethelowestnodenumber.Regardlessonwhichnodeaprivateinterconnectcableispulled,node1
shouldsurviveandhencenode2isrebooted.

WhichprocessesaccesstheOCR?
OracleClusterRegistry(OCR)isusedtostoretheclusterconfigurationinformationamongotherthings.OCRneedstobeaccessiblefromallnodesinthecluster.IfOCRbecame
inaccessibletheCSSdaemonwouldsoonfail,andtakedownthenode.PMONneverneedstowritetoOCR.ToconfirmifOCRisaccessible,tryocrcheckfromyourORACLE_HOMEand
OracleClusterware/GRID_HOME.

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

46/72

8/24/2015

OracleClusterware/GRID_HOME.

Document220970.1

WithOracleClusterware10g,howdoyoubackuptheOCR?
ThereisanautomaticbackupmechanismforOCR.Thedefaultlocationis:$CLUSTERWARE_HOME\cdata\"clustername"\
Todisplaybackups:
#ocrconfigshowbackup
Torestoreabackup:
#ocrconfigrestore
Theautomaticbackupmechanismkeepsuptoaboutaweekoldcopy.So,ifyouwanttoretainabackupcopymorethanthat,thenyoushouldcopythat"backup"filetosomeother
name.
UnfortunatelywithOracleRAC10gRelease1thereareacoupleofbugsregardingbackupfilemanipulation,andchangingdefaultbackupdironWindows.Thesewerefixedin10.1.0.4.
OCRbackuponWindowsareabsent.Onlyfileinthebackupdirectoryistemp.ocrwhichwouldbethelastbackup.Youcanrestorethismostrecentbackupbyusingthecommandocr
restoretemp.ocr
WithOracleRAC10gRelease2orlater,youcanalsousetheexportcommand:
#ocrconfigexportsonline,anduseimportoptiontorestorethecontentsback.
WithOracleRAC11gRelease1,youcandoamanaualbackupoftheOCRwiththecommand:
#ocrconfigmanualbackup

Mycustomerhasnoticedtonsoflogfilesgeneratedunder$CRS_HOME/log//client,isthereanywayautomatedwaywecansetupthroughOracleClusterwareto
prevent/minimize/removethoseaggressivelygeneratedfiles?
CheckNote.5187351.8Youcaneitherapplythepatchsetifitisavailableforyourplatformorhaveacronjobthatremovesthesefilesuntilthepatchisavailable.

WhatdoIdo,IhaveacorruptOCRandnovalidbackup?
Note:399482.1describeshowtorecreateyourOCR/VotingDiskwhichyouhaveaccidentallydeletedandcannotrecoverfrombackups

WhyisthehomeforOracleClusterware/OracleGridInfrastructurenotrecommendedtobeasubdirectoryoftheOraclebasedirectory?
IfanyoneotherthanroothaswritepermissionstotheparentdirectoriesoftheOracleClusterwarehome/OracleGridInfrastructureforaClusterhome,thentheycangivethemselves
rootescalations.Thisisasecurityissue.
Consequenely,itisstronglyrecommendedtoplacetheOracleGridInfrastructure/OracleClusterwarehomeoutsideoftheOracleBase.TheOracleUniversalInstallerwillconfirm
deviatingsettingsduringtheOracleGridInfrastructure11gRelease2andlaterinstallation.
TheOracleClusterwarehomeitselfisamixofrootandnonrootpermissions,asappropriatetothesecurityrequirements.Please,followtheinstallationguidesregardingOSusersand
groupsandhowtostructuretheOraclesoftwareinstallationsonagivensystem.

DoIneedtohaveuserequivalence(ssh,etc...)setupafterGRID/RACisalreadyinstalled?
Yes.Manyassistantsandscriptsdependonuserequivalencebeingsetup.

WithGNS,doALLpublicaddresseshavetobeDHCPmanaged(publicIP,publicVIP,publicSCANVIP)?
No,ThechoicetouseDHCPforthepublicIPsisoutsideofOracle.OracleClusterwareandOracleRACwillworkwithbothstaticandDHCPassignedIPforthehostnames.Whenusing
GNS,OracleClusterwrewilluseDHCPforallVIPsinthecluster,whichmeansnodevipsandSCANvips.

HowistheOracleClusterRegistry(OCR)storedwhenIuseASM?
TheOCRisstoredsimilartohowOracleDatabasefilesarestored.Theextentsarespreadacrossallthedisksinthediskgroupandtheredundancy(whichisattheextentleve)isbased
ontheredundancyofthediskgroup.YoucanonlyhaveoneOCRinadiskgroup.BestPracticeforASMistohave2diskgroups.BestPracticeforOCRinASMistohaveacopyoftheOCR
ineachdiskgroup.

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

47/72

8/24/2015

Document220970.1

WhendoestheOraclenodeVIPfailovertoanothernodeandsubsequentlyreturntoitshomenode?
ThehandlingoftheVIPwithrespecttoafailovertoanothernodeandsubsequentreturntoitshomenodeishandleddifferentlydependingontheOracleClusterwareversion.In
general,onecandistinguishbetweenOracleClusterware10g&11gRelease1andOracleClusterware11gRelease2behavior.
ForOracleClusterware10g&11gRelease1theVIPwillfailovertoanothernodeeitherafteranetworkoranodefailure.However,theVIPwillautomaticallyreturntoitshomenode
onlyafteranodefailureandasubsequentrestartofthenode.SincethenetworkisnotconstantlymonitoredinthisOracleClusterwareversion,thereisnowaythatOracleClusterware
candetecttherecoveryofthenetworkandinitiateanautomaticreturnofthenodeVIPtoitshomenode.
Exception:WithOraclePatchSet10.2.0.3anewbehaviorwasintroducedthatallowedthenodeVIPtoreturntoitshomenodeafterthenetworkrecovered.Therequirednetwork
checkwaspartofthedatabaseinstancecheck.However,thisnewcheckintroducedquitesomesideeffectsandhence,wasdisabledwithsubsequentbundlepatchesandtheOracle
PatchSet10.2.0.4
Startingwith10.2.0.4andforOracleClusterware11gRelease1thedefaultbehavioristoavoidanautomaticreturnofthenodeVIPtoitshomenodeafterthe
networkrecovered.Thisbehaviorcanbeactivated,ifrequired,usingthe"ORA_RACG_VIP_FAILBACK"parameter.Thisparametershouldonlybeusedafterreviewingsupportnote
805969.1(VIPdoesnotrelocatebacktotheoriginalnodestartingfrom10.2.0.4and11.1evenafterthepublicnetworkproblemisresolved.)
WithOracleClusterware11gRelease2thedefaultbehavioristoautomaticallyinitiateareturnofthenodeVIPtoitshomenodeassoonasthenetworkrecoveredafterafailure.It
needstobenotedthatthisbehaviorisnotbasedontheparametermentionedaboveandthereforedoesnotinducethesamesideeffects.Instead,anewnetworkresourceisusedin
OracleClusterware11gRelease2,whichmonitorsthenetworkconstantly,evenafterthenetworkfailedandtheresourcebecame"OFFLINE".Thisfeatureiscalled"OFFLINEresource
monitoring"andisperdefaultenabledforthenetworkresource.

HowdoIprotecttheOCRandVotingincaseofmediafailure?
InOracleDatabase10gRelease1theOCRandVotingdevicearenotmirroredwithinOracle,hencebothmustbemirroredviaastoragevendormethod,likeRAID1.
StartingwithOracleDatabase10gRelease2OracleClusterwarewillmultiplextheOCRandVotingDisk(twofortheOCRandthreefortheVoting).
PleasereadNote:279793.1andNote:268937.1regardingbackupandrestorealostVoting/OCR
@andFAQ10734regardingOCRbackup.

HowdoIusemultiplenetworkinterfacestoprovideHighAvailabilityand/orLoadBalancingformyinterconnectwithOracleClusterware?
ThisneedstobedoneexternallytoOracleClusterwareusuallybysomeOSprovidednicbondingwhichgivesOracleClusterwareasingleipaddressfortheinterconnectbutprovide
failover(HighAvailability)and/orloadbalancingacrossmultipleniccards.ThesesolutionsareprovidedexternallytoOracleatamuchlowerlevelthantheOracleClusterware,hence
Oraclesupportsusingthem,thesolutionsareOSdependentandthereforethebestsourceofinformationisfromyourOSVendor.However,thereareseveralarticlesinMetalinkonhow
todothis.ForexampleforSunSolarissearchforIPMP(IPnetworkMultiPathing).
Note:Customershouldpaycloseattentiontothebondingsetup/configuration/featuresandensuretheirobjectivesaremet,sincesomesolutionsprovideonlyfailoverandsomeonly
loadbalancingstillothersclaimtoprovideboth.Asalways,it'salwaysimportanttotestyoursetuptoensureitdoeswhatitwasdesignedtodo.
@ForLinux,readthedoconrac.us:
@ConfigureRedundantNetworkCards/SwitchesforOracleDatabase10gRelease1RealApplicationClusteronLinux
WhenbondingwithNetworkInterfacesthatconnecttoseparateswitches(forredundancy)youmusttestiftheNIC'sareconfiguredforactive/activemode.Themostreliable
configurationforthisarchitectureistoconfiguretheNIC'sforActive/Passive.

CantheNetworkInterfaceCard(NIC)devicenamesbedifferentonthenodesinacluster,forbothpublicandprivate?
AllpublicNICsmusthavethesamenameonallnodesinthecluster
@(ER5439875filedtoremovethisrequirement).
Similarly,allprivateNICsmustalsohavethesamenamesonallnodes
@(ER6785792filedtoremovethisrequirement).
DonotmixNICswithdifferentinterfacetypes(infiniband,ethernet,hyperfabric,etc.)forthesamesubnet/network.

CanIruna10.1.0.xdatabasewithOracleClusterware10.2?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

48/72

8/24/2015

CanIruna10.1.0.xdatabasewithOracleClusterware10.2?

Document220970.1

Yes.OracleClusterware10.2willsupportboth10.1and10.2databases(andASMtoo!).AdetailedmatrixisavailableinNote:337737.1

Isitsupportedtorerunroot.shfromtheOracleClusterwareinstallation?
ForOracleRAC10grerunningroot.shaftertheinitialsuccessfulinstalloftheOracleClusterwareisexpresslydiscouragedandunsupported.Westronglyrecommendnotdoingit.
Incasewhereroot.shisfailingtoexecuteforonaninitialinstall(oranewnodejoininganexistingcluster),itisOKtorerunroot.shafterthecauseofthefailureiscorrected
(permissions,paths,etc.).Inthiscase,pleaserunrootdelete.shtoundothelocaleffectsofroot.shbeforererunningroot.sh.

Whenctrunthecommand'onsctlstart'receivesthemessage"Unabletoopenlibhasgen10.so".Anyideawhythemessage"unabletoopenlibhasgen10.so"?
MostlikelyyouaretryingtostartONSfromORACLE_HOMEinsteadofOracleClusterware(orGridInfrastructurein11.2)home.PleasetrytostartitfromtheOracleClusterwarehome.

VotingFilesstoredinASMHowmanydisksperdiskgroupdoIneed?
IfVotingFilesarestoredinASM,theASMdiskgroupthathoststheVotingFileswillplacetheappropriatenumberofVotingFilesinaccordancetotheredundancylevel.OnceVoting
FilesaremanagedinASM,amanualaddition,deletion,orreplacementofVotingFileswillfail,sinceusersarenotallowedtomanuallymanageVotingFilesinASM.
Iftheredundancylevelofthediskgroupissetto"external",1VotingFileisused.
Iftheredundancylevelofthediskgroupissetto"normal",3VotingFilesareused.
Iftheredundancylevelofthediskgroupissetto"high",5VotingFilesareused.
NotethatOracleClusterwarewillstorethediskwithinadiskgroupthatholdstheVotingFiles.OracleClusterwaredoesnotrelyonASMtoaccesstheVotingFiles.
Inaddition,notethattherecanbeonlyoneVotingFileperfailuregroup.Intheabovelistofrules,itisassumedthateachdiskthatissupposedtoholdaVotingFileresidesinitsown,
dedicatedfailuregroup.
Inotherwords,adiskgroupthatissupposedtoholdtheabovementionednumberofVotingFilesneedstohavetherespectivenumberoffailuregroupswithatleastonedisk.(1/3/5
failuregroupswithatleastonedisk)
Consequently,anormalredundancyASMdiskgroup,whichissupposedtoholdVotingFiles,requires3disksinseparatefailuregroups,whileanormalredundancy
ASMdiskgroupthatisnotusedtostoreVotingFilesrequiresonly2disksinseparatefailuregroups.

OCRstoredinASMWhathappens,ifmyASMinstancefailsonanode?
IfanASMinstancefailsonanynode,theOCRbecomesunavailableonthisparticularnode,butthenoderemainsoperational.
Ifthe(RAC)databasesuseASM,too,theycannotaccesstheirdataonthisnodeanymoreduringthetimetheASMinstanceisdown.IfaRACdatabaseisused,accesstothesamedata
canbeestablishedfromanothernode.
IftheCRSDprocessrunningonthenodeaffectedbytheASMinstancefailureistheOCRwriter,ANDthemajorityoftheOCRlocationsisstoredinASM,ANDanIOisattemptedonthe
OCRduringthetimetheASMinstanceisdownonthisnode,THENCRSDstopsandbecomesinoperable.Henceclustermanagementisaffectedonthisparticularnode.
UndernocircumstanceswillthefailureofoneASMinstanceononenodeaffectthewholecluster.

IsitpossibletouseASMfortheOCRandvotingdisk?
Yes.AsofOracleRealApplicationClusters11gRelease2,theOCRandVotingDiskscanbestoredinASM.Thisistherecommendedbestpracticeforthisrelease.
Forreleasespriorto11gRelease2,theOCRandvotingdiskmustbeonRAWdevicesorCFS(clusterfilesystem).
RAWdevices(orblockdevicesonLinux)isthebestpracticeforOracleRAC10gorOracleRAC11gRelease1.

Iamtryingtomovemyvotingdisksfromonediskgrouptoanotherandgettingtheerror"crsctlreplacevotedisknotpermittedbetweenASMDiskGroups."
Why?

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

49/72

8/24/2015

Why?

Document220970.1

YouneedtoreviewtheASMandcrsctllogstoseewhythecommandisfailing.
ToputyourvotingdisksinASM,youmusthavethediskgroupsetupproperly.Theremustbeenoughfailuregroupstosupporttheredundancyofthevotingdisksassetbythe
redundancyonthediskgroup.EG:Normalredundancy,3failuregroupsarerequried,Highredundancy,5failuregroups.Note:bydefaulteachdiskinadiskgroupisputinitsown
failuregroup.Thecompatible.asmattributeofthediskgroupmustbesetto11.2andyoumustbeusing11.2versionofOracleClusterwareandASM.

CanIrunthefixupscriptgeneratedbythe11.2OUIorCVUonarunningsystem?
Itdependsonwhattheproblemthatwerelistedtobefixed.Thefixupscriptscanchangesystemparameterssoyoushouldnotchangesystemparameterswhileapplicationsare
running.However,ifanearlierversionofOracleDatabaseisalreadyrunningonthesystem,thereshouldnotbeanyneedtochangethesystemparameters.

WhatshouldthepermissionsbesettoforthevotingdiskandocrwhendoinganOracleRACInstall?
TheOracleRealApplicationClustersinstallguideiscorrect.ItdescribesthePREINSTALLownership/permissionrequirementsforocrandvotingdisk.Thisstepisneededtomakesure
thattheOracleClusterwareinstallsucceeds.Pleasedon'tusethosevaluestodeterminewhattheownership/permmissionshouldbePOSTINSTALL.Therootscriptwillchangethe
ownership/permissionofocrandvotingdiskaspartofinstall.ThePOSTINSTALLpermissionswillendupbeing:OCRroot:oinstall640VotingDiskoracle:oinstall644

HowtomovetheOCRlocation?
ForOracleRAC10gRelease1
stoptheCRSstackonallnodesusing"init.crsstop"
Edit/var/opt/oracle/ocr.loconallnodesandsetupocrconfig_loc=newOCRdevice
Restorefromoneoftheautomaticphysicalbackupsusingocrconfigrestore.
Runocrchecktoverify.
reboottorestarttheCRSstack.
ForOracleRAC10gRelease2orlaterPleaseusetheOCRcommandtoreplacetheOCRwiththenewlocation:
#ocrconfigreplaceocr/dev/newocr
#ocrconfigreplaceocrmirror/dev/newocrmirror
Manualeditingofocr.locorequivalentisnotrecommended,andwillnotwork.

IamtryingtoinstallOracleClusterware(10.2)andwhenIruntheOUI,attheSpecifyClusterConfigurationscreen,theAdd,EditandRemovebuttonsaregrayed
out.Nothingcomesupintheclusternodeseither.Why?
Checkfor3rdPartyVendorclusterware(suchasSunClusterorVeritasCluster)thatwasnotcompletelyremoved.IELookfor/opt/ORCLclusterdirectory,itshouldberemoved.

WhathappensifIlosemyvotingdisk(s)?
Ifyoulose1/2ormoreofallofyourvotingdisks,thennodesgetevictedfromthecluster,ornodeskickthemselvesoutofthecluster.Itdoesn'tthreatendatabasecorruption.
AlternativelyyoucanuseexternalredundancywhichmeansyouareprovidingredundancyatthestoragelevelusingRAID.
ForthisreasonwhenusingOraclefortheredundancyofyourvotingdisks,Oraclerecommendsthatcustomersuse3ormorevotingdisksinOracleRAC10gRelease2.Note:Forbest
availability,the3votingfilesshouldbephysicallyseparatedisks.Itisrecommendedtouseanoddnumberas4diskswillnotbeanymorehighlyavailablethan3disks,1/2of3is
1.5...roundedto2,1/2of4is2,oncewelose2disks,ourclusterwillfailwithboth4votingdisksor3votingdisks.
Restoringcorruptedvotingdisksiseasysincethereisn'tanysignificantpersistentdatastoredinthevotingdisk.SeetheOracleClusterwareAdminandDeploymentGuideforinformation
onbackupandrestoreofvotingdisks.

IaminstallingOracleClusterwarewitha3rdpartyvendorclusterwarehoweverinthe"SpecifyClusterConfigurationPage",OracleClusterwareinstallerdoesn't
showtheexistingnodes.Why?
ThisshowsthatOracleClusterwaredoesnotdetectthe3rdPartyclusterwareisinstalled.Makesureyouhavefollowedtheinstallationinstructionsprovidedbythevendorforintegrating
withOracleRAC.MakesureLD_LIBRARY_PATHisnotset.
ForexamplewithSunCluster,makesurethelibskgxn*filestothe/opt/ORCLclusterdirectory.CheckthatlsnodesreturnsthecorrectlistofnodesintheSunCluster.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

50/72

8/24/2015

Document220970.1

ImadeamistakewhenIcreatedtheVIPduringtheinstallofOracleClusterware,canIchangetheVIP?
YesThedetailsofhowtodothisaredescribedinMetalinkNote.276434.1

HowshouldItestthefailureofthepublicnetwork(IEOracleVIPfailover)inmyOracleRACenvironment?
Priorto10.2.0.3,ItwaspossibletotestVIPfailoverbysimplyrunning
ifconfig<interface_name>down.
TheintendedbehaviourwasthattheVIPwouldfailovertotheanothernode.In10.2.0.3thisisthesamebehaviouronLinux,howeveronotheroperatingsystemstheVIPwillNOT
failover,insteadtheinterfacewillbeplumbedagain.TotestVIPfailoveronplatformsotherthanLinux,theswitchcanbeturnedofforthephysicalcablepulled.
Theisbestwaytotest.NOTE:ifyouhaveotherDBsthatsharethesameIPsthentheywillbeaffected.YourtestsshouldsimulateProductionfailureswhicharegenerallySwitcherrors
orinterfaceerrors.

Whatisthevotingdiskusedfor?
AvotingdiskisabackupcommunicationsmechanismthatallowsCSSdaemonstonegotiatewhichsubclusterwillsurvive.Thesevotingdiskskeepastatusofwhoiscurrentlyaliveand
countsvotesincaseofaclusterreconfiguration.Itworksasfollows:
a)Ensuresthatyoucannotjointheclusterifyoucannotaccessthevotingdisk(s)
b)Leavetheclusterifyoucannotcommunicatewithit(toensurewedonothaveaberrantnodes)
c)Shouldmultiplesubclustersform,itwillonlyallowonetocontinue.Itprefersagreaternumberofnodes,andsecondlythenodewiththelowestincarnationnumber.
d)IskeptredundantbyOraclein10gRelease2(youneedtoaccessamajorityofexistingvotingdisks)
Atmostonlyonesubclusterwillcontinueandasplitbrainwillbeavoided.

CanIconfigureafirewall(iptables)ontheclusterinterconnect?
Disableallfirewallsontheclusterinterconnect.Seenote:554781.1fordetails.

Doesthehostnamehavetomatchthepublicnameorcanitbeanythingelse?
Whenthereisnovendorclusterware,onlyOracleClusterware,thenthepublicnodenamemustmatchthehostname.Whenvendorclusterwareispresent,itdeterminesthepublicnode
names,andtheinstallerdoesn'tpresentanopportunitytochangethem.So,whenyouhaveachoice,alwayschoosethehostname.

Ihavea2nodeRACrunning.Inoticethatitisalwaysnode2thatisevictedwhenItestprivatenetworkfailurescenariobydisconnectingtheprivatenetwork
cable.Doesn'tmatterwhetheritisnode1'sornode2'sprivatenetworkcablethatisdisconnected,itisalwaysthenode2thatisevicted.Whathappensina3
nodesRACclusterifnode1'scableisdisconnected?
Thenodewiththelowernodenumberwillsurvive(Thefirstnodetojointhecluster).Incaseof3nodes,2nodeswillsurviveandtheoneyoupulledthecablewillgoaway.4nodes
thesubclusterwiththelowernodenumberwillsurvive.

CanIuseOracleClusterwaretoprovidecoldfailoverofmysingleinstanceOracleDatabases?
OracledoesnotprovidethenecessarywrapperstofailoversingleinstancedatabasesusingOracleClusterware.It'spossibleforcustomerstouseOracleClusterwaretowraparbitrary
applications,it'dbepossibleforthemtowrapsingleinstancedatabasesthisway.AsamplecanbefoundintheDEMOsthataredistributedwithOracleDatabase11g.

WhatarethelicensingrulesforOracleClusterware?CanIrunitwithoutRAC?
ChecktheOracleDatabaseLicensingInformation11gRelease1(11.1)PartNumberB2828701LookintheSpecialUsesectionunderOracleDatabaseEditions.

InthecourseoffailuretestinginanextendedRACenvironmentwefindentriesinthecssdlogfilewhichindicateactionslike'diskShortTimeoutsetto(value)'
and'diskLongTimeoutsetto(value)'.
Cananyonepleaseexplainthemeaningofthesetwotimeoutsinadditiontodisktimeout?

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

51/72

8/24/2015

Document220970.1

Cananyonepleaseexplainthemeaningofthesetwotimeoutsinadditiontodisktimeout?
Havingashortandlongdisktimeout,andnolongerjustonedisktimeout,isduetopatchforbug4748797(includedin10.2.0.2).Thelongdisktimeoutis200secbydefaultunlessset
differentlyvia'crsctlsetcssdisktimeout',andappliestotimeoutsideareconfiguration.Theshortdisktimeoutisineffectduringareconfigurationandismisscount3s.Thepointisthat
wecantoleratealongdisktimeoutwhenallnodesarejustrunningfine,buthavetorevertbacktoashortdisktimeoutifthere'sareconfiguration.

DuringOracleClusterwareinstallation,Iamaskedtodefineaprivatenodename,andthenonthenextscreenaskedtodefinewhichinterfacesshouldbeused
asprivateandpublicinterfaces.Whatinformationisrequiredtoanswerthesequestions?
TheprivatenamesonthefirstscreendeterminewhichprivateinterconnectwillbeusedbyCSS.
ProvideexactlyonenamethatmapstoaprivateIPaddress,orjusttheIPaddressitself.Ifalogicalnameisused,thentheIPaddressthismapstocanbechangedsubsequently,butif
youIPaddressisspecifiedCSSwillalwaysusethatIPaddress.CSScannotusemultipleprivateinterconnectsforitscommunicationhenceonlyonenameorIPaddresscanbespecified.
TheprivateinterconnectenforcementpagedetermineswhichprivateinterconnectwillbeusedbytheRACinstances.
It'sequivalenttosettingtheCLUSTER_INTERCONNECTSinit.oraparameter,butismoreconvenientbecauseitisaclusterwidesettingthatdoesnothavetobeadjustedeverytimeyou
addnodesorinstances.RACwillusealloftheinterconnectslistedasprivateinthisscreen,andtheyallhavetobeup,justastheirIPaddresseshavetobewhenspecifiedintheinit.ora
paramter.RACdoesnotfailoverbetweenclusterinterconnectsifoneisdownthentheinstancesusingthemwon'tstart.

CanIchangethenameofmyclusterafterIhavecreateditwhenIamusingOracleClusterware?
No,youmustproperlydeinstallOracleClusterwareandthenreinstall.ToproperlydeinstallOracleClusterware,youMUSTfollowthedirectionsinthe"OracleGridInfrastructure
InstallationGuide"forOracleRAC11gRelease2,chapter6"HowtoModifyorDeinstallOracleGridInfrastructure".Thiswillensuretheocrgetscleanedout.

WhydoesOracleClusterwareuseanadditional'heartbeat'viathevotingdisk,whenotherclustersoftwareproductsdonot?
OracleusesthisimplementationbecauseOracleclustersalwayshaveaccesstoashareddiskenvironment.Thisisdifferentfromclassicalclusteringwhichassumessharednothing
architectures,andchangesthedecisionofwhatstrategiesareoptimalwhencomparedtootherenvironments.Oraclealsosupportsawidevarietyofstoragetypes,insteadoflimitingit
toaspecificstoragetype(likeSCSI),allowingthecustomerquitealotofflexibilityinconfiguration.

WhydoesOraclestillusethevotingdiskswhenotherclustersofwareispresent?
Votingdisksarestillusedwhen3rdpartyvendorclusterwareispresent,becausevendorclusterwareisnotabletomonitor/detectallfailuresthatmattertoOracleClusterwareandthe
database.ForexampleoneknowncaseiswhenthevendorclusterwareissettohaveitsheartbeatgooveradifferentnetworkthanRACtraffic.Continuingtousethevotingdisksallows
CSStoresolvesituationswhichwouldotherwiseendupinclusterhangs.

Customerishittingbug4462367withanerrormessagesayinglowopenfiledescriptor,howdoIworkaroundthisuntilthefixisreleasedwiththeOracle
ClusterwareBundlefor10.2.0.3or10.2.0.4isreleased?
Thefixfor"lowopenfiledescriptor"problemistoincreasetheulimitforOracleClusterware.Pleasebecarefulwhenyoumakethistypeofchangeandmakeabackupcopy
oftheinit.crsdbeforeyoustart!Todothis,youcanmodifytheinit.crsdasfollows,whileyouwaitforthepatch:1.StopOracleClusterwareonthenode(crsctlstopcrs)
2.copythe/etc/init.d/init.crsd
3.Modifythefilechanging:
#Allowthedaemontodropadiagnosticcorefile/
ulimitcunlimited
ulimitnunlimited
to
#Allowthedaemontodropadiagnosticcorefile/
ulimitcunlimited
ulimitn65536
4.restartOracleClusterwareinthenode(crsctlstartcrs)

HowdoIidentifythevotingfilelocation?
Runthefollowingcommandfrom/bin
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

52/72

8/24/2015

Document220970.1

"crsctlquerycssvotedisk"

HowmuchI/Oactivityshouldthevotingdiskhave?
Approximately2read+1writepersecondpernode.

DoesOracleClusterwarehavetobethesameorhigherreleasethanallinstancesrunningonthecluster?
YesOracleClusterwaremustbethesameorahigherreleasewithregardstotheRDBMSorASMHomes.
PleaserefertoNote#337737.1

CanIuseOracleClusterwaretomonitormyEMAgent?
CheckoutChapter3oftheEMadvancedconfigurationguide,specificallythesectiononactivepassiveconfigurationofagents.Youshouldbeabletomodelthosetoyourrequirements.
Thereisnothingspecialaboutthecommands,butyoudoneedtofollowthestartup/shutdownsequencetoavoidanydiscontinuityofmonitoring.Theagentdoesstartawatchdogthat
monitorsthehealthoftheactualmonitoringprocess.Thisisdoneautomaticallyatagentstart.ThereforeyoucoulduseOracleClusterwarebutyoushouldnotneedto.

WhataretheIPrequirementsfortheprivateinterconnect?
TheinstallguidewilltellyouthefollowingrequirementsprivateIPaddressmustsatisfythefollowingrequirements:
1.Mustbeseparatefromthepublicnetwork
2.Mustbeaccessibleonthesamenetworkinterfaceoneachnode
3.Musthaveauniqueaddressoneachnode
4.Mustbespecifiedinthe/etc/hostsfileoneachnode
TheBestPraticesrecommendationistousetheTCP/IPstandardfornonrouteablenetworks.Reservedaddressrangesforprivate(nonrouted)use(seeTCP/IPRFC1918):
*10.0.0.0>10.255.255.255
*172.16.0.0>172.31.255.255
*192.168.0.0>192.168.255.255
Cluvfywillgiveyouanerrorifyoudonothaveyourprivateinterconnectintherangesabove.
Youshouldnotignorethiserror.IfyouareusinganIPaddressintherangeusedforthepublicnetworkfortheprivatenetworkinterfaces,youareprettymuchmessinguptheIP
addressing,andpossiblytheroutingtables,fortherestofthecorporation.IPaddressesareasparsecommodity,usethemwisely.Ifyouusethemonanonroutablenetwork,thereis
nothingtopreventsomeoneelsetogoandusetheminthenormalcorporatenetwork,andthenwhenthoseRACnodesfindoutthatthereisanotherpathtothataddressrange(through
RIP),theyjustmightstartsendingtraffictothoseotherIPaddressesinsteadoftheinterconnect.Thisisjustabadidea.

CanIsetupfailoveroftheVIPtoanothercardinthesamemachineorwhatdoIdoifIhavedifferentnetworkinterfacesondifferentnodesinmycluster(I.E.
eth0onnode1,2andeth1onnode3,4)?
Withsrvctl,youcanmodifythenodeappfortheVIPtolisttheNICsitcanuse.ThenVIPwilltrytostartoneth0interfaceandifitfails,tryeth1interface.
./srvctlmodifynodeappsnA//eth0\|eth1
Notehowtheinterfacesarealistseparatedbythe|symbolandhowyouneedtoquotethiswitha\characterortheUnixshellwillinterpretthecharacterasapipe.Soonanode
calledukdh364withaVIPaddressofukdh364vipandwewantanetmask(say)of255.255.255.0thenwehave:
./srvctlmodifynodeappsnukdh364Aukdh364vip/255.255.255.0/eth0\|eth1
Tocheckwhichinterfacesareconfiguredaspublicorprivateuseoifcfggetif
exampleoutput:
eth0138.2.238.0globalpublic
eth1138.2.240.0globalpublic
eth2138.2.236.0globalcluster_interconnect
Anifconfigonyourmachinewillshowwhatthehardwarenamesfortheinterfacecardsinstalled.

HowtoRestoreaLostVotingDiskusedbyOracleClusterware10g
PleasereadNote:279793.1andforOCRNote:268937.1

AslongasyoucanconfirmviatheCSSdaemonlogfilethatitthinksthevotingdiskisbad,youcanrestorethevotingdiskfrombackupwhiletheclusterisonline.Thisisthebackupthat
youtookwithdd(bythemanual'srequest)afterthemostrecentaddnode,deletenode,orinstalloperation.IfbyaccidentyourestoreavotingdiskthattheCSSdaemonthinksisNOT
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

53/72

8/24/2015

Document220970.1

youtookwithdd(bythemanual'srequest)afterthemostrecentaddnode,deletenode,orinstalloperation.IfbyaccidentyourestoreavotingdiskthattheCSSdaemonthinksisNOT
bad,thentheentireclusterwillprobablygodown.
crsctladdcssvotediskaddsanewvotingdisk
crsctldeletecssvotediskremovesavotingdisk
Note:theclusterhastobedown.Youcanalsorestorethebackupviaddwhentheclusterisdown.

HowcanIregisterthelistenerwithOracleClusterwareinRAC10gRelease2?
NetCAistheonlytoolthatconfigureslistenerandyoushouldbealwaysusingit.ItwillregisterthelistenerwithOracleClusterware.Therearenoothersupportedalternatives.

HowisthevotingdiskusedbyOracleClusterware?
ThevotingdiskisaccessedexclusivelybyCSS(oneoftheOracleClusterwaredaemons).Thisistotallydifferentfromadatabasefile.Thedatabaselooksatthedatabasefilesand
interactswiththeCSSdaemon(atasignificantlyhigherlevelconceptuallythananynotionof"votingdisk").
"Nonsynchronizedaccess"(i.e.databasecorruption)ispreventedbyensuringthattheremotenodeisdownbeforereassigningitslocks.Thevotingdisk,network,andthecontrolfile
areusedtodeterminewhenaremotenodeisdown,indifferent,parallel,indepdendentwaysthatalloweachtoprovideadditionalprotectioncomparedtotheother.Thealgorithms
usedforeachofthesethreethingsarequitedifferent.
Asfarasvotingdisksareconcerned,anodemustbeabletoaccessstrictlymorethanhalfofthevotingdisksatanytime.Soifyouwanttobeabletotolerateafailureofnvotingdisks,
youmusthaveatleast2n+1configured.(n=1means3votingdisks).Youcanconfigureupto32votingdisks,providingprotectionagainst15simultaneousdiskfailures,howeverit's
unlikelythatanycustomerwouldhaveenoughdisksystemswithstatisticallyindependentfailurecharacteristicsthatsuchaconfigurationismeaningful.Atanyrate,configuringmultiple
votingdisksincreasesthesystem'stoleranceofdiskfailures(i.e.increasesreliability).
ConfiguringasmallernumberofvotingdisksonsomekindofRAIDsystemcanallowacustomertousesomeothermeansofreliabilitythantheCSS'smultiplevotingdiskmechanisms.
HoweverthereseemtobequiteafewRAIDsystemsthatdecidethat3060second(or45minutesinthecaseofveritas)IOlatenciesareacceptable.Howeverwehavetowaitforat
leastthelongestIOlatencybeforewecandeclareanodedeadandallowthedatabasetoreassigndatabaseblocks.SowhileusinganindependentRAIDsystemforthevotingdiskmay
appearappealing,sometimestherearefailoverlatencyconsequenecs.

DoesOracleClusterwaresupportapplicationvips?
Yes,withOracleDatabase10gRelease2,OracleClusterwarenowsupportsan"application"vip.ThisistosupportputtingapplicationsunderthecontrolofOracleClusterwareusingthe
newhighavailabilityAPIandallowtheusertousethesameURLorconnectionstringregardlessofwhichnodeintheclustertheapplicationisrunningon.Theapplicationvipisanew
resourcedefinedtoOracleClusterwareandisafunctionalvip.Itisdefinedasadependentresourcetotheapplication.Therecanbemanyvipsdefined,typicallyoneperuserapplication
underthecontrolofOracleClusterware.Youmustfirstcreateaprofile(crs_profile),thenregisteritwithOracleClusterware(crs_register).Theusrvipscriptmustrunasroot.

HowdoIputmyapplicationunderthecontrolofOracleClusterwaretoachievehigheravailability?
Firstwriteacontrolagent.Itmustaccept3differentparameters:startThecontrolagentshouldstarttheapplication,checkThecontrolagentshouldchecktheapplication,stopThe
Controlagentshouldstoptheapplication.Secondlyyoumustcreateaprofileforyourapplicationusingcrs_profile.ThirdlyyoumustregisteryourapplicationasaresourcewithOracle
Clusterware(crs_register).SeetheRACAdminandDeploymentGuidefordetails.

Isitsupportedtoallow3rdPartyClusterwaretomanageOracleresources(instances,listeners,etc)andturnoffOracleClusterwaremanagementofthese?
In10gwedonotsupportusing3rdPartyClusterwareforfailoverandrestartofOracleresources.OracleClusterwareresourcesshouldnotbedisabled.

WhatistheHighAvailabilityAPI?
AnapplicationprogramminginterfacetoallowprocessestobeputundertheHighAvailabilityinfrastructurethatispartoftheOracleClusterwaredistributedwithOracleDatabase10g.
AuserwrittenscriptdefineshowOracleClusterwareshouldstart,stopandrelocatetheprocesswhentheclusternodestatuschanges.Thisextendsthehighavailabilityservicesofthe
clustertoanyapplicationrunninginthecluster.OracleDatabase10gRealApplicationClusters(RAC)databasesandassociatedOracleprocesses(E.G.listener)areautomatically
managedbytheclusterware.

IsitarequirementtohavethepublicinterfacelinkedtoETH0ordoesitonlyneedtobeonaETHlowerthantheprivateinterface?:publiconETH1privateon
ETH2
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

54/72

8/24/2015

Document220970.1

Thereisnorequirementforinterfacenameordering.YoucouldhavepubliconETH2privateonETH0JustmakesureyouchoosethecorrectpublicinterfaceinVIPCA,andinthe
installer'sinterconnectclassificationscreen.

HowdoIrestoreOCRfromabackup?OnWindows,canIuseocopy?
TheonlyrecommendedwaytorestoreanOCRfromabackupis"ocrconfigrestore".TheocopycommandwillnotbeabletoperformtherestoreactionforOCR.

Whatarethenetworkrequirementsforthethirdvotingfile(inathirdlocation)inanextendedOracleRACsetup?
InanextendedOracleRACsetup,OraclerecommendstohostthreeOracleClusterwaremanagedVotingFilesoneineachdatacenterhostingtheOracleRACserversandoneinathird
location,independentofthosedatacenters.
AsmostcustomersmaynothavetheabilitytohostathirdSANintheindependentthirdlocation,OracleallowsaccessingthethirdVotingFileusingstandardNFSasdescribedinthis
paperUsingstandardNFStosupportathirdvotingfileforextendedclusterconfigurations
Note:ThissetupissupportedasmentionedinthepaperandforextendedOracleRACclusterconfigurationsonly.Itisassumedthatthethirdlocationisnotusedfordatastorage,just
forhostingathirdVotingFileinaboveconfiguration.FromaclusterperspectiveallVotingFilesarealike.ThethirdlocationbasedVotingFilecannotbedistinguishedandwilltherefore
notbetreateddifferently.
Giventheseassumptions,thethirdlocationdoesnotneedtoobservethesamenetworkrequirementsastheinterdatacenterconnectivity,whichneedstomeetOracleRACstandards
asoutlinedintheOracleRACdocumentationorinthispaperOracleRACandOracleClusterwareInterconnectVLANDeploymentConsiderations
OraclehasnottestedtheimpactofdistanceonthethirdlocationbasedVotingFile,asitisgenerallyassumedthatthethirdlocationisideallyinsimilardistancetothedatacentersas
thetwodatacentersarelocatedtoeachother.However,customerscanchooseanydistanceforthethirdlocation,aslongastheaccessibilitytothethirdVotingFileisassuredand
consideringthatincreasingdistancetothethirdlocationfromthedatacentersincreasestheriskforfailuresonthepaththatnotonlycausedelays,butalsootherfailuresthatmight
eitherberecognizedasIOerrorsorsimplyasa(NFS)freeze.
MinimumnetworkrequirementsfortheconnectivitytothethirdlocationcanthereforebederivedfromtheaccesspatternstoVotingFilesinanOraclecluster.Ingeneral,
OracleClusterwareaccessesaVotingFileeverysecondforreadandwritewithlessthanakilobyteofdatareadorwritten.AnacknowledgementofthewriteI/Oneedstobereceivedin
200secondsundernormaloperations(longdisktimeout)and27secondsduringareconfigurationinthecluster(shortdisktimeout).
Inordertomeetthoseresponsetimerequirements(latencies)foraccessingthethirdVotingFilehostedonstandardNFSorusingiSCSIbasedconnectivity,Oraclerecommendssetting
upthesystemtomeethalfofthelowerlatencyrequirementinaverage.Inotherwords,theconnectivitytothethirdlocationshouldensurethattheVotingFilewriteI/O
canbeacknowledgedin27/2seconds(approx.14seconds)inaverage,providingaminimumaveragethroughputofatleast128Kbps..

CanIuseASMtomirrorOracledatainanextendedRACenvironment?
OracleASMbasedmirroringcanbeusedtomirrordatabetweenthesitesinanextendedOracleRACenvironmentstartingwithOracleASM10gRelease2.Thefollowingrestrictions
apply:
1.AsinanyextendedRACenvironments,theadditionallatencyinducedbydistancewillaffectI/Oandcachefusionperformance.Thiseffectwillvarybydistanceandthecustomeris
responsibleforensuringthattheimpactattainedintheirenvironmentisacceptablefortheirapplication.
2.OCRmustbemirroredacrossbothsitesusingOracleprovidedmechanisms.
3.VotingDiskredundancymustexistacrossbothsitesandata3rdlocationtoactasanarbitrage.ThisthirdsitemaybeconnectedviaaWANusingstandardNFSonaLinuxbased
servertomountthe3rdVotingDiskonthecluster.
4.StoragemustbemanagedbyOracleASMandsetupasnormalredundancydiskgroupswithseparatefailuregroupsdefinedforeachsite.ThisenablesASMtomirrordatabasefiles
acrossbothsites.
5.Itisstronglyrecommendedthatcustomershaveaseparateanddedicatedtestclusteralsoinanextendedconfigurationandusingthesamesoftwareandhardwarecomponents(can
befewerorsmallernodes).AsOraclesupportsmultipleextendedRACconfigurations,anypotentialsolutionverificationincaseofissuesneedstobeperformedatcustomerside.
6.CustomermustbeawarethatpriortoOracleASM11gRel.1,ASMdoesnotprovidepartialresilvering.Shouldalossofconnectivitybetweenthesitesoccur,oneofthefailuregroups
willbemarkedinvalid.Whenthesiterejoinsthecluster,thefailuregroupswillneedtobemanuallydroppedandadded.OracleASM11gRel.1providesforFastMirrorResyncwhere
ASMkeepstrackofthechangedextentsthatneedtobeappliedtoofflinedisks.Oncethedisksareavailable,onlythechangedextentsarewrittentoresynchronizethedisks,ratherthan
overwritingthecontentsoftheentiredisks.

WhatarethenetworkrequirementsforanextendedRACcluster?
NecessaryConnections
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

55/72

8/24/2015

Document220970.1
NecessaryConnections
Interconnect,SAN,andIPNetworkingneedtobekeptonseparatechannels,eachwithrequiredredundancy.RedundantconnectionsmustnotsharethesameDarkFiber(ifused),
switch,path,orevenbuildingentrances.Keepinmindthatcablescanbecut.
TheSANandInterconnectconnectionsneedtobeondedicatedpointtopointconnections.NoWANorSharedconnectionallowed.Traditionalcablesarelimitedtoabout10kmifyou
aretoavoidusingrepeaters.DarkFibernetworksallowthecommunicationtooccurwithoutrepeaters.Sincelatencyislimited,DarkFibernetworksallowforagreaterdistancein
separationbetweenthenodes.ThedisadvantageofDarkFibernetworksaretheycancosthundredsofthousandsofdollars,sogenerallytheyareonlyanoptioniftheyalreadyexist
betweenthetwosites.
Ifdirectconnectionsareused(forshortdistances)thisisgenerallydonebyjuststringinglongcablesfromaswitch.IfaDWDMorCWDMisusedthenthenthesearedirectlyconnected
viaadedicatedswitchoneitherside.
Noteofcaution:DonotdoRACInterconnectoveraWAN.Thisisathesameasdoingitoverthepublicnetworkwhichisnotsupportedandotherusesofthenetwork(i.e.largeFTPs)
cancauseperformancedegradationsorevennodeevictions.
ForSANnetworksmakesureyouareusingSANbuffercreditsifthedistanceisover10km.
IfOracleClusterwareisbeingused,wealsorequirethatasinglesubnetbesetupforthepublicconnectionssowecanfailoverVIPsfromonesidetoanother.

CanacustomeruseSERACtoimplementan"ExtendedRACCluster"?
YES.Effectivewith11gRel.1theformerrestrictiontohaveallnodescolocatedinoneroomwhenusingSERAChasbeenlifted.CustomerscannowuseSERACclustersinextended
environments.However,otherSERACrestrictionsstillapply(e.g.compulsoryusageofASM,nothirdpartyclusternorvolumemanagermustbeinstalled).Please,refertothelicensing
documentationformoreinformation.

WhatisthemaximumdistancebetweennodesinanextendedRACenvironment?
Thehighimpactoflatencycreatepracticallimitationsastowherethisarchitecturecanbedeployed.Whilethereisnotfixeddistancelimitation,theadditionallatencyonroundtripon
I/Oandaonewaycachefusionwillhaveanaffectonperformanceasdistanceincreases.Forexampletestsat100kmshoweda34msimpactonI/Oand1msimpactoncachefusion,
thusthefartherdistanceisthegreatertheimpactonperformance.Thisarchitecturefitsbestwherethe2datacentersarerelativelyclose(

CanIuseASMasmechanismtomirrorthedatainanExtendedRACcluster?
Yes,butitcannotreplicateeverythingthatneedsreplication.
ASMworkswelltoreplicateanyobjectyoucanputinASM.ButyoucannotputtheOCRorVotingDiskinASM.
In10gR1theycaneitherbemirroredusingadifferentmechanism(whichcouldthenbeusedinsteadofASM)ortheOCRneedstoberestoredfrombackupandtheVotingDiskcanbe
recreated.
InthefuturewearelookingatprovidingOracleredundancyforboth.

Howshouldvotingdisksbeimplementedinanextendedclusterenvironment?CanIusestandardNFSforthethirdsitevotingdisk?
http://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfStandardNFSisonlysupportedforthetiebreakingvotingdiskinanextendedcluster
environment.Seeplatformandmountoptionrestrictionsat:http://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfOtherwisejustaswithdatabase
files,weonlysupportvotingfilesoncertifiedNASdevices,withtheappropriatemountoptions.PlsrefertoMetalinkNote359515.1forafulldescriptionoftherequiredmountoptions.
ForacompletelistofsupportedNASvendorsrefertoOTNat:http://www.oracle.com/technology/deploy/availability/htdocs/vendors_nfs.html

WherecanIfindtheCVUtracefiles?
CVUlogfilescanbefoundunder$CV_HOME/cv/logdirectory.Thelogfilesareautomaticallyrotatedandthelatestlogfilehasthenamecvutrace.log.0.Itisagoodideatocleanup
unwantedlogfilesorarchivethemtoreclaimdiskplace.
Inrecentreleases,CVUtracefilesaregeneratedbydefault.SettingSRVM_TRACE=falsebeforeinvokingcluvfydisablesthetracegenerationforthatinvocation.

WhyisvalidateUserEquivfailingduringinstall(orcluvfyrun)?
SSHmustbesetupasperthepreinstallationtasks.ItisalsonecessarytohavefilepermissionssetasdescribedbelowforfeaturessuchasPublicKeyAuthorizationtowork.Ifyour
permissionsarenotcorrect,publickeyauthenticationwillfail,andwillfallbacktopasswordauthenticationwithnohelpfulmessageastowhy.Thefollowingserverconfigurationfiles
and/ordirectoriesmustbeownedbytheaccountownerorbyrootandGROUPandWORLDWRITEpermissionmustbedisabled.
$HOME
$HOME/.rhosts
$HOME/.shosts

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

56/72

8/24/2015

Document220970.1

$HOME/.shosts
$HOME/.ssh
$HOME/.ssh.authorizedkeys
$HOME/.ssh/authorizedkeys2#Opensshspecificforssh2protocol.
SSH(fromOUI)willalsofailifyouhavenotconnectedtoeachmachineinyourclusterasperthenoteintheinstallationguide:
ThefirsttimeyouuseSSHtoconnecttoanodefromaparticularsystem,youmayseeamessagesimilartothefollowing:
Theauthenticityofhost'node1(140.87.152.153)'can'tbeestablished.RSAkeyfingerprintis7z:ez:e7:f6:f4:f2:4f:8f:9z:79:85:62:20:90:92:z9.
Areyousureyouwanttocontinueconnecting(yes/no)?
Enter|yes|attheprompttocontinue.Youshouldnotseethismessageagainwhenyouconnectfromthissystemtothatnode.Answeringyestothisquestioncausesanentrytobe
addedtoa"knownhosts"fileinthe.sshdirectorywhichiswhysubsequentconnectionrequestsdonotreask.
ThisisknowntoworkonSolarisandLinuxbutmayworkonotherplatformsaswell.

HowdoIturnontracing?
SettheenvironmentalvariableSRVM_TRACEtotrue.Forexample,intcsh"setenvSRVM_TRACEtrue"willturnontracing.Alsoitmayhelptoruncluvfywithverboseattribute
$scriptrun.log
$exportSRVM_TRACE=TRUE
$cluvfyblahverbose
$exit

CanIcheckifthestorageissharedamongthenodes?
Yes,youcanuse'compssa'commandtocheckthesharednessofthestorage.Pleaserefertotheknownissuessectionforthetypeofstoragesupportedbycluvfy.

WhenIrun10.2CLUVFYonasystemwhereRAC10gRelease1isrunningIgetfollowingoutput:
Packageexistencecheckfailedfor"SUNWscucm:3.1".
Packageexistencecheckfailedfor"SUNWudlmr:3.1".
Packageexistencecheckfailedfor"SUNWudlm:3.1".
Packageexistencecheckfailedfor"ORCLudlm:Dev_Release_06/11/04,_64bit_3.3.4.8_reentrant".
Packageexistencecheckfailedfor"SUNWscr:3.1".
Packageexistencecheckfailedfor"SUNWscu:3.1".
CheckingthisSolarissystemIdon'tseethosepackagesinstalled.CanIcontinuemyinstall?
Notethatcluvfychecksallpossibleprerequisitesandtellsyouwhetheryoursystempassesthecheckornot.Youcanthencrossreferencewiththeinstallguidetoseeifthechecksthat
failedarerequiredforyourtypeofinstallation.Ittheabovecase,ifyouarenotplanningonusingSunCluster,thenyoucancontinuetheinstall.Thechecksthatfailedarethechecksfor
SunClusterrequiredpackagesandarenotneededonyourcluster.Aslongaseverythingelsechecksoutsuccessfully,youcancontinue.

Whatarethedefaultvaluesforthecommandlinearguments?
Herearethedefaultvaluesandbehaviorfordifferentstageandcomponentcommands:
Forcomponentnodecon:
Ifnoioraargumentsisprovided,thencluvfywillgetintothediscoverymode.
Forcomponentnodereach:
Ifnosrcnodeisprovided,thenthelocal(nodeofinvocation)willbeusedasthesourcenode.
Forcomponentscfs,ocr,crs,space,clumgr:
Ifnonargumentisprovided,thenthelocalnodewillbeused.
Forcomponentssysandadmprv:
Ifnonargumentisprovided,thenthelocalnodewillbeused.
Ifnoosdbaargumentisprovided,then'dba'willbeused.Ifnoorainvargumentisprovided,then'oinstall'willbeused.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

57/72

8/24/2015

Document220970.1

Forcomponentpeer:
Ifnoosdbaargumentisprovided,then'dba'willbeused.
Ifnoorainvargumentisprovided,then'oinstall'willbeused.
Forstageposthwos:
Ifnosargumentisprovided,thencluvfywillgetintothediscoverymode.
Forstagepreclusvc:
Ifnocargumentisprovided,thencluvfywillskipOCRrelatedchecks.
Ifnoqargumentisprovided,thencluvfywillskipvotingdiskrelatedchecks.
Ifnoosdbaargumentisprovided,then'dba'willbeused.
Ifnoorainvargumentisprovided,then'oinstall'willbeused.
Forstagepredbinst:
Ifcfs_ohflagisnotspecified,thencluvfywillassumeOraclehomeisnotonasharedfilesystem.
Ifnoosdbaargumentisprovided,then'dba'willbeused.
Ifnoorainvargumentisprovided,then'oinstall'willbeused.

HowdoIchecktheOracleClusterwarestackandothersubcomponentsofit?
CluvfyprovidescommandstocheckaparticularsubcomponentoftheCRSstackaswellasthewholeCRSstack.Youcanusethe'compocr'commandtochecktheintegrityofOCR.
Similarly,youcanuse'compcrs'and'compclumgr'commandstocheckintegrityofcrsandclustermanagersubcomponents.TochecktheentireCRSstack,runthestagecommand
'clucvystagepostcrsinst'.

IsthereawaytoverifythattheOracleClusterwareisworkingproperlybeforeproceedingwithRACinstall?
Yes.Youcanusethepostcheckcommandforclusterservicessetup(postclusvc)toverifyCRSstatus.Amoreappropriatetestwouldbetousetheprecheckcommandfordatabase
installation(predbinst).ThiswillcheckwhetherthecurrentstateofthesystemissuitableforRACinstall.

Atwhatpointcluvfyisusable?CanIusecluvfybeforeinstallingOracleClusterware?
Youcanruncluvfyatanytime,evenbeforeCRSinstallation.Infact,cluvfyisdesignedtoassisttheuserassoonasthehardwareandOSisup.Ifyouinvokeacommandwhichrequires
CRSorRAConlocalnode,cluvfywillreportanerrorifthoserequiredproductsarenotyetinstalled.

WhatisCVU?Whatareitsobjectivesandfeatures?
CVUbringseasetoRACusersbyverifyingalltheimportantcomponentsthatneedtobeverifiedatdifferentstagesinaRACenvironment.ThewidedomainofdeploymentofCVU
rangesfrominitialhardwaresetupthroughfullyoperationalclusterforRACdeploymentandcoversalltheintermediatestagesofinstallationandconfigurationofvariouscomponents.
Thecommandlinetooliscluvfy.Cluvfyisanonintrusiveutilityandwillnotadverselyaffectthesystemoroperationsstack.

Whatisastage?
CVUsupportsthenotionofStageverification.ItidentifiesalltheimportantstagesinRACdeploymentandprovideseachstagewithitsownentryandexitcriteria.Theentrycriteriafora
stagedefineaspecificsetofverificationtaskstobeperformedbeforeinitiatingthatstage.Thisprechecksavestheuserfromenteringintoastageunlessitsprerequisiteconditionsare
met.Theexitcriteriaforastagedefineanotherspecificsetofverificationtaskstobeperformedaftercompletionofthestage.Thepostcheckensuresthattheactivitiesforthatstage
havebeencompletedsuccessfully.Itidentifiesanystagespecificproblembeforeitpropagatestosubsequentstagesthusmakingitdifficulttofinditsrootcause.Anexampleofastage
is"precheckofdatabaseinstallation",whichcheckswhetherthesystemmeetsthecriteriaforRACinstall.

Whatisacomponent?
CVUsupportsthenotionofComponentverification.Theverificationsinthiscategoryarenotassociatedwithanyspecificstage.Theusercanverifythecorrectnessofaspecificcluster
component.Acomponentcanrangefromabasicone,likefreediskspacetoacomplexonelikeCRSStack.TheintegritycheckforCRSstackwilltransparentlyspanoververificationof
multiplesubcomponentsassociatedwithCRSstack.Thisencapsulationofasetoftaskswithinspecificcomponentverificationshouldbeofagreateasetotheuser.

Whatisnodelist?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

58/72

8/24/2015

Whatisnodelist?

Document220970.1

Nodelistisacommaseparatedlistofhostnameswithoutdomain.Cluvfywillignoreanydomainwhileprocessingthenodelist.Ifduplicateentitiesafterremovingthedomainexist,cluvfy
willeliminatetheduplicatenameswhileprocessing.Whereversupported,youcanuse'nall'tocheckonalltheclusternodes.Checkthisformoreinformationonnodelistandshortcuts.

DoIhavetoberoottouseCVU?
No.CVUisintendedfordatabaseandsystemadministrators.CVUassumesthecurrentuserasoracleuser.

Whataboutdiscovery?DoesCVUdiscoverinstalledcomponents?
Atpresent,CVUdiscoveryislimitedtothesecomponents.CVUdiscoversavailablenetworkinterfacesifyoudonotspecifyanyinterfaceorIPaddressinitscommandline.Forstorage
relatedverification,CVUdiscoversallthesupportedstoragetypesifyoudonotspecifyaparticularstorage.CVUdiscoversCRSHOMEifoneisavailable.

WhataretherequirementsforCVU?
CVUrequires:1._Anareawithatleast30MBforcontainingsoftwarebitsontheinvocationnode.2._Java1.4.1locationontheinvocationnode.3._Aworkdirectorywithatleast25MB
onallthenodes.CVUwillattempttocopythenecessarybitsasrequiredtothislocation.Makesure,thelocationexistsonallnodesandithaswritepermissionforCVUuser.Thisdiris
setthroughtheCV_DESTLOCenvironmentvariable.Ifthisvariabledoesnotexist,CVUwilluse"/tmp"astheworkdir.4._OnRedHatLinux3.0,anoptionalpackage'cvuqdisk'is
requiredonallthenodes.ThisassistsCVUinfindingscsidisksandhelpsCVUtoperformstoragechecksondisks.PleaserefertoWhatis'cvuqdisk'rpm?fordetail.Notethat,this
packageshouldbeinstalledonlyonRedHatLinux3.0distribution.

Whatis'cvuqdisk'rpm?WhyshouldIinstallthisrpm?
CVUrequiresrootprivilegetogatherinformationaboutthescsidisksduringdiscovery.Asmallbinaryusesthesetuidmechanismtoquerydiskinformationasroot.Notethatthisprocess
ispurelyareadonlyprocesswithnoadverseimpactonthesystem.Tomakethissecured,thisbinaryispackagedinthecvuqdiskrpmandneedrootprivilegetoinstallonamachine.If
thispackageisinstalledonallthenodes,CVUwillbeabletoperformdiscoveryandsharedstorageaccessibilitychecksforscsidisks.Otherwise,itcomplainsaboutthemissingpackage
'cvuqdisk'.Notethat,thispackageshouldbeinstalledonlyonRedHatLinux3.0distribution.DiscoveryofscsidisksforRedHatLinux2.1isnotsupported.

HowdoIinstall'cvuqdisk'package?
Herearethestepstoinstallcvuqdiskpackage.1._Becomerootuser2._Copytherpm(cvuqdisk1.0.11.i386.rpm,currentversionis1.0.1)toalocaldirectory.Youcanfindtherpmin
Oracle'sOTNsite.3._Settheenvironmentvariabletoagroup,whoshouldownthisbinary.Typicallyitisthe"dba"group.exportCVUQDISK_GRP=dba4._Eraseanyexistingpackage
rpmecvuqdisk5._Installtherpmrpmivcvuqdisk1.0.11.i386.rpm

HowdoIknowaboutcluvfycommands?Theusagetextofcluvfydoesnotshowindividualcommands.
Cluvfyhascontextsensitivehelpbuiltintoit.Cluvfyshowsthemostappropriateusagetextbasedonthecluvfycommandlinearguments.Ifyoutype'cluvfy'onthecommandprompt,
cluvfydisplaysthehighlevelgenericusagetext,whichtalksaboutvalidstageandcomponentsyntax.Ifyoutype'cluvfycomplist',cluvfywillshowvalidcomponentswithbrief
descriptiononeachofthem.Ifyoutype'cluvfycomphelp',cluvfywillshowdetailsyntaxforeachofthevalidcomponents.Similarly,'cluvfystagelist'and'cluvfystagehelp'willlist
validstagesandtheirsyntaxrespectively.Ifyoutypeaninvalidcommand,cluvfywillshowtheappropriateusageforthatparticularcommand.Forexample,ifyoutype'cluvfystagepre
dbinst',cluvfywillshowthesyntaxforprecheckofdbinststage.

DoIhavetotypethenodelisteverytimefortheCVUcommands?Isthereanyshortcut?
YoudonothavetotypethenodelisteverytimefortheCVUcommands.Typingthenodelistforalargeclusterispainfulanderrorprone.Herearefewshortcuts.Toprovideallthe
nodesofthecluster,type'nall'.Cluvfywillattempttogetthenodelistinthefollowingorder:1.Ifavendorclusterwareisavailable,itwillpickalltheconfigurednodesfromthevendor
clusterwareusinglsnodesutility.2.IfCRSisinstalled,itwillpickalltheconfigurednodesfromOracleclusterwareusingolsnodesutility.3.Innoneoftheabove,itwilllookforthe
CV_NODE_ALLenvironmentalvariable.Ifthisvariableisnotdefined,itwillcomplain.Toprovideapartiallist(someofthenodesofthecluster)ofnodes,youcansetanenvironmental
variableanduseitintheCVUcommand.Forexample:setenvMYNODESnode1,node3,node5cluvfycompnodeconn$MYNODES

HowdoIgetdetailoutputofacheck?
Cluvfysupportsaverbosefeature.Bydefault,cluvfyreportsinnonverbosemodeandjustreportsthesummaryofatest.Togetdetailedoutputofacheck,usetheflag'verbose'inthe
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

59/72

8/24/2015

Document220970.1
Cluvfysupportsaverbosefeature.Bydefault,cluvfyreportsinnonverbosemodeandjustreportsthesummaryofatest.Togetdetailedoutputofacheck,usetheflag'verbose'inthe
commandline.Thiswillproducedetailoutputofindividualchecksandwhereapplicablewillshowpernoderesultinatabularfashion.

HowdoIchecknetworkornodeconnectivityrelatedissues?
Usecomponentverificationscommandslike'nodereach'or'nodecon'forthispurpose.Fordetailsyntaxofthesecommands,typecluvfycomphelponthecommandprompt.Ifthe
'cluvfycompnodecon'commandisinvokedwithouti,cluvfywillattempttodiscoveralltheavailableinterfacesandthecorrespondingIPaddress&subnet.Thencluvfywilltrytoverify
thenodeconnectivitypersubnet.Youcanrunthiscommandinverbosemodetofindoutthemappingsbetweentheinterfaces,IPaddressesandsubnets.Youcanchecktheconnectivity
amongthenodesbyspecifyingtheinterfacename(s)throughiargument.

HowdoIcheckwhetherOCFSisproperlyconfigured?
Youcanusethecomponentcommand'cfs'tocheckthis.ProvidetheOCFSfilesystemyouwanttocheckthroughthefargument.Notethat,thesharednesscheckforthefilesytemis
supportedforOCFSversion1.0.14orhigher.

HowdoIcheckuseraccountsandadministrativepermissionsrelatedissues?
Useadmprvcomponentverificationcommand.Refertotheusagetextfordetailinstructionandtypeofsupportedoperations.Tocheckwhethertheprivilegeissufficientforuser
equivalence,use'ouser_equiv'argument.Similarly,the'ocrs_inst'willverifywhethertheuserhasthecorrectpermissionsforinstallingCRS.The'odb_inst'willcheckfor
permissionsrequiredforinstallingRACand'odb_config'willcheckforpermissionsrequiredforcreatingaRACdatabaseormodifyingaRACdatabaseconfiguration.

HowdoIcheckminimalsystemrequirementsonthenodes?
Thecomponentverificationcommandsysismeantforthat.TocheckthesystemrequirementforRAC,use'pdatabase'argument.TocheckthesystemrequirementforCRS,use'p
crs'argument.

Isthereawaytocomparenodes?
Youcanusethepeercomparisonfeatureofcluvfyforthispurpose.Thecommand'comppeer'willlistthevaluesofdifferentnodesforseveralpreselectedproperties.Youcanusethe
peercommandwithrefnodeargumenttocomparethosepropertiesofothernodesagainstthereferencenode.

Whythepeercomparisonwithrefnodesayspassedwhenthegrouporuserdoesnotexist?
Peercomparisonwiththerefnodefeatureactslikeabaselinefeature.Itcomparesthesystempropertiesofothernodesagainstthereferencenode.Ifthevaluedoesnotmatch(not
equaltoreferencenodevalue),thenitflagsthatasadeviationfromthereferencenode.Ifagrouporuserdoesnotexistonreferencenodeaswellasontheothernode,itwillreport
thisas'matched'sincethereisnodeviationfromthereferencenode.Similarly,itwillreportas'mismatched'foranodewithhighertotalmemorythanthereferencenodefortheabove
reason.

Whycluvfyreports"unknown"onaparticularnode?
Cluvfyreportsunknownwhenitcannotconcludeforsureifthecheckpassedorfailed.AcommoncauseofthistypeofreportingisanonexistentlocationsetfortheCV_DESTLOC
variable.Pleasemakesurethedirectorypointedbythisvariableexistsonallnodesandiswritablebytheuser.

Whataretheknownissueswiththisrelease?
1._Sharedstorageaccessibility(ssa)checkreportsCurrentreleaseofcluvfyhasthefollowinglimitationsonLinuxregardingsharedstorageaccessibilitycheck.a.CurrentlyNASstorage(
r/w,noattributecaching),OCFS(version1.0.14orhigher)andscsidisks(ifcvuqdiskpackageisinstalled)aresupported.Notethat,'cvuqdisk'packageshouldbeinstalledonlyon
RedHatLinux3.0distribution.DiscoveryofscsidisksforRedHatLinux2.1isnotsupported.b.ForsharednesscheckonNAS,cluvfyrequirestheusertohavewritepermissiononthe
specifiedpath.Ifthecluvfyuserdoesnothavewritepermission,cluvfyreportsthepathasnotshared.2._WhatdatabaseversionissupportedbyCVU?CurrentCVUreleasesupports
only10gRACandCRSandisnotbackwardcompatible.Inotherwords,CVUcannotcheckorverifypre10gproducts.3._WhatLinuxdistributionsaresupported?Thisreleasesupports
onlyRedHat3.0Update2andRedHat2.1ASdistributions.Notethat,theCVUdistributionforRedHat3.0Update2andRedHat2.1ASaredifferenttheyarenotbinarycompatible.In
otherwords,CVUbitsforRedHat3.0andRedHat2.1arenotthesame.4._Thecomponentcheckfornodeapplication(cluvfycompnodeapp...)commandreportsnodeappcreation
errorifthelocalCRSstackisdown.Thisisaknownissueandwillbeaddressedshortly.5._CVUdoesnotrecongnizethediskbindings(e.g./dev/raw/raw1)asvalidstoragepathsor
identifiers.Pleaseusetheunderlyingdisk(e.g./dev/sdmetc)forthestoragepathoridentifiers.6._CurrentversionofCVUforRedHat2.1complainsaboutthemissingcvuqdisk
package.Thiswillbecorrectedinthefuturerelease.Usershouldignorethiserror.Notethat,'cvuqdisk'packageshouldbeinstalledonlyonRedHatLinux3.0distribution.Discoveryof

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

60/72

8/24/2015

Document220970.1

package.Thiswillbecorrectedinthefuturerelease.Usershouldignorethiserror.Notethat,'cvuqdisk'packageshouldbeinstalledonlyonRedHatLinux3.0distribution.Discoveryof
scsidisksforRedHatLinux2.1isnotsupported.

HowdoesRACOneNodecomparewithdatabaseDRproductslikeDataGuardorGoldenGate?
Theproductsareentirelycomplementary.RACOneNodeisdesignedtoprotectasingledatabase.Itcanbeusedforrollingdatabasepatches,OSupgrades/patches,andgrid
infrastructure(ASM/Clusterware)rollingupgradesandpatches.Thisislessdisruptivethanswitchingtoadatbasereplica.Switchingtoareplicaforpatching,orforupgradingtheOSor
gridinfrastructurerequiresthatyouchoosetorunActive/Active(anddealwithpotentialconflicts)orActive/Passive(andwaitforworkontheactiveprimarydatabasetodrainbefore
allowingworkonthereplica).Youneedtomakesurereplicationsupportsalldatatypesyouareusing.Youneedtomakesurethereplicacankeepupwithyourload.Youneedtofigure
outhowtorepointyourclientstothereplica(notanissuewithRACOneNodebecauseit'sthesamedatabase,andweuseVIPs).Andlastly,RACOneNodeallowsasparenodetobe
used10daysperyearwithoutlicensing.OurrecommendationistouseRACorRACOneNodetoprotectfromlocalfailuresandtosupportrollingmaintenanceactivities.UseDataGuard
orreplicationtechnologyforDR,dataprotection,andforrollingdatabaseupgrades.BotharerequiredaspartofacomprehensiveHAsolution.

IfacurrentcustomerhasanEnterpriseLicenseAgreement(ELA),aretheyentitledtouseOracleRACOneNode?
Yes,assumingtheexistingELA/ULAincludesOracleRAC.ThelicenseguidestatesthatallOracleRACoptionlicenses(notSERAC)includeallthefeaturesofOracleRACOneNode.
CustomerswithexistingRAClicensesorOracleRACELA'scanusethoselicensesasOracleRACOneNode.Thisamountsto"burning"aOracleRAClicenseforOracleRACOneNode,
whichisexpensivelongterm.ObviouslyiftheELA/ULAdoesnotincludeOracleRAC,thentheyarenotentitledtouseOracleRACOneNode.

HowisOracleRACOneNodelicensedandpriced?
OracleRACOneNodeisanoptiontotheOracleDatabaseEnterpriseEditionandlicensedbaseduponthenumberofCPU'sintheserveronwhichitisinstalled.Currentlistpriceis
$10,000perCPU(Checkpricelist).
UnliketheOracleRACfeature,OracleRACOneNodeisnotavailablewiththeOracleStandardEdition.
OracleRACOneNodelicensingalsoincludesthe10dayrule,allowingadatabasetorelocatetoanothernodeforupto10daysperyear,withoutincurringadditionallicensingfees.This
ismostoftenusedinthecaseoffailover,orforplannedmaintenanceandupgrading.Onlyonenodeintheclustercanbeusedforthe10dayrule.

IsOracleRACOneNodesupportedwith3rdpartyclusterwareand/or3rdpartyCFS?
No.OracleRACOneNodeisonlysupportedwithwithversion11.2(andabove)ofOraclegridinfrastructure.

HowdoesRACOneNodecomparewithtraditionalcoldfailoversolutionslikeHPServiceguard,IBMHACMP,SunClusterandSymantec,andVeritasCluster
Server?
RACOneNodeisabetterhighavailabilitysolutionthantraditionalcoldfailoversolutions.
RACOneNodeoperatesinaclusterbutonlyasingleinstanceofthedatabaseisrunningononenodeinthecluster.Ifthatdatabaseinstancehasaproblem,RACOneNodedetectsthat
andcanattempttorestarttheinstanceonthatnode.Ifthewholenodefails,RACOneNodewilldetectthatandwillbringupthatdatabaseinstanceonanothernodeinthecluster.
Unliketraditionalcoldfailoversolutions,OracleClusterwarewillsendoutnotifications(FANevents)toclientstospeedreconnectionafterfailover.3rdpartysolutionsmaysimplywaitfor
potentiallylengthytimeoutstoexpire.
RACOneNodegoesbeyondthetraditionalcoldfailoverfunctionalitybyofferingadministratorstheabilitytoproactivelymigrateinstancesfromonenodeintheclustertoanother.For
example,letssayyouwantedtodoanupgradeoftheoperatingsystemonthenodethattheRACOneNodedatabaseisrunningon.Theadministratorwouldactivate"OMotion,"anew
Oraclefacilitythatwouldmigratetheinstancetoanothernodeinthecluster.Oncetheinstanceandalloftheconnectionshavemigrated,theservercanbeshutdown,upgradedand
restarted.OMotioncanthenbeinvokedagaintomigratetheinstanceandtheconnectionsbacktothenowupgradednode.Thisnondisruptiverollingupgradeandpatchingcapability
ofRACOneNodeexceedsthecurrentfunctionalityofthetraditionalcoldfailoversolutions.
Also,RACOneNodeprovidesaloadbalancingcapabilitythatisattractivetoDBAsandSysAdmins.Forexample,ifyouhavetwodifferentdatabaseinstancesrunningonaRACOne
NodeServeranditbecomesapparentthattheloadagainstthesetwoinstancesisimpactingperformance,theDBAcaninvokeOMotionandmigrateoneoftheinstancestoanotherless
usednodeinthecluster.RACOneNodeoffersthisloadbalancingcapability,somethingthatthetraditionalcoldfailoversolutionsdonot.
Lastly,many3rdpartysolutionsdonotsupportASMstorage.Thiscanslowdownfailover,andpreventconsolidationofstorageacrossmultipledatabases,increasingthemanagement
burdenontheDBA.
ThefollowingtablesummarizesthedifferencesbetweenRACOneNodeand3rdpartyfailoversolutions:
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

61/72

8/24/2015

Document220970.1

Feature

RACOneNode

EEplus3rdPartyClusterware

Outofthe
box
experience

RACOneNodeprovideseverything
necessarytoimplementdatabase
failover.

3rdpartyfailoversolutionsrequireaseparate
installandaseparatemanagement
infrastructure.

Single
Vendor

RACOneNodeis100%supportedby
Oracle

EEissupportedbyOracle,butthecustomer
mustrelyonthe3rdpartytosupporttheir
clusterware.

Fastfailover RACOneNodesupportsFANEvents,to 3rdpartyfailoversolutionsrelyontimeouts

sendnotificationstoclientsafter
forclientstodetectfailoverandinitiatea
failoversandtospeedreconnection
reconnection.Itcouldtakeseveralminutes
foraclienttodetecttherehadbeenafailover.

RollingDB
patching,
OS,
Clusterware,
ASM
patchingand
upgrades

RACOneNodecanmigrateadatabase
fromoneservertoanothertoenable
onlinerollingpatching.Most
connectionsshouldmigratewithno
disruption.

3rdpartysolutionsmustbefailedoverfrom
onenodetoanother,whichmeansall
connectionswillbedroppedandmust
reconnect.Sometransactionswillbedropped
andmustreconnect.Reconnectioncouldtake
severalminutes.

Workload
RACOneNodecanmigrateadatabase
Management fromoneservertoanotherwhileonline
toenableloadbalancingofdatabases
acrossserversinthecluster.Most
connectionsshouldmigratewithno
disruption.

3rdpartysolutionsmustbefailedoverfrom
onenodetoanother,whichmeansall
connectionswillbedroppedandmust
reconnect.Sometransactionswillbedropped
andmustreconnect.Reconnectioncouldtake
severalminutes.

Onlinescale
out

CompletereinstallincludingOracleGrid
Infrastructureisrequired.

OnlineupgradetomultinodeRAC

Standardized RACandRACOneNodeusethesame
toolsand
tools,managementinterfaces,and
processes
processes.

EEandRACusedifferenttools,management
interfaces,andprocesses.3rdparty
clusterwarerequiresadditionalinterfaces.

Storage
RACOneNodesupportsuseofASMto
virtualization virtualizeandconsolidatestorage.
Becauseitssharedacrossnodes,it
eliminatesthelengthyfailoverof
volumesandfilesystems

Traditional3rdpartysolutionsrelyonlocalfile
systemsandvolumesthatmustbefailed
over.Largevolumescantakealongtimeto
failover.Dedicatedstorageisalsomore
difficulttomanage.

HowdoesRACOneNodecomparewithasingleinstanceOracleDatabaseprotectedwithOracleClusterware?
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

62/72

8/24/2015

Document220970.1

Feature

RACOneNode

EEplusOracleClusterware

Outofthebox
experience

RACOneNodeisacomplete
solutionthatprovides
everythingnecessaryto
implementadatabase
protectedfromfailuresbya
failoversolution.

UsingOracleClusterwaretoprotectanEE
databaseispossiblebycustomizingsome
samplescriptsweprovidetoworkwithEE.
Thisrequirescustomscriptdevelopmentby
thecustomer,andtheyneedtosetupthe
environmentandinstallthescripts
manually.

Supportability

RACOneNodeis100%
supported

WhileEEis100%supported,thescripts
customizedbythecustomerarenot
supportedbyOracle.

DBControlsupport

RACOneNodefullysupports
failoverofDBControlina
transparentmanner

DBControlmustbereconfiguredaftera
failover(unlessthecustomerscriptsare
modifiedtosupportDBControlfailover)

RollingDBpatching,
OS,Clusterware,
ASMpatchingand
upgrades

RACOneNodecanonline
migrateadatabasefromone
servertoanothertoenable
onlinerollingpatching.Most
connectionsshouldmigrate
withnodisruption

EEmustbefailedoverfromonenodeto
another,whichmeansallconnectionswill
bedroppedandmustreconnect.Some
transactionswillbedroppedandmust
reconnect.Reconnectioncouldtakeseveral
minutes.

Workload
Management

RACOneNodecanonline
migrateadatabasefromone
servertoanothertoenable
loadbalancingofdatabases
acrossserversinthecluster.
Mostconnectionsshould
migratewithnodisruption

EEmustbefailedoverfromonenodeto
another,whichmeansallconnectionswill
bedroppedandmustreconnect.Some
transactionswillbedroppedandmust
reconnect.Reconnectioncouldtakeseveral
minutes.

Onlinescale
out

Onlineupgradetomultinode
RAC

TakeDBoutageandrelinktoupgradeto
multinodeRAC,restartDB.

Standardizedtools
andprocesses

RACandRACOneNodeuse
sametools,management
interfaces,andprocesses

EEandRACusedifferenttools,
managementinterfaces,andprocesses

WhatisOracleRealApplicationClustersOneNode(RACOneNode)?
OracleRACOneNodeisanoptionavailablewithOracleDatabase11gRelease2.OracleRACOneNodeisasingleinstanceofOracleRACrunningononenodeinacluster.
ThisoptionaddstotheflexibilitythatOracleoffersforreducingcostsviaconsolidation.Itallowscustomerstomoreeasilyconsolidatetheirlessmissioncritical,singleinstancedatabases
intoasinglecluster,withmostofthehighavailabilitybenefitsprovidedbyOracleRealApplicationClusers(automaticrestart/failover,rollingpatches,rollingOSandclusterware
upgrades),andmanyofthebenefitsofservervirtualizationsolutionslikeVMware.
RACOneNodeoffersbetterhighavailabilityfunctionalitythantraditionalcoldfailoverclustersolutionsbecauseofanewOracletechnologyOmotion,whichisabletointelligentlyrelocate
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

63/72

8/24/2015

Document220970.1
RACOneNodeoffersbetterhighavailabilityfunctionalitythantraditionalcoldfailoverclustersolutionsbecauseofanewOracletechnologyOmotion,whichisabletointelligentlyrelocate
databaseinstancesandconnectionstootherclusternodesforhighavailabilityandsystemloadbalancing.

IfIaddorremovenodesfromthecluster,howdoIinformRACOneNode?
YoumustrerunraconeinittoupdatethecandidateserverlistforeachRACOneNodeDatabase.

IsRACOneNodesupportedwithdatabaseversionspriorto11.2?
No.RACOneNoderequiresatleastversion11.2ofOracleGridInfrastructure,andtheRACOneNodedatabasemustbeatleast11.2.Earlierversionsoftherdbmscancoexistwith
11.2RACOneNodedatabases.

HowdoIgetOracleRealApplicationClustersOneNode(OracleRACOneNode)?
OracleRACOneNodeisonlyavailablewithOracleDatabase11gRelease2.OracleGridInfrastructurefor11gRelease2mustbeinstalledasaprerequisite.DownloadandapplyPatch
9004119toyourOracleRAC11gRelease2homeinordertoobtainthecodeassociatedwithRACOneNode.(thispatchwasreleasedafter11.2.0.1wasreleasedandisonlyavailable
forLinux).Supportforotherplatformswillbeaddedwith11.2.0.2.ThedocumentationistheOracleRACOneNodeUserGuide

WheredoIfindthedocumentationforRACOneNode?
RACOneNodewasreleasedasapatchaftertheoriginalGAreleaseofOracleDatabase11gRelease2.RACOneNodedocumentationwillbeincludedinthenextdocsetrefresh.
PleaserefertoOracleRACOneNodeUserGuide

DoesEnterpriseManagerSupportRACOneNode?
Yes,youcanuseEnterpriseManagerDBConsoletomanageRACOneNodedatabases.Notethatin11.2.0.1,whenyourunraconeinit,theinstancenameischanged.youshouldeither
configureEMDBConsoleafterrunningraconeinit,andaftereveryinstancerelocation(Omotion)orfailover,theEMDBConsolewillneedtobereconfiguredtoseethenewinstanceon
thenewnode.ThiscanbedoneusingemcaandisthesameaswithaddinganynewDBtotheconfiguration.Inthefuture,11.2.0.2,EMwillsupportRACOneNodedatabaseoutofthe
box.soEMwillbeabletodetectwhentheinstanceismigratedorfailedovertoanothernode.

HowdoIinstallthecommandlinetoolsforRACOneNode?
ThecommandlinetoolsareinstalledwhenyouinstalltheRACOneNodepatch9004119ontopof11.2.0.1.

ArewecertifyingapplicationsspecificallyforRACOneNode?
No.Ifthe3rdpartyapplicationiscertifiedforOracleDatabase11gRelease2EnterpriseEdition,itiscertifiedforRACOneNode.

DoesRacOneNodemakesenseinastretchclusterenvironment?
Yes.However,rememberthatmoststretchclusterimplementationsalsoimplementdeparatestoragearraysatbothlocations.Sowritelatencyisstillanissuethatmustbeconsidered
sinceASMisstillwritingblockstobothsites.AnythingbeyondametroareaconfigurationislikelytointroducetoomuchlatencyfortheapplicationtomeetperformanceSLAs.

HowdoesRACOneNodecomparewithvirtualizationsolutionslikeVMware?
RACOneNodeoffersgreaterbenefitsandperformancethanVMwareinthefollowingways:
ServerConsolidation:VMwareoffersphysicalserverconsolidationbutimposesa10%+processingoverheadtoenablethisconsolidationandhavethehypervisorcontrolaccess
tothesystemsresources.RACOneNodeenablesbothphysicalserverconsolidationaswellasdatabaseconsolidationwithouttheadditionaloverheadofahypervisorbased
solutionlikeVMware.
HighAvailability:VMwareofferstheabilitytofailoverafailedvirtualmachineeverythingrunninginthatvmmustberestartedandconnectionsreestablishedintheeventofa
virtualmachinefailure.VMwarecannotdetectafailedprocesswithinthevmjustafailedvirtualmachine.RACOneNodeoffersafinergrained,moreintelligentandless
disruptivehighavailabilitymodel.RACOneNodecanmonitorthehealthofthedatabasewithinaphysicalorvirtualserver.Ifitfails,RACOneNodewilleitherrestartitor
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

64/72

8/24/2015

Document220970.1

disruptivehighavailabilitymodel.RACOneNodecanmonitorthehealthofthedatabasewithinaphysicalorvirtualserver.Ifitfails,RACOneNodewilleitherrestartitor
migratethedatabaseinstancetoanotherserver.Oftentimes,databaseissuesorproblemswillmanifestthemselvesbeforethewholeserverorvirtualmachineisaffected.RAC
OneNodewilldiscovertheseproblemsmuchsoonerthanaVMwaresolutionandtakeactiontocorrectit.Also,RACOneNodeallowsdatabaseandOSpatchesorupgradesto
bemadewithouttakingacompletedatabaseoutage.RACOneNodecanmigratethedatabaseinstancetoanotherserver,patchesorupgradescanbeinstalledontheoriginal
serverandthenRACOneNodewillmigratetheinstanceback.VMwareoffersafacility,Vmotion,thatwilldoamemorytomemorytransferfromonevirtualmachineto
another.ThisDOESNOTallowforanyOSorotherpatchesorupgradestooccurinanondisruptivefashion(anoutagemustbetaken).Itdoesallowforthehardwaretobe
dustedandvacuumed,however.
Scalability:VMwareallowsyoutoscaleonasinglephysicalserverbyinstantiatingadditionalvirtualmachinesuptoan8corelimitpervm.RACOneNodeallowsonline
scalingbymigratingaRACOneNodeimplementationfromoneservertoanother,morepowerfulserverwithouttakingadatabaseoutage.Additionally,RACOneNodeallows
furtherscalingbyallowingtheRACOneNodetobeonlineupgradedtoafullRealApplicationClustersimplementationbyaddingadditionaldatabaseinstancestothecluster
therebygainingalmostunlimitedscalability.
OperationalFlexibilityandStandardization:VMwareonlyworksonx86basedservers.RACOneNodewillbeavailableforalloftheplatformsthatOracleRealApplication
ClusterssupportsincludingLinux,Windows,Solaris,andAIX,HPUX.

CanIuseOracleRACOneNodeforStandardEditionOracleRAC?
No,OracleRACOneNodeisonlypartofOracleDatabase11gRelease2EnterpriseEdition.Itisnotlicensedorsupportedforusewithanyothereditions.

WhatisRACOneNodeOmotion?
OmotionisautilitythatisdistributedaspartofOracleRACOneNode.TheOmotionutilityallowsyoutomovetheOracleRACOneNodeinstancefromonenodetoanotherinthe
cluster.Thereareseveralreasonsyoumaywanttomovetheinstancesuchasthenodeisoverloadedsoyouneedtobalancetheworkloadbymovingtheinstance,oryouneedtodo
someoperatingsystemmaintenanceonthenodehoweveryouwanttoeliminatetheoutageforapplicationusersbymovingtheinstancetoanothernodeinthecluster.

HowtochangetheOracleGridInfrastructuresoftwareowner?
AnychangetotheOracleGridInfrastructureorOracleClusterwaresoftwareowner(theOSuserthatwasusedtoinstallthesoftware)currentlyrequiresreinstallationoftheOracle
GridInfrastructureorOracleClusterwaresoftware.TheOracleGridInfrastructure/OracleClusterwaresoftwareownercannotbechangedduringanupgrade,either.

IsitrecommendedthatweputtheOCR/VotingDisksinOracleASMand,ifso,isitpreferabletocreateaseparatediskgroupforthem?
WithOracleGridInfrastructure11gRelease2,itisrecommendedtoputtheOCRandVotingDisksinOracleASM,usingthesamediskgroupyouuseforyourdatabasedata.Forthe
OCRitisalsorecommendedtoputanotherOCRlocationintoadifferentdiskgroup(typically,theFastRecoveryAreadiskgroup,a.k.a.FRA)toprovideadditionalprotectionagainst
logicalcorruption,ifavailable.
UsingthesamediskgroupsfortheOracleClusterwarefiles(OCRandVotingDisks)simplifies(youdonothavetocreatespecialdevicestostorethosefiles)andcentralizesthestorage
management(allOraclerelatedfilesarestoredandmanagedinOracleASM),usingthesamecharacteristicsforthedatastored.
IftheVotingDisksarestoredinanOracleASMdiskgroup,thenumberofVotingDisksthatwillbecreatedinthisdiskgroupandfortheclusterisdeterminedbytheredundancylevelof
therespectivediskgroup.Formoreinformation,seeVotingFilesstoredinASMHowmanydisksperdiskgroupdoIneed?TheVotingDisksforaparticularclustercanonlyresidein
onediskgroup.
Incase"externalredundancy"hasbeenchosenforthediskgroupthatholdsthedatabasedata,itisassumedthatanexternalmechanism(e.g.RAID)isusedtoprotectthedatabase
dataagainstdiskfailures.ThesamemechanismcanthereforebeusedtoprotecttheOracleClusterwarefiles,includingtheVotingDisk(onlyoneVotingDiskiscreatedinan"external
redundancydiskgroup").
Undercertaincircumstances,onemaywanttocreateadedicateddiskgroupfortheOracleClusterwarefiles(OCRandVotingDisks),separatedfromtheexistingdatabasedata
containingdiskgroups.Whilehavingadedicateddiskgroupisnotrequired,itcanbeconfiguredandmightbeusefulundercertaincircumstances.Potentialscenarios
include,butarenotlimitedto:
A1:1relationshipbetweendiskgroupsanddatabasesispreferredanddiskgroupsaregenerallynotsharedamongstdatabases.
Thebackupandrecoveryforindividualdatabases(morethanoneinthecluster)isbasedonasnapshotrestoremechanism(BCVs).Thisapproachismostlikelyusedinconjunctionwith
a1:1diskgrouptodatabaserelationshipasmentionedbefore.
Certainandfrequentsystemspecificmaintenancetasksuncommonlyrequiretounmountspecific,databasedatacontainingdiskgroups.Thisscenariocanmostlikelybeavoidedusinga
differentapproachforthosemaintenancetasks.
Whenusingadiskgroupthatcontainsaquorumfailgroupinadditiontoregularfailgroups(typicallyusedinextendeddistanceOracleRACclusters)andwhenaccesstothedisk(s)inthe

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

65/72

8/24/2015

Document220970.1

Whenusingadiskgroupthatcontainsaquorumfailgroupinadditiontoregularfailgroups(typicallyusedinextendeddistanceOracleRACclusters)andwhenaccesstothedisk(s)inthe
quorumfailgroupmaybesubjecttofailuretogetherwithanyotherfailgroupinthesamediskgrouporaccesstothedisk(s)inthequorumfailgroupisotherwiseconsideredunreliable.
Ahigherprotectionlevelthantheoneprovidedforthe"externalredundancydiskgroups"andthereforeforthedatabasedataisforsomereasonrequiredfortheOracleClusterware
files.

WhatdoIneedtodowhenIneedtochangethehostnameforaclusternoderunningOracleClusterware?
ThehostnameofaclusternodebecomesanintegralpartoftheclusterconfigurationandfolderstructureduringtheinstallationofOracleGridInfrastructure/OracleClusterware.
AhostnamechangeofaclusternodesubsequenttotheinitialinstallationwillatleastrequireareconfigurationoftheOracleGridInfrastructureorOracleClusterwareconfigurationon
thisparticularnode.MoreinformationcanbefoundintheOracleDocumentation,"OracleGridInfrastructureInstallationGuides"forOracleRAC11gRelease2,chapter6"Howto
ModifyorDeinstallOracleGridInfrastructure",subsection:"DeconfiguringOracleClusterwareWithoutRemovingBinaries".
NotethatthereconfigurationofOracleGridInfrastructurewithoutremovingthebinariesdoesnotremovefolderstructuresundertheOracleGridInfrastructurehomecontainingthe
oldhostname.
Ifyouneedtoremovethose,youcaneithermanuallydeletethemoralternativelyperformadeletenode,followedbyanaddnodeprocedure.Afteryoudeletedthenodefromthe
configuration,youcanremovetheoldOracleGridInfrastructureorOracleClusterwarehome,makingsurethatyouperformtheaddnodeproceduresothatitcopiesnewbinaries(this
isthedefaultincurrentreleases).

HowtoefficientlyrecoverfromalossofanOracleASMdiskgroupcontainingtheOracleClusterwarefiles?
IfanOracleASMdiskgroupcontainingOracledatabasedataandtheOracleClusterwarefilesislostcompletely,thesystemneedstoberestoredstartingwiththerestoreoftheOracle
Clusterwarefilesaffected.
Oraclerecommendstohavetwodiskgroupsasastandarddeploymentscenario:thedatabasedatacontainingdiskgroup(commonlyreferredtoastheDATAdiskgroup)
andthebackupdatacontainingdiskgroup(commonlyreferredtoastheFRAdiskgroup).Inthisconfiguration,theOracleVotingFiles(s)andthefirstOracleClusterRegistry(OCR)
locationshouldsharethesamediskgroupastheOracleDatabasedata,heretheDATAdiskgroup.AsecondOCRlocationshouldbeplacedintotheseconddiskgroup,hereFRA,using
"ocrconfigadd+FRA"asroot,whiletheclusterisrunning.
AcompletefailureoftheFRAdiskgroupwouldbewithouteffectfortheoverallclusteroperationinthiscase.AcompletefailureoftheDATAdiskgroupinsteadwillrequirearestoreof
theOracleVotingFilesandtheOracledatabasedatathatwereformerlystoredinthisdiskgroup.
Inanycase,themostefficientproceduretorestorefromalossofanOCRstoredinOracleASMistdescribedinMyOracleSupport(MOS)note1062983.1HowtorestoreASM
basedOCRaftercompletelossoftheCRSdiskgrouponLinux/Unixsystems

CanIremoveOracleresourcesfromthecluster?HowcanIhideunusedresourceswhenlistingallresourcesinthecluster?
OnemustnotremoveordeleteanyOracleresourcesfromthecluster.OracleresourcesaredefinedaretypicallypreconfiguredduringtheinstallationofOracleClusterware/
GridInfrastructureoraddedinthecourseofadefaultinstallationorconfigurationprocessofotherOracleproducts.WithOracleDatabase11gRelease2Oraclehastakentheapproach
topreconfiguresomeresources,butactivatethem(orhavethemactivated)onlyoncerequired.Aslongasthecomponentsrepresentedbytheseresourcesarenotused,theOracle
Clusterwareproxyresourcesaresettooffline/aredisabled,whichmeans,theyarenotrunningandshouldthereforenotcauseanyissues.
HowcanIhideunusedresourceswhenlistingallresourcesinthecluster?Ifonewantstolistonlythoseresourcesthatareactivelyusedinthecluster,thefollowing
commandcanbeused:
crsctlstatresw'ENABLED!=0'thetoptioncanbeusedtogeta"tabularview"
Note:IfASMisnotusedfortheclusteratall,disabletheASMproxyresourceinOracleClusterwareinordertonotlistitusingthiscommand.TodisabletheASMproxyresourcein
OracleClusterware,the"srvctldisableasm[n(node_name)]"commandcanbeused.IfASMisusedtostoretheVotingDisksand/orOCRs,DBfiles,oranACFSfilesystem,ASM
mustbeenabled!

HowtouseSCANandnodelistenerswithdifferentports?
Forthepurposeofallowingapplicationstomaintainadistinguishableconnectstringonhost/portlevelusingoneSCANasthehostortoenableportbasedfirewallconfigurations,SCAN
canbesetuptosupportmorethanoneport.Itneedstobenoticedthatdifferentportsareonlyusedforaninitialconnect.
OnceaconnectionusingSCANonanySCANporthasbeenestablished,portassignmentisnotmaintainedandroutingofconnectionswithintheOracleRACclusterisperformedusing
databaseservicesregisteredwithoneormorelocallisteners.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

66/72

8/24/2015

Document220970.1

databaseservicesregisteredwithoneormorelocallisteners.
ThismeansthatallSCANportsaretreatedthesame,sothataconnectionthatcomesinonanySCANportcanbedirectedtoanyregisteredlocallisteneronanyport.
Formoreinformationon"HowtouseSCANandnodelistenerswithdifferentports?",seeMyOracleSupportnote1585786.1

HowtochangetheSCANconfigurationaftertheOracleGridInfrastructure11gRelease2installationiscomplete?
UseSRVCTLtomodifytheSCAN.
InordertomaketheclusterawareofthemodifiedSCANconfiguration,deletetheentryinthehostsfileormakesurethatthenewDNSentryreflects(dependingonwhereyouhave
setupyourSCANnameresolutioninthefirstplace)andthenissue:"srvctlmodifyscann"astherootuserononenodeinthecluster.
Thescan_nameprovidedcanbetheexistingfullyqualifiedname(oranewname),butshouldberesolvedthroughDNS,having3IPsassociatedwithit.Theremainingreconfigurationis
thenperformedautomatically.
Asuccessfulreconfigurationwillresultin3SCANVIPsand3SCAN_LISTENERSinthecluster,enablingtoloadbalancingofconnectionstodatabasesrunninginthecluster.Each
SCAN_LISTENERlistensononeoftheSCANVIPaddresses.
MostchangestotheSCANconfigurationcanbeperformedusing'srvctlmodifyscan'.Thisincludesnamechanges(changestotheSCANname)andIPaddresschanges(assumingthat
thenewIPaddressesaretakenfromthesamesubnetastheoldones).RemovingandaddingbacktheSCANconfigurationshouldnotberequired.However,theSCANlistenersmay
needtoberestartedusing'srvctlstop/startscan'toreflectanIPaddresschange,iftheIPaddresseswerechanged.
AlsonotethatupdatingtheSCANnamemightrequiretochangetheremote_listenersettingsforthevariousOracleRACdatabasesinthecluster,sincethedefaultconfigurationwouldbe
tohavetheremote_listenerparameterforanOracleRACdatabasepointtotheSCANname.IftheSCANnamechanges,theparameterneedstobeupdatedmanuallyforeach
database.

WhyamIonlyusing1outof3SCANIPaddresses?
TheSCANnamemustbesetuptoroundrobinacross3IPaddresses.ThisrequiresaSCANnameresolutionviaeitherDNSorthenewOracleGridNamingService(GNS).
Usingthehostsfile(Linux:/etc/hosts),youwillonlyget1SCANIPandyoucannotworkaroundthisotherthanusingtheformerlymentionedDNSorGNSbasednameresolution.
Tryingtoworkaroundthisrestrictionbysettingupahostsfileentrylikethefollowingonewillnotworkasexpectedandshouldthereforebeavoided,sinceitisanonconformantuseof
thehostsfile:
#SCANaddr
192.21.101.74rac16cluster.example.comrac16cluster
192.21.101.75rac16cluster.example.comrac16cluster
192.21.101.76rac16cluster.example.comrac16cluster
Evenwithsuchahostsfileentry,youwillonlyget1SCANVIPand1SCANListener.
IFyouhavesetupaDNSbasedSCANnameresolutionandyoustillnoticethattheclientwouldonlyuseoneIPaddress(outofthethreeIPaddressesthatareresolvedviaSCAN),
makesurethattheSCANaddressesarereturnedbytheDNSinaroundrobinmanner.YoucanchecktheSCANconfigurationinDNSusingnslookup.IfyourDNSissetuptoprovide
roundrobinaccesstotheIPsresolvedbytheSCANentry,thenrunthenslookupcommandatleasttwicetoseetheroundrobinalgorithmwork.Theresultshouldbethateachtime,
thenslookupwouldreturnasetof3IPsinadifferentorder.

HowtoinstallOracleGridInfrastructureusingSCANwithoutusingDNS?
OracleUniversalInstaller(OUI)enforcesprovidingaSCANresolutionduringtheOracleGridInfrastructureinstallation,sincetheSCANconceptisanessentialpartduringthecreationof
OracleRAC11gRelease2databasesinthecluster.AllOracleDatabase11gRelease2toolsusedtocreateadatabase(e.g.theDatabaseConfigurationAssistant(DBCA),orthe
NetworkConfigurationAssistant(NetCA))wouldassumeitspresence.Hence,OUIwillnotletyoucontinuewiththeinstallationuntilyouhaveprovidedasuitableSCANresolution.
However,inordertoovercometheinstallationrequirementwithoutsettingupaDNSbasedSCANresolution,youcanuseahostsfilebasedworkaround.Inthiscase,youwouldusea
typicalhostsfileentrytoresolvetheSCANtoonly1IPaddressandoneIPaddressonly.ItisnotpossibletosimulatetheroundrobinresolutionthattheDNSserverdoesusingalocal
hostfile.ThehostfilelookuptheOSperformswillonlyreturnthefirstIPaddressthatmatchesthename.Neitherwillyoubeabletodosoinoneentry(onelineinthehostsfile).Thus,
youwillcreateonly1SCANforthecluster.(Notethatyouwillhavetochangethehostsfileonallnodesintheclusterforthispurpose.)
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

67/72

8/24/2015

Document220970.1

Thisworkaroundmightalsobeusedwhenperforminganupgradefromformer(preOracleDatabase11gRelease2)releases.However,itisstronglyrecommendedtoenabletheSCAN
configurationasdescribedunderOption1orOption2aboveshortlyaftertheupgradeortheinitialinstallation.InordertomaketheclusterawareofthemodifiedSCAN
configuration,deletetheentryinthehostsfileandthenissue:"srvctlmodifyscann"astherootuserononenodeinthecluster.Thescan_nameprovidedcanbetheexistingfully
qualifiedname(oranewname),butshouldberesolvedthroughDNS,having3IPsassociatedwithit,asdiscussed.Theremainingreconfigurationisthenperformedautomatically.

HowcanIaddmoreSCANVIPsorlistenersnotusingDNS?
Youcanonlycreatethe3SCANVIPsand3SCANListenersacrossthecluster,ifyouhaveaDNSaliaseitheratinstallationtimeorlater.YouneedtoresolvetheSCANNametothose
formerlymentioned3IPaddressesatthemomentofcreationorwhenmodifyingtheSCAN.ThisishowtheygetcreatedtheIPsthatareresolvedbytheSCANDNSentryarereadand
therespectiveVIPsgetcreated.
IFyouhavenoDNSatallathandatanytime,especiallynotfortheserversinyourcluster,youwillnotget3SCANVIPsinyourclusterandhenceyouwillhaveonly1VIP,whichcanbe
consideredasinglepointoffailure.
Thismeansthatyouhave2choices:YoucaneitherlivewiththisconfigurationandtherespectiveconsequencesORyoucanfallbacktousingthenodesVIPsoftheclustertoconnect
yourclientsto,neitherofwhichisrecommended,asmentionedinMyOracleSupportnotewithDOCId.887522.1forexample.
FormoreinformationonhowtochangetheSCANconfiurationaftertheinstallationiscompleteusingsrvctl,please,seetheRACFAQentrytitled:"HowtochangetheSCAN
configurationaftertheOracleGridInfrastructure11gRelease2installationiscomplete?"

HowdoIexplainthefollowingphraseinthe"OracleClusterwareAdministrationandDeploymentGuide11gRelease2(11.2)"toacustomer?
Page227:"IfOracleASMfails,thenOCRisnotaccessibleonthenodeonwhichOracleASMfailed,buttheclusterremainsoperational.Theentireclusteronly
failsiftheOracleASMinstanceontheOCRmasternodefails,ifthemajorityoftheOCRlocationsareinOracleASM,andifthereisanOCRreadorwrite
access,thenthecrsdstopsandthenodebecomesinoperative."
Thiswasadocumentationbugandhasbeenfixed.
Hereistheupdatedwriteup(postedintheonlineversion):
IfanOracleASMinstancefailsonanynode,thenOCRbecomesunavailableonthatparticularnode.IfthecrsdprocessrunningonthenodeaffectedbytheOracleASMinstancefailure
istheOCRwriter,themajorityoftheOCRlocationsarestoredinOracleASM,andyouattemptI/OonOCRduringthetimetheOracleASMinstanceisdownonthisnode,thencrsd
stopsandbecomesinoperable.Clustermanagementisnowaffectedonthisparticularnode.UndernocircumstanceswillthefailureofoneOracleASMinstanceononenodeaffectthe
wholecluster.

Iftheroot.shscriptfailsonanodeduringtheinstalloftheGridInfrastructurewithOracleDatabase11gRelease2,canIrerunit?
Yes,howeveryoushouldfirstfixtheproblemthatcausedittofail,onlythenrun:
GRID_HOME/crs/install/rootcrs.pldeleteforce
Thenrerunroot.sh

IstheGNSrecommendedformostOracleRACinstallations?
TheGridNamingService(GNS)isapartoftheGridPlugandPlayfeatureofOracleRAC11gRelease2.Itprovidesnameresolutionforthecluster.Ifyouhavealargercluster(greater
than46nodes)orarequirementtohaveadynamiccluster(youexpecttoaddorremovenodesinthecluster),thenyoushouldimplementGNS.Ifyouareimplementingasmallcluster
4nodesorless,youdonotneedtoaddGNS.Note:SelectGNSduringinstallassumesthatyouhaveaDHCPserverrunningonthepublicsubnetwhereOracleClusterwarecanobtainIP
addressesfortheNodeVIPsandtheSCANVIPs.

WhatisClusterHealthMonitor(IPD/OS)?
Thistool(formerlyknownasInstantaneousProblemDetectiontool)isdesignedtodetectandanalyzeoperatingsystem(OS)andclusterresourcerelateddegradationandfailuresin
ordertobringmoreexplanatorypowertomanyissuesthatoccurinclusterswhereOracleClusterwareandOracleRACarerunningsuchasnodeeviction.
IttrackstheOSresourceconsumptionateachnode,process,anddevicelevelcontinuously.Itcollectsandanalyzestheclusterwidedata.Inrealtimemode,whenthresholdsarehit,
analertisshowntotheoperator.Forrootcauseanalysis,historicaldatacanbereplayedtounderstandwhatwashappeningatthetimeoffailure.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

68/72

8/24/2015

Document220970.1
analertisshowntotheoperator.Forrootcauseanalysis,historicaldatacanbereplayedtounderstandwhatwashappeningatthetimeoffailure.

FormoreinformationonClusterHealthMonitor(IPD/IO),seethispubliclyavailableTechnicalWhitePaperonOTN:OverviewofClusterHeathMonitor(IPD/OS)

WhatOSdoesClusterHealthMonitor(IPD/OS)support?
ClusterHealthMonitor(IPD/OS)isastandalonetoolthatshouldbeinstalledonallclusterswhereyouareusingOracleRealApplicationClusters(RAC).ItisindependentoftheOracle
DatabaseorOracleClusterwareversionused.
ClusterHealthMonitor(IPD/OS)iscurrentlysupportedonLinux(requiresLinuxKernelversiongreaterthanorequalto2.6.9)andWindows(requiresatleastWindowsServer2003with
servicepack2).
Itsupportsboth,32bitand64bitinstallations.Theclientinstallationrequiresthe32bitJavaSDK.

WhatisOraclesgoalindevelopingQoSManagement?
QoSManagementisafullOraclestackdevelopmentefforttoprovideeffectiveruntimemanagementofdatacenterSLAsbyensuringwhentherearesufficientresourcestomeetall
objectivestheyareproperlyallocatedandshoulddemandorfailuresexceedcapacitythatthemostbusinesscriticalSLAsarepreservedatthecostoflesscriticalones.

WhattypeofapplicationsdoesOracleQoSManagementmanage?
QoSManagementiscurrentlyabletomanageOLTPopenworkloadtypesfordatabaseapplicationswhereclientsormiddletiersconnecttotheOracledatabasethroughOCIorJDBC.
OpenworkloadsarethosewhosedemandisunaffectedbyincreasesinresponsetimeandaretypicalofInternetfacingapplications.

WhatdoesQoSManagementmanage?
Indatacenterswhereapplicationssharedatabasesordatabasesshareservers,performanceismadeupofthesumofthetimespentusingandwaitingtouseresources.Sincean
applicationsuseofresourcesiscontrolledduringdevelopment,test,andtuningitcannotbemanagedatruntimehoweverthewaitforresourcescan.QoSManagementmanages
resourcewaittimes.

WhattypesofresourcesdoesQoSManagementmanage?
CurrentlyQoSManagementmanagesCPUresourcesbothwithinadatabaseandbetweendatabasesrunningonsharedordedicatedservers.ItalsomonitorswaittimesforI/O,Global
Cache,andOtherdatabasewaits.

WhattypeofuserinterfacesdoesQoSManagementsupport?
QoSManagementisintegratedintoEnterpriseManagerDatabaseControl11gRelease2andEnterpriseManager12cCloudControlandisaccessiblefromtheclusteradministration
page.

WhatQoSManagementfunctionalityisinOracleEnterpriseManager?
EnterpriseMangersupportsthefullrangeofQoSManagementfunctionalityorganizedbytask.APolicyEditorwizardpresentsasimpleworkflowthatspecifiestheserverpoolsto
managedefinesperformanceclassesthatmaptothedatabaseapplicationsandassociatedSLAsorobjectives,andspecifiesperformancepoliciesthatcontainperformanceobjectives
andrelativerankingforeachperformanceclassandbaselineserverpoolresourceallocations.Aneasytomonitordashboardpresentstheentireclusterperformancestatusataglance
aswellasrecommendedactionsshouldresourcesneedtobereallocatedduetoperformanceissues.Finallyasetofcomprehensivegraphstracktheperformanceandmetricsofeach
performanceclass.

Whattypesofperformanceobjectivescanbeset?
QoSManagementcurrentlysupportsresponsetimeobjectives.Responsetimeobjectivesuptoonesecondfordatabaseclientrequestsaresupported.Additionalperformanceobjectives
areplannedforfuturereleases.

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

69/72

8/24/2015

Document220970.1

DoesQoSManagementrequireanyspecificdatabasedeployment?
OracledatabasesmustbecreatedasRACorRACOneNodePolicyManageddatabases.Thismeansthedatabasesaredeployedinoneormoreserverpoolsandapplicationsandclients
connectusingCRSmanageddatabaseservices.EachmanageddatabasemustalsohaveResourceManagerenabledandbeenabledforQoSManagement.Itisalsorecommendedthat
connectionpoolsthatsupportFastApplicationNotification(FAN)eventsbeusedformaximumfunctionalityandperformancemanagement.

WhatareServerPools?
ServerPoolsareanewmanagemententityintroducedinOracleClusterware11gtogiveITadministratorstheabilitytobettermanagetheirapplicationsanddatacentersalongactual
workloadlines.ServerPoolsarealogicalcontainer,wherelikehardwareandworkcanbeorganizedandgivenimportanceandavailabilitysemantics.Thisallowsadministratorsaswell
asQoSManagementtoactivelygrowandshrinkthesegroupstomeetthehourtohour,daytodayapplicationdemandswithoptimumutilizationofavailableresources.Theuseof
ServerPoolsdoesnotrequireanyapplicationcodechanges,recompilingorrelinking.ServerPoolsalsoallowoldernonQoSManagementsupporteddatabasesandmiddlewaretoco
existinasingleclusterwithoutinterferingwiththemanagementofnewersupportedversions.

WhatmethodsdoesQoSManagementsupportforclassifyingapplicationsandworkloads?
QoSManagementusedatabaseentrypointstotagtheapplicationorworkloadwithuserspecifiednames.DatabasesessionsareevaluatedagainstclassifiersthataresetsofBoolean
expressionsmadeupofServiceName,Program,User,ModuleandAction.

WhatistheoverheadofusingQoSManagement?
TheQoSManagementServerisasetofJavaMBeansthatruninasingleJ2EEcontainerrunningononenodeinthecluster.Metricsareretrievedfromeachdatabaseonceeveryfive
seconds.Workloadclassificationandtaggingonlyoccursatconnecttimeorwhenaclientchangessessionparameters.Thereforetheoverheadisminimalandisfullyaccountedforin
themanagementofobjectives.

DoesQoSManagementnegativelyaffectanapplicationsavailability?
No,theQoSManagementserverisnotinthetransactionpathandonlyadjustsresourcesthroughalreadyexistingdatabaseandclusterinfrastructure.Infact,itcanimproveavailability
bydistributingworkloadswithinandclusterandpreventnodeevictionscausedmymemorystresswithitsautomaticMemoryGuardfeature.

WhathappensshouldtheQoSManagementServerfail?
TheQoSManagementServerisamanagedClusterwaresingletonresourcethatisrestartedorfailedovertoanothernodeintheclustershouldithangorcrash.Evenifafailureoccurs,
thereisnodisruptiontothedatabasesandtheirworkloadsrunninginthecluster.Oncetherestartcompletes,QoSManagementwillcontinuemanagingintheexactstateitwaswhen
thefailureoccurred.

WhatisMemoryGuardandhowdoesitwork?
MemoryGuardisanexclusiveQoSManagementfeaturethatusesmetricsfromClusterHealthMonitortoevaluatethestressofeachserverintheclusteronceaminute.Shoulditdetect
anodehasovercommittedmemory,itwillpreventnewdatabaserequestsfrombeingsenttothatnodeuntilthecurrentloadisrelieved.Itdoesthismyturningofftheservicestothat
nodetransactionallyatwhichpointexistingworkwillbegintodrainoff.Oncethestressisnolongerdetected,serviceswillautomaticallybestartedandnewconnectionswillresume.

HowdoesQoSManagementenablethePrivateDatabaseCloud?
ThePrivateDatabaseCloudfundamentallydependsuponsharedresources.Whetherdeployingadatabaseserviceoraseparatedatabase,bothdependuponbeingabletodeliver
performancewithcompetingworkloads.QoSManagementprovidesboththemonitoringandmanagementofthesesharedresources,thuscomplementingtheflexibledeploymentof
databasesasaservicetoalsomaintainaconsistentlevelofperformanceandavailability.

WhichversionsofOracledatabasesdoesQoSManagementsupport?
QoSManagementissupportedonOracleRACEEandRACOneEEdatabasesfrom11gRelease2(11.2.0.2)forwarddeployedonOracleExadataDatabaseMachine.Itisalsosupported
inMeasureOnlyModewithMemoryGuardsupportonOracleRACEEandRACOneEEdatabasesfrom11gRelease2(11.2.0.3)forward.PleaseconsulttheOracleDatabaseLicense
Guidefordetails.
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

70/72

8/24/2015

Document220970.1

IsthisaproducttobeusedbyanITadministratororDBA?
TheprimaryuserofQoSManagementisexpectedtobetheITorsystemsadministratorthatwillhaveQoSadministrativeprivilegesontheRACcluster.AsQoSManagementactively
managesallofthedatabasesinaclusteritisnotdesignedforusebytheDBAunlessthatindividualalsohastheclusteradministrationresponsibility.DBAlevelexperienceisnot
requiredtobeaQoSManagementadministrator.

WherecanIfinddocumentationforQoSManagement?
TheOracleDatabaseQualityofServiceManagementUsersGuideisthesourcefordocumentationandcoversallaspectsofitsuse.ItiscurrentlydeliveredaspartoftheOracle
DatabaseDocumentationLibrarystartingin11gRelease2.

REFERENCES
NOTE:444134.1UsingIBMGPFS2.3withOracleRealApplicationClusters10g
NOTE:454607.1NewPartitionsinWindows2003RACEnvironmentsNotVisibleonRemoteNodes
NOTE:301138.1RACDDTUserGuide
NOTE:77370.1WhataretheeffectsofchangingthesystemclockonanOracleServerinstance?
BUG:4537790CLUSTERNODESHANGANDTHENBUGCHECKONREBOOTS
NOTE:151051.1Init.oraParameter"CLUSTER_INTERCONNECTS"ReferenceNote
NOTE:787420.1ClusterInterconnectinOracle10gand11gR1RAC
NOTE:805969.110g/11gR1:VIPdoesnotrelocatebacktothehomenodeevenafterthepublicnetworkproblemisresolved
NOTE:298891.1Linux:ConfiguringBondingForOracle10gVIPorprivateinterconnect
NOTE:726833.1Linux:HangcheckTimerModuleRequirementsforOracle9i,10g,and11gR1RAC
NOTE:732683.1CannotStartInstanceUsingSRVCTLbutSQLPLUSCan
NOTE:743649.1WillanOperatingSystemUpgradeAffectOracleClusterware?
NOTE:465001.1Configuringrawdevices(singlepath)forOracleClusterware10gRelease2(10.2.0)onRHEL5/OEL5
NOTE:5187351.8Bug5187351ManyclscNN.logfilescreatedcanaffectperformance
NOTE:458485.1HowtofindwhethertheoneoffPatcheswillconflictornot?
NOTE:559365.1UsingDiagwaitasadiagnostictogetmoreinformationfordiagnosingOracleClusterwareNodeevictions
NOTE:751343.1OracleClusterwareandRACSupportforRDSOverInfiniband
NOTE:759143.1NTPleapsecondeventcausingOracleClusterwarenodereboot
NOTE:554781.1RACinstabilitiesduetofirewall(netfilter/iptables)enabledontheclusterinterconnect
NOTE:279793.1HowtoRestoreaLostVotingDiskin10g
NOTE:139436.1Understanding9iRealApplicationClustersCacheFusion
NOTE:180608.1AutomaticSpaceSegmentManagementinRACEnvironments
NOTE:358156.1AutomaticStartupofClusterServicesMayHangorFailonWindows
BUG:4024251ALLOWDBCATOALLOWFORMORETHAN5CHARACTERSFORSIDPREFIX
BUG:4462367CRSSTARTSINSTANCEWITHTOOLOWFILELIMIT
NOTE:342443.110.2.0.xOracleDatabaseandNetworkingPatchesforMicrosoftPlatforms
NOTE:464683.1UnexplainedDatabaseSlowdownSeenonWindows2003ServicePack1
NOTE:283107.1ConfiguringSolarisIPMultipathing(IPMP)fortheCRS10gVIP
NOTE:238278.1Linux:What'sOCFSorOCFS2
NOTE:249212.1SupportPositionforOracleProductsRunningonVMWareVirtualizedEnvironments
NOTE:219361.1TroubleshootingORA29740inaRACEnvironment
NOTE:314422.1RemoteDiagnosticAgent(RDA)GettingStarted
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

71/72

8/24/2015

NOTE:314422.1RemoteDiagnosticAgent(RDA)GettingStarted
NOTE:332257.1UsingOracleClusterwarewithVendorClusterwareFAQ
NOTE:337737.1OracleClusterware(CRS/GI)ASMDatabaseVersionCompatibility

Document220970.1

NOTE:1062983.1HowtorestoreASMbasedOCRaftercompletelossoftheCRSdiskgrouponLinux/Unixsystems
NOTE:183340.1FrequentlyAskedQuestionsAbouttheCLUSTER_INTERCONNECTSParameterin9i.
NOTE:341788.1RecommendationfortheRealApplicationClusterInterconnectandJumboFrames
NOTE:184875.1HowToCheckTheCertificationMatrixforRealApplicationClusters
NOTE:359515.1MountOptionsforOraclefileswhenusedwithNFSonNASdevices
NOTE:368464.1HowtoSetupIPMPasClusterInterconnect
NOTE:291958.1SettingUpBondinginSuseSLES8
NOTE:291962.1SettingUpBondinginSLES9
NOTE:296874.1ConfiguringtheHPUXOperatingSystemfortheOracle10gandOracle11gVIP
NOTE:296856.1ConfiguringtheIBMAIX5LOperatingSystemfortheOracle10gVIP
NOTE:300479.110gRollingUpgradeswithLogicalStandby
NOTE:269320.1RemovingaNodefroma10gR1RACCluster
NOTE:276434.1HowtoModifyPublicNetworkInformationincludingVIPinOracleClusterware
NOTE:397460.1Oracle'sPolicyforSupportingOracleRACwithSymantecSFRAC
NOTE:399482.1HowtoRecreateOCR/VotingDiskAccidentallyDeleted
NOTE:405820.110.2.0.XCRSBundlePatchInformation
NOTE:414897.1Linux:HowToSetupUDEVRulesForRACOCRAndVotingDevicesOnSLES10,RHEL5,OEL5,OL5
NOTE:420977.1HowtoUselistener.orainNondefaultLocationfor10gRAC
Didn'tfindwhatyouarelookingfor?

AskinCommunity...

Related
Products
OracleDatabaseProducts>OracleDatabaseSuite>OracleDatabase>OracleDatabaseEnterpriseEdition>RealApplicationCluster

Keywords
CLUSTERMANAGER CONNECTIONLOADBALANCING FAILOVER FASTAPPLICATIONNOTIFICATION FASTCONNECTIONFAILOVER LOADBALANCING NOTIFICATION ORACLEPROCESSMANAGERANDNOTIFICATION STANDARDEDITION

Errors
CRS215 ORA1031 ORA29740 ORA3113 RFC1918 SDK.CRS
BacktoTop

Copyright(c)2015,Oracle.Allrightsreserved.

LegalNoticesandTermsofUse

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=195010624546244&id=220970.1&displayIndex=3&_afrWindowMode=0&_adf.ctrlstate=1bnizem9m7_72#A10154

PrivacyStatement

72/72