Sharing experience implementing

Direct NFS
Yury & Oracle Community
Contributors list
From Twitter
John Piwowar (@jpiwowar)
Leighton L. Nelson
(@leight0nn)
Kevin Closson(@kevinclosson)
Guenadi Jilevski (@gjilevski)
Simon Haslam
(@simon_haslam)
neto from Brazil
(@netofrombrazil)
Frits Hoogland (@fritshoogland)
Martin Bach (@MartinDBA) https://dl.dropbox.com/u/28277
Gwen (Chen) Shapira 005/DNFS%20exp%2004.pptx
(@gwenshap)
Avi Miller (@Djelibeybi) 2012 Pythian
Why Pythian
Recognized Leader:
Global industry leader in data infrastructure managed services and consulting with
expertise in Oracle, Oracle Applications, MicrosotSQL Server, MySQL, big data and
systems administration
Work with over 200 multinational companies such as Forbes.com, Fox Sports,
Nordion and Western Union to help manage their complex IT deployments

Expertise:
One of the worlds largest concentrations of dedicated, full-time DBA expertise.
Employ 8 Oracle ACEs/ACE Directors
Hold 7 Specializations under Oracle Platinum Partner program, including Oracle
Exadata, Oracle GoldenGate & Oracle RAC

Global Reach & Scalability:

24/7/365 global remote support for DBA and consulting, systems administration,
2012 Pythian
special projects or emergency response
 Why Yury
Google: Yury Oracle [phone|email]

Twitter, LinkedIn, Blog, Slideshare,YouTube

Oracle ACE (RAC SIG international chair, Sydney Oracle Meetups)

Oracle Certified Master
Oracle DBA with 15+ years experience
I like my job, I like what I do, I like to share knowledge, I like to help others to share knowledge, I like to learn, I
like my job.

2012 Pythian
Why Direct NFS

Network IO
SPEED
Easy to use +RAC

2012 Pythian
 k NFS d NFS

Load Profile Per Second Load Profile Per Second

~~~~~~~~~~~~ --------------- ~~~~~~~~~~~~ ---------------
DB Time(s): 19.6 DB Time(s): 19.6
DB CPU(s): 0.8 DB CPU(s): 5.2
Logical reads: 12,219.1 Logical reads: 38,277.5
Physical reads: 12,141.3 Physical reads: 38,156.1
1.7ms 0.5ms

100% 300%
Oracle VM, PIO SLOB test, ram disk mounted via 127.0.0.1 backloop device, other bottlenecks but kernel have been excluded, db file sequential read aka
random reads
Direct storage 0.36 ms
2012 Pythian
 It isnt about data files only !
File Type Supported
Control file YES Direct NFS: FAQ [ID 954425.1]
Data file YES
Redo log file YES RDBMS file type support matrix for
Direct NFS client
Archive/Flashback log file YES
Backup files YES
Temp file YES
Datapump dump file YES blog Direct NFS speeds up
Data Pump
OCR files NO
spfile YES
passwd file YES
ASM files YES blog Reasons for using ASM
on NFS
Voting files NO
Audit files NO
Database trace files NO
External tables NO
2012 Pythian
Direct NFS Concepts
 Simplified
User
k NFS d NFS
IO Client IO Client IO Client IO Client IO Client IO Client
Oracle Oracle Oracle Oracle Oracle Oracle
process process process process process process

user mode
NFS client NFS client NFS client

kerne
l
NFS client
Kernel mode

NFS service NFS service

kernel
File A File B File C File A File B File C

2012 Pythian
dNFS Setup simple version

mount NFS volumes

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dnfs_on

ls -l $ORACLE_HOME/lib/lib*od*11.so
-rw-r--r--. 1 oracle oinstall 60431 Sep 17 2011 $OH/lib/libnfsodm11.so << dnfs
-rw-r--r--. 1 oracle oinstall 60431 Oct 20 19:45 $OH/lib/libodm11.so # Oracle used
-rw-r--r--. 1 oracle oinstall 12259 Sep 17 2011 $OH/lib/libodmd11.so << no dnfs

Oracle processes use /etc/mtab to access NFS settings

http://www.pythian.com/news/34425/oracle-direct-nfs-how-to
-start/
odm stands for Oracle Disk Manager
2012 Pythian
How can the dNFS usage be
verified?
Direct NFS: FAQ [ID 954425.1]

1. Just after the initialization parameters are

listed in the alert log, you will see the
following entry
Oracle instance running with ODM: Oracle
Direct NFS ODM Library Version 2.0
2. Along with the message from the alert log,
this query on v$dnfs_servers ensures that dNFS
is truly being used (returns !=0 value):
select count(*) from v$dnfs_servers
2012 Pythian
How can the dNFS usage be
verified?
In my experience, the easiest and most reliable way is

1. lsof p < dbw pid >

# DNFS OFF
lsof -p 725 | grep data01.dbf
oracle 725 oracle 262u REG ... /nfsimp/data01.dbf (192.168.51.21:/u01)
# DNFS ON
lsof -p 6540 | grep 192.168.51.21
oracle 6540 oracle 32u IPv4 ... TCP dbhost:26171->nfsserver:nfs (ESTABLISHED)

2. alert.log
Direct NFS: channel id [0] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP
Direct NFS: channel id [1] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP

2012 Pythian
Direct NFS advanced
implementation
 dNFS setup is documented
Step by Step - Configure Direct NFS Client on Linux [ID 762374.1]
Step by Step - Configure Direct NFS Client on Windows [ID 1468114.1]
=======================================================
Set filesystemio_options at least to directIO. ALL database files.

PART A -- SETTING UP THE NFS SERVER ON LINUX

...
iv) Make sure the NFS server will get started during boot of this server.
...

MOUNTING NFS ON THE CLIENT NODE / CLUSTER NODES ON LINUX

...
stgasm:/oraclenfs /oradata1 nfs
rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0
# Please contact your NAS vendor for NFS mount option recommendations.
...

PART B -- Configure Direct NFS Client (DNFS)

i) Configure oranfstab file
...
Direct NFS Client can use a new configuration file or the mount tab file (/etc/mtab on Linux)
2012 Pythian
to determine the mount point settings for NFS storage devices.
 = Performance, Scalability, High Availability, Cost
dNFS oranfstabSavings
1Gbit 1Gbit

NIC

NIC`
RAC Node 1
4 TCP/IP connections

IO Client
Round
Robin
1Gbit Switc File A

NIC
Oracle
1Gbit h
1Gbit 1Gbit

NIC
NIC
process
1Gbit

NFS service
NIC
NIC
4 Gbit File B
RAC Node 2

1Gbit
Switc

NIC
IO Client
NIC

Oracle
1Gbit h
NIC

process 1Gbit
File C

NIC
NIC

2012 Pythian
dNFS oranfstab (things to remember)

You can live without it (simple implementations)

DB Hangs When DNFS is Enabled on UEK kernel [ID 1460787.1]

You may get confused reading though documentations

Enabling Direct NFS Client Oracle Disk Manager Control of NF

S
Server, Local, Path, Export, Mount, Mnt_timeout, Dontroute

Server any name you like (alias for a channel)

Local, Path - You can specify other IPs than in mtab
Export, Mount local and remote mount points
Mnt_timeout sessions drops a connection after the timeout
Dontroute dont use OS routeing table to send TCP/IP packages

2012 Pythian
Oracle Direct NFS configuration file explained
dNFS oranfstab (things to remember)

Direct NFS searches for mount entries in the following order

1. $ORACLE_HOME/dbs/oranfstab # an instance specific
2. /etc/oranfstab
3. /etc/mtab

You must restart an instance after adjusting oranfstab file

2012 Pythian
Troubleshooting & War stories
One DNFS disadvantage

It is relatively difficult to troubleshoot

You will not find many DNFS messages in alert.log or

trace files

Sometimes Oracle SILINTLY switches from dNFS to

kNFS
2012 Pythian
Dont mix old and new methods
Old method
cd $ORACLE_HOME/lib

mv libodm11.so libodm11.so_bak

ln -s libnfsodm11.so libodm11.so

New/current method
make -f ins_rdbms.mk dnfs_on

If you enable dNFS using one method disable it the

same way
2012 Pythian
Magic words to remember
Oracle Grid Infrastructure Installation Guide 11g Re
lease 2 (11.2) for Linux

If Oracle Database cannot open an NFS server

using Direct NFS, then Oracle Database SILENTLY
uses the platform operating system kernel NFS
client.

1. lsof p < dbw pid >

# dNFS OFF
lsof -p 725 | grep data01.dbf
oracle 725 oracle 262u REG ... /nfsimp/data01.dbf (192.168.51.21:/u01)
# dNFS ON
lsof -p 6540 | grep 192.168.51.21
oracle 6540 oracle 32u IPv4 ... TCP dbhost:26171->nfsserver:nfs (ESTABLISHED)
2012 Pythian
NFS server reserved ports
restriction
Oracle Grid Infrastructure Installation Guide 11g Release
2 (11.2) for Linux

Some NFS file servers require NFS clients to connect using

reserved ports. If your filer is running with reserved
port checking, then you must disable it for Direct NFS to
operate. To disable reserved port checking, consult your
NFS file server documentation.

In a simple NFS implementation that means:

cat /etc/exports
2012 Pythian

< dir > IP(...,insecure,...)

dNFS: wtmax =
/proc/fs/nfsd/max_block_size
Oracle Grid Infrastructure Installation Guide 11g Release
2 (11.2) for Linux
Caution: Direct NFS will not serve an NFS server with
write size values (wtmax) less than 32768.

...
ALTER DATABASE OPEN
Direct NFS: attempting to mount /u02 on filer KUKARACHA2 defined in oranfstab
Direct NFS: channel config is:
channel id [0] local [192.168.51.30] path [192.168.51.21]
Direct NFS: mount complete dir /u02 on KUKARACHA2 mntport 963 nfsport 2049
Direct NFS: Invalid filer wtmax 525232 on filer KUKARACHA2
Direct NFS: Filer wtmax 525232 must be an even multiple of 32768
Thread 1 opened at log sequence 26
...

2012 Pythian
dNFS: wtmax =
/proc/fs/nfsd/max_block_size
Pythians dNFS related blog posts
root@nfsfiler01 sysconfig# cat /proc/fs/nfsd/max_block_size
524288
root@nfsfiler01 sysconfig# echo 1048576 > /proc/fs/nfsd/max_block_size
root@nfsfiler01 sysconfig# cat /proc/fs/nfsd/max_block_size
1048576

...
ALTER DATABASE OPEN
Direct NFS: attempting to mount /u02 on filer KUKARACHA2 defined in oranfstab
Direct NFS: channel config is:
channel id [0] local [192.168.51.30] path [192.168.51.21]
Direct NFS: mount complete dir /u02 on KUKARACHA2 mntport 883 nfsport 2049
Direct NFS: channel id [0] path [192.168.51.21] to filer [KUKARACHA2] via local [192.168.51.30] is UP
Direct NFS: channel id [1] path [192.168.51.21] to filer [KUKARACHA2] via local [192.168.51.30] is UP
Beginning crash recovery of 1 threads
...

2012 Pythian
init.ora events to set
If you are confused and need more information
then set the following events and access NFS files
event "19392 trace name context forever, level 8" # kgnfs

event "19394 trace name context forever, level 8" # skgnfs

event "19396 trace name context forever, level 2" # kgodm

event="19398 trace name context forever, level 128"

alter system set event="19392 trace name context forever, level

8","19394 trace name context forever, level 8","19396 trace name
context forever, level 2","19398 trace name context forever, level
128" scope=spfile;

Read trace file, search MOS/Google, ask others

2012 Pythian
 References
Direct NFS: FAQ [ID 954425.1]
Example About How To Setup DNFS On Oracle Release 11.2 [ID 1452614.1]
How to Setup Direct NFS client multipaths in same subnet [ID 822481.1]
DB Hangs When DNFS is Enabled on UEK kernel [ID 1460787.1]
DB hang and ORA-600 [2116] errors when enabling DNFS [ID 1484422.1]
DNFS CANNOT MOUNT FILESYSTEM AND DATABASE FAILED WITH ORA-600 [2116] AFTER A REBOOT OF SERVER
DUE TO POWER FAILURE [ID 1480788.1]
DATABASE STARTUP HANGS AT MOUNTING CONTROLFILE WHEN DNFS IS ENABLED. [ID 971406.1]
Database Startup Failed with "Direct NFS: please check that oradism is setuid [1430654.1]
ORA-600 [2116] Using The Veritas Odm Lib Oracle Fails To Mount [ID 418603.1]
OERI [2116] [900] during database mount with > 13 instances [ID 9790947.8]
TESTCASE Step by Step - Configure Direct NFS Client (DNFS) on Windows [ID 1468114.1]
Database Alert Log entries: Direct NFS: Failed to set socket buffer size.wtmax=[1048576]
rtmax=[1048576], errno=-1 [ID 1352886.1]

@kevinclosson Oracle 11g, Direct NFS Client, An Oracle White Paper + http://bit.ly/QU3w82
@leight0nn http://blogs.griddba.com/2012/02/direct-nfs-speeds-up-data-pump.html
@rene_kundersma
https://blogs.oracle.com/XPSONHA/entry/using_dnfs_for_test_purposes
@yvelikanov http://www.pythian.com/news/tag/dnfs/
2012 Pythian
Your best friends OR things to keep in
mind

efficiency

2012 Pythian
Thank you and Q&A
To contact us
sales@pythian.com

1-877-PYTHIAN
To follow us
http://www.pythian.com/news/

http://www.facebook.com/pages/The-Pythian-
Group/163902527671
@pythian

@pythianjobs

http://www.linkedin.com/company/pythian

2012 Pythian