Database Layout On SAN

Mrs. Dee (Team Leader: Server Virtualization) posted 11/21/2006 | Comments (7)

I used to have this very simple question on database that nobody could provide a satisfactory
answer. I have been asking many database and UNIX experts on how to do the proper layout for
Oracle databases. There have been so many suggestions and scenarios discussed and we always
ended up having more doubts and disagreement among us. After reading some best practice
papers, I still couldn't come up with a general formula that I could use during database
deployments.

The following content was the result of an intense discussion we had during a product update
class we had a few weeks ago.

Let's take Oracle 9i database requirements for datafiles as an example. You'll most likely have
to create SYSTEM, UNDO, TEMPORARY, ONLINE REDO, DATA and INDEX tablespaces in the
database. There will be many data and index tablespaces defined to cater for the application
data storage requirement. The tablespaces for data could be created based on expected
application tables' behaviour for example:

1. ts_data01 for tables that will contain many large rows with heavy
select/insert/update/delete
2. ts_data02 for tables that will contain many small rows with heavy insert/update
3. ts_data03 for reference tables that will contain many small rows with infrequent inserts

The DBA can also choose to create data tablespaces based on departmental applications for
example:
1. ts_acc01 for tables that are used by inventory applications accessed by the accounting
department.
2. ts_log01 for tables that are used by logistics application accessed by the logistic
department.

The above decisions are usually made during application design phase of each application
software assuming that they are all developed in-house. However, for applications that are
already in production, the DBA can always relocate tables to a more suitable tablespaces
whenever needed.

For tablespaces that are defined according to table growth behaviour, it would be best for the
DBA and storage administrator to strategize the database layout based on the expected type of
I/O. In this case, both party must sit down together during planning sessions.

It has been almost like a ritual for the DBA to have a small disagreement with the storage
administrator when it comes to defining their requirement for LUNs to be created. In many
cases, the storage administrator will happily suggest to have a single big LUN for a medium
sized database.

If that database is mainly used for development and testing purposes, creating it on top of a
single LUN may not cause any performance issue to the system. The system administrator will
later on create separate filesystems for the DBA to organize the database files. Although in this
case, filesystem hierarchy won't help improve disk performance, it will help reduce the System
Administrator's headaches.

So, what if it's going to be a huge database with heavy read write operations around the clock?
The DBA is not usually aware of the application response time promised to the customer. This
is one of the classic cases when ignorance is not bliss. The database layout somehow plays a
significant role to determine I/O performance.

In this case, the following things should be considered by the storage administrator:

1. Having a few RAID groups in a storage subsystem.

The storage subsystem could come with 14, 28 or more hard disks. Based on normal practise,
and referring to the storage documentation, you could find that each RAID type can have a
certain maximum number of hard drives. However it is best not to choose the maximum
number while creating the RAID group. We should also realise that for storage that supports
dynamic LUN sizing or dynamic array expansion, there will be times when you'll need the RAID
array to perform block re-arrangement. Imagine how long it's going to take if it involves 20
hard drives.
However, we must also deliver the usable storage capacity according to what has been
committed to the customer. If the initial hardware sizing was done for 1TB usable with 2 arrays
of RAID-5, creating 4 RAID groups will reduce the usable storage capacity.

2. Having more than one RAID type in a storage subsystem.

There are a few types of database files with different I/O requirements. For example the DBA
can classify them like the following:

SYSTEM tablespace's datafiles : not frequently accessed : low I/O

UNDO tablespace's datafiles : medium access frequency : medium I/O
ONLINE REDO files : medium access frequency : medium I/O
TEMP tablespaces' datafiles : medium access frequency : medium I/O
DATA tablespaces' datafiles : very frequently accessed: high I/O
INDEX tablespaces' datafiles : very frequently accessed: high I/O

Based on this, the storage administrator can decide which RAID should be created. For
example:
RAID-5 for SYSTEM, UNDO, TEMP and ONLINE REDO
RAID-10 (1+0) for DATA and INDEX
So, the storage administrator will have to create at least 2 RAID groups.

Note: RAID-10 is the best choice for tablespaces with very high I/O requirement as it doesn't
involve parity calculating, thus speeding up the I/O.

By now you might be thinking that it's going to be such a waste to just have a few datafiles on
one LUN. Well, it is possible to share LUNs or array among the tablespaces, but remember the
following tips:

a. The INDEX and DATA tablespaces for the same table shouldn't reside on the same RAID
group.
 b. SYSTEM tablespace can reside on the same RAID group as INDEX.
c. ONLINE REDO files can reside on the same RAID group as TEMP.
d. UNDO tablespace can reside on the same RAID group as SYSTEM, TEMP or ONLINE REDO files.
e. In huge databases, you will probably have many big LUNs for DATA, they could occupy one or
two RAID groups.

3. Have at least one global hot spare disk in the storage subsystem.
As much as you drag the feeling of paying your monthly insurance premium installment, that's
exactly how customer felt looking at the global hot spare disk drives in the enclosure.
However, it is still a requirement. There maybe a 146 GB of unused space in the storage
subsystem all of the time, but that 146 GB could save an organisation millions of dollars worth
of database transactions during disk failures.

4. Buy disks with smaller capacity, but in larger quantity.

Regardless of how sophisticated the storage subsystem you have, when it comes to I/O we have
to accept the fact that we are still depending on the mechanical ability of each hard disk, that
is the drive spindles.
Having more disks means we're having more spindles that will do the actual I/O works. Big
capacity disks are more suitable for multimedia and file sharing applications where contigous
read/writes are performed. For most databases (be it OLTP or datawarehouse type), there will
be a very high rate of random read/writes.

Well, it was a fun and fruitful discussion we had that day. I also had a lot of fun documenting
this. You may have heard other tips or found out that things have been working fine otherwise.
It would be even a lot more fun if you could share them with me here.

Previous Entry / Next Entry

More on Vendors
Oracle Agile Product Lifecycle Management, Oracle Database, UNIX OS, Oracle Development,
Oracle PeopleSoft Workforce Planning Solutions,

Related White Papers

 Performance Automation in DB2 LUW

 A Technical Overview of HP Oracle Exadata Storage Server
 A Technical Overview of HP Oracle Exadata Storage Server

More White Papers...