InnoDB tablespace management

By default InnoDB uses a shared tablespace for all tables that consists of one or
more files. These
files are used to store metadata for all the InnoDB tables across all databases on
the server, and
are usually referred to as ibdata files, because the default name for the first
file is ibdata1.
These ibdata files are, by default, stored in the datadir directory.

By default, ibdata files also contain all the data and indexes for all the InnoDB
tables. When
an InnoDB table is defragmented in a shared tablespace configuration, the ibdata
files will not
shrink, even though the data and indexes are successfully defragmented. The good
news is that
the space is not lost InnoDB will add that space to its pool of free space and
put new rows
in it. The amount of InnoDB free space that is reported in the TABLE_COMMENT field
of the
INFORMATION_SCHEMA.TABLES system view and the Comment field of SHOW TABLE STATUS
for a defragmented InnoDB table will increase. The bad news is that the operating
system
cannot reclaim that disk space and use it for other purposes if needed. The space
is used by the
tablespace files.

A second option is to configure a per-table tablespace using the

innodb_file_per_table
option which, as the name suggests, stores the indexes and data for each table in a
separate file,
named with the table name and an extension of ibd. There is still some metadata
stored in the
ibdata files but the end result of using innodb_file_per_table is that the overall
system
can be more manageable. When an InnoDB table is defragmented (using OPTIMIZE TABLE,
for
example), the associated .ibd file will shrink and disk space will automatically be
reclaimed.

Defragmenting of InnoDB tablespace, regardless of configuration, is typically done

through the
previously mentioned OPTIMIZE TABLE command. This command is discussed in the
Table
Maintenance Commands section of Chapter 4. An ALTER TABLE that rebuilds the
entire table
will also defragment the table.

Working with ibdata files

By default ibdata files are located in datadir. Any options for the InnoDB storage
engine
should be listed in the [mysqld] directive of the configuration file. The following
example
shows two ibdata files in the default data directory. One file is a gigabyte in
size, and the
second is configured to begin at a gigabyte in size but grow larger as needed:

innodb_data_file_path=ibdata1:1024M;ibdata2:1024M:autoextend
The innodb_data_home_dir option is used to specify the location of the ibdata files
explicitly,
to store them somewhere other than datadir. In the following example two ibdata
files
are in the /data directory:

innodb_data_home_dir = /data
innodb_data_file_path=ibdata1:1024M;ibdata2:1024M:autoextend

A third option would be to have the ibdata files located in different directories
from each
other. Using this method the files can be placed on separate hard drives for
performance gains
or because of low storage space. Here is how it is done:

innodb_data_home_dir =
innodb_data_file_path=/d1/ibdata1:1024M;/d2/ibdata2:1024M:autoextend

Adding an additional ibdata file to a shared tablespace

A single ibdata file can grow very large (over 300 gigabytes). In these cases, it
may be desirable
to add a new tablespace. Maybe the partition where the ibdata file is located is
running out of
space. The procedure to add an additional ibdata file is straightforward:

Configure the innodb_data_home_dir (if necessary) and the innodb_data_file_

path variable. If the current configuration is the following:

innodb_data_file_path=ibdata1:1024M:autoextend

Determine the actual size of ibdata1. This must be done to the nearest megabyte.
On
Unix-based systems a simple ls lh command executed in the data directory will
return
the current size in megabytes.
Once this is known the new innodb_data_file_path can be written. If the size of
the
ibdata1 file is 1824 megabytes and the second ibdata is to begin at 2 gigabytes in
size,
the new line would look this:

innodb_data_file_path=ibdata1:1824M;ibdata2:2G:autoextend

The autoextend option on the innodb_data_file path allows a file to grow as

needed.
Once this change to innodb_data_file_path is completed a restart of mysqld is
necessary.
It will take some time to initialize the ibdata2 file, but after a minute or two
the
file should be present in the data directory.

Adding this file is not very complicated. The previous section shows how you can
configure
your innodb_data_file_path

NB:
Do not think that just because an additional ibdata file is added that data will be
balanced across the files. If there were only 400 megabytes of data in ibdata1 (out
of a configured 1 gigabyte of space) when ibdata2 was added no table data will be
written to
ibdata2 until ibdata1 is completely full. Similarly, when ibdata1 is full, new data
is written
to ibdata2, and tables can be thus split across data files.

Removing or reorganizing shared tablespace files

If you overestimated the size of the shared tablespace or it has grown much larger
than your
data (for example, if you have dropped some larger tables or your data is heavily
fragmented),
you may wish to remove an ibdata file from a shared tablespace. You may also wish
to
reorganize the current ibdata files for example, if one ibdata file is 100Mb and
another is
500 Mb, you may wish to have each ibdata file be 300 Mb. Unfortunately, this is not
as easy
as adding an ibdata file. To remove or reorganize shared tablespace files:

Perform a logical export (using a logical export tool such as mysqldump) of all
the InnoDB
tables.
Shut down mysqld.
Change the configuration options as desired to remove or reorganize a tablespace.
Move the existing ibdata files, ib_logfile log files, and .frm table definition
files to a
backup location.
Restart mysqld. The new ibdata file(s) will take some time to initialize.
Import the InnoDB tables.
After a successful import, you can delete the backup ibdata, ib_logfile, and .frm
files. Should you need to roll back to the previous ibdata files, shut down mysqld,
change the configuration options back to the previous values, and move the ibdata,
ib_logfile, and .frm files back.

Moving from a shared tablespace file to per-table tablespace

If you decide to move from a shared tablespace to using per-table tablespaces the
process is somewhat
involved. Moving to a per-table tablespace allows you to easily see on-disk how
large the data
and indexes for the tables are. If a tablespace is significantly larger than the
metadata from SHOW
TABLE STATUS and the TABLES system view of the INFORMATION_SCHEMA database, it is
very
likely the table is fragmented. As previously stated, with per-table tablespaces,
defragmentation
reclaims disk space; storing all the data and indexes in the shared tablespace does
not.
Converting your database to use per-table data and index files is very similar to
reorganizing or
removing shared tablespace files:

Perform a logical export (using a logical export tool such as mysqldump) of all
the InnoDB
tables.
Shut down mysqld.
Edit the configuration file and add innodb_file_per_table in the [mysqld]
directive.
Though it is still necessary to retain an ibdata1 file it will remain relatively
small
compared to the overall amount of data in InnoDB tables. This is because it will
only contain
metadata about the tables. Because of this you will almost certainly want to edit
your
innodb_data_file_path. Something like this will probably be more appropriate:
innodb_data_file_path=ibdata1:256M:autoextend
Move the current InnoDB table format files (.frm files), ibdata file(s), and the
InnoDB
log files (ib_logfile) to a backup location until you are assured everything works
properly.
Restart mysqld. At this point a new central tablespace and InnoDB log files will
be
initialized.
Verify that you can create an InnoDB table and that an individual .ibd data file
is created
in the database directory.
Restore your files from the backup. Once this is done the InnoDB tables are all
stored in
per-table tablespace.
After a successful import, you can delete the backup ibdata and .frm files.
Should you
need to roll back to the previous ibdata files, shut down mysqld, change the
configuration
options back to the previous values, and move the ibdata and .frm files back.

InnoDB log files and crash recovery

The unsung heroes of InnoDB are the log files. They are what makes InnoDB automatic
crash recovery possible.

Database administrators of other DBMSs may be familiar with the concept of a

redo log.
When data is changed, affected data pages are changed in the innodb_buffer_pool.
Then,
the change is written to the redo log, which in mysqld is stored in the InnoDB log
files
(ib_logfile0 and ib_logfile1). The pages are marked as dirty, because they have
been
changed, and eventually get flushed and written to disk.
If mysqld crashes, there may be data that is changed that has not been written to
disk. Those
data pages were marked as dirty in the innodb_buffer_pool, but the
innodb_buffer_pool
is an in-memory cache, so after a crash the information inside the buffer pool no
longer exists.
However, all changes were written to the redo log. On crash recovery, mysqld can
read the redo
log (InnoDB log files) and apply any changes that were not written to disk.
Some of the different parameters for the InnoDB log files and their ramifications
are:
innodb_log_files_in_group is set with a default of 2. The log files are written
in a
circular manner ib_logfile0 is written first, and when it is full of changed data,
ib_logfile1 will be written to. The log files are initialized to their full size
when
mysqld starts up, if they are not already initialized.
innodb_log_file_size is the size of each log file in the log group. The total,
combined
size of all the log files has to be less than 4 Gb (according to the MySQL
Manualthis has
nothing to do with the size of files an operating system can handle). Because the
log files
contain changes in the buffer pool that have not been written to disk, the total,
combined
size of all the log files should not be more than the innodb_buffer_pool_size. If
so,
the log files are wasting space that will never be used.
If all the log files in the group are full of changes that have not been written
to disk,
mysqld will start to flush dirty pages from the InnoDB buffer pool, writing the
changes to
disk. If the log files are small, changes will be written to disk more often, which
can cause
more disk I/O. When InnoDB does a crash recovery, it reads the log files. If the
log files
are large, it will take longer to recover from a crash.
If innodb_fast_shutdown is set to 0, the log files are purged when mysqld
shuts downlarger files mean a longer shutdown time. The default for innodb_
fast_shutdown is 1, which means that the log files are not purged before a
shutdown. A value is 2 simulates a crash, and at the next startup InnoDB will do a
crash
recovery. This can be useful when testing to see approximately how long InnoDB
takes to
recover from a crash, to determine if the size of your log files is too large.
innodb_flush_log_at_trx_commit controls how often the log files are written to.
A value of 0 causes the log files to be written and flushed to disk once per
second. The
default is 1, which causes the log buffer to be written and flushed to disk after
every transaction
commit. The value can also be set to 2, which causes the log buffer to be written
after every transaction commit and flushes the log files to disk once per second. A
value of
2 means that MySQL might think that some changes are written to the log file, but
do not
persist in the log file after an operating system crash, because the log file was
not flushed
to disk before a crash.
Note that some file systems are not honest about flushing to disk, so even though
you may
have the default value of 1, your system may be acting as if it has a value of 2.
Setting this
parameter to 2 means that there will be less I/O, at the cost of not being able to
recover
data from a crash.
innodb_flush_method changes how InnoDB opens and flushes data and log files. See
Chapter 10 for more information; the end result is a tradeoff in I/O performance
versus
whether or not an operating system crash would leave the InnoDB log files in an
inconsistent
state.
innodb_log_buffer_size is the write buffer for InnoDB log files. The larger the
buffer
is, the less often the log files are written to. This can save I/O.