A Checklist for Preventing

Common But Deadly

MySQL Problems
Part 1

A Percona White Paper

October 2, 2014

Contents
Introduction ................................................................................................................ 3
The Busy MySQL DBA .............................................................................................. 3
The Impact on Your Business .................................................................................. 3
The Common but Deadly MySQL Problems ................................................................ 4
Out of Date MySQL Version .................................................................................... 4
Inadequate MySQL Configuration Settings ............................................................. 5
Unmindful Deployment Practices ........................................................................... 6
How Percona Can Help You......................................................................................... 7
About Percona ............................................................................................................ 8

October 7, 2014
Percona. All rights reserved.

Introduction
This is the first of a series of three white papers intended to give busy DBAs and their
management an understanding of some of the most common MySQL implementation,
performance, and recovery problems that can be minimized or prevented by proactive
and diligent MySQL administration best practices. The issues covered in this white paper
are:

Out of Date MySQL Version

Inadequate MySQL Configuration Settings
Unmindful Deployment Practices

This information is based on collected advice from the Percona Support, Consulting, and
Managed Services teams. This group includes top experts in MySQL operations who
have helped thousands of clients over the past 8 years.

The Busy MySQL DBA

Busy MySQL DBAs often fall into one of three categories:

You may be filling the role of your organizations MySQL DBA because there is
no one else available. You may have limited DBA experience and might be an IT
professional, a SysAdmin, the CTO, or on the Engineering and Development
staff.
You may be a DBA by profession but be overloaded due to lack of resources in
your organization. Your time might be divided between database operations
and other tasks such as application development. You may be addressing issues
reactively or when time allows, hopefully before disaster strikes.
You may be a DBA transitioning into a more strategic role in which the handling
of common operational tasks is no longer your main focus or desire. In this
capacity, common but necessary database administrative tasks are performed
ad hoc, as time allows, or perhaps not at all.

The Impact on Your Business

The busy DBA and their managers must also consider the problems caused by part-time
or reactive MySQL administration and the impact those issues may have on your overall
business. It is managements responsibility to maintain the resources necessary to
manage their organizations MySQL environments. A lack of needed resources inevitably
leads to:

October 7, 2014
Percona. All rights reserved.

Unacceptable downtime
Unnecessary costs
Potential malicious attacks

Frustrated users
Lost business

Percona services can help. Percona Consulting and Support services for MySQL offer
customized help and immediate assistance from MySQL experts. Percona Managed
Services for MySQL can take over all of your organizations MySQL operational tasks,
letting you concentrate on other projects.

The Common but Deadly MySQL Problems

We will now take a look at some of the most common MySQL implementation,
performance, and recovery problems.

Out of Date MySQL Version

You may adopt or inherit an environment where the MySQL version is out of date. There
are many scenarios, such as relying on the version available from your Linux package
manager, where you or your team may be unknowingly using a legacy version of MySQL.
Remaining with a legacy version can lead to risks:

Security you can be more vulnerable to security exploits with an old version
of MySQL which may increase the surface area for potential attacks
Performance you will usually experience better performance with the latest
version of MySQL
Support your version might be so old that it is no longer supported by your
vendor or it is difficult to find tools or a consultant to help

For example, Percona had an online consumer business client that was running MySQL
5.1. They were experiencing edge case query performance issues where index hints
were unable to provide relief from poor response time. The problem was critically
impacting sales due to frustrated users. Testing the same queries on Percona Server 5.6
proved that the performance enhancements of the Percona Server 5.6 query optimizer
reduced their overall load by around 40%. Upgrading to Percona Server 5.6 provided
capacity for growth on their existing systems, avoided costly hardware purchases, and
delivered better response times and a superior customer experience.
Minor version upgrades may be something you can handle with little risk. However, for
a major upgrade, you may want to consider using outside help that has experience
dealing with a multitude of upgrade scenarios. Percona uses a carefully managed
approach to upgrades which combines unique tools such as Percona Toolkit with our
decades of experience. For our Managed Services clients, the team works hand-in-hand
with our clients to test, plan, execute, and re-test so there are no surprises when
moving from a legacy version of MySQL to the latest release.

October 7, 2014
Percona. All rights reserved.

Related Percona Resources

There are several Percona resources on the Percona website that you can use to
help better prepare for a version upgrade including:

Webinar: Upgrading to MySQL 5.6: Best Practices given by Peter Zaitsev,

Percona CEO and Founder
Technical Presentation: Upgrading to MySQL 5.6: Best Practices, by Peter
Zaitsev, Percona CEO and Founder
Blog Post on the Percona MySQL Performance Blog: MySQL Upgrade Best
Practices, by Percona Software Engineer Muhammad Irfan
Blog Post on the Percona MySQL Performance Blog: Doing a rolling upgrade of
Percona XtraDB Cluster from 5.5 to 5.6, by Percona Managing Consultant Jay
Janssen

Inadequate MySQL Configuration Settings

MySQL can run right out of the box. While this makes it easy to install and evaluate
MySQL at a small scale, many of the default MySQL configuration values are not
appropriate for a large production environment.
You might use the default MySQL configuration values because you do not have the
time or experience to tune them appropriately. Tuning MySQL without understanding
how to measure the results can cause more harm than good. This can lead to:

A highly inefficient usage of system resources

A system that does not scale well
Unpredictable behavior for application end users

Here are some examples of how default settings can impact your system:

The overhead of fully synchronous disk writes may be unnecessarily throttling

your sites rate of write traffic
The size of the InnoDB buffer pool and transaction log may be keeping your
server from fully using its hardware capacity
Using the outdated MyISAM storage engine (instead of the preferred InnoDB
engine) risks data loss and costly table repairs

The reality is that organizations can often use their database servers a lot longer than
they think. Adding more infrastructure should only occur after MySQL is tuned
appropriately. A new Percona client in the ecommerce industry asked us to investigate
performance issues because their current architecture couldn't cope with their query
workload. Upon investigation, we found that the variables configured to sync the

October 7, 2014
Percona. All rights reserved.

replication metadata were all set to 1 and would sync after every event written. This
generated a huge amount of fsyncs on their environment and put significant pressure
on their storage. Changing these variables to values that were more suitable to their
application cut storage usage by over 50% and instantly resolved their problems with
query performance.
Percona approaches MySQL configuration tuning by measuring performance, identifying
bottlenecks, applying tuning changes, and measuring the results. This approach provides
a record of the reasons behind tuning changes so they can be applied with confidence.
Related Percona Resources
There are several resources on the Percona website that you can use to help you better
configure MySQL including:

Webinar: MySQL 5.6 Configuration Optimization given by Peter Zaitsev,

Percona CEO and Founder
Technical Presentation: MySQL 5.6 Configuration Optimization, by Peter
Zaitsev, Percona CEO and Founder
The MySQL Configuration Wizard, a free online tool by Percona
Blog Post on the Percona MySQL Performance Blog: Measure the impact of
MySQL configuration changes with Percona Cloud Tools, by Percona Principal
Architect Alexander Rubin

Unmindful Deployment Practices

Updating the code in your production application is a frequent necessity. This may occur
every few weeks or even every few minutes. Sometimes, deploying new code requires
changes to the database for schema updates, adding indexes, or deleting rows.
Making changes to a database, such as deleting many records, can cause problems if not
thoughtfully planned and handled. For example, many database design and
configuration activities can block all users of an application for unknown periods of time.
Unplanned downtime can occur when operations that require various types of locks run
in the production environment at the wrong time or using the wrong approach. MyISAM
can be a perpetrator in this context. However, even with a dataset entirely using
InnoDB, some operations can demand global locking. For example, executing ALTER
table on a large scale can require a complete table rebuild. This action will block any
reads and writes on the affected tables and can take a significant amount of time to
complete on a multi-gigabyte scale.
There are tricks and tools that Percona uses to mitigate the effect of an ALTER. For
example, you might be able to perform the actions on a slave and then promote it to

October 7, 2014
Percona. All rights reserved.

become the master or you might use pt-online-schema-change from Percona Toolkit.
You might even be able to directly run ALTER on small tables or run the larger ALTERs
during off hours. The point is to pay careful attention in each deployment, not only the
code changes but also how the databases are impacted.
Percona works with clients to thoroughly inspect SQL to ensure that there are no hidden
surprises. When we have identified database changes in a clients code deployment that
could impact performance, we have found that gently administering the needed
changes over an elongated period of time reduces risk. In addition, we have deployment
tools that enable us to perform dry runs before actually executing the changes on the
data. Running tests and manually checking changes has saved our clients from potential
disasters in the past.
Related Percona Resources
There are several resources located on the Percona website that you can use to prepare
to deploy new code including:

Webinar: Architecture and Design of MySQL Powered Applications, given by

Peter Zaitsev, Percona CEO and Founder
Blog Post on the Percona MySQL Performance Blog: Integrating pt-onlineschema-change with a Scripted Deployment, by Percona Managing Consultant
Jay Janssen
Blog Post on the Percona MySQL Performance Blog: How to monitor ALTER
TABLE progress in MySQL, by Percona Support Engineer Nilnandan Joshi

How Percona Can Help You

Managing your organizations MySQL operations requires in-depth knowledge of
potential issues plus diligent, dedicated practice. It is a vital competency if your business
depends on MySQL. Being aware of the issues above will help protect your
organizations MySQL-based applications. It will also significantly enhance both
performance and scalability to deliver a better user experience.
Keeping abreast of best practices for preventing these dangers takes time and expertise.
If you are your organizations busy MySQL DBA and you worry about the level of
attention being given to your databases, Percona can help.
For the over-extended DBA who requires expert help to optimize and provide additional
support for MySQL, Percona can help. Percona Support services are accessible 24x7
online or by phone to ensure that your MySQL installation is running optimally. We can
also provide onsite or remote Percona Consulting for current or planned projects, or in
emergency situations. Every engagement is unique and we will work with you to create
the most effective solution for your business.

October 7, 2014
Percona. All rights reserved.

If you dont have the resources to adequately manage your MySQL operations, you can
outsource the full operational control of your MySQL servers to Percona. Percona is a
leader in MySQL managed services, offering both outsourced database administration
services and backup and recovery services. We can evaluate your current MySQL
operations, your goals, and your resources to collaboratively implement a solution that
will:

Prevent common operational problems

Keep your MySQL performing optimally
Lower your database operations and administration costs

Organizations that rely on Percona Managed Services benefit from our deep
operational expertise, 24x7x365 coverage from our worldwide team of experts, SLA
commitments, and unlimited incidents without additional hourly fees.
Percona Managed Services can support your existing database infrastructure whether it
is hosted on premise or at a colocation facility or you purchase services from a cloud
provider or database-as-a-service provider.
The Percona Remote DBA service provides full operational management of your MySQL
servers. We apply best practices to configure your servers, set up monitoring and alerts,
configure and manage your backups, respond to alerts 24x7x365, perform routine
operational tasks, and help you plan for future growth. Our Remote DBA service allows
you to concentrate on enhancing your applications while Percona takes care of database
operations.
The Percona Backup Service ensures that your backups are properly configured, run
successfully on schedule, and that any backup issues or production data recovery needs
are handled promptly by expert MySQL technicians. The system uses proprietary tools
and methods which reduce the cost and complexity of MySQL backup and recovery
through advanced data storage management. It also provides full visibility into the
status of all of your backups through a convenient online portal.
To learn about any of our services, please contact us at (208) 473-2904 or +44 208 1330309 in Europe or email us at sales@percona.com.

October 7, 2014
Percona. All rights reserved.

About Percona
Percona has made MySQL faster and more reliable for over
2,000 clients worldwide since 2006. Percona provides
enterprise grade MySQL Support, Consulting, Training, and
Managed Services. Percona's founders authored the
definitive book High Performance MySQL from O'Reilly
Press and the widely read Percona MySQL Performance
Blog. Percona also develops software for MySQL users,
including Percona Server, Percona XtraBackup, Percona
XtraDB Cluster, and Percona Toolkit. The popular Percona
Live conferences draw attendees and acclaimed speakers
from around the world. For more information, visit
www.percona.com.

October 7, 2014
Percona. All rights reserved.