Best Practices for Deploying MySQL on Solaris

Franz Haberhauer
Chief Technologist & Principal Engineer Sun Microsystems GmbH

Overview of MySQL

11 million product installations 50,000 downloads each day Part of the rapidly growing open source LAMP stack Reduces Database TCO by 90% MySQL Enterprise is our Commercial Offering

#3 Most Deployed Database

66% Are Deploying MySQL or Are Planning To Deploy

Gartner 2006

Source: Gartner

Applications

Web 2.0: Wikipedia

Application Wikipedia is one of the top 10 most trafficked web sites in the world, hosting over 7 million articles in 200 languages. Key Business Benefit MySQL has enabled Wikipedias annual visitors to grow 50,000 to over 154 million. Why MySQL? Wikipedia relies upon MySQL replication to scaleout their database infrastructure and accommodate more visitors, more articles and more contributors.

MySQL's Industry-Leading Customers

Web / Web 2.0

OEM / ISV's

On Demand, SaaS, Hosting

Telecommunications

Enterprise 2.0

Open-source is powering the Web

6

MySQL Architecture

MySQL Enterprise

Added Value of MySQL Enterprise

Comprehensive offering of production support, monitoring tools, and MySQL database software Optimal performance, reliability, security, and uptime
Load balancer Query analysis Enterprise manager Advanced backup

Best practices rules Monitoring

24x7 support Hot fixes

Knowledge base

Open-source server with pluggable APIs

Service packs
9

Why MySQL on Solaris?

Don MacAskill SmugMug's CEO & Chief Geek blogs.smugmug.com/don/2008/10/10

Leveraging unique features in Solaris for MySQL: ZFS & DTrace Leveraging Solaris as stable enterprise OS on x86/x64 and SPARC systems
10

Fotolog

Application Photo sharing and social networking site. Top 50 most trafficked sites on the Internet. Key Business Benefit Scaled Fotolog site to 17 million members and 500 million photos using existing architecture Why MySQL & Sun? - Big Performance Improvement using Sun SPARC Enterprise T5120 Servers - Significantly reduced power consumption
Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community Farhan Mashraqi, Director of Database Infrastructure forge.mysql.com/wiki/MySQLConf2008ThursdayNotes

Sun Servers were our dream come true in terms of power consumption. Our investment in theT1000 and T2000 Servers have really paid off. Frank Mash, Director IT Operations

11

One OS for Both Server Scalability Architectures

Proc Proc Mem I/O I/O

Vertical Scaling / Scale Up / Data Facing

Mem

Memory Switch One OS Instance Several

... we can scale horizontally, and we can scale vertically. And we don't have to change one line of our software code to run dual-core, quad-core, or sixteen-core machines ... Jean-Luc Vaillant, CTO, LinkedIn
http://www.sun.com/customers/servers/linkedin.xml

Proc Mem I/O

Proc Mem I/O

Proc Mem I/O

OS Instances

Network Switch

Horizontal Scaling / Scale Out / Network Facing

12

Dual Release Strategy

Single Solaris Platform, Two Distributions Mechanisms

Enterprises, Business Critical Applications Unsurpassed Support > 10+ Years Long Release Cycles > 3-5 Years with Quarterly Updates

Project Indiana Developers, Early Adopter Latest Innovations Short Release Cycles > semiannually

13

OpenSolaris 2008.05 Project Indiana

Install from a single Live CD Core OS > Kernel, Libraries, Utilities Network based software package management > Repositories for packages Familiar desktop environment > GNU utilities, standard path ZFS as root filesystem
> initially on x86 / x64, on SPARC with next release
14

Development in a Goldfish Bowl New Features Moving into Solaris

Nevada Solaris Express, Community Release
Development version binary distribution for download Support options available Image Packaging System (IPS) Network-Repositories opensolaris.com Source code of current version under development opensolaris.org

every 2 weeks
snv_99, snv_100, ...

every 6 months
2008.05, 2008.11, ...

Solaris 10
Download or media kit Long term release and support cycles

every 3-9 months

5/08, 10/08, ...
15

MySQL Packages for Solaris

Via www.sun.com/mysql > GetIt Tab > MySQL Enterprise Subscription > MySQL Community Server
> 5.0.67 available SPARC 32-bit and 64-bit x86 32-bit and x64 64-bit > 32/64 bit comparison on coalface.mcslp.com minor on SPARC, significant on x86/x64 > Solaris 10 (and 9 and 8) > SVR4-Package and Tar-Archive Distributions

> Optimized builds > Currently using gcc, soon using Sun Studio 12 > Link in or preload libmtmalloc
> LD_PRELOAD_64=/usr/lib/amd64/libmtmalloc.so (for x64)
16

MySQL Community Server for Solaris

Recommended builds for optimal performance (dev.mysql.com) > Shipping versions of MySQL: 5.0.56 or later > Beta/RC versions of MySQL: 5.1.24 or later > Alpha versions of MySQL: 6.0.5 or later > If still using 4.0 (4.0.15 in Solaris 10 before 10/08) apply (security-)patch 120292-02/120293-02 for 4.0.31 Solaris Express Community Edition (SXCE) > MySQL 5.0.45 in build 87 in /usr/mysql > Optimized builds with Sun Studio > MySQL 5.1 coming soon (32/64 bit)
> MySQL 5.0 and 5.1 will coexist

> MySQL 4.x in /usr/sfw/mysql being retired

17

MySQL Packages for Solaris

Cool Stack > Most commonly used open source applications optimized for the Sun Solaris OS platform
> MySQL 5.1.25 (1.3.1), 5.0.45(1.2) > CSKamp package which in turn includes :

CSKapache2: Apache2 HTTP Server CSKphp5: PHP 5 CSKmysql32: MySQL5 32bit

> CSKmysql: MySQL5 64bit database server

> http://cooltools.sunsource.net/coolstack/ > http://opensolaris.org/os/project/webstack/

18

OpenSolaris Web Stack Project

http://opensolaris.org/os/project/webstack/
MediaWiki, Drupal, Wordpress, Joomla all deploy in minutes! Project repository at http://pkg.opensolaris.org/webstack Web Stack is well integrated and easy to integrate!

GlassFish Application Server Apache 2.2.x mod_proxy/ mod_jk Interoperability with NetBeans and GlassFish App Server, Tomcat
19

MySQL Leveraging Solaris Technologies

Service Management (SMF) Storage-Management and Filesystems > ZFS and UFS Performance analysis with DTrace Scalability with Solaris Containers

20

SMF and MySQL

MySQL on Solaris integrated in SMF see mysql(1)
>

# svcs mysql STATE disabled

STIME FMRI 17:59:52 svc:/application/database/mysql:version_50

# svccfg svc>select mysql:version_50 svc:/application/database/mysql:version_50>setprop mysql/enable_64bit=true # svcadm refresh mysql # svcadm enable mysql

Dynamically configured properties mysql/bin Installation , Default : /usr/mysql/5.0/bin mysql/data Data Directory , Default: /var/mysql/5.0/data mysql/enable_64bit Flag to select 32bit or 64bit , Default : false

21

ZFS

More than just the Next Generation of Local Filesystems

End-to End Data Integrity Immense Data Software Capacity Developer

With check-summing and copy-on-write transactions. Integrated fault management across abstraction layers.
Easier Administration

ZFS

ZFS Storage Pool

ZFS

The world's first 128-bit file system.

Huge Performance Gains

A pooled storage model no volume manager. Snapshots, clones, compression, filesystem quota and reservations included. Part of Solaris 10 and OpenSolaris now also for root file system

Architected for speed today's disk characteristics. Enables transparent use of SSDs.
22

MySQL on ZFS Best Practices

ZFS eliminates management overhead > ZFS and InnoDB-Tablespaces
>

Instead of innodb_data_file_path = /disk1/ibdata1:10G;/disk2/ibdata2:10G;/disk3/ibdata3:10G:autoextend do use innodb_data_file_path = /dbzpool/data/ibdatafile:10G:autoextend

May use compression > safe I/Os, not just disk space Transactional nature of ZFS obsoletes the need for InnoDB double write buffer More on http://dev.mysql.com/tech-resources/articles/mysql-zfs.html
23

MySQL on ZFS Best Practices

Effective MySQL Backups can be complex Snapshot a running system > May require recovery for InnoDB, Falcon, Maria Snapshot a halted system > No recovery, but requires a halt Zmanda Recovery Manager supports ZFS snapshots on Solaris

24

MySQL on ZFS Performance Best Practices

Best practices for DBMS in general > http://www.solarisinternals.com/ wiki/index.php/ZFS_Best_Practices_Guide > Section on MySQL > zfs set recordsize=8K zp1/data
> Note: Before any datafile creation! > use separate zpool for InnoDB logs with default recordsize (128KB)

> With MyISAM create separate ZFS Intent Log (ZIL) for WAL
# zpool create p <pool devices> mirror c1t10d0 c1t11d0 \

log mirror c1t8d0 c1t9d0

> before Solaris 10 10/08 use separate zpools for data and log (WAL)

> Limit ARC cache size > set zfs:zfs_arc_max

= #bytes in /etc/system
25

MySQL on UFS

from http://dev.mysql.com/tech-resources/articles/mysql-zfs.html

Avoiding Double Buffering with InnoDB

> innodb_flush_method = O_DIRECT in my.cnf

need forcedirectio for UFS MyISAM caches indexes only > Relies on UFS buffercache for data caching
26

> Before MySQL 5.0.42 and 5.1.18

More on MySQL Performance on Solaris

MySQL InnoDB Performance Tuning for the Solaris 10 OS
> http://developers.sun.com/solaris/articles/mysql_perf_tune.html

Neelakanth Nadgir's blog

> http://blogs.sun.com/realneel/category/MySQL

Sun Studio Compiler Options for MySQL on Solaris 10 x64 OS : Performance Study
> http://blogs.sun.com/krishs/entry/sun_studio_compiler_options_for1

27

DTrace
# dtrace -n 'syscall::write:entry/execname == "iotest"/{@ = quantize(arg2)}'
value 0 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 ------------- Distribution ------------- count | 0 |@ 48 | 1 |@ 26 |@ 43 |@@@ 115 |@@ 82 | 13 | 7 |@@ 73 | 5 | 16 | 7 | 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1280 | 0

a.d

b.d

D Programs lockstat(1M) ... libdtrace(3LIB) dtrace(7D)

DTrace Consumer plockstat(1M) Userland Kernel DTrace mib Provider ... fbt

dtrace(1M)

DTrace
sysinfo lockstat vminfo syscall sdt

28

DTrace Tools
Collection of useful scripts and oneliners great introduction http://www.BrendanGregg.com/dtrace.html OpenSolaris Community: DTrace (DTrace Scripts, DTrace Toolkit, DExplorer) > http://opensolaris.org/os/community/dtrace/ /usr/demo/dtrace
dappprof profiles user and library function usage
# dappprof -ceoT banner hello # # # # ###### # # # # # # CALL __fsr main banprt banner banset convert banfil TOTAL: CALL banset banfil convert banner __fsr banprt TOTAL: CALL banset convert banfil __fsr banner banprt TOTAL: ###### # ##### # # ###### # # # # # ###### # # # # # ###### # # # # #### # # # #

####

COUNT 1 1 1 1 1 5 5 15 ELAPSED 38733 150280 152113 907212 1695068 1887674 4831080 CPU 7710 9566 11931 15199 52685 776429 873520

Plot of iosnoop output iosnoop traced output from tar -cvf /dev/null /var

29

Using DTrace with MySQL

Using DTrace to observe the SQL statements on a live running MySQL database > http://blogs.sun.com/angelo/entry/observing_mysql_sql_statements_in
#!/usr/sbin/dtrace -qs pid$1::*dispatch_command*:entry { printf("%d::%s\n",tid,copyinstr(arg2)); } 11::sugarcrm 11::SET CHARACTER SET utf8 11::SET NAMES 'utf8' 11::SELECT id, name, symbol, conversion_rate FROM currencies WHERE status = 'Active' and deleted = 0 11::SELECT category, name, value FROM config 11::SELECT id FROM outbound_email WHERE type = 'system' 11::SELECT * FROM outbound_email WHERE id = '592d612d-3fd2-fa02-f067-48d0467e2da0'
30

Using DTrace with MySQL

Check out for how to use today ... > DTrace and MySQL by Ben Rockwood (Joyent Inc)
> http://en.oreilly.com/mysql2008/public/schedule/detail/1193 > requires knowledge about MySQL internals

> Best Practices for Deploying MySQL on the Solaris Platform

by Ritu Kamboj and Luojia Chen (Sun Microsystems, Inc.)

> http://en.oreilly.com/mysql2008/public/schedule/detail/806 > instrument MySQL source code

... making use of DTrace will soon become much easier

31

DTrace in MySQL
Instrumenting MySQL with DTrace statically defined probes DTrace will allow performance and trace monitoring > Monitor execution time for
> SELECT, INSERT, UPDATE, DELETE and variants > Total query execution time > Time in locks

> Lock triggers/types > Filesort exection/time > Query Cache hits/misses

MySQL 6.0 will include an extended set of probes MySQL 5.1 will have a patch for the same probes
32

DTrace Example
Comparing execution for SELECT, 50 million rows
QC Dur ms 0 0 0 0 0 0 0 0 8289 0 0 0 8290 331594 0 Matched 0 0 0 0 20 0 0 0 Changed 0 0 0 0 Query select @@version_comment limit 1 SELECT DATABASE() show databases show tables Filesort for select * from s where i > 10 order by i limit 20 0 select * from s where i > 10 order by i limit 20 0 create index si on s (i) 0 select * from s where i > 10 order by i limit 20
33

DTrace Benefits
Dtrace is passive Always available Low overhead to get the information No overhead if you dont use it Allows you to trace and monitor any time Or monitor constantly and highlight

34

Solaris Containers
Virtual application environments within a single OS instance Zones Isolation - Security - Faults - Licensing + Resource Management - CPU, Memory
192.9.9.1

Network

192.9.9.2

192.9.9.3

www

store 192.9.9.4

B2B 192.9.9.5 oltp

appserver

Global Zone
Independent Users Separate Networks Independent Storage Isolated Containers

Disk Storage and File Systems

35

Using MySQL with Solaris Containers

Combine multiple low-utilisation systems onto larger single system > Isolate applications from faults > Maintain service levels
> Fine tune resources and response times

Effectively use MySQL on systems with many cores (CMT) > Run multiple instances of MySQL on one system
> one instance per zone > easy setup

36

MySQL Cluster
Transactional database for fast, always-on access to data under high throughput conditions > Scale-out architecture > Most often used as in-memory database
> can be configured to use disk-based data as well > uses NDB storage engine

Setting Up MySQL Cluster Software Using Solaris Zones Partitioning Technology > Hashamkha Pathan, August 2008
> http://www.sun.com/bigadmin/features/articles/mysql_cluster_zones.jsp

37

Solaris Cluster
Proven HA solution
> 10+ years in the enterprise > Wide and varied customer install base

Solaris Cluster
> Cross platform (SPARC/x64) > Deeply integrated with Solaris kernel

> > > > >

> e.g. for cluster membership management, disk fencing Shared storage, shared nothing Active-active and active-passive Failover between Solaris Containers Open sourced as Open HA Cluster http://www.sun.com/cluster
38

Using MySQL on Solaris Cluster

MySQL cluster agent available
> Probing steps include periodically executing

> connect to the MySQL instance mysqladmin command with argument ping > if instance is slave configuration check whether instance is connected to its master > verify that the probe can list all databases and tables (not contents) > function test defined test-database Create Table, Insert into Table, Update Table, Delete from Table and Drop Table > Automatic action upon failure: restart, failover > Sun Cluster Data Service for MySQL Guide for Solaris OS > http://docs.sun.com/app/docs/doc/820-2571 > Deployment and Failover Study of HA MySQL on a Solaris Cluster > http://blogs.sun.com/krishs/entry/deployment_and_failover_study_of

39

MySQL on Solaris Summary

Solaris or OpenSolaris > Optimised binaries on all platforms > Use Webstack/Coolstack for complete integrated solution MySQL integrated as a service in SMF Use ZFS to simplify disk management and backup, and help performance Use DTrace to monitor queries and anomalies Use Zones/Containers for isolation, consolidation, and scalability Use Solaris Cluster for HA
40

Best Practices for Deploying MySQL on Solaris

Franz Haberhauer
Chief Technologist & Principal Engineer Sun Microsystems GmbH Franz.Haberhauer@Sun.com sun.com/mysql