PostgreSQL High Performance Cookbook

Table of Contents

PostgreSQL High Performance Cookbook

Credits

About the Authors

About the Reviewers

www.PacktPub.com

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it...

How it works...

There's more...

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Database Benchmarking

Introduction

CPU benchmarking

Getting ready

How to do it...

Phoronix

sysbench

How it works...

Phoronix

sysbench

Memory benchmarking

Getting ready

How to do it...

Phoronix

tmpfs

Write test

Read test

How it works...

Disk benchmarking

Getting ready

How to do it...

Phoronix

bonnie++

How it works...

bonnie++

Performing a seek rate test

Getting ready

How to do it...

How it works...

Working with the fsync commit rate

Getting ready

How to do it...

How it works...

Checking IOPS

Getting ready

How to do it...

Sequential mixed read and write

Random mixed read and write

How it works...

Storage sizing

Getting ready

How to do it...

How it works...

Discussing RAID levels

Getting ready

How to do it...

How it works...

RAID 0

RAID 1

RAID 5

RAID 6

RAID 10

Configuring pgbench

Getting ready

How to do it...

How it works...

Running read/write pgbench test cases

Getting ready

How to do it...

Read-only

Write-only

How it works...

2. Server Configuration and Control

Introduction

Starting the server manually

Getting ready

How to do it...

How it works...

Stopping the server quickly

Getting ready

How to do it...

How it works...

Stopping the server in an emergency

How to do it...

How it works...

Reloading server configuration

Getting ready

How to do it...

How it works...

Restarting the database server quickly

How to do it...

How it works...

Tuning connection-related parameters

How to do it...

How it works...

Tuning query-related parameters

How to do it...

How it works...

Tuning logging-related parameters

How to do it...

How it works...

3. Device Optimization

Introduction

Understanding memory units in PostgreSQL

Getting ready

How to do it...

shared_buffers

temp_buffers

work_mem

maintenance_work_mem

wal_buffers

max_stack_depth

effective_cache_size

How it works...

Handling Linux/Unix memory parameters

Getting ready

How to do it...

kernel.shmmax

kernel.shmall

kernel.shmmni

vm.swappiness

vm.overcommit_memory

vm.overcommit_ratio

vm.dirty_background_ratio

vm.dirty_ratio

How it works...

CPU scheduling parameters

Getting ready

How to do it...

kernel.sched_autogroup_enabled

kernel.sched_min_granularity_ns

kernel.sched_latency_ns

kernel.sched_wakeup_granularity_ns

kernel.sched_migration_cost_ns

How it works...

Disk tuning parameters

Getting ready

How to do it...

CFQ

noop

Deadline

How it works...

Identifying checkpoint overhead

Getting ready

How to do it...

How it works...

Analyzing buffer cache contents

Getting ready

How to do it...

How it works...

4. Monitoring Server Performance

Introduction

Monitoring CPU usage

Getting ready

How to do it...

How it works...

Monitoring paging and swapping

Getting ready

How to do it...

How it works...

Tracking CPU consuming processes

Getting ready

How to do it...

How it works...

Monitoring CPU load

How to do it...

How it works...

Identifying CPU bottlenecks

How to do it...

How it works...

Identifying disk I/O bottlenecks

How to do it...

How it works...

Monitoring system load

How to do it...

How it works...

Tracking historical CPU usage

Getting ready

How to do it...

How it works...

There's more...

Tracking historical memory usage

Getting ready

How to do it...

How it works...

Monitoring disk space

How to do it...

How it works...

Monitoring network status

Getting ready

How to do it...

How it works...

5. Connection Pooling and Database Partitioning

Introduction

Installing pgpool-II

Getting Ready

How to do it...

How it works...

Configuring pgpool and testing the setup

Getting ready

How to do it...

How it works...

There's more...

Installing PgBouncer

Getting ready

How to do it...

How it works...

Connection pooling using PgBouncer

Getting ready

How to do it...

How it works...

There's more...

Managing PgBouncer

Getting ready

How to do it...

How it works...

Implementing partitioning

Getting ready

How to do it...

How it works...

Managing partitions

Getting ready

How to do it...

How it works...

Installing PL/Proxy

How to do it...

How it works...

Partitioning with PL/Proxy

Getting ready

How to do it...

How it works...

6. High Availability and Replication

Introduction

Setting up hot streaming replication

Getting ready

How to do it...

How it works...

Replication using Slony

Getting ready

How to do it...

How it works...

Replication using Londiste

Getting ready

How to do it...

How it works...

Replication using Bucardo

Getting ready

How to do it...

How it works...

Replication using DRBD

Getting ready

How to do it...

How it works...

Setting up a Postgres-XL cluster

Getting ready

How to do it...

7. Working with Third-Party Replication Management Utilities

Introduction

Setting up Barman

Getting ready

How to do it...

How it works...

Backup and recovery using Barman

Getting ready

How to do it...

How it works...

Setting up OmniPITR

Getting ready

How to do it...

How it works...

WAL management with OmniPITR

Getting ready

How to do it...

How it works...

Setting up repmgr

How to do it...

How it works...

Using repmgr to create replica

How to do it...

How it works...

Setting up walctl

How to do it...

How it works...

Using walctl to create replica

How to do it...

How it works...

8. Database Monitoring and Performance

Introduction

Checking active sessions

Getting ready

How to do it...

How it works...

There's more...

Finding out what the users are currently running

Getting ready

How to do it...

How it works...

Finding blocked sessions

Getting ready

How to do it...

How it works...

There's more...

Transactional locks

Table level locks

Prepared transaction locks

Usage of SKIP LOCKED

Dealing with deadlocks

Getting ready

How to do it...

Using FOR UPDATE

Advisory locks

Table access statistics

Getting ready

How to do it...

pg_stat_user_tables

pg_statio_user_tables

How it works...

Logging slow statements

Getting ready

How to do it...

How it works...

Determining disk usage

Getting ready

How to do it...

Column size

Relation size

Database size

Tablespace size

How it works...

Preventing page corruption

Getting ready

How to do it...

How it works...

There's more...

Routine reindexing

Getting ready

How to do it...

How it works...

There's more...

Generating planner statistics

Getting ready

How to do it...

How it works...

null_frac

avg_width

n_distinct

most_common_vals

most_common_freqs

histogram_bounds

correlation

most_common_elems

most_common_elem_freqs

Tuning with background writer statistics

Getting ready

How to do it...

How it works...

checkpoints_timed

checkpoints_req

checkpoint_write_time

checkpoint_sync_time

buffers_checkpoint

buffers_clean

max_written_clean

buffers_backend

buffers_backend_fsync

buffers_alloc

stats_reset

There's more...

9. Vacuum Internals

Introduction

Dealing with bloating tables and indexes

Getting ready

How to do it...

How it works...

Vacuum and autovacuum

Getting ready

How to do it...

How it works...

xmin

xmax

Autovacuum

Freezing and transaction ID wraparound

Getting ready

How to do it...

How it works...

Fixing transaction ID wraparound

Preventing transaction ID wraparound

Monitoring vacuum progress

Getting ready

How to do it...

How it works...

Control bloat using transaction age

Getting ready

How to do it...

How it works...

10. Data Migration from Other Databases to PostgreSQL and Upgrading the PostgreSQL Cluster

Introduction

Using pg_dump to upgrade data

Getting ready

How to do it...

How it works...

Using the pg_upgrade utility for version upgrade

Getting ready

How to do it...

How it works...

Replicating data from other databases to PostgreSQL using Goldengate

Getting ready

How to do it...

How it works...

There's more...

11. Query Optimization

Introduction

Using sample data sets

Getting ready

How to do it…

How it works…

Timing overhead

Getting ready

How to do it…

How it works…

Studying hot and cold cache behavior

Getting ready

How to do it…

Cold cache

Hot cache

How it works…

Clearing the cache

Getting ready

How to do it…

How it works…

There's more…

Query plan node structure

Getting ready

How to do it…

How it works…

Generating an explain plan

Getting ready

How to do it…

How it works…

There's more…

Computing basic cost

Getting ready

How to do it…

How it works…

Sequential scan

Index scan

Phase I

Phase II

Running sequential scans

Getting ready

How to do it…

How it works…

Running bitmap heap and index scan

Getting ready

How to do it…

How it works…

Aggregate and hash aggregate

Getting ready

How to do it…

Aggregate

Hash aggregate

How it works…

Aggregate

State transition functions

Hash aggregates

Running CTE scan

Getting ready

How to do it…

How it works…

CTE scan

Recursive union

Nesting loops

Getting ready

How to do it…

How it works…

Phase I

Phase II

Working with hash and merge join

Getting ready

How to do it…

Hash join

Merge join

How it works…

Hash join

Phase I

Phase II

Merge join

Phase I

Phase II

Grouping

Getting ready

How to do it…

How it works…

Working with set operations

Getting ready

How to do it…

How it works…

INTERSECT

EXCEPT

UNION

Working on semi and anti joins

Getting ready

How to do it…

How it works…

12. Database Indexing

Introduction

Measuring query and index block statistics

Getting ready

How to do it...

How it works...

idx_scan

idx_tup_read

idx_tup_fetch

idx_blks_read

idx_blks_hit

Index lookup

Getting ready

How to do it...

How it works...

Index scan

Bitmap heap scan

Index only scan

Comparing indexed scans and sequential scans

Getting ready

How to do it...

How it works...

Sequential scan

Index scan

Index only scan

Bitmap heap scan

There's more...

Clustering against an index

Getting ready

How to do it...

How it works...

There's more...

Concurrent indexes

Getting ready

How to do it...

How it works...

Combined indexes

Getting ready

How to do it...

How it works...

Partial indexes

Getting ready

How to do it...

How it works...

Finding unused indexes

Getting ready

How to do it...

How it works...

Forcing a query to use an index

Getting ready

How to do it...

How it works...

Detecting a missing index

Getting ready

How to do it...

How it works...

PostgreSQL High Performance Cookbook

PostgreSQL High Performance Cookbook

Copyright © 2017 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: March 2017

Production reference: 1240317

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham 

B3 2PB, UK.

ISBN 978-1-78528-433-5

www.packtpub.com

Credits

About the Authors

Chitij Chauhan currently works as a senior database administrator at an IT-based MNC in Chandigarh. He has over 10 years of work experience in the field of database and system administration, with specialization in MySQL clustering, PostgreSQL, Greenplum, Informix DB2, SQL Server 2008, Sybase, and Oracle. He is a leading expert in the area of database security, with expertise in database security products such as IBM InfoSphere Guardium, Oracle Database Vault, and Imperva.

Dinesh Kumar is an enthusiastic open source developer and has written several open source tools for PostgreSQL. He recently announced pgBucket, a brand new job scheduler for PostgreSQL. He is also a frequent blogger at manojadinesh.blogpsot.com, where he talks more about PostgreSQL. He is currently working as a senior database engineer in OpenSCG and building the PostgreSQL cloud operations. He has more than 6 years of experience as an Oracle and PostgreSQL database administrator and developer, and is currently focusing on PostgreSQL.

Thanks to my loving parents, Sreenivasulu and Vanamma, who raised me in a small village called Viruvuru, which is in the Nellore district of India. Thanks to my loving wife, Manoja, who enlightens my life with her wonderful support. Also, thanks to my friend Baji Shaik and coordinators, Mayur Pawanikar and Nitin Dasan, for their excellent support. Finally, thanks to every PostgreSQL contributor, author, and blogger.

About the Reviewers

Baji Shaik is a database administrator and developer. He is a co-author of PostgreSQL Development Essentials and has tech-reviewed Troubleshooting PostgreSQL by Packt Publishing. He is currently working as a database consultant at OpenSCG. He has an engineering degree in telecommunications, and had started his career as a C# and Java developer. Baji started working with databases in 2011, and over the years he has worked with Oracle, PostgreSQL, and Greenplum. His background spans a length and breadth of expertise and experience in SQL/NoSQL database technologies. He has good knowledge of automation, orchestration, and DevOps in a cloud environment. He likes to watch movies, read books, and write technical blogs. He also loves to spend time with family. Baji is a certified PostgreSQL professional.

Feng Tan is from China. His nickname is Francs. He was a PostgreSQL DBA at SkyMobi (NASDAQ: MOBI) for more than 5 years, where he was maintaining more than 100 PostgreSQL instances. He gave presentations at the China PostgreSQL conference on topics such as Oracle VS PostgreSQL and PostgreSQL 9.4 new features.

Feng Tan likes to share PostgreSQL technology in his blog at http://francs3.blog.163.com/. He is also one of the translators of PostgreSQL 9 Administration Cookbook Chinese Edition.

Currently, he serves as the open source database administrator at China Mobile Group Zhejiang Co. Ltd.

www.PacktPub.com

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www.packtpub.com/mapt