FreeBSD Mastery: Storage Essentials: IT Mastery, #4

Brief Contents

Acknowledgements

0: Disks Suck

1: FreeBSD and Disks

2: Using GUID Partition Tables

3: Using MBR Partitions

4: The Unix File System

5: Filesystem Encryption 101

6: GELI

7: GBDE

8: Swap Space

9: GEOM RAID

10: SMART

11: Complex Installation

Afterword

Detailed Contents

Acknowledgements

This book would not exist without the support, encouragement, and tolerance of the FreeBSD community over the last eighteen years. Many people have helped me understand disks, filesystems, and exactly how I destroyed my data. Sadly, I’m not man enough to trawl a decade and a half of FreeBSD mailing list archives to gather everyone’s names, but you either know who you are or you can write a Perl script to extract the relevant data.

The bank that holds my mortgage wants me to pass along their gratitude to iX Systems (http://www.ixsystems.com). I normally assemble test equipment from donations and hand-me-downs. Thanks to iX Systems’ reasonable rates, I was able to skip the recharge fire extinguisher step of my writing process.

My thanks to the people who bought this book before it was published. I especially want to thank those who bought the book early, read it, and sent me corrections. They are, in alphabetical order: Will Andrews, Lars Engels, Eli Janssen, Peter Jeremy, Ruslan Makhmatkhanov, Alexander Motin, Jens Schweikhardt, Dag-Erling Smørgrav, Jeff Russo, Romain Tartière, and Garrett Wollman.

Lastly, my gratitude to Poul-Henning Kamp. Not only did he write GEOM, he volunteered to tech review this book. For those unfamiliar with the process, tech review pretty much means educate the author about all the stuff he was either too lazy to read or too stupid to understand.

This book would suck a whole lot more without these people.

For Liz

Chapter 0: Disks Suck

You can arbitrarily replace everything in your computer—except the hard disk. Memory or mainboard dies? Throw it out, get another. Processor overdoses on Bitcoin mining and explodes? Vacuum up the debris. Slot another in. Preferably one with a bigger fan. But when a hard disk has trouble, even experienced systems administrators sweat. Did the last backup run recently enough? Has the disk problem corrupted the data? Has the data corruption persisted long enough to render all of the backups unusable?

Nothing can completely remove all concerns about disks, but understanding the tools and options for disk management minimizes your worries. This book takes you through FreeBSD’s disk management tools and filesystems. FreeBSD has a whole bunch of tools to examine disks and filesystems, and several filesystems to choose from. No filesystem or disk system is perfect, but you can choose the options that best fit your environment.

Prerequisites

This book is written for system administrators interested in FreeBSD filesystems. No previous FreeBSD experience is necessary. We will cover FreeBSD 9 and later, although much of the material is relevant for earlier FreeBSD versions.

You must know something about disk technologies. We won’t discuss the differences between SATA, SAS, PATA, EIDE, iSCSI, and their innumerable brethren. Anything I write about hardware would go obsolete more quickly than anything else in this book. I’ll mention these technologies in passing, but won’t make any specific recommendations. Just because I really like my SATA-3 setup in February 2014 doesn’t mean you should rush out and buy one in 2016, or even November 2014. Do your own hardware research or, better still, bribe a friend who already has done the work.

I’m going assume you have a FreeBSD system on hand for testing. A virtual machine suffices to explore filesystems, and might actually be preferable. Studying filesystems carries unique risks. A single command can destroy the operating system. It doesn’t matter if you enter that command from ignorance, curiosity, or malice—the system will still vanish. You should test any disk changes on your disposable system, and be prepared to reinstall the disposable system when you screw up. A snapshot of a cleanly working virtual machine greatly helps in learning disk management. The more systems administration experience you have, the less you need a test system—and the more you want a system to test changes on before you try them in production. Get a test system.

When you blow up your test system, don’t panic. As an educational exercise, attempt to recover the system. If that fails, reinstall.

An operating system’s storage systems are intimately tied to the underlying hardware. In the last half-century, storage hardware has evolved from punchcards to spinning platters to solid state disks. All hardware, and the operating systems supporting it, carries traces of this history. Before you can understand filesystems, you need to know something about the hardware.

Disks, Limits, and Filesystems

For many years, software authors created filesystems with the idea that the filesystem would go on a physical disk, such as a hard drive or a floppy disk (or a thing called a floppy disk that was actually fairly hard). They designed filesystems to take advantage of the drive’s physical characteristics. Factors such as the drive’s rotational speed, number of platters, and spin-up time influenced their designs. Filesystems were often tied tightly to the operating system—remember, DOS meant Disk Operating System!

These assumptions no longer apply, except when they do. A filesystem might reside on a disk, or it might exist as an image file, such as an ISO or a virtual machine disk. You still must understand the basics of how the disk works, however. Even ultramodern filesystems like the Z File System (ZFS) demand that the systems administrator understand physical disks.

Back in the stone age of computing, disks had a clearly defined geometry. Disks were round. They spun. The axle in the middle was called a spindle. An individual disk was called a platter, and each disk drive contained a stack of platters. Each platter had concentric rings of magnetic material, called tracks. As all of the individual disks inside a hard drive were identical, the tracks lined up above one another. A stack of tracks was called a cylinder. Each disk drive had a number of heads, devices that read and wrote data on a track as the platter spun beneath them. Each cylinder was divided into little logical units, called sectors. Each sector had a number, with sector 0 at the beginning of the disk, and increasing by one as you circled the disk. Sector 0 is reserved for the partition table. The number of sectors, tracks, heads, and cylinders combined to describe the disk geometry. Disk drive manufacturers still use these terms, but they no longer apply to anything physical.

Hardware engineers set all kinds of hard limits in system design. The 640KB memory limit in the original IBM PC is the butt of any number of jokes, but one day they’ll tell the same jokes about the 4GB i386 memory limit. Those big industrial-grade SCSI disks had a maximum size of 1GB (2²¹ blocks). Hardware has had critical disk limits at 504 megabytes, 8 gigabytes, at 2 terabytes, and more.

If a hardware manufacturer can cleverly figure out how to cram 126 sectors into a track when everybody else can only fit 63, congratulate them! They’ve doubled the capacity of their disk. But if the most popular operating system insists that disks can only have 63 sectors per track, nobody will buy that hard drive. The easy solution is not to fix the operating system, but to teach the hard drive to claim that it has half as many sectors per track and twice as many platters or twice as many tracks. Either works, so long as the numbers still add up.

Is this a new development? No. Not all tracks are the same length, but filesystems expect an equal number of sectors per tracks. The tracks at the edge of the disk are about twice as long as the tracks on the inside, which means you can cram more sectors in the outside tracks than the inside. Back in the 1970s, manufacturers adjusted the hard drive controller to permit this and taught the disk to give wrong answers.

When newer standards increased built-in limits, manufacturers could tell the truth again. Or not. Once you accumulate enough layers of lies, and everyone expects you to lie, coming clean is hard.

Every hard drive manufacturer has faced these problems. And they’ve all chosen their preferred solution. Flash drives claim to have cylinders, sectors, and tracks, but they’re not round and they don’t spin. Hardware RAID devices combine several disks into one disk. Virtual disk images consist of sectors scattered across another filesystem. You could fill entire hard drives documenting the ways hard drives torment numbers to tell useful lies.

When you read a statement like Partitions must end on a cylinder boundary, what you’re really reading is This filesystem is designed for a disk that tells a certain type of lie, and it throws a tantrum when used other than as directed. This doesn’t mean that you cannot use the filesystem on a virtual disk, but you must respect the filesystem’s limitations.

Logical Block Addressing

Hardware manufacturers got tired of teaching hard drives to lie believably, and operating system manufacturers got tired of working around those lies. That’s why all SCSI hard drives, and all modern hard drives, use Logical Block Addressing (LBA).

With LBA, every sector is assigned a number, starting with 0. LBA lets you stop worrying about the cylinders, sectors, platters, and heads on a hard drive, and access arbitrary sector numbers. The computer can think of sectors on a disk as points on a line, with no regard for the physical layout. LBA drives still report a traditional geometry, but neither the hardware nor operating system manufacturers expect that geometry to have any resemblance to reality.

Many filesystems still expect to find a disk geometry, however. LBA’s convenient sector numbering doesn’t release you from those filesystems’ restrictions.

Sector Size

One critical factor on a disk is sector size. Up through the 1990s, sector sizes varied from 128 bytes to 2kB. Even the original IBM PC could understand different sector sizes on floppy disks. FreeBSD supports whatever sector size the hardware supported.

In the early 2000s, manufacturers settled on 512-byte sectors. Today’s hard drives are much larger, and files are similarly larger. In the last few years, the 512-byte sectors have mostly been replaced with 4096-byte (4KB) sectors. This is more appropriate for today’s computers and file sizes. While the size of an MP3 file might not have changed, when a hard drive was only 200 MB I would have kept far fewer of them around.

The problem is, operating systems like Windows XP know that a disk sector had always been and always would be 512 bytes. They would not accept hard drives with 4KB sectors. If you manufacture 4KB-sector disks and you want to sell your hard drives to Windows XP users, what do you do?

The same thing you always do.

You teach your hard drive to lie.

Best of all, different 4KB-sector hard drives lie in different ways. If you ask a drive its sector size, most state that they have 512-byte sectors. Some claim they simultaneously have both 512-byte and 4KB sectors. Very few admit to having 4KB sectors. To complicate matters even more, some solid state drives (SSDs) have sectors as large as 8KB or 16KB, or have multiple sector sizes.

Both of FreeBSD’s main filesystems must know the sector size of the underlying disk and where the sectors lie on the disk. If you use the wrong sector size on the disk, or you don’t align your filesystem partitions with sector boundaries, performance suffers. You’ll learn how to avoid these problems throughout this book.

Disk Size

Not all disks sold as a certain size are the same size. The computing industry can’t agree on what the unit measurements mean, and then manufacturers round things off anyway.

Some computing purists declare that a one-terabyte hard drive should have 1,024 gigabytes. Each gigabyte should be made up of 1,024 megabytes. Hard drive manufacturers normally measure storage size in multiples of 1000 rather than powers of 2, so your 1TB hard drive has an even million megabytes rather than 1,048,576MB.

Differences in manufacturing process and drive design mean that hard drives don’t necessarily have the same number of sectors on them. So long as the total number of sectors rounds off to about the right size, the manufacturer is content. Most of us don’t care if one disk has a few megabytes more than another… until we start partitioning disks, that is. You cannot copy a hard drive’s partition table onto a smaller hard drive. Two hard drives might both claim to be 2TB drives, but if one is a few sectors larger than the other you cannot duplicate the larger drive on the smaller drive.

I recommend leaving at least a few megabytes open at the end of your hard drive. This leaves you room to duplicate a drive or restore a failed drive without having to peer at the fine print on a prospective replacement drive.

Disk Performance

Hardware quality and age determines your performance more than any other factor. A 10k RPM drive processes requests more quickly than a 5k RPM drive. A SATA-3 drive works more quickly than a SATA-2. SSD outperforms spinning disks. New hardware outperforms old. That’s just how things work. The best way to improve performance is to improve your hardware. With that in mind, there are a few things you can do to improve disk performance within the limits of your hardware.

On spinning disks the beginning of the disk, with low sector numbers, is supposed to be faster than the end of the disk. Put partitions that you want to be fast, such as your swap space, closer to the beginning of the disk. Some partitions, such as boot partitions, must go at the very beginning of the disk, but you can put the fast partitions right after that.

If you have a sufficiently large spinning disk, you might even leave the slower part of the disk empty. Leaving empty space on a hard drive doesn’t improve the drive’s raw performance, but using only the faster part of the drive can make it perform better for your application. You’re probably better off adding more spindles to your storage.

SSD storage is slightly different. No part of an SSD is faster than any other part. Leaving empty space on a solid state disk improves performance and longevity, for different reasons than a spinning disk. Also, enabling TRIM (Chapter 4) on UFS filesystems on SSD storage prolongs disk life.

When should you add hardware? Disks are limited in the number of input/output operations per second (IOPS) they can perform. The exact number varies between disks, but on hard disks it’s around 100-200 IOPS. SSD disks can reach tens of thousands of IOPS with certain loads. When you run up against an IOPS limit but require more performance, either add disks or replace slow disks with faster ones.

Book Overview

This chapter covered the basics of disk terminology and technology. If you already understand disks you can skip this chapter, but since you’ve come this far you might as well read the next few paragraphs.

Chapter 1 covers FreeBSD-specific storage technologies and concepts, including the GEOM storage transformation layer, FreeBSD’s device nodes, and the Common Access Method. I’ll also touch on how FreeBSD interacts with standard storage partitioning schemes and FreeBSD’s handling of disk alignment issues. Finally I’ll discuss how to manage filesystems on FreeBSD.

Chapter 2