The Mozaic

The Mozaic
Mark Safronov (compiled only)
Dedication
Dedicated to all newbs in programming.
Table of Contents
Preface ................................................................................................................... iii
Youre just another carriage return line feed in the wall ........................................ 1
What every programmer absolutely, positively needs to know about encodings
and character sets to work with text ..................................................................... 6
Everything Is Broken ............................................................................................ 27
Dont go live with simple security problems - 10 tips to help .............................. 43
Extreme Programming, a Reflection .................................................................... 47
Surviving Legacy Code With Golden Master and Sampling .................................. 51
Hexagonal Architecture ........................................................................................ 57
Stop. Write a learning test ................................................................................... 74
The Long, Painful History of Time ........................................................................ 78
The make it work, make it right, make it fast misconception ............................ 103
The Myth of Schema-less ...................................................................................
NoDB ..................................................................................................................
7 coding tasks you should probably not write yourself ......................................
You Need This One Skill to Succeed in IT ..........................................................
ORM Is Not a Choice and How to Make It Suck Less ..........................................
Programming Sucks ...........................................................................................
Revenge of the Nerds ........................................................................................
Screaming Architecture ......................................................................................
How to securely hash passwords? .....................................................................
Smart Guy Productivity Pitfalls ...........................................................................
TDD, Straw Men, and Rhetoric ...........................................................................
Teach Yourself Programming in Ten Years ........................................................
Untested code is broken code: test automation in enterprise software delivery ..
The Three Laws of TDD. ....................................................................................
ii
105
109
113
118
120
123
131
148
151
162
171
176
184
193
Preface
This book is a collection of essays from various authors and times. Those essays are
about different themes of the software engineering practice. When I was starting
my career as a software developer, I was reading a lot of books, blogs and chat
discussions trying to catch up with all the experience accumulated in this profession.
Some articles were so groundbreaking to me then that I started to collect them over
the years. And now, here they are.
This "fan-made" book is completely non-commercial and has been made in good faith
that it hadnt offended any of the authors. I didnt ask any of the authors about the
permission to reproduce their text here. All articles are unchanged, with carefully
preserved layout, where it was possible.
I put this work under Creative Commons Attribution-NonCommercial-ShareAlike 3.0
1
Unported license . Some of articles probably were licensed under different terms,
but I strongly believe CC BY-NC-SA 3.0 is a best possible common ground for any
of such articles.
Hope this collection will help somebody get a better understanding of our craft.
https://creativecommons.org/licenses/by-nc-sa/3.0/
iii
Youre just another carriage return

line feed in the wall
Written by: Scott Hanselman at 2013-02-20
I love getting pull requests on GitHub. Its such a lovely gift when someone wants
to contribute their code to my code. However, it seems there are three kinds of pull
requests that I get.
1. Awesome, appreciated and wanted.
2. Not so good, thanks for trying, but perhaps another time.
3. THE WALL OF PINK
Id like to talk about The Wall of Pink. This is a pull request that is possibly useful,
possibly awesome, but Ill never know because 672 lines (GitHub tells me) changed
because they used CRs and I used LFs or I used CRLF and they used LF, or I used
well, you get the idea.
There is definitely a problem here. But whats the problem? Well, its kind of like
endianness, except were still talking about it in 2013.
1
http://www.hanselman.com/blog/YoureJustAnotherCarriageReturnLineFeedInTheWall.aspx
Youre just another carriage return line feed in the wall

"A big-endian machine stores the most significant byte firstat the
lowest byte addresswhile a little-endian machine stores the least
significant byte first."
Wikipedia http://en.wikipedia.org/wiki/Endianness
Did you know for a long time Apple computers were big endian and Intel computers
were little endian? The Java VM is big endian. I wrote shareware code generator
16 years ago that generated a byte array on an Intel PC that was later entered
into a PalmPilot running a Motorola 68328. This was the last time I thought about
endianness in my career. Folks working on lower-level stuff do think about this
sometimes, admittedly, but the majority of folks dont sweat endianness day to day..
TCP/IP itself is, in fact, big endian. There was a time when we had to really think about
the measurable performance hit involved in using TCP/IP on a little-endian processor.
But we dont think about that anymore. Its there but the abstraction is not very leaky.
Its years later, but CR/LF issues plague us weekly. That Wall of Pink I mentioned?
It looks like this. I had to scroll 672 lines before I saw the +green where the added
lines were added. Who knows what really changed here though? Cant tell since this
diff tool thinks every line changed.
Sigh. Whose fault is this?

2
Perhaps we blame mile Baudot in 1870 and Donald Murray in 1899 for adding
control characters to instruct a typewriter carriage to return to the home position
2
http://en.wikipedia.org/wiki/%C3%89mile_Baudot
3
http://en.wikipedia.org/wiki/Donald_Murray_(inventor)

plus a line feed to advance the paper on the roller. Or we blame Teletype machines.
Or the folks at DEC, or perhraps Gary Kidall and CP/M for using DEC as a convention.
Then the bastards at IBM who moved to ASCII from EBCDIC and needed a carriage
return when punch-cards fell out of favor.
The text files we have to day on Windows still have a CR LF (0D 0A) after every line.
But Apple uses just uses a line feed (LF) character. Theres no carriage to return, but
there are lines to advance so its a logical savings.
Macs and PCs are sharing text more than ever. We live in a world where Git is FTP
for code, were up a level, above TCP/IP where Endianness is hidden, but still in text
where CR LFs arent.
We store our text files in different formats on disk, but later when the files are
committed to Git, how are they stored? It depends on your settings and the defaults
are never whats recommended.
You can setup a .gitattributes per repo to do things like this:
*.txt -crlf
Or you can do what GitHub for Windows suggests with text=auto .

# Auto detect text files and perform LF normalization
* text=auto
Whats text=auto
4
do?
http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

This ensures that all files that git considers to be text will have
normalized (LF) line endings in the repository. The core.eol
configuration variable controls which line endings git will use for
normalized files in your working directory; the default is to use the
native line ending for your platform, or CRLF if core.autocrlf is set.
It uses the native line ending for your platform. But if you spend a few minutes
googling around youll find arguments several ways with no 100% clear answer,
5
although most folks seem to believe GitHub has the right one .
If this is the right answer, why isnt it a default? Is it time to make this the default?
This is such a problem that did you know GitHub for Windows has dedicated
"normalize your repos CRLF" code? Theyll fix them all and make a one-time commit
to fix the line endings.
I think a more complete solution would also include improvements to the online diff
tool. If the GitHub repro and server knows something is wrong, thats a great chance
for the server to suggest a fix, proactively.
SolutionsHeres some possible solutions as I see it.
Make Windows switch all text files and decades of convention to use just LF
Git needs to install with correct platform-specific defaults without
needing .gitattributes file
Have the GitHub web application be more proactive in suggesting solutions and
preventing badness
5
http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git

Have the GitHub for Windows desktop application proactively notice issues
(before I go to settings) and offer to help
Make the diff tool CR/LF aware and "do the right thing" like desktop diff tools that
can ignore line ending issues
Until something is done, Ill always tense up when I see an incoming pull request and
hope its not a Wall of Pink.
What every programmer absolutely,

positively needs to know about
encodings and character sets to work
with text
Written by: David C. Zentgraf at 2013-08-05
If you are dealing with text in a computer, you need to know about encodings. Period.
Yes, even if you are just sending emails. Even if you are just receiving emails. You
dont need to understand every last detail, but you must at least know what this
whole "encoding" thing is about. And the good news first: while the topic can get
messy and confusing, the basic idea is really, really simple.
This article is about encodings and character sets. An article by Joel Spolsky entitled
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know
2
About Unicode and Character Sets (No Excuses!) is a nice introduction to the topic
and I greatly enjoy reading it every once in a while. I hesitate to refer people to it who
have trouble understanding encoding problems though since, while entertaining, it
is pretty light on actual technical details. I hope this article can shed some more light
on what exactly an encoding is and just why all your text screws up when you least
need it. This article is aimed at developers (with a focus on PHP), but any computer
user should be able to benefit from it.
Getting the basics straight

Everybody is aware of this at some level, but somehow this knowledge seems to
suddenly disappear in a discussion about text, so lets get it out first: A computer
cannot store "letters", "numbers", "pictures" or anything else. The only thing it can
store and work with are bits. A bit can only have two values: yes or no , true
or false , 1 or 0 or whatever else you want to call these two values. Since a
computer works with electricity, an "actual" bit is a blip of electricity that either is
or isnt there. For humans, this is usually represented using 1 and 0 and Ill stick
with this convention throughout this article.
1
http://kunststube.net/encoding/
2
http://www.joelonsoftware.com/articles/Unicode.html
What every programmer absolutely, positively needs to
know about encodings and character sets to work with text

To use bits to represent anything at all besides bits, we need rules. We need to
convert a sequence of bits into something like letters, numbers and pictures using
an encoding scheme, or encoding for short. Like this:
01100010 01101001 01110100 01110011
b
i
t
s
In this encoding, 01100010 stands for the letter "b", 01101001 for the letter "i",
01110100 stands for "t" and 01110011 for "s". A certain sequence of bits stands
for a letter and a letter stands for a certain sequence of bits. If you can keep this in
your head for 26 letters or are really fast with looking stuff up in a table, you could
read bits like a book.
The above encoding scheme happens to be ASCII. A string of `1`s and `0`s is broken
down into parts of eight bit each (a byte for short). The ASCII encoding specifies a
table translating bytes into human readable letters. Heres a short excerpt of that
table:
bits
character
01000001
01000010
01000011
01000100
01000101
01000110
There are 95 human readable characters specified in the ASCII table, including the
letters A through Z both in upper and lower case, the numbers 0 through 9, a handful
of punctuation marks and characters like the dollar symbol, the ampersand and a
few others. It also includes 33 values for things like space, line feed, tab, backspace
and so on. These are not printable per se, but still visible in some form and useful
to humans directly. A number of values are only useful to a computer, like codes
to signify the start or end of a text. In total there are 128 characters defined in the
ASCII encoding, which is a nice round number (for people dealing with computers),
3
Yes, that means ASCII can be stored and transferred using only 7 bits and it often is. No, this is not
within the scope of this article and for the sake of argument well assume the highest bit is "wasted" in
ASCII.

since it uses all possible combinations of 7 bits ( 0000000 , 0000001 , 0000010
3
through 1111111 ).
And there you have it, the way to represent human-readable text using only `1`s
and `0`s.
01001000 01100101 01101100 01101100 01101111 00100000
01010111 01101111 01110010 01101100 01100100
"Hello World"
Important terms
To encode something in ASCII, follow the table from right to left, substituting letters
for bits. To decode a string of bits into human readable characters, follow the table
from left to right, substituting bits for letters.
encode |enkd|
verb [ with obj. ]
convert into a coded form
code |kd|
noun
a system of words, letters, figures, or other symbols substituted for other
words, letters, etc.
To encode means to use something to represent something else. An encoding is the
set of rules with which to convert something from one representation to another.
Other terms which deserve clarification in this context:
character set, charset
The set of characters that can be encoded. "The ASCII encoding encompasses a
character set of 128 characters." Essentially synonymous to "encoding".
code page
A "page" of codes that map a character to a number or bit sequence. A.k.a. "the
table". Essentially synonymous to "encoding".
8

string
A string is a bunch of items strung together. A bit string is a bunch of bits,
like 01010011 . A character string is a bunch of characters, like this .
Synonymous to "sequence".
Binary, octal, decimal, hex

There are many ways to write numbers. 10011111 in binary is 237 in octal is 159 in
decimal is 9F in hexadecimal. They all represent the same value, but hexadecimal is
shorter and easier to read than binary. I will stick with binary throughout this article
to get the point across better and spare the reader one layer of abstraction. Do not
be alarmed to see character codes referred to in other notations elsewhere, its all
the same thing.
Excusez-moi?
Now that we know what were talking about, lets just say it: 95 characters really
isnt a lot when it comes to languages. It covers the basics of English, but what about
writing a risqu letter in French? A Straenbergangsnderungsgesetz in German?
An invitation to a smrgsbord in Swedish? Well, you couldnt. Not in ASCII. Theres
no specification on how to represent any of the letters , , , , or in ASCII, so
you cant use them.
"But look at it," the Europeans said, "in a common computer with 8 bits to the byte,
ASCII is wasting an entire bit which is always set to 0 ! We can use that bit to squeeze
a whole 'nother 128 values into that table!" And so they did. But even so, there
are more than 128 ways to stroke, slice, slash and dot a vowel. Not all variations of
letters and squiggles used in all European languages can be represented in the same
table with a maximum of 256 values. So what the world ended up with is a wealth
4
of encoding schemes , standards, de-facto standards and half-standards that all

cover a different subset of characters. Somebody needed to write a document about
Swedish in Czech, found that no encoding covered both languages and invented one.
Or so I imagine it went countless times over.
And not to forget about Russian, Hindi, Arabic, Hebrew, Korean and all the other
languages currently in active use on this planet. Not to mention the ones not in
use anymore. Once you have solved the problem of how to write mixed language
documents in all of these languages, try yourself on Chinese. Or Japanese. Both
4
http://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings

contain tens of thousands of characters. You have 256 possible values to a byte
consisting of 8 bit. Go!
Multi-byte encodings
To create a table that maps characters to letters for a language that uses more than
256 characters, one byte simply isnt enough. Using two bytes (16 bits), its possible
5
to encode 65,536 distinct values. BIG-5 is such a double-byte encoding. Instead

of breaking a string of bits into blocks of eight, it breaks it into blocks of 16 and
has a big (I mean, BIG) table that specifies which character each combination of
bits maps to. BIG-5 in its basic form covers mostly Traditional Chinese characters.
6
GB18030 is another encoding which essentially does the same thing, but includes
both Traditional and Simplified Chinese characters. And before you ask, yes, there
are encodings which cover only Simplified Chinese. Cant just have one encoding
now, can we?
Here a small excerpt from the GB18030 table:
bits
character
10000001 01000000
10000001 01000001
10000001 01000010
10000001 01000011
10000001 01000100
GB18030 covers quite a range of characters (including a large part of latin

characters), but in the end is yet another specialized encoding format among many.
5
http://en.wikipedia.org/wiki/Big-5
6
http://en.wikipedia.org/wiki/GB18030
10
Unicode to the confusion
Finally somebody had enough of the mess and set out to forge a ring to bind them
all create one encoding standard to unify all encoding standards. This standard is
Unicode. It basically defines a ginormous table of 1,114,112 code points that can
be used for all sorts of letters and symbols. Thats plenty to encode all European,
Middle-Eastern, Far-Eastern, Southern, Northern, Western, pre-historian and future
7
characters mankind knows about. Using Unicode, you can write a document
containing virtually any language using any character you can type into a computer.
This was either impossible or very very hard to get right before Unicode came along.
8
Theres even an unofficial section for Klingon in Unicode. Indeed, Unicode is big
enough to allow for unofficial, private-use areas.
So, how many bits does Unicode use to encode all these characters? None. Because
Unicode is not an encoding.
Confused? Many people seem to be. Unicode first and foremost defines a table of
code points for characters. Thats a fancy way of saying "65 stands for A, 66 stands
for B and 9,731 stands for " (seriously, it does). How these code points are actually
encoded into bits is a different topic. To represent 1,114,112 different values, two
bytes arent enough. Three bytes are, but three bytes are often awkward to work
7
And if it isnt, it will be extended. It already has been several times.
8
http://www.evertype.com/standards/csur/klingon.html
11

with, so four bytes would be the comfortable minimum. But, unless youre actually
using Chinese or some of the other characters with big numbers that take a lot of
bits to encode, youre never going to use a huge chunk of those four bytes. If the
letter "A" was always encoded to 00000000 00000000 00000000 01000001 , "B"
always to 00000000 00000000 00000000 01000010 and so on, any document
would bloat to four times the necessary size.
To optimize this, there are several ways to encode Unicode code points into bits.
UTF-32 is such an encoding that encodes all Unicode code points using 32 bits. That
is, four bytes per character. Its very simple, but often wastes a lot of space. UTF-16
and UTF-8 are variable-length encodings. If a character can be represented using a
single byte (because its code point is a very small number), UTF-8 will encode it with
a single byte. If it requires two bytes, it will use two bytes and so on. It has elaborate
ways to use the highest bits in a byte to signal how many bytes a character consists
of. This can save space, but may also waste space if these signal bits need to be
used often. UTF-16 is in the middle, using at least two bytes, growing to up to four
bytes as necessary.
character encoding
bits
UTF-8
UTF-16
00000000 01000001
UTF-32
00000000 00000000 00000000

01000001
UTF-8
11100011 10000001 10000010
UTF-16
00110000 01000010
UTF-32
00000000 00000000 00110000

01000010
01000001
And thats all there is to it. Unicode is a large table mapping characters to numbers
and the different UTF encodings specify how these numbers are encoded as bits.
Overall, Unicode is yet another encoding scheme. Theres nothing special about it, its
just trying to cover everything while still being efficient. And thats A Good Thing.
12
Code points
Characters are referred to by their "Unicode code point". Unicode code points are
written in hexadecimal (to keep the numbers shorter), preceded by a "U+" (thats
just what they do, it has no other meaning than "this is a Unicode code point"). The
character has the Unicode code point U+1E00. In other (decimal) words, it is the
7680th character of the Unicode table. It is officially called "LATIN CAPITAL LETTER
A WITH RING BELOW".
TL;DR
A summary of all the above: Any character can be encoded in many different bit
sequences and any particular bit sequence can represent many different characters,
depending on which encoding is used to read or write them. The reason is simply
because different encodings use different numbers of bits per characters and
different values to represent different characters.
bits
encoding
characters
11000100 01000010
Windows
Latin 1
11000100 01000010
Mac Roman
11000100 01000010
GB18030
13

characters encoding
bits
Windows
Latin 1
01000110 11111000 11110110
Mac Roman
01000110 10111111 10011010
UTF-8
01000110 11000011 10111000 11000011

10110110
Misconceptions, confusions and problems

Having said all that, we come to the actual problems experienced by many users
and programmers every day, how those problems relate to all of the above and what
their solution is. The biggest problem of all is:
Why in gods name are my characters garbled?!

GR[fBO
If you open a document and it looks like this, theres one and only one reason for it:
Your text editor, browser, word processor or whatever else thats trying to read the
document is assuming the wrong encoding. Thats all. The document is not broken
(well, unless it is, see below), theres no magic you need to perform, you simply need
to select the right encoding to display the document.
The hypothetical document above contains this sequence of bits:
10000011
10000011
10000010
10000010
01000111
01100110
11001101
11001000
10000011
10000011
10010011
10000010
10010011 10000011 01010010 10000001 01011011

01000010 10000011 10010011 10000011 01001111
11101111 10000010 10110101 10000010 10101101
10100010
Now, quick, what encoding is that? If you just shrugged, youd be correct. Who knows,
right
9
Well, lets try to interpret this as ASCII. Hmm, most of these bytes start with a
1 bit. If you remember correctly, ASCII doesnt use that bit. So its not ASCII. What
9
Please note that when Im using the term "starting" together with "byte", I mean it from the human-
readable point of view.
14

10
about UTF-8? Hmm, no, most of these sequences are not valid UTF-8. So UTF-8 is
out, too. Lets try "Mac Roman" (yet another encoding scheme for them Europeans).
Hey, all those bytes are valid in Mac Roman. 10000011 maps to "", 01000111
to "G" and so on. If you read this bit sequence using the Mac Roman encoding,
the result is "GR[fBO". That looks like a valid string, no?
Yes? Maybe? Well, hows the computer to know? Maybe somebody meant to write
11
"GR[fBO". For all I know that could be a DNA sequence.
Unless you have a better suggestion, lets declare this to be a DNA sequence, say
this document was encoded in Mac Roman and call it a day.
Of course, that unfortunately is complete nonsense. The correct answer is that this
text is encoded in the Japanese Shift-JIS encoding and was supposed to read "
". Well, whodve thunk?
The primary cause of garbled text is: Somebody is trying to read a byte sequence
using the wrong encoding. The computer always needs to be told what encoding
some text is in. Otherwise it cant know. There are different ways how different kinds
of documents can specify what encoding theyre in and these ways should be used.
A raw bit sequence is always a mystery box and could mean anything.
10
Peruse the UTF-8 specification if you want to follow this with pen and paper.
11
Hey, Im a programmer, not a biologist.
15

Most browsers allow the selection of a different encoding in the View menu under the
menu option "Text Encoding", which causes the browser to reinterpret the current
page using the selected encoding. Other programs may offer something like "Reopen
using encoding" in the File menu, or possibly an "Import" option which allows the
user to manually select an encoding.
My document doesnt make sense in any encoding!

If a sequence of bits doesnt make sense (to a human) in any encoding, the document
has mostly likely been converted incorrectly at some point. Say we took the above
text "GR[fBO" because we didnt know any better and
saved it as UTF-8. The text editor assumed it correctly read a Mac Roman encoded
text and you now want to save this text in a different encoding. All of these characters
are valid Unicode characters after all. That is to say, theres a code point in Unicode
that can represent "", one that can represent "G" and so on. So we can happily save
this text as UTF-8:
11000011
10001001
11000011
10001001
11000011
11100010
10000111
10001001
01010010
10001001
01001111
10010100
10001001
11000010
01000111
11000011
01000010
11000011
11000011
10100000
10100010
11000011
10000101
11000011
10000111
10000111
11000011
10001001
01011011
10001001
11000011
11000010
10000111
11000011
11000011
11000011
10010101
10110101
11000010
10101100
10001001
10101100
11000011
11000011
10111011
11000011
01100110
11000011
10101100
10000111
11000011
This
is
now
the
UTF-8
bit
sequence
representing
the
text
"GR[fBO". This bit sequence has absolutely nothing to do
with our original document. Whatever encoding we try to open it in, we wont ever
get the text "" from it. It is completely lost. It would be
possible to recover the original text from it if we knew that a Shift-JIS document was
misinterpreted as Mac Roman and then accidentally saved as UTF-8 and reversed
this chain of missteps. But that would be a lucky fluke.
Many times certain bit sequences are invalid in a particular encoding. If we tried
to open the original document using ASCII, some bytes would be valid in ASCII and
map to a real character and others wouldnt. The program youre opening it with
may decide to silently discard any bytes that arent valid in the chosen encoding, or
possibly replace them with ? . Theres also the "Unicode replacement character"
(U+FFFD) which a program may decide to insert for any character it couldnt decode
correctly when trying to handle Unicode. If a document is saved with some characters
16

gone or replaced, then those characters are really gone for good with no way to
reverse-engineer them.
If a document has been misinterpreted and converted to a different encoding, its
broken. Trying to "repair" it may or may not be successful, usually it isnt. Any manual
bit-shifting or other encoding voodoo is mostly that, voodoo. Its trying to fix the
symptoms after the patient has already died.
So how to handle encodings correctly?

Its really simple: Know what encoding a certain piece of text, that is, a certain byte
sequence, is in, then interpret it with that encoding. Thats all you need to do. If youre
writing an app that allows the user to input some text, specify what encoding you
accept from the user. For any sort of text field, the programmer can usually decide
its encoding. For any sort of file a user may upload or import into a program, there
needs to be a specification what encoding that file should be in. Alternatively, the
user needs some way to tell the program what encoding the file is in. This information
may be part of the file format itself, or it may be a selection the user has make (not
that most users would usually know, unless they have read this article).
If you need to convert from one encoding to another, do so cleanly using tools
that are specialized for that. Converting between encodings is the tedious task
of comparing two code pages and deciding that character 152 in encoding A is
the same as character 4122 in encoding B, then changing the bits accordingly.
This particular wheel does not need reinventing and any mainstream programming
language includes some way of converting text from one encoding to another without
needing to think about code points, pages or bits at all.
Say, your app must accept files uploaded in GB18030, but internally you are handling
all data in UTF-32. A tool like iconv can cleanly convert the uploaded file with a
one-liner like iconv('GB18030', 'UTF-32', $string) . That is, it will preserve
the characters while changing the underlying bits:
character
GB18030 encoding
UTF-32 encoding
10111111
01101100
00000000 00000000 01111110

00100111
Thats all there is to it. The content of the string, that is, the human readable
characters, didnt change, but its now a valid UTF-32 string. If you keep treating
it as UTF-32, theres no problem with garbled characters. As discussed at the very
17

beginning though, not all encoding schemes can represent all characters. Its not
possible to encode the character "" in any encoding scheme designed for European
languages. Something Bad would happen if you tried to.
Unicode all the way

Precisely because of that, theres virtually no excuse in this day and age not to be
using Unicode all the way. Some specialized encodings may be more efficient than
the Unicode encodings for certain languages. But unless youre storing terabytes
and terabytes of very specialized text (and thats a lot of text), theres usually no
reason to worry about it. Problems stemming from incompatible encoding schemes
are much worse than a wasted gigabyte or two these days. And this will become
even truer as storage and bandwidth keeps growing larger and cheaper.
If your system needs to work with other encodings, convert them to Unicode upon
input and convert them back to other encodings on output as necessary. Otherwise,
be very aware of what encodings youre dealing with at which point and convert as
necessary, if thats possible without losing any information.
Flukes
I have this website talking to a database. My app handles
everything as UTF-8 and stores it as such in the database and
everything works fine, but when I look at my database admin
interface my text is garbled.
Anonymous code monkey

There are situations where encodings are handled incorrectly but things still work.
An often-encountered situation is a database thats set to latin-1 and an app that
works with UTF-8 (or any other encoding). Pretty much any combination of 1`s and
`0`s is valid in the single-byte `latin-1 encoding scheme. If the
database receives text from an application that looks like 11100111 10111000
10100111 , itll happily store it, thinking the app meant to store the three latin
characters "". After all, why not? It then later returns this bit sequence back to the
app, which will happily accept it as the UTF-8 sequence for "", which it originally
stored. The database admin interface automatically figures out that the database is
set to latin-1 though and interprets any text as latin-1, so all values look garbled only
in the admin interface.
18

Thats a case of fools luck where things happen to work when they actually arent.
Any sort of operation on the text in the database may or may not work as intended,
since the database is not interpreting the text correctly. In a worst case scenario, the
database inadvertently destroys all text during some random operation two years
after the system went into production because it was operating on text assuming
12
the wrong encoding.
UTF-8 and ASCII

The ingenious thing about UTF-8 is that its binary compatible with ASCII, which is
the de-facto baseline for all encodings. All characters available in the ASCII encoding
only take up a single byte in UTF-8 and theyre the exact same bytes as are used in
ASCII. In other words, ASCII maps 1:1 unto UTF-8. Any character not in ASCII takes up
two or more bytes in UTF-8. For most programming languages that expect to parse
ASCII, this means you can include UTF-8 text directly in your programs:
$string = "";
Saving this as UTF-8 results in this bit sequence:

00100100 01110011 01110100 01110010 01101001 01101110 01100111 00100000
00111101 00100000 00100010 11100110 10111100 10100010 11100101 10101101
10010111 00100010 00111011
Only bytes 12 through 17 (the ones starting with 1 ) are UTF-8 characters (two
characters with three bytes each). All the surrounding characters are perfectly good
ASCII. A parser would read this as follows:
$string = "11100110 10111100 10100010 11100101 10101101 10010111";
To the parser, anything following a quotation mark is just a byte sequence which it
will take as-is until it encounters another quotation mark. If you simply output this
byte sequence, youre outputting UTF-8 text. No need to do anything else. The parser
does not need to specifically support UTF-8, it just needs to take strings literally.
Naive parsers can support Unicode this way without actually supporting Unicode.
Many modern languages are explicitly Unicode-aware though.
12
And of course therell be no recent backup.
19
Encodings and PHP

This last section deals with issues surrounding Unicode and PHP.
Some portions of it are applicable to programming languages in
general while others are PHP specific. Nothing new will be revealed
about encodings, but concepts described above will be rehashed
in the light of practical application.
PHP doesnt natively support Unicode. Except it actually supports it quite well. The
previous section shows how UTF-8 characters can be embedded in any program
directly without problems, since UTF-8 is backwards compatible with ASCII, which is
all PHP needs. The statement "PHP doesnt natively support Unicode" is true though
and it seems to cause a lot of confusion in the PHP community.
False promises
13
One specific pet-peeve of mine are the functions utf8_encode

and
14
utf8_decode . I often see nonsense along the lines of "To use Unicode in PHP you
need to utf8_encode your text on input and utf8_decode on output
". These
two functions seem to promise some sort of automagic conversion of text to UTF-8
which is "necessary" since "PHP doesnt support Unicode". If youve been following
this article at all though, you should know by now that
1. theres nothing special about UTF-8 and
2. you cannot encode text to UTF-8 after the fact
To clarify that second point: All text is already encoded in some encoding. When you
type it into the source code, it has some encoding. Specifically, whatever you saved
it as in your text editor. If you get it from a database, its already in some encoding.
If you read it from a file, its already in some encoding.
Text is either encoded in UTF-8 or its not. If its not, its encoded in ASCII, ISO-8859-1,
UTF-16 or some other encoding. If its not encoded in UTF-8 but is supposed to contain
15
"UTF-8 characters", then you have a case of cognitive dissonance. If it does contain
13
http://php.net/utf8_encode
14
http://php.net/utf8_decode
15
A "Unicode character" is a code point in the Unicode table. "" is not a Unicode character, its the
Hiragana letter . There is a Unicode code point for it, but that doesnt make the letter itself a Unicode
character. A "UTF-8 character" is an oxymoron, but may be stretched to mean whats technically called
a "UTF-8 sequence", which is a byte sequence of one, two, three or four bytes representing one Unicode
20

actual characters encoded in UTF-8, then its actually UTF-8 encoded. Text cant
contain Unicode characters without being encoded in one of the Unicode encodings.
So what in the world does utf8_encode do then?
"Encodes an ISO-8859-1 string to UTF-8"
16
Aha! So what the author actually wanted to say is that it converts the encoding of text
from ISO-8859-1 to UTF-8. Thats all there is to it. utf8_encode must have been
named by some European without any foresight and is a horrible, horrible misnomer.
The same goes for utf8_decode . These functions are useless for any purpose other
than converting between ISO-8859-1 and UTF-8. If you need to convert a string from
17
any other encoding to any other encoding, look no further then iconv .
utf8_encode is not a magic wand that needs to be swung over any and all text
because "PHP doesnt support Unicode". Rather, it seems to cause more encoding
problems than it solves thanks to terrible naming and unknowing developers.
Native-schmative
So what does it mean for a language to natively support or not support Unicode?
It basically refers to whether the language assumes that one character equals one
byte or not. For example, PHP allows direct access to the characters of a string using
array notation:
echo $string[0];
If that $string was in a single-byte encoding, this would give us the first character.
But only because "character" coincides with "byte" in a single-byte encoding. PHP
simply gives us the first byte without thinking about "characters". Strings are byte
sequences to PHP, nothing more, nothing less. All this "readable character" stuff is
a human thing and PHP doesnt care about it.
01000100
D
01100011
c
01101111
o
01100001
a
01101110
n
01110010
r
00100111
'
01100101
e
01110100
t
00100001
!
character. Both terms are often used in the sense of "any letter that aint part of my keyboard" though,
which means absolutely nothing.
16
http://www.php.net/manual/en/function.utf8-encode.php
17
http://www.php.net/manual/en/function.iconv.php
21

The same goes for many standard functions such as substr , strpos , trim and
so on. The non-support arises if theres a discrepancy between the length of a byte
and a character.
11100110 10111100 10100010 11100101 10101101 10010111
Using $string[0] on the above string will, again, give us the first byte, which is
11100110 . In other words, a third of the three-byte character "". 11100110 is,
by itself, an invalid UTF-8 sequence, so the string is now broken. If you felt like it,
you could try to interpret that in some other encoding where 11100110 represents
a valid character, which will result in some random character. Have fun, but dont
use it in production.
And thats actually all there is to it. "PHP doesnt natively support Unicode" simply
means that most PHP functions assume one byte = one character, which may
lead to it chopping multi-byte characters in half or calculating the length of strings
incorrectly if youre naively using non-multi-byte-aware functions on multi-byte
strings. It does not mean that you cant use Unicode in PHP or that every Unicode
string needs to be blessed by utf8_encode or other such nonsense.
18
Luckily, theres the Multibyte String extension , which replicates all important
string functions in a multi-byte aware fashion. Using mb_substr($string, 0,
1, 'UTF-8') on the above string correctly returns 11100110 10111100
10100010 , which is the whole "" character. Because the mb_ functions
now have to actually think about what theyre doing, they need to know what
encoding theyre working on. Therefore every mb_ function accepts an $encoding
parameter as well. Alternatively, this can be set globally for all mb_ functions using
mb_internal_encoding .
18
http://www.php.net/manual/en/book.mbstring.php
22
Using and abusing PHPs handling of encodings

The whole issue of PHPs (non-)support for Unicode is that it just doesnt care.
Strings are byte sequences to PHP. What bytes in particular doesnt matter. PHP
doesnt do anything with strings except keeping them stored in memory. PHP simply
doesnt have any concept of either characters or encodings. And unless it tries
to manipulate strings, it doesnt need to either; it just holds onto bytes that may
or may not eventually be interpreted as characters by somebody else. The only
19
requirement PHP has of encodings is that PHP source code needs to be saved in
an ASCII compatible encoding. The PHP parser is looking for certain characters that
tell it what to do. $ ( 00100100 ) signals the start of a variable, = ( 00111101 ) an
assignment, " ( 00100010 ) the start and end of a string and so on. Anything else
that doesnt have any special significance to the parser is just taken as a literal byte
sequence. That includes anything between quotes, as discussed above. This means
the following:
1. You cant save PHP source code in an ASCII-incompatible encoding. For example,
in UTF-16 a " is encoded as 00000000 00100010 . To PHP, which tries to read
everything as ASCII, thats a NUL byte followed by a " . PHP will probably get a
hiccup if every other character it finds is a NUL byte.
2. You can save PHP source code in any ASCII-compatible encoding. If the first 128
code points of an encoding are identical to ASCII, PHP can parse it. All characters
that are in any way significant to PHP are within the 128 code points defined by
ASCII. If string literals contain any code points beyond that, PHP doesnt care.
You can save PHP source code in ISO-8859-1, Mac Roman, UTF-8 or any other
ASCII-compatible encoding. The string literals in your script will have whatever
encoding you saved your source code as.
3. Any external file you process with PHP can be in whatever encoding you like. If
PHP doesnt need to parse it, there are no requirements to meet to keep the PHP
parser happy.
$foo = file_get_contents('bar.txt');
The above will simply read the bits in bar.txt into the variable $foo . PHP
doesnt try to interpret, convert, encode or otherwise fiddle with the contents.
The file can even contain binary data such as an image, PHP doesnt care.
19
http://www.php.net/manual/en/mbstring.php4.req.php
23

4. If internal and external encodings have to match, they have to match. A common
case is localization, where the source code contains something like echo
localize('Foobar') and an external localization file contains something
along the lines of this:
msgid "Foobar"
msgstr ""
Both "Foobar" strings need to have an identical bit representation if you want
to find the correct localization. If the source code was saved in ASCII but
the localization file in UTF-16, the strings wouldnt match. Either some sort of
encoding conversion would be necessary or the use of an encoding-aware string
matching function.
The astute reader might ask at this point whether its possible to save a, say, UTF-16
byte sequence inside a string literal of an ASCII encoded source code file, to which
the answer would be: absolutely.
echo "UTF-16";
If you can bring your text editor to save the echo " and "; parts in ASCII and
only UTF-16 in UTF-16, this will work just fine. The necessary binary representation
for that looks like this:
01100101 01100011
e
c
11111110 11111111
(UTF-16 marker)
00000000 01000110
F
00000000 00110110
6
01101000
h
00000000
U
00000000
00100010
"
01101111 00100000 00100010

o
"
01010101 00000000 01010100
T
00101101 00000000 00110001
1
00111011
;
The first line and the last two bytes are ASCII. The rest is UTF-16 with two bytes per
character. The leading 11111110 11111111 on line 2 is a marker required at the
start of UTF-16 encoded text (required by the UTF-16 standard, PHP doesnt give a
damn). This PHP script will happily output the string "UTF-16" encoded in UTF-16,
because it simple outputs the bytes between the two double quotes, which happens
to represent the text "UTF-16" encoded in UTF-16. The source code file is neither
24

completely valid ASCII nor UTF-16 though, so working with it in a text editor wont
be much fun.
Bottom line
PHP supports Unicode, or in fact any encoding, just fine, as long as certain
requirements are met to keep the parser happy and the programmer knows what hes
doing. You really only need to be careful when manipulating strings, which includes
slicing, trimming, counting and other operations that need to happen on a character
level rather than a byte level. If youre not "doing anything" with your strings besides
reading and outputting them, you will hardly have any problems with PHPs support
of encodings that you wouldnt have in any other language as well.
Encoding-aware languages
What does it mean for a language to support Unicode then? Javascript for example
supports Unicode. In fact, any string in Javascript is UTF-16 encoded. In fact, its the
only thing Javascript deals with. You cannot have a string in Javascript that is not
UTF-16 encoded. Javascript worships Unicode to the extent that theres no facility
to deal with any other encoding in the core language. Since Javascript is most often
run in a browser thats not a problem, since the browser can handle the mundane
logistics of encoding and decoding input and output.
Other languages are simply encoding-aware. Internally they store strings in a
particular encoding, often UTF-16. In turn they need to be told or try to detect the
encoding of everything that has to do with text. They need to know what encoding
the source code is saved in, what encoding a file theyre supposed to read is in,
what encoding you want to output text in; and they convert encodings on the fly as
needed with some manifestation of Unicode as the middleman. Theyre doing the
same thing you can/should/need to do in PHP semi-automatically behind the scenes.
Thats neither better nor worse than PHP, just different. The nice thing about it is
that standard language functions that deal with strings Just Work, while in PHP one
needs to spare some attention to whether a string may contain multi-byte characters
or not and choose string manipulation functions accordingly.
The depths of Unicode

Since Unicode deals with many different scripts and many different problems, it
has a lot of depth to it. For example, the Unicode standard contains information
25

20
for such problems as CJK ideograph unification . That means, information that two
or more Chinese/Japanese/Korean characters actually represent the same character
in slightly different writing methods. Or rules about converting from lower case to
upper case, vice-versa and round-trip, which is not always as straight forward in
all scripts as it is in most Western European Latin-derived scripts. Some characters
can also be represented using different code points. The letter "" for example
can be represented using the code point U+00F6 ("LATIN SMALL LETTER O WITH
DIAERESIS") or as the two code points U+006F ("LATIN SMALL LETTER O") and U
+0308 ("COMBINING DIAERESIS"), that is the letter "o" combined with "". In UTF-8
thats either the double-byte sequence 11000011 10110110 or the three-byte
sequence 01101111 11001100 10001000 , both representing the same human
readable character. As such, there are rules governing Normalization within the
Unicode standard, i.e. how either of these forms can be converted into the other.
This and a lot more is outside the scope of this article, but one should be aware of it.
Final TL;DR
Text is always a sequence of bits which needs to be translated into human
readable text using lookup tables. If the wrong lookup table is used, the wrong
character is used.
Youre never actually directly dealing with "characters" or "text", youre always
dealing with bits as seen through several layers of abstractions. Incorrect results
are a sign of one of the abstraction layers failing.
If two systems are talking to each other, they always need to specify what
encoding they want to talk to each other in. The simplest example of this is this
website telling your browser that its encoded in UTF-8.
In this day and age, the standard encoding is UTF-8 since it can encode virtually
any character of interest, is backwards compatible with the de-facto baseline
ASCII and is relatively space efficient for the majority of use cases nonetheless.
Other encodings still occasionally have their uses, but you should have a
concrete reason for wanting to deal with the headaches associated with
character sets that can only encode a subset of Unicode.
The days of one byte = one character are over and both programmers and
programs need to catch up on this.
Now you should really have no excuse anymore the next time you garble some text.
20
http://en.wikipedia.org/wiki/CJK_Unified_Ideographs
26
Everything Is Broken
Written by: Quinn Norton at May 20, 2014
Once upon a time, a friend of mine accidentally took over thousands of computers.
He had found a vulnerability in a piece of software and started playing with it. In the
process, he figured out how to get total administration access over a network. He
put it in a script, and ran it to see what would happen, then went to bed for about
four hours. Next morning on the way to work he checked on it, and discovered he
was now lord and master of about 50,000 computers. After nearly vomiting in fear he
killed the whole thing and deleted all the files associated with it. In the end he said
he threw the hard drive into a bonfire. I cant tell you who he is because he doesnt
want to go to Federal prison, which is what could have happened if hed told anyone
that could do anything about the bug hed found. Did that bug get fixed? Probably
eventually, but not by my friend. This story isnt extraordinary at all. Spend much
time in the hacker and security scene, youll hear stories like this and worse.
Its hard to explain to regular people how much technology barely works, how much
the infrastructure of our lives is held together by the IT equivalent of baling wire.
Computers, and computing, are broken.
Build it badly, and they will come.

For a bunch of us, especially those who had followed security and the warrantless
wiretapping cases, the revelations werent big surprises. We didnt know the
specifics, but people who keep an eye on software knew computer technology was
sick and broken. Weve known for years that those who want to take advantage of
that fact tend to circle like buzzards. The NSA wasnt, and isnt, the great predator
of the internet, its just the biggest scavenger around. It isnt doing so well because
they are all powerful math wizards of doom.
The NSA is doing so well because software is bullshit.
Eight months before Snowdens first revelation I tweeted this:
It was my exasperated acknowledgement that looking for good
software to count on has been a losing battle. Written by people with
1
https://medium.com/message/everything-is-broken-81e5f33a24e1
27
either no time or no money, most software gets shipped the moment it
works well enough to let someone go home and see their family. What
we get is mostly terrible.
@quinnnorton https://twitter.com/quinnnorton/
status/236655133046501376
Software is so bad because its so complex, and because its trying to talk to other
programs on the same computer, or over connections to other computers. Even your
computer is kind of more than one computer, boxes within boxes, and each one of
those computers is full of little programs trying to coordinate their actions and talk to
each other. Computers have gotten incredibly complex, while people have remained
the same gray mud with pretensions of godhood.
Your average piece-of-shit Windows desktop is so complex that no one person on
Earth really knows what all of it is doing, or how.
Now imagine billions of little unknowable boxes within boxes constantly trying
to talk and coordinate tasks at around the same time, sharing bits of data and
passing commands around from the smallest little program to something huge, like
28
a browser thats the internet. All of that has to happen nearly simultaneously and
smoothly, or you throw a hissy fit because the shopping cart forgot about your movie
tickets.
We often point out that the phone you mostly play casual games on and keep
dropping in the toilet at bars is more powerful than all the computing we used to go
to space for decades.
NASA had a huge staff of geniuses to understand and care for their software. Your
phone has you.
Plus a system of automatic updates you keep putting off because youre in the middle
of Candy Crush Saga every time it asks.
Because of all this, security is terrible. Besides being riddled with annoying bugs and
impossible dialogs, programs often have a special kind of hackable flaw called 0days
by the security scene. No one can protect themselves from 0days. Its their defining
feature 0 is the number of days youve had to deal with this form of attack. There
are meh, not-so-terrible 0days, there are very bad 0days, and there are catastrophic
0days that hand the keys to the house to whomever strolls by. I promise that right
now you are reading this on a device with all three types of 0days. But, Quinn, I can
hear you say, If no one knows about them how do you know I have them? Because
even okay software has to work with terrible software. The number of people whose
job it is to make software secure can practically fit in a large bar, and Ive watched
them drink. Its not comforting. It isnt a matter of if you get owned, only a matter
of when.
29
Figure1.This is a thing that actually happened several years

2
ago. To get rid of a complaining message from another piece
of software, a Debian developer just commented out a line of
code without realizing that it left their encryption open to easy
attack (https://www.xkcd.com/424/)
Look at it this wayevery time you get a security update (seems almost daily on my
Linux box), whatever is getting updated has been broken, lying there vulnerable, for
who-knows-how-long. Sometimes days, sometimes years. Nobody really advertises
that part of updates. People say You should apply this, its a critical patch! and
leave off the because the developers fucked up so badly your childrens identities
are probably being sold to the Estonian Mafia by smack addicted script kiddies right
now.
https://www.schneier.com/blog/archives/2008/05/random_number_b.html
30
The really bad bugs (and who knows which ones those are when they click the
Restart Later button?) can get swept up by hackers, governments, and other
horrors of the net that are scanning for versions of software they know they can
exploit. Any computer that shows up in a scan saying Hey! Me! Im vulnerable!
can become part of a botnet, along with thousands, or hundreds of thousands of
other computers. Often zombied computers get owned again and become part of yet
another botnet. Some botnets patch computers to throw out the other botnets so they
dont have to share you with other hackers. How can you tell if this is happening? You
cant! Have fun wondering if youre getting your online life rented out by the hour!
Next time you think your grandma is uncool, give her credit for her time helping
dangerous Russian criminals extort money from offshore casinos with DDoS attacks.
31
Figure2.A map of things which were hacked for the Internet

Census.
Recently an anonymous hacker wrote a script that took over embedded Linux
3
devices. These owned computers scanned the whole rest of the internet and
created a survey that told us more than wed ever known about the shape of the
internet. The little hacked boxes reported their data back (a full 10 TBs) and quietly
deactivated the hack. It was a sweet and useful example of someone who hacked
the planet to shit. If that malware had actually been malicious, we would have been
so fucked.
This is because all computers are reliably this bad: the ones in hospitals and
governments and banks, the ones in your phone, the ones that control light switches
and smart meters and air traffic control systems. Industrial computers that maintain
infrastructure and manufacturing are even worse. I dont know all the details, but
those who do are the most alcoholic and nihilistic people in computer security.
Another friend of mine accidentally shut down a factory with a malformed ping at
the beginning of a pen test. For those of you who dont know, a ping is just about the
smallest request you can send to another computer on the network. It took them a
day to turn everything back on.
Computer experts like to pretend they use a whole different, more awesome class of
software that they understand, that is made of shiny mathematical perfection and
whose interfaces happen to have been shat out of the business end of a choleric
donkey. This is a lie. The main form of security this offers is through obscurity so
3
http://internetcensus2012.bitbucket.org/paper.html
32
few people can use this software that theres no point in building tools to attack it.
Unless, like the NSA, you want to take over sysadmins.
A well written encrypted chat, what could go

wrong?
Lets take an example computer experts like to stare down their noses at normal
people for not using: OTR. OTR, or Off The Record messaging, sneaks a layer of
encryption inside normal plain text instant messaging. Its like you got on AIM or
Jabber or whatever and talked in code, except the computer is making the code for
you. OTR is clever and solid, its been examined carefully, and were fairly sure it
hasnt got any of those nasty 0days.
Except, OTR isnt a program you use, as such.
There is a standard for OTR software, and a library, but it doesnt do anything on
its own. It gets implemented in software for normal human shlubs to use by other
normal human shlubs. By now, you know this ends in tears.
The main thing that uses OTR is another piece of software that uses a library called
4
libpurple . If you want to see infosec snobs look as distressed as the donkeys that
shit out their interfaces, bring up libpurple. Libpurple was written in a programming
language called C.
C is good for two things: being beautiful and creating catastrophic 0days in memory
management.
https://developer.pidgin.im/wiki/WhatIsLibpurple
33
Figure3.Heartbleed attack
Heartbleed , the bug that affected the world over, leaking password and encryption
keys and who knows what? Classic gorgeous C.
5
http://xkcd.com/1354/
6
http://heartbleed.com/
34
Libpurple was written by people who wanted their open source chat client to talk to
every kind of instant messaging system in the world, and didnt give a shit about
security or encryption. Security people who have examined the code have said there
are so many possible ways to exploit libpurple there is probably no point in patching
it. It needs to be thrown out and rewritten from scratch. These arent bugs that let
someone read your encrypted messages, they are bugs that let someone take over
your whole computer, see everything you type or read and probably watch you pick
your nose on your webcam.
This lovely tool, OTR, sits on top of libpurple on most systems that use it. Let me
make something clear, because even some geeks dont get this: it doesnt matter
how good your encryption is if your attacker can just read your data off the screen
with you, and I promise they can. They may or may not know how to yet, but they can.
There are a hundred libpurples on your computer: little pieces of software written on
a budget with unrealistic deadlines by people who didnt know or didnt care about
keeping the rest of your system secure.
Any one of these little bugs will do when it comes to taking over everything else on
your computer. So we update and update, and maybe that throws any intruders out,
and maybe it doesnt. No one knows!
When we tell you to apply updates we are not telling you to mend your ship. We are
telling you to keep bailing before the water gets to your neck.
To step back a bit from this scene of horror and mayhem, let me say that things
are better than they used to be. We have tools that we didnt in the 1990s, like
sandboxing, that keep the idiotically written programs where they cant do as much
harm. (Sandboxing keeps a program in an artificially small part of the computer,
cutting it off from all the other little programs, or cleaning up anything it tries to do
before anything else sees it.)
Certain whole classes of terrible bugs have been sent the way of smallpox. Security
is taken more seriously than ever before, and theres a network of people responding
to malware around the clock. But they cant really keep up. The ecosystem of these
problems is so much bigger than it was even ten years ago that its hard to feel like
were making progress.
35
People, as well, are broken.
I trust you was my least favorite thing to hear from my sources in Anonymous.
Inevitably it was followed by some piece of information they shouldnt have been
telling me. It is the most natural and human thing to share something personal with
someone you are learning to trust. But in exasperation I kept trying to remind Anons
they were connecting to a computer, relaying though countless servers, switches,
routers, cables, wireless links, and finally to my highly targeted computer, before
they were connecting to another human being. All of this was happening in the time
it takes one person to draw in a deep, committal breath. Its obvious to say, but bears
repeating: humans were not built to think this way.
Everyone fails to use software correctly. Absolutely everyone fucks up. OTR doesnt
encrypt until after the first message, a fact that leading security professionals
and hackers subject to 20-country manhunts consistently forget. Managing all the
encryption and decryption keys you need to keep your data safe across multiple
devices, sites, and accounts is theoretically possible, in the same way performing
36
an appendectomy on yourself is theoretically possible. This one guy did it once in
7
Antarctica , why cant you?
Every malware expert I know has lost track of what some file is, clicked on it to
see, and then realized theyd executed some malware they were supposed to be
examining. I know this because I did it once with a PDF I knew had something bad in
it. My friends laughed at me, then all quietly confessed theyd done the same thing.
If some of the best malware reversers around cant keep track of their malicious files,
what hope do your parents have against that e-card that is allegedly from you?
Executable mail attachments (which includes things like Word, Excel, and PDFs) you
get just about everyday could be from anyone people can write anything they want
in that From: field of emails, and any of those attachments could take over your
computer as handily as an 0day. This is probably how your grandmother ended up
working for Russian criminals, and why your competitors anticipate all your product
plans. But if you refuse to open attachments you arent going to be able to keep
an office job in the modern world. Theres your choice: constantly risk clicking on
dangerous malware, or live under an overpass, leaving notes on the lawn of your
former house telling your children you love them and miss them.
Security and privacy experts harangue the public about metadata and networked
sharing, but keeping track of these things is about as natural as doing blood panels on
yourself every morning, and about as easy. The risks on a societal level from giving
up our privacy are terrible. Yet the consequences of not doing so on an individual
basis are immediately crippling. The whole thing is a shitty battle of attrition between
7
http://www.southpolestation.com/trivia/igy1/appendix.html
37
what we all want for ourselves and our families and the ways we need community to
survive as humans a Mexican stand off monetized by corporations and monitored
by governments.
I live in this stuff, and Im no better. Once I had to step through a process to verify
myself to a secretive source. I had to take a series of pictures showing my location
and the date. I uploaded them, and was allowed to proceed with my interview. It
turns out none of my verification had come through, because Id failed to let the
upload complete before nervously shutting down my computer. Why did you let me
through? I asked the source. Because only you would have been that stupid, my
source told me.
Touch.
But if I cant do this, as a relatively well trained adult who pays attention to these
issues all the damn time, what chance do people with real jobs and real lives have?
In the end, its culture thats broken.

A few years ago, I went to several well respected people who work in privacy and
security software and asked them a question.
38
First, I had to explain something:
Most of the world does not have install privileges on the computer they
are using.
That is, most people using a computer in the world dont own the computer they
are using. Whether its in a cafe, or school, or work, for a huge portion of the world,
installing a desktop application isnt a straightforward option. Every week or two, I
was being contacted by people desperate for better security and privacy options,
and I would try to help them. Id start, Download th and then wed stop. The next
thing people would tell me was that they couldnt install software on their computers.
Usually this was because an IT department somewhere was limiting their rights as a
part of managing a network. These people needed tools that worked with what they
had access to, mostly a browser.
So the question I put to hackers, cryptographers, security experts, programmers, and
so on was this: Whats the best option for people who cant download new software
to their machines? The answer was unanimous: nothing. They have no options. They
are better off talking in plaintext I was told, so they dont have a false sense of
security. Since they dont have access to better software, I was told, they shouldnt
do anything that might upset the people watching them. But, I explained, these are
the activists, organizers, and journalists around the world dealing with governments
and corporations and criminals that do real harm, the people in real danger. Then
they should buy themselves computers, I was told.
That was it, that was the answer: be rich enough to buy your own computer, or
literally drop dead. I told people that wasnt good enough, got vilified in a few
inconsequential Twitter fights, and moved on.
Not long after, I realized where the disconnect was. I went back to the same experts
and explained: in the wild, in really dangerous situations even when people are
being hunted by men with guns when encryption and security fails, no one stops
talking. They just hope they dont get caught.
The same human impulse that has kept lotteries alive for thousands of years keeps
people fighting the man against the long odds. Maybe Ill get away with it, might
as well try!
As for self-censoring their conversations in the face of hostile infrastructure, nontechnical activists are just as good at it as Anons are, or people told to worry about
metadata, or social media sharing, or that first message before OTR encryption kicks
in. They blow.
39
This conversation was a wake-up call for some security people who hadnt realized
that people who become activists and journalists routinely do risky things. Some
of them joined my side of the time-wasting inconsequential Twitter fights, realizing
that something, even something imperfect, might be better than nothing. But many
in the security scene are still waiting for a perfect world into which to deploy their
perfect code.
Then theres the Intelligence Community, who call themselves the IC. We might like
it if they stopped spying on everyone all the time, while they would like us to stop
whining about it.
After spending some time with them, I am pretty sure I understand why they dont
care about the complaining. The IC are some of the most surveilled humans in history.
They know everything they do is gone over with a fine-toothed comb by their peers,
their bosses, their lawyers, other agencies, the president, and sometimes Congress.
They live watched, and they dont complain about it.
In all the calls for increased oversight, the basics of human nature gets neglected.
Youre not going to teach the spooks this is wrong by doing it to them more.
There will always be loopholes and as long as loopholes exist or can be constructed
or construed, surveillance will be as prevalent as it possibly can be. Humans are
mostly egocentric creatures. Spooks, being humans, are never going to know why
living without privacy is bad as long as they are doing it.
Yet thats the lesser problem. The cultural catastrophe is what theyre doing to make
their job of spying on everyone easier. The most disturbing parts of the revelations
are the 0day market, exploit hoarding, and weakening of standards. The question is
who gets to be part of the we that are being kept allegedly safe by all this exploiting
40
and listening and decrypting and profiling. When they attacked Natanz with Stuxnet
and left all the other nuclear facilities vulnerable, we were quietly put on notice that
the we in question began and ended with the IC itself. Thats the greatest danger.
When the IC or the DOD or the Executive branch are the only true Americans, and the
rest of us are subordinate Americans, or worse the non-people that arent associated
with America, then we can only become lesser people as time goes on.
As our desires conflict with the IC, we become less and less worthy of rights and
considerations in the eyes of the IC. When the NSA hoards exploits and interferes
with cryptographic protection for our infrastructure, it means using exploits against
people who arent part of the NSA just doesnt count as much. Securing us comes
after securing themselves.
In theory, the reason were so nice to soldiers, that we have customs around honoring
and thanking them, is that theyre supposed to be sacrificing themselves for the
good of the people. In the case of the NSA, this has been reversed. Our wellbeing is
sacrificed to make their job of monitoring the world easier. When this is part of the
culture of power, it is well on its way to being capable of any abuse.
But the biggest of all the cultural problems still lies with the one group I havent taken
to task yet the normal people living their lives under all this insanity.
The problem with the normals and tech is the same as the problem with the normals
and politics, or society in general. People believe they are powerless and alone, but
the only thing that keeps people powerless and alone is that same belief. People,
working together, are immensely and terrifyingly powerful.
There is certainly a limit to what an organized movement of people who share a
mutual dream can do, but we havent found it yet.
Facebook and Google seem very powerful, but they live about a week from total
ruin all the time. They know the cost of leaving social networks individually is high,
but en masse, becomes next to nothing. Windows could be replaced with something
better written. The US government would fall to a general revolt in a matter of
days. It wouldnt take a total defection or a general revolt to change everything,
because corporations and governments would rather bend to demands than die.
These entities do everything they can get away withbut weve forgotten that were
the ones that are letting them get away with things.
Computers dont serve the needs of both privacy and coordination not because
its somehow mathematically impossible. There are plenty of schemes that could
41
federate or safely encrypt our data, plenty of ways we could regain privacy and make
our computers work better by default. It isnt happening now because we havent
demanded that it should, not because no one is clever enough to make that happen.
So yes, the geeks and the executives and the agents and the military have fucked
the world. But in the end, its the job of the people, working together, to unfuck it.
42
Dont go live with simple security

problems - 10 tips to help
Written by: eviltester at Tuesday, 23 July 2013
I feel anger when I stumble across very, very, very simple security issues. Especially
when they compromise my data.
Yes I do. And I hope, as a tester, that you do too.
But I face a problem As a tester, I cant say "Did no-one test this!" because I know
that they might have done, and someone else might have chosen to go live anyway.
But on the off chance that no-one did "test this", I offer you this post.
Security by obscurity
If I visit your site and I can gain access to top level pages that I shouldnt have, then
I get angry, because that should never happen.
Even if you havent told me about the URL, I can try to guess it from your other
naming conventions.
Please make sure you secure your URLs.
Security by obscurity doesnt work for very long.
Validate Parameters on the server

And if I see parameters in your URLs. Ill change them.
Yes I will.
I will:
Because I dont want the list of items to stay limited at 25, I want to see 2500
No I dont care about the performance impact on your database - fix that a
different way
1
http://blog.eviltester.com/2013/07/dont-go-live-with-simple-security.html
43
Dont go live with simple security problems - 10 tips to help

Because I want to skip ahead more pages than you have listed on the page. "I
want to go to page 15 now!"
No I dont care about the user experience you want me to have, I care about
the user experience that I want to have
Because I can
Yes, because I can
Security by ignorance
When I visit your site, I look at the traffic you issue when I hit a top level URL.
I look at what requests you make to build the page.
Yes I do.
Most browsers have the functionality to view traffic built in now. Any user can view
the traffic and see the URLs that you access.
And then I use that information
I take the URLs youve used. And I use them.
Sometimes I change them. Simple idea, but sadly all too effective.
So if you access
http://site/api/123456/report (note: not a real domain)
Ill access
http://site/api/123455/report (note: I changed the number)
Yes I will.
And if you havent locked down the API to specific users, with specific role level
controls. Then Ill probably see another users report. Then I get annoyed, because
as a user it means that other people can see my reports. And I dont like that.
No I dont.
Assume that anything you do automatically someone else will do manually.
Just because they can.
44

Make sure you have low level permissions on your API, dont assume that no-one
will notice it.
I frequently do this because I want to bypass the horrendous GUIs that web sites put
in my face, when I want to achieve a result, rather than experience your broken GUI.
So I script it. Or if I can get away by posting a URL with some data, then Ill do it.
I bet other people do this too.
Testers should do this too, because
Security by ignorance doesnt work.
Security through sequential numbering

And if youre using sequential IDs for reports, or users, or accounts, etc. you actively
encouraged people to hack you.
Yes you did.
No-one has ever recommended - Security through Sequential numbering.
No they havent. Never Ever.
Security through sequential numbering doesnt work.
Tips for Testing

So now, the inevitable 10 tips for testing:
1. Play with the URLs
2. Change URL Parameters
a. to check that permissions surround the public level
b. to check that request validation takes place
3. Try URLs when logged out to make sure permissions apply
4. Guess some URLs
5. Use an HTTP Debug proxy and look at the traffic
6. Investigate the traffic and see what the requests do
7. Issue the traffic requests out of context on a page to understand the 'real' state
rules in place
45

8. Change the URL parameters in the traffic URLs
a. to check that permissions surround the AP
b. to check that request validation acts at the API level, not just the GUI level
9. Issue the requests when logged out to check the permissions still apply
10.And if you do test like this, and your organisation keeps ignoring these types of
defects, check if you reported them effectively, and if you did, then leave because
that company doesnt deserve you.
You wouldnt like me when Im Angry

I didnt even describe security testing above. I described functionality testing.
And really basic functionality testing at that, just simple input variations. I havent
messed with cookies, I havent done anything hard (because cookie editing aint
easy, 'right kids).
If you dont include this "really really simple stuff" level of test activity, then please
let me know so that I can avoid your site and find a competitor quickly before we
develop a user/supplier relationship.
I really dont like getting angry when I act as a user.
Trust me, you wouldnt want me as an Angry user.
P. S.:
Yes, this blog post does describe problems found at a specific web site.
No I will not name that site.
Yes, I have already told them and more than once.
Yes I have started looking at alternatives, sigh.
46
Extreme Programming, a Reflection

Written by: Robert Martin at 2013-12-10
In my hand I am holding a little white book that, fourteen years ago, changed the
software world forever. The title of that book is: Extreme Programming Explained;
and the subtitle is: Embrace Change. The author is Kent Beck, and the copyright date
is 1999.
The book is small, less than 200 pages. The print is large and widely spaced. The
writing style is casual and accessible. The chapters are short. The concepts are
simple.
The implications were an Earthquake whose tremors havent even begun to die down.
Chapter 10, which begins on page 53, lays out the 12 practices that caused our
industry to erupt into controversy; and spawned a revolution that has changed
everything about the way we write software. Those practices are:
The Planning Game: Nowadays known as SCRUM. The idea that software is
produced in short increments from a prioritized list of work items.
Small Releases: The notion that deployments should be frequent and incremental.
Metaphor: Finally crystalized by Eric Evans in his book Domain Driven Design. The
notion that the structure of the system is based upon a simple mental model of
the problem domain.
Simple Design: The notion that it is best to keep the system as simple as possible
at all times regardless of what we fear about the future.
Testing: The notion that programmers, and customers, write automated tests
that verify that the production code actually does what they think it should.
Nowadays we call this Test Driven Development (TDD) and Acceptance Test
Driven Development (ATDD).
Refactoring: The notion that the internal structure of software can, and should,
be continuously improved.
Pair Programming: The notion that members of a team cannot be a team if they
work separately. To be a team they must regularly collaborate, at the keyboard. In
1
http://blog.8thlight.com/uncle-bob/2013/12/10/Thankyou-Kent.html
47

so doing they share knowledge sufficient to cover for each other as team members
should.
Collective Ownership: The notion that the code belongs to the team, not to the
individual.
40 Hour week: The notion that teams who consistently work overtime are failing.
On Site Customer: The notion that someone from the business, who is responsible
for requirements, must be readily and consistently available to the programming

team.
Coding Standards: The notion that the team adopts a consistent style in their code
emphasizing cleanliness and communication.
Controversial?
Strange isnt it? This doesnt seem all that controversial does it? But fourteen years
ago it was wildly controversial. Indeed, it was so controversial that whole books were
published describing how this couldnt possibly work, and how all the proponents
were knuckle-dragging, money-grubbing, nitwits who never wrote a line of code in
their lives and.
Ah, but I shouldnt let those old feelings overtake me Because, after all, theyre
gone and were still here.
Look at those twelve practices. Which ones dont you do? Most of you, my gentle
readers, likely do most of these practices on a regular basis. While its certainly a
stretch to say that they have become universal, it is by no means a stretch to say
that they are now considered main-stream. Whats more, those teams that dont do
all these practices today, are trying to move towards them. These practices have
become an ideal, a goal to be achieved as opposed to a heresy to be reviled.
The Churn
The last fourteen years have been strange. The Agile movement, which was spawned
out of the controversy over Extreme Programming, skyrocketed into success, and
was subsequently taken over by the project managers who all but pushed the
programmers out. Weve seen the creation, the wild success, and the corresponding
(and predictable) impotence, of certifications. We saw the adoption of the planning
game (i.e. SCRUM) without the other eleven practices; and we saw that strategy
fail becoming what Martin Fowler called: Flaccid Scrum. Weve experienced
48

continuous and vocal process churn as consultants and authors split and competed
over Kanban, Lean, and every new project-management prefix-of-the-day. Weve
seen the growth of the software craftsmanship movement, and the slow degradation
and dilution of the Agile meme.
But in the midst of all that hype and churn, those twelve practices have remained.
Some of their names have changed a bit. 40 Hour Week became Sustainable Rate.
Testing became TDD. Metaphor became DDD. Small Releases became Continuous
Integration and Continuous Deployment. But despite these changes the practices
remain very much as they were described fourteen years ago.
We also saw the name Extreme Programming fade almost entirely out of use. Very
few people use that term nowadays. Some still use the abbreviation XP; but for the
most part the name has evaporated. It is very rare for me to hear a team describe
what they do as Extreme Programming, even when they are practicing all twelve
practices as described. The names change. The practices remain. The practices are
persistent.
Amidst the churn, the hype, the controversy, the bluster and blather. Amidst all the
chaos of humans jockeying for position over one-another. Amidst all the messiness
of human avarice, passion, and pride. Amidst all that politics, the practices persist.
Stable Values
I believe the practices persist because they are based on a firm foundation of stable
values. Values that Kent Beck described in Chapter 7 on page 29 of his book:
Communication
Simplicity
Feedback
Courage.
I could try to argue why these are the right values; but I think they speak for
themselves. What software craftsman would reject any one of those values? What
software craftsman would not strive to ensure that each one of those values were
represented in their work? These values are values of software craftsmanship.
I could try to argue that the twelve practices embrace and exemplify these
values, but their persistence despite the churn and dissolution of the names and
movements that surrounded them, is evidence enough.
49
Success
Extreme Programming succeeded! It succeeded beyond the wildest dreams of its
proponents. It succeeded because it survived the controversy of its birth and the
subsequent, and inevitable, churn of its advocacy. It succeeded because it outlived
even its own name!
Extreme Programming has succeeded in the way that Structured Programming
succeeded. Nobody even thinks about structured programming any more they just
do it. Nobody even thinks about Extreme Programming any more, we are all just
trying to do it.
Thats success! An idea succeeds when it outlives the movement that spawns it and
simply becomes part of our everyday lives. Thats SUCCESS!
Looking Back
So today, in these last weeks of 2013, take a moment to reflect back on 1999. A time
when Kent Beck wrote a ground-breaking book. A book that changed everything.
Look back and remember: Extreme Programming; and recognize it as the core of
what we, today, simply think of as:
Good Software Practice.
50
Surviving Legacy Code With Golden

Master and Sampling
Written by: J. B. Rainsberger at 2014-09-28
You have inherited some code. Congratulations. Now you need to change it.
There, there.
Michael Feathers once wrote that legacy code is code without unit tests. I use a
slightly more general definition.
Legacy code is valuable code that we feel afraid to change.
I think that both parts matter. You probably accepted the afraid to change part
without any need for convincing. (If not, then this article probably wont interest you.)
Moreover, if the code doesnt generate significant value, then I dont see much risk in
changing it. If the cost of getting it wrong doesnt significantly outweigh the profit
we derive from getting it right, then who cares? Probably not I.
I treat valuable code with considerable respect. It provides food for families. I treat
difficult-to-change code also with consider respect, although this comes more from
fear than admiration. If we put these two things together, then, quite simply, one
false move and I might destroy an order of magnitude more profit than the yearly
cost to keep me around.
This brings me to Rule Number Zero of Surviving Legacy Code:
Maximise safety.
We find ourselves in the typical chicken-and-egg problem: we want to write tests

in order to refactor more safely, but then we remember that integrated tests are a
2
scam and decide that wed rather break things apart a little in order to write lesscostly-to-maintain tests. So which do we do first?
1
http://blog.thecodewhisperer.com/2014/09/28/surviving-legacy-code-with-golden-master-andsampling/
2
You can find a series of articles on that topic at http://link.jbrains.ca/17NH9w3
51
Surviving Legacy Code With Golden Master and Sampling

In a situation like this, I like to go back to my guiding principles.
Integrated tests are a scam in part because they dont put enough positive pressure
on my designs and thereby dont give me enough useful design feedback. Right now,
I dont care about this. I already know that the design needs significant work. I also
know that I cant handle the torrent of feedback that microtests would give me about
34
the design. If I want to use this principle to guide my behavior, then I need to find
another justification.
Integrated tests remain a scam in part because of the combinatoric explosion

in the number of tests I need to achieve a strong level of coverage, which in this
case correlates to confidence. I might have to write millions of tests to achieve high
coverage. I probably only have time to write hundreds of tests, in which case I have
to gamble about the level of coverage. Perchance, could I not care about coverage
in this situation?
Test coverage however one measures or defines it links directly to safety in
changing code. I want to use those tests as change detectors. I want the red light
that flashes the moment I make a mistake. Microtests, especially if I write them first,
give me that. They help me find mistakes immediately. They help drive down the
5
cost of making a mistake, an essential technique for managing risk. If I cant write
microtests cost-effectively, then what can I do?
What if, instead of a red light that flashes the moment I make (almost) any mistake,
I had a pink light that flashes when I make a really obvious mistake? I cant have
what I want, but I can afford this; will it do? It will help more than doing nothing. I
will simply buy as much of this confidence as I can afford. To do this, I combine two
simple ideas: Golden Master and sampling.
When diving into legacy code, I find it more important than ever to keep stuff out of my head. During
the two hours it takes to safely refactor some large function, Im probably going to spot 14 potentiallyuseful refactorings. I cant chase every bunny, no matter how cute they are. I need to write those ideas
down, get them out of my head, and get back to the tricky surgery at hand.
4
I see little point in spending energy generating a backlog knowing full well that I will never get around
to doing about 80% of it. Who would volunteer to do that? (Ask your project manager if value-driven
product [http://www.jbrains.ca/training/value-driven-product-development] development is right for
em.)
5
I claim that the agile approach to risk management complements the typical approach to risk
management of limiting the probability of failure in order to limit exposure. The agile way, if you will
permit me to use this shorthand, involves limiting the cost of failure instead. Eventually I will replace
this sentence with a link to an article that goes into this topic in more detail.
52
Golden Master
I use Golden Master to help me detect changes in the behavior of a system when I
cant justify writing the typical kind of assertion that youve grown used to seeing in
tests. I use this trick, for example, when I find it difficult to articulate the expected
result of a test. Imagine a function whose output consists of an image. It happens
quite often that a binary comparison between actual and expected result yields a
hyperactive assertionone which frequently fails even when a human would judge
that the test had passed. I suppose some people know tricks to make it easier to
articulate looks similar enough for images, but I dont know how to do that, and that
leaves me to choose either a hyperactive bit-by-bit comparison or ongoing, manual
6
inspection. Rather than revert to the Guru Checks Output antipattern , however, I
take a snapshot of the last-known acceptable outputI call that the golden master
and save it for future use. When I run the test again, I compare the output to the
golden master, and if they match, then the test passes; if they dont match, then
the test fails. This doesnt make the code wrong, but it means that I need to check
the result and decide whether the code needs fixing or the golden master needs
replacing.
You can use Golden Master wherever you already have some output to check, even
if you find the form of that output particularly challenging. With this technique,
you simply diff the output and inspect the situation only when you find differences
between the current test run and the golden master. If your system already sends
text to an output stream that you can capture, then you have the tools to use this
technique.
I warn you, however, not to give in to the temptation to start scraping your output
for specific information to check. Unless you have no other alternative, you will
probably find it more cost-effective to carefully extract that information from the
code and check it directly using good, old-fashioned assertEquals() . Dont build
a parser for an arbitrary, unplanned, probably context-sensitive grammar. That way
lies madness. (Of course, if a context-free grammar happens to describe the format,
then go for it. Youve always wanted to learn lexx and yacc , havent you?)
Marcia, the guru, looks at the output, pauses for a moment, then says, Yep. Thats it. If you want to
re-run the test, then you need Marcia. That doesnt seem to scale particularly well.
53
Sampling
I find one glaring problem with the Golden Master technique: if the output describes a
long-running algorithm, process, or path through the system, then the golden master
itself might describe only one of a thousand, million, or even billion potentiallyinteresting possible outputs. Welcome back to the combinatoric explosion problem
that makes integrated tests such a scam. How do we proceed when we cant
possibly check the variety of paths through the system that we need to check?
Ideally, we refactor! I know that if I can break my system into many smaller,
composable pieces, then I turn products into sums: instead of checking combinations
of paths through multiple parts of the system at once, I can check the handful of
pairwise connexions between parts of the system in relative isolation. I could turn
millions of tests into hundreds. Unfortunately, in our current situation, I dont feel
comfortable refactoring, so that means that I have to sample the inputs and hope
for the best.
You can find more sophisticated sampling systems out there among blogs written by
experienced testers, but they all amount to sampling: if I cant try every combination
of inputs, then I try some combinations of some of the inputs and aim for the best
coverage that I can.
This shouldnt surprise you. Youve done this before. Youve written a function
that operates on an integer, and you knew enough about the algorithm to identify
boundary cases at, for example, -1, 0, and 1, as well as around 100 and 1000, so you
check on the order of ten inputs and feel satisfied that the algorithm will work for the
remaining few billion inputs. You were sampling.
In the case of legacy code, however, sometimes we cant sample quite so
intentionally. Sometimes even when we limit our scope to characteristic inputs, we
have so many combinations of those inputs that we still cant afford to write and run
all those tests. In some cases, we dont even know how to identify the characteristic
inputs. In other cases, the algorithm itself has a random element, defeating our goal
of writing nice, deterministic, repeatable tests. Random sampling to the rescue.
If you can use the random number generator to generate a stream of inputs to
your system, then you can use this generate a collection of output files, and that
collection can act as your golden master. You only need to control the random
number generator by seeding it with the same stream of seeds every time. I use
a simple linear generating function like m + p * i where m and p represent
arbitrarily-chosen numbers and i represents a loop index. Now I simply have to
54

decide how big a sample to take. Generally speaking, a larger sample gives me more
confidence in the sensitivity of the pink flashing light that signals danger.
I adjust the size of the sample depending on how long it takes to execute a test run,
and how much significantly that affects my flow while programming. I also adjust
the size of the sample to match my fear level: the more worried I feel about getting
something wrong, the larger sample I take while working, and I accept the cost of
slowing down. Id usually rather go a little too slow than a little too fast, because
I know that the cost of making a mistake would likely dominate the savings from
going more quickly.
The Techniques in Action

7
You can see an example of this technique in action by reading this code . If youd
8
like to see how I added this behavior to some legacy code, then start at this commit
and follow the process step by step.
Although these techniques do not, on their own, guarantee success, when I combine
Golden Master and Sampling, I can usually find a way to proceed safely. When I
9
combine these with microcommitting , I can proceed at an even quicker pace. They
help me avoid the Catch-22 problem that arises from needing to refactor dangerously
unsafely in order to be able to refactor safely and sensibly. Where might you use
Golden Master and Sampling to help get your arms (at least a little) around your
legacy code?
References
Michael Feathers, Working Effectively with Legacy Code
on winding your way through legacy code.
10
. Still the classic work

11
J. B. Rainsberger, Getting Started with Getting Things Done . You dont have
time to read Getting Things Done? Start here. Four pages. Itll be fine.
7
https://github.com/jbrains/SurvivingLegacyCode-solutions/
tree/08bb50de492cd5edc089dd358693bd00652306bd
8
https://github.com/jbrains/SurvivingLegacyCode-solutions/
tree/110dc1308c05a7c43a1d991c66f7dae7633e921a
9
Really frequent committing, like after changing a single line of code. No, really. Eventually I will
replace this sentence with a reference to an article that explores this topic in more detail.
10
http://www.amazon.com/gp/product/0131177052/ref=as_li_qf_sp_asin_il_tl?
ie=UTF8&tag=jbrains.ca-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131177052
11
http://articles.jbrains.ca/GettingStartedWithGettingThingsDone.pdf
55

12
David Allen, Getting Things Done . I use it. Not all of it, and not all the time, but
I use its core principles quite significantly in managing my work and home lives.
No cult, I promise.
13
J. B. Rainsberger, A Proven Path to Effectiveness . A method that combines

Getting Things Done and Test-Driven Development aimed specifically at the
beleaguered programmer.
14
texttest.org . A library to help you write text-based tests, such as I would use to
provide golden masters. Do not download this tool until you have written
your own golden master at least once. That is an order. After that, use
TextTest, because it really helps.
12
ie=UTF8&camp=1789&creative=9325&creativeASIN=0142000280&linkCode=as2&tag=jbrains.ca-20
13
ie=UTF8&camp=1789&creative=9325&creativeASIN=0142000280&linkCode=as2&tag=jbrains.ca-20
14
http://texttest.sourceforge.net/
56
Hexagonal Architecture
Written by: Alistair. Cockburn at 2005-01-04
Create your application to work without either a UI or a database so you can

run automated regression-tests against the application, work when the database
becomes unavailable, and link applications together without any user involvement.
The Pattern: Ports and Adapters (Object

Structural)
Alternative name: Ports & Adapters
Alternative name: Hexagonal Architecture
Intent
Allow an application to equally be driven by users, programs, automated test or batch
scripts, and to be developed and tested in isolation from its eventual run-time devices
and databases.
As events arrive from the outside world at a port, a technology-specific adapter
converts it into a usable procedure call or message and passes it to the application.
The application is blissfully ignorant of the nature of the input device. When the
1
http://alistair.cockburn.us/Hexagonal+architecture
57
application has something to send out, it sends it out through a port to an adapter,
which creates the appropriate signals needed by the receiving technology (human or
automated). The application has a semantically sound interaction with the adapters
on all sides of it, without actually knowing the nature of the things on the other side
of the adapters.
Motivation
One of the great bugaboos of software applications over the years has been
infiltration of business logic into the user interface code. The problem this causes
is threefold:
First, the system cant neatly be tested with automated test suites because part
of the logic needing to be tested is dependent on oft-changing visual details such
as field size and button placement;
For the exact same reason, it becomes impossible to shift from a human-driven
use of the system to a batch-run system;
For still the same reason, it becomes difficult or impossible to allow the program
to be driven by another program when that becomes attractive.
The attempted solution, repeated in many organizations, is to create a new layer in
the architecture, with the promise that this time, really and truly, no business logic
58
will be put into the new layer. However, having no mechanism to detect when a
violation of that promise occurs, the organization finds a few years later that the new
layer is cluttered with business logic and the old problem has reappeared.
Imagine now that every piece of functionality the application offers were available
through an API (application programmed interface) or function call. In this situation,
the test or QA department can run automated test scripts against the application
to detect when any new coding breaks a previously working function. The business
experts can create automated test cases, before the GUI details are finalized, that
tells the programmers when they have done their work correctly (and these tests
become the ones run by the test department). The application can be deployed in
headless mode, so only the API is available, and other programs can make use
of its functionality this simplifies the overall design of complex application suites
and also permits business-to-business service applications to use each other without
human intervention over the web. Finally, the automated function regression tests
detect any violation of the promise to keep business logic out of the presentation
layer. The organization can detect, and then correct, the logic leak.
An interesting similar problem exists on what is normally considered the other side
of the application, where the application logic gets tied to an external database
or other service. When the database server goes down or undergoes significant
rework or replacement, the programmers cant work because their work is tied to the
presence of the database. This causes delay costs and often bad feelings between
the people.
It is not obvious that the two problems are related, but there is a symmetry between
them that shows up in the nature of the solution.
Nature of the Solution

Both the user-side and the server-side problems actually are caused by the same
error in design and programming the entanglement between the business logic
and the interaction with external entities. The asymmetry to exploit is not that
between left and right sides of the application but between inside and
outside of the application. The rule to obey is that code pertaining to the inside
part should not leak into the outside part.
Removing any left-right or up-down asymmetry for a moment, we see that the
application communicates over ports to external agencies. The word port is
supposed to evoke thoughts of ports in an operating system, where any device that
59
adheres to the protocols of a port can be plugged into it; and ports on electronics
gadgets, where again, any device that fits the mechanical and electrical protocols
can be plugged in.
The protocol for a port is given by the purpose of the conversation between the two
devices.
The protocol takes the form of an application program interface (API).
For each external device there is an adapter that converts the API definition to
the signals needed by that device and vice versa. A graphical user interface or GUI
is an example of an adapter that maps the movements of a person to the API of
the port. Other adapters that fit the same port are automated test harnesses such
as FIT or Fitnesse, batch drivers, and any code needed for communication between
applications across the enterprise or net.
On another side of the application, the application communicates with an external
entity to get data. The protocol is typically a database protocol. From the
applications perspective, if the database is moved from a SQL database to a flat file
or any other kind of database, the conversation across the API should not change.
Additional adapters for the same port thus include an SQL adapter, a flat file adapter,
and most importantly, an adapter to a mock database, one that sits in memory
and doesnt depend on the presence of the real database at all.
Many applications have only two ports: the user-side dialog and the database-side
dialog. This gives them an asymmetric appearance, which makes it seem natural
to build the application in a one-dimensional, three-, four-, or five-layer stacked
architecture.
There are two problems with these drawings. First and worst, people tend not to take
the lines in the layered drawing seriously. They let the application logic leak across
the layer boundaries, causing the problems mentioned above. Secondly, there may
be more than two ports to the application, so that the architecture does not fit into
the one-dimensional layer drawing.
The hexagonal, or ports and adapters, architecture solves these problems by noting
the symmetry in the situation: there is an application on the inside communicating
over some number of ports with things on the outside. The items outside the
application can be dealt with symmetrically.
The hexagon is intended to visually highlight
60
1. the inside-outside asymmetry and the similar nature of ports, to get away from
the one-dimensional layered picture and all that evokes, and
2. the presence of a defined number of different ports two, three, or four (four is
most I have encountered to date).
The hexagon is not a hexagon because the number six is important, but rather to
allow the people doing the drawing to have room to insert ports and adapters as
they need, not being constrained by a one-dimensional layered drawing. The term
hexagonal architecture comes from this visual effect.
The term port and adapters picks up the purposes of the parts of the drawing.
A port identifies a purposeful conversation. There will typically be multiple adapters
for any one port, for various technologies that may plug into that port. Typically,
these might include a phone answering machine, a human voice, a touch-tone phone,
a graphical human interface, a test harness, a batch driver, an http interface, a
direct program-to-program interface, a mock (in-memory) database, a real database
(perhaps different databases for development, test, and real use).
In the Application Notes, the left-right asymmetry will be brought up again. However,
the primary purpose of this pattern is to focus on the inside-outside asymmetry,
pretending briefly that all external items are identical from the perspective of the
application. Structure
61
Figure 2 shows an application having two active ports and several adapters for each
port. The two ports are the application-controlling side and the data-retrieval side.
This drawing shows that the application can be equally driven by an automated,
system-level regression test suite, by a human user, by a remote http application,
or by another local application. On the data side, the application can be configured
to run decoupled from external databases using an in-memory oracle, or mock,
database replacement; or it can run against the test- or run-time database. The
functional specification of the application, perhaps in use cases, is made against the
inner hexagons interface and not against any one of the external technologies that
might be used.
Figure 3 shows the same application mapped to a three-layer architectural drawing.

To simplify the drawing only two adapters are shown for each port. This drawing
is intended to show how multiple adapters fit in the top and bottom layers, and
the sequence in which the various adapters are used during system development.
The numbered arrows show the order in which a team might develop and use the
application:
62
1. With a FIT test harness driving the application and using the mock (in-memory)
database substituting for the real database;
2. Adding a GUI to the application, still running off the mock database;
3. In integration testing, with automated test scripts (e.g., from Cruise Control)
driving the application against a real database containing test data;
4. In real use, with a person using the application to access a live database.
Sample Code
The simplest application that demonstrates the ports & adapters fortunately comes
with the FIT documentation. It is a simple discount computing application:
discount(amount) = amount * rate(amount);
In our adaptation, the amount will come from the user and the rate will come from a
database, so there will be two ports. We implement them in stages:
With tests but with a constant rate instead of a mock database,
then with the GUI,
then with a mock database that can be swapped out for a real database.
Thanks to Gyan Sharma at IHC for providing the code for this example.
Stage 1: FIT App constant-as-mock-database

First we create the test cases as an HTML table (see the FIT documentation for this):
Table1.TestDiscounter
amount
discount()
100
200
10
Note that the column names will become class and function names in our program.
FIT contains ways to get rid of this programmerese, but for this article it is easier
just to leave them in.
63
Knowing what the test data will be, we create the user-side adapter, the
ColumnFixture that comes with FIT as shipped:
import fit.ColumnFixture;
public class TestDiscounter extends ColumnFixture
{
private Discounter app = new Discounter();
public double amount;
public double discount()
{ return app.discount(amount); }
}
Thats actually all there is to the adapter. So far, the tests run from the command
line (see the FIT book for the path youll need). We used this one:
set FIT_HOME=/FIT/FitLibraryForFit15Feb2005
java -cp %FIT_HOME%/lib/javaFit1.1b.jar;%FIT_HOME%/dist/
fitLibraryForFit.jar;src;bin
fit.FileRunner test/Discounter.html TestDiscount_Output.html
FIT produces an output file with colors showing us what passed (or failed, in case we
made a typo somewhere along the way).
At this point the code is ready to check in, hook into Cruise Control or your automated
build machine, and include in the build-and-test suite.
Stage 2: UI App constant-as-mock-database

Im going to let you create your own UI and have it drive the Discounter application,
since the code is a bit long to include here. Some of the key lines in the code are
these:
...
Discounter app = new Discounter();
public void actionPerformed(ActionEvent event)
{
...
String amountStr = text1.getText();
double amount = Double.parseDouble(amountStr);
discount = app.discount(amount));
text3.setText( "" + discount );
...
64
At this point the application can be both demoed and regression tested. The userside adapters are both running.
Stage 3: (FIT or UI) App mock database

To create a replaceable adapter for the database side, we create an interface to a
repository, a RepositoryFactory that will produce either the mock database or the
real service object, and the in-memory mock for the database.
public interface RateRepository

{
double getRate(double amount);
}
public class RepositoryFactory
{
public RepositoryFactory() { super(); }
public static RateRepository getMockRateRepository()
{
return new MockRateRepository();
}
}
public class MockRateRepository implements RateRepository
{
public double getRate(double amount)
{
if(amount <= 100) return 0.01;
if(amount <= 1000) return 0.02;
return 0.05;
}
}
To hook this adapter into the Discounter application, we need to update the
application itself to accept a repository adapter to use, and the have the (FIT or UI)
user-side adapter pass the repository to use (real or mock) into the constructor of
the application itself. Here is the updated application and a FIT adapter that passes
in a mock repository (the FIT adapter code to choose whether to pass in the mock or
real repositorys adapter is longer without adding much new information, so I omit
that version here).
import repository.RepositoryFactory;
import repository.RateRepository;
public class Discounter
{
65
private RateRepository rateRepository;
public Discounter(RateRepository r)
{
super();
rateRepository = r;
}
public double discount(double amount)
{
double rate = rateRepository.getRate( amount );
return amount * rate;
}
}
import app.Discounter;
import fit.ColumnFixture;
public class TestDiscounter extends ColumnFixture
{
private Discounter app =
new Discounter(RepositoryFactory.getMockRateRepository());
public double amount;
public double discount()
{
return app.discount( amount );
}
}
That concludes implementation of the simplest version of the hexagonal architecture.
Application Notes
The Left-Right Asymmetry
The ports and adapters pattern is deliberately written pretending that all ports
are fundamentally similar. That pretense is useful at the architectural level.

In implementation, ports and adapters show up in two flavors, which Ill call
primary and secondary, for soon-to-be-obvious reasons. They could be also
called driving adapters and driven adapters.
The alert reader will have noticed that in all the examples given, FIT fixtures are used
on the left-side ports and mocks on the right. In the three-layer architecture, FIT sits
in the top layer and the mock sits in the bottom layer.
This is related to the idea from use cases of primary actors and secondary actors.
A primary actor is an actor that drives the application (takes it out of quiescent
66
state to perform one of its advertised functions). A secondary actor is one that
the application drives, either to get answers from or to merely notify. The distinction
between primary and secondary lies in who triggers or is in charge of the
conversation.
The natural test adapter to substitute for a primary actor is FIT, since that
framework is designed to read a script and drive the application. The natural test
adapter to substitute for a secondary actor such as a database is a mock, since
that is designed to answer queries or record events from the application.
These observations lead us to follow the systems use case context diagram and draw
the primary ports and primary adapters on the left side (or top) of the hexagon,
and the secondary ports and secondary adapters on the right (or bottom) side
of the hexagon.
The relationship between primary and secondary ports/adapters and their respective
implementation in FIT and mocks is useful to keep in mind, but it should be used as
a consequence of using the ports and adapters architecture, not to short-circuit it.
The ultimate benefit of a ports and adapters implementation is the ability to run the
application in a fully isolated mode.
Use Cases And The Application Boundary

It is useful to use the hexagonal architecture pattern to reinforce the preferred way
of writing use cases.
A common mistake is to write use cases to contain intimate knowledge of the
technology sitting outside each port. These use cases have earned a justifiably bad
name in the industry for being long, hard-to-read, boring, brittle, and expensive to
maintain.
Understanding the ports and adapters architecture, we can see that the use cases
should generally be written at the application boundary (the inner hexagon), to
specify the functions and events supported by the application, regardless of external
technology. These use cases are shorter, easier to read, less expensive to maintain,
and more stable over time.
How Many Ports?

What exactly a port is and isnt is largely a matter of taste. At the one extreme,
every use case could be given its own port, producing hundreds of ports for many
67
applications. Alternatively, one could imagine merging all primary ports and all
secondary ports so there are only two ports, a left side and a right side.
Neither extreme appears optimal.
The weather system described in the Known Uses has four natural ports: the weather
feed, the administrator, the notified subscribers, the subscriber database. A coffee
machine controller has four natural ports: the user, the database containing the
recipes and prices, the dispensers, and the coin box. A hospital medication system
might have three: one for the nurse, one for the prescription database, and one for
the computer-controller medication dispensers.
It doesnt appear that there is any particular damage in choosing the wrong number
of ports, so that remains a matter of intuition. My selection tends to favor a small
number, two, three or four ports, as described above and in the Known Uses.
Known Uses
Figure 4 shows an application with four ports and several adapters at each port. This
was derived from an application that listened for alerts from the national weather
service about earthquakes, tornadoes, fires and floods, and notified people on their
telephones or telephone answering machines. At the time we discussed this system,
the systems interfaces were identified and discussed by technology, linked to
purpose. There was an interface for trigger-data arriving over a wire feed, one
68
for notification data to be sent to answering machines, an administrative interface
implemented in a GUI, and a database interface to get their subscriber data.
The people were struggling because they needed to add an http interface from
the weather service, an email interface to their subscribers, and they had to find a
way to bundle and unbundle their growing application suite for different customer
purchasing preferences. They feared they were staring at a maintenance and testing
nightmare as they had to implement, test and maintain separate versions for all
combinations and permutations.
Their shift in design was to architect the systems interfaces by purpose rather
than by technology, and to have the technologies be substitutable (on all sides) by
adapters. They immediately picked up the ability to include the http feed and the
email notification (the new adapters are shown in the drawing with dashed lines).
By making each application executable in headless mode through APIs, they could
add an app-to-add adapter and unbundle the application suite, connecting the subapplications on demand. Finally, by making each application executable completely
in isolation, with test and mock adapters in place, they gained the ability to regression
test their applications with stand-alone automated test scripts.
Mac, Windows, Google, Flickr, Web 2.0

In the early 1990s, MacIntosh applications such as word processor applications were
required to have API-drivable interfaces, so that applications and user-written scripts
could access all the functions of the applications. Windows desktop applications have
evolved the same ability (I dont have the historical knowledge to say which came
first, nor is that relevant to the point).
The current (2005) trend in web applications is to publish an API and let other web
applications access those APIs directly. Thus, it is possible to publish local crime data
over a Google map, or create web applications that include Flickrs photo archiving
and annotating abilities.
All of these examples are about making the primary ports APIs visible. We see no
information here about the secondary ports.
Stored Outputs
This example written by Willem Bogaerts on the C2 wiki:
I encountered something similar, but mainly because my application layer had a
strong tendency to become a telephone switchboard that managed things it should
69
not do. My application generated output, showed it to the user and then had some
possibility to store it as well. My main problem was that you did not need to store
it always. So my application generated output, had to buffer it and present it to the
user. Then, when the user decided that he wanted to store the output, the application
retrieved the buffer and stored it for real.
I did not like this at all. Then I came up with a solution: Have a presentation control
with storage facilities. Now the application no longer channels the output in different
directions, but it simply outputs it to the presentation control. Its the presentation
control that buffers the answer and gives the user the possibility to store it.
The traditional layered architecture stresses UI and storage to be different. The
Ports and Adapters Architecture can reduce output to being simply output again.
Anonymous example from the C2-wiki

In one project I worked on, we used the SystemMetaphor of a component stereo
system. Each component has defined interfaces, each of which has a specific
purpose. We can then connect components together in almost unlimited ways using
simple cables and adapters.
Distributed, Large-Team Development

This one is still in trial use and so does not properly count as a use of the pattern.
However, it is interesting to consider.
Teams in different locations all build to the Hexagonal architecture, using FIT and
mocks so the applications or components can be tested in standalone mode. The
CruiseControl build runs every half hour and runs all the applications using the FIT
+mock combination. As application subsystem and databases get completed, the
mocks are replaced with test databases.
Separating Development of UI and Application Logic

This one is still in early trial use and so does not count as a use of the pattern.
However, it is interesting to consider.
The UI design is unstable, as they havent decided on a driving technology or a
metaphor yet. The back-end services architecture hasnt been decided, and in fact
will probably change several times over the next six months. Nonetheless, the project
has officially started and time is ticking by.
70
The application team creates FIT tests and mocks to isolate their application, and
creates testable, demonstrable functionality to show their users. When the UI and
back-end services decisions finally get met, it should be straightforward to add
those elements the application. Stay tuned to learn how this works out (or try it
yourself and write me to let me know).
Related Patterns
Adapter
The Design Patterns book contains a description of the generic Adapter pattern:
Convert the interface of a class into another interace clients expect. The ports-andadapters pattern is a particular use of the Adapter pattern.
Model-View-Controller
The MVC pattern was implemented as early as 1974 in the Smalltalk project. It has
been given, over the years, many variations, such as Model-Interactor and ModelView-Presenter. Each of these implements the idea of ports-and-adapters on the
primary ports, not the secondary ports.
Mock Objects and Loopback

A mock object is a double agent used to test the behaviour of other objects. First,
a mock object acts as a faux implementation of an interface or class that mimics the
external behaviour of a true implementation. Second, a mock object observes how
other objects interact with its methods and compares actual behaviour with preset
expectations. When a discrepancy occurs, a mock object can interrupt the test and
report the anomaly. If the discrepancy cannot be noted during the test, a verification
method called by the tester ensures that all expectations have been met or failures
reported. From http://MockObjects.com
Fully implemented according to the mock-object agenda, mock objects are used
throughout an application, not just at the external interface The primary thrust of the
mock object movement is conformance to specified protocol at the individual class
and object level. I borrow their word mock as the best short description of an inmemory substitute for an external secondary actor.
The Loopback pattern is an explicit pattern for creating an internal replacement for
an external device.
71
Pedestals
In Patterns for Generating a Layered Architecture, Barry Rubel describes a pattern
about creating an axis of symmetry in control software that is very similar to ports
and adapters. The Pedestal pattern calls for implementing an object representing
each hardware device within the system, and linking those objects together in a
control layer. The Pedestal pattern can be used to describe either side of the
hexagonal architecture, but does not yet stress the similarity across adapters. Also,
being written for a mechanical control environment, it is not so easy to see how to
apply the pattern to IT applications.
Checks
Ward Cunninghams pattern language for detecting and handling user input errors,
is good for error handling across the inner hexagon boundaries.
Dependency Inversion (Dependency Injection) and

SPRING
Bob Martins Dependency Inversion Principle (also called Dependency Injection by
Martin Fowler) states that High-level modules should not depend on low-level
modules. Both should depend on abstractions. Abstractions should not depend on
details. Details should depend on abstractions. The Dependency Injection pattern
by Martin Fowler gives some implementations. These show how to create swappable
secondary actor adapters. The code can be typed in directly, as done in the sample
code in the article, or using configuration files and having the SPRING framework
generate the equivalent code.
Acknowledgements
Thanks to Gyan Sharma at Intermountain Health Care for providing the sample code
used here. Thanks to Rebecca Wirfs-Brock for her book Object Design, which when
read together with the Adapter pattern from the Design Patterns book, helped
me to understand what the hexagon was about. Thanks also to the people on Wards
wiki, who provided comments about this pattern over the years (e.g., particularly
Kevin Rutherfords http://silkandspinach.net/blog/2004/07/hexagonal_soup.html).
72
References and Related Reading

FIT, A Framework for Integrating Testing: Cunningham, W., online at http://
fit.c2.com, and Mugridge, R. and Cunningham, W., Fit for Developing Software,
Prentice-Hall PTR, 2005.
The Adapter pattern: in Gamma, E., Helm, R., Johnson, R., Vlissides, J., Design
Patterns, Addison-Wesley, 1995, pp. 139-150.
The Pedestal pattern: in Rubel, B., Patterns for Generating a Layered
Architecture, in Coplien, J., Schmidt, D., PatternLanguages of Program Design,
Addison-Wesley, 1995, pp. 119-150.
The Checks pattern: by Cunningham, W., online at http://c2.com/ppr/
checks.html
The Dependency Inversion Principle : Martin, R., in Agile Software
Development Principles Patterns and Practices, Prentice Hall, 2003,
Chapter 11: The Dependency-Inversion Principle, and online at http://
www.objectmentor.com/resources/articles/dip.pdf
The Dependency Injection pattern: Fowler,
www.martinfowler.com/articles/injection.html
M.,
online
at
http://
The Mock Object pattern: Freeman, S. online at http://MockObjects.com

The Loopback pattern: Cockburn, A., online at http://c2.com/cgi/wiki?LoopBack
Use cases: Cockburn, A., Writing Effective Use Cases, Addison-Wesley,
2001, and Cockburn, A., Structuring Use Cases with Goals, online at http://
alistair.cockburn.us/crystal/articles/sucwg/structuringucswithgoals.htm
73
Stop. Write a learning test

Written by: J. B. Rainsberger at 2011-12-14 (updated 2013-10-28)
The 30-second version

Where did that yak come from?
When you try to learn a new library at the same time as explore the behavior and
design of your application, you slow down more than you think.
When you cant figure out how to make the new library work for this thing you
want to build, you might spend hours fighting, debugging, swearing.
Stop. Write a Learning Test.
1. Start a new test suite, test class, spec file, whatever you want to call it.
2. Write a test that checks the things you tried to check earlier with debug
statements.
3. Write a test that has nothing to do with your application and its domain.
4. Remove unnecessary details from your test.
When this test passes, then you understand what that part of the library does. If it
behaves strangely, then you have the perfect test to send to the maintainers of the
3
library .
The Details
4
I just did this on a project using the context-free grammar parser treetop . Of course,
I hadnt used treetop before, so I had to learn it at the same time as design the
grammar for the language I wanted to parse. I reached the point where I couldnt
write a grammar rule correctly, and spent probably an hour trying to figure out get
5
it to work . Fortunately, at that moment, my laptop ran out of power, so I left the
1
http://blog.thecodewhisperer.com/2011/12/14/when-to-write-learning-tests/
2
http://www.google.com/search?q=yak+shaving
3
Remember, we dont call them bugs anymore: we call them mistakes. In this case, we cant call it a
mistake yet, because we might simply have a difference of opinion or mindset.

4
https://github.com/nathansobo/treetop
5
Note the wording: I already assuming that I have it right and they have it wrong. Bad programmer.
74

6
coffee shop and did the usual thing: I explained the problem to my wife so that
I could hear myself doing that. After about 15 minutes away from the problem, I
decided to write some Learning Tests.
Summary of what I did

1. I wrote a Learning Test for a simple case that I thought I already understood well.
2. I wrote a Learning Test similar to the problem I had to deal with, to make sure
I understood that well.
3. I wrote a Learning Test for the exact case that behaved unexpectedly.
The whole thing took an hour, and I understood the problem well enough to explain it
to my wife. She understood it and agreed that it sounded like a mistake in the library
7
. I used this Learning Test to open an issue at github. Now I can proceed without
pulling my own hair out.
Do you want to see the Learning Tests?

The case I already understood well (single_simple_rule_spec.rb)
require 'treetop'
describe "Grammar with a simple rule" do
let(:subject) { Treetop.load_from_string(
<<GRAMMAR
grammar SimpleRule
rule word
[A-Za-z]+
end
end
GRAMMAR
)}
let (:parser) { subject.new }
it "doesn't match empty string" do
parser.parse("").should be_false
end
6
They have a Second Cup in Romania. Canadians get why Id find that weird.
7
Smart woman, my wife.
75

context "matching single letter, the match result" do
let(:result) { parser.parse("a") }
it
it
it
it
end
{
{
{
{
result.should be_true }
result.text_value.should == "a" }
result.to_s.should_not == "a" }
result.should_not respond_to(:word) }
context "matching many letters, the match result" do

let(:result) { parser.parse("aBcDeF") }
it
it
it
it
end
end
{
{
{
{
result.should be_true }
result.text_value.should == "aBcDeF" }
result.to_s.should_not == "aBcDeF" }
result.should_not respond_to(:word) }
The cases I wasnt sure I understood (single_rule_using_labels_spec.rb)

require 'treetop'
describe "Grammar with a simple rule that uses a label" do
context "Labeled subexpression followed by another expression" do
let(:subject) { Treetop.load_from_string(
<<GRAMMAR
grammar SimpleRuleWithLabel
rule word
letters:[A-Za-z]+ [A-Za-z]*
end
end
GRAMMAR
)}
let (:parser) { subject.new }
context "matching many letters, the match result" do
let(:result) { parser.parse("aBcDeF") }
it { result.should respond_to(:letters) }
it { result.letters.text_value.should == "aBcDeF" }
end
end
context "Labeled subexpression without another expression" do
76

it "does not represent a valid grammar, even though I think it
should" do
lambda {
Treetop.load_from_string(
<<GRAMMAR
grammar SimpleRuleWithLabel
rule word
letters:[A-Za-z]+
end
end
GRAMMAR
)}.should raise_error(RuntimeError, /Expected \#/)
end
it "really should let me refer to the expression as #letters" do
pending "https://github.com/nathansobo/treetop/issues/21"
end
end
end
77
The Long, Painful History of Time

Written by: Erik Naggum at 1999-10-11
The programming language Common Lisp offers a few functions to support the
concept of time as humans experience it, including GET-UNIVERSAL-TIME ,
ENCODE-UNIVERSAL-TIME , DECODE-UNIVERSAL-TIME , and GET-DECODEDTIME . These functions assume the existence of a timezone and a daylight saving
time regime, such that they can support the usual expression of time in the
environment in which a small number of real-life applications run. The majority of
applications, however, need more support to be able to read and write dates and
times, calculate with time, schedule events at specific clock times daily, and work
with several time zones and daylight saving time regimes. This paper discusses
some of the problems inherent in processing time suitable to humans and describes
a solution employed by the author in a number of applications, the LOCAL-TIME
concept.
0 Introduction
The measurement of time has a very long history, dating back to the first records of
human civilization. Yet, the archeological evidence suggests that the concept of time
evolved no further than ordinary human needs, and any notion of time remained
confined to a fairly short time frame, such as a lifetime past and future. Expressions
of measurements of time were brief and imprecise, rife with the numerous and
nefarious assumptions humans bring into their communication, consistent with our
tendency to suppress information believed to be redundant.
For instance, everyone knows which century they are in or that some two-digit
year refers to. Until computers came along, the assumptions held by people were
either recoverable from the context or shared by contemporary communicators. After
computers came to store information for us, we still held onto the context as if the
computers were as able to recover it as we are. Quite obviously, they arent, and in
about three months, we will see whether other humans were indeed able to recover
the context left unstated by other humans when they wrote down their dates with two
digits and assumed it would never be a problem. The infamous Y2K problem is one
of the few opportunities mankind will get to tally the costs of lack of precision in our
common forms of communication. The lesson learned will not be that our notations
1
http://naggum.no/lugm-time.html
78

of time need to be precise and include their context, unless the general public stops
refusing to be educated in the face of dire experience. That so much attention has
been granted this silly problem is fortunate for those of us who argue against legacy
notations of time. However, the inability of most people to deal with issues of such
extraordinary importance when they look "most harmless" means that those who do
understand them must be inordinately careful in preparing their information such
that loss of real information can be minimized.
The basic problem with time is that we need to express both time and place whenever
we want to place some event in time and space, yet we tend to assume spatial
coordinates even more than we assume temporal coordinates, and in the case of
time in ordinary communication, it is simply left out entirely. Despite the existence of
time zones and strange daylight saving time regimes around the world, most people
are blithely unaware of their own time zone and certainly of how it relates to standard
references. Most people are equally unaware that by choosing a notation that is close
to the spoken or written expression of dates, they make it meaningless to people who
may not share the culture, but can still read the language. It is unlikely that people
will change enough to put these issues to rest, so responsible computer people need
to address the issues and resist the otherwise overpowering urge to abbreviate and
drop context.
This paper is almost all about how we got ourselves into trouble by neglecting to
think about time frames longer than a human lifetime, how we got all confused by
the difference between time as an orderly concept in science and a mess in the rest
of human existence, and how we have missed every opportunity to fix the problems.
This paper proposes a fix to the most glaring problems in a programming language
that should not have been left without a means to express time for so long.
1 Scientific Time
How long does it take the earth to face the Sun at the same angle? This simple
question has a definite and fairly simple scientific answer, and from this answer, we
can work out a long list of answers about what time is and how we want to deal with
astronomical events. The SI units (Systme International dUnits), probably better
known as "metric units", define the second as the fundamental unit of time, and this,
too, has a very good scientific definition. Time progresses continuously and is only
chopped up into units for human convenience. Agreement on a single reference point
within a scientific community has always been easy, and it is useful to count basic
units, like days in the (Modified) Julian Day system, or seconds since some arbitrary
epoch in computers.
79

Scientific time also lends itself to ease of computation; after all, that is what we do
with it. For instance, we have a world-wide standard for time, called the Coordinated
Universal Time, or UTC. (The C used to be subscripted, UTC, just like the digits in UT0
and UT1 which are universal time concepts with slightly different reference points,
but "UTC" has become the preferred form.) Scientific time naturally has origin 0, as
usual with scientific measures, even though the rest of human time notations tend
to have origin 1, the problems of which will be treated below.
Most computer-related references to time deal with periods of time, which lend
themselves naturally to use scientific time, and therefore, it makes sense to most
programmers to treat the period of time from some epoch until some other time to
be the best way to express said other time. This is the path taken by Common Lisp in
its UNIVERSAL-TIME concept, with time 0 equal to 1900-01-01 00:00:00 UTC, and
the Unix time concept, with time 0 equal to 1970-01-01 00:00:00 UTC. This approach
works well as long as the rules for converting between relative and absolute time are
stable. As it turns out, they are not.
Not all languages and operating systems use this sensible an approach. Some have
used local time as the point of reference, some use decoded local time as the
reference, and some use hardware clocks that try to maintain time suitable for direct
human consumption. There is no need to make this issue more complex than it
already is, so they will not be granted any importance.
2 Political Time
How long does it take for the clock to show the same value? The answer to this
question is only weakly related to the time the planet takes to make a complete
rotation. Normally, we would say the political rotation takes 24 hours, just like the
scientific, but one day out of the year, it takes only 23 hours, and another day out
of the year, it takes 25 hours, thanks to the wonders of daylight saving time. Which
days these are is a decision made by politicians. It used to be made by the military to
conserve fuel, but was taken over by labor unions as a means to get more daylight
in the workers' spare time, and most countries have gone through an amazing list
of strange decision-making in this area during this century. Short of coming to their
senses and abolishing the whole thing, we might expect that the rules for daylight
saving time will remain the same for some time to come, but there is no guarantee.
(We can only be glad there is no daylight loan time, or we would face decades of too
much daylight, only to be faced with a few years of total darkness to make up for it.)
80

Political time is closely related to territory, power, and collective human irrationality.
There is no way you can know from your location alone which time zone applies at
some particular point on the face of the earth: you have to ask the people who live
there what they have decided. This is very different from scientific time, which could
tell you with great ease and precision what the mean sidereal time at some location
should be. In some locations, this is as much as three hours off from what the local
population has decided, or has had decided for them. The Sun is in zenith at noon
at very few places on earth, instead being eclipsed or delayed by political decisions
where the randomness never ends.
Yet, it is this political time that most people want their computers to produce when
they ask for the date or the time of day, so software will have to comply with the
randomness and produce results consistent with political decisions. The amount of
human input into this process is very high, but that is the price we have to pay
for our willingness to let politicians dictate the time. However, once the human
input has been provided, it is annoying to find that most programming languages
and supporting systems do not work with more than one timezone at a time, and
consequently do not retain timezone information with time data.
The languages we use tend to shape the ideas we can talk about. So, too, the way
we write dates and times influence our concepts of time, as they were themselves
influenced by the way somebody thought about time a long time ago. Calendars
and decisions like which year is the first, when the year starts, and how to deal with
astronomical irregularities were made so long ago that the rationale for them has not
survived in any form, but we can still look at what we have and try to understand.
In solving the problem of dealing with time in computers, a solid knowledge of the
legacy we are attending to is required.
3 Notations for Time

The way we write down time coordinates appears to have varied little over the
years in only one respect: we tend to write them differently depending on the
smallest perceived unit of time that needs to be communicated. For instance, it
seems sufficiently redundant to include AD or BC in the dates of birth of contemporary
people that they are always omitted. Should some being with age >2000 years come
to visit us, it is also unlikely that writing its date of birth correctly would be a pressing
concern. However, we tend to include these markers for the sign of the year when
the possibility of ambiguity reaches a certain level as determined by the reader. This
process is itself fraught with ambiguity and inconsistency, but when computers need
to deal with dates this far back, it does not seem worthwhile to calculate them in
81

terms of standard reference points, so we can ignore the problem for now, but may
need to deal with it if a system of representation is sufficiently useful to be extended
to the ancient past.
Not only do we omit information that is deemed redundant, it is not uncommon for
people to omit information out of sheer laziness. A particularly flagrant example of
the omission of information relative to the current time is the output from the Unix
ls program which lists various information about files. The customary date and
time format in this program is either month-day-hour-minute or month-day-year.
The cutoff for tolerable precision is six months ago, which most implementations
approximate with 180 days. This reduction in precision appears to have been
motivated by horizontal space requirements, a necessary move after wasting a lot of
space on irrelevant information, but for some reason, precision in time always suffers
when people are short of space.
The infamous Y2K problem, for instance, is said to have started when people wanted
to save two columns on punched cards, but there is strong evidence of other, much
better alternatives at the time, so the decision to lose the century was not predicated
on the need for space, but rather on the culturally acceptable loss of information from
time coordinates. The details of this mess are sufficiently involved to fill a separate
paper, so the conclusion that time loses precision first when in need or perceived
need of space should be considered supported by the evidence.
3.1 Natural-Language Notations

People tend to prefer words to numbers, and go out of their way to name things.
Such names are frequently symbolic because they are inherently arbitrary, which
implies that we can learn much from studying what people call numbers. (French has
a number which means "arbitrarily many": 36, used just like English "umpteen", but
it is fascinating that a number has meaning like that. Other numbers with particular
meaning include 69, 666, and 4711. The number 606 has been used to refer to
arsphenamine, because it was the 606th compound tested by Paul Ehrlich to treat
syphilis.) In the present context, the names of the Roman months have been adopted
by all Western languages, while the names of days of the week have more recent
and diverse names, probably because weeks are a fairly recent concept.
Using names for numeric entities complicates processing a natural language
specification of time tremendously, yet this is what people seem more comfortable
with. In some cultures, months have only names, while in others, they are nearly
always written as numbers. The way the names of months and the days of the week
82

are abbreviated varies from language to language, as well, so software that wants
to be international needs to maintain a large repository of names and notations to
cater to the vanity of human users. However, the names are not the worst we have
to deal with in natural language notations.
Because dates and times are frequently spoken and because the written forms are
often modeled after the spoken, we run into the problem of ordering the elements
of time and the omission of perceived redundancy becomes a much more serious
problem, because each language and each culture have handled these problems so
differently. The orders in use for dates are
year-month-day
day-month-year
month-day-year
day-month
month-day
year-month
month-year
As long as the year is zero or greater than 31 or the day greater than 12, it is usually
possible to disambiguate these orders, but we are about to experience renewed
problems in 2001, when the year will probably be still be written with two digits by
some people regardless of the experience of mankind as a whole at 2000-01-01
00:00:00 . We live in interesting times, indeed.
Time is fortunately specified with a uniform hour-minute-second order, but the
assumption of either AM or PM even in cultures where there is no custom for their
specification provides us with an ambiguity that computers are ill equipped to deal
with. This and other historic randomness will be treated in full below.
Most of the time people refer to is in their immediate vicinity, and any system
intended to capture human-friendly time specifications will need to understand
relative times, such as "yesterday", "this time tomorrow", "two hours ago", "in fifteen
minutes". All of these forms vary considerably from culture to culture and from
language to language, making the process of reading these forms as input nontrivial. The common forms of expression for periods of time is also fuzzy in human
communication, with units that fail to convert to intervals of fixed length, but instead
are even more context-sensitive than simple points in time.
83
3.2 Language-Neutral Notations

Various attempts have been made to overcome the problems of human-to-human
forms of communication between human and machine and in machine-to-machine
communication. Machine-to-machine communication generally falls into one of three
categories:
1. Nave binary
2. Formatted or encoded binary
3. Character sequences (text)
Binary formats in general suffer from a huge number of problems that there is little
value in discussing here, but it is worth noting that a binary format that is as robust
as a textual format is frequently just as verbose as a textual format, so in the interest
of robustness and legibility, this discussion will restrict itself to textual formats
Obviously, a language-neutral notation will have to consist of standardized elements
and possibly codes. Fortunately, a standard like this already exists: ISO 8601. Since
all the work with a good language-neutral notation has already been done, it would
be counter-productive in the extreme to reinvent one. However, ISO 8601 is fairly
expensive from the appropriate sources and also chock full of weird options, like
most compromise standards, so in the interest of solving some problems with its use,
only the extended format of this standard will be employed in this paper.
A language-neutral notation will need to satisfy most, if not all, of the needs satisfied
by natural language notations, but some latitude is necessary when dealing with
relative times after all, the purpose of the language-neutral notation is to remove
ambiguity and make assumptions more if not completely explicit. ISO 8601 is
sufficient to cover these needs:
absolute positions in time
duration
period with absolute start and end
period with absolute start or end and duration
The needs not covered are mostly related to user convenience with respect to the
present and absolute positions in time in its immediate vicinity. E.g., the omission
of the date when referring to yesterday, tomorrow, the most recent occurrence of
84

a time of day, and the forthcoming occurrence of a time of day. To make this more
convenient, the notation employed in the LOCAL-TIME concept described below
has special syntax for these relative times.
The full, extended format of ISO 8601 is as follows:
1999-10-11T11:10:30,5-07:00
The elements are, in order:

1. the year with four digits
2. a hyphen (omitted in the basic format)
3. the month with two digits
4. a hyphen (omitted in the basic format)
5. the day of month with two digits
6. the letter T to separate date and time
7. the hour in the 24-hour system with two digits
8. a colon (omitted in the basic format)
9. the minute with two digits
10.a colon (omitted in the basic format)
11.the second with two digits
12.a comma
13.the fraction of the second with unlimited precision
14.a plus sign or hyphen (minus) to indicate sign of time zone
15.the hours of the time zone with two digits
16.a colon (omitted in the basic format)
17.the minutes of the time zone with two digits
The rules for omission of elements are quite simple. Elements from the time of
day may be omitted from the right and take their immediately preceding delimiter
with them. Elements from the date may be omitted from the left, but leave the
immediately following delimiter behind. When the year is omitted, it is replaced by a
hyphen. Elements of the date may also be omitted from the left, provided no other
85

elements follow, in which case they take their immediately preceding delimiter with
them. The letter T is omitted if the whole of the time of day or the whole of the date are
omitted. If an element is omitted from the left, it is assumed to be the current value.
(In other words, omitting the century is really dangerous, so I have even omitted the
possibility of doing so.) If an element is omitted from the right, it is assumed to cover
the whole range of values and thus be indeterminate.
Every element in the time specification needs to be within the normal bounds. There
is no special consideration for leap seconds, although some might want to express
them using this standard.
A duration of time has a separate notation entirely, as follows:
P1Y2M3DT4H5M6S>
P7W
The elements are, in order:

1. the letter P to indicate a duration
2. the number of years
3. the letter Y to indicate years
4. the number of months
5. the letter M to indicate months
6. the number of days
7. the letter D to indicate days
8. the letter T to separate dates from times
9. the number of hours
10.the letter H to indicate hours
11.the number of minutes
12.the letter M to indicate minutes
13.the number of seconds
14.the letter S to indicate seconds
86

or for the second form, usually used alone
1. the letter P to indicate a duration
2. the number of weeks
3. the letter W to indicate weeks
Any element (number) may be omitted from this specification and if so takes its
following delimited with it. Unlike the absolute time format, there is no requirement
on the number of digits, and thus no requirement for leading zeros.
A period of time is indicated by two time specifications, at least one of which has to be
absolute, separated by a single solidus (slash), and has the general forms as follows:
start/end
start/duration
duration/end
the end form may have elements of the date omitted from the left with the
assumption that the default is the corresponding value of the element from the start
form. Omissions in the start form follow the normal rules.
The standard also has specifications for weeks of the year and days of the week, but
these are used so rarely and are aesthetically displeasing so are gracefully elided
from the presentation.
When discussing the read/write syntax of the LOCAL-TIME concept below, the
above formats will be employed with very minor modifications and extensions.
4 Geography
It is amusing that when people specify a time, they tend to forget that they looked
at their watches or asked other time-keeping devices at a particular geographic
location. The value they use for "current time" is colored by this location so much that
the absence of a location at which we have the current time, renders it completely
useless it could be specified in any one of the about 30 (semantically different)
timezones employed around the planet. This is particularly amusing with statements
you find on the web:
This page was updated 7/10/99 2:00 AM.
87

This piece of information is amazingly useless, yet obviously not so to the person
who knows where the machine is located and who wrote it in the first place. Only by
monitoring for changes to this statement does it have any value at all. Specifications
of time often has this purpose, but the belief that they carry information, too, is quite
prevalent. The only thing we know about this time specification is that it was made
in the past, which may remove most of the ambiguity, but not quite all it could
be 1999-07-10 .
The geographical origin of a time specification is in practice necessary to understand
it. Even with the standard notation described above, people will want to know
the location of the time. Unfortunately, there is no widely adopted standard for
geographical locations. Those equipped with GPS units may use ICBM or grid
coordinates, but this is almost as devoid of meaning as raw IP addresses on the
Internet. Above all, geography is even more rife with names and naming rules that
suffer from translation than any other information that cries for a precise standard.
Time zones therefore double as indicators of geographical location, much to the
chagrin of anyone who is not from the same location, because they use names and
abbreviations of names with local meaning. Of course. Also, the indication of the
daylight saving time in the timezone is rather amusing in the probably unintentional
complexity they introduce. For instance, the Middle or Central European Time can be
abbreviated MET or CET, but the "summer time" as it is called here is one of MEST,
CEST, MET DST, or CET DST. Add to this that the "S for summer" in the former two
choices is often translated, and then we have the French.
The only good thing about geography is that most names can be translated into
geographical coordinates, and a mapping from coordinates to time zone and daylight
saving time rules is fairly easy to collect, but moderately difficult to maintain. This
work has been done, however, and is distributed with most Unix systems these
days, most notably the free ones, for some value of "free". In order for a complete
time representation to work fully with its environment, access to this information
is necessary. The work on the LOCAL-TIME concept includes an interface to the
various databases available under most Unix systems.
5 Perspective
An important part of the Y2K problem has been that the information about the
perspective on the time stored was lost. Trivialities like the fact that people were
born in the past, bills were paid in the past and fall due in the future, deliveries will be
made in the future, etc, and most of the time, meaningful specifications of time have
88

hard boundaries that they cannot cross. Few people have problems with credit cards
that expire 02/02 , say. This was very obviously not 1902-02 . The perspective we
bring to time specifications usually last beyond the particular time specified.
When dealing with a particular time, it is therefore necessary to know, or to be
told, whether it refers to the past or the future, and whether the vantage point
is different from the present. If, for instance, a delivery is due 10/15/99 , and it
fails to be delivered that day, only a computer would assume that it was now due
2099-10-15 . Unfortunately, there is no common practice in this area at all, and
most people are satisfied with a tacit assumption. That is in large part what caused
the Y2K problem to become so enormously expensive to fix. Had the assumed, but
now missing information been available, the kinds of upgrades required would have
been different, and most likely much less expensive.
There is more to the perspective than just past and future, however. Most computer
applications that are concerned with time are so with only one particular time: the
present. We all expect a log file to be generated along with the events, and that it
would be disastrous if the computer somehow recorded a different time than the time
at which an event occurred, or came back to us and revised its testimony because it
suddenly remembered it better. Modern society is disproportionately dependent on
a common and coordinated concept of the present time, and we have increasingly let
computers take care of this perspective for us. Telephones and computers, both voice
and electronic radio broadcasts, watches, wall clocks, the trusty old time clocks in
factories where the workers depended on its accuracy, they all portray this common
concept of a coordinated understanding of which time it is. And they all disagree
slightly. A reportedly Swiss saying goes: "A man with one clock knows the time. A
man with two clocks does not."
Among the many unsolved problems facing society is an infrastructure for time-
keeping that goes beyond individual, uncoordinated providers, and a time-keeping

technology that actually works accurately and is so widely available that the
differences in opinion over what time it is can be resolved authoritatively. The
technology is actually here and the infrastructure is almost available to everyone,
but it is not used by the multitude of purported sources of the current time. On the
Internet, NTP (the Network Time Protocol) keeps fully connected systems in sync, and
most telecommunications and energy providers have amazingly accurate clocks, but
mere mortals are still left with alarming inaccuracies. This fact alone has a tendency
to reduce the interest in accurate representation of time, for the obvious reason that
the more accurate the notation and representation, the less trustworthy the value
expressed.
89
6 Calculations with Time

The notation for duration and periods bounded by one absolute position in time
and one duration described above have intuitive meaning, but when pressed for
actual meaning, suffer somewhat from the distressing effects of political time. For
instance, a period of one year that starts 1999-03-01 would end on 2000-02-29
or 2000-03-01 with equal probability of being correct. More common problems
occur with the varying lengths of months, but those are also more widely understood
and the heuristics are in place to deal with them.
Less obvious is the problem of adding one day to a particular time of day. This was
the original problem that spurred the development of the LOCAL-TIME concept and
its implementation. In brief, the problem is to determine which two days of the year
the day is not 24 hours long. One good solution is to assume the day is 24 hours long
and see if the new time has a different timezone than the original time. If so, add
the difference between the timezones to the internal time. This, however, is not the
trivial task it sounds like it should be.
The first complication is that none of the usual time functions can report the absolute
time that some timezone identifier will cause a change in the value of timezone as
applicable to the time of day. Resolving this complications means that we do not have
to test for a straddled timezone boundary the hard way with every calculation, but
could just compare with the edge of the current timezone. Most software currently
does this the hard way, including the Unix cron scheduler. However, if we accept
the limitation that we can work with only one timezone at a time, this becomes much
less of a problem, so Unix and C people tend to ignore this problem.
The second complication is that there really is no way around working with an internal
time representation in any calculation attempts to adjust elements of a decoded
time generally fail, not only because programmers are forgetful, but also because
the boundary conditions are hard to enumerate.
Most often, however, calculations fall into two mutually exclusive categories:
1. calculations with the time of day possibly including days
2. calculations with the date with no concept of a time of day
When time is represented internally in terms of seconds since an epoch, only the
former is easy the latter is irrevocably linked with all the timezone problems. The
latter may in particular be calculated without reference to timezones at all, and
90

indeed should be conducted in UTC . As far as the author knows, there are no tools or
packages available in modern programming languages or environments that provide
significant support for calculations with dates apart from calculation with times of
day these are usually deferred to the application-level, and appear not to have
been solved as far as the application programmer is concerned.
7 Historic Randomness
The Roman tradition of using Ante Meridiem and Post Meridiem to refer to the two
halves have survived into English, despite the departure from the custom of changing
the day of the month at noon. The Meridiem therefore has a very different role in
modern usage than in ancient usage. This legacy notation also carries a number
system that is fairly unusual. As seen from members of the 24-hour world, the order
12,1,2,11,12,1,2,,11 as mapped onto 0,1,2,23 is not only confusing, it is nearly
impossible to make people believe that 13 hours have elapsed from 11 AM to 12 AM.
For instance, several Scandinavian restaurants are open only 1 hour a day to tourists
from the world of the 12-hour clock, but open 13 hours a day to natives of the world
of the 24-hour clock.
The Roman tradition of starting the year in the month of March has also been lost.
Most agrarian societies were far more interested in the onset of spring than in the
winter solstice, even though various deities were naturally celebrated when the sun
returned Most calendars were designed by people who made no particular effort to
be general or accurate outside their own lifetime or needs, but Julius Csar decided
to move the Roman calendar back two months, and thus it came to be known as the
Julian calendar. This means that month number 7, 8, 9, and 10 suddenly came in as
number 9, 10, 11, and 12, but kept their names: September, October, November,
December. This is of interest mostly to those who remember their Latin but far more
important was the decision to retain the leap day in February. In the old calendar,
the leap day was added at the end of the year, as makes perfect sense, when the
month was already short, but now it is squeezed into the middle of the first quarter,
complicating all sorts of calculations, and affecting how much people work. In the old
days, the leap day was used as an extra day for the various fertility festivities. You
would just have to be a csar to find this unappealing.
The Gregorian calendar improved on the quadrennial leap years in the Julian
calendar by making only every fourth centennial a leap year, but the decision was
unexpectedly wise for a calendar decision. It still is not accurate, so in a few thousand
years, they may have to insert an extra leap day the way we introduce leap seconds
now, but the simplicity of the scheme is quite amazing: a 400-year cycle not only
91

starts 2000-03-01 (as it did 1600-03-01 ), it contains an even number of weeks:
20,871. This means that we can make do with a single 400-year calculation for all
time within the Gregorian calendar with respect to days of week, leap days, etc. Pope
Gregory XIII may well have given a similar paper to this one to another unsuspecting
audience that probably also failed to appreciate the elegance of his solution., and
400 more years will pass before it is truly appreciated.
Other than the unexpected elegance of the Gregorian calendar, the world is now
quite fortunate to have reached consensus on its calendars. Other calendars are still
used, but we now have a global reference calendar with complete convertibility. This
is great news for computers. It is almost as great news as the complete intercurrency
convertibility that the monetary markets achieved only as late as 1992. Before that
time, you could wind up with a different amount of money depending on which
currencies you traded obscure currencies like the ruble through. The same applied
to calendars: not infrequently, you could wind up on different dates according as
you converted between calendar systems, similar to the problem of adding a year to
February 29 any year and then subtracting a year.
8 The LOCAL-TIME Concept

The groundwork should now have been laid for the introduction of the several
counter-intuitive decisions made in the design of the LOCAL-TIME concept and its
implementation.
8.1 Time Elements as Fixnums

Unix time has the "advantage" that it is representable as a 32-bit
machine integer. It has the equal disadvantage of not working if the
time is not representable as a 32-bit machine integer, and thus can only
represent times in the interval 1901-12-13T20:45:52/2038-01-19T03:14:07 .
If
we
choose
an
unsigned
machine
integer,
the
interval
is
1970-01-01T00:00:00/2106-02-07T06:28:16 . The Common Lisp
UNIVERSAL-TIME concept has the disadvantage that it turned into a bignum on
most 32-bit machines on 1934-01-10T13:37:04 and runs out of 32 bits two years
earlier than Unix time, on 2036-02-07T06:28:16 . I find these restrictions to be
uncomfortable, regardless of whether there are any 32-bit computers left in 2036
to share my pain.
Bignum operations are generally far more expensive than fixnum operations, and
they have to be, regardless of how heavily the Common Lisp implementation has
92

optimized them. It therefore became a pronounced need to work with fixnums
in time-intensive applications. The decision fell on splitting between days and
seconds, which should require no particular explanation, other than to point out that
calculation with days regardless of the time of day is now fully supported and very
efficient.
Because we are very close to the beginning of the next 400-year leap-year cycle,
thanks to Pope Gregory, day 0 is defined to be 2000-03-01 , which much less

arbitrary than other systems, but not obviously so. Each 400-year cycle contains
146,097 days, so an arbitrary decision was made to limit the day to a maximal
negative value of -146,097, or 1600-03-01 . This can be changed at the peril of
accurately representing days that do not belong to the calendar used at the time. No
attempt has been made to accurately describe dates not belonging to the Gregorian
calendar, as that is an issue resolvable only with reference to the borders between
countries and sometimes counties at the many different times throughout history
that monarchs, church leaders, or other power figures decided to change to the
Gregorian calendar. Catering to such needs is also only necessary with dates prior
to the conversion of the Russian calendar to Gregorian, a decision made by Lenin as
late as 1918, or any other conversion, such as 1582 in most of Europe, 1752 in the
United States, and even more embarrassingly late in Norway.
Not mention above is the need for millisecond resolution. Most events on modern
computers fall within the same second, so it is now necessary to separate them by
increasing the granularity of the clock representation. This part is obviously optional
in most time processing functions.
The LOCAL-TIME concept therefore represents time as three disjoint fixnums:
1. the number of days since (or until, when negative) 2000-03-01
2. the number of seconds since the start of the day in Coordinated UniversalTime
3. the number of milliseconds since the start of the second.
All numbers have origin 0. Only the number of days may be negative.
The choice of epoch needs some more explanation. Conversion to this system only
requires subtracting two from the month and making January and February part of
the previous year.
The moderate size of the fixnums allows us another enormous advantage over
customary ways to represent time. Since the leap year is now always at the end of
the year, it has no bearing on the decoding of the year, month, day, and day-of-week
93

of the date. By choosing this odd-looking epoch, the entire problem with computing
leap years and days evaporates. This also means that a single, moderately large
table of decoded date elements may be pre-computed for 400 years, providing a
tremendous speed-up over the division-based calculations used by other systems.
Similarly, a table of the decoded values of the 86400 possible seconds in a day
(86401 if we allow leap seconds) yields a tremendous speedup over division-based
calculations. (Depending on your processor and memory speeds, a factor of 10 to 50

may be expected. for a complete decoding)
8.2 Timezone Representation

David Olsen of Digital Equipment Corporation has laid down a tremendous amount
of work in collecting the timezones of the world and their daylight saving time
boundaries. Contrary to the Unix System V approach from New Jersey (insert
appropriate booing for best effect), which codifies a daylight saving time regime only
for the current year, and apply it to all years, David Olsens approach is to maintain
tables of all the timezone changes. A particular timezone thus has a fairly long table
of periods of applicability of the specific number of seconds of to add to get local
time. Each interval is represented by the start and end times of the specific value,
the specific value, a daylight saving time flag, and the customary abbreviation of the
timezone. On most Unix systems, this is available in compiled files in /usr/share/
zoneinfo/ under names based on the continent and capital of the region in most
cases, or more general names in other cases. While not perfect, this is probably a
scheme good as any it is fairly easy to figure out which to use. Usually, a table is
also provided with geographic coordinates mapped to the timezone file.
For the timezone information, the LOCAL-TIME concept implements a package,
TZ , or TIMEZONE in full, which contains symbols named after the files, whose
values are lazy-loaded timezone objects. Because the source files for the zoneinfo
files are generally not as available as the portably coded binary information, the
information are loaded into memory from the compiled files, thus maintaining
maximum compatibility with the other timezone functions on the system.
In the LOCAL-TIME instances, the timezone is represented as a symbol to aid in the

ability to save literal time objects in compiled Lisp files. The package TZ can easily
be autoloaded in systems that support such facilities, in order to reduce the loadorder complexity.
In order to increase efficiency substantially once again, each timezone object holds
the last few references to timezone periods in it, in order to limit the search time.
94

Empirical studies of long-running systems have showed that more than 98% of the
lookups on a given timezone were for time in the same period, with more than 80%
of the remaining lookups at the neighboring periods, so caching these values made
ample sense.
8.3 Efficiency Considerations in Table Storage

In order to store 146,072 entries for the days of a 400-year cycle with the decoded
year, month, day, and day-of-week and 86401 entries for the seconds of a day
with the decoded hour, minute and second efficiently, various optimizations were
employed. The nave approach, to uses lists, consumes approximately 6519K on
a 32-bit machine. Due to their overhead, vectors did worse. Since the decoded
elements are small, well-behaved unsigned integers, encoding them in bit fields
within a fixnum turns out to save a lot of memory:
+----------+----+-----+---+
|
yyyy
| mm | day |dow|
+----------+----+-----+---+
10
4
5
3
+-----+------+------+
|hour | min | sec |
+-----+------+------+
5
6
6
This simple optimization meant 7 times more compact storage of the exact same
data, with significantly improved access times, to boot (depending on processor and
memory speeds as well as considerations for caching strategies, a factor of 1.5 to 3
has been measured in production).
Still, 909K of storage to keep tables of precomputed dates and times may seem
a steep price to pay for the improved performance. Unsurprisingly, more empirical
evidence confirmed that most dates decoded were in the same century. Worst case
over the next few years, we will access two centuries frequently, but it is still a waste
to store four full centuries. A reduction to 100 years per table also meant the number
of years were representable in 7 bits, meaning that an specialized vector of type
(UNSIGNED-BYTE 16) could represent them all. The day of week would be lost
in this optimization, but a specialized vector of type (UNSIGNED-BYTE 4) of the
full length (146097) could hold them if a single division to get the day of week was
too expensive. It turns out that the day of week is much less used than the other
decoded elements, so the specialized vector was dropped and an option included
with the call to the decoder to skip the day of week.
Similarly, by representing only 12 hours in a specialized vector of type (UNSIGNEDBYTE 16) , the hour would need only 4 bits and the lookup could do the 12-hour
95

shift in code. This reduces the table memory needs to only 156K, and it is still faster
than access to the full list representation. This compaction yields almost a factor 42
improvement over the nave approach
For completeness, the bit field layout is now simplified as follows.
+-------+----+-----+
| 0-100 |1-12| 1-31|
+-------+----+-----+
7
4
5
+----+------+------+
|0-11| 0-59 | 0-59 |
+----+------+------+
4
6
6
Decoding the day now means finding the 400-year cycle for the day of week, the
century within it for the table lookup, and adding together the values of the centuries
and the year from the table, which may be 100 to represent January and February of
the following century. All of this can be done with very inexpensive fixnum operations
for about 2,939,600 years, after which the day will incur a bignum subtraction to
bring it into fixnum space for the next 2,939,600> years. (This optimization has not
actually been implemented.)
9 Reading and Printing Time

Common Lisp is renowned for the ability to print and read back almost all of its
data types. The motivation for the LOCAL-TIME concept included the ability to
save human-readable timestamps in files, as well as the ability to store literal time
objects efficiently in compiled Lisp files. The former has been accomplished through
the use of the reader macros. Ignoring all other possible uses of the @ character,
it was chosen to be the reader macro for the full representation of a LOCAL-TIME
object. Considering the prevalence of software that works with the UNIVERSALTIME concept, especially in light of the lack of alternatives until now, #@ was
chosen to be the reader macro for the UNIVERSAL-TIME representation of a time
object. This latter notation obviously loses the original time zone information and
any milliseconds.
9.1 Timestring Syntax

The Lisp reader is instructed to parse a timestring following the reader macro
characters. Other functions may call PARSE-TIMESTRING directly. Such a
timestring follows ISO 8601 closely, but allows for a few enhancements and an
additional option: the ability to choose between comma and period for the fractional
second delimiter.
96

Supported formats of the timestring syntax include
1. absolute time with all elements, the default printed format
2. absolute time with some elements omitted, as per ISO 8601
3. absolute time with date omitted, defaulting to the current
4. absolute time with time omitted, defaulting to 00:00:00Z .
5. the most recent occurrence of a time of day, with a leading < .
6. the forthcoming occurrence of a time of day, with a leading > .
7. the time of day specified, but yesterday, with a leading - .
8. the time of day specified, but tomorrow, with a leading + .
9. the current time of day, with a single = .
Work in progress includes adding and subtracting a duration from the specified time,
such as the present, explaining the use of the = , which is also needed to represent
periods with one anchor at the present. The duration syntax is, however, rife with
assumptions that are fairly hard to express concisely and to use without causing
unexpected and unwanted results.
The standard syntax from ISO 8601 is fairly rich with options. These are mostly
unsupported due to the ambiguity they introduce. The goal with the timestring
syntax is that positions and periods of time shall be so easy to read and write in an
information-preserving syntax that there will be no need to cater to the informationlosing formats preferred by some only because of their attempt at similarity to their
spoken forms.
9.2 Formatting Timestrings

Considering that the primary problem with time formats is randomness in the order
of the elements, the timestring formatter for LOCAL-TIME objects allows no options
in that regard, but allows elements to be omitted as per the standard. The loss of
12-hour clocks will annoy a few people for a time, but there is nothing quite like
shaking a bad habit for good. Of course, the persistent programmer will write his own
formatter, anyway, so the default should be made most sensible for representing
time in programs and in lisp-oriented input files.
At present, the interface to the timestring formatter is well suited for a call from
FORMAT control strings with the ~// construct, and takes arguments a follows:
97

1. stream the stream to receive the formatter timestring
2. local-time the LOCAL-TIME instance
3. universal if true, ignore the timezone and use UTC>. This is the colon
modifier.
4. timezone if true, print a timezone specification at the end. This is the atsign
modifier.
5. date-elements the number of elements of the date to write, counted from
the right. This is a number from 0 to 4 (the default if omitted or NIL ).
6. time-elements the number of elements of the time to write, counted from
the left. This is a number from 0 to 4 (the default if omitted or NIL ).
7. date-separator the character to print between elements of the date. If
omitted or NIL , defaults to the hyphen.
8. time-separator the character to print between elements of the time. If
omitted or NIL , defaults to the colon. This argument also applies to the timezone
when it is printed, and when it has a minute component.
9. internal-separator the character to print between the date and the time
elements. May also be specified as the number 0, to omit it entirely, which is the
default if either the date or the time elements are entirely omitted, or the letter
T otherwise.
9.3 Exported LOCAL-TIME Symbols

LOCAL-TIME
[Type]
[Constructor] Arguments: (&key universal internal unix (msec
0) (zone 0) .
Produce a LOCAL-TIME
representation.
instance
from
the
provided
MAKE-LOCAL-TIME
[Constructor] Arguments: (&key day sec msec zone)
LOCAL-TIME-DAY
LOCAL-TIME-SEC
LOCAL-TIME-MSEC
98
numeric
time

LOCAL-TIME-ZONE
[Accessors]
LOCAL-TIME<
LOCAL-TIME
LOCAL-TIME>
LOCAL-TIME>=
LOCAL-TIME=
LOCAL-TIME/=
[Functions] Comparison, just like STRING<op> .
LOCAL-TIME-ADJUST
[Function] Arguments: (source timezone &optional destination)
Returns two values, the values of new day and sec slots, or, if destination is
a LOCAL-TIME instance, fills the slots with the new values and returns the
destination.
LOCAL-TIME-DESIGNATOR
[Function] Convert a designator (real number) into a LOCAL-TIME instance.
GET-LOCAL-TIME
[Function] Return the current time as a LOCAL-TIME instance.
ENCODE-LOCAL-TIME
[Function] Arguments: (ms ss mm hh day month year &optional
timezone)
Return a new LOCAL-TIME instance corresponding to the specified time
elements.
DECODE-LOCAL-TIME
[Function] Argument: (local-time)
Returns the decoded time as multiple values: ms , ss , mm , hh , day , month ,
year , day-of-week , daylight-saving-time-p , timezone , and the
customary timezone abbreviation.
99

PARSE-TIMESTRING
[Function] Arguments: (timestring &key start end junk-allowed)
Parse a timestring and return the corresponding LOCAL-TIME .
FORMAT-TIMESTRING
[Function] Arguments: (stream local-time universal-p timezonep date-elements time-elements date-separator time-separator
internal-separator)
Produces on stream the timestring corresponding to the LOCAL-TIME with the
given options.
UNIVERSAL-TIME
[Function] Return the UNIVERSAL-TIME corresponding to the LOCALTIME .
INTERNAL-TIME
[Function] Return the internal system time corresponding to the LOCALTIME .
UNIX-TIME
[Function] Return the Unix time corresponding to the LOCAL-TIME .
TIMEZONE
[Function] Arguments: (local-time &optional timezone)
Return as multiple values the time zone as the number of seconds east of UTC ,
a boolean daylight-saving-p, the customary abbreviation of the timezone, the
starting time of this timezone, and the ending time of this timezone
LOCAL-TIMEZONE
[Function] Arguments: (adjusted-local-time &optional timezone)
Return the local timezone adjustment applicable at the already adjusted-localtime. Used to reverse the effect of TIMEZONE and LOCAL-TIME-ADJUST .
DEFINE-TIMEZONE
[Macro] Arguments: (zone-name zone-file &key load)
100

Define zone-name (a symbol or a string) as a new timezone, lazy-loaded from
zone-file (a pathname designator relative to the zoneinfo directory on this
system). If load is true, load immediately.
DEFAULT-TIMEZONE
+ [Variable] Holds the default timezone for all time operations needing a default.
10 Conclusions
1. The absence of a standard notation for time in Common Lisp required all this work.
2. The presence of International Standards for the representation of time made it
all a lot easier.
3. Time basically has the most messed-up legacy you can imagine.
4. Pope Gregory XIII made it a little easier on us all.
5. Adoption of this proposal in Common Lisp systems and applications would make
time a lot easier for almost everyone involved, except users who cling to the
habits that caused the Y2K problems.
6. This package is far from complete.
11 Credits and Acknowledgments

This work has been funded by the author and by NHST, publishers of Norways
financial daily, and TDN, their electronic news agency, and has been a work
in progress since late 1997. My colleagues and managers have been extremely
supportive in bringing this fundamental work to fruition. In particular, Linn Ir;n
Humlekj;r and Erik Haugan suffered numerous weird proposals and false starts
but encouraged the conceptual framework and improved on the execution with
their ideas and by lending me an ear. My management line, consisting of OleMartin Halden, Bjrn Hole, and Hasse Farstad, have encouraged the quality of the
implementation and were willing listeners to the many problems and odd ideas that
preceded the realization that this had to be done.
The great guys at Franz Inc have helped with internal details in Allegro CL and have
of course made a wonderful Common Lisp environment to begin with. Thanks in
particular to Samantha Cichon and Anna McCurdy for taking care of all the details
and making my stays so carefree, and to Liliana Avila for putting up with my total
lack of respect for deadlines.
101

Many thanks to Pernille Nylehn for reading and commenting on drafts, nudging me
towards finishing this work, and for taking care of my cat Xyzzy so I could write
this in peace and deliver it at LUGM '99 without worrying about the little furballs
constant craving for attention, but also without both their warmth and comfort when
computers simply refuse to behave rationally.
102
The make it work, make it right, make

it fast misconception
Written by: Henrique Bastos at 2009-08-18
On the agile way of doing software development, one of developers favorite mantras
is: Make it Work, Make it Right, Make it Fast.
However, its not uncommon to see people complaining that by following this
directions, their project never get to the make it fast stage. That happens because
of a misconception where people treats these three steps as isolated actions of a
project. And they are three stages of one development task.
While you start a development task, you have an early stage where you need to
explore possibilities. You dont quite know what is the best approach but have
some ideas. If during that stage you go big looking for THE piece of code that
will fit the problem perfectly, you will end up spending a considerable amount
of time on that quest. Thats because you have too many uncertainties on your
solution and you need to discover through experimentation to reduce those
uncertainties to a comfortable amount so you can focus and implement your code.
This experimentation process is the Make it Work stage where you just code to
achieve the desired behavior or effect.
Just after that, youd probably discovered what kind of libraries you will use, how
the rest of your code base interacts with this new behavior, etc. Thats when you go
for refactoring and Make it Right. Now you dry things out and organize your code
properly to have a good design and be easily maintainable.
As Software Developers, we know its true that restrictions are everywhere. And for
this we know that there are tradeoffs between design and performance. This is when
you go for the last stage where you Make it Fast by optimizing your code to achieve
the needed performance. By doing this at this point, you will see that it will require
much less effort. Thats because you can focus on the small 20% of code you can
tune to end up with a 80% performance increase.
Ending all those three stages, you will reach the end of your development task and
can integrate this code into your project, delivering the desired value to your Product
Owner.
1
http://henriquebastos.net/the-make-it-work-make-it-right-make-it-fast-misconception/
103
The make it work, make it right, make it fast misconception

If you ignore one of those steps while youre coding that one task, you will probably
suffer the effects of doing big design up front, increasing technical debit, and having
a awful system performance.
That said, a last piece of advice: Be careful not to compensate bad time estimations
delivering code that only "works". If you do it as a rule, I assure you that you will end
up with some non-tested spaghetti project where your cost of change will be so high
that can put your business at risk and make your "agile" useless.
104
The Myth of Schema-less

Written by: Jon Haddad at Wed 09 July 2014
I have grown increasingly frustrated with the world as people have become more
and more convinced that "schema-less" is actually a feature to be proud of (or even
exists). For over ten years Ive worked with close to a dozen different databases in
production and have not once seen "schemaless" truly manifest. Whats extremely
frustrating is seeing this from vendors, who should really know better. At best, we
should be using the description "provides little to no help in enforcing a schema" or
"youre on your own, good luck."
Before we dig any further into this topic, I think its important to understand how we
got here. Lets go back 5-10 years. The default database for a majority of projects
was typically a RDBMS. MySQL is all the rage and people cant get enough PHP. ORMs
like hibernate made it easier to work with a RDBMS without actually treating it like
one. Everyone is sort of happy, but the tools really arent all that great and mostly do
a poor job of abstracting the DB away, or require XML files or some other nonsense.
In the end, the ORM could only take away some of the annoyance, mostly manually
written SQL queries. You still had to deal with the other (perceived) downsides of
the RDBMS:
It gets slow to add columns in the most popular implementations as your data
set grows.
Joins dont perform well with large datasets.
People wanted to think in terms of objects, and see having a schema as pointless
overhead.
SQL is perceived as hard
If youre interested in OLTP workloads, the RDBMS requires sharding and manual
management
Once you shard you lose out on most the fun SQL features
Looking at each of these limitations, the classic "lets just throw everything out and
start from scratch" mentality kicked in. This seems to be an effort to solve the
root problem, but the problem is misdiagnosed. The decision to eliminate schema
1
http://rustyrazorblade.com/2014/07/the-myth-of-schema-less/
105

from the database because there are some problems with the implementations of
schemas is total nonsense.
Slow alterations is sometimes hailed as a reason to go schema-less. Logically though,
there is no reason why changing your schema inherently must result in downtime or
take hours. The only reason why this is the case with the RDBMS world is because
it generally requires a rewrite of the db files to include the new column. This is an
2
implementation detail, and a choice, not a requirement. Cassandra , for instance,
uses a schema, and can perform CQL table alterations without rewriting the entire
3
table, as does sqlite . So this reason is a crock and can be dismissed as an excuse
to not support a typed schema.
Joins dont perform spectacularly across machines due to a number of reasons. For
OLTP workloads the common advice is to denormalize your data. Copying your data
once incurs a penalty on writes but returns many fold on reads. For OLAP workloads,
the industry has been doing a lot of map reduce, but this is most likely a means to
an end. Writing map reduce requires too much developer work and will mostly be
replaced by tools that either utilize a M/R framework or something better. The current
4
best candidate for this is SparkSQL , which integrates well with existing data stores,
and uses a DAG (directed acyclic graph) instead of map/reduce. Im not sure how
you could use a tool like Spark and glean any useful information out of a database
without any sort of defined structure somewhere. I think this is a clear case where
explicit schema is highly beneficial, so Ill add this to my list of "features that dont
need to be thrown out just because theyre associated w/ an RDBMS."
The third point, "schema as pointless overhead" is nonsense. In any non trivial
application, there is going to be a defined data model. The purpose of a schema is
not for the developer writing an MVP by himself, its for the team that has to deal
with his code for the foreseeable future. Schema is important, its the only way that
everyone that didnt write the original code can sanely understand the data model.
Providing a central location (the database!) to look at this schema, and a human
parsable / writable query language to examine it is undeniably useful. The overhead
with any non-trivial application will quickly shift to the developers maintaining
5
the codebase and away from managing the schema. Tools like Alembic make it
straightforward to manage RDBMS schema changes, as well as the sync_table
functionality in cqlengine, for Cassandra. The overhead of managing schema at this
2
3
http://cassandra.apache.org/
http://www.sqlite.org/lang_altertable.html
4
http://databricks.com/blog/2014/03/26/spark-sql-manipulating-structured-data-using-spark-2.html
5
http://alembic.readthedocs.org/en/latest/
106

point is significantly less than the mental overhead your dev team must exert. Not
maintaining your schema in a database becomes a form of technical debt.
SQL can be confusing for someone new to the language, but thats hardly an excuse
to throw either it or explicit schema out. SQL was designed to be human readable and
writable, and a solid understand of it wields its user incredibly power and flexibility.
Yes, you can write some really nasty queries, but theres no reason to use a dull
knife all the time just because you are afraid someone might cut themselves. This is
strongly supported if you consider the alternatives so far. As a replacement for SQL,
query languages based on JSON are total crap. They discourage you to actually look
at your database directly, requiring code to compose queries frequently based on
nested data structures. At a glance, its easy enough to compose simple queries for a
handful of rows. As the feature set grows however, you end up with something thats
completely unmanageable and a nightmare to edit. Even if we decided to ditch SQL
forever in favor of some other query language, it STILL does not force our hand to
even consider removing explicit schema.
Manual sharding with an RDBMS is a huge pain. Having to expand your production
database cluster via home grown migrations is at best an annoyance and at worst
a massive liability. This certainly has contributed to the decline of the RDBMS, but
theres no reason why this should lead to a wild west-esque data model. Like the
other excuses above, it has nothing to do with the case against schema.
So, for the above reasons, there is a stigma against the RDBMS, and with it, a rejection
of explicit database schema. How the two got to be tangled so close together is a
mystery - in any of the above points is there a solid technical reason to ditch schema?
Not one.
If there isnt a technical reason, there must be an advantage for a development
team. Without schema were free to do what exactly? Create a new class per
record? That sounds tedious after a handful of classes, and impossible with millions.
Put heterogeneous objects in the same container? Thats useful, but nothing new.
6
7
Sqlalchemy , a Python ORM has had this for a while now, so has cqlengine .
We dont seem to gain much in terms of database flexibility. Is our application more
flexible? I dont think so. Even without our schema explicitly defined in our database,
its there somewhere. You simply have to search through hundreds of thousands of
lines to find all the little bits of it. It has the potential to be in several places, making
6
http://docs.sqlalchemy.org/en/rel_0_9/orm/inheritance.html
7
https://cqlengine.readthedocs.org/en/latest/topics/models.html#table-polymorphism
107

it harder to properly identify. The reality of these codebases is that they are error
prone and rarely lack the necessary documentation. This problem is magnified when
there are multiple codebases talking to the same database. This is not an uncommon
practice for reporting or analytical purposes.
Finally, all this "flexibility" rears its head in the same way that PHP and Javascripts
"neat" weak typing stabs you right in the face. There are some somethings you can
be cavalier about, and some things you should be strict about. Your data model is
one you absolutely need to be strict on. If a field should store an int, it should store
nothing else. Not a string, not a picture of a horse, but an integer. Its nice to know
that I have my database doing type checking for me and I can expect a field to be
the same type across all records.
All this leads us to an undeniable fact: There is always a schema. Wearing "I
dont do schema" as a badge of honor is a complete joke and encourages a terrible
development practice.
108
NoDB
In the United States, in 1920, the manufacture, sale, and importation of alcoholic
beverages was prohibited by a constitutional amendment. That amendment was
repealed thirteen years later. During that period of prohibition, the beer industry
died.
In 1933, when prohibition was lifted, a few giant grain companies started brewing
beer. They completely cornered the market. And for nearly 50 years, we in the United
State drank this fizzy bodily effluent and called it beer. The only way to tolerate
the flavor was to drink it very cold.
As a teenager in the 60s, I never understood the attraction. Why beer? It was a pale,
yellow, distasteful fluid derived from the urine of sick boars, and had no redeeming
qualities that I could see.
In 1984, I went to England; and the scales dropped from my eyes. At last I understood.
I had tasted beer for the first time; and I found it to be good.
Since those days the beer situation in the United States has improved dramatically.
New beer companies are springing up all over the country; and in many cases the
beer they make is actually quite good. We dont have anything quite so nice as a
good english bitter; but were getting close.
In the 80s a few giant database companies cornered the market. They did this by
promulgating fear, uncertainty, and doubt amongst managers and marketing people.
The word relational became synonymous with good; and any other kind of data
storage mechanism was prohibited.
I was the lead developer in a startup in those days. Our product measured the quality
of T1 communications lines. Our data model was relatively simple, and we kept the
data in flat files. It worked fine.
But our marketing guy kept on telling us that we had to have a relational database.
He said that customers would demand it. I found that to be a strange claim since we
hadnt sold even one system at that time, and no customer had ever mentioned our
1
http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html
109
NoDB
data storage technology. But the marketing guy was adamant. We just had to have
a relational database. Flat files were prohibited.
As the lead developer, responsible for the quality of the software, my view of a
relational database was that it would be a big, stogy, slow, expensive pain in the rear.
We didnt have complex queries. We didnt need massive reporting capabilities. We
certainly didnt need a process with a multi-megabyte footprint sitting in memory
and burning cycles. (Remember, this was the 80s). So I fought against this idea with
everything I had; because it was the wrong technical solution.
This was not a politically astute move for me. Over a period of several months,
a hardware engineer who managed to write a few lines of code, was moved into
the software group. He was gradually given more and more responsibility, and was
eventually named my co-manager. He and I would share the responsibility for
leading the software team.
Uh huh. Sure. Right. A hardware guy with no real software experience was going to
help me lead the team. And what do you think his first issue was? Why it was to
get a relational database into our system!
I left a month later and started my consulting career. It was best career move I have
ever made. The company I left no longer exists. I dont think they ever made a dime.
I watched the relational database market grow during the 90s. I watched as all
other data storage technologies, like the object databases, and the B-tree databases
dwindled and died; like the beer companies in the 20s. By the end of the 90s, only
the giants were left.
Those giants were marketing up a storm. They were gods. They were rulers. During
the dot com bubble, one of them actually had the audacity to buy television ads that
claimed that their product was the power that drove the internet. That reminded
me of a beer slogan from the 70s Ya gotta grab for all the gusto in life ya can.
Oh brother.
During this time I watched in horror as team after team put the database at the
center of their system. They had been convinced by the endless marketing hype
that the data model was the most important aspect of the architecture, and that the
database was the heart and soul of the design.
I witnessed the rise of a new job function. The DBA! Mere programmers could not be
entrusted with the data so the marketing hype told us. The data is too precious,
too fragile, too easily corrupted by those undisciplined louts. We need special people
110
NoDB
to manage the data. People trained by the database companies. People who would
safeguard and promulgate the giant database companies marketing message: that
the database belongs in the center. The center of the system, the enterprise, the
world, the very universe. MUAHAHAHAHAHAHA!
I watched as SQL slipped through every crack and crevice in the system. I ran
screaming from systems in which SQL had leaked into the UI. I railed endlessly
against the practice of moving all business rules into stored procedures. I quailed
and quaked and ranted and raved as I read through entire mail-merge programs
written in SQL.
I hammered and hammered as I saw tables and rows permeating the source code
of system after system. I hammered out danger. I hammered out a warning. I
hammered out that the schema had become The Blob, consuming everything in
sight. But I knew all my hammering was just slinging pebbles at a behemoth.
And then, in the first decade of the 21st century, the prohibition was lifted, and the
NOSQL movement was born. I considered it a kind of miracle, a light shining forth
in the wilderness. Finally, someone realized that there might just be some systems
in the world that did not require a big, fat, horky, slow, expensive, bodily effluent,
memory hog of a relational database!
I watched in glee as I saw BigTable, Mongo, CouchDB, and all the other cute little
data storage systems begin to spring up; like little micro-breweries in the 80s. The
beer was back! And it was starting to taste good.
But then I noticed something. Some of the systems using these nice, simple,
tasty, non-relational databases were being designed around those databases. The
database, wrapped in shiny new frameworks, was still sitting at the center of the
design! That poisonous old relational marketing hype was still echoing through the
minds of the designers. They were still making the fatal mistake.
Stop! I yelled. Stop! You dont understand. You dont understand. But the
momentum was too great. An enormous wave of frameworks rose up and smashed
down on our industry, washing over the land. Those frameworks wrapped up the
databases and fought to grab and hold the center of our applications. They claimed
to master and tame the databases. They even claimed to be able to turn a relational
database into a NoSQL database. And the frameworks cried out with a great voice
heard all over the land: Depend on me, and Ill set you free!
The name of this article is No DB. Perhaps after that rant you are getting an inkling
of why I named it that.
111
NoDB
The center of your application is not the database. Nor is it one or more of the
frameworks you may be using. The center of your application are the use cases of
your application.
It makes me crazy when I hear a software developer describe his system as a
Tomcat system using Spring and Hibernate using Oracle. The very wording puts
the frameworks and the database at the center.
What do you think the architecture of that system would look like? Do you think youd
find the use cases at the center of the design? Or would you find the source code
arranged to fit nicely into the pattern of the frameworks? Would you find business
objects that looked suspiciously like database rows? Would the schema and the
frameworks pollute everything?
Heres what an application should look like. The use cases should be the highest
level and most visible architectural entities. The use cases are at the center. Always!
Databases and frameworks are details! You dont have to decide upon them up front.
You can push them off until later, once youve got all the use cases and business
rules figured out, written, and tested.
What is the best time to determine your data model? When you know what the data
entities are, how they are related, and how they are used. When do you know that?
When youve gotten all the use cases and business rules written and tested. By
that time you will have identified all the queries, all the relationships, all the data
elements, and youll be able to construct a data model that fits nicely into a database.
Does this change if you are using a NoSql database? Of course not! You still focus on
getting the use cases working and tested before you even think about the database;
no matter what kind of database it ends up being.
If you get the database involved early, then it will warp your design. Itll fight to gain
control of the center, and once there it will hold onto the center like a scruffy terrier.
You have to work hard to keep the database out of the center of your systems. You
have to continuously say No to the temptation to get the database working early.
We are heading into an interesting time. A time when the prohibition against different
data storage mechanisms has been lifted, and we are free to experiment with many
novel new approaches. But as we play with our CouchDBs and our Mongos and
BigTables, remember this: The database is just a detail that you dont need to figure
out right away.
112
7 coding tasks you should probably

not write yourself
Written by: Andy Lester at 2014-07-08
As programmers, we like to solve problems. We like to get ideas to spring from our
heads, channel through our fingertips, and create magical solutions.
But sometimes we are too quick to jump in and start cranking out code to solve
the problem without considering all the implications of the issues were trying to
solve. We dont consider that someone else might have already solved this problem,
with code available for our use that has already been written, tested, and debugged.
Sometimes we just need to stop and think before we start typing.
For example, when you encounter these seven coding problems, youre almost
always better off looking for an existing solution than trying to code one yourself:
1. Parsing HTML or XML

The task whose complexity is most often underestimatedat least based on how
many times its asked about on StackOverflow is parsing HTML or XML. Extracting
data from arbitrary HTML looks deceptively simple, but really should be left to
libraries. Say youre looking to extract a URL from an <img> tag like
<img src="foo.jpg">
Its a simple regular expression to match a pattern.

/<img src="(.+?)">/
The string foo.jpg will be in capture group #1 and can be assigned to a string. But
will your code handle tags with other attributes like:
<img id="bar" src="foo.jpg">
And after you change your code to handle that, will it handle alternate quotes like:
1
http://blog.newrelic.com/2014/07/08/7-things-never-code/
113
7 coding tasks you should probably not write yourself
<img
src= 'foo.jpg'>
or no quotes at all, like:

<img src=foo.jpg>
What about if the tag spans multiple lines and is self-closing, like:
<img id="bar"
src="foo.jpg"
/>
And will your code know to ignore this commented-out tag:

<!-<img src="foo.jpg">
-->
By the time youve gone through the cycle of finding yet another valid case your
code doesnt handle, modifying the code, retesting it, and trying it again, you could
have used a proper library and been done with it.
Thats the story with all of these examples: Youll spend far less time finding an
existing library and learning to use it rather than trying to roll your own code, then
debug it, then extend it to fit all the cases you hadnt thought of when you started.
2. Parsing CSV and JSON

CSV files are deceptively simple, yet fraught with peril. Files of comma-separated
values are trivial to parse, right?
# ID, name, city
1, Queen Elizabeth II, London
Sure, until you have double-quoted values to handle embedded commas:

2, J. R. Ewing, "Dallas, Texas"
114

Once you get around those double-quoted values, what happens when you have
embedded double quotes in a string that have to be escaped:
3, "Larry \"Bud\" Melman", "New York, New York"
You can get around those, too, until you have to deal with embedded newlines in
the middle of a record.
JSON has all the same data type hazards of CSV, with the added headache of being
able to store multi-level data structures.
Save yourself the hassle and inaccuracy. Any data that cant be handled with splitting
the string on a comma should be left to a library.
If its bad to read structured data in an unstructured way, its even worse to try to
modify it in place. People often say things like, I want to change all the <img>
tags with such-and-such a URL so they have a new attribute. But even something
as seemingly simple as I want to change any fifth field in this CSV with the
name Bob to Steve is dangerous because as noted above, you cant just count
commas. To be safe you need to read the datausing a comprehensive libraryinto an
internal structure, modify the data, and then write it back out with the same library.
Anything less risks corrupting the data if its structure doesnt precisely match your
expectations.
3. Email address validation

There are two ways you can validate an email address. You can have a very simple
check like, I need to have some characters before an @ sign, and then some
characters after, matching it against this regular expression:
/.+@.+/
Its not complete and it lets invalid stuff through, but at least youve got an @ sign
in the middle.
2
Or you can validate it against the rules in RFC 822. Have you read RFC 822 ? It covers
all sorts of things that you rarely see but are still legal. A simple regular expression
isnt going to cut it. Youre going to need to use a library that someone else has
already written.
2
http://www.ietf.org/rfc/rfc0822.txt
115

If youre not going to validate against RFC822, then anything else you do is going
to be a matter of using a subset of rules that seem reasonable but might not be
correct. Thats a valid design tradeoff to make many times, but dont fool yourself
into thinking that youve covered all the cases unless you go back to the RFC, or use
a library written by someone who has.
(For far more discussion about validation of email addresses, see this StackOverflow
3
thread .)
4. Processing URLs
URLs arent nearly as odious as email addresses, but theyre still full of annoying
little rules you have to remember. What characters need to be encoded? How do
you handle spaces? How about + signs? What characters are valid to go in that part
after the # sign?
Whatever language youre working in, theres code to break apart URLs into the
components you need, and to reassemble them from parts, properly formatted.
5. Date/time manipulation
Date/time manipulation is the king of problem sets with rules you cant possibly wrap
your head around all at once. Date/time handling has to account for time zones,
daylight saving time, leap years, and even leap seconds. In the United States, we
have only four time zones to think about, and theyre all an hour apart. The rest of
the world is not so simple.
Whether its date arithmetic where youre figuring out what three days after another
day is, or youre validating that an input string is in fact a valid date, use an existing
library.
6. Templating systems
Its almost a rite of passage. A junior programmer has to create lots of boilerplate
text and comes up with a simple little format like:
Dear #user#,
Thank you for your interest in #product#...
3
http://stackoverflow.com/questions/201323
116

It works for a while, but then she winds up having to add multiple output formats,
and numeric formatting, and outputting structured data in tables, and on and on until
shes built an ad hoc monster requiring endless care and feeding.
If youre doing anything more complex than simple string-for-string substitution, step
back and find yourself a good templating library. To make things even simpler, if
youre writing in PHP, the language itself is a templating system (though that is often
forgotten these days).
7. Logging frameworks
Logging tools are another example of projects that start small and grow into
behemoths. One little function for logging to a file soon needs to be able to log to
multiple files, or to send email on completion, or have varying log levels and so on.
Whatever language youre using, there are at least three log packages that have
already been around for years and will save you no end of aggravation.
But isnt a library overkill?

Before you pooh-pooh the idea of a module, take a hard look at your objections. The
Number 1 objection is usually, Why do I need an entire library just to do (validate this
date/parse this HTML/etc.), My response is, Whats the harm in using it? Chances
are youre not writing microcontroller code for a toaster where you have to squeeze
out every byte of code space.
If you have speed concerns, consider that avoiding a library may be premature
optimization. Maybe loading up an entire date/time library makes your date
validation take 10 times as long as your mostly correct homemade solution, but
profile your code first to see if it actually matters.
We programmers are proud of our skills, and we like the process of creating code.
Thats fine. Just remember that your job as a programmer is not to write code but to
solve problems, and often the best way to solve a problem is to write as little code
as possible.
117
You Need This One Skill to Succeed in

IT
Written by: Jes Schultz Borland at July 17, 2013
The ability to program in five languages, including one machine-level? Not it.
Project management skills, up to and including a PMP certification? Not that either.
Excellent oral and written communication skills, as noted on every job description
ever? That doesnt hurt, but can be learned.
All of the best IT professionals I have worked with have excellent problem solving
skills.
Thats it.
We face problems in IT on a regular basis. From the help desk technicians who are
asked, Why am I locked out of my computer? to the SAN administrators who have
to balance the needs of different servers and workloads to the DBA who is asked,
Why is the server so slow? we are all given problems to solve.
How we go about solving those problems is what sets the great professionals apart
from the good or the average.
Problem Solving Methodology

In high school, I was introduced to the scientific method. The process is: Does this
remind you of your server room?
Does this remind you of your server room?
1. Formulate a question.
2. Make a hypothesis.
3. Make a prediction.
4. Test the hypothesis. Measurements are emphasized.
5. Analyze the results.
1
http://www.brentozar.com/archive/2013/07/you-need-this-one-skill-to-succeed-in-it/
118
You Need This One Skill to Succeed in IT

Can this be applied to your problems? Yes, it can.
Formulate a question usually, the question is asked of you. Why is the server
slow? Why cant I connect to the database? Why is this report execution timing
out?
Make a hypothesis perhaps a patch was installed on the server or SQL Server the
night before. Maybe a network cable could have been unplugged. Maybe a developer
changed a line of code in the stored procedure. Make a list of what could affect the
system in question, so you have items to test.
Make a prediction take a guess at what your results will be. If this is an error
or problem youve encountered before, youll probably have a good idea of what
to expect. If its a new problem, use your past experiences and deductive skills to
determine what changes you make will do to the system.
Test the hypothesis make a plan, make a change, and check if the problem is
solved. Dont make three changes and wonder which one fixed it fix one at a time.
Know what success looks like. If a query is slow, know what performance was before
the problem occurred, what performance is when the problem is happening, and
what acceptable performance looks like. Metrics are important here. You must be
able to measure if things improved, stayed the same, or got worse.
Analyze the results check those metrics. Did you get the results you expected?
If so, is the problem resolved? If not, what is the next item on your list to check?
Continue to iterate through your list until the problem is solved.
Anyone Can Do This

It doesnt require a PhD in computer science. It doesnt require a masters degree
in chemistry. What it does take is a consistent approach to problems every time. It

takes curiosity and an ability to see patterns.
With practice, this becomes much easier. Practice your problem-solving skills often.
They will make you a great IT professional, and set you apart from the rest of the
crowd.
119
ORM Is Not a Choice and How to

Make It Suck Less
Written by: Elnur Abdurrakhimov at 26 Apr 2014
There is a lot of ORM hate lately. Some people even call it The Vietnam of Computer
2
Science .
ORM vs ORM Tools

First, lets clarify the difference between ORM and ORM tools.
ORM object-relational mapping is a technique used to map object-oriented
constructs like classes and objects to relational database structures like tables and
rows.
An ORM tool is a particular tool like Hibernate or Doctrine that does that mapping
for you.
ORM Is Not a Choice

If the model in your application is implemented in object-oriented style and you use
a relational database to persist data, ORM is not a choice. Whether you like it or not,
you have to map the application code to the relational constructs of the database.
The only choice you have in this case is whether to use an existing ORM tool or roll
up your own. Thats it.
Of course, there are other options like using NoSQL tools for persistence or
abandoning OO style in the model, but thats another story.
Making ORM Suck Less

While ORM itself is not a choice, there is a choice in using an ORM tool in a way that
makes it suck less.
1
http://elnur.pro/orm-is-not-a-choice-and-how-to-make-it-suck-less/
2
http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx
120
ORM Is Not a Choice and How to Make It Suck Less
Do Not Let an ORM Tool Generate Database Schema

I see this too often. A lot of developers generate their database schemas with an
ORM tool and thats a bad idea.
First, ORM tools suck in schema generation. They generate schemas in probably the
most blunt way one could imagine. They also name constraints with useless names
like FK_D95AB405F92F3E70 or IDX_9C0600CAF6BD1646 . If you dont name a
constraint explicitly, sane DBMSes like PostgreSQL will come up with much better
names on their own.
Second, ORM tools try to support as many DB vendors as possible, so they use only a
portable subset of DDL. This means they cant take advantage of powerful features
of a particular DBMS you went with. For example, PostgreSQL has a CHECK constraint
that an ORM wont generate:
CREATE TABLE person
(
sex varchar NOT NULL,
CHECK (sex IN ('male', 'female'))
);
Take back the control of your database schema. Write database migrations in plain
SQL and adapt ORM mapping to it not the other way around.
Do Not Let an ORM Tool Dictate a Particular Schema

Design your schema using relation databases idioms and then use an ORM tool to
map the schema to the application code. Do not bend your schema to please a
particular ORM tool youre using.
For example, there are ORM tools that force you to have an autoincrement ID field
playing the role of they primary key in each table. From the relational database
design point of view, it doesnt always make sense to add a surrogate key when you
already have a natural one. For example, ISBN is a great natural key for a book; you
dont have to add a surrogate ID column to the books table.
If your ORM tool forces to bend your schema to its own will, look for a better tool.
121
ORM Is Not a Choice and How to Make It Suck Less
Do Not Let an ORM Tool Dictate a Particular Model

Structure
Some ORM tools force you to extend some base class or do something else to please
them. Those are mostly implementations of the Active Record pattern.
There are other ORM tools that implement the Data Mapper pattern and they let you
keep your model structure the way you would if you were not using an ORM tool at
all. Hibernate and Doctrine are examples of that.
Do not let an ORM bend your model to its own will. If it does, throw it away and find
a better alternative.
Use ORM Tools for Mapping Only

Since marshalling objects to rows and back is mostly boilerplate code and very boring
to write, its a good idea to use an existing ORM tool to do that for you. Just dont let
it do anything else because thats out of ORM and ORM tools concern.
Conclusion
ORM is not a choice when you have OO code on one side and RDBMS on another. But
there is a choice of ORM tools and ways to use them that suck less.
122
Programming Sucks
Every friend I have with a job that involves picking up something heavier than a
laptop more than twice a week eventually finds a way to slip something like this into
1
conversation: "Bro, you dont work hard. I just worked a 4700-hour week digging a
tunnel under Mordor with a screwdriver."
They have a point. Mordor sucks, and its certainly more physically taxing to dig
a tunnel than poke at a keyboard unless youre an ant. But, for the sake of the
argument, can we agree that stress and insanity are bad things? Awesome. Welcome
to programming.
All programming teams are constructed by and

of crazy people
Imagine joining an engineering team. Youre excited and full of ideas, probably just
out of school and a world of clean, beautiful designs, awe-inspiring in their aesthetic
unity of purpose, economy, and strength. You start by meeting Mary, project leader
for a bridge in a major metropolitan area. Mary introduces you to Fred, after you get
1
It always starts with "Bro"
123
Programming Sucks
through the fifteen security checks installed by Dave because Dave had his sweater
stolen off his desk once and Never Again. Fred only works with wood, so you ask
why hes involved because this bridge is supposed to allow rush-hour traffic full of
cars full of mortal humans to cross a 200-foot drop over rapids. Dont worry, says
Mary, Freds going to handle the walkways. What walkways? Well Fred made a good
case for walkways and theyre going to add to the bridges appeal. Of course, theyll
have to be built without railings, because theres a strict no railings rule enforced by
Phil, whos not an engineer. Nobodys sure what Phil does, but its definitely full of
synergy and has to do with upper management, whom none of the engineers want to
deal with so they just let Phil do what he wants. Sara, meanwhile, has found several
hemorrhaging-edge paving techniques, and worked them all into the bridge design,
so youll have to build around each one as the bridge progresses, since each one
means different underlying support and safety concerns. Tom and Harry have been
working together for years, but have an ongoing feud over whether to use metric
or imperial measurements, and its become a case of "whoever got to that part of
the design first." This has been such a headache for the people actually screwing
things together, theyve given up and just forced, hammered, or welded their way
through the day with whatever parts were handy. Also, the bridge was designed as
a suspension bridge, but nobody actually knew how to build a suspension bridge,
so they got halfway through it and then just added extra support columns to keep
the thing standing, but they left the suspension cables because theyre still sort of
holding up parts of the bridge. Nobody knows which parts, but everybodys pretty
sure theyre important parts. After the introductions are made, you are invited to
come up with some new ideas, but you dont have any because youre a propulsion
engineer and dont know anything about bridges.
Would you drive across this bridge? No. If it somehow got built, everybody involved
would be executed. Yet some version of this dynamic wrote every single program
you have ever used, banking software, websites, and a ubiquitously used program
that was supposed to protect information on the internet but didnt.
All code is bad

Every programmer occasionally, when nobodys home, turns off the lights, pours
a glass of scotch, puts on some light German electronica, and opens up a file on
their computer. Its a different file for every programmer. Sometimes they wrote it,
sometimes they found it and knew they had to save it. They read over the lines, and
weep at their beauty, then the tears turn bitter as they remember the rest of the files
and the inevitable collapse of all that is good and true in the world.
124
Programming Sucks
This file is Good Code. It has sensible and consistent names for functions and
variables. Its concise. It doesnt do anything obviously stupid. It has never had to
live in the wild, or answer to a sales team. It does exactly one, mundane, specific
thing, and it does it well. It was written by a single person, and never touched by
another. It reads like poetry written by someone over thirty.
Every programmer starts out writing some perfect little snowflake like this. Then
theyre told on Friday they need to have six hundred snowflakes written by Tuesday,
so they cheat a bit here and there and maybe copy a few snowflakes and try to stick
them together or they have to ask a coworker to work on one who melts it and then all
the programmers' snowflakes get dumped together in some inscrutable shape and
somebody leans a Picasso on it because nobody wants to see the cat urine soaking
into all your broken snowflakes melting in the light of day. Next week, everybody
shovels more snow on it to keep the Picasso from falling over.
Theres a theory that you can cure this by following standards, except there are more
"standards" than there are things computers can actually do, and these standards are
all variously improved and maligned by the personal preferences of the people coding
them, so no collection of code has ever made it into the real world without doing a
few dozen identical things a few dozen not even remotely similar ways. The first few
weeks of any job are just figuring out how a program works even if youre familiar with
every single language, framework, and standard thats involved, because standards
are unicorns.
There will always be darkness

I spent a few years growing up with a closet in my bedroom. The closet had an
odd design. It looked normal at first, then you walked in to do closet things, and
discovered that the wall on your right gave way to an alcove, making for a handy
little shelf. Then you looked up, and the wall at the back of the alcove gave way
again, into a crawlspace of utter nothingness, where no light could fall and which you
immediately identified as the daytime retreat for every ravenous monster you kept
at bay with flashlights and stuffed animals each night.
This is what it is to learn programming. You get to know your useful tools, then you
look around, and there are some handy new tools nearby and those tools show you
the bottomless horror that was always right next to your bed.
For example, say youre an average web developer. Youre familiar with a dozen
programming languages, tons of helpful libraries, standards, protocols, what have
125
Programming Sucks
you. You still have to learn more at the rate of about one a week, and remember to
check the hundreds of things you know to see if theyve been updated or broken and
make sure they all still work together and that nobody fixed the bug in one of them
that you exploited to do something you thought was really clever one weekend when
you were drunk. Youre all up to date, so thats cool, then everything breaks.
"Double you tee eff?" you say, and start hunting for the problem. You discover that
one day, some idiot decided that since another idiot decided that 1/0 should equal
infinity, they could just use that as a shorthand for "Infinity" when simplifying their
code. Then a non-idiot rightly decided that this was idiotic, which is what the original
idiot should have decided, but since he didnt, the non-idiot decided to be a dick and
make this a failing error in his new compiler. Then he decided he wasnt going to tell
anyone that this was an error, because hes a dick, and now all your snowflakes are
urine and you cant even find the cat.
You are an expert in all these technologies, and thats a good thing, because that
expertise let you spend only six hours figuring out what went wrong, as opposed to
losing your job. You now have one extra little fact to tuck away in the millions of little
facts you have to memorize because so many of the programs you depend on are
written by dicks and idiots.
And thats just in your own chosen field, which represents such a tiny fraction of all the
things there are to know in computer science you might as well never have learned
anything at all. Not a single living person knows how everything in your five-year-old
MacBook actually works. Why do we tell you to turn it off and on again? Because we
dont have the slightest clue whats wrong with it, and its really easy to induce coma
in computers and have their built-in team of automatic doctors try to figure it out for
us. The only reason coders' computers work better than non-coders' computers is
coders know computers are schizophrenic little children with auto-immune diseases
and we dont beat them when theyre bad.
A lot of work is done on the internet and the

internet is its own special hellscape
Remember that stuff about crazy people and bad code? The internet is that except its
literally a billion times worse. Websites that are glorified shopping carts with maybe
three dynamic pages are maintained by teams of people around the clock, because
the truth is everything is breaking all the time, everywhere, for everyone. Right now
someone who works for Facebook is getting tens of thousands of error messages and
126
Programming Sucks
frantically trying to find the problem before the whole charade collapses. Theres a
team at a Google office that hasnt slept in three days. Somewhere theres a database
programmer surrounded by empty Mountain Dew bottles whose husband thinks shes
dead. And if these people stop, the world burns. Most people dont even know what
sysadmins do, but trust me, if they all took a lunch break at the same time they
wouldnt make it to the deli before you ran out of bullets protecting your canned
goods from roving bands of mutants.
You cant restart the internet. Trillions of dollars depend on a rickety cobweb of
unofficial agreements and "good enough for now" code with comments like "TODO:
FIX THIS ITS A REALLY DANGEROUS HACK BUT I DONT KNOW WHATS WRONG" that
were written ten years ago. I havent even mentioned the legions of people attacking
various parts of the internet for espionage and profit or because theyre bored. Ever
heard of 4chan? 4chan might destroy your life and business because they decided
they didnt like you for an afternoon, and we dont even worry about 4chan because
another nuke doesnt make that much difference in a nuclear winter.
On the internet, its okay to say, "You know, this kind of works some of the time if
youre using the right technology," and BAM! its part of the internet now. Anybody
with a couple of hundred dollars and a computer can snag a little bit of the internet
and put up whatever awful chunks of hack code they want and then attach their little
bit to a bunch of big bits and everything gets a little bit worse. Even the good coders
dont bother to learn the arcane specifications outlined by the organizations people
set up to implement some unicorns, so everybody spends half their time coping with
the fact that nothing matches anything or makes any sense and might break at any
time and we just try to cover it up and hope no one notices.
Here are the secret rules of the internet: five minutes after you open a web browser
for the first time, a kid in Russia has your social security number. Did you sign up for
something? A computer at the NSA now automatically tracks your physical location
for the rest of your life. Sent an email? Your email address just went up on a billboard
in Nigeria.
These things arent true because we dont care and dont try to stop them, theyre
true because everything is broken because theres no good code and everybodys
just trying to keep it running. Thats your job if you work with the internet: hoping
the last thing you wrote is good enough to survive for a few hours so you can eat
dinner and catch a nap.
127
Programming Sucks
We didnt start out crazy, were being driven

crazy
ERROR: Attempted to parse HTML with regular expression; system returned Cthulhu.
Funny, right? No? How about this exchange:
"Is that called arrayReverse?"
"s/camel/_/"
"Cool thanks."
Wasnt that guy helpful? With the camel? Doesnt that seem like an appropriate
response? No? Good. You can still find Jesus. You have not yet spent so much of
your life reading code that you begin to talk in it. The human brain isnt particularly
good at basic logic and now theres a whole career in doing nothing but really, really
complex logic. Vast chains of abstract conditions and requirements have to be picked
through to discover things like missing commas. Doing this all day leaves you in a
state of mild aphasia as you look at peoples faces while theyre speaking and you
dont know theyve finished because theres no semicolon. You immerse yourself in
a world of total meaninglessness where all that matters is a little series of numbers
went into a giant labyrinth of symbols and a different series of numbers or a picture
of a kitten came out the other end.
The destructive impact on the brain is demonstrated by the programming languages
people write. This is a program:
#include <iostream>
int main( int argc, char** argv ) {
std::cout << "Hello World!" << std::endl;
return 0;
}
That program does exactly the same thing as this program:

`r```````````.H.e.l.l.o. .w.o.r.l.di
And this program:

128
Programming Sucks
>+++++++++[<++++++++>-]<.>+++++++[<++++>-]<+.+++++++..+++.[-]
>++++++++[<++++>-] <.>+++++++++++[<++++++++>-]<-.--------.+++
.------.--------.[-]>++++++++[<++++>- ]<+.[-]++++++++++.
And this one:

Ook. Ook?
Ook.
Ook. Ook.
Ook.
Ook. Ook.
Ook.
Ook! Ook.
Ook.
Ook. Ook.
Ook?
Ook! Ook!
Ook.
Ook. Ook.
Ook.
Ook. Ook.
Ook.
Ook. Ook.
Ook.
Ook. Ook.
Ook.
Ook. Ook?
Ook.
Ook. Ook.
Ook.
Ook. Ook.
Ook.
Ook. Ook?
Ook.
Ook? Ook.
Ook.
Ook! Ook.
Ook.
Ook! Ook!
Ook!
Ook! Ook.
Ook!
Ook. Ook.
Ook.
Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
Ook. Ook. Ook! Ook? Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook? Ook! Ook! Ook? Ook! Ook?
Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
Ook! Ook? Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
Ook? Ook! Ook? Ook. Ook. Ook. Ook! Ook. Ook. Ook. Ook. Ook. Ook.
Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook. Ook! Ook. Ook. Ook. Ook.
Ook! Ook. Ook. Ook? Ook. Ook? Ook. Ook? Ook. Ook. Ook. Ook. Ook.
Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook? Ook? Ook. Ook.
Ook. Ook. Ook. Ook. Ook. Ook? Ook! Ook! Ook? Ook! Ook? Ook. Ook!
Ook. Ook? Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook? Ook? Ook. Ook.
Ook! Ook! Ook? Ook! Ook? Ook. Ook! Ook! Ook! Ook! Ook! Ook! Ook!
Ook? Ook. Ook? Ook. Ook? Ook. Ook! Ook. Ook. Ook. Ook. Ook. Ook.
Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook!
Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook!
Ook. Ook? Ook. Ook? Ook. Ook. Ook! Ook. Ook! Ook? Ook! Ook! Ook?
129
Programming Sucks
Ook. Ook. Ook. Ook. Ook! Ook.
And once somebody wrote a programming language that let somebody else write
this:
#:: ::-| ::-| .-. :||-:: 0-| .-| ::||-| .:|-. :||
open(Q,$0);while("){if(/^#(.*)$/){for(split('-',$1)){$q=0;for(split){s/|
/:.:/xg;s/:/../g;$Q=$_?length:$_;$q+=$q?$Q:$Q*20;}print
chr($q);}}}print"n";
#.: ::||-| .||-| :|||-| ::||-| ||-:: :|||-| .:|
"
According to the author, that program is "two lines of code that parse two lines
of embedded comments in the code to read the Mayan numbers representing the
individual ASCII characters that make up the magazine title, rendered in 90-degree
rotated ASCII art."
That program won a contest, because of course it did. Do you want to live in a world
like this? No. This is a world of where you can smoke a pack a day and nobody even
questions it. "Of course he smokes a pack a day, who wouldnt?" Eventually every
programmer wakes up and before theyre fully conscious they see their whole world
and every relationship in it as chunks of code, and they trade stories about it as if
sleepiness triggering acid trips is a normal thing that happens to people. This is a
world where people eschew sex to write a programming language for orangutans.
All programmers are forcing their brains to do things brains were never meant to do
in a situation they can never make better, ten to fifteen hours a day, five to seven
days a week, and every one of them is slowly going mad.
</rant>
So no, Im not required to be able to lift objects weighing up to fifty pounds. I traded
that for the opportunity to trim Satans pubic hair while he dines out of my open skull
so a few bits of the internet will continue to work for a few more days.
130
Revenge of the Nerds

Written by: Paul Graham at May 2002
We were after the C++ programmers. We managed to drag a lot of

them about halfway to Lisp.
Guy Steele co-author of the Java spec
In the software business there is an ongoing struggle between the pointy-headed
academics, and another equally formidable force, the pointy-haired bosses. Everyone
knows who the pointy-haired boss is, right? I think most people in the technology
world not only recognize this cartoon character, but know the actual person in their
company that he is modelled upon.
The pointy-haired boss miraculously combines two qualities that are common by
themselves, but rarely seen together: (a) he knows nothing whatsoever about
technology, and (b) he has very strong opinions about it.
Suppose, for example, you need to write a piece of software. The pointy-haired boss
has no idea how this software has to work, and cant tell one programming language
from another, and yet he knows what language you should write it in. Exactly. He
thinks you should write it in Java.
Why does he think this? Lets take a look inside the brain of the pointy-haired boss.
What hes thinking is something like this. Java is a standard. I know it must be,
because I read about it in the press all the time. Since it is a standard, I wont
get in trouble for using it. And that also means there will always be lots of Java
programmers, so if the programmers working for me now quit, as programmers
working for me mysteriously always do, I can easily replace them.
Well, this doesnt sound that unreasonable. But its all based on one unspoken
assumption, and that assumption turns out to be false. The pointy-haired boss
believes that all programming languages are pretty much equivalent. If that were
true, he would be right on target. If languages are all equivalent, sure, use whatever
language everyone else is using.
But all languages are not equivalent, and I think I can prove this to you without even
getting into the differences between them. If you asked the pointy-haired boss in
1
http://www.paulgraham.com/icad.html
131

1992 what language software should be written in, he would have answered with as
little hesitation as he does today. Software should be written in C++. But if languages
are all equivalent, why should the pointy-haired bosss opinion ever change? In fact,
why should the developers of Java have even bothered to create a new language?
Presumably, if you create a new language, its because you think its better in some
way than what people already had. And in fact, Gosling makes it clear in the first Java
white paper that Java was designed to fix some problems with C++. So there you
have it: languages are not all equivalent. If you follow the trail through the pointyhaired bosss brain to Java and then back through Javas history to its origins, you
end up holding an idea that contradicts the assumption you started with.
So, whos right? James Gosling, or the pointy-haired boss? Not surprisingly, Gosling is
right. Some languages are better, for certain problems, than others. And you know,
that raises some interesting questions. Java was designed to be better, for certain
problems, than C++. What problems? When is Java better and when is C++? Are
there situations where other languages are better than either of them?
Once you start considering this question, you have opened a real can of worms. If
the pointy-haired boss had to think about the problem in its full complexity, it would
make his brain explode. As long as he considers all languages equivalent, all he has
to do is choose the one that seems to have the most momentum, and since that
is more a question of fashion than technology, even he can probably get the right
answer. But if languages vary, he suddenly has to solve two simultaneous equations,
trying to find an optimal balance between two things he knows nothing about: the
relative suitability of the twenty or so leading languages for the problem he needs to
solve, and the odds of finding programmers, libraries, etc. for each. If thats whats
on the other side of the door, it is no surprise that the pointy-haired boss doesnt
want to open it.
The disadvantage of believing that all programming languages are equivalent is that
its not true. But the advantage is that it makes your life a lot simpler. And I think
thats the main reason the idea is so widespread. It is a comfortable idea.
We know that Java must be pretty good, because it is the cool, new programming
language. Or is it? If you look at the world of programming languages from a distance,
it looks like Java is the latest thing. (From far enough away, all you can see is the
large, flashing billboard paid for by Sun.) But if you look at this world up close, you
find that there are degrees of coolness. Within the hacker subculture, there is another
language called Perl that is considered a lot cooler than Java. Slashdot, for example,
132

is generated by Perl. I dont think you would find those guys using Java Server Pages.
But there is another, newer language, called Python, whose users tend to look down
2
on Perl, and more waiting in the wings.
If you look at these languages in order, Java, Perl, Python, you notice an interesting
pattern. At least, you notice this pattern if you are a Lisp hacker. Each one is
progressively more like Lisp. Python copies even features that many Lisp hackers
consider to be mistakes. You could translate simple Lisp programs into Python line
for line. Its 2002, and programming languages have almost caught up with 1958.
Catching Up with Math

What I mean is that Lisp was first discovered by John McCarthy in 1958, and popular
programming languages are only now catching up with the ideas he developed then.
Now, how could that be true? Isnt computer technology something that changes
very rapidly? I mean, in 1958, computers were refrigerator-sized behemoths with
the processing power of a wristwatch. How could any technology that old even be
relevant, let alone superior to the latest developments?
Ill tell you how. Its because Lisp was not really designed to be a programming
language, at least not in the sense we mean today. What we mean by a programming
language is something we use to tell a computer what to do. McCarthy did eventually
intend to develop a programming language in this sense, but the Lisp that we
actually ended up with was based on something separate that he did as a theoretical
3
exercise an effort to define a more convenient alternative to the Turing Machine.
As McCarthy said later,
Another way to show that Lisp was neater than Turing machines was
to write a universal Lisp function and show that it is briefer and more
comprehensible than the description of a universal Turing machine.
4
This was the Lisp function eval , which computes the value of a Lisp
expression. Writing eval required inventing a notation representing
Lisp functions as Lisp data, and such a notation was devised for the
purposes of the paper with no thought that it would be used to express
Lisp programs in practice.
2
http://www.paulgraham.com/accgen.html
3
http://www.paulgraham.com/rootsoflisp.html
4
http://lib.store.yahoo.net/lib/paulgraham/jmc.lisp
133

What happened next was that, some time in late 1958, Steve Russell, one of
McCarthys grad students, looked at this definition of eval and realized that if he
translated it into machine language, the result would be a Lisp interpreter.
This was a big surprise at the time. Here is what McCarthy said about it later in an
interview:
Steve Russell said, look, why dont I program this eval, and I said to
him, ho, ho, youre confusing theory with practice, this eval is intended
for reading, not for computing. But he went ahead and did it. That is,
he compiled the eval in my paper into [IBM] 704 machine code, fixing
bugs, and then advertised this as a Lisp interpreter, which it certainly
was. So at that point Lisp had essentially the form that it has today.
Suddenly, in a matter of weeks I think, McCarthy found his theoretical exercise
transformed into an actual programming language-- and a more powerful one than
he had intended.
So the short explanation of why this 1950s language is not obsolete is that it was not
technology but math, and math doesnt get stale. The right thing to compare Lisp to
is not 1950s hardware, but, say, the Quicksort algorithm, which was discovered in
1960 and is still the fastest general-purpose sort.
There is one other language still surviving from the 1950s, Fortran, and it represents
the opposite approach to language design. Lisp was a piece of theory that
unexpectedly got turned into a programming language. Fortran was developed
intentionally as a programming language, but what we would now consider a very
low-level one.
5
Fortran I , the language that was developed in 1956, was a very different animal
from present-day Fortran. Fortran I was pretty much assembly language with math.
In some ways it was less powerful than more recent assembly languages; there were
no subroutines, for example, only branches. Present-day Fortran is now arguably
closer to Lisp than to Fortran I.
Lisp and Fortran were the trunks of two separate evolutionary trees, one rooted in
math and one rooted in machine architecture. These two trees have been converging
ever since. Lisp started out powerful, and over the next twenty years got fast. Socalled mainstream languages started out fast, and over the next forty years gradually
5
http://www.paulgraham.com/history.html
134

got more powerful, until now the most advanced of them are fairly close to Lisp.
Close, but they are still missing a few things.
What Made Lisp Different

When it was first developed, Lisp embodied nine new ideas. Some of these we now
take for granted, others are only seen in more advanced languages, and two are still
unique to Lisp. The nine ideas are, in order of their adoption by the mainstream,
1. Conditionals. A conditional is an if-then-else construct. We take these for granted

now, but Fortran I didnt have them. It had only a conditional goto closely based
on the underlying machine instruction.
2. A function type. In Lisp, functions are a data type just like integers or strings.
They have a literal representation, can be stored in variables, can be passed as
arguments, and so on.
3. Recursion. Lisp was the first programming language to support it.
4. Dynamic typing. In Lisp, all variables are effectively pointers. Values are what
have types, not variables, and assigning or binding variables means copying
pointers, not what they point to.
5. Garbage-collection.
6. Programs composed of expressions. Lisp programs are trees of expressions, each
of which returns a value. This is in contrast to Fortran and most succeeding
languages, which distinguish between expressions and statements.
It was natural to have this distinction in Fortran I because you could not nest
statements. And so while you needed expressions for math to work, there was no
point in making anything else return a value, because there could not be anything
waiting for it.
This limitation went away with the arrival of block-structured languages, but by
then it was too late. The distinction between expressions and statements was
entrenched. It spread from Fortran into Algol and then to both their descendants.
7. A symbol type. Symbols are effectively pointers to strings stored in a hash table.
So you can test equality by comparing a pointer, instead of comparing each
character.
8. A notation for code using trees of symbols and constants.
135

9. The whole language there all the time. There is no real distinction between readtime, compile-time, and runtime. You can compile or run code while reading, read
or run code while compiling, and read or compile code at runtime.
Running code at read-time lets users reprogram Lisps syntax; running code at
compile-time is the basis of macros; compiling at runtime is the basis of Lisps use
as an extension language in programs like Emacs; and reading at runtime enables
programs to communicate using s-expressions, an idea recently reinvented as
XML.
When Lisp first appeared, these ideas were far removed from ordinary programming
practice, which was dictated largely by the hardware available in the late 1950s.
Over time, the default language, embodied in a succession of popular languages, has
gradually evolved toward Lisp. Ideas 1-5 are now widespread. Number 6 is starting
to appear in the mainstream. Python has a form of 7, though there doesnt seem to
be any syntax for it.
As for number 8, this may be the most interesting of the lot. Ideas 8 and 9 only
became part of Lisp by accident, because Steve Russell implemented something
McCarthy had never intended to be implemented. And yet these ideas turn out to
be responsible for both Lisps strange appearance and its most distinctive features.
Lisp looks strange not so much because it has a strange syntax as because it has
no syntax; you express programs directly in the parse trees that get built behind the
scenes when other languages are parsed, and these trees are made of lists, which
are Lisp data structures.
Expressing the language in its own data structures turns out to be a very powerful
feature. Ideas 8 and 9 together mean that you can write programs that write
programs. That may sound like a bizarre idea, but its an everyday thing in Lisp. The
most common way to do it is with something called a macro.
The term "macro" does not mean in Lisp what it means in other languages. A Lisp
macro can be anything from an abbreviation to a compiler for a new language. If you
want to really understand Lisp, or just expand your programming horizons, I would
6
learn more about macros.
Macros (in the Lisp sense) are still, as far as I know, unique to Lisp. This is partly
because in order to have macros you probably have to make your language look as
strange as Lisp. It may also be because if you do add that final increment of power,
6
http://www.paulgraham.com/onlisp.html
136

you can no longer claim to have invented a new language, but only a new dialect
of Lisp.
I mention this mostly as a joke, but it is quite true. If you define a language that has
car, cdr, cons, quote, cond, atom, eq, and a notation for functions expressed as lists,
then you can build all the rest of Lisp out of it. That is in fact the defining quality of
Lisp: it was in order to make this so that McCarthy gave Lisp the shape it has.
Where Languages Matter

So suppose Lisp does represent a kind of limit that mainstream languages are
approaching asymptotically-- does that mean you should actually use it to write
software? How much do you lose by using a less powerful language? Isnt it wiser,
sometimes, not to be at the very edge of innovation? And isnt popularity to some
extent its own justification? Isnt the pointy-haired boss right, for example, to want
to use a language for which he can easily hire programmers?
There are, of course, projects where the choice of programming language doesnt
matter much. As a rule, the more demanding the application, the more leverage you
get from using a powerful language. But plenty of projects are not demanding at
all. Most programming probably consists of writing little glue programs, and for little
glue programs you can use any language that youre already familiar with and that
has good libraries for whatever you need to do. If you just need to feed data from
one Windows app to another, sure, use Visual Basic.
You can write little glue programs in Lisp too (I use it as a desktop calculator), but
the biggest win for languages like Lisp is at the other end of the spectrum, where
you need to write sophisticated programs to solve hard problems in the face of fierce
7
competition. A good example is the airline fare search program that ITA Software
licenses to Orbitz. These guys entered a market already dominated by two big,
entrenched competitors, Travelocity and Expedia, and seem to have just humiliated
them technologically.
The core of ITAs application is a 200,000 line Common Lisp program that searches
many orders of magnitude more possibilities than their competitors, who apparently
are still using mainframe-era programming techniques. (Though ITA is also in a sense
using a mainframe-era programming language.) I have never seen any of ITAs code,
but according to one of their top hackers they use a lot of macros, and I am not
surprised to hear it.
7
http://www.paulgraham.com/carl.html
137
Centripetal Forces
Im not saying there is no cost to using uncommon technologies. The pointyhaired boss is not completely mistaken to worry about this. But because he doesnt
understand the risks, he tends to magnify them.
I can think of three problems that could arise from using less common languages.
Your programs might not work well with programs written in other languages. You
might have fewer libraries at your disposal. And you might have trouble hiring
programmers.
How much of a problem is each of these? The importance of the first varies depending
on whether you have control over the whole system. If youre writing software that
has to run on a remote users machine on top of a buggy, closed operating system
(I mention no names), there may be advantages to writing your application in the
same language as the OS. But if you control the whole system and have the source
code of all the parts, as ITA presumably does, you can use whatever languages you
want. If any incompatibility arises, you can fix it yourself.
In server-based applications you can get away with using the most
advanced technologies, and I think this is the main cause of what Jonathan
Erickson calls the "http://www.byte.com/documents/s=1821/byt20011214s0003/
[programming language renaissance]." This is why we even hear about new
languages like Perl and Python. Were not hearing about these languages because
people are using them to write Windows apps, but because people are using them
8
on servers. And as software shifts off the desktop and onto servers (a future even
Microsoft seems resigned to), there will be less and less pressure to use middle-ofthe-road technologies.
As for libraries, their importance also depends on the application. For less demanding
problems, the availability of libraries can outweigh the intrinsic power of the
language. Where is the breakeven point? Hard to say exactly, but wherever it is,
it is short of anything youd be likely to call an application. If a company considers
itself to be in the software business, and theyre writing an application that will be
one of their products, then it will probably involve several hackers and take at least
six months to write. In a project of that size, powerful languages probably start to
outweigh the convenience of pre-existing libraries.
8
http://www.paulgraham.com/road.html
138

The third worry of the pointy-haired boss, the difficulty of hiring programmers, I think
is a red herring. How many hackers do you need to hire, after all? Surely by now we
all know that software is best developed by teams of less than ten people. And you
shouldnt have trouble hiring hackers on that scale for any language anyone has ever
heard of. If you cant find ten Lisp hackers, then your company is probably based in
the wrong city for developing software.
In fact, choosing a more powerful language probably decreases the size of the team
you need, because (a) if you use a more powerful language you probably wont need
as many hackers, and (b) hackers who work in more advanced languages are likely
to be smarter.
Im not saying that you wont get a lot of pressure to use what are perceived as
"standard" technologies. At Viaweb (now Yahoo Store), we raised some eyebrows
among VCs and potential acquirers by using Lisp. But we also raised eyebrows by
using generic Intel boxes as servers instead of "industrial strength" servers like
Suns, for using a then-obscure open-source Unix variant called FreeBSD instead of a
real commercial OS like Windows NT, for ignoring a supposed e-commerce standard
9
called SET that no one now even remembers, and so on.
You cant let the suits make technical decisions for you. Did it alarm some potential
acquirers that we used Lisp? Some, slightly, but if we hadnt used Lisp, we wouldnt
have been able to write the software that made them want to buy us. What seemed
like an anomaly to them was in fact cause and effect.
If you start a startup, dont design your product to please VCs or potential acquirers.
Design your product to please the users. If you win the users, everything else will
follow. And if you dont, no one will care how comfortingly orthodox your technology
choices were.
The Cost of Being Average

How much do you lose by using a less powerful language? There is actually some
data out there about that.
10
The most convenient measure of power is probably code size . The point of highlevel languages is to give you bigger abstractions-- bigger bricks, as it were, so you
9
http://news.com.com/2100-1017-225723.html
10
http://www.paulgraham.com/power.html
139

dont need as many to build a wall of a given size. So the more powerful the language,
the shorter the program (not simply in characters, of course, but in distinct elements).
How does a more powerful language enable you to write shorter programs? One
technique you can use, if the language will let you, is something called bottom-up
11
programming . Instead of simply writing your application in the base language,
you build on top of the base language a language for writing programs like yours,
then write your program in it. The combined code can be much shorter than if you
had written your whole program in the base language-- indeed, this is how most
compression algorithms work. A bottom-up program should be easier to modify as
well, because in many cases the language layer wont have to change at all.
Code size is important, because the time it takes to write a program depends mostly
on its length. If your program would be three times as long in another language,
it will take three times as long to write-- and you cant get around this by hiring
more people, because beyond a certain size new hires are actually a net lose. Fred
Brooks described this phenomenon in his famous book The Mythical Man-Month, and
everything Ive seen has tended to confirm what he said.
So how much shorter are your programs if you write them in Lisp? Most of the
numbers Ive heard for Lisp versus C, for example, have been around 7-10x. But a
12
recent article about ITA in New Architect magazine said that "one line of Lisp can
replace 20 lines of C," and since this article was full of quotes from ITAs president,
I assume they got this number from ITA. If so then we can put some faith in it; ITAs
software includes a lot of C and C++ as well as Lisp, so they are speaking from
experience.
My guess is that these multiples arent even constant. I think they increase when you
face harder problems and also when you have smarter programmers. A really good
hacker can squeeze more out of better tools.
As one data point on the curve, at any rate, if you were to compete with ITA and chose
to write your software in C, they would be able to develop software twenty times
faster than you. If you spent a year on a new feature, theyd be able to duplicate it in
less than three weeks. Whereas if they spent just three months developing something
new, it would be five years before you had it too.
And you know what? Thats the best-case scenario. When you talk about code-size
ratios, youre implicitly assuming that you can actually write the program in the
11
http://www.paulgraham.com/progbot.html
12
http://www.newarchitectmag.com/documents/s=2286/new1015626014044/
140

weaker language. But in fact there are limits on what programmers can do. If youre
trying to solve a hard problem with a language thats too low-level, you reach a point
where there is just too much to keep in your head at once.
So when I say it would take ITAs imaginary competitor five years to duplicate
something ITA could write in Lisp in three months, I mean five years if nothing goes
wrong. In fact, the way things work in most companies, any development project that
would take five years is likely never to get finished at all.
I admit this is an extreme case. ITAs hackers seem to be unusually smart, and C is
a pretty low-level language. But in a competitive market, even a differential of two
or three to one would be enough to guarantee that youd always be behind.
A Recipe
This is the kind of possibility that the pointy-haired boss doesnt even want to think
about. And so most of them dont. Because, you know, when it comes down to it,
the pointy-haired boss doesnt mind if his company gets their ass kicked, so long as
no one can prove its his fault. The safest plan for him personally is to stick close to
the center of the herd.
Within large organizations, the phrase used to describe this approach is "industry
best practice." Its purpose is to shield the pointy-haired boss from responsibility: if
he chooses something that is "industry best practice," and the company loses, he
cant be blamed. He didnt choose, the industry did.
I believe this term was originally used to describe accounting methods and so on.
What it means, roughly, is dont do anything weird. And in accounting thats probably
a good idea. The terms "cutting-edge" and "accounting" do not sound good together.
But when you import this criterion into decisions about technology, you start to get
the wrong answers.
Technology often should be cutting-edge. In programming languages, as Erann Gat
has pointed out, what "industry best practice" actually gets you is not the best, but
merely the average. When a decision causes you to develop software at a fraction
of the rate of more aggressive competitors, "best practice" is a misnomer.
So here we have two pieces of information that I think are very valuable. In fact, I
know it from my own experience. Number 1, languages vary in power. Number 2,
most managers deliberately ignore this. Between them, these two facts are literally
141

a recipe for making money. ITA is an example of this recipe in action. If you want to
win in a software business, just take on the hardest problem you can find, use the
most powerful language you can get, and wait for your competitors' pointy-haired
bosses to revert to the mean.
Appendix: Power
As an illustration of what I mean about the relative power of programming languages,
consider the following problem. We want to write a function that generates
accumulators-- a function that takes a number n, and returns a function that takes
another number i and returns n incremented by i.
(Thats incremented by, not plus. An accumulator has to accumulate.)
In Common Lisp this would be
(defun foo (n)
(lambda (i) (incf n i)))
and in Perl 5,
sub foo {
my ($n) = @_;
sub {$n += shift}
}
which has more elements than the Lisp version because you have to extract
parameters manually in Perl.
In Smalltalk the code is slightly longer than in Lisp
foo: n
|s|
s := n.
^[:i| s := s+i. ]
because although in general lexical variables work, you cant do an assignment to a

parameter, so you have to create a new variable s.
142

In Javascript the example is, again, slightly longer, because Javascript retains the
distinction between statements and expressions, so you need explicit return
statements to return values:
function foo(n) {
return function (i) {
return n += i } }
(To be fair, Perl also retains this distinction, but deals with it in typical Perl fashion
by letting you omit `return`s.)
If you try to translate the Lisp/Perl/Smalltalk/Javascript code into Python you run into
some limitations. Because Python doesnt fully support lexical variables, you have
to create a data structure to hold the value of n. And although Python does have a
function data type, there is no literal representation for one (unless the body is only
a single expression) so you need to create a named function to return. This is what
you end up with:
def foo(n):
s = [n]
def bar(i):
s[0] += i
return s[0]
return bar
Python users might legitimately ask why they cant just write
def foo(n):
return lambda i: return n += i
or even
def foo(n):
lambda i: n += i
and my guess is that they probably will, one day. (But if they dont want to wait for
Python to evolve the rest of the way into Lisp, they could always just)
In OO languages, you can, to a limited extent, simulate a closure (a function that
refers to variables defined in enclosing scopes) by defining a class with one method
143

and a field to replace each variable from an enclosing scope. This makes the
programmer do the kind of code analysis that would be done by the compiler in
a language with full support for lexical scope, and it wont work if more than one
function refers to the same variable, but it is enough in simple cases like this.
Python experts seem to agree that this is the preferred way to solve the problem in
Python, writing either
def foo(n):
class acc:
def __init__(self, s):
self.s = s
def inc(self, i):
self.s += i
return self.s
return acc(n).inc
or
class foo:
def __init__(self, n):
self.n = n
def __call__(self, i):
self.n += i
return self.n
I include these because I wouldnt want Python advocates to say I was

misrepresenting the language, but both seem to me more complex than the first
version. Youre doing the same thing, setting up a separate place to hold the
accumulator; its just a field in an object instead of the head of a list. And the use of
these special, reserved field names, especially call , seems a bit of a hack.
In the rivalry between Perl and Python, the claim of the Python hackers seems to be
that that Python is a more elegant alternative to Perl, but what this case shows is that
power is the ultimate elegance: the Perl program is simpler (has fewer elements),
even if the syntax is a bit uglier.
How about other languages? In the other languages mentioned in this talk-- Fortran,
C, C++, Java, and Visual Basic-- it is not clear whether you can actually solve this
problem. Ken Anderson says that the following code is about as close as you can
get in Java:
144
public interface Inttoint {

public int call(int i);
}
public static Inttoint foo(final int n) {
return new Inttoint() {
int s = n;
public int call(int i) {
s = s + i; return s;
}
};
}
This falls short of the spec because it only works for integers. After many email
exchanges with Java hackers, I would say that writing a properly polymorphic version
that behaves like the preceding examples is somewhere between damned awkward
and impossible. If anyone wants to write one Id be very curious to see it, but I
personally have timed out.
Its not literally true that you cant solve this problem in other languages, of course.
The fact that all these languages are Turing-equivalent means that, strictly speaking,
you can write any program in any of them. So how would you do it? In the limit case,
by writing a Lisp interpreter in the less powerful language.
That sounds like a joke, but it happens so often to varying degrees in large
programming projects that there is a name for the phenomenon, Greenspuns Tenth
Rule:
Any sufficiently complicated C or Fortran program contains an ad hoc
informally-specified bug-ridden slow implementation of half of Common
Lisp.
If you try to solve a hard problem, the question is not whether you will use a powerful
enough language, but whether you will (a) use a powerful language, (b) write a
de facto interpreter for one, or (c) yourself become a human compiler for one. We
see this already begining to happen in the Python example, where we are in effect
simulating the code that a compiler would generate to implement a lexical variable.
This practice is not only common, but institutionalized. For example, in the OO world
you hear a good deal about "patterns". I wonder if these patterns are not sometimes
evidence of case (c), the human compiler, at work. When I see patterns in my
programs, I consider it a sign of trouble. The shape of a program should reflect only
145

the problem it needs to solve. Any other regularity in the code is a sign, to me at least,
that Im using abstractions that arent powerful enough-- often that Im generating
by hand the expansions of some macro that I need to write.
Notes
The IBM 704 CPU was about the size of a refrigerator, but a lot heavier. The CPU
weighed 3150 pounds, and the 4K of RAM was in a separate box weighing another
4000 pounds. The Sub-Zero 690, one of the largest household refrigerators,
weighs 656 pounds.
Steve Russell also wrote the first (digital) computer game, Spacewar, in 1962.
If you want to trick a pointy-haired boss into letting you write software in Lisp,
you could try telling him its XML.
Here is the accumulator generator in other Lisp dialects:
Scheme: (define (foo n)
(lambda (i) (set! n (+ n i)) n))
Goo: (df foo (n) (op incf n _)))
Arc: (def foo (n) [++ n _])
Erann Gats sad tale about "industry best practice" at JPL inspired me to address
this generally misapplied phrase.
Peter Norvig found that 16 of the 23 patterns in Design Patterns were "http://
www.norvig.com/design-patterns/[invisible or simpler]" in Lisp.
Thanks to the many people who answered my questions about various languages
and/or read drafts of this, including Ken Anderson, Trevor Blackwell, Erann Gat,
Dan Giffin, Sarah Harlin, Jeremy Hylton, Robert Morris, Peter Norvig, Guy Steele,
and Anton van Straaten. They bear no blame for any opinions expressed.
Related:
Many people have responded to this talk, so I have set up an additional page to deal
13
with the issues they have raised: Re: Revenge of the Nerds .
14
It also set off an extensive and often useful discussion on the LL1 mailing list. See
particularly the mail by Anton van Straaten on semantic compression.
13
http://www.paulgraham.com/icadmore.html
14
http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/threads.html
146

Some of the mail on LL1 led me to try to go deeper into the subject of language
15
power in Succinctness is Power .
A larger set of canonical implementations of the accumulator generator
16
benchmark are collected together on their own page.
15
http://www.paulgraham.com/power.html
16
http://www.paulgraham.com/accgen.html
147
Screaming Architecture
Imagine that you are looking at the blueprints of a building. This document, prepared
by an architect, tells you the plans for the building. What do these plans tell you?
If the plans you are looking at are for a single family residence, then youll likely
see a front entrance, a foyer leading to a living room and perhaps a dining room.
Therell likely be a kitchen a short distance away, close to the dining room. Perhaps
a dinette area next to the kitchen, and probably a family room close to that. As you
looked at those plans, thered be no question that you were looking at a house. The
architecture would scream: house.
Or if you were looking at the architecture of a library, youd likely see a grand
entrance, an area for check-in-out clerks, reading areas, small conference rooms, and
gallery after gallery capable of holding bookshelves for all the books in the library.
That architecture would scream: library.
So what does the architecture of your application scream? When you look at the
top level directory structure, and the source files in the highest level package;
do they scream: Health Care System, or Accounting System, or Inventory
Management System? Or do they scream: Rails, or Spring/Hibernate, or ASP?
The Theme of an Architecture

Go back and read Ivar Jacobsons seminal work on software architecture: Object
Oriented Software Engineering. Notice the subtitle of the book: A use case driven
approach. In this book Ivar makes the point that software architectures are structures
that support the use cases of the system. Just as the plans for a house or a library
scream about the use cases of those buildings, so should the architecture of a
software application scream about the use cases of the application.
Architectures are not (or should not) be about frameworks. Architectures should not
be supplied by frameworks. Frameworks are tools to be used, not architectures to be
conformed to. If your architecture is based on frameworks, then it cannot be based
on your use cases.
1
http://blog.8thlight.com/uncle-bob/2011/09/30/Screaming-Architecture.html
148
The Purpose of an Architecture

The reason that good architectures are centered around use-cases is so that
architects can safely describe the structures that support those use-cases without
committing to frameworks, tools, and environment. Again, consider the plans for a
house. The first concern of the architect is to make sure that the house is usable, it
is not to ensure that the house is made of bricks. Indeed, the architect takes pains
to ensure that the homeowner can decide about bricks, stone, or cedar later, after
the plans ensure that the use cases are met.
A good software architecture allows decisions about frameworks, databases, webservers, and other environmental issues and tools, to be deferred and delayed. A
good architecture makes it unnecessary to decide on Rails, or Spring, or Hibernate,
or Tomcat or MySql, until much later in the project. A good architecture makes it easy
to change your mind about those decisions too. A good architecture emphasizes the
use-cases and decouples them from peripheral concerns.
But what about the Web?

Is the web an architecture? Does the fact that your system is delivered on the
web dictate the architecture of your system? Of course not! The Web is a delivery
mechanism, and your application architecture should treat it as such. The fact that
your application is delivered over the web is a detail and should not dominate your
system structure. Indeed, the fact that your application is delivered over the web
is something you should defer. Your system architecture should be as ignorant as
possible about how it is to be delivered. You should be able to deliver it as a console
app, or a web app, or a thick client app, or even a web service app, without undue
complication or change to the fundamental architecture.
Frameworks are tools, not ways of life.

Frameworks can be very powerful and very useful. Framework authors often believe
in their frameworks. The examples they write for how to use their frameworks are
told from the point of view of a true believer. Other authors who write about the
framework also tend to be disciples of the true belief. They show you the way to use
the framework. Often it is an all-encompassing, all-pervading, let-the-framework-doeverything position. This is not the position you want to take.
Look at each framework with a jaded eye. View it skeptically. Yes, it might help,
but at what cost. How should I use it, and how should I protect myself from it. How
149
can I preserve the use-case emphasis of my architecture? How can I prevent the
framework from taking over that architecture.
Testable Architectures.
If you system architecture is all about the use cases, and if you have kept your
frameworks at arms-length. Then you should be able to unit-test all those use cases
without any of the frameworks in place. You shouldnt need the web server running
in order to run your tests. You shouldnt need the database connected in order
to run your tests. Your business objects should be plain old objects that have no
dependencies on frameworks or databases or other complications. Your use case
objects should coordinate your business objects. And all of them together should be
testable in-situ, without any of the complications of frameworks.
Conclusion
Your architectures should tell readers about the system, not about the frameworks
you used in your system. If you are building a health-care system, then when new
programmers look at the source repository, their first impression should be: Oh,
this is a heath-care system. Those new programmers should be able to learn all the
use cases of the system, and still not know how the system is delivered. They may
come to you and say: We see some things that look sorta like models, but where are
the views and controllers, and you should say: Oh, those are details that neednt
concern you at the moment, well show them to you later.
For more on this topic, see Episode VII Architecture, Use-cases, and High Level
Design, at cleancoders.com.
150
How to securely hash passwords?

Written by: Thomas Pornin at 2nd March 2013
The Theory
We need to hash passwords as a second line of defence. A server which can
authenticate users necessarily contains, somewhere in its entrails, some data which
can be used to validate a password. A very simple system would just store the
passwords themselves, and validation would be a simple comparison. But if an
hostile outsider gains a simple glimpse at the contents of the file or database
table which contains the passwords, then that attacker learns a lot. Unfortunately,
such partial, read-only breaches do occur in practice (a mislaid backup tape, a
decommissioned but not wiped out hard disk, as an aftermath of a SQL injection
2
attack the possibilities are numerous). See this blog post for a detailed discussion.
Since the overall contents of a server which can validate passwords are necessarily
sufficient to, indeed, validate passwords, an attacker who got a read-only snapshot
3
of the server is in position to make an offline dictionary attack : he tries potential
passwords until a match is found. We cannot avoid that. So we will want to make
that kind of attack as hard as possible. Our tools are the following:
4
Cryptographic hash functions : these are fascinating mathematical objects

which everybody can compute efficiently, and yet nobody knows how to invert
them. This looks good for our problem: the server could store a hash of a
password; when presented with a putative password, the server just has to hash
it to see if it gets the same value; and yet, knowing the hash does not reveal the
password itself.
Salts: among the advantages of the attacker over the defender, is parallelism.
The attacker usually grabs a whole list of hashed passwords, and is interested in
breaking as many of them as possible. He may try to attack several in parallels.
For instance, the attacker may consider one potential password, hash it, and then
compare the value with 100 hashed passwords; this means that the attacker
shares the cost of hashing over several attacked passwords. A similar optimisation
1
2
http://security.stackexchange.com/a/31846
http://security.blogoverflow.com/2011/11/why-passwords-should-be-hashed/
3
http://en.wikipedia.org/wiki/Dictionary_attack
4
http://en.wikipedia.org/wiki/Cryptographic_hash_function
151

5
is precomputed tables, including rainbow tables ; this is still parallelism, with a

space-time change of coordinates.
The common characteristic of all attacks which use parallelism is that they work
over several passwords which were processed with the exact same hash function.
Salting is about using not one hash function, but a lot of distinct hash functions;
ideally, each instance of password hashing should use its own hash function. A salt
is a way to select a specific hash function among a big family of hash functions.
Properly applied salts will completely thwart parallel attacks (including rainbow
tables).
Slowness: computers become faster over time (Gordon Moore, co-founder of
6
Intel, theorized it in his famous law ). Human brains do not. This means that
attackers can "try" more and more potential passwords as years pass, while users
cannot remember more and more complex passwords (or flatly refuse to). To
counter that trend, we can make hashing inherently slow by defining the hash
function to use a lot of internal iterations (thousands, possibly millions).
7
We have a few standard cryptographic hash functions; most famous are MD5 and
8
the SHA family . Building a secure hash function out of elementary operations is
far from easy. When cryptographers want to do that, they think hard, then harder,
and organize a tournament where the functions fight each other fiercely. When
hundreds of cryptographers gnawed and scraped and punched at a function for
several years and found nothing bad to say about it, then they begin to admit that
maybe that specific function could be considered as more or less secure. This is just
9
what happened in the SHA-3 competition . We have to use this way of designing
hash function because we know no better way. Mathematically, we do not know if
secure hash functions actually exist; we just have "candidates" (thats the difference
between "it cannot be broken" and "nobody in the world knows how to break it").
A basic hash function, even if secure as a hash function, is not appropriate for
password hashing, because:
it is unsalted, allowing for parallel attacks (rainbow tables for MD5 or SHA-1
can be obtained for free, you do not even need to recompute them yourself);
5
http://en.wikipedia.org/wiki/Rainbow_table
6
http://en.wikipedia.org/wiki/Moore%27s_law
7
http://en.wikipedia.org/wiki/MD5
8
http://en.wikipedia.org/wiki/Secure_Hash_Algorithm
9
http://en.wikipedia.org/wiki/NIST_hash_function_competition
10
http://www.freerainbowtables.com/tables/
152
10

it is way too fast, and gets faster with technological advances. With a recent GPU
(i.e. off-the-shelf consumer product which everybody can buy), hashing rate is
11
counted in billions of passwords per second .
So we need something better. It so happens that slapping together a hash function
and a salt, and iterating it, is not easier to do than designing a hash function at
least, if you want the result to be secure. There again, you have to rely on
standard constructions which have survived the continuous onslaught of vindicative
cryptographers.
Good Password Hashing Functions

PBKDF2
12
13
PBKDF2
comes from PKCS#5 . It is parameterized with an iteration count (an
integer, at least 1, no upper limit), a salt (an arbitrary sequence of bytes, no
constraint on length), a required output length (PBKDF2 can generate an output of
configurable length), and an "underlying PRF". In practice, PBKDF2 is always used
14
with HMAC , which is itself a construction which is built over an underlying hash
function. So when we say "PBKDF2 with SHA-1", we actually mean "PBKDF2 with
HMAC with SHA-1".
Advantages of PBKDF2:
Has been specified for a long time, seems unscathed for now.
Is already implemented in various framework (e. g. it is provided with .NET
15
).
Highly configurable (although some implementations do not let you choose the
hash function, e.g. the one in .NET is for SHA-1 only).
Received NIST blessings
derivation; see later on).
16
(modulo the difference between hashing and key
Configurable output length (again, see later on).

Drawbacks of PBKDF2:
11
http://www.golubev.com/hashgpu.htm
12
http://en.wikipedia.org/wiki/PBKDF2
13
http://tools.ietf.org/html/rfc2898
14
http://en.wikipedia.org/wiki/Hash-based_message_authentication_code
15
http://msdn.microsoft.com/en-us/library/system.security.cryptography.rfc2898derivebytes.aspx
16
http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf
153

CPU-intensive only, thus amenable to high optimization with GPU (the defender
is a basic server which does generic things, i.e. a PC, but the attacker can spend
his budget on more specialized hardware, which will give him an edge).
You still have to manage the parameters yourself (salt generation and
storage, iteration count encoding). There is a standard encoding for PBKDF2
17
parameters
but it uses ASN.1 so most people will avoid it if they can (ASN.1
can be tricky to handle for the non-expert).
bcrypt
18
bcrypt was designed by reusing and expanding elements of a block cipher called
19
Blowfish . The iteration count is a power of two, which is a tad less configurable
than PBKDF2, but sufficiently so nevertheless. This is the core password hashing
20
mechanism in the OpenBSD operating system.
Advantages of bcrypt:
Many available implementations in various languages (see the links at the end
of the Wikipedia page).
More resilient to GPU; this is due to details of its internal design. The bcrypt
authors made it so voluntarily: they reused Blowfish because Blowfish was based
on an internal RAM table which is constantly accessed and modified throughout
the processing. This makes life much harder for whoever wants to speed up bcrypt
with a GPU (GPU are not good at making a lot of memory accesses in parallel).
21
See here for some discussion.
Standard output encoding which includes the salt, the iteration count and the
output as one simple to store character string of printable characters.
Drawbacks of bcrypt:
Output size is fixed: 192 bits.
While bcrypt is good at thwarting GPU, it can still be thoroughly optimized with
22
FPGA : modern FPGA chips have a lot of small embedded RAM blocks which are
17
http://tools.ietf.org/html/rfc2898#appendix-A.2
18
http://en.wikipedia.org/wiki/Bcrypt
19
http://en.wikipedia.org/wiki/Blowfish_%28cipher%29
20
http://www.openbsd.org/
21
http://security.stackexchange.com/questions/4781/do-any-security-experts-recommend-bcrypt-forpassword-storage/6415#6415
22
http://en.wikipedia.org/wiki/Field-programmable_gate_array
154

very convenient for running many bcrypt implementations in parallel within one
23
chip. It has been done .
Input password size is limited to 51 characters. In order to handle longer
24
passwords, one has to combine bcrypt with a hash function

(you hash the
password and then use the hash value as the "password" for bcrypt). Combining
cryptographic primitives is known to be dangerous (see above) so such games
cannot be recommended on a general basis.
scrypt
25
scrypt is a much newer construction (designed in 2009) which builds over PBKDF2
26
and a stream cipher called Salsa20/8 , but these are just tools around the core
strength of scrypt, which is RAM. scrypt has been designed to inherently use a
lot of RAM (it generates some pseudo-random bytes, then repeatedly read them
in a pseudo-random sequence). "Lots of RAM" is something which is hard to make

parallel. A basic PC is good at RAM access, and will not try to read dozens of unrelated
RAM bytes simultaneously. An attacker with a GPU or a FPGA will want to do that,
and will find it difficult.
Advantages of scrypt:
A PC, i.e. exactly what the defender will use when hashing passwords, is the most
efficient platform (or close enough) for computing scrypt. The attacker no longer
gets a boost by spending his dollars on GPU or FPGA.
One more way to tune the function: memory size.
Drawbacks of scrypt:
Still new (my own rule of thumb is to wait at least 5 years of general exposure, so
no scrypt for production until 2014 but, of course, it is best if other people try
scrypt in production, because this gives extra exposure).
Not as many available, ready-to-use implementations for various languages.
Unclear whether the CPU / RAM mix is optimal. For each of the pseudo-random
RAM accesses, scrypt still computes a hash function. A cache miss will be about
23
http://openwall.info/wiki/john/FPGA
24
http://security.stackexchange.com/questions/6623/pre-hash-password-before-applying-bcrypt-to-
avoid-restricting-password-length
25
http://www.tarsnap.com/scrypt.html
26
http://en.wikipedia.org/wiki/Salsa20
155

200 clock cycles, one SHA-256 invocation is close to 1000. There may be room
for improvement here.
Yet another parameter to configure: memory size.
OpenPGP Iterated And Salted S2K

I cite this one because you will use it if you do password-based file encryption with
27
28
GnuPG . That tool follows the OpenPGP format which defines its own password
hashing functions, called "Simple S2K", "Salted S2K" and "http://tools.ietf.org/html/
rfc4880#section-3.7.1.3[Iterated and Salted S2K]". Only the third one can be
deemed "good" in the context of this answer. It is defined as the hash of a very long
string (configurable, up to about 65 megabytes) consisting of the repetition of an 8byte salt and the password.
As far as these things go, OpenPGPs Iterated And Salted S2K is decent; it can
be considered as similar to PBKDF2, with less configurability. You will very rarely
encounter it outside of OpenPGP, as a stand-alone function.
Unix "crypt"
Recent Unix-like systems (e.g. Linux), for validating user passwords, use iterated
29
and salted variants of the crypt()
function based on good hash functions, with
thousands of iterations. This is reasonably good. Some systems can also use bcrypt,
which is better.
The old crypt() function, based on the DES block cipher
30
, is not good enough:
It is slow in software but fast in hardware, and can be made fast in software too but
31
only when computing several instances in parallel (technique known as SWAR
or "bitslicing"). Thus, the attacker is at an advantage.
It is still quite fast, with only 25 iterations.
It has a 12-bit salt, which means that salt reuse will occur quite often.
It truncates passwords to 8 characters (characters beyond the eighth are ignored)
and it also drops the upper bit of each character (so you are more or less stuck
with ASCII).
27
http://www.gnupg.org/
28
http://tools.ietf.org/html/rfc4880
29
http://en.wikipedia.org/wiki/Crypt_%28C%29
30
http://en.wikipedia.org/wiki/Data_Encryption_Standard
31
http://en.wikipedia.org/wiki/SWAR
156

But the more recent variants, which are active by default, will be fine.
Bad Password Hashing Functions

About everything else, in particular virtually every homemade method that people
relentlessly invent.
For some reason, many developers insist on designing function themselves, and
seem to assume that "secure cryptographic design" means "throw together every
kind of cryptographic or non-cryptographic operation that can be thought of". See
32
this question for an example. The underlying principle seems to be that the sheer
complexity of the resulting utterly tangled mess of instruction will befuddle attackers.
In practice, though, the developer himself will be more confused by his own creation
than the attacker.
Complexity is bad. Homemade is bad. New is bad. If you remember that,
youll avoid 99% of problems related to password hashing, or cryptography, or even
security in general.
Password hashing in Windows operating systems used to be mindbogglingly awful
and now is just terrible (unsalted, non-iterated MD4).
33
Key Derivation
Up to now, we considered the question of hashing passwords. A close problem
is about transforming a password into a symmetric key which can be used for
34
encryption; this is called key derivation
and is the first thing you do when you
"encrypt a file with a password".
It is possible to make contrived examples of password hashing functions which are
secure for the purpose of storing a password validation token, but terrible when it
comes to generating symmetric keys; and the converse is equally possible. But these
examples are very "artificial". For practical functions like the one described above:
The output of a password hashing function is acceptable as a symmetric key, after
possible truncation to the required size.
32
http://security.stackexchange.com/questions/25585/is-my-developers-home-brew-password-
security-right-or-wrong-and-why
33
http://security.stackexchange.com/questions/2881/is-there-any-advantage-to-splitting-apassword/2883#2883
34
http://en.wikipedia.org/wiki/Key_derivation_function
157

A Key Derivation Function can serve as a password hashing function as long as
the "derived key" is long enough to avoid "generic preimages" (the attacker is
just lucky and finds a password which yields the same output). An output of more
than 100 bits or so will be enough.
Indeed, PBKDF2 and scrypt are KDF, not password hashing function and NIST
"approves" of PBKDF2 as a KDF, not explicitly as a password hasher (but it is possible,
with only a very minute amount of hypocrisy, to read NISTs prose in such a way that
it seems to say that PBKDF2 is good for hashing passwords).
35
Conversely, bcrypt is really a block cipher

(the bulk of the password processing
36
is the "key schedule") which is then used in CTR mode
to produce three blocks
(i.e. 192 bits) of pseudo-random output, making it a kind of hash function. bcrypt
can be turned into a KDF with a little surgery, by using the block cipher in CTR
mode for more blocks. But, as usual, we cannot recommend such homemade
transforms. Fortunately, 192 bits are already more than enough for most purposes
37
38
(e.g. symmetric encryption with GCM or EAX only needs a 128-bit key).
Miscellaneous Topics
How many iterations ?
As much as possible ! This salted-and-slow hashing is an arms race between the
attacker and the defender. You use many iterations to make the hashing of a
password harder for everybody. To improve security, you should set that number
as high as you can tolerate on your server, given the tasks that your server must
otherwise fulfill. Higher is better.
Collisions and MD5

MD5 is broken: it is computationally easy to find a lot of pairs of distinct inputs which
hash to the same value. These are called collisions.
However, collisions are not an issue for password hashing. Password hashing
requires the hash function to be resistant to preimages, not to collisions. Collisions
are about finding pairs of messages which give the same output without restriction,
35
36
http://en.wikipedia.org/wiki/Block_cipher
http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Counter_.28CTR.29
37
http://en.wikipedia.org/wiki/Galois/Counter_Mode
38
http://en.wikipedia.org/wiki/EAX_mode
158

whereas in password hashing the attacker must find a message which yields a given
output that the attacker does not get to choose. This is quite different. As far as we
known, MD5 is still (almost) as strong as it has ever been with regards to preimages
39
(there is a theoretical attack
which is still very far in the ludicrously impossible
to run in practice).
The real problem with MD5 as it is commonly used in password hashing is that it
is very fast, and unsalted. However, PBKDF2 used with MD5 would be robust. You
should still use SHA-1 or SHA-256 with PBKDF2, but for Public Relations. People get
nervous when they hear "MD5".
Salt Generation
The main and only point of the salt is to be as unique as possible. Whenever a salt
value is reused anywhere, this has the potential to help the attacker.
For instance, if you use the user name as salt, then an attacker (or several colluding
attackers) could find it worthwhile to build rainbow tables which attack the password
hashing function when the salt is "admin" (or "root" or "joe") because there will be
several, possibly many sites around the world which will have a user named "admin".
Similarly, when a user changes his password, he usually keeps his name, leading
to salt reuse. Old passwords are valuable targets, because users have the habit of
reusing passwords in several places (thats known to be a bad idea, and advertised
as such, but they will do it nonetheless because it makes their life easier), and also
because people tend to generate their passwords "in sequence": if you learn that
Bobs old password is "SuperSecretPassword37", then Bobs current password is
probable "SuperSecretPassword38" or "SuperSecretPassword39".
The cheap way to obtain uniqueness is to use randomness. If you generate your
40
salt as a sequence of random bytes from the cryptographically secure PRNG that
your operating system offers ( /dev/urandom , CryptGenRandom() ) then you
will get salt values which will be "unique with a sufficiently high probability". 16 bytes
are enough so that you will never see a salt collision in your life, which is overkill
but simple enough.
41
UUID are a standard way of generating "unique" values. Note that "version 4" UUID
just use randomness (122 random bits), like explained above. A lot of programming
39
http://www.iacr.org/archive/eurocrypt2009/54790136/54790136.pdf
40
http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator
41
http://en.wikipedia.org/wiki/Universally_unique_identifier
159

frameworks offer simple to use functions to generate UUID on demand, and they can
be used as salts.
Salt Secrecy
Salts are not meant to be secret; otherwise we would call them keys. You do not need
to make salts public, but if you have to make them public (e.g. to support client-side
hashing), then dont worry too much about it. Salts are there for uniqueness. Strictly
speaking, the salt is nothing more than the selection of a specific hash function within
a big family of functions.
"Pepper"
Cryptographers can never let a metaphor alone; they must extend it with further
analogies and bad puns. "Peppering" is about using a secret salt, i.e. a key. If you
use a "pepper" in your password hashing function, then you are switching to a quite
different kind of cryptographic algorithm; namely, you are computing a Message
42
Authentication Code over the password. The MAC key is your "pepper".
Peppering makes sense if you can have a secret key which the attacker will not be
able to read. Remember that we use password hashing because we consider that an
attacker could grab a copy of the server database, or possible of the whole disk of
43
the server. A typical scenario would be a server with two disks in RAID 1 . One disk
fails (electronic board fries this happens a lot). The sysadmin replaces the disk,
the mirror is rebuilt, no data is lost due to the magic of RAID 1. Since the old disk
is dysfunctional, the sysadmin cannot easily wipe its contents. He just discards the
disk. The attacker searches through the garbage bags, retrieves the disk, replaces
the board, and lo! He has a complete image of the whole server system, including
database, configuration files, binaries, operating system the full monty, as the
British say. For peppering to be really applicable, you need to be in a special setup
44
where there is something more than a PC with disks; you need a HSM . HSM are very
expensive, both in hardware and in operational procedure. But with a HSM, you can
45
just use a secret "pepper" and process passwords with a simple HMAC (e.g. with
SHA-1 or SHA-256). This will be vastly more efficient than bcrypt/PBKDF2/scrypt and
42
43
http://en.wikipedia.org/wiki/Message_authentication_code
http://en.wikipedia.org/wiki/RAID#RAID_1
44
http://en.wikipedia.org/wiki/Hardware_security_module
45
http://en.wikipedia.org/wiki/Hash-based_message_authentication_code
160

their cumbersome iterations. Also, usage of a HSM will look extremely professional
46
when doing a WebTrust audit .
Client-side hashing
Since hashing is (deliberately) expensive, it could make sense, in a client-server
situation, to harness the CPU of the connecting clients. After all, when 100 clients
connect to a single server, the clients collectively have a lot more muscle than the
server.
To perform client-side hashing, the communication protocol must be enhanced to
support sending the salt back to the client. This implies an extra round-trip, when
compared to the simple client-sends-password-to-server protocol. This may or may
not be easy to add to your specific case.
Client-side hashing is difficult in a Web context because the client uses Javascript,
which is quite anemic for CPU-intensive tasks.
In the context of SRP
47
, password hashing necessarily occurs on the client side.
Conclusion
Use bcrypt. PBKDF2 is not bad either. If you use scrypt you will be a "slightly early
adopter" with the risks that are implied by this expression; but it would be a good
move for scientific progress ("crash dummy" is a very honourable profession).
46
http://www.webtrust.org/item64428.aspx
47
http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol
161
Smart Guy Productivity Pitfalls

Written by: Hook at 13th March 2013
Productivity is one of my pet topics, because its always dogged me a bit, especially
early in my career. Id pull long days and nights and then realize I only actually
worked (as in, typing in code, debugging stuff, and thinking about problems and
their solutions) maybe 20% of the time. Upon talking to coworkers, this seemed
to be normal, a part of the expected friction costs incurred working in an office
environment. Meetings, shooting the shit with coworkers, lunch, email, and, er, stuff.
Figure4.This is what I mean by "stuff" in 1996

Eventually working around high-productivity professionals like John Carmack made
me realize that if you want to excel, then you have to work hard and focus the whole
time.
http://bookofhook.blogspot.de/2013/03/smart-guy-productivity-pitfalls.html
162
Figure 5. John Carmack Productivity Measurement Tool ca.

1998
I remember Carmack talking about productivity measurement. While working he
would play a CD, and if he was not being productive, hed pause the CD player. This
meant any time someone came into his office to ask him a question or he checked
email hed pause the CD player. Hed then measure his output for the day by how
many times he played the CD (or something like that maybe it was how far he got
down into his CD stack). I distinctly remember him saying "So if I get up to go to the
bathroom, I pause the player".
You know whats pretty hardcore? Thinking that going to the bathroom is essentially
the same as fucking off.
(Sidenote: my memory is hazy on this since it was long ago CD PLAYERS yo).
That level of work ethic and focus is rare, particularly with my generation of
programmers who were raised with the vile "work smart, not hard" mantra, coupled
with the sense that we were somehow significantly brighter than most of our peers.
2
Combine those two notions and add the Scotty Principle and you get.me.
http://www.urbandictionary.com/define.php?term=Scotty%20Principle
163
If my coworkers were grinding through stuff that took them 20 hours, Id be all "I
work smart, not hard" with accompanying snarky eye roll, and Id assume I could
bust through an equivalent work load in four hours and still look like a goddamn
superhero.
And sometimes I could, so, hey, validation!
And you can skate by like this (God knows I did) for a long time before a couple things
eventually rear their heads and bite you in your entitled face.
Productivity Deficit: Your Attitude Writing

Checks Your Work Ethic Cant Cash
An overinflated sense of your own abilities creates a constant state of production
deficit, because you assume that you can make it up with a burst of brilliance and/
or crunch.
But there is no countering surplus to offset the deficit. The only way surpluses show
up is when you finish a (presumably) hard task much faster than you anticipated. But
instead of banking the surplus (i.e. moving on immediately to your next task), you
spend it relaxing and screwing off because, whew, you just earned a small vacation
by busting shit out in an hour that you thought would take all day. Hey, whats on
Facebook and Twitter right now? Do I have any new mail? No? I wonder if anyone has
tweeted something recently since the last time I checkedwhat about Reddit? Oh,
164

an inspirational/funny/cool YouTube video, its only eight minutes long, let me watch
it now! (Eight minutes later) Sweetwhats up Twitter? Oh man, I have to make a
snarky response, but first I have to Google for that XKCD comic that totally makes
my point for me
And before you know it, the day is done, and youre still feeling good because you
finished in one hour what should have taken all day, so all good, right?
Trap of the Easy Task

And yeah, its often all good, but when operating at a slight deficit things can go
pear shaped quickly when you accidentally spring the trap of the easy task. Tasks
so trivial that they barely register as work. Update an SDK? Hour, tops. Implement
this feature that Ive already done on every other platform? Hour, tops. This is all
mindless grunt work that anyone could do, its just clicking through dialog boxes,
maybe changing a few lines of source, rebuilding, whatever.
In other words, an easy task like this is so easy that its a constant time cost
for everyone irrespective of ability, so theres no opportunity nor need for crazy
overestimation since what could possibly go wrong?
Well, as with so many things, its the unexpected things that screw you the hardest.
The things that should be trivial end up taking days because, shit, that SDK install
required a new compiler install, and the new compiler install failed and I cant
uninstall it, and now I have to fucking WIPE MY WHOLE GODDAMN HARD DRIVE
TO RECOVER. Or you need to implement something that was trivial on nine other
platforms, and it should be 30 minutes on platform number ten, except it lacks
a base feature that is present on everything else, the compiler is dying with an
"Internal compiler failure 2033", the API call you need to use hangs the dev console
immediately and, oh, the dev support site you use to get release notes is totally down
for the foreseeable future.
So what should have taken less than an hour took a week.
Its like having a perfect monetary budget that assumes no crazy "one time" only
bills, except life is full of crazy one time only bills and the only way you can keep those
under control is by giving yourself a budgetary capacitor to dampen the fluctuations.
And now youre defensive about losing a week to something stupid because you
budgeted an hour for it and waited until the last second to do it and now the schedule
has gone to hell, but its not your fault, because it could have happened to anyone!
165

But if you had banked your surplus hours before and/or worked at closer to your
theoretical peak effectiveness then this type of thing would get absorbed in the wash.
And now you live in a state of mild panic because youre way smarter than all this
but youre never actually getting things done on time.
Identity Recalibration Crisis

Which leads to a potential identity recalibration crisis upon landing at a company
with high performers that work hard and smart. And that will happen if youre good.
Now youre no longer at the top of the curve. In fact, shit, youre in the middle or
bottom of the curve, a situation your brain probably never considered as an option.
Figure6.This dude was my wake up call with respect to my

over inflated sense of skill
I went through this when I went to id software. I had rationalized that John Carmacks
success was just a factor of timing and luck since, hell, he was my age, and I was
pretty smart, so what else could it be? Upon my arrival I had a crash course in
humility, because he was way smarter than me and way more productive as well.
This took a while to sink in, because until then I was used to believing I was one of
the better programmers at a company. And then working with Carmack I realized he
166

was not just a little better, he was orders of magnitude better. And I couldnt dismiss
it with a "But I write a lot of code" hand wave, because he also wrote like 80% of
the Quake code base.
Sometimes it takes a while for this to sink in. Sometimes it doesnt sink in at all, and
youre let go because youre not very productive compared to your new team of high
performers. Sometimes it sinks in, and you panic and get depressed because your
self-image has always been that of being the strongest swimmer in the school, and
right now youre just trying not to drown, much less keep up with everyone else.
But ideally you shrug off the old counter productive mentality and habits and emerge
as another one of the high functioning team members, but that can take a lot of
work, particularly if you have to get over years of giving it twenty percent.
Killing the Underachiever

If my pithy advice were little more than "Be more like John Carmack" then I can
imagine a lot of readers throwing up their hands and saying "Well, fuck it, Im a lost
cause, because thats not going to happen." But what I can do is relate some of the
things that helped me kill off some of my underachieving habits. The point isnt to
become a superstar, its to become better, since thats always the first step.
I dont believe in mechanical solutions ("Turn off the internet", "Listen to music",
and stuff like that) because I dont think they address the core issues, which are
psychological. Instead I found that I had to do the following.
1. Develop self-awareness. It took working with John Carmack and other high
productivity programmers and admitting that they were way more productive
than me before I really understood how much more productive I could be. In other
words, if you dont admit its something worth improving, then obviously youre
not going to search for a solution, and if thats the case, you should go here
4
or here .
2. Give a shit. Originally I called this "develop a sense of urgency", but really its
just about caring about getting your work done. It doesnt even matter what you
specifically care about! It can be your professional image, your product, your
team, your customers, whatever. You just have to care about something that will
drive you to getting things done, because if you dont, apathy will occupy that
void.
3
http://www.reddit.com/
4
https://www.facebook.com/
167

3. Minimize uncertainty. In another blog article, Productivity vs. Uncertainty and
5
Apathy , I talk about how poorly defined goals can lead to poor productivity. If
its unclear what you need to get done today, then theres a reasonable chance
you wont actually do anything today.
4. Commit to getting something done every day. When you show up in the
morning have a well defined set of things to finish that day. Stay as late as you
have to in order to finish. By committing to finishing that task other distractions
will naturally fall by the wayside. For example, I have jiu-jitsu training at 7pm.
If I screw off too much during the day, I dont get to train. Also, by committing
to a task, you avoid "being busy" instead of "getting work done", they are not
6
the same thing .
5. Never say "Ill finish it up tomorrow" or "Ill make up for it by coming
in early/staying late/working the weekend". This is an easy trap to get into,
where you keep incurring time debt until at some point you realize youre now
three weeks behind on a task that should have taken two days. This is like racking
up credit card bills assuming you can pay them off later. Which is fine, until "later"
arrives and youve only accumulated more debt.
6. Do not overpromise to make up for poor productivity. Theres a tendency
when were falling behind to try to overcompensate with future promises. "When
Im done, itll be AWESOME" or "I know Im late, but Im positive Ill be done by
Monday". By doing those things we just build more debt we cant pay off, and that
will eventually lead to a catastrophic melt down when the super final absolutely
last deadline date shows up. Just get shit done, dont talk about how youre going
to get shit done.
7. Have an objective productivity metric. This is a mechanical thing, but it acts
as a reasonable backstop. If you have changelogs you can reference, then its
easy to sit down on Friday and ask yourself "What have I done this week?" And if
you know that its possible for others to check on you, then it makes you take each
day a lot more seriously. If you judge your value solely on output, not subjective
things like "being smart", you will be more productive.
8. Accept that "the grind" is part of the job. A friend of mines father has a
great quote: "Son, i dont wake up every day and go to a place called fun. I wake
up and go to a place called work" You cant get irate or frustrated that the bulk
of the day is typing in boring code and dealing with bugs and other people.
5
http://bookofhook.blogspot.com/2012/09/productivity-vs-uncertainty-apathy.html
6
http://bookofhook.blogspot.com/2013/01/being-busy-isnt-same-as-getting-work.html
168

I still screw off during the day. I am not a code grinding automaton. I read Facebook,
chat with others, Tweet, shop on Amazon, get coffee, read forums, write blog posts
like this one and Im totally fine with that.
Because I know Im committed to getting things done as well. So even though Im

writing this right now at 3pm, if I have to work until 8pm to get my shit done and
checked in, Ill do that, so that tomorrow is a fresh start, not a day of "Oh goddamit,
lets try again to finish that task I started a week ago."
169

Once I developed that sense of daily urgency my productivity went up considerably.
And if Im really productive in the morning and afternoon and feel like I can go home
at 4pm with a clean conscience, I do that too, so I dont burn out.
Its a push/pull between banking surplus and generating a deficit, just make sure you
dont do just one or the other.
170
TDD, Straw Men, and Rhetoric

Written by: Gary Bernhardt at 2014-04-30
1
2
In a blog post called Slow database test fallacy , David Heinemeier Hansson, the
creator of Rails, begins:
The classical definition of a unit test in TDD lore is one that doesnt
touch the database.
First, you can immediately tell that this piece of writing is going to be heavily
rhetorical. He refers to "TDD lore" as opposed to, say, "TDD practice". By using the
word "lore", he positions it as subjective, unreliable, mythological.
Second, that sentence is false. Isolation from the database, or anything else, is
generally done with mocks, but mocks didnt even exist when TDD was rediscovered
3
by Kent Beck in 1994-1995 . They were introduced at XP2000 in a paper called Endo4
Testing: Unit Testing with Mock Objects , and it took a long time after that for them
to gain popularity. Their role in software development was still being fleshed out in
5
2004 when Mock Roles, Not Objects was published.
Classical TDD does not involve mocking or other forms of synthetic isolation by
definition. We even use the term "classical TDD" to mean "TDD without isolation".
David used the word "classical" not because its correct, but because it implies "old".
This is the beginning of a series of rhetorical techniques that he uses to incorrectly
associate isolated unit testing with "oldness". He continues:
Connecting to external services like that would be too slow to get the
feedback cycle you need. That was probably true in 1997 when you
were connecting to a mainframe database or some Oracle abomination.
In 1997, TDD was only known to a small number of people. Mocks did not exist.
Certainly no one was isolating unit tests. Why would David invoke that year? Its a
disingenuous rhetorical technique: by implying that a modern movement actually
occurred far in the past, he positions it as outdated, thereby discrediting it.
1
https://www.destroyallsoftware.com/blog/2014/tdd-straw-men-and-rhetoric
2
http://david.heinemeierhansson.com/2014/slow-database-test-fallacy.html
3
http://c2.com/cgi/wiki?TenYearsOfTestDrivenDevelopment
4
http://xunitpatterns.com/EndoTesting.html
5
http://jmock.org/oopsla2004.pdf
171

The reality is that interest in sub-millisecond tests, and spending a lot of effort to get
6
them, is fairly new. In 2011, when I published the Fast Tests With and Without Rails
screencast, almost no one was talking about doing it in Rails. At that point, if my
memory serves me, only Corey Haines and I were talking about it loudly in public.
The history that David implies is completely false.
Ill stop quoting in a moment, but we need one more. He now repeats a claim from
another post: that TDD leads to worse design. He extends that idea, tying it to test
speed:
Inflicting such damage may well had been worth it back in the old days
when a full suite of tests hitting the database would have taken hours to
run. It so much certainly is not when were talking about a few minutes.
Again, you see reinforcement of the falsehood with "the old days": that was "then",
but today is "now"! However, with this passage, you finally get to see whats really
going on. Davids tests run in a few minutes, and hes fine with that.
Im not fine with that. A lot of other people are not fine with that.
This is the fundamental issue. Its possible that Im the most impatient programmer
on earth; I want my feedback to be so fast that I cant think before it shows up. If I
can think, then Ill sometimes lose attention, and I dont want to lose attention.
I aim for test feedback in 300 ms, from the time I press enter to the time I have a
result. Thats not an abstract desire; I frequently achieve it. Here are the tests for
Destroy All Softwares catalog class:
$ time rspec spec/lib/catalog_spec.rb
......
Finished in 0.00723 seconds
8 examples, 0 failures, 2 pending
0.24s elapsed
Thats 105 lines of test covering 59 lines of production code running in 240
milliseconds end to end, with only 7 of those milliseconds being actual tests. The
test:code ratio here is slightly higher than DAS' overall ratio, which is about 1.4:1,
including both the unit and acceptance test suites. Most of those tests are fully
6
https://www.destroyallsoftware.com/screencasts/catalog/fast-tests-with-and-without-rails
172

isolated, meaning that they only interact with a single class. Some tests use mocks
to achieve that; many dont need to use them.
These tests are fast enough that I can hit enter (my test-running keystroke) and
have a response before I have time to think. It means that the flow of my thoughts
7
never breaks. If youve watched Destroy All Software screencasts , you know that
Ill sometimes run tests ten times per minute. All of my screencasts are recorded live
after doing many takes to smooth the presentation out, so youre seeing my actual
speed, not the result of editing. (They also cost money. I apologize for that, but this
is how I buy food and pay rent.)
Walking through a part of my TDD process will make this whole process more
concrete. I first write a new test and run it by hitting enter. This runs only the current
test file. It also sets that file as "active", meaning that hitting enter from anywhere
in the production code will re-run it.
The tests run in a tmux pane to the right of my editor, so they stick around even while
Im editing code. I expect them to fail, since I just wrote a new hopefully-failing test.
A quarter of a second after I hit enter, the tests have finished running. I flick my eyes
to the output for less than a second; I know where the exception name will appear in
the traceback. Usually, Ill see the exception name that I expect. Because the tests
run so fast, I literally do not have a chance to do anything before they finish, so I cant
get distracted. By the time my eyes have moved to the test pane, the output is there.
While my eyes were moving to the test pane to confirm that the correct exception
was raised, my fingers were switching to the production code file. I make whatever
change will cause the tests to pass. Its usually a small change, which is the nature
of TDD when done well. I then kick off another test by hitting enter. This runs the
same test file as before because it was implicitly set as the active test. I expect it to
pass, and it does, but I only know this because I see green in my peripheral vision;
theres no reason to actually focus my eyes on the test output.
Ive seen my tests fail, made a production code change, and seen the tests pass. A
few seconds have elapsed since I ran the first failing test. Youve spent an order of
magnitude more time reading these few paragraphs about it. Unlike David, Im not
exaggerating here for rhetorical purposes: Im literally talking about a single-digit
number of seconds between running the failing test and seeing it pass.
https://www.destroyallsoftware.com/screencasts
173

My TDD loop isnt always that tight, of course, but most of the second-to-second
steps in building a software system are mundane and dont require careful thought.
If theres a design question, Ill stop and think, possibly aided by pen and paper. If
the tests fail in an unexpected way, Ill stop and analyze the failure. For many tests,
I dont have to do either of these.
Staying under the 300 ms mark necessarily requires isolation. I dont have time
to load Rails, or to load fixtures, or to connect to databases. The test runtime is
dominated by RSpec startup, not the application. In fact, many of these tests load
only a single source file.
That file is going to be a plain old Ruby class: not a Rails model, controller, etc.
Isolating those would lead to pain, as anyone whos tried to do it knows. I keep models
very simple and allow them to integrate with the database; I keep controllers very
simple and generally dont unit test them at all, although I do integration test them.
When Im at full speed, as I was in that story, Im typing at 120 words per
minute. Thats 10 keystrokes per second, with most of those keystrokes being
Vim commands. I dont leave home row; I dont touch a mouse; I dont stop and
read; I dont wait for tests to run. This process allows me to do the trivial parts of
programming as quickly as physically possible. Some people dont believe this, but
there are 90 screencasts in the Destroy All Software catalog that document it.
David misses all of this. Hes never done it. It takes a lot of practice with TDD and
a seriously fast editor that you can use with your eyes closed, but he has neither.
(Again, no rhetorical exaggeration: when Im at full speed, I sometimes close my eyes
because removing visual stimulus aids my concentration.)
Unsurprisingly, there are other parts of Davids post that I disagree with. Most
notably, application preloaders like Spring are band-aids that introduce tremendous
complexity. They cause confusing failures that are fundamental to their nature. Not
to mention his straw man attacks on the design feedback of TDD, and especially
isolated TDD. Ive focused on test speed here because it provides a nice microcosm
of our disagreement and of his tactics. He hasnt actually done the thing that hes
decrying, and hes arguing against it by making things up.
8
Hes been at this for a couple of weeks: if you watch his RailsConf keynote , youll
hear him say that he has very little theoretical computer science knowledge. Then
he pooh-poohs the value of that same theoretical knowledge: knowledge which
8
http://www.justin.tv/confreaks/b/522089408
174

he just said that he doesnt have! Its normal human behavior to form opinions
before understanding something. I certainly wouldnt want to deny anyone their
human fallibility. However, to stand in front of thousands of eager followers that will
believe anything you say, and to proclaim radical, outlandish positions, using madeup history as justification, on topics that you admit to not understanding that just
seems bad.
Finally, I should say that despite being one of the early proponents of sub-millisecond
tests in Rails, I dont have any kind of capital-B Belief in isolation, in mocking, or in
TDD. I use TDD perhaps 75% percent of the time on web apps, and probably less
9
than 50% of the time on console tools like Selecta . I suspect that test doubles,
including mocks, are a stop-gap solution: an intermediate evolutionary form. Ive
10
11
talked about this in conference talks and in screencasts . Both of them contain
a lot of code. The screencast is a tour through an actual software system designed
12
using the scheme in question. Selecta is also designed using the same principles
(although its design is imperfect in many ways, as designs tend to be).
The straw man that David has been propagating may apply to people who picked up
TDD a year ago, and by whom its viewed as a silver bullet that should be applied in
an extreme way. I was one of those people around 2006, so I know that they exist.
There may be other, more experienced people who talk about it in an extreme way
for rhetorical purposes. (I dont know; I dont read intro-to-TDD material, having done
it for almost ten years now.)
If (if!) those things are true, they say nothing about TDD as a technical practice.
Despite writing prose so dripping with rhetoric, David doesnt seem to separate the
rhetorical presentation of a topic from the facts of its use.
TDD is useful and test isolation is useful, but they both involve making trade-offs.
Unfortunately, doing them 100% of the time seems to be the best way to learn what
those trade-offs are, and that can temporarily lead beginners toward extremism.
TDD and isolation both break down in some situations, and learning to detect those
situations in advance takes a lot of time. This is true of advanced techniques in any
discipline, programming or otherwise. That is the honest, non-exaggerated, no-liesinvolved truth.
https://github.com/garybernhardt/selecta
10
https://www.destroyallsoftware.com/talks/boundaries
11
https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell
12
https://github.com/garybernhardt/selecta
175
Teach Yourself Programming in Ten

Years
Written by: Peter Norvig
Why is everyone in such a rush?

Walk into any bookstore, and youll see how to Teach Yourself Java in 24 Hours
alongside endless variations offering to teach C, SQL, Ruby, Algorithms, and so on in
a few days or hours. The Amazon advanced search for [ title: teach, yourself, hours,
2
since: 2000 ] and found 512 such books. Of the top ten, nine are programming
books (the other is about bookkeeping). Similar results come from replacing "teach
yourself" with "learn" or "hours" with "days."
The conclusion is that either people are in a big rush to learn about programming, or
that programming is somehow fabulously easier to learn than anything else. Felleisen
3
et al. give a nod to this trend in their book How to Design Programs , when they say
"Bad programming is easy. Idiots can learn it in 21 days, even if they are dummies."
4
The Abtruse Goose comic also had their take .
5
Lets analyze what a title like Teach Yourself C++ in 24 Hours could mean:
Teach Yourself: In 24 hours you wont have time to write several significant
programs, and learn from your successes and failures with them. You wont have
time to work with an experienced programmer and understand what it is like to
live in a C++ environment. In short, you wont have time to learn much. So the
book can only be talking about a superficial familiarity, not a deep understanding.
As Alexander Pope said, a little learning is a dangerous thing.
1
http://norvig.com/21-days.html
2
http://www.amazon.com/gp/search/ref=sr_adv_b/?search-alias=stripbooks&unfiltered=1&field-
keywords=&field-author=&field-title=teach+yourself+hours&field-isbn=&fieldpublisher=&node=&field-p_n_condition-type=&field-feature_browse-bin=&field-subject=&field-
language=&field-dateop=After&field-datemod=&field-dateyear=2000&sort=relevanceexprank&AdvSrch-Books-Submit.x=16&Adv-Srch-Books-Submit.y=5
3
http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html
4
http://abstrusegoose.com/249
5
http://www.amazon.com/Sams-Teach-Yourself-Hours-5th/dp/0672333317/ref=sr_1_6?
s=books&ie=UTF8&qid=1412708443&sr=1-6&keywords=learn+c%2B%2B+days
176
Teach Yourself Programming in Ten Years

C++: In 24 hours you might be able to learn some of the syntax of C++ (if you
already know another language), but you couldnt learn much about how to use
the language. In short, if you were, say, a Basic programmer, you could learn to
write programs in the style of Basic using C++ syntax, but you couldnt learn
6
what C++ is actually good (and bad) for. So whats the point? Alan Perlis once
said: "A language that doesnt affect the way you think about programming, is
not worth knowing". One possible point is that you have to learn a tiny bit of C+
+ (or more likely, something like JavaScript or Processing) because you need to
interface with an existing tool to accomplish a specific task. But then youre not
learning how to program; youre learning to accomplish that task.
in 24 Hours: Unfortunately, this is not enough, as the next section shows.

7
Researchers ( Bloom (1985) , Bryan & Harter (1899), Hayes (1989) , Simmon
& Chase (1973)) have shown it takes about ten years to develop expertise
in any of a wide variety of areas, including chess playing, music composition,
telegraph operation, painting, piano playing, swimming, tennis, and research in
neuropsychology and topology. The key is deliberative practice: not just doing it
again and again, but challenging yourself with a task that is just beyond your current
ability, trying it, analyzing your performance while and after doing it, and correcting
any mistakes. Then repeat. And repeat again. There appear to be no real shortcuts:
even Mozart, who was a musical prodigy at age 4, took 13 more years before he
began to produce world-class music. In another genre, the Beatles seemed to burst
onto the scene with a string of #1 hits and an appearance on the Ed Sullivan show
in 1964. But they had been playing small clubs in Liverpool and Hamburg since
1957, and while they had mass appeal early on, their first great critical success,
9
Sgt. Peppers, was released in 1967. Malcolm Gladwell has popularized the idea,
although he concentrates on 10,000 hours rather than 10 years.
It may be that 10,000 hours, not 10 years, is the magic number. Or it might be some
other metric; Henri Cartier-Bresson (1908-2004) said "Your first 10,000 photographs
are your worst." True expertise may take a lifetime: Samuel Johnson (1709-1784)
said "Excellence in any department can be attained only by the labor of a lifetime;
it is not to be purchased at a lesser price." And Chaucer (1340-1400) complained
6
7
http://www-pu.informatik.uni-tuebingen.de/users/klaeren/epigrams.html
http://www.amazon.com/exec/obidos/ASIN/034531509X/
8
http://www.amazon.com/exec/obidos/ASIN/0805803092
9
http://www.amazon.com/Outliers-Story-Success-Malcolm-Gladwell/dp/0316017922
177

"the lyf so short, the craft so long to lerne." Hippocrates (c. 400BC) is known for
the excerpt "ars longa, vita brevis", which is part of the longer quotation "Ars longa,
vita brevis, occasio praeceps, experimentum periculosum, iudicium difficile", which
in English renders as "Life is short, [the] craft long, opportunity fleeting, experiment
treacherous, judgment difficult." Of course, no single number can be the final answer:
it doesnt seem reasonable to assume that each of programming, chess playing,
checkers playing, and music playing could all require exactly the same amount of
time to master, nor that all people will take exactly the same amount of time.
So You Want to be a Programmer

Heres my recipe for programming success:
Get interested in programming, and do some because it is fun. Make sure that it
keeps being enough fun so that you will be willing to put in your ten years/10,000
hours.
10
Program. The best kind of learning is learning by doing . To put it more

technically, "the maximal level of performance for individuals in a given domain
is not attained automatically as a function of extended experience, but the level
of performance can be increased even by highly experienced individuals as a
11
result of deliberate efforts to improve." (p. 366) and "the most effective learning
requires a well-defined task with an appropriate difficulty level for the particular
individual, informative feedback, and opportunities for repetition and corrections
of errors." (p. 20-21) The book Cognition in Practice: Mind, Mathematics, and
12
Culture in Everyday Life is an interesting reference for this viewpoint.
Talk with other programmers; read other programs. This is more important than
any book or training course.
If you want, put in four years at a college (or more at a graduate school). This
will give you access to some jobs that require credentials, and it will give you a
deeper understanding of the field, but if you dont enjoy school, you can (with
some dedication) get similar experience on your own or on the job. In any case,
book learning alone wont be enough. "Computer science education cannot make
anybody an expert programmer any more than studying brushes and pigment
can make somebody an expert painter" says Eric Raymond, author of The New
10
http://www.engines4ed.org/hyperbook/nodes/NODE-120-pg.html
11
http://www2.umassd.edu/swpi/DesignInCS/expertise.html
12
178

Hackers Dictionary. One of the best programmers I ever hired had only a High
13
14
School degree; hes produced a lot of great
software , has his own news
15
16
group , and made enough in stock options to buy his own nightclub .
Work on projects with other programmers. Be the best programmer on some
projects; be the worst on some others. When youre the best, you get to test your
abilities to lead a project, and to inspire others with your vision. When youre the
worst, you learn what the masters do, and you learn what they dont like to do
(because they make you do it for them).
Work on projects after other programmers. Understand a program written by
someone else. See what it takes to understand and fix it when the original
programmers are not around. Think about how to design your programs to make
it easier for those who will maintain them after you.
Learn at least a half dozen programming languages. Include one language
that emphasizes class abstractions (like Java or C++), one that emphasizes
functional abstraction (like Lisp or ML or Haskell), one that supports syntactic
abstraction (like Lisp), one that supports declarative specifications (like Prolog or
C++ templates), and one that emphasizes parallelism (like Clojure or Go).
Remember that there is a "computer" in "computer science". Know how long it
takes your computer to execute an instruction, fetch a word from memory (with
and without a cache miss), read consecutive words from disk, and seek to a new
location on disk. (Answers here.)
Get involved in a language standardization effort. It could be the ANSI C++
committee, or it could be deciding if your local coding style will have 2 or 4
space indentation levels. Either way, you learn about what other people like in
a language, how deeply they feel so, and perhaps even a little about why they
feel so.
Have the good sense to get off the language standardization effort as quickly
as possible.
With all that in mind, its questionable how far you can get just by book learning.
Before my first child was born, I read all the How To books, and still felt like a clueless
novice. 30 Months later, when my second child was due, did I go back to the books for
13
14
http://www.xemacs.org/
http://www.mozilla.org/
15
http://groups.google.com/groups?q=alt.fan.jwz&meta=site%3Dgroups
16
http://en.wikipedia.org/wiki/DNA_Lounge
179

a refresher? No. Instead, I relied on my personal experience, which turned out to be
far more useful and reassuring to me than the thousands of pages written by experts.
Fred Brooks, in his essay No Silver Bullet
great software designers:
17
identified a three-part plan for finding
1. Systematically identify top designers as early as possible.

2. Assign a career mentor to be responsible for the development of the prospect
and carefully keep a career file.
3. Provide opportunities for growing designers to interact and stimulate each other.
This assumes that some people already have the qualities necessary for being a
18
great designer; the job is to properly coax them along. Alan Perlis
put it more
succinctly: "Everyone can be taught to sculpt: Michelangelo would have had to be
taught how not to. So it is with the great programmers". Perlis is saying that the
greats have some internal quality that transcends their training. But where does the
quality come from? Is it innate? Or do they develop it through diligence? As Auguste
Gusteau (the fictional chef in Ratatouille) puts it, "anyone can cook, but only the
fearless can be great." I think of it more as willingness to devote a large portion of
ones life to deliberative practice. But maybe fearless is a way to summarize that.
Or, as Gusteaus critic, Anton Ego, says: "Not everyone can become a great artist,
but a great artist can come from anywhere."
So go ahead and buy that Java/Ruby/Javascript/PHP book; youll probably get some
use out of it. But you wont change your life, or your real overall expertise as a
programmer in 24 hours or 21 days. How about working hard to continually improve
over 24 months? Well, now youre starting to get somewhere
References
Bloom, Benjamin (ed.) Developing Talent in Young People
Brooks, Fred, No Silver Bullets
20
19
, Ballantine, 1985.
, IEEE Computer, vol. 20, no. 4, 1987, p. 10-19.
Bryan, W.L. & Harter, N. "Studies on the telegraphic language: The acquisition of a
hierarchy of habits. Psychology Review, 1899, 8, 345-375
17
18
http://en.wikipedia.org/wiki/No_Silver_Bullet
http://www-pu.informatik.uni-tuebingen.de/users/klaeren/epigrams.html
19
http://www.amazon.com/exec/obidos/ASIN/034531509X
20
http://citeseer.nj.nec.com/context/7718/0
180

Hayes, John R., Complete Problem Solver
21
Lawrence Erlbaum, 1989.
Chase, William G. & Simon, Herbert A. "Perception in Chess"

1973, 4, 55-81.
22
Cognitive Psychology,
Lave, Jean, Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life
Cambridge University Press, 1988.
23
Answers
Approximate timing for various operations on a typical PC:
execute typical instruction
1/1,000,000,000 sec = 1 nanosec
fetch from L1 cache memory
0.5 nanosec
branch misprediction
5 nanosec
fetch from L2 cache memory
7 nanosec
Mutex lock/unlock
25 nanosec
fetch from main memory
100 nanosec
send 2K bytes over 1Gbps network
20,000 nanosec
read 1MB sequentially from memory
250,000 nanosec
fetch from new disk location (seek)
8,000,000 nanosec
read 1MB sequentially from disk
20,000,000 nanosec
send packet US to Europe and back
150 milliseconds = 150,000,000

nanosec
Appendix: Language Choice

Several people have asked what programming language they should learn first.
There is no one answer, but consider these points:
Use your friends. When asked "what operating system should I use, Windows,
Unix, or Mac?", my answer is usually: "use whatever your friends use." The
advantage you get from learning from your friends will offset any intrinsic
difference between OS, or between programming languages. Also consider your
21
22
http://books.google.com/books?id=dYPSHAAACAAJ&dq=%22perception+in+chess
%22+simon&ei=z4PyR5iIAZnmtQPbyLyuDQ
23
181

future friends: the community of programmers that you will be a part of if you
continue. Does your chosen language have a large growing community or a small
dying one? Are there books, web sites, and online forums to get answers from?
Do you like the people in those forums?
Keep it simple. Programming languages such as C++ and Java are designed for
professional development by large teams of experienced programmers who are
concerned about the run-time efficiency of their code. As a result, these languages
have complicated parts designed for these circumstances. Youre concerned with
learning to program. You dont need that complication. You want a language that
was designed to be easy to learn and remember by a single new programmer.
Play. Which way would you rather learn to play the piano: the normal, interactive
way, in which you hear each note as soon as you hit a key, or "batch" mode, in
which you only hear the notes after you finish a whole song? Clearly, interactive
mode makes learning easier for the piano, and also for programming. Insist on a
language with an interactive mode and use it.
Given these criteria, my recommendations for a first programming language would
24
25
be Python or Scheme . Another choice is Javascript, not because it is perfectly
well-designed for beginners, but because there are so many online tutorials for it,
26
such as Khan Academys tutorial . But your circumstances may vary, and there are
27
28
other good choices. If your age is a single-digit, you might prefer Alice or Squeak
29
or Blockly (older learners might also enjoy these). The important thing is that you
choose and get started.
Appendix: Books and Other Resources

Several people have asked what books and web pages they should learn from.
I repeat that "book learning alone wont be enough" but I can recommend the
following:
Scheme: Structure and Interpretation of Computer Programs (Abelson &
30
Sussman)
is probably the best introduction to computer science, and it does
24
http://python.org/
25
http://www.schemers.org/
26
https://www.khanacademy.org/computing/cs/programming
27
http://alice.org/
28
http://www.squeak.org/
29
https://blockly-demo.appspot.com/static/apps/index.html
30
http://www.amazon.com/gp/product/0262011530
182

teach programming as a way of understanding the computer science. You can see
31
32
online videos of lectures
on this book, as well as the complete text online .
The book is challenging and will weed out some people who perhaps could be
successful with another approach.
33
Scheme: How to Design Programs (Felleisen et al.) is one of the best books on
how to actually design programs in an elegant and functional way.
Python: Python Programming: An Intro to CS (Zelle)
using Python.
Python: Several online tutorials
35
34
is a good introduction
are available at Python.org
36
Oz: Concepts, Techniques, and Models of Computer Programming (Van Roy &
37
Haridi) is seen by some as the modern-day successor to Abelson & Sussman.
It is a tour through the big ideas of programming, covering a wider range than
Abelson & Sussman while being perhaps easier to read and follow. It uses a
language, Oz, that is not widely known but serves as a basis for learning other
languages.
Notes
38
T\. Capey points out that the Complete Problem Solver page on Amazon now has
the "Teach Yourself Bengali in 21 days" and "Teach Yourself Grammar and Style"
books under the "Customers who shopped for this item also shopped for these items"
section. I guess that a large portion of the people who look at that book are coming
from this page. Thanks to Ross Cohen for help with Hippocrates.
31
http://www.swiss.ai.mit.edu/classes/6.001/abelson-sussman-lectures/
32
http://mitpress.mit.edu/sicp/full-text/book/book.html
33
34
35
http://wiki.python.org/moin/BeginnersGuide
36
http://python.org/
37
38
183
Untested code is broken code: test

automation in enterprise software
delivery
Written by: Martin Aspeli at October 17th, 2013
Test automation is a key technical practice of modern software development, yet it

is sometimes viewed as an unnecessary luxury in enterprise software development.
In this article, Martin Aspeli looks at the reasons why.
In 1975, Fred Brooks wrote The Mythical Man Month, arguably the only book you ever
need to read about software development. In his essay No Silver Bullet, he wrote:
Software entities are more complex for their size than perhaps any other human
construct. Even a moderately complex piece of software could be engineered in an
almost infinite number of different ways, and no two developers will produce identical
outputs for a non-trivial set of requirements. The upshot of all this: we are human
and we make mistakes. Our software contains defects and bugs.
To remove (or at least reduce) the bugs, we rely on testing. Testing takes two main
forms:
1. Validate that the software performs the functions expected by its users. This is
what we refer to as external quality have we built the right thing?
2. Checking the correctness of implementation of each component of a piece of
software. For each possible input, confirm the expected output. We call internal
quality have we built it right?
The first type of testing is hugely important and cannot be automated. It should be
done all the time, with as representative a group of users as possible.
It is the second type of testing that should be automated as much as possible, for
a number of reasons:
It is a labour-intensive process that increases in effort exponentially as the
complexity of the software under test grows.
1
http://www.deloittedigital.com/eu/blog/untested-code-is-broken-code-test-automation-in-enterprise-
software-deliver
184
automation in enterprise software delivery

It is highly repetitive even boring. Consider a simple function that divides one
number by another. To test that this performs as expected, we may want to
test conditions where the numerator is negative, positive, zero, or a very large
number; ditto the denominator, leading to 24 possible conditions that could be
tested. If manual test execution means clicking through a user interface to set up
a condition where the code is executed to produce each of those, it could take
many minutes. Compound that to the thousands of operations performed in even
moderately complex software, and you quickly run out of time.
If we want to be Agile to be responsive to genuinely business-driven change; to
allow architecture and technical foundations to evolve as we learn more about a
project; to catch problems as early as possible to minimise the cost of remediation
we will want to test not just once at the end of a project, but every week or even
every day. Lengthy test execution is profoundly limiting to agility.
Lack of appropriate, timely testing inevitably leads to regressions. Regressions
seriously undermine business confidence in a software team and the ability to
predict project completion. We find seemingly fundamental bugs, and we are
terrified of what else may be broken.
A complex, yet mundane process makes it likely that human error will occur in
test execution, reducing the value of testing.
Test automation is actually a very simple idea: we take this labour-intensive,
repetitive and largely predictable activity and we let a computer do as much of it
as possible, letting the humans focus on the value-adding activities with greater
confidence that the underlying solution is sound.
There are two main types of automated tests: unit tests and acceptance tests.
Unit tests
Automated unit tests assure internal quality. They exercise a single function or unit
of a program. A very simple harness calls the unit under test with a range of inputs
and expected outputs. Few or no additional components are involved, to make it as
easy as possible to trace a failure to its root cause.
Writing the test and the code it is testing goes hand-in hand, and should be done
by the same person. To borrow an analogy from Uncle Bob Martin: automated
unit tests are akin to double-entry bookkeeping. Just like you would never trust
an accountant to make zero mistakes in a complex set of accounts without the
185

confidence provided by dual entry, you should never trust a programmer to write
bug-free code based only on his or her own reading of that code.
A unit test is a very simple piece of code (dumb tests are good tests, as I like to
say) that describes the expected behaviour of a component for a single set of inputs.
By describing the required logic in two different ways one elegant, performant and
featureful for production use, the other nave, repetitive and easy to understand for
testing purposes we radically improve the chances that the code that ends up in
production is actually correct.
Unit tests also serve to inspire confidence. If we believe that the individual units
of our code are highly likely to be correct, then we can more easily alter the way
we assemble those units into larger components and systems. We then test those
integrations, again using automation as far as possible, but here out of necessity
avoiding the full combinatorial complexity of possible inputs and outputs. This is
akin to having the confidence that every component that goes into a car has been
individually quality assured before the car is assembled.
Finally, unit tests can be viewed as executable developer documentation. The test
suite explains, unambiguously, how the programmer intended a function to work. If
in future we attempt to change the innards of that function, say to fix an escaped
defect or improve performance, we can run the test suite again and have confidence
that we did not accidentally break an edge case. Good programmers write specific
tests for intentional conditions that may not be obvious in the future, for this very
reason.
There are three main ways to write unit tests:
1. Test-last: Write some code, then write some tests. This is how most inexperienced
unit testers begin. The problem is that it is easy to lose concentration or succumb
to procrastination and not write as many or as useful tests as we would like.
2. Test-during: Write the code and tests at the same time, and use the test suite as
the starting point for debugging and detailed code analysis. This is how I code
most of the time.
3. Test-first: Write tests that describe the thing you intend to build. Verify that they
fail you havent built it yet! Then write the unit under test, running the tests
repeatedly until they pass. This is the most disciplined, rigorous approach to
unit testing and is known by its acronym TDD Test Driven Development. Some
people find it difficult to get into this habit, but those who do it well write some
of the best code you will ever find.
186
Acceptance tests
Automated acceptance tests assure external quality: are we building the right thing?
The practice of Acceptance Test Driven Development (ATDD, sometimes called
Behaviour Driven Development, or BDD) is based on the idea of writing detailed
requirements in the form of acceptance tests. The business warrants that when those
tests pass, the associated feature can be deemed complete.
In our projects, we typically write the acceptance test scenarios just-in-time as a
medium-level requirement (a user story) is about to scheduled for development. We
then ask the client to sign off these acceptance criteria, to remove any ambiguity
about what we are building. We have found that this not only increases our shared
understanding of requirements, it also leads to radically better requirements and
saves us from having to define separate, possibly inconsistent test cases later.
It is not strictly necessary to automate acceptance tests, but it is a very good idea. We
created a tool called CoreJet specifically to support our workflow of writing, signing
off, automating and reporting against these acceptance tests. It presents a twodimensional visualisation of the project that shows the amount of scope implemented
and covered by passing automated acceptance tests. The more green, the closer
we are to completion.
Automated acceptance tests do not replace unit tests, and typically test key business
scenarios, but not every combination of inputs and outputs. They often involve
scripting a web browser or other user interface, and can be significantly slower to
run than lightweight unit tests. The two approaches are very much complementary.
Acceptance test automation serves another purpose as well: they make sure
developers actually read the specification. Using a tool like CoreJet, the exact text of
the most detailed expression of a requirement is right there in the developers line
of sight, woven into the code of a test suite. There is no separate document to read
and understand and no context switching. We are more likely to build the right thing
the first time around.
Continuous integration
Test automation underpins another key agile practice: continuous integration. It
works like this: a server monitors the source code repository for changes. As soon
as something is changed, the full suite of automated tests is run. If they all pass:
great! If not, a warning sign goes on (quite literally, in our office, where weve rigged
187

a red light to the CI server), and the exact change that caused a test to regress is
highlighted. The developer responsible is made to fix it, buy coffee for the team and
optionally wear a funny hat.
Continuous integration is all about fail fast. The cost of fixing a defect incurred
minutes ago is miniscule compared to the cost of fixing it six months down the
line after thousands of code changes. We all make mistakes, but we can redeem
ourselves quickly by fixing them immediately, whilst the errant code is still fresh in
our minds.
Good testing discipline

Mature development teams adopt a great deal of discipline around their automated
testing:
Monitor and report on test coverage (there are tools that can report on the
percentage of non-test code exercised during a test run). Somewhere north of
75% is considered good. However, this is merely one indicator. It is possible to
have high coverage with poor tests and vice-a-versa, so be pragmatic.
Review the coverage and correctness of tests as part of manual code reviews.
Always run the tests locally before checking in new code to detect regressions.
Never break the build for others on the team.
If you do break the build, dont go home without fixing it.
For the best developers I know, great automated testing is an obsession, and a source
of genuine pride.
But
Alas, not everyone is sold. Below are some of the excuses weve heard for not doing
testing, in increasing order of validity.
We have a team of manual testers. They do the testing

for us.
As outlined, intelligent quality assurance by humans is hugely important. Manual and
automated testing are complementary. However, the somebody elses problem
mentality simply doesnt fly. Each developer should feel accountable for the code
they write being correct. Writing something half-baked and waiting for someone else
188

to find problems is a recipe for messy code and long delays between incurring a
defect and detecting it. This leads to significantly higher overall costs.
Were on a deadline. We dont have time to write tests.

Fear of failure can sometimes create perverse incentives. We think it is better to
meet a deadline with something that doesnt work, than to miss it by a week, but
have something that does.
The answer to this is simple: do less, to a consistently high level of quality. A
looming deadline that looks unachievable is usually a systemic failure to provide the
environment in which the team has enough capacity and realistic goals. Negotiate
scope down and dont attempt to start features that wont be complete in time.
Having regular releases (i.e. short iterations in a Scrum team) and making sure you
work on high priority features first makes this significantly easier.
We charge by the hour. The client wont pay for tests.

Good test automation discipline has the potential to significantly reduce overall effort
expended on a software development project.
The constraining factor of a developers productivity is certainly not the rate at which
he or she can type out code. Measures based on lines of code are largely useless,
but even quite productive developers may not produce more than, say, 100 lines of
non-generated code, net, in a day (the best developers tend to remove a lot of code
by replacing it with more elegant constructs). Typing that out flat would easily take
less than half an hour.
The challenge, of course, is producing the right 100 lines of code. Good automated
unit tests tend to lead to cleaner, better-factored code, because well-factored code
is easier to test. The act of writing a suite of tests tend to help the developer explore
and understand how a piece of logic should behave to be correct in a way that simply
staring at the screen does not. Simply put, test automation accelerates the problemsolving thought process that is software development.
Finally, there is death by a thousand cuts impact to productivity of manual testing.
Writing a simple component may take a few minutes. To check that it works, you
start up an application server and click through the user interface to exercise the
code. This may take another minute or two. You find a problem. Go back and change
the code, restart the server, click through the user interface again. Repeat. The first
couple of times, the time to write a decent automated unit test may be longer than
189

the time to click through the user interface. Once you have a test, though, creating
variants of it to test boundary conditions is normally just copy-and-paste with minor
edits. The test should run in less than a second. Economies of scale kick in.
Now consider a situation where you want to improve or refactor a fundamental,
shared component, with potentially open-ended consequences. A fully manual
regression test of an application to have confidence that the change hasnt broken
something could take hours. It is simply unsustainable.
Im a great developer. I dont need to test my code.

The best developers in the world are acutely aware of the inevitability that they
will introduce defects. That is probably why they are so passionate about test
automation. That said, if you know of a programmer who writes perfect code, tell
them we want to offer them a job.
The tests take too long to run.

As a project grows, it is not uncommon to have thousands of unit tests and hundreds
of acceptance tests, if not more. Sometimes these can take a long time to run. It
is prudent to monitor test execution times. If it takes longer than a few minutes to
run automated unit tests and longer than ten minutes to run automated acceptance
tests, it is worth investing in ways to speed up the tests. Alternatively, it may be
necessary to take a risk-based approach and partition the tests so that some tests
(i.e. those at greater risk of failing) are run more frequently than others. At the very
least, however, you should run all the tests once per day.
We dont know how to write test.

Test automation is a skill. Just as a developer needs to learn the prevailing languages
and frameworks, he or she should learn the test tools used on a project. For people
who often work with cutting-edge technology, programmers can be surprisingly
adverse to learning new things. Have none of it, but invest in training and mentoring
where required.
Our solution doesnt support test automation.

This is probably the only valid reason in this whole list. Some software, especially
older software, can be difficult to automate, for example due to excessive
dependencies between modules or poor abstraction. It may take significant
investment to make custom code testable.
190

However, there are libraries and tools that can reduce this burden. Mocking
frameworks can be used to stub out interfaces, even deep into a third party
component. Tools like Selenium can be used to automate at the user interface level
for web applications. It may not be possible to test perfectly, but even a small number
of tests are infinitely better than none.
One might also suggest that ability to support test automation should be a key
deciding factor when choosing technologies to adopt.
What shouldnt we test?

Test automation is not simply a case of more is more. There are some things that
dont make sense to test. For example:
The layout of a user interface. This is much better judged by a human being than
a computer.
Tautologies, e.g. checking that a constant equals its value.
Excessively repeated tests, e.g. checking that a shared component behaves the
same in every single instance where it is used. Instead, test the component as a
single unit, and then test the other units that rely on it with the assumption that
the shared component works.
Externalised configuration that could change between environments.
Automatically generated code, e.g. getters and setters in Java.
Things unrelated to your code. It is sometimes easy to get a little carried away and
test, for example, that a database knows how to execute a simple query, or that
a web browser can render HTML. Its probably safe to assume those things work.
Conditions that depend on specific data in an external system that cannot be
controlled as part of the test fixture. The problem with this is that the test may
not be easily repeatable across environments.
Conditions that depend on dates. I call these New Years Regressions, as there is
usually some poorly written, date-dependent tests that fail once the system clock
ticks over into another year.
Conclusion
Im going to end this article with a bold statement: Untested code is broken code.
We simply should not accept software solutions that do not have a decent number
191

of appropriate, passing automated tests. It is one of the first things I look for when I
review a software project, and one of the best indicators of project quality.
192
The Three Laws of TDD.

Written by: Robert Martin at Oct 2005
Over the years I have come to describe Test Driven Development in terms of three
simple rules. They are:
1. You are not allowed to write any production code unless it is to make a failing
unit test pass.
2. You are not allowed to write any more of a unit test than is sufficient to fail; and
compilation failures are failures.
3. You are not allowed to write any more production code than is sufficient to pass
the one failing unit test.
You must begin by writing a unit test for the functionality that you intend to write.
But by rule 2, you cant write very much of that unit test. As soon as the unit test
code fails to compile, or fails an assertion, you must stop and write production code.
But by rule 3 you can only write the production code that makes the test compile
or pass, and no more.
If you think about this you will realize that you simply cannot write very much code
at all without compiling and executing something. Indeed, this is really the point. In
everything we do, whether writing tests, writing production code, or refactoring, we
keep the system executing at all times. The time between running tests is on the
order of seconds, or minutes. Even 10 minutes is too long.
2
Too see this in operation, take a look at The Bowling Game Kata .
Now most programmers, when they first hear about this technique, think: "This is
stupid!" "Its going to slow me down, its a waste of time and effort, It will keep me
from thinking, it will keep me from designing, it will just break my flow." However,
think about what would happen if you walked in a room full of people working this
way. Pick any random person at any random time. A minute ago, all their code
worked.
Let me repeat that: A minute ago all their code worked! And it doesnt matter who you
pick, and it doesnt matter when you pick. A minute ago all their code worked!
1
http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd
2
http://butunclebob.com/ArticleS.UncleBob.TheBowlingGameKata
193

If all your code works every minute, how often will you use a debugger? Answer,
not very often. Its easier to simply hit ^Z a bunch of times to get the code back
to a working state, and then try to write the last minutes worth again. And if you
arent debugging very much, how much time will you be saving? How much time do
you spend debugging now? How much time do you spend fixing bugs once youve
debugged them? What if you could decrease that time by a significant fraction?
But the benefit goes far beyond that. If you work this way, then every hour you are
producing several tests. Every day dozens of tests. Every month hundreds of tests.
Over the course of a year you will write thousands of tests. You can keep all these
tests and run them any time you like! When would you run them? All the time! Any
time you made any kind of change at all!
Why dont we clean up code that we know is messy? Were afraid well break it.
But if we have the tests, we can be reasonably sure that the code is not broken, or
that well detect the breakage immediately. If we have the tests we become fearless
about making changes. If we see messy code, or an unclean structure, we can clean
it without fear. Because of the tests, the code becomes malleable again. Because of
the tests, software becomes soft again.
But the benefits go beyond that. If you want to know how to call a certain API, there
is a test that does it. If you want to know how to create a certain object, there is
a test that does it. Anything you want to know about the existing system, there is
a test that demonstrates it. The tests are like little design documents, little coding
examples, that describe how the system works and how to use it.
Have you ever integrated a third party library into your project? You got a big manual
full of nice documentation. At the end there was a thin appendix of examples. Which
of the two did you read? The examples of course! Thats what the unit tests are!
They are the most useful part of the documentation. They are the living examples of
how to use the code. They are design documents that are hideously detailed, utterly
unambiguous, so formal that they execute, and they cannot get out of sync with the
production code.
But the benefits go beyond that. If you have ever tried to add unit tests to a system
that was already working, you probably found that it wasnt much fun. You likely
found that you either had to change portions of the design of the system, or cheat
on the tests; because the system you were trying to write tests for was not designed
to be testable. For example, youd like to test some function f . However, f calls
another function that deletes a record from the database. In your test, you dont
194

want the record deleted, but you dont have any way to stop it. The system wasnt
designed to be tested.
When you follow the three rules of TDD, all your code will be testable by definition!
And another word for "testable" is "decoupled". In order to test a module in isolation,
you must decouple it. So TDD forces you to decouple modules. Indeed, if you follow
the three rules, you will find yourself doing much more decoupling than you may be
used to. This forces you to create better, less coupled, designs.
Given all these benfits, these stupid little rules of TDD might not actually be so stupid.
They might actually be something fundemental, something profound. Indeed, I had
been a programmer for nearly thirty years before I was introduced to TDD. I did not
think anyone could teach me a low level programming practice that would make a
difference. Thirty years is a lot of experience after all. But when I started to use TDD,
I was dumbfounded at the effectiveness of the technique. I was also hooked. I can no
longer concieve of typing in a big long batch of code hoping it works. I can no longer
tolerate ripping a set of modules apart, hoping to reassemble them and get them all
working by next Friday. Every decision I make while programming is driven by the
basic need to be executing again a minute from now.
195
Colophon
All articles were manually converted from HTML to AsciiDoc
and the final work

4
was built into the single electronic publication in PDF format using AsciiDoctor and
5
DocBook toolchains.
http://asciidoctor.org/docs/asciidoc-writers-guide/
4
http://asciidoctor.org/
5
http://www.docbook.org/
196

The Mozaic

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

The Mozaic

Încărcat de

Drepturi de autor:

Formate disponibile

The Mozaic

Mark Safronov (compiled only)

Youre just another carriage return

Youre just another carriage return line feed in the wall

Sigh. Whose fault is this?

Youre just another carriage return line feed in the wall

Or you can do what GitHub for Windows suggests with text=auto .

Youre just another carriage return line feed in the wall

Youre just another carriage return line feed in the wall

What every programmer absolutely,

Getting the basics straight

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

Binary, octal, decimal, hex

of encoding schemes , standards, de-facto standards and half-standards that all

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

to encode 65,536 distinct values. BIG-5 is such a double-byte encoding. Instead

GB18030 covers quite a range of characters (including a large part of latin

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

Unicode to the confusion

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

00000000 00000000 00000000

11100011 10000001 10000010

00000000 00000000 00110000

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

01000110 11111000 11110110

01000110 10111111 10011010

01000110 11000011 10111000 11000011

Misconceptions, confusions and problems

Why in gods name are my characters garbled?!

10010011 10000011 01010010 10000001 01011011

readable point of view.

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

My document doesnt make sense in any encoding!

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

So how to handle encodings correctly?

00000000 00000000 01111110

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

Unicode all the way

Anonymous code monkey

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

UTF-8 and ASCII

Saving this as UTF-8 results in this bit sequence:

And of course therell be no recent backup.

What every programmer absolutely, positively needs to

know about encodings and character sets to work with text

Encodings and PHP

One specific pet-peeve of mine are the functions utf8_encode