First Monday Read related articles on Computer industry, Internet economics, Linux and Software development The Cathedral

and the Ba aar !y "ric S# Raymond I anatomi e a successful open$source pro%ect, fetchmail, that was run as a deli!erate test of some surprisin& theories a!out software en&ineerin& su&&ested !y the history of Linux# I discuss these theories in terms of two fundamentally different development styles, the 'cathedral' model of most of the commercial world versus the '!a aar' model of the Linux world# I show that these models derive from opposin& assumptions a!out the nature of the software$de!u&&in& tas(# I then ma(e a sustained ar&ument from the Linux experience for the proposition that ')iven enou&h eye!alls, all !u&s are shallow', su&&est productive analo&ies with other self$correctin& systems of selfish a&ents, and conclude with some exploration of the implications of this insi&ht for the future of software# Contents The Cathedral and the Ba aar The Mail Must )et Throu&h The Importance of *avin& +sers Release "arly, Release ,ften -hen Is . Rose /ot . Rose0 1opclient !ecomes Fetchmail Fetchmail )rows +p . Few More Lessons From Fetchmail /ecessary 1reconditions for the Ba aar Style The Social Context of ,pen$Source Software .c(nowled&ements For Further Readin& "pilo&2 /etscape "m!races the Ba aar3 4ersion and Chan&e *istory The Cathedral and the Ba aar Linux is su!versive# -ho would have thou&ht even five years a&o that a world$class operatin& system could coalesce as if !y ma&ic out of part$time hac(in& !y several thousand developers scattered all over the planet, connected only !y the tenuous strands of the Internet0 Certainly not I# By the time Linux swam onto my radar screen in early 5667, I had already !een involved in +nix and open$source development for ten years# I was one of the first )/+ contri!utors in the mid$5689s# I had released a &ood deal of open$source software onto the net, developin& or co$developin& several pro&rams :nethac(, "macs 4C and )+; modes, xlife, and others< that are still in wide use today# I thou&ht I (new how it was done# Linux overturned much of what I thou&ht I (new# I had !een preachin& the +nix &ospel of small tools, rapid prototypin& and evolutionary pro&rammin& for years# But I also !elieved there was a certain critical complexity a!ove which a more centrali ed, a priori approach was re=uired# I !elieved that the most important software :operatin& systems and really lar&e tools li(e "macs< needed to !e !uilt li(e cathedrals, carefully crafted !y individual wi ards or small !ands of ma&es wor(in& in splendid isolation, with no !eta to !e released !efore its time# Linus Torvalds>s style of development $ release early and often, dele&ate everythin& you can, !e open to the point of promiscuity $ came as a surprise# /o =uiet, reverent cathedral$!uildin& here $ rather, the Linux community seemed to resem!le a &reat !a!!lin& !a aar of differin& a&endas and approaches :aptly sym!oli ed !y the Linux archive sites, who>d ta(e su!missions

from anyone< out of which a coherent and sta!le system could seemin&ly emer&e only !y a succession of miracles# The fact that this !a aar style seemed to wor(, and wor( well, came as a distinct shoc(# .s I learned my way around, I wor(ed hard not %ust at individual pro%ects, !ut also at tryin& to understand why the Linux world not only didn>t fly apart in confusion !ut seemed to &o from stren&th to stren&th at a speed !arely ima&ina!le to cathedral$!uilders# By mid$566? I thou&ht I was !e&innin& to understand# Chance handed me a perfect way to test my theory, in the form of an open$source pro%ect which I could consciously try to run in the !a aar style# So I did $ and it was a si&nificant success# In the rest of this article, I>ll tell the story of that pro%ect, and I>ll use it to propose some aphorisms a!out effective open$source development# /ot all of these are thin&s I first learned in the Linux world, !ut we>ll see how the Linux world &ives them particular point# If I>m correct, they>ll help you understand exactly what it is that ma(es the Linux community such a fountain of &ood software $ and help you !ecome more productive yourself# The Mail Must )et Throu&h Since 5667 I>d !een runnin& the technical side of a small free$access IS1 called Chester County InterLin( :CCIL< in -est Chester, 1ennsylvania :I co$founded CCIL and wrote our uni=ue multiuser !ulletin$!oard software $ you can chec( it out !y telnettin& to loc(e#ccil#or&# Today it supports almost three thousand users on nineteen lines<# The %o! allowed me @A$hour$a$day access to the Internet throu&h CCIL>s B?C line $ in fact, it practically demanded it3 .ccordin&ly, I had &otten =uite used to instant Internet e$mail# For complicated reasons, it was hard to &et SLI1 to wor( !etween my home machine :snar(#thyrsus#com< and CCIL# -hen I finally succeeded, I found havin& to periodically telnet over to loc(e to chec( my mail annoyin&# -hat I wanted was for my mail to !e delivered on snar( so that I would !e notified when it arrived and could handle it usin& all my local tools# Simple sendmail forwardin& wouldn>t wor(, !ecause my personal machine isn>t always on the /et and doesn>t have a static I1 address# -hat I needed was a pro&ram that would reach out over my SLI1 connection and pull across my mail to !e delivered locally# I (new such thin&s existed, and that most of them used a simple application protocol called 1,1 :1ost ,ffice 1rotocol<# .nd sure enou&h, there was already a 1,17 server included with loc(e>s BS;D,S operatin& system# I needed a 1,17 client# So I went out on the net and found one# .ctually, I found three or four# I used pop$perl for a while, !ut it was missin& what seemed an o!vious feature, the a!ility to hac( the addresses on fetched mail so replies would wor( properly# The pro!lem was this2 suppose someone named E%oe> on loc(e sent me mail# If I fetched the mail to snar( and then tried to reply to it, my mailer would cheerfully try to ship it to a nonexistent >%oe> on snar(# *and$editin& reply addresses to tac( on >Fccil#or&> =uic(ly &ot to !e a serious pain# This was clearly somethin& the computer ou&ht to !e doin& for me# But none of the existin& 1,1 clients (new how3 .nd this !rin&s us to the first lesson2 5# "very &ood wor( of software starts !y scratchin& a developer>s personal itch# 1erhaps this should have !een o!vious :it>s lon& !een prover!ial that '/ecessity is the mother of invention'< !ut too often software developers spend their days &rindin& away for pay at pro&rams they neither need nor love# But not in the Linux world $ which may explain why the

avera&e =uality of software ori&inated in the Linux community is so hi&h# So, did I immediately launch into a furious whirl of codin& up a !rand$new 1,17 client to compete with the existin& ones0 /ot on your life3 I loo(ed carefully at the 1,1 utilities I had in hand, as(in& myself 'which one is closest to what I want0'# Because @# )ood pro&rammers (now what to write# )reat ones (now what to rewrite :and reuse<# -hile I don>t claim to !e a &reat pro&rammer, I try to imitate one# .n important trait of the &reat ones is constructive la iness# They (now that you &et an . not for effort !ut for results, and that it>s almost always easier to start from a &ood partial solution than from nothin& at all# Linus Torvalds, for example, didn>t actually try to write Linux from scratch# Instead, he started !y reusin& code and ideas from Minix, a tiny +nix$li(e ,S for 78? machines# "ventually all the Minix code went away or was completely rewritten $ !ut while it was there, it provided scaffoldin& for the infant that would eventually !ecome Linux# In the same spirit, I went loo(in& for an existin& 1,1 utility that was reasona!ly well coded, to use as a development !ase# The source$sharin& tradition of the +nix world has always !een friendly to code reuse :this is why the )/+ pro%ect chose +nix as a !ase ,S, in spite of serious reservations a!out the ,S itself<# The Linux world has ta(en this tradition nearly to its technolo&ical limitG it has tera!ytes of open sources &enerally availa!le# So spendin& time loo(in& for some else>s almost$&ood$enou&h is more li(ely to &ive you &ood results in the Linux world than anywhere else# .nd it did for me# -ith those I>d found earlier, my second search made up a total of nine candidates $ fetchpop, 1opTart, &et$mail, &wpop, pimp, pop$perl, popc, popmail and upop# The one I first settled on was >fetchpop> !y Seun&$*on& ,h# I put my header$rewrite feature in it, and made various other improvements which the author accepted into his 5#6 release# . few wee(s later, thou&h, I stum!led across the code for >popclient> !y Carl *arris, and found I had a pro!lem# Thou&h fetchpop had some &ood ori&inal ideas in it :such as its daemon mode<, it could only handle 1,17 and was rather amateurishly coded :Seun&$*on& was a !ri&ht !ut inexperienced pro&rammer, and !oth traits showed<# Carl>s code was !etter, =uite professional and solid, !ut his pro&ram lac(ed several important and rather tric(y$to$implement fetchpop features :includin& those I>d coded myself<# Stay or switch0 If I switched, I>d !e throwin& away the codin& I>d already done in exchan&e for a !etter development !ase# . practical motive to switch was the presence of multiple$protocol support# 1,17 is the most commonly used of the post$office server protocols, !ut not the only one# Fetchpop and the other competition didn>t do 1,1@, R1,1, or .1,1, and I was already havin& va&ue thou&hts of perhaps addin& IM.1 :Internet Messa&e .ccess 1rotocol, the most recently desi&ned and most powerful post$office protocol< %ust for fun# But I had a more theoretical reason to thin( switchin& mi&ht !e as &ood an idea as well, somethin& I learned lon& !efore Linux# 7# '1lan to throw one awayG you will, anyhow#' :Fred Broo(s, The Mythical Man$Month, Chapter 55<

,r, to put it another way, you often don>t really understand the pro!lem until after the first time you implement a solution# The second time, may!e you (now enou&h to do it ri&ht# So if you want to &et it ri&ht, !e ready to start over at least once# -ell :I told myself< the chan&es to fetchpop had !een my first try# So I switched# .fter I sent my first set of popclient patches to Carl *arris on @B Hune 566?, I found out that he had !asically lost interest in popclient some time !efore# The code was a !it dusty, with minor !u&s han&in& out# I had many chan&es to ma(e, and we =uic(ly a&reed that the lo&ical thin& for me to do was ta(e over the pro&ram# -ithout my actually noticin&, the pro%ect had escalated# /o lon&er was I %ust contemplatin& minor patches to an existin& 1,1 client# I too( on maintainin& an entire one, and there were ideas !u!!lin& in my head that I (new would pro!a!ly lead to ma%or chan&es# In a software culture that encoura&es code$sharin&, this is a natural way for a pro%ect to evolve# I was actin& out this2 A# If you have the ri&ht attitude, interestin& pro!lems will find you# But Carl *arris>s attitude was even more important# *e understood that B# -hen you lose interest in a pro&ram, your last duty to it is to hand it off to a competent successor# -ithout ever havin& to discuss it, Carl and I (new we had a common &oal of havin& the !est solution out there# The only =uestion for either of us was whether I could esta!lish that I was a safe pair of hands# ,nce I did that, he acted with &race and dispatch# I hope I will act as well when it comes my turn# The Importance of *avin& +sers .nd so I inherited popclient# Hust as importantly, I inherited popclient>s user !ase# +sers are wonderful thin&s to have, and not %ust !ecause they demonstrate that you>re servin& a need, that you>ve done somethin& ri&ht# 1roperly cultivated, they can !ecome co$developers# .nother stren&th of the +nix tradition, one that Linux pushes to a happy extreme, is that a lot of users are hac(ers too# Because source code is availa!le, they can !e effective hac(ers# This can !e tremendously useful for shortenin& de!u&&in& time# )iven a !it of encoura&ement, your users will dia&nose pro!lems, su&&est fixes, and help improve the code far more =uic(ly than you could unaided# ?# Treatin& your users as co$developers is your least$hassle route to rapid code improvement and effective de!u&&in&# The power of this effect is easy to underestimate# In fact, pretty well all of us in the open$ source world drastically underestimated how well it would scale up with num!er of users and a&ainst system complexity, until Linus Torvalds showed us differently# In fact, I thin( Linus> cleverest and most conse=uential hac( was not the construction of the Linux (ernel itself, !ut rather his invention of the Linux development model# -hen I expressed this opinion in his presence once, he smiled and =uietly repeated somethin& he has often said2 'I>m !asically a very la y person who li(es to &et credit for thin&s other people actually do#' La y

li(e a fox# ,r, as Ro!ert *einlein mi&ht have said, too la y to fail# In retrospect, one precedent for the methods and success of Linux can !e seen in the development of the )/+ "macs Lisp li!rary and Lisp code archives# In contrast to the cathedral$ !uildin& style of the "macs C core and most other FSF tools, the evolution of the Lisp code pool was fluid and very user$driven# Ideas and prototype modes were often rewritten three or four times !efore reachin& a sta!le final form# .nd loosely$coupled colla!orations ena!led !y the Internet, a la Linux, were fre=uent# Indeed, my own most successful sin&le hac( previous to fetchmail was pro!a!ly "macs 4C mode, a Linux$li(e colla!oration !y e$mail with three other people, only one of whom :Richard Stallman, the author of "macs and founder of the Free Software Foundation or FSF< I have met to this day# It was a front$end for SCCS, RCS and later C4S from within "macs that offered 'one$ touch' version control operations# It evolved from a tiny, crude sccs#el mode some!ody else had written# .nd the development of 4C succeeded !ecause, unli(e "macs itself, "macs Lisp code could &o throu&h releaseDtestDimprove &enerations very =uic(ly# ,ne unexpected side$effect of FSF>s policy of tryin& to le&ally !ind code into the )1L is that it !ecomes procedurally harder for FSF to use the !a aar mode, since they !elieve they must &et a copyri&ht assi&nment for every individual contri!ution of more than twenty lines in order to immuni e )1Led code from challen&e under copyri&ht law# 1eople who copyri&ht usin& the BS; and MIT I Consortium licenses don>t have this pro!lemG they>re not tryin& to reserve ri&hts that anyone mi&ht have an incentive to challen&e# Release "arly, Release ,ften "arly and fre=uent releases are a critical part of the Linux development model# Most developers :includin& me< used to !elieve this was !ad policy for lar&er than trivial pro%ects, !ecause early versions are almost !y definition !u&&y versions and you don>t want to wear out the patience of your users# This !elief reinforced the &eneral commitment to a cathedral$!uildin& style of development# If the overridin& o!%ective was for users to see as few !u&s as possi!le, why then you>d only release one every six months :or less often<, and wor( li(e a do& on de!u&&in& !etween releases# The "macs C core was developed this way# The Lisp li!rary, in effect, was not $ !ecause there were active Lisp archives outside the FSF>s control, where you could &o to find new and development code versions independently of "macs>s release cycle# The most important of these, the ,hio State elisp archive, anticipated the spirit and many of the features of today>s !i& Linux archives# But few of us really thou&ht very hard a!out what we were doin&, or a!out what the very existence of that archive su&&ested a!out pro!lems in FSF>s cathedral$!uildin& development model# I made one serious attempt around 566@ to &et a lot of the ,hio code formally mer&ed into the official "macs Lisp li!rary# I ran into political trou!le and was lar&ely unsuccessful# But !y a year later, as Linux !ecame widely visi!le, it was clear that somethin& different and much healthier was &oin& on there# Linus> open development policy was the very opposite of cathedral$!uildin&# The sunsite and tsx$55 archives were !ur&eonin&, multiple distri!utions were !ein& floated# .nd all of this was driven !y an unheard$of fre=uency of core system releases# Linus was treatin& his users as co$developers in the most effective possi!le way2 J# Release early# Release often# .nd listen to your customers#

Linus> innovation wasn>t so much in doin& this :somethin& li(e it had !een +nix$world tradition for a lon& time<, !ut in scalin& it up to a level of intensity that matched the complexity of what he was developin&# In those early times :around 5665< it wasn>t un(nown for him to release a new (ernel more than once a day3 Because he cultivated his !ase of co$developers and levera&ed the Internet for colla!oration harder than anyone else, this wor(ed# But how did it wor(0 .nd was it somethin& I could duplicate, or did it rely on some uni=ue &enius of Linus Torvalds0 I didn>t thin( so# )ranted, Linus is a damn fine hac(er :how many of us could en&ineer an entire production$=uality operatin& system (ernel0<# But Linux didn>t represent any awesome conceptual leap forward# Linus is not :or at least, not yet< an innovative &enius of desi&n in the way that, say, Richard Stallman or Hames )oslin& :of /e-S and Hava< are# Rather, Linus seems to me to !e a &enius of en&ineerin&, with a sixth sense for avoidin& !u&s and development dead$ ends and a true (nac( for findin& the minimum$effort path from point . to point B# Indeed, the whole desi&n of Linux !reathes this =uality and mirrors Linus> essentially conservative and simplifyin& desi&n approach# So, if rapid releases and levera&in& the Internet medium to the hilt were not accidents !ut inte&ral parts of Linus> en&ineerin&$&enius insi&ht into the minimum$effort path, what was he maximi in&0 -hat was he cran(in& out of the machinery0 1ut that way, the =uestion answers itself# Linus was (eepin& his hac(erDusers constantly stimulated and rewarded $ stimulated !y the prospect of havin& an e&o$satisfyin& piece of the action, rewarded !y the si&ht of constant :even daily< improvement in their wor(# Linus was directly aimin& to maximi e the num!er of person$hours thrown at de!u&&in& and development, even at the possi!le cost of insta!ility in the code and user$!ase !urnout if any serious !u& proved intracta!le# Linus was !ehavin& as thou&h he !elieved somethin& li(e this2 8# )iven a lar&e enou&h !eta$tester and co$developer !ase, almost every pro!lem will !e characteri ed =uic(ly and the fix o!vious to someone# ,r, less formally, ')iven enou&h eye!alls, all !u&s are shallow#' I du! this2 'Linus> Law'# My ori&inal formulation was that every pro!lem 'will !e transparent to some!ody'# Linus demurred that the person who understands and fixes the pro!lem is not necessarily or even usually the person who first characteri es it# 'Some!ody finds the pro!lem', he says, 'and some!ody else understands it# .nd I>ll &o on record as sayin& that findin& it is the !i&&er challen&e#' But the point is that !oth thin&s tend to happen =uic(ly# *ere, I thin(, is the core difference underlyin& the cathedral$!uilder and !a aar styles# In the cathedral$!uilder view of pro&rammin&, !u&s and development pro!lems are tric(y, insidious, deep phenomena# It ta(es months of scrutiny !y a dedicated few to develop confidence that you>ve win(led them all out# Thus the lon& release intervals, and the inevita!le disappointment when lon&$awaited releases are not perfect# In the !a aar view, on the other hand, you assume that !u&s are &enerally shallow phenomena $ or, at least, that they turn shallow pretty =uic( when exposed to a thousand ea&er co$developers poundin& on every sin&le new release# .ccordin&ly you release often in order to &et more corrections, and as a !eneficial side effect you have less to lose if an occasional !otch &ets out the door# .nd that>s it# That>s enou&h# If 'Linus> Law' is false, then any system as complex as the

Linux (ernel, !ein& hac(ed over !y as many hands as the Linux (ernel, should at some point have collapsed under the wei&ht of unforseen !ad interactions and undiscovered 'deep' !u&s# If it>s true, on the other hand, it is sufficient to explain Linux>s relative lac( of !u&&iness# .nd may!e it shouldn>t have !een such a surprise, at that# Sociolo&ists years a&o discovered that the avera&ed opinion of a mass of e=ually expert :or e=ually i&norant< o!servers is =uite a !it more relia!le a predictor than that of a sin&le randomly$chosen one of the o!servers# They called this the ';elphi effect'# It appears that what Linus has shown is that this applies even to de!u&&in& an operatin& system $ that the ;elphi effect can tame development complexity even at the complexity level of an ,S (ernel# I am inde!ted to Heff ;ut(y Kdut(yFwam#umd#eduL for pointin& out that Linus> Law can !e rephrased as ';e!u&&in& is paralleli a!le'# Heff o!serves that althou&h de!u&&in& re=uires de!u&&ers to communicate with some coordinatin& developer, it doesn>t re=uire si&nificant coordination !etween de!u&&ers# Thus it doesn>t fall prey to the same =uadratic complexity and mana&ement costs that ma(e addin& developers pro!lematic# In practice, the theoretical loss of efficiency due to duplication of wor( !y de!u&&ers almost never seems to !e an issue in the Linux world# ,ne effect of a 'release early and often policy' is to minimi e such duplication !y propa&atin& fed$!ac( fixes =uic(ly# Broo(s even made an off$hand o!servation related to Heff>s2 'The total cost of maintainin& a widely used pro&ram is typically A9 percent or more of the cost of developin& it# Surprisin&ly this cost is stron&ly affected !y the num!er of users# More users find more !u&s'# :my emphasis<# More users find more !u&s !ecause addin& more users adds more different ways of stressin& the pro&ram# This effect is amplified when the users are co$developers# "ach one approaches the tas( of !u& characteri ation with a sli&htly different perceptual set and analytical tool(it, a different an&le on the pro!lem# The ';elphi effect' seems to wor( precisely !ecause of this variation# In the specific context of de!u&&in&, the variation also tends to reduce duplication of effort# So addin& more !eta$testers may not reduce the complexity of the current 'deepest' !u& from the developer>s point of view, !ut it increases the pro!a!ility that someone>s tool(it will !e matched to the pro!lem in such a way that the !u& is shallow to that person# Linus coppers his !ets, too# In case there are serious !u&s, Linux (ernel version are num!ered in such a way that potential users can ma(e a choice either to run the last version desi&nated 'sta!le' or to ride the cuttin& ed&e and ris( !u&s in order to &et new features# This tactic is not yet formally imitated !y most Linux hac(ers, !ut perhaps it should !eG the fact that either choice is availa!le ma(es !oth more attractive# -hen Is . Rose /ot . Rose0 *avin& studied Linus> !ehavior and formed a theory a!out why it was successful, I made a conscious decision to test this theory on my new :admittedly much less complex and am!itious< pro%ect# But the first thin& I did was reor&ani e and simplify popclient a lot# Carl *arris>s implementation was very sound, !ut exhi!ited a (ind of unnecessary complexity common to many C pro&rammers# *e treated the code as central and the data structures as support for the code# .s a result, the code was !eautiful !ut the data structure desi&n ad$hoc and rather u&ly :at least !y the hi&h standards of this old LIS1 hac(er<# I had another purpose for rewritin& !esides improvin& the code and the data structure

desi&n, however# That was to evolve it into somethin& I understood completely# It>s no fun to !e responsi!le for fixin& !u&s in a pro&ram you don>t understand# For the first month or so, then, I was simply followin& out the implications of Carl>s !asic desi&n# The first serious chan&e I made was to add IM.1 support# I did this !y reor&ani in& the protocol machines into a &eneric driver and three method ta!les :for 1,1@, 1,17, and IM.1<# This and the previous chan&es illustrate a &eneral principle that>s &ood for pro&rammers to (eep in mind, especially in lan&ua&es li(e C that don>t naturally do dynamic typin&2 6# Smart data structures and dum! code wor(s a lot !etter than the other way around# Broo(s, Chapter 62 'Show me your McodeN and conceal your Mdata structuresN, and I shall continue to !e mystified# Show me your Mdata structuresN, and I won>t usually need your McodeNG it>ll !e o!vious#' .ctually, he said 'flowcharts' and 'ta!les'# But allowin& for thirty years of terminolo&icalDcultural shift, it>s almost the same point# .t this point :early Septem!er 566?, a!out six wee(s from ero< I started thin(in& that a name chan&e mi&ht !e in order $ after all, it wasn>t %ust a 1,1 client any more# But I hesitated, !ecause there was as yet nothin& &enuinely new in the desi&n# My version of popclient had yet to develop an identity of its own# That chan&ed, radically, when fetchmail learned how to forward fetched mail to the SMT1 port# I>ll &et to that in a moment# But first2 I said a!ove that I>d decided to use this pro%ect to test my theory a!out what Linus Torvalds had done ri&ht# *ow :you may well as(< did I do that0 In these ways2 5# I released early and often :almost never less often than every ten daysG durin& periods of intense development, once a day<# @# I &rew my !eta list !y addin& to it everyone who contacted me a!out fetchmail# 7# I sent chatty announcements to the !eta list whenever I released, encoura&in& people to participate# A# .nd I listened to my !eta testers, pollin& them a!out desi&n decisions and stro(in& them whenever they sent in patches and feed!ac(# The payoff from these simple measures was immediate# From the !e&innin& of the pro%ect, I &ot !u& reports of a =uality most developers would (ill for, often with &ood fixes attached# I &ot thou&htful criticism, I &ot fan mail, I &ot intelli&ent feature su&&estions# -hich leads to2 59# If you treat your !eta$testers as if they>re your most valua!le resource, they will respond !y !ecomin& your most valua!le resource# ,ne interestin& measure of fetchmail>s success is the sheer si e of the pro%ect !eta list, fetchmail$friends# .t time of writin& it has @A6 mem!ers and is addin& two or three a wee(# .ctually, as I revise in late May 566J the list is !e&innin& to lose mem!ers from its hi&h of close to 799 for an interestin& reason# Several people have as(ed me to unsu!scri!e them !ecause fetchmail is wor(in& so well for them that they no lon&er need to see the list traffic3 1erhaps this is part of the normal life$cycle of a mature !a aar$style pro%ect# 1opclient !ecomes Fetchmail

The real turnin& point in the pro%ect was when *arry *ochheiser sent me his scratch code for forwardin& mail to the client machine>s SMT1 port# I reali ed almost immediately that a relia!le implementation of this feature would ma(e all the other delivery modes next to o!solete# For many wee(s I had !een twea(in& fetchmail rather incrementally while feelin& li(e the interface desi&n was servicea!le !ut &ru!!y $ inele&ant and with too many exi&uous options han&in& out all over# The options to dump fetched mail to a mail!ox file or standard output particularly !othered me, !ut I couldn>t fi&ure out why# -hat I saw when I thou&ht a!out SMT1 forwardin& was that popclient had !een tryin& to do too many thin&s# It had !een desi&ned to !e !oth a mail transport a&ent :MT.< and a local delivery a&ent :M;.<# -ith SMT1 forwardin&, it could &et out of the M;. !usiness and !e a pure MT., handin& off mail to other pro&rams for local delivery %ust as sendmail does# -hy mess with all the complexity of confi&urin& a mail delivery a&ent or settin& up loc($and$ append on a mail!ox when port @B is almost &uaranteed to !e there on any platform with TC1DI1 support in the first place0 "specially when this means retrieved mail is &uaranteed to loo( li(e normal sender$initiated SMT1 mail, which is really what we want anyway# There are several lessons here# First, this SMT1$forwardin& idea was the !i&&est sin&le payoff I &ot from consciously tryin& to emulate Linus> methods# . user &ave me this terrific idea $ all I had to do was understand the implications# 55# The next !est thin& to havin& &ood ideas is reco&ni in& &ood ideas from your users# Sometimes the latter is !etter# Interestin&ly enou&h, you will =uic(ly find that if you are completely and self$deprecatin&ly truthful a!out how much you owe other people, the world at lar&e will treat you li(e you did every !it of the invention yourself and are %ust !ein& !ecomin&ly modest a!out your innate &enius# -e can all see how well this wor(ed for Linus3 :-hen I &ave this paper at the 1erl conference in .u&ust 566J, Larry -all was in the front row# .s I &ot to the last line a!ove he called out, reli&ious$revival style, 'Tell, it, tell it, !rother3'# The whole audience lau&hed, !ecause they (new it had wor(ed for the inventor of 1erl too#< .fter a very few wee(s of runnin& the pro%ect in the same spirit, I !e&an to &et similar praise not %ust from my users !ut from other people to whom the word lea(ed out# I stashed away some of that e$mailG I>ll loo( at it a&ain sometime if I ever start wonderin& whether my life has !een worthwhile 2$<# But there are two more fundamental, non$political lessons here that are &eneral to all (inds of desi&n# 5@# ,ften, the most stri(in& and innovative solutions come from reali in& that your concept of the pro!lem was wron&# I had !een tryin& to solve the wron& pro!lem !y continuin& to develop popclient as a com!ined MT.DM;. with all (inds of fun(y local delivery modes# Fetchmail>s desi&n needed to !e rethou&ht from the &round up as a pure MT., a part of the normal SMT1$spea(in& Internet mail path# -hen you hit a wall in development $ when you find yourself hard put to thin( past the next patch $ it>s often time to as( not whether you>ve &ot the ri&ht answer, !ut whether you>re as(in&

the ri&ht =uestion# 1erhaps the pro!lem needs to !e reframed# -ell, I had reframed my pro!lem# Clearly, the ri&ht thin& to do was :5< hac( SMT1 forwardin& support into the &eneric driver, :@< ma(e it the default mode, and :7< eventually throw out all the other delivery modes, especially the deliver$to$file and deliver$to$standard$output options# I hesitated over step 7 for some time, fearin& to upset lon&$time popclient users dependent on the alternate delivery mechanisms# In theory, they could immediately switch to #forward files or their non$sendmail e=uivalents to &et the same effects# In practice the transition mi&ht have !een messy# But when I did it, the !enefits proved hu&e# The cruftiest parts of the driver code vanished# Confi&uration &ot radically simpler $ no more &rovellin& around for the system M;. and user>s mail!ox, no more worries a!out whether the underlyin& ,S supports file loc(in&# .lso, the only way to lose mail vanished# If you specified delivery to a file and the dis( &ot full, your mail &ot lost# This can>t happen with SMT1 forwardin& !ecause your SMT1 listener won>t return ,C unless the messa&e can !e delivered or at least spooled for later delivery# .lso, performance improved :thou&h not so you>d notice it in a sin&le run<# .nother not insi&nificant !enefit of this chan&e was that the manual pa&e &ot a lot simpler# Later, I had to !rin& delivery via a user$specified local M;. !ac( in order to allow handlin& of some o!scure situations involvin& dynamic SLI1# But I found a much simpler way to do it# The moral0 ;on>t hesitate to throw away superannuated features when you can do it without loss of effectiveness# .ntoine de Saint$"xupery :who was an aviator and aircraft desi&ner when he wasn>t !ein& the author of classic children>s !oo(s< said2 57# '1erfection :in desi&n< is achieved not when there is nothin& more to add, !ut rather when there is nothin& more to ta(e away#' -hen your code is &ettin& !oth !etter and simpler, that is when you (now it>s ri&ht# .nd in the process, the fetchmail desi&n ac=uired an identity of its own, different from the ancestral popclient# It was time for the name chan&e# The new desi&n loo(ed much more li(e a dual of sendmail than the old popclient hadG !oth are MT.s, !ut where sendmail pushes then delivers, the new popclient pulls then delivers# So, two months off the !loc(s, I renamed it fetchmail# Fetchmail )rows +p There I was with a neat and innovative desi&n, code that I (new wor(ed well !ecause I used it every day, and a !ur&eonin& !eta list# It &radually dawned on me that I was no lon&er en&a&ed in a trivial personal hac( that mi&ht happen to !e useful to few other people# I had my hands on a pro&ram every hac(er with a +nix !ox and a SLI1D111 mail connection really needs# -ith the SMT1 forwardin& feature, it pulled far enou&h in front of the competition to potentially !ecome a 'cate&ory (iller', one of those classic pro&rams that fills its niche so competently that the alternatives are not %ust discarded !ut almost for&otten# I thin( you can>t really aim or plan for a result li(e this# Oou have to &et pulled into it !y

desi&n ideas so powerful that afterward the results %ust seem inevita!le, natural, even foreordained# The only way to try for ideas li(e that is !y havin& lots of ideas $ or !y havin& the en&ineerin& %ud&ment to ta(e other peoples> &ood ideas !eyond where the ori&inators thou&ht they could &o# .ndrew Tanen!aum had the ori&inal idea to !uild a simple native +nix for the 78?, for use as a teachin& tool# Linus Torvalds pushed the Minix concept further than .ndrew pro!a!ly thou&ht it could &o $ and it &rew into somethin& wonderful# In the same way :thou&h on a smaller scale<, I too( some ideas !y Carl *arris and *arry *ochheiser and pushed them hard# -e were not >ori&inal> in the romantic way people thin( is &enius# But then, most science and en&ineerin& and software development isn>t done !y ori&inal &enius, hac(er mytholo&y to the contrary# The results were pretty heady stuff all the same $ in fact, %ust the (ind of success every hac(er lives for3 .nd they meant I would have to set my standards even hi&her# To ma(e fetchmail as &ood as I now saw it could !e, I>d have to write not %ust for my own needs, !ut also include and support features necessary to others !ut outside my or!it# .nd do that while (eepin& the pro&ram simple and ro!ust# The first and overwhelmin&ly most important feature I wrote after reali in& this was multidrop support $ the a!ility to fetch mail from mail!oxes that had accumulated all mail for a &roup of users, and then route each piece of mail to its individual recipients# I decided to add the multidrop support partly !ecause some users were clamorin& for it, !ut mostly !ecause I thou&ht it would sha(e !u&s out of the sin&le$drop code !y forcin& me to deal with addressin& in full &enerality# .nd so it proved# )ettin& RFC 8@@ parsin& ri&ht too( me a remar(a!ly lon& time, not !ecause any individual piece of it is hard !ut !ecause it involved a pile of interdependent and fussy details# But multidrop addressin& turned out to !e an excellent desi&n decision as well# *ere>s how I (new2 5A# .ny tool should !e useful in the expected way, !ut a truly &reat tool lends itself to uses you never expected# The unexpected use for multi$drop fetchmail is to run mailin& lists with the list (ept, and alias expansion done, on the client side of the SLI1D111 connection# This means someone runnin& a personal machine throu&h an IS1 account can mana&e a mailin& list without continuin& access to the IS1>s alias files# .nother important chan&e demanded !y my !eta testers was support for 8$!it MIM" operation# This was pretty easy to do, !ecause I had !een careful to (eep the code 8$!it clean# /ot !ecause I anticipated the demand for this feature, !ut rather in o!edience to another rule2 5B# -hen writin& &ateway software of any (ind, ta(e pains to distur! the data stream as little as possi!le $ and PneverP throw away information unless the recipient forces you to3 *ad I not o!eyed this rule, 8$!it MIM" support would have !een difficult and !u&&y# .s it was, all I had to do is read RFC 5?B@ and add a trivial !it of header$&eneration lo&ic# Some "uropean users !u&&ed me into addin& an option to limit the num!er of messa&es retrieved per session :so they can control costs from their expensive phone networ(s<# I resisted this for a lon& time, and I>m still not entirely happy a!out it# But if you>re writin& for the world, you have to listen to your customers $ this doesn>t chan&e %ust !ecause they>re not payin& you in money#

. Few More Lessons From Fetchmail Before we &o !ac( to &eneral software$en&ineerin& issues, there are a couple more specific lessons from the fetchmail experience to ponder# The rc file syntax includes optional >noise> (eywords that are entirely i&nored !y the parser# The "n&lish$li(e syntax they allow is considera!ly more reada!le than the traditional terse (eyword$value pairs you &et when you strip them all out# These started out as a late$ni&ht experiment when I noticed how much the rc file declarations were !e&innin& to resem!le an imperative minilan&ua&e# :This is also why I chan&ed the ori&inal popclient Eserver> (eyword to >poll><# It seemed to me that tryin& to ma(e that imperative minilan&ua&e more li(e "n&lish mi&ht ma(e it easier to use# /ow, althou&h I>m a convinced partisan of the 'ma(e it a lan&ua&e' school of desi&n as exemplified !y "macs and *TML and many data!ase en&ines, I am not normally a !i& fan of '"n&lish$li(e' syntaxes# Traditionally pro&rammers have tended to favor control syntaxes that are very precise and compact and have no redundancy at all# This is a cultural le&acy from when computin& resources were expensive, so parsin& sta&es had to !e as cheap and simple as possi!le# "n&lish, with a!out B9Q redundancy, loo(ed li(e a very inappropriate model then# This is not my reason for normally avoidin& "n&lish$li(e syntaxesG I mention it here only to demolish it# -ith cheap cycles and core, terseness should not !e an end in itself# /owadays it>s more important for a lan&ua&e to !e convenient for humans than to !e cheap for the computer# There are, however, &ood reasons to !e wary# ,ne is the complexity cost of the parsin& sta&e $ you don>t want to raise that to the point where it>s a si&nificant source of !u&s and user confusion in itself# .nother is that tryin& to ma(e a lan&ua&e syntax "n&lish$li(e often demands that the '"n&lish' it spea(s !e !ent seriously out of shape, so much so that the superficial resem!lance to natural lan&ua&e is as confusin& as a traditional syntax would have !een# :Oou see this in a lot of so$called 'fourth &eneration' and commercial data!ase$=uery lan&ua&es#< The fetchmail control syntax seems to avoid these pro!lems !ecause the lan&ua&e domain is extremely restricted# It>s nowhere near a &eneral$purpose lan&ua&eG the thin&s it says simply are not very complicated, so there>s little potential for confusion in movin& mentally !etween a tiny su!set of "n&lish and the actual control lan&ua&e# I thin( there may !e a wider lesson here2 5?# -hen your lan&ua&e is nowhere near Turin&$complete, syntactic su&ar can !e your friend# .nother lesson is a!out security !y o!scurity# Some fetchmail users as(ed me to chan&e the software to store passwords encrypted in the rc file, so snoopers wouldn>t !e a!le to casually see them# I didn>t do it, !ecause this doesn>t actually add protection# .nyone who>s ac=uired permissions to read your rc file will !e a!le to run fetchmail as you anyway $ and if it>s your password they>re after, they>d !e a!le to rip the necessary decoder out of the fetchmail code itself to &et it# .ll #fetchmailrc password encryption would have done is &ive a false sense of security to

people who don>t thin( very hard# The &eneral rule here is2 5J# . security system is only as secure as its secret# Beware of pseudo$secrets# /ecessary 1reconditions for the Ba aar Style "arly reviewers and test audiences for this paper consistently raised =uestions a!out the preconditions for successful !a aar$style development, includin& !oth the =ualifications of the pro%ect leader and the state of code at the time one &oes pu!lic and starts to try to !uild a co$ developer community# It>s fairly clear that one cannot code from the &round up in !a aar style# ,ne can test, de!u& and improve in !a aar style, !ut it would !e very hard to ori&inate a pro%ect in !a aar mode# Linus didn>t try it# I didn>t either# Oour nascent developer community needs to have somethin& runna!le and testa!le to play with# -hen you start community$!uildin&, what you need to !e a!le to present is a plausi!le promise# Oour pro&ram doesn>t have to wor( particularly well# It can !e crude, !u&&y, incomplete, and poorly documented# -hat it must not fail to do is convince potential co$developers that it can !e evolved into somethin& really neat in the foreseea!le future# Linux and fetchmail !oth went pu!lic with stron&, attractive !asic desi&ns# Many people thin(in& a!out the !a aar model as I have presented it have correctly considered this critical, then %umped from it to the conclusion that a hi&h de&ree of desi&n intuition and cleverness in the pro%ect leader is indispensa!le# But Linus &ot his desi&n from +nix# I &ot mine initially from the ancestral popclient :thou&h it would later chan&e a &reat deal, much more proportionately spea(in& than has Linux<# So does the leaderDcoordinator for a !a aar$style effort really have to have exceptional desi&n talent, or can he &et !y on levera&in& the desi&n talent of others0 I thin( it is not critical that the coordinator !e a!le to ori&inate desi&ns of exceptional !rilliance, !ut it is a!solutely critical that the coordinator !e a!le to reco&ni e &ood desi&n ideas from others# Both the Linux and fetchmail pro%ects show evidence of this# Linus, while not :as previously discussed< a spectacularly ori&inal desi&ner, has displayed a powerful (nac( for reco&ni in& &ood desi&n and inte&ratin& it into the Linux (ernel# .nd I have already descri!ed how the sin&le most powerful desi&n idea in fetchmail :SMT1 forwardin&< came from some!ody else# "arly audiences of this paper complimented me !y su&&estin& that I am prone to undervalue desi&n ori&inality in !a aar pro%ects !ecause I have a lot of it myself, and therefore ta(e it for &ranted# There may !e some truth to thisG desi&n :as opposed to codin& or de!u&&in&< is certainly my stron&est s(ill# But the pro!lem with !ein& clever and ori&inal in software desi&n is that it &ets to !e a ha!it $ you start reflexively ma(in& thin&s cute and complicated when you should !e (eepin& them ro!ust and simple# I have had pro%ects crash on me !ecause I made this mista(e, !ut I mana&ed not to with fetchmail# So I !elieve the fetchmail pro%ect succeeded partly !ecause I restrained my tendency to !e cleverG this ar&ues :at least< a&ainst desi&n ori&inality !ein& essential for successful !a aar pro%ects# .nd consider Linux# Suppose Linus Torvalds had !een tryin& to pull off fundamental innovations in operatin& system desi&n durin& the developmentG does it seem at all li(ely that the resultin& (ernel would !e as sta!le and successful as what we have0

. certain !ase level of desi&n and codin& s(ill is re=uired, of course, !ut I expect almost any!ody seriously thin(in& of launchin& a !a aar effort will already !e a!ove that minimum# The open$source community>s internal mar(et in reputation exerts su!tle pressure on people not to launch development efforts they>re not competent to follow throu&h on# So far this seems to have wor(ed pretty well# There is another (ind of s(ill not normally associated with software development which I thin( is as important as desi&n cleverness to !a aar pro%ects $ and it may !e more important# . !a aar pro%ect coordinator or leader must have &ood people and communications s(ills# This should !e o!vious# In order to !uild a development community, you need to attract people, interest them in what you>re doin&, and (eep them happy a!out the amount of wor( they>re doin&# Technical si le will &o a lon& way towards accomplishin& this, !ut it>s far from the whole story# The personality you pro%ect matters, too# It is not a coincidence that Linus is a nice &uy who ma(es people li(e him and want to help him# It>s not a coincidence that I>m an ener&etic extrovert who en%oys wor(in& a crowd and has some of the delivery and instincts of a stand$up comic# To ma(e the !a aar model wor(, it helps enormously if you have at least a little s(ill at charmin& people# The Social Context of ,pen$Source Software It is truly written2 the !est hac(s start out as personal solutions to the author>s everyday pro!lems, and spread !ecause the pro!lem turns out to !e typical for a lar&e class of users# This ta(es us !ac( to the matter of rule 5, restated in a perhaps more useful way2 58# To solve an interestin& pro!lem, start !y findin& a pro!lem that is interestin& to you# So it was with Carl *arris and the ancestral popclient, and so with me and fetchmail# But this has !een understood for a lon& time# The interestin& point, the point that the histories of Linux and fetchmail seem to demand we focus on, is the next sta&e $ the evolution of software in the presence of a lar&e and active community of users and co$developers# In The Mythical Man$Month, Fred Broo(s o!served that pro&rammer time is not fun&i!leG addin& developers to a late software pro%ect ma(es it later# *e ar&ued that the complexity and communication costs of a pro%ect rise with the s=uare of the num!er of developers, while wor( done only rises linearly# This claim has since !ecome (nown as 'Broo(s>s Law' and is widely re&arded as a truism# But if Broo(s>s Law were the whole picture, Linux would !e impossi!le# . few years later )erald -ein!er&>s classic The 1sycholo&y ,f Computer 1ro&rammin& supplied what, in hindsi&ht, we can see as a vital correction to Broo(s# In his discussion of 'e&oless pro&rammin&', -ein!er& o!served that in shops where developers are not territorial a!out their code, and encoura&e other people to loo( for !u&s and potential improvements in it, improvement happens dramatically faster than elsewhere# -ein!er&>s choice of terminolo&y has perhaps prevented his analysis from &ainin& the acceptance it deserved $ one has to smile at the thou&ht of descri!in& Internet hac(ers as 'e&oless'# But I thin( his ar&ument loo(s more compellin& today than ever# The history of +nix should have prepared us for what we>re learnin& from Linux :and what I>ve verified experimentally on a smaller scale !y deli!erately copyin& Linus> methods<# That is, that while codin& remains an essentially solitary activity, the really &reat hac(s come from harnessin& the attention and !rainpower of entire communities# The developer who uses only his or her own !rain in a closed pro%ect is &oin& to fall !ehind the developer who (nows how to create

an open, evolutionary context in which !u&$spottin& and improvements &et done !y hundreds of people# But the traditional +nix world was prevented from pushin& this approach to the ultimate !y several factors# ,ne was the le&al contraints of various licenses, trade secrets, and commercial interests# .nother :in hindsi&ht< was that the Internet wasn>t yet &ood enou&h# Before cheap Internet, there were some &eo&raphically compact communities where the culture encoura&ed -ein!er&>s 'e&oless' pro&rammin&, and a developer could easily attract a lot of s(illed (i!it ers and co$developers# Bell La!s, the MIT .I La!, +C Ber(eley $ these !ecame the home of innovations that are le&endary and still potent# Linux was the first pro%ect to ma(e a conscious and successful effort to use the entire world as its talent pool# I don>t thin( it>s a coincidence that the &estation period of Linux coincided with the !irth of the -orld -ide -e!, and that Linux left its infancy durin& the same period in 5667$ 566A that saw the ta(eoff of the IS1 industry and the explosion of mainstream interest in the Internet# Linus was the first person who learned how to play !y the new rules that pervasive Internet made possi!le# -hile cheap Internet was a necessary condition for the Linux model to evolve, I thin( it was not !y itself a sufficient condition# .nother vital factor was the development of a leadership style and set of cooperative customs that could allow developers to attract co$developers and &et maximum levera&e out of the medium# But what is this leadership style and what are these customs0 They cannot !e !ased on power relationships $ and even if they could !e, leadership !y coercion would not produce the results we see# -ein!er& =uotes the auto!io&raphy of the 56th$century Russian anarchist 1yotr .lexeyvich Cropot(in>s 'Memoirs of a Revolutionist' to &ood effect on this su!%ect2 '*avin& !een !rou&ht up in a serf$owner>s family, I entered active life, li(e all youn& men of my time, with a &reat deal of confidence in the necessity of commandin&, orderin&, scoldin&, punishin& and the li(e# But when, at an early sta&e, I had to mana&e serious enterprises and to deal with MfreeN men, and when each mista(e would lead at once to heavy conse=uences, I !e&an to appreciate the difference !etween actin& on the principle of command and discipline and actin& on the principle of common understandin&# The former wor(s admira!ly in a military parade, !ut it is worth nothin& where real life is concerned, and the aim can !e achieved only throu&h the severe effort of many conver&in& wills#' The 'severe effort of many conver&in& wills' is precisely what a pro%ect li(e Linux re=uires $ and the 'principle of command' is effectively impossi!le to apply amon& volunteers in the anarchist>s paradise we call the Internet# To operate and compete effectively, hac(ers who want to lead colla!orative pro%ects have to learn how to recruit and ener&i e effective communities of interest in the mode va&uely su&&ested !y Cropot(in>s 'principle of understandin&'# They must learn to use Linus> Law# "arlier I referred to the ';elphi effect' as a possi!le explanation for Linus> Law# But more powerful analo&ies to adaptive systems in !iolo&y and economics also irresista!ly su&&est themselves# The Linux world !ehaves in many respects li(e a free mar(et or an ecolo&y, a collection of selfish a&ents attemptin& to maximi e utility which in the process produces a self$ correctin& spontaneous order more ela!orate and efficient than any amount of central plannin& could have achieved# *ere, then, is the place to see( the 'principle of understandin&'# The 'utility function' Linux hac(ers are maximi in& is not classically economic, !ut is the intan&i!le of their own e&o satisfaction and reputation amon& other hac(ers# :,ne may call their

motivation 'altruistic', !ut this i&nores the fact that altruism is itself a form of e&o satisfaction for the altruist<# 4oluntary cultures that wor( this way are not actually uncommonG one other in which I have lon& participated is science fiction fandom, which unli(e hac(erdom explicitly reco&ni es 'e&o!oo' :the enhancement of one>s reputation amon& other fans< as the !asic drive !ehind volunteer activity# Linus, !y successfully positionin& himself as the &ate(eeper of a pro%ect in which the development is mostly done !y others, and nurturin& interest in the pro%ect until it !ecame self$ sustainin&, has shown an acute &rasp of Cropot(in>s 'principle of shared understandin&'# This =uasi$economic view of the Linux world ena!les us to see how that understandin& is applied# -e may view Linus> method as a way to create an efficient mar(et in 'e&o!oo' $ to connect the selfishness of individual hac(ers as firmly as possi!le to difficult ends that can only !e achieved !y sustained cooperation# -ith the fetchmail pro%ect I have shown :al!eit on a smaller scale< that his methods can !e duplicated with &ood results# 1erhaps I have even done it a !it more consciously and systematically than he# Many people :especially those who politically distrust free mar(ets< would expect a culture of self$directed e&oists to !e fra&mented, territorial, wasteful, secretive, and hostile# But this expectation is clearly falsified !y :to &ive %ust one example< the stunnin& variety, =uality and depth of Linux documentation# It is a hallowed &iven that pro&rammers hate documentin&G how is it, then, that Linux hac(ers &enerate so much of it0 "vidently Linux>s free mar(et in e&o!oo wor(s !etter to produce virtuous, other$directed !ehavior than the massively$funded documentation shops of commercial software producers# Both the fetchmail and Linux (ernel pro%ects show that !y properly rewardin& the e&os of many other hac(ers, a stron& developerDcoordinator can use the Internet to capture the !enefits of havin& lots of co$developers without havin& a pro%ect collapse into a chaotic mess# So to Broo(s>s Law I counter$propose the followin&2 562 1rovided the development coordinator has a medium at least as &ood as the Internet, and (nows how to lead without coercion, many heads are inevita!ly !etter than one# I thin( the future of open$source software will increasin&ly !elon& to people who (now how to play Linus> &ame, people who leave !ehind the cathedral and em!race the !a aar# This is not to say that individual vision and !rilliance will no lon&er matterG rather, I thin( that the cuttin& ed&e of open$source software will !elon& to people who start from individual vision and !rilliance, then amplify it throu&h the effective construction of voluntary communities of interest# .nd perhaps not only the future of open$source software# /o commercial developer can match the pool of talent the Linux community can !rin& to !ear on a pro!lem# 4ery few could afford even to hire the more than two hundred people who have contri!uted to fetchmail3 1erhaps in the end the open$source culture will triumph not !ecause cooperation is morally ri&ht or software 'hoardin&' is morally wron& :assumin& you !elieve the latter, which neither Linus nor I do<, !ut simply !ecause the commercial world cannot win an evolutionary arms race with open$source communities that can put orders of ma&nitude more s(illed time into a pro!lem# .c(nowled&ements This paper was improved !y conversations with a lar&e num!er of people who helped de!u& it# 1articular than(s to Heff ;ut(y Kdut(yFwam#umd#eduL, who su&&ested the 'de!u&&in& is paralleli a!le' formulation, and helped developed the analysis that proceeds from it# .lso to /ancy Le!ovit KnancylFuniverse#di&ex#netL for her su&&estion that I emulate -ein!er& !y

=uotin& Cropot(in# 1erceptive criticisms also came from Hoan "slin&er Kwom!atF(iliman%aro#en&r#s&i#comL and Marty Fran KmartyFnet$lin(#netL of the )eneral Technics list# 1aul "&&ert Ke&&ertFtwinsun#comL noticed the conflict !etween )1L and the !a aar model# I>m &rateful to the mem!ers of 1L+), the 1hiladelphia Linux +ser>s &roup, for providin& the first test audience for the first pu!lic version of this paper# Finally, Linus Torvalds>s comments were helpful and his early endorsement very encoura&in&# For Further Readin& I =uoted several !its from Frederic( 1# Broo(s>s classic The Mythical Man$Month !ecause, in many respects, his insi&hts have yet to !e improved upon# I heartily recommend the @Bth .nniversary addition from .ddison$-esley :ISB/ 9$@95$87B6B$6<, which adds his 568? '/o Silver Bullet' paper# The new edition is wrapped up !y an invalua!le @9$years$later retrospective in which Broo(s forthri&htly admits to the few %ud&ements in the ori&inal text which have not stood the test of time# I first read the retrospective after this paper was su!stantially complete, and was surprised to discover that Broo(s attri!utes !a aar$li(e practices to Microsoft3 )erald 1# -ein!er&>s The 1sycholo&y ,f Computer 1ro&rammin& :/ew Oor(2 4an /ostrand Reinhold, 56J5< introduced the rather unfortunately$la!eled concept of 'e&oless pro&rammin&'# -hile he was nowhere near the first person to reali e the futility of the 'principle of command', he was pro!a!ly the first to reco&ni e and ar&ue the point in particular connection with software development# Richard 1# )a!riel, contemplatin& the +nix culture of the pre$Linux era, reluctantly ar&ued for the superiority of a primitive !a aar$li(e model in his 5686 paper Lisp2 )ood /ews, Bad /ews, and *ow To -in Bi&# Thou&h dated in some respects, this essay is still ri&htly cele!rated amon& Lisp fans :includin& me<# . correspondent reminded me that the section titled '-orse Is Better' reads almost as an anticipation of Linux# The paper is accessi!le on the -orld -ide -e! at http2DDalp ha$!its#ai#mit#eduDarticlesD&ood$newsD&ood$news#html# ;e Marco and Lister>s 1eopleware2 1roductive 1ro%ects and Teams :/ew Oor(2 ;orset *ouse, 568JG ISB/ 9$67@?77$9B$?< is an underappreciated &em which I was deli&hted to see Fred Broo(s cite in his retrospective# -hile little of what the authors have to say is directly applica!le to the Linux or free$software communities, the authors> insi&ht into the conditions necessary for creative wor( is acute and worthwhile for anyone attemptin& to import some of the !a aar model>s virtues into a more commercial context# Finally, I must admit that I very nearly called this paper 'The Cathedral and the .&ora', the latter term !ein& the )ree( for an open mar(et or pu!lic meetin& place# The seminal 'a&oric systems' papers !y Mar( Miller and "ric ;rexler, !y descri!in& the emer&ent properties of mar(et$li(e computational ecolo&ies, helped prepare me to thin( clearly a!out analo&ous phenomena in the free$software culture when Linux ru!!ed my nose in them five years later# These papers are availa!le on the -e! at http2DDwww#a&orics#comDa&orpapers# html# "pilo&2 /etscape "m!races the Ba aar3 It>s a stran&e feelin& to reali e you>re helpin& ma(e history ### # ,n Hanuary @@ 5668, approximately seven months after I first pu!lished this paper, /etscape Communications, Inc# announced plans to &ive away the source for /etscape Communicator# I had had no clue this was &oin& to happen !efore the day of the announcement#

"ric *ahn, "xecutive 4ice 1resident and Chief Technolo&y ,fficer at /etscape, wrote me shortly afterwards as follows2 ',n !ehalf of everyone at /etscape, I want to than( you for helpin& us &et to this point in the first place# Oour thin(in& and writin&s were fundamental inspirations to our decision#' The followin& wee( I flew out to Silicon 4alley at /etscape>s invitation for a day$lon& strate&y conference :on Fe! A 5668< with some of their top executives and technical people# -e desi&ned /etscape>s source$release strate&y and license to&ether, and laid some more plans that we hope will eventually have far$reachin& and positive impacts on the open$source community# .s I write, it is a !it too soon to !e more specificG !ut details should !e forthcomin& within wee(s# /etscape is a!out to provide us with a lar&e$scale, real$world test of the !a aar model in the commercial world# The open$source culture now faces a dan&erG if /etscape>s execution doesn>t wor(, the open$source concept may !e so discredited that the commercial world won>t touch it a&ain for another decade# ,n the other hand, this is also a spectacular opportunity# Initial reaction to the move on -all Street and elsewhere has !een cautiously positive# -e>re !ein& &iven a chance to prove ourselves, too# If /etscape re&ains su!stantial mar(et share throu&h this move, it %ust may set off a lon&$overdue revolution in the computer industry# The next year should !e a very instructive and interestin& time# 4ersion and Chan&e *istory RId2 cathedral$!a aar#s&ml,v 5#77 5668D9@D5? 582@@2A@ esr "xp R I &ave 5#5? at the Linux Con&ress, May @5 566J# I added the !i!lio&raphy Huly J 566J in 5#@9# I added the 1erl Conference anecdote /ovem!er 58 566J in 5#@J# I chan&ed 'free software' to 'open source' Fe!ruary 6 5668 in 5#@6# I added '"pilo&2 /etscape "m!races the Ba aar3' on Fe!ruary 59 5668 in 5#75 ,ther revision levels incorporate minor editorial and mar(up fixes# .!out the .uthor "ric S# Raymond is Co$Founder and Technical ;irector of Chester County InterLin( which provides free Internet access to the residents of Chester County, 1ennsylvania# *is home pa&e is located at http2DDsa&an#earthspace#netDSesrD "$mail2 esrFthyrsus#com Contents Index Copyri&ht T 5668, U V W s X $ m Y Z d F [