Netscape Documentary

Tape 49

Its not its not brain surgery but theres lot of it

OK

And still have to hire DJs

Uh just sos you know if youre interested think the NSA has copies of the

interesting files in Libnet So that if they have any really really serious attitude they

could us Urn
yell at that has been two days ago and havent heard anything

back So Im presuming that theyre giving it some attention Its going to be really

interesting how this whole crypto thing plays out Weinsteins already telling like

zillion people Im sorry cant answer your technical questions That would be

illegal Im sorry cant cant tell

you the difference between bit and byte

that would be technical information OK Lets be done

...It doesnt sound to me like theres theres too much time in in this
any

scenario to do any type of tests or checks its just lets just do

or like it

Its been happening

Well its already been Beta tested

What about the FTP servers think Jamies right mean

think were done think were done now

LT We could probably were really close to rolling it out

...if this were .. community why are we waiting till

Monday for these bits

How many bits

Weve got the were done

FTP is changed Since time

long ago

page 3\
Netscape Documentary
Tape 49

Since the old days OK

Now they have these big machines they have an unbelievable amount of memory

so all the files that live on disks that have to the

normally paging through

to it in in
operating system screw Theyre theyre memory not on disks

And these machines are going to be housed in the in the uh IS server room

Well were going to have one machine

One machine

...and they say they tell me that the FTP machines we use these days can handle

5000 concurrent users And but they never actually get up to that number of

users because the T3 saturates first

So people just have to wait

T3s are big fat pipes

Big fat pipes and if

youre saturating T3s enough

bet that happens though

The question is will the people be able to get mean these are these are what

10 times the size the download size that

theyre theyre

And one tenth the population

And hopefully yeah hopefully there are lot less people who are interested in

them mean the thing that on that be

is first day theres going to bunch of idiots

downloading this source who dont have compilers You know theyre just not

going to understand and theyre to you know

going use up our bandwidth

anyway...So mean the

thing Im worried about is is getting the stuff to the

page 024
Netscape Documentary
Tape 49

mirrors early enough

LT How many meg is it Did it end up being

dont know

Thought we would ask you that question

Yeah youre one of the tarballs you tell me

LT .1 never looked at the size should know this

Well Bob was pretty happy when he did it Did you see his change

LT His didnt see his change saw that he built it

...Bob Lisbon had change

Whatd he do

Guess

Yup

He made the Bob-a-gator

Oh come on Try harder

About Bob

He did huh

Its so easy even the do it

vice-president can

LT Oh he did he put

Yeah He submitted the line of code asked Michael to it in

get

LT Did he really

Yeah He asked He didnt tell

Thats great But thats the rule You have to actually modify the code yourself.

page
Netscape Documentary
Tape 49

Yeah he did he did it he figured out how it worked wrote the line of code the

whole bit

think Mozilla.org is going to need to formulate policy about that

Oh yes we are

LT You know think we ought to think its kind of like knighthood you got to

actually you work up and you get knighted by your..

Thats interesting

You know think its Do you

are you golden

We could raffle one off at the party

Thats great idea

LT think we ought mean ..thats funny Thats funny

That that is good that is good Thats probably even better prize than giving

them CD of the source which was going to be the other you know

Go for it Jamie raffle that off...Easter

eggs .are not going to endure..

No theyre not Well put it

in and itll last however long it lasts

Li What we ought to do though is have it when you to the CVS

get rights server

mean thats when..

think the rule doesnt change Module owners get to ..

LT But somebody owns the module on the Net right

Rathole need to talk to Terry so were done OK

OK have bunch of names that can come yell at you about1-

Do you want me to
stay
for this either of you

page4
Netscape Documentary
Tape 49

Go away mean wish was going away

OK so whats the deal now

CUT

Whats Lloyds job

Yeah IS well one of the things Lloyd has been really good at in the past

likeShe wrote Tinderbox and Bonsai which are these aPrHTML pages that monitor

our build process and all that stuff and and so hes seeing his job in Mozilla.org as

making sure that the

processes are in place so that people can their work done and
get

dealing with CVS and and checking in files and and all that sort of thing Urn think one

of the reasons that that hard time

Lloyds having pretty is that...1 see client eng and

Mozilla.org as being very separate organizations think they have to be And Lloyd is

like trying to do two jobs Hes trying to look out for both of their interests at the same

time Um which is
great mean someone needs to do that but think the distinctions

between the two have to be really really clear And Im not sure hes them
like keeping as

clear as he should be but urn you know it is really hard job no about it
question

What are the distinctions between Netscape and Mozilla.org

Well...the interest of Mozilla.org is secvintCall the who want do something

developers to

with code which and people outside

the includes Netscape And the interest of client eng

is shipping the Netscape Communicator product And these two goals have lot ins

common but theyre not the same and theyre going to have different schedules and

theyre going to have different

working on them And theyre
people going to come into

conflict sometimes

pages
L4.-7
 Netscape Documentary
Tape 49

Describe the challenges of having both Nscape Mo.org making changes

to the code at the same time

Well thats really hard problem Thats you know one of the things that Lloyd is

worrying about is how do we make it not be just total disaster in the past people at

Netscape have as the companys grown its gotten gotten even hairier but its its

been hard times mean things like Tinderbox

to keep the thats
thing building at all why

are at of have been has been to

necessary all and\e sort way people operating

think it
basically works check it in see if it broke then fix it And you know that just

doesnt scale when youve got you know thousand one thing at at

develojne
another guess it was JG at at his previous company youve seen the buy sign And

they
had they had that up and theyll show it and it would splash message

whenever someone broke the build saying so and so broke the build so the the joke

weve been making is some guy in Idaho broke the build You know we cant we cant

live like have have

that Mozilla.org is going to to stricter rules if
you break the build

your change gets backed out and if

you do this regularly you lose your permission to

Which is not rule that client eng has ever operated under before have
cheeks People

done it in in previous lives but its not how weve done here so the works
it
way that

has to change dramatically And Lloydss trying to figure out how to do it

So client eng is going to have to the

live by same rules as outside

developers

f04ctt
J- really
22S
strongly believe that if
Netscape gets special exemptions of any kind because

theyre Netscape then will fail Not for for of reasons

Mozilla.org couple first of all it

page
vt Z/ 2i\
Netscape Documentary
Tape 49

looks bad and no one

just will trust us and no one will believe were trying to do what we

say were trying to do theyll think theyre being conned Urn second if the only way

Netscape can get their job done is by being Netscape and having this magic relationship

were doing something wrong because were were going to our bigger
fail at goal

which is making it
possible for other people to do what Netscape has been
doin-1 5Z
This should be an to the culture in
interesting shift developing general

Yeah...its different world But mean at some level everyones always competing

with people outside right thats if you didnt have competition youd be Microsoft Uh

its its little different in that not competing

people are via secrets They have to compete

via innovation instead And if

you ask me that healthier
way to do things
fl

No one ever talks about Microsoft here why is that

\\ 7\
Well Im kind of surprised that you say that because my impression going back year

and half or two years has been that the company is reactive to Microsoft And
just utterly

think thats disastrously bad And largely responsible for lot of the problems weve

been havingr though thinking about it think youre right and and its because the

free source thing is just you know so out there that its its not something Microsoft is

already domg...no one believes that Microsoft

really is going to give away their source

right So it doesnt even come They do come

up definitely up like Microsoft comes up

lot in license discussions like oh well pre we giving our crown

iust away jewels to

Microsoft you hear people ask that question in various forms all the timet ii Lj /i
Do you think that shipping the free source will help in
you competition

with Microsoft or is it more to the culture

change development

Ci
page
 Netscape Documentary
Tape 49

wouldnt say its an unnecessary distraction you ignore whats going on in the world at

your peril but being

focus on making
overly

good product
focused

rather
on it is

than doing
bad.. is

slightly
-__
not healthy

better
you

than the
know

guy
you

next door
need to

-7
.---------------
you know Urn think another reason something Ive noticed about the environment

around here since we started doing this free source is is think there are lot of people

who are just kind of shell-shocked and and theyre suggestible think thats large part

of the reason that that weve gotten away with as much as we have in Mozilla.org is that
-C-

oh look the rules changed and dont what to do so Im just going to smile and nod

you know Not entirely

abad ma5b1iis not the best reason think the right thing is

happening anyway -a1 have my reasons for why think the free source way of doing

things is you know the path to success or whatever Im sure other people have

different take on thatIdont Idont know what the motivations of you know upper

management were when they decided we would do this -k

ci

How does this affect client eng

.ct
LP% ell things are little different right now for for two reasons First of all theres lot of

like kind of weird little structural on as of the sanitization

changes going part process

which would not normally be happening every day in in the steady state And the other

thing is that were on this externally imposed really tight schedule So that thats going to

make things bit more chaotic now then they would have been week month
ago or

from now or but things are to be have to be different in the future

whatever.4J4 going

and of the reason for that that when

part is client eng was really small organization it

worked well Client eng has gotten

pretty really big and its
pretty much trying to operate

page
H4 17
iL4

-r.1
I-
Netscape Documentary
Tape 49

in the same way and its not scaling well and its going its were were on the

brink of getting lot bigger still So something needs to be fixed And part of what

Lloyds doing

Is in software one of the hurdles

scaling development greatest to

innovation

Yeah Its definitely one of the hardest parts dont know if theres dont
something

know if it embodies some fundamental limit to the complexity of software or anything like

that but you know in our specific case its problem we have and problem that could

get lot worse if were not really careful about it Were to be involving
certainly going

lot more people So in this situation its really important Im not sure what think about

thatin general

QL4
Well Ive got lot of running around to do to to follow on the
get people through

conmiitments theyve made of documents theyre going to write and machines that are

be and make
going to up that sort of thing sure all that stuff is actually ready to go in time

umjcm not that clear on what exactly

all is going to happen on the 31st yet was trying to

get some answers to that in in the meeting hope that what happens is late Friday

night sit at home and make all the changes and the documents and
get ready to go

rth in the
morning Iget up and the switch and the site live and then probably
Ifljp oes

whats going to happen is the entire universe to download and

is going try to everything

our is going to melt down there are going to be an awful lot of comments about_.a

about the fact that people cant download and then

anything couple hours later there are

page9
Netscape Documentary
Tape 49

going to be lots of comments about people having

been unable to build it And ..theres

going to be lot of noise and lot of chaos and wont be atrain wreck
hopefully it

When are you going to flip the switch

Tuesday morning

Midnight between Monday and Tuesday hope

Will you do it from home

yeah probably But well dont know it

depends they might want me to be

like in meetings like when theyre calling the

press and stuff they did last time when the

press release went out4 so it the the kind of work have to do leading up to to

flipping the switch is much easier for me to do home because know my machines
at
you

there and when Im accessing my machine remotely its really slow So if cane if

everyones ready to go by midnight on Monday cane can set everything up and

then Ill be able to flip the switch from here without any trouble to 2_S

And thats when the files go from client eng to Mozilla.org FTP site

Yeah itll
probably be some combination of of physically from one machine
moving files

to another and rem on access

just dropping restrictions to machine you know

unlocking the door

Well want to be there for that

Yeah And and youll be the party

at too assume

CUT new interview Lloyd Tab starts here

Urn Mozilla.org is going to be accepting changes from the outside world and the

page 10
CI Lc
Netscape Documentary
Tape 49

the problem is the rate of change urn that urn you take various
at points in project can

be different so urn at the beginning of project rate of change is high and as

your very

you get towards the end its you taper down and you urn you you slow down

the...rate of change Jim was concerned that with Mozilla.org and Netscape having

two different ideas about what that rate of change should be could create real problern

and would would force Netscape to be frorn the tip of the tree for of
away long period

time Urn its my goal that that doesnt happen think would be bad thing
urn that for

both because believe that who with Mozilla.org

parties people are working want

Netscapes in
help shipping something and so for example if urn Netscape ships the 5.0

uh thats very stable urn...there needs to be sornething that correlates to that in

Mozilla.org also And so its difference of of premises have different assumption

than Jim does But was trying to explain mine my prernise

Whats the difference between branch and the tip of the tree

So whenever fundamentally when developer works on something alone what they do is

they take copy of the tree they make modifications to it and when those

modifications are tested well enough...theres of comfort

certain level what they do is

they commit those changes back into the tree Or the tip of the tree And what
generally

ends up happening is that they they verify that those changes went in OK by looking at

the various tools we have like when

Tinderbox group of engineers wants to
go off

and work on feature and that feature not be in state state thats done
rnay

enough for if youre working on if youre working alone then you can always work

on in
your own copy of the tree and then conunit it but if youre in
group sometirnes

page 11
o-j5
73 oO
Netscape Documentary
Tape 49

that group needs to

go off and work together and so what they do is they go off and they

work on branch together and what this is is shared copy of the tree They go off and

they each commit into their of the tree and when theyre done. calIecanding the
copy .its

branch they take that entire branch and bnng that branch back into the of the tree and it
tip

can be looked at as if single developers changes are coming in at 2-

2-\

\SS
Is landing

...Generally
brnch

groups
\S
are
riskier

as careful
than

or more
if one

careful
person makes

than individuals
change

so its not bustage that

you worry about What you worry about when you have somebody off on branch

is of goals And what Jim was worried

misalignment thats about So for example uh

they have feature theyre off on branch this is great new branch new feature
great

everybody should want it but the goal of Mozilla.org is to make something stable And

so the branch is forced to be orphaned out there for period of time until Mozilla.org

is...in position to accept new features Andifl and only then can that branch land

because if if the nature of it.. is complex enough and...possibly destabilizing enough

you would set back your stabilization date And so but that happens on just any on any

project So ..theres time for features and theres time for stabilization And ..youve

be careful
got to about when you accept things
01.15 1F t59

So if Mozilla.org doesnt wilt slow the

accept feature it
development

process down is that the problem Or

Well every feature has money riding on it OK So mean its. .money-driven right So

theres customer somewhere who wants this feature badly and theres
very marketing

person who very much who very badly wants this feature into released So
product

page 12

1-1 19.L451
Netscape Documentary
Tape 49

what ends up happening is that sometimes uh Netscape will

say OK were. .going into

stabilization mode Mozilla.org is going into stabilization mode and believe its
my

belief that
Mozilla.org will leave stabilization mode before Netscape does So Mozilla.org

will
say OK were done being stable now were coming into accepting new features

Netscape might want that feature in their release so they might actually drop that feature

what theyll do is theyll branch at that stabilization point when ahead

Mozilla.org goes

4I% and j$theyll drop that feature into the stable into Netscapes stable copy But it

wont be in Mozilla.org until after

Netscape uh finishes shipping their 5.0 And Jim was

actually worried about if the reverse happened if chose stabilization and

Netscape

Mozilla.org wasnt interested in ..stabilization at that And. .he would be doomed

point

And um because he would have hard time getting ship on and he

something stable to

might be forced to go back to prior stable version from Mozilla.org and then cull pieces

out of it that he wanted He and thats thats what hes considering disaster scenario

Um believe that if we can keep our development cycles short enough in Mozilla.org that

wont be problem So we need to shorter than Netscape

keep development cycles and

well be OK But were making this

up as we go There is what what ends up

happening is that were going to figure out what works for and work that
everybody way
it-iOir
So Mozilla.org has to be faster than in
always Netscape order for this to

work
L-1 i5
...highŁr than Netscapes or higher And think were
frequency equal to so...I
prepared

forthat ut
3t
Why is it so hard to create good software

page 13
5o
 ---
-7
Netscape Documentary
Tape 49

its hard...software just hard to do well mean

is
traditionally Right urn guess

that the the large part of it the main problem with doing software in the that
large is you

over each other rnean thats the problem You too

essentially really get many people

in
room you trip over each other And urn so figuring out ways to keep the path clear

keep its also like any civilization if you have you have to have rules and if people

dont if there arent enough rules you end up with chaos You can have without
family

very many rules and it works pretty well you you start getting to be larger

organization you have to have rules and and the rules have to make sense because if

you have bad set of rules what you have is people doing the
wrong thing it forces

people into doing the

wrong thing and if
you have good set of rules it it clears the

way. .for success So urn actually figuring out the the right set of rules is really

important

hJ ii
fb

page 14