Go generate: A Proposal

Introduction

ThegobuildcommandautomatestheconstructionofGoprogramsbutsometimes
preliminaryprocessingisrequired,processingthatgobuilddoesnotsupport.Motivating
examplesinclude:

yacc:generating.gofilesfromyaccgrammar(.y)files
protobufs:generating.pb.gofilesfromprotocolbufferdefinition(.proto)files
Unicode:generatingtablesfromUnicodeData.txt
HTML:embedding.htmlfilesintoGosourcecode
bindata:translatingbinaryfilessuchasJPEGsintobytearraysinGosource

Thereareotherprocessingstepsonecanimagine:

stringmethods:generatingString() stringmethodsfortypesusedasenumerated
constants
macros:generatingcustomizedimplementationsgivengeneralizedpackages,suchas
sort.Intsfromints

Thisproposaloffersadesignforsmoothautomationofsuchprocessing.

Non-goal

ItisnotagoalofthisproposaltobuildageneralizedbuildsystemliketheUnixmake(1)utility.
Wedeliberatelyavoiddoinganydependencyanalysis.Thetooldoeswhatisaskedofit,
nothingmore.

Itishoped,however,thatitmayreplacemanyexistingusesofmake(1)intheGorepoat
least.

Design

Therearetwobasicelements,anewsubcommandforthegocommand,calledgogenerate,
anddirectivesinsideGosourcefilesthatcontrolgeneration.

Whengogenerateruns,itscansGosourcefileslookingforthosedirectives,andforeach
oneexecutesageneratorthattypicallycreatesanewGosourcefile.Thegogeneratetool

alsosetsthebuildtag"generate"sothatfilesmaybeexaminedbygogeneratebutignored
duringbuild.

Theusageis:

go generate [-run regexp] [file.go...|packagePath...]

(Plustheusual-x,-n,-vand-tagsoptions.)Ifpackagesarenamed,eachGosourcefilein
eachpackageisscannedforgeneratordirectives,andforeachdirective,thespecified
generatorisruniffilesarenamed,theymustbeGosourcefilesandgenerationhappensonly
fordirectivesinthosefiles.Givennoarguments,generatorprocessingisappliedtotheGo
sourcefilesinthecurrentdirectory.

The-runflagtakesaregularexpression,analogoustothatofthegotestsubcommand,that
restrictsgenerationtothosedirectiveswhosecommand(seebelow)matchestheregular
expression.

GeneratordirectivesmayappearanywhereintheGosourcefileandareprocessed
sequentially(noparallelism)insourceorderaspresentedtothetool.Eachdirectiveisa//
commentbeginningaline,withsyntax

//go:generatecommandarg...

wherecommandisthegenerator(suchasyacc)toberun,correspondingtoanexecutablefile
thatcanberunlocallyitmusteitherbeintheshellpath(gofmt)orfullyqualified
(/usr/you/bin/mytool)andisruninthepackagedirectory.

Theargumentsarespaceseparatedtokens(ordoublequotedstrings)passedtothe
generatorasindividualargumentswhenitisrun.Shelllikevariableexpansionisavailablefor
anyenvironmentvariablessuchas$HOME.Also,thespecialvariable$GOFILEreferstothe
nameofthefilecontainingthedirective.(Wemayneedotherspecialvariablessuchas
$GOPACKAGE.Whenthegeneratorisrun,thesearealsoprovidedintheshellenvironment.)No
otherspecialprocessing,suchasglobbing,isprovided.

Nofurthergeneratorsarerunifanygeneratorreturnsanerrorexitstatus.

Asanexample,saywehaveapackage"my/own/gopher"thatincludesayaccgrammarin
filegopher.y.Insidemain.go(notgopher.y)weplacethedirective

//go:generate yacc -o gopher.go gopher.y

(Moreaboutwhat"yacc"meansinthenextsection.)Wheneverweneedtoupdatethe
generatedfile,wegivetheshellcommand,


% go generate my/own/gopher

or,ifwearealreadyinthesourcedirectory,

% go generate

Ifwewanttomakesurethatonlytheyaccgeneratorisrun,weexecute

% go generate -run yacc

Ifwehavefixedabuginyaccandwanttoupdateallyaccgeneratedfilesinourtree,wecan
run

% go generate -run yacc all

Thetypicalcycleforapackageauthordevelopingsoftwarethatusesgogenerateis

% edit
% go generate
% go test

andoncethingsaresettled,theauthorcommitsthegeneratedfilestothesourcerepository,
sothattheyareavailabletoclientsthatusegoget:

% git add *.go

% git commit

Commands

Theyaccprogramisofcoursenotthestandardversion,butisaccessedfromthecommand
lineby

go tool yacc args

Tomakeiteasytousetoolslikeyaccthatarenotinstalledin$PATH,havecomplexaccess
methods,orbenefitfromextraflagsorotherwrapping,thereisaspecialdirectivethatdefines
ashorthandforacommand.Itisago:generatedirectivefollowedbythekeyword/flag
"command"andwhichgeneratoritdefinestherestofthelineissubstitutedforthecommand
namewhenthegeneratorisrun.Thustodefine"yacc"asageneratorcommandweaccess
normallybyrunning"gotoolyacc",wefirstwritethedirective

//go:generate -command yacc go tool yacc

andthenallothergeneratordirectivesusingyaccthatfollowinthatfile(only)canbewritten
asabove:

//go:generate yacc -o gopher.go gopher.y

whichwillbetranslatedto

go tool yacc -o gopher.go gopher.y

whenrun.

Discussion

Thisdesignisunusualbutisdrivenbyseveralmotivatingprinciples.

First,go generateisintendedtoberunbytheauthorofapackage,nottheclientofit.The
authorofthepackagegeneratestherequiredGofilesandincludestheminthepackagethe
clientdoesaregulargogetorgobuild.Generationthroughgogenerateisnotpartofthe
build,justatoolforpackageauthors.Thisavoidscomplicatingthedependencyanalysisdone
byGobuild.

Second,gobuildshouldnevercausegenerationtohappenautomaticallybytheclientofthe
package.Generatorsshouldrunonlywhenexplicitlyrequested.

Third,theauthorofthepackageshouldhavegreatfreedominwhatgeneratortouse(thatisa
keygoaloftheproposal),buttheclientmightnothavethatprocessoravailable.Asasimple
example,ifitisashellscript,itwillnotrunonWindows.Itisimportantthatautomated
generationnotbreakclientsbutbeinvisibletothem,whichisanotherreasonitshouldberun
onlybytheauthorofthepackage.

Finally,itmustfitwellwiththeexistinggocommand,whichmeansitappliesonlytoGo
sourcefilesandpackages.ThisiswhythedirectivesareinGofilesbutnot,forexample,in
the.yfileholdingayaccgrammar.

Onecanimaginescenarioswheretheauthorwishestheclienttorunthegenerator,butinsuchcasesthe
authormustguaranteethattheclienthasthegeneratoravailable.Regardless,gogetwillnotautomatethe
runningoftheprocessor,sofurtherinstallationinstructionswillneedtobeprovidedbytheauthor.

Examples

Herearesomehypotheticalworkedexamples.Therearecountlessmorepossibilities.

String methods

WewishtogenerateaStringmethodforanamedconstanttype.Wewriteatool,say
strmeth,thatreadsadefinitionforasingleconstanttypeandvaluesandprintsacomplete
Gosourcefilecontainingamethoddefinitionforthattype.

InourGosourcefile,main.go,wedecorateeachconstantdeclarationlikethis(withsome
blanklinesinterposedsothegeneratordirectivedoesnotappearinthedoccomment):

//go:generate strmeth Day -o day_string.go $GOFILE

// Day represents the day of the week
type Day int
const (
Sunday Day = iota
Monday

)
ThestrmethgeneratorparsestheGosourcetofindthedefinitionoftheDaytypeandits
constants,andwritesoutaString()stringmethodforthattype.Fortheuser,generation
ofthestringmethodistrivial:justrungo generate.

Yacc

Asoutlinedabove,wedefineacustomcommand

//go:generate -command yacc go tool yacc

andthenanywhereinmain.go(say)wewrite

//go:generate yacc -o foo.go foo.y

Protocol buffers

Theprocessisthesameaswithyacc.Insidemain.go,wewrite,foreachprotocolbufferfile
wehave,alinelike

//go:generate protoc -go_out=. file.proto

Becauseofthewayprotocworks,wecouldgeneratemultipleprotodefinitionsintoasingle
.pb.gofilelikethis:

//go:generate protoc -go_out=. file1.proto file2.proto

Sincenoglobbingisprovided,onecannotsay*.proto,butthisisintentional,forsimplicity
andclarityofdependency.

Caveat:Theprotocprogrammustberunattherootofthesourcetreewewouldneedto
provideacdoptiontoitorwrapitsomehow.

Binary data

AtoolthatconvertsbinaryfilesintobytearraysthatcanbecompiledintoGobinarieswould
worksimilarly.Again,intheGosourcewewritesomethinglike

//go:generate bindata -o jpegs.go pic1.jpg pic2.jpg pic3.jpg

ThisisalsodemonstratesanotherreasontheannotationsareinGosource:thereisnoeasy
waytoinjectthemintobinaryfiles.

Sort

Onecouldimagineavariantsortimplementationthatallowsonetospecifyconcretetypes
thathavecustomsorters,justbyautomaticrewritingofmacrolikesortdefinition.Todothis,
wewriteasort.gofilethatcontainsacompleteimplementationofsortonanexplicitbut
undefinedtypespelled,say,TYPE.Inthatfileweprovideabuildtagsoitisnevercompiled
(TYPEisnotdefined,soitwon'tcompile)butisprocessedbygogenerate:

// +build generate

Thenwewriteangeneratordirectiveforeachtypeforwhichwewantacustomsort:

//go:generate rename TYPE=int

//go:generate rename TYPE=strings

orperhaps

//go:generate rename TYPE=int TYPE=strings

Therenameprocessorwouldbeasimplewrappingofgofmt-r,perhapswrittenasashell
script.

Therearemanymorepossibilities,anditisagoalofthisproposaltoencourage
experimentationwithprebuildtimecodegeneration.