Metaprogramming Go Weng Wei, Nov 2014

About Me

• Weng Wei, Chinese in Singapore

• Full-stack developer

• .net -> python -> go

• Passionate about code & life

Metaprogramming (MP)

• “writing of computer programs with the ability to treat programs as their data...program could be designed to read, generate, analyse and/or transform other programs”

• allows programmers to minimize the number of lines of code to express a solution => better produtivity

MP is everywhere

• Compiler is the most common MP tool, it compile high level code to lower one

• MP for manipulating other code is also common, for example: C Macro

C Macro

#define list_for_each(pos, head) \ for (pos = (head)->next; pos != (head); pos = pos->next)

Metaprogramming Go？• No macro

• No generic

• Reflection in Go is awful

• “ gopher has been hard working these years… to solve problems don’t exist in other languages……” - @laiyonghao

How hard working gopher solve generic with py?

• StringSet& IntSet has exact code structure, only types are different

• Classic usage of generic

Simple & Effective

auto generate• MakeFile？

• Real-time file monitor: github.com/gorakhargosh/watchdog

How about “ORM”?

We need DSL:thrift

Same struct definition

• CRUD

• Full-text search

• User Permission

• RPC

Lots of boilerplate code for obj passing, value

assign, display etc

• Generate them all!

• Generated code won’t have typo

Implementation

Struct Parsing

• ptsd: github.com/wickman/ptsd

• thrift lexer/parser using ply

• ply: www.dabeaz.com/ply/

• An implementation of lex and yacc parsing tools for Python

code-gen template• www.cheetahtemplate.org

YAML is very handy

Define workflow

• Similar to ORM: change DSL from thrift to YAML

• YAML

• More flexible

• Non-coder friendly

API• Needed in different places：

• web

• mobile web

• mobile client

• Business are the same, but may need different wrapping

Web Form

JSON HTTP

Mobile API

We should be lazy about writing boilerplate

wrapping code

So, we have 300k80line of code, mostly generated

• Above are “demo codes”

• Actual code are much more complicated

• ORM：github.com/sipin/xuanwu

• Just for referencing of idea; don’t use it

• No documentation

• No maintain

• We’ve restarted

Customisation？

• Generate once, and change

• Generate whenever template changes：

• Provide callback interface

• Method overwriting

• May use MP for generic, ORM, workflow, API wrapping, and more

• Consider MP as code-refactoring

• Write the dirty code first

• Abstract similar code repeated to template

• Choose an DSL

• Python is simple & resourceful; handy for metaprogramming prototype/PoC

Challenge of MP• Choosing the right DSL

• Provide both:

• high level of abstraction

• low level customization

• www.joelonsoftware.com/articles/LeakyAbstractions.html

Code generation

• Parse (DSL) & Bind (code-template) are easy

• just like HTML template

• Generated codes are simple, thus easy to debug

• Compiler checking for free

Reflection for LISP way？

• No compilation

• Error checking at runtime

• Implementation needs more challenger

• Codes are more elegant

• Yup, that could be better; so, how?

–Ken Thompson

“When in doubt, use brute force.”

Actually• Python is not so fast

• After experiencing git’s extreme speed, you can’t tolerate hg’s “not so fast”

• It’s easier to use & analyse the same language in MP

• golang.org/pkg/go/parser/

• go generate in go 1.4

• docs.google.com/document/d/1V03LUfjSADDooDMhe-_K59EgpTEm3V8uvQRuNMAEnjg

• Metaprogramming Go with GO is the way

• with py only for quick prototype/POC

Q & A