metaprogramming for erlang. abstract format & core

7/30/2019 Metaprogramming for Erlang. Abstract Format & Core

1/42

Metaprogramming for Erlang

Abstract Format & Core

Salvador Tamarit Munoz

Universitat Politecnica de Valencia
http://find/http://goback/


2/42

Contents

1 Abstract Format

Description

How to get itManipulating Abstract Format

Get the Abstract Format from an Erlang expression

Smerl

Evaluating Abstract Format

Libraries

2 Core Erlang

Description

Syntax

How to get it

From Erlang to Core

Manipulating Core ErlangGet the Core Erlang from an Erlang expression

Get the Core Erlang from an external module

Retrieving Erlang Code from Core Erlang

Libraries

3 Conclusions
http://find/


3/42

Abstract Format
http://find/


4/42

Abstract Format Description

Description

The following Erlang expression:

it is represented in abstract format by the following tuple:

The tuple forms a tree structure:

Salvador Tamarit Munoz (UPV) Abstract Format & Core Erlang U. Politecnica de Valencia 4 / 29

Ab F D i i


5/42


Description





Ab t t F t D i ti
http://find/


6/42


Description





http://goforward/http://find/http://goback/


7/42


Description

A module is represented with a list [F1...Fn], where each Frepresents a form.

A form is either an attribute or a function declaration. Concretely:

For instance, the abstract format corresponding to the attribute-module(Mod) is {attribute,LINE,module,Mod}.The abstract format corresponding to a function declaration is{function,LINE,Name,Arity,[FC1...FCn]} where each FC is theabstract format of a function clause, wich in turn is represented by{clause,LINE,[P1...Pn],[G1...Gn],[E1...En]} where each P, Gand E is the abstract representation of one of its pattern, one of its

guards and one of its bodys expressions respectively.

The whole abstract format description is publicly available at:

http://www.erlang.org/doc/apps/erts/absform.html


Abstract Format How to get it
http://www.erlang.org/doc/apps/erts/absform.htmlhttp://www.erlang.org/doc/apps/erts/absform.htmlhttp://find/


8/42


How to get it

One line of code:

Lets see an example:

factorial.erl


https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/factorial.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/factorial.erlhttp://find/


9/42


How to get it

One line of code:

Lets see an example:

factorial.erl


Abstract Format Manipulating Abstract Format


10/42


Manipulating Abstract Format

We want to write a program transformation to replace each occurrence of

an append operator (++) where its left operand is single element list (e.g.[1]) with a list construction. For instance:

[1] ++ [2,3] [1 | [2,3]]

Additionally, we will replace append operations where one of their operand

is an empty list with its other operand. For instance:

[] ++ [2,3] [2,3][2,3] ++[] [2,3]

The code is in:refactorer.erl

And a test with some cases in:

refactorer test.erl


https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer.erlhttp://find/


11/42

p g


We want to write a program transformation to replace each occurrence of

an append operator (++) where its left operand is single element list (e.g.[1]) with a list construction. For instance:

[1] ++ [2,3] [1 | [2,3]]

Additionally, we will replace append operations where one of their operand

is an empty list with its other operand. For instance:

[] ++ [2,3] [2,3][2,3] ++[] [2,3]

The code is in:

refactorer.erl

And a test with some cases in:

refactorer test.erl


https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer.erlhttp://find/


12/42

p g


We can write a simpler version using function erl syntax lib:map/2:

refactorer simpler.erl

We have been accessing directly to the raw abstract format. We can uselibrary erl syntax to avoid having to do so:

refactorer simpler alt.erl


https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler.erlhttp://www.erlang.org/doc/man/erl_syntax.htmlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler_alt.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler_alt.erlhttp://www.erlang.org/doc/man/erl_syntax.htmlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler.erlhttp://goforward/http://find/http://goback/


13/42

g


We can write a simpler version using function erl syntax lib:map/2:

refactorer simpler.erl

We have been accessing directly to the raw abstract format. We can uselibrary erl syntax to avoid having to do so:

refactorer simpler alt.erl


https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler.erlhttp://www.erlang.org/doc/man/erl_syntax.htmlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler_alt.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler_alt.erlhttp://www.erlang.org/doc/man/erl_syntax.htmlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_simpler.erlhttp://find/


14/42


If we want easily reuse our transformations, we can use parse

transformations. In order to implement a parse transformation, we need towrite in our module a function namedparse transform/2 and export it.

refactorer pt.erl

Then, we can reuse this transformation via a complier option:c(refactorer test,[{parse transform,refactorer pt}]).

The loaded code will have its content changed according to thetransformation defined in refactorer pt.

When a program transformation is faced for first time, it is a good idea totake a look in the following module to consult the cases to be treated:

erl id trans.erl


https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_pt.erlhttps://github.com/erlang/otp/blob/maint/lib/stdlib/examples/erl_id_trans.erlhttps://github.com/erlang/otp/blob/maint/lib/stdlib/examples/erl_id_trans.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_pt.erlhttp://find/


15/42




refactorer pt.erl




erl id trans.erl


https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_pt.erlhttps://github.com/erlang/otp/blob/maint/lib/stdlib/examples/erl_id_trans.erlhttps://github.com/erlang/otp/blob/maint/lib/stdlib/examples/erl_id_trans.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_pt.erlhttp://find/


16/42




refactorer pt.erl




erl id trans.erl


Abstract Format Get the Abstract Format from an Erlang expression
https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_pt.erlhttps://github.com/erlang/otp/blob/maint/lib/stdlib/examples/erl_id_trans.erlhttps://github.com/erlang/otp/blob/maint/lib/stdlib/examples/erl_id_trans.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/refactorer_pt.erlhttp://find/http://goback/


17/42


It is very easy to get the abstract format corresponding to any expression:


Abstract Format Get the Abstract Format from an Erlang expression
http://find/


18/42


It is very easy to get the abstract format corresponding to any expression:


Abstract Format Smerl
http://find/


19/42

Smerl

Module smerl.erl provides functionality to simplify themetaprogramming task, along with some interesting features.

Additionally, with smerl it is possible to create and compile easily a newmodule allowing to call its functions:

smerl test.erl


Abstract Format Evaluating Abstract Format
https://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/smerl.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/smerl_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/smerl_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/abstract_format/smerl.erlhttp://find/


20/42


We can use the functionality of module erl eval to evaluate abstract

format structures:


Abstract Format Evaluating Abstract Format


21/42


We can use the functionality of module erl eval to evaluate abstract

format structures:


Abstract Format Libraries
http://find/


22/42

Libraries

There are several useful modules to work with the abstract form:

erl syntax

This module defines an abstract data type for representing Erlangsource code as syntax trees, in a way that is backwards compatiblewith the data structures created by the Erlang standard library

parser module erl parse.erl syntax lib

This module contains utility functions for working with the abstractdata type defined in the module erl syntax.

erl prettyprPretty printing of abstract Erlang syntax trees.

erl id trans

An Identity Parse Transform.


Abstract Format Libraries
http://find/


23/42

Libraries

There are several useful modules to work with the abstract form:

erl tidy

Tidies and pretty-prints Erlang source code, removing unusedfunctions, updating obsolete constructs and function calls, etc.

epp dodgerBypasses the Erlang preprocessor - avoids macro expansion, fileinclusion, conditional compilation, etc. Allows to find/modifyparticular definitions/applications of macros, etc.

igor It merges the source code of one or more Erlang modules into asingle module, which can then replace the original set of modules.

http://find/


24/42

Core Erlang

Core Erlang Description


25/42

Description

A program operating on source code must handle so many cases as tobecome impractical in general. Core Erlang was designed to overcomethis issue.

The compiler uses it as an intermediate representation between the

source code and the byte code. Additionally it helps to perform someoptimizations.

Some interesting features are:

A strict, higher-order functional language.

Simple and unambiguous grammar.Human-readable textual representation.Language easy to work with.


Core Erlang Syntax
http://find/


26/42

Syntax

fname ::= Atom / Integer

lit ::= Atom | Integer | Float | Char | String | [ ]fun ::= fun(var1 , . . . , varn) -> exprsclause ::= pats when exprs1 -> exprs2pat ::= var | lit | [ pats | pats ] | { pats1, . . . , patsn } | var = patspats ::= pat | < pat, . . . , pat >exprs ::= expr | < expr, . . . , expr >expr ::= var | fname | fun | [ exprs | exprs ] | { exprs1, . . . , exprsn }

| let vars = exprs1 in exprs2| letrec fname1 = fun1 ...fnamen = funn in exprs| apply exprs ( exprs1 , . . . , exprsn )| call exprsn+1:exprsn+2 ( exprs1 , . . . , exprsn )| primop Atom ( exprs1 , . . . , exprsn )| try exprs1 of -> exprs2 catch -> exprs3

| case exprs of clause1 ...clausen end | do exprs1 exprs2 | catch exprs ::= Exception(valm)val ::= lit | fname | fun | [ vals | vals ] | {vals1, . . . , valsn} | vals ::= val | < val, . . . , val >vars ::= var | < var, . . . , var >


Core Erlang How to get it
http://find/


27/42

How to get it

There are two main ways to get the core representation of a module:

In the interpreter, with the following instruction:

> c(factorial,to core).

In a source code, using the functionality of the module compile:

{ok, ,Core} =compile:file("factorial.erl",[to core, binary])

Try it:

factorial.erl


Core Erlang How to get it


28/42

How to get it

There are two main ways to get the core representation of a module:

In the interpreter, with the following instruction:

> c(factorial,to core).

In a source code, using the functionality of the module compile:

{ok, ,Core} =compile:file("factorial.erl",[to core, binary])

Try it:

factorial.erl


Core Erlang From Erlang to Core

C


29/42

From Erlang to Core

It is important to know some basis of how the translation from Erlang toCore Erlang is done. It is going to be definitely very useful when treatingthe Core Erlang code. Let see an example:

core transformations.erl

There is a compiler option that let the resulting Core Erlang closer to theoriginal Erlang source. We should use it when we are facing a problemwhere the correspondence between Core and Erlang is important.

> c(core transformations, [to core, no copt])


Core Erlang From Erlang to Core

F E l C
https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_transformations.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_transformations.erlhttp://goforward/http://find/http://goback/


30/42

From Erlang to Core

It is important to know some basis of how the translation from Erlang toCore Erlang is done. It is going to be definitely very useful when treatingthe Core Erlang code. Let see an example:

core transformations.erl

There is a compiler option that let the resulting Core Erlang closer to theoriginal Erlang source. We should use it when we are facing a problemwhere the correspondence between Core and Erlang is important.

> c(core transformations, [to core, no copt])


Core Erlang Manipulating Core Erlang

M i l i C E l
https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_transformations.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_transformations.erlhttp://goforward/http://find/http://goback/


31/42

Manipulating Core Erlang

Suppose we need to know all the variables names used in a module. We

should only return the names of user defined variables, in other words, weshould ignore variables introduces by Core (i.e. cor variables). Functionnames are also considered as variables names and they are represented bytuples of two elements. We want these function names to be ignored also.The code is in:

core vars.erl

As in the case of Abstract Format, there are some libraries that can

simplify this code. In this case, librarycerl trees

contain some usefulmethods. We can get a simpler version of our previous code:

core vars simpler.erl


Core Erlang Manipulating Core Erlang

M i l ti C E l
https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars_simpler.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars_simpler.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars.erlhttp://find/http://goback/


32/42

Manipulating Core Erlang

Suppose we need to know all the variables names used in a module. We

should only return the names of user defined variables, in other words, weshould ignore variables introduces by Core (i.e. cor variables). Functionnames are also considered as variables names and they are represented bytuples of two elements. We want these function names to be ignored also.The code is in:

core vars.erl

As in the case of Abstract Format, there are some libraries that cansimplify this code. In this case, library cerl trees contain some usefulmethods. We can get a simpler version of our previous code:

core vars simpler.erl


Core Erlang Get the Core Erlang from an Erlang expression

G t th C E l f E l i
https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars_simpler.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars_simpler.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_vars.erlhttp://find/


33/42

Get the Core Erlang from an Erlang expression

We could need to get the Core Erlang corresponding to a single expression.We can build an auxiliary module, and then compile it to core. The

following module shows how to get this:

core from expression.erl


Core Erlang Get the Core Erlang from an external module

G t th C E l f t l d l
https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_from_expression.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_from_expression.erlhttp://find/


34/42


Sometimes we need to load Core Erlang from a module included in theOTP libraries. The following code allow us to get this:

core from module.erl

Note that this code can also load Abstract Format changing the call tofunction compile:file/2 in the the last expression to a call to functionepp:parse file/3.


Core Erlang Get the Core Erlang from an external module

https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_from_module.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_from_module.erlhttp://find/


35/42


Sometimes we need to load Core Erlang from a module included in theOTP libraries. The following code allow us to get this:

core from module.erl

Note that this code can also load Abstract Format changing the call tofunction compile:file/2 in the the last expression to a call to functionepp:parse file/3.


Core Erlang Retrieving Erlang Code from Core Erlang

https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_from_module.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_from_module.erlhttp://find/


36/42


It is not easy to return from Core Erlang to Erlang. There are differentproblems:

The language is different, and some Core constructions are not inErlang and vice versa.

Providing that the expression that we want to retrieve could beretrieved, there is still a problem. Its line is the only way to identifyan expression as well as its type. Therefore, if two expression of thesame type share the same line is impossible to know which is the onethat we want to retrieve. For example, suppose we have this

expression in one line of a Erlang module {{1,2},{3,4}}. During theCore processing, we want to retrieve a tuple from this line. We willhave three possibilities.



http://find/


37/42


The problem can be seen using this module:

core to abstract bad.erl

with this module as test:

core to abstract test.erl



https://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_to_abstract_bad.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_to_abstract_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_to_abstract_test.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_to_abstract_bad.erlhttp://find/http://goback/


38/42


We can build a parse transformation to give to each individual expression aunique line, and then compile to Core from the resulting forms. Themodule implementing this transformation is:

line changer pt.erl

We can now modify the previous module to retrieve correctly the Erlangexpression for a given Core Erlang:

core to abstract.erl


Core Erlang Libraries

Libraries
https://github.com/tamarit/erlang_metaprogramming/blob/master/core/line_changer_pt.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_to_abstract.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/core_to_abstract.erlhttps://github.com/tamarit/erlang_metaprogramming/blob/master/core/line_changer_pt.erlhttp://find/


39/42

Libraries

Interesting modules when manipulating Core Erlang are:

cerl

This module defines an abstract data type for representing CoreErlang source code as syntax trees.

cerl treesBasic functions on Core Erlang abstract syntax trees.

cerl clauses

Utility functions for Core Erlang case/receive clauses.

cerl inlineAn implementation of the algorithm by Waddell and Dybvig (Fastand Effective Procedure Inlining, 1997), adapted to the Core Erlang.


Core Erlang Libraries

Libraries
http://find/


40/42

Libraries

Interesting modules when manipulating Core Erlang are:

cerl prettypr

Core Erlang prettyprinter.

cerl closureanClosure analysis of Core Erlang programs.

cerl pmatch

Core Erlang pattern matching compiler.

cerl typean

Type analysis of Core Erlang programs.


Conclusions

Conclusions


41/42

Conclusions

Abstract formatIt is easy to manipulate and understand.

We have a lot of libraries to ease our work.

We can evaluate its forms in two different ways.

Core Erlang

It is suitable for program analysis.

It is simpler that Abstract Format but it stills human-readable.

There exists a lot of libraries helping to work with it.

It is possible to retrieve the corresponding Erlang expressions usingsome tricky methods.



42/42

Thanks for listening
http://find/

metaprogramming for erlang. abstract format & core

Documents