.
.
. ..
.
.
Development of SML# –making ML an ordinary practical languaage
Atsushi Ohorijoint work with Katsuhiro Ueno
, Tohoku University
Atsushi Ohori Development of SML# – making ML an ordinary practical language 1 / 59
Motivation and backgrounds
This talk is about SML#
Its motivation:
to make ML an ordinary practical language.
Some immediate reactions from eminent ML/FP researchers wouldbe:
We should be taking about the glorious future of ML or otherfunctional languages ...
ML has already been a super-advanced and highly practicallanguage.
What’s the issue?
Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59
Motivation and backgrounds
This talk is about SML#
Its motivation:
to make ML an ordinary practical language.
Some immediate reactions from eminent ML/FP researchers wouldbe:
We should be taking about the glorious future of ML or otherfunctional languages ...
ML has already been a super-advanced and highly practicallanguage.
What’s the issue?
Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59
Motivation and backgrounds
This talk is about SML#
Its motivation:
to make ML an ordinary practical language.
Some immediate reactions from eminent ML/FP researchers wouldbe:
We should be taking about the glorious future of ML or otherfunctional languages ...
ML has already been a super-advanced and highly practicallanguage.
What’s the issue?
Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59
Motivation and backgrounds
This talk is about SML#
Its motivation:
to make ML an ordinary practical language.
Some immediate reactions from eminent ML/FP researchers wouldbe:
We should be taking about the glorious future of ML or otherfunctional languages ...
ML has already been a super-advanced and highly practicallanguage.
What’s the issue?
Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59
Motivation and backgrounds
This talk is about SML#
Its motivation:
to make ML an ordinary practical language.
Some immediate reactions from eminent ML/FP researchers wouldbe:
We should be taking about the glorious future of ML or otherfunctional languages ...
ML has already been a super-advanced and highly practicallanguage.
What’s the issue?
Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59
Motivation and backgrounds
Backgrounds
ML is indeed a great language. it is:
Highly Productive
We all know it.
Highly Reliable, andPolymorphic type inference is the only verification tool made intopractice for programmers’ daily use.
SafeIt is type safe and memory safe; “buffer overflow” is not even an issue.
You get all of them for free by just using ML ....
Then, everyone should have been using ML by now,...
Atsushi Ohori Development of SML# – making ML an ordinary practical language 3 / 59
Motivation and backgrounds
Backgrounds
ML is indeed a great language. it is:
Highly Productive
We all know it.
Highly Reliable, andPolymorphic type inference is the only verification tool made intopractice for programmers’ daily use.
SafeIt is type safe and memory safe; “buffer overflow” is not even an issue.
You get all of them for free by just using ML ....
Then, everyone should have been using ML by now,...
Atsushi Ohori Development of SML# – making ML an ordinary practical language 3 / 59
Motivation and backgrounds Weaknesses of ML
Weaknesses of ML
Despite these advantages, ML has not yet been among the popularproduction languages of the industry.
There may be historical or cultural backgrounds/prejudice.
It is a toy invented in academia...
Functional language is difficult/inefficient/impractical...
But, more fundamentally, several important practical problems have beenleft unsolved.
improper treatment of records
lack of interoperability with C
lack of database support
lack of native thread support
lack of separate compilation and linking
lack of tools · · ·Atsushi Ohori Development of SML# – making ML an ordinary practical language 4 / 59
Motivation and backgrounds Weaknesses of ML
Weakness (1): improper treatment of records
Record is the most basic data constructor, which must be fullysupported.
However, record operations are not quite first-class citizens in ML.In SML:- fn x => map #name x;stdIn:1.1-1.17 Error: unresolved flex record(can’t tell what fields there are besides #name)
This is actually better than some of the other systems.
A minor comment: there are several solutions. I nonetheless address thisissue first, since our solution to this problem turns out to be an importantstep in the development of SML#.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 5 / 59
Motivation and backgrounds Weaknesses of ML
Weakness (2) : lack of interoperability with C
Seamless (or at least smooth) interoperability with C is aprerequisite for a new language to be accepted in practice.
Unfortunately, in current ML, using C libraries requires:to understand runtime representations of ML objects,to write a low-level stub for data conversion, andperhaps to call mysterious coordination-functions for GC
which make using C not only cumbersome but also dangerous.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 6 / 59
Motivation and backgrounds Weaknesses of ML
Weakness (3) : lack of database support
There are few real-world applications that do not use databases.
The current (ordinary) practice is string interface to SQL, where theprogrammer
generates an SQL command string,sends it to an SQL server, andconverts the result to appropriate data structure.
This is cumbersome, error prone and unsafe.
ML is worse: programmers cannot even use string interface to SQL.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 7 / 59
Motivation and backgrounds Weaknesses of ML
Weakness (4) : lack of native thread support
Exploiting emerging multi/many-core CPUs is the key to futurehigh-performance application development.
In the excising typical ML compilers, threads are implemented bytheir runtime system using a timer.
This implies that only one core can be used, no matter how manythreads the program code generates.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 8 / 59
Motivation and backgrounds Weaknesses of ML
Weakness (5) : lack of separate compilation
In a large software project, separate compilation and linking isessential.
Unfortunately, no ML compiler supports separate compilation andlinking in the ordinary sense, i.e. the process:
to compile a source file into an object file consisting ofa set of machine code blocksexternal name definitions of the exported variablesexternal name refereces of the imported variables
and to link an object file withother ML object files independenly compiled, andsystem libraries (libc, libm etc)
There seem to be no theoretical foundation for separately compilingStandard ML (except for typecheking.)
Atsushi Ohori Development of SML# – making ML an ordinary practical language 9 / 59
Motivation and backgrounds Sources of the weakness
Probable sources of these weakness
Widely accepted assumptions/beliefs/folklore around ML:Parametric polymorphism is the principle.Copying collector is the choice.Interfaces are Types
Taking them as axioms, ML attempts to re-construct the world.
This attempt may work well in theory, but creates some seriousproblems in practice.
Let us examine their implication and impact.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 10 / 59
Motivation and backgrounds Sources of the weakness
Parametric Polymorphism Principle
Since ’a in ’a -> ’a ranges over any types, we should only have onerepresentation, namely 32/64 bit word (possibly with an inline tag).
In the ordinary daily computation (on any hardware), however, we have:
Some programmers would surely give up polymorphism for these data.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59
Motivation and backgrounds Sources of the weakness
Parametric Polymorphism Principle
Since ’a in ’a -> ’a ranges over any types, we should only have onerepresentation, namely 32/64 bit word (possibly with an inline tag).
In the ordinary daily computation (on any hardware), however, we have:
Some programmers would surely give up polymorphism for these data.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59
Motivation and backgrounds Sources of the weakness
Parametric Polymorphism Principle
Since ’a in ’a -> ’a ranges over any types, we should only have onerepresentation, namely 32/64 bit word (possibly with an inline tag).
In the ordinary daily computation (on any hardware), however, we have:
Some programmers would surely give up polymorphism for these data.Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59
Motivation and backgrounds Sources of the weakness
Copy GC Folklore
Functional programs require fast allocation (true, indeed), on whichfolklore has it that:
“Cheney’s copying GC is the only option for primary (minor) GC”.
This implies that
there is one allocation pointer for the thread-shared global heap, andobjects are moved at any unpredictable time.
In such an absurd environment, one reasonable strategy for the MLruntime is to act as a monitor for thread scheduling and memory allocation.
However, if we are free from the folklore, then there is a possibility of:
having a non-moving GC with multiple allocation pointers, anddirectly using POSIX threads provided by the OS kernel.
Then you get true efficient multi-core threads for free.This is the way it should be; OS kernels are designed to serve us.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 12 / 59
Motivation and backgrounds Sources of the weakness
Copy GC Folklore
Functional programs require fast allocation (true, indeed), on whichfolklore has it that:
“Cheney’s copying GC is the only option for primary (minor) GC”.
This implies that
there is one allocation pointer for the thread-shared global heap, andobjects are moved at any unpredictable time.
In such an absurd environment, one reasonable strategy for the MLruntime is to act as a monitor for thread scheduling and memory allocation.
However, if we are free from the folklore, then there is a possibility of:
having a non-moving GC with multiple allocation pointers, anddirectly using POSIX threads provided by the OS kernel.
Then you get true efficient multi-core threads for free.This is the way it should be; OS kernels are designed to serve us.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 12 / 59
Motivation and backgrounds Sources of the weakness
Interface as Types Assumption
There are some proposals for (a theory of?) “separate compilation”system for ML, but as far as I can tell they only talk about type-checking,and do not address separate compilation.
In the ordinary understanding, compilation is the process to translate asource program into an object file consisting of
machine code blocks,a set of defined symbols for the set of exported variables, anda set of external symbols (to be resolved by a linker).
Separate compilation is a process to compile (in the above sense) asource without consulting any source programs that they are referenced.The reslulting object files are linked by a system linker (ld) with
other object files separately compiled, andsystem libraries (libc, libm etc).
In this ordinary sense, there is no system that can separately compileStandard ML (other than SML#.)
Atsushi Ohori Development of SML# – making ML an ordinary practical language 13 / 59
Motivation and backgrounds Sources of the weakness
Interface as Types Assumption
There are some proposals for (a theory of?) “separate compilation”system for ML, but as far as I can tell they only talk about type-checking,and do not address separate compilation.
In the ordinary understanding, compilation is the process to translate asource program into an object file consisting of
machine code blocks,a set of defined symbols for the set of exported variables, anda set of external symbols (to be resolved by a linker).
Separate compilation is a process to compile (in the above sense) asource without consulting any source programs that they are referenced.The reslulting object files are linked by a system linker (ld) with
other object files separately compiled, andsystem libraries (libc, libm etc).
In this ordinary sense, there is no system that can separately compileStandard ML (other than SML#.)
Atsushi Ohori Development of SML# – making ML an ordinary practical language 13 / 59
Motivation and backgrounds Sources of the weakness
Interface as Types Assumption II
For example, consider compiling the following into linkable machine code:
functor F (type foodatatype bar = A of foo * int) =
struct datatype hoge = X of int * X.barfun f (a, b) = X (a, X.A (b, a))fun g (X (_, X.A c)) = c
end
It is rather trivial to do the following:...1 separately typecheck sources and keep them (or elaboration of them)...2 when all the programs become available, then do code generation
from the sources (or their elaborated intermediate forms).but I would not consider such process as separate compilation.
Blurring separate compilation with typecheking would curtail theopportunity for making ML an ordinary practical language.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 14 / 59
Motivation and backgrounds The SML# Project
My comments on the weakness of ML
Of course researchers have been aware that
one-bit tag in integers causes problems
heap-allocating floating point numbers is undesirable
threads should be supported by the OS thread library, · · ·On this situation, two comments I would like to make:
...1 They are ingenious mechanisms researchers have developed torealize ML, to which my thorough respects are due.
...2 Overcoming them has been shown to be quite difficult. They are theprices to pay for ML to obtain its reliability and productivity.
Unfortunately, these prices are too high to make ML an ordinary practicallanguage.
As researchers and hackers∗ who love ML, we cannot stop here!(*) A person with an enthusiasm for programming or using computers as an end in itself.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59
Motivation and backgrounds The SML# Project
My comments on the weakness of ML
Of course researchers have been aware that
one-bit tag in integers causes problems
heap-allocating floating point numbers is undesirable
threads should be supported by the OS thread library, · · ·On this situation, two comments I would like to make:
...1 They are ingenious mechanisms researchers have developed torealize ML, to which my thorough respects are due.
...2 Overcoming them has been shown to be quite difficult. They are theprices to pay for ML to obtain its reliability and productivity.
Unfortunately, these prices are too high to make ML an ordinary practicallanguage.
As researchers and hackers∗ who love ML, we cannot stop here!(*) A person with an enthusiasm for programming or using computers as an end in itself.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59
Motivation and backgrounds The SML# Project
My comments on the weakness of ML
Of course researchers have been aware that
one-bit tag in integers causes problems
heap-allocating floating point numbers is undesirable
threads should be supported by the OS thread library, · · ·On this situation, two comments I would like to make:
...1 They are ingenious mechanisms researchers have developed torealize ML, to which my thorough respects are due.
...2 Overcoming them has been shown to be quite difficult. They are theprices to pay for ML to obtain its reliability and productivity.
Unfortunately, these prices are too high to make ML an ordinary practicallanguage.
As researchers and hackers∗ who love ML, we cannot stop here!(*) A person with an enthusiasm for programming or using computers as an end in itself.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59
Motivation and backgrounds The SML# Project
We must develop a new ML language that can be a possible choice of theindustry for serious software development.
The SML# Project
Its goal is to develop a new language in the ML family that is:a proper extension of Standard ML:
record polymorphism (and others, e.g rank-1 polymorphism, andlimited form of overloading)complete downward compatibility with Standard ML
with practically important features:seamless interoperability with Cseamless database integrationseparate compilation and linkingdirect and light-weight native thread supporteasy-to-use parallelism (being developed)
Atsushi Ohori Development of SML# – making ML an ordinary practical language 16 / 59
Motivation and backgrounds The SML# Project
SML# team and collaborators
SML# Development Team:
Atsushi Ohori (RIEC, Tohoku University)
Katsuhiro Ueno (RIEC, Tohoku University)
in (past) collaboration with (current affiliation):
Kiyoshi Yamatodani (Sanpu Kobo Inc.)
Nguen Duc Huu (Hanoi University of Science and Technology)
Liu Bochao (CNCERT, China)
Satoshi Osaka (Advantest)
with contributions from many peoples including:
Nobuaki Yoshida, Isao Sasano, Yutaka Matsuno,Kwanghoon Choi
Atsushi Ohori Development of SML# – making ML an ordinary practical language 17 / 59
Quick overview of SML# through examples Record polymorphism
Rest of the Talk
.. .1 Motivation and backgrounds
.. .2 Quick overview of SML# through examples
Record polymorphismInteroperability with CSeamless database integrationSeparate compilation
.. .3 Outline of the Technical Developments
Atsushi Ohori Development of SML# – making ML an ordinary practical language 18 / 59
Quick overview of SML# through examples Record polymorphism
Record polymorphism: basic record operations
fn x => map #X x;val it = fn : [’a#{X:’b},’b.’a list -> ’b list]
# fun getX x = #X x;val getX = fn : [’a#{X:’b},’b.’a -> ’b]
# fun setX x y= x # {X = y};val setX = fn : [’a#{X:’b},’b.’a -> ’b -> ’a]
Atsushi Ohori Development of SML# – making ML an ordinary practical language 19 / 59
Quick overview of SML# through examples Record polymorphism
Record polymorphism: objects
val methods ={getX = fn self => #X (!self),setX = fn S => fn x => S := (!S # {X = x}),getY = fn S => #Y (!S),setY = fn S => fn x => S := (!S # {Y = x}),getColor = fn S => #Color (!S),setColor = fn S => fn x => S := (!S # {Color = x})
};localval state = ref { X = 0.0, Y = 0.0 }
in val p1 = fn method => method methods stateend;p1 # setX 1.0;p1 # getX;
Atsushi Ohori Development of SML# – making ML an ordinary practical language 20 / 59
Quick overview of SML# through examples Record polymorphism
Record polymorphism: polymorphic variants
val P1 = fn M => #CPoint M {x=10.0, y = 10.0};val P2 = fn M => #PPoint M {r=14.1421356, theta = 45.0};val myPointList = [P1, P2];val distance ={CPoint = fn {x,y,...} => Real.Math.sqrt (x *x + y* y),PPoint = fn {r, theta,...} => r};
P1 distance ;P2 distance ;fun iterator L f = map (fn x => x f) L;val myPointLstIter = fn x => iterator myPointList x;myPointLstIter distance;
See (Ohori, 1995) for formal accounts of polymorphic variants.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 21 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (1): the “Hello World”
SML# source program hello.sml:val print = _import "printf" : string -> intval _ = print "こんにちは,日本へようこそ.\n"
Compile, link and execution:% smlsharp hello.sml% ./a.outこんにちは,日本へようこそ.
The above “printf” is the printf in libc library, of course.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 22 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (2): printf
Source program printf.sml:val printReal = _import "printf"
: (string, real) -> intval printTriple = _import "printf"
: (string, real, real, real) -> intval _ = printReal("%f\n", 3.14)val _ = printTriple("if a=%f, b=%f then a/b=%f\n,
3.14, 1.41, 3.14/1.41)
Compile, link and execute:% smlsharp printf.sml% ./a.out3.140000if a=3.140000, b=1.410000 then a/b=2.226950
Atsushi Ohori Development of SML# – making ML an ordinary practical language 23 / 59
Quick overview of SML# through examples Interoperability with C
Some comment on “printf”
There are some proposals for printf in ML. They are elegant,entertaining, and educational. They also roughly works as analternative to printf, but not quite.
val printYen = _import "printf" : (string,int) -> intval _ = printYen("%dYen\n", 1024)val _ = printYen("%10dYen\n", 1024)val _ = printYen("%-10dYen\n", 1024)val _ = printYen("%+10dYen\n", 1024)val _ = printYen("%#10xYen\n", 1024) ...
and many more alternatives.
Remember what you have wanted: it’s printf.Then use printf.
It’s free and fully functional (of course since it is printf).
Atsushi Ohori Development of SML# – making ML an ordinary practical language 24 / 59
Quick overview of SML# through examples Interoperability with C
Some comment on “printf”
There are some proposals for printf in ML. They are elegant,entertaining, and educational. They also roughly works as analternative to printf, but not quite.
val printYen = _import "printf" : (string,int) -> intval _ = printYen("%dYen\n", 1024)val _ = printYen("%10dYen\n", 1024)val _ = printYen("%-10dYen\n", 1024)val _ = printYen("%+10dYen\n", 1024)val _ = printYen("%#10xYen\n", 1024) ...
and many more alternatives.
Remember what you have wanted: it’s printf.Then use printf.
It’s free and fully functional (of course since it is printf).Atsushi Ohori Development of SML# – making ML an ordinary practical language 24 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (3): a small real example
mt.sml using Mersenne Twister:
val printWord = _import "printf" : (string, word) -> intval printReal = _import "printf" : (string, real) -> intval init = _import "init_genrand" : (int) -> unitval genWord = _import "genrand_int32" : () -> wordval genReal = _import "genrand_res53" : () -> realval _ = init 0val _ = printWord ("%d\n", genWord ())val _ = printReal ("%lf\n",genReal ())
% gcc -c mt19937ar.c -o mtlib.o # compile Mersenne Twister% smlsharp mt.sml mtlib.o% a.out23571360440.592845
Atsushi Ohori Development of SML# – making ML an ordinary practical language 25 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (4): higher-order functions I
(Just for your entertainment)
In SML#:
val f1 = _import "f1": (((((()->unit)->unit)->unit)->unit)->unit)->unit
val g1 = _import "g1" : ()->()->()->()->unitval () = f1 (fn h1 => (print "h1\n";
h1 (fn h2 =>(print "h2\n";h2 (fn h3 => print "h3\n")))))
val g2 = g1 ()val g3 = g2 ()val g4 = g3 ()val () = g4 ()
Atsushi Ohori Development of SML# – making ML an ordinary practical language 26 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (4): higher-order functions II
In C:
void f4(void){ printf("f4\n");}
void f3(void(*f)(void(*)(void))){printf("f3\n"); f(f4); }
void f2(void(*f)(void(*)(void(*)(void(*)(void))))){printf("f2\n");f(f3); }
void f1(void(*f)(void(*)(void(*)(void(*)(void(*)(void(*)(void)))))))
{printf("f1\n");f(f2);}
Atsushi Ohori Development of SML# – making ML an ordinary practical language 27 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (4): higher-order functions III
void g4(void){printf("g4\n");}
void (*g3(void))(void){printf("g3\n");return g4;}
void (*(*g2(void))(void))(void){printf("g2\n");return g3;}
void (*(*(*g1(void))(void))(void))(void){printf("g1\n");return g2;}
Atsushi Ohori Development of SML# – making ML an ordinary practical language 28 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (4): higher-order functions IV
Compile, link and execute:
% smlsharp callback.sml callback.o% gcc -c callback.c% smlsharp callback.sml callback.o% ./a.outf1 h1 f2 h2 f3 h3 g1 g2 g3 g4
Atsushi Ohori Development of SML# – making ML an ordinary practical language 29 / 59
Quick overview of SML# through examples Interoperability with C
Interoperability with C (5): polymorphic functions
(For your further entertainment)
val libc = DynamicLink.dlopen libcval c_qsort = DynamicLink.dlsym (libc, "qsort")fun qsort (a, f) =
_ffiapply c_qsort (a : ’a array,Array.length a : int,_sizeof(’a),f : (’a ptr, ’a ptr) -> int) : unit
In the interpreter mode:
# use "qsort.sml";...
val qsort =fn : [’a. ’a array * (’a ptr * ’a ptr -> int) -> unit]
Atsushi Ohori Development of SML# – making ML an ordinary practical language 30 / 59
Quick overview of SML# through examples Seamless database integration
Seamless database integration in SML#
SML# achieves seamless databases integration whereSQL expressions are first-class citizens in the language, andthey are evaluated by a database server.
As an example, consider the following SQL command:
Q = SELECT name, ageFROM peopleWHERE age <= 25
which is a shorthand for the following verbose syntax:
Q = SELECT person.name AS name, person.age AS ageFROM people AS personWHERE (person.age <= 25)
Atsushi Ohori Development of SML# – making ML an ordinary practical language 31 / 59
Quick overview of SML# through examples Seamless database integration
Simple demo: seamless database integration
# val Q = _sql db => select #person.name as name,#person.age as age
from #db.people as personwhere SQL.<= (#person.age, 25);
val q = fn: [’a#{people:’b},’b#{age:int, name:’d},’c,’d::{int, word, char, string, real, ’e option},’e::{int, word, char, bool, string, real}.(’a, ’c) SQL.db -> {age: int, name: ’d} SQL.query]
Of course, you get a lot of type errors when you first write acomplicated query.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 32 / 59
Quick overview of SML# through examples Seamless database integration
Simple demo: seamless database integration
# val Q = _sql db => select #person.name as name,#person.age as age
from #db.people as personwhere SQL.<= (#person.age, 25);
val q = fn: [’a#{people:’b},’b#{age:int, name:’d},’c,’d::{int, word, char, string, real, ’e option},’e::{int, word, char, bool, string, real}.(’a, ’c) SQL.db -> {age: int, name: ’d} SQL.query]
Of course, you get a lot of type errors when you first write acomplicated query.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 32 / 59
Quick overview of SML# through examples Seamless database integration
Simple demo: seamless database integration
# val server = _sqlserver "host=127.0.0.1 dbname=test": {people: {name: string, age: int}};
val server = "host=127.0.0.1 dbname=test": {people: {age: int, name: string}} SQL.server
# val db = SQL.connect server;val db = <conn> : {people:{age: int, name: string}} SQL.conn
# val r = _sqleval q db;val r = <rel> : {age: int, name: string} SQL.rel
# val x = SQL.fetchAll r;val x = [{age = 25, name = "Bob"}]: {age: int, name: string} list
Atsushi Ohori Development of SML# – making ML an ordinary practical language 33 / 59
Quick overview of SML# through examples Separate compilation
Separate compilation in SML#
SML# compiles a sequence of any oftype and datatype bindingsvariable and exception bindingsstructure bindingsfunctor bindings
into an object code in the standard ELF format, and links it withSML# runtime library,C libraries (linbc.a, libm.a, libgmp.a, etc), anduser compiled C functions
by invoking the system linker (ld) to produce an executable file.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 34 / 59
Quick overview of SML# through examples Separate compilation
SML# compiler command
Some typical usages:
smlsharp -c foo.smlcompiles foo.sml to an object file foo.o, consulting with its interfacefile specified implicitly or explicitly.
smlsharp -c foo.smi -o foolocates the object files referenced in foo.smi, invokes the systemlinker to produce an executable file foo.
smlsharp foo.smlcompiles foo.sml to foo.o, links it with all the files referenced in foo.smiand generates a.out.
An interface file foo.smi of foo.sml specifies:
the resources required by foo.sml and,
those foo.sml provides.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 35 / 59
Quick overview of SML# through examples Separate compilation
Separate compilation example I
top.sml:
structure B = F(type foo = realdatatype bar = A of foo * int)
val printPair = _import "printPair" : real * int -> unitval _ = printPair (B.g (B.f (2011, 3.14)))
top.smi:
_require "functor.smi", "printPair.smi"
Atsushi Ohori Development of SML# – making ML an ordinary practical language 36 / 59
Quick overview of SML# through examples Separate compilation
Separate compilation example II
functor.smi:
functor F(type foo; datatype bar = A of foo * int) =struct datatype hoge = X of int * X.bar
val f : int * X.foo -> hogeval g : hoge -> X.foo * int
end
functor.sml:
functor F (type foodatatype bar = A of foo * int) =
struct datatype hoge = X of int * X.barfun f (a, b) = X (a, X.A (b, a))fun g (X (_, X.A c)) = c
end
Atsushi Ohori Development of SML# – making ML an ordinary practical language 37 / 59
Quick overview of SML# through examples Separate compilation
Separate compilation example III
printPair.sml:
val printPair = _import "c_printPair" : (real,int) -> int
printPair.smi
val printPair : real * int -> int
C file
struct PAIR {double a; int b;};void c_printPair(struct PAIR *X) {printf ("(%4.2lf,%4ld)\n", X->a, X->b);}
Atsushi Ohori Development of SML# – making ML an ordinary practical language 38 / 59
Quick overview of SML# through examples Separate compilation
Separate compilation example IV
Compile, link and execute:
% smlsharp -c top.sml% smlsharp -c functor.sml% smlsharp -c printPair.sml% gcc -c printPair.c% smlahrp top.smi printPaur.o% a.out(3.14, 2011)
Atsushi Ohori Development of SML# – making ML an ordinary practical language 39 / 59
Outline of the Technical Developments
Contents of the Talk
.. .1 Motivation and backgrounds
.. .2 Quick overview of SML# through examples
.. .3 Outline of the Technical Developments
Polymorphic Record CalculusNatural data representationSeparate compilationSeamless database integrationEfficient Non-moving GC
Atsushi Ohori Development of SML# – making ML an ordinary practical language 40 / 59
Outline of the Technical Developments
Technical Development of SML#
SML# is realized by combining the following...1 Polymorphic record calculus...2 Natural data representation...3 Separate compilation...4 Polymorphic type system for databases...5 Efficient non-moving GC
Atsushi Ohori Development of SML# – making ML an ordinary practical language 41 / 59
Outline of the Technical Developments Polymorphic Record Calculus
The key technique: polymorphic record compilation
The key insight underlying the development of SML# compiler is
“Polymorphism” is just an abstraction, and not acomputational reality. So it must be compiled out.
The first example is the polymorphic record calculus and itscompilation (Ohori 1992, 1995).
Consider the term:
fn x => #name x
This is apparently polymorphic, but it is also apparent that this mustbe compiled out to efficient load instruction.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 42 / 59
Outline of the Technical Developments Polymorphic Record Calculus
Compiling polymorphic operation using types
First, we reconstruct a typed term:
val f : [’a,’b#{name:’a}. ’a -> ’b]= [’a, ’b#{name:’a}. fn )x:’b) => #name x]
f {name:string, age:int} {name = "smlsharp", age=7}
where ’b#{name:’a} is a type variable with a record kind.We then insert abstraction and application to pass necessaryattributes for compilation.
val f : [’a,’b#{name:’a}. index(’b,name) -> ’b -> ’a]= [’a, ’b#{name:’a}.fn (I:index(’b,name)) => fn (x:’b) => x[I]]
f {name:string, age:int} 2 [| 7,"smlsharp" |]
Atsushi Ohori Development of SML# – making ML an ordinary practical language 43 / 59
Outline of the Technical Developments Polymorphic Record Calculus
Review of record compilation
It is based on the following ideas:...1 Information needed to compile a polymorphic function is
abstracted at type abstraction and applied at type application....2 The needed information is encoded as a type that denotes the
unique needed value. In the example, index(τ, name)denotes the index value of the name field of a record type τ.
This mechanism is far-reaching in compiling out various aspects ofpolymorphism, which we have used to develop SML# throughout.
Note: These concepts have later been variously called such as“dictionary passing”“evidences”“singleton types”
Although literature did not seem to properly compare or even mention, theessence is the same as the one first presented in the 1992 article.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 44 / 59
Outline of the Technical Developments Polymorphic Record Calculus
Review of record compilation
It is based on the following ideas:...1 Information needed to compile a polymorphic function is
abstracted at type abstraction and applied at type application....2 The needed information is encoded as a type that denotes the
unique needed value. In the example, index(τ, name)denotes the index value of the name field of a record type τ.
This mechanism is far-reaching in compiling out various aspects ofpolymorphism, which we have used to develop SML# throughout.
Note: These concepts have later been variously called such as“dictionary passing”“evidences”“singleton types”
Although literature did not seem to properly compare or even mention, theessence is the same as the one first presented in the 1992 article.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 44 / 59
Outline of the Technical Developments Natural data representation
Natural data representation
In SML#, we have:
This is obviously desired. How the pointer-tracing GC works?
Atsushi Ohori Development of SML# – making ML an ordinary practical language 45 / 59
Outline of the Technical Developments Natural data representation
Natural data representation
In SML#, we have:
This is obviously desired. How the pointer-tracing GC works?Atsushi Ohori Development of SML# – making ML an ordinary practical language 45 / 59
Outline of the Technical Developments Natural data representation
Natural data representation: object bitmap
We make the following heap blocks.
-th bit
0 1
non-
pointerpointer
-th word
object pointer
header(includingsize )
words
layout bitmap
bits
Note: This is only for records (tuples); atomic data (except forstrings) are always unboxed.
How such a bit map can be computed for polymorphic functions?
We adopt the idea of record compilation.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59
Outline of the Technical Developments Natural data representation
Natural data representation: object bitmap
We make the following heap blocks.
-th bit
0 1
non-
pointerpointer
-th word
object pointer
header(includingsize )
words
layout bitmap
bits
Note: This is only for records (tuples); atomic data (except forstrings) are always unboxed.
How such a bit map can be computed for polymorphic functions?
We adopt the idea of record compilation.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59
Outline of the Technical Developments Natural data representation
Natural data representation: object bitmap
We make the following heap blocks.
-th bit
0 1
non-
pointerpointer
-th word
object pointer
header(includingsize )
words
layout bitmap
bits
Note: This is only for records (tuples); atomic data (except forstrings) are always unboxed.
How such a bit map can be computed for polymorphic functions?
We adopt the idea of record compilation.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59
Outline of the Technical Developments Natural data representation
Natural data representation: compilation
We introduce the typestag(τ) to denote one bit indicating whether τ is boxed or notsize(τ) to denote the size of τ
The rest is essentially the same as record compilation, but requirescertain amount of careful development, especially in
closure conversion that creates records which require bitmaps,a call frame that also requires a bitmap,properly aligning record fields.
See (Nguen Duc Huu and Atsushi Ohori, 2006) for the details.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 47 / 59
Outline of the Technical Developments Separate compilation
Separate compilation
I’d repeat:
to make ML an ordinary practical language, it is essential to haveseparate compilation and linking in the ordinary sense, which isthe process
to compile a source file into an object file consisting ofa set of machine code blocks,external name definitions of the exported variables, andexternal name refereces of the imported variables,
and to link an object file withother ML object files independenly compiled, andsystem libraries (libc, libm etc)
As far as I can see, separate compilation in this ordinary sense has neverbeen done before for Standard ML or equivalent one.Indeed, separately compiling Standard ML declarations into an object fileis inherently difficult.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 48 / 59
Outline of the Technical Developments Separate compilation
Issues in separately compiling Standard ML (I)
To compile a program separately from those it references, it must beclosed both in its type and in its representation.
Suppose we compile a source program that references a structure
S :> sigtype fooval x : foo
end
Since foo is unknown, we must specify its interface such as
structure S =structtype foo (= real * int) (* opaque type interface *)val x : foo
end
Atsushi Ohori Development of SML# – making ML an ordinary practical language 49 / 59
Outline of the Technical Developments Separate compilation
Interface language
By this way, we design an interface language whose syntax is roughly:
<itopdec> ::= <idecl> | <ifundecl><idecl> ::= val <valdesc>
| type <itypbind>| eqtype <tyvarSeq> id (= <ty>)| datatype <idatbind>| datatype <id> = datatype <longid>| exception <exbind>| structure <id> = <istrexp>
<itypbind> ::= <tyvarSeq> <id> = <ty>| <tyvarSeq> <id> ( = <ty> )
<idatbind> ::= <tyvarSeq> <id> = <conbinds> [and <idatbind>]| <tyvarSeq> <id> ( = <conbinds> [and<idatbind>])
<istrexp> ::= struct <idecl-seq> end<ifundec> ::= functor <id> ( <id> : <sigexp> ) = <istrexp>
Atsushi Ohori Development of SML# – making ML an ordinary practical language 50 / 59
Outline of the Technical Developments Separate compilation
Separately compiling a functor
Then how to deal with functor, which is by definition open in its typeparameters.
functor F (X:sigtype foodatatype bar = A of foo * int
end) =structdatatype hoge = X of int * X.barfun f (a, b) = X (a, X.A (b, a))fun g (X (_, X.A c)) = c
end
Since foo’s is a parameter, its representation varies.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 51 / 59
Outline of the Technical Developments Separate compilation
Lifting functor arguments
We have solved this problem based roughly on the following idea:
lifting static (type) parameters as type abstractions,
lambda lift dynamic parameters (variables)
A functor becomes a polymorphic function and functor application becomeapplication with type instantiation.
We can then use the idea of record compilation again to pass necessarycompile attributes through type abstraction and type application.
The example code is transformed to the code of the form:
val ’foo F = fn (id : ’foo -> ’foo) =>let fun f (a, b) = ...
fun g v = ...in (f, g) end
Atsushi Ohori Development of SML# – making ML an ordinary practical language 52 / 59
Outline of the Technical Developments Separate compilation
Other issues in separate compilation
There are a number of other issues that had to be worked out, including
flattening nested structures (mostly mechanical),
complete signature checking before type inference
proper treatment of exceptions
statically allocating constants and records in top-level declarations,
generating initialization code, and
generating system linkable object files.
We have solved all the problems and have a separate compile system thatcompile any sequence of declarations.
The details will (hopefully) be published somewhere in near future,we hope.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 53 / 59
Outline of the Technical Developments Separate compilation
Separate compilation demo : lexgen
We have wrote interface file for each file in the basis library.basis.smi
include "basis/main/ArrayStructures.smi"include "basis/main/BinIO.smi"include "basis/main/Bool.smi"....
lexgen benchmark can be compiled by writing the interface file
_require "basis.smi"structure Main = struct val doit : unit -> unit end
without consulting any of the actual source files.The resulting object file can be liked with those of the basis to form anexecutable file by the command line:
% smlsharp lexgen.smi
Atsushi Ohori Development of SML# – making ML an ordinary practical language 54 / 59
Outline of the Technical Developments Seamless database integration
Seamless database integration in SML#
SML# achieves seamless databases integration where
SQL expressions are first-class citizens in the language that can befreely combined with all the other components of SML#, and
they are evaluated by a database server.
using the following results and new technical developments:
database type system a la Machiavelli [Ohori, et.al. 1989, 1996],
record polymorphism,
C FFI,
light-weight first-class overloading (through record compilation again),
limited introduction of existential types, and
a uniform approach to implement a DSL, bridging static and dynamicsemantics.
The details at the talk tomorrow.Atsushi Ohori Development of SML# – making ML an ordinary practical language 55 / 59
Outline of the Technical Developments Efficient Non-moving GC
Efficient non-moving GC
It is absurd to assume that objects are moved at any unpredictable time.
Indeed, there are many objects you can’t move....
My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.
Near the C-5 exit ofOtemachi st. About 20min. walk from here.
Photography by Katsuhiro Ueno
Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59
Outline of the Technical Developments Efficient Non-moving GC
Efficient non-moving GC
It is absurd to assume that objects are moved at any unpredictable time.
Indeed, there are many objects you can’t move....
My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.
Near the C-5 exit ofOtemachi st. About 20min. walk from here.
Photography by Katsuhiro Ueno
Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59
Outline of the Technical Developments Efficient Non-moving GC
Efficient non-moving GC
It is absurd to assume that objects are moved at any unpredictable time.
Indeed, there are many objects you can’t move....
My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.
Near the C-5 exit ofOtemachi st. About 20min. walk from here.
Photography by Katsuhiro Ueno
Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59
Outline of the Technical Developments Efficient Non-moving GC
Efficient non-moving GC
It is absurd to assume that objects are moved at any unpredictable time.
Indeed, there are many objects you can’t move....
My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.
Near the C-5 exit ofOtemachi st. About 20min. walk from here.
Photography by Katsuhiro Ueno
Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59
Outline of the Technical Developments Efficient Non-moving GC
Efficient non-moving GC
If we are sure that objects once created will not be moved (as in anordinary decent world), then interoperability with C and native threadsupport become lot easier.
We have successfully developed a non-moving GC with generationalextension that is as efficient as the legendary Cheney’s generationalcopying collector.
The details at the talk this afternoon by Ueno san.
Atsushi Ohori Development of SML# – making ML an ordinary practical language 57 / 59
Outline of the Technical Developments Efficient Non-moving GC
Conclusions
SML# is available fromSML# 0.62 the interactive compilerSML# 0.90 the separate compiler (released on 2011-09-14)
We hope to release SML# 1.00 in near future by integrating SML#0.62 into SML# 0.90.
Near Future Plan64 bit support (easy)concurrent GCoptimization (type propagation)
Atsushi Ohori Development of SML# – making ML an ordinary practical language 58 / 59
Outline of the Technical Developments Efficient Non-moving GC
Thank You
Atsushi Ohori Development of SML# – making ML an ordinary practical language 59 / 59