next generation software systems and programming languages research matthias felleisen

Next Generation Software Systems and Programming Languages Research

Matthias Felleisen

Our Experience

• PLT has built a bunch of software, including PDE, Web server, IMAP clients, and many more.

• These products share a number of characteristics with other modern software products.

• What does our experience suggest for PL research?

My Thesis, My Goal

• Software systems have become extensible collections of components.

• What does this mean for programming languages: linguistics, pragmatics, virtual machines/compilers, and run-time environments?

Outline

• Two Examples: DrScheme, Web Server

• Language Support for COP• Virtual Machine Support • Contracts for Components

• Summary, Advice

Disclaimer

• PLT’s experience

• My thesis and trajectory

• these are not universal concerns

Examples: Extensible Systems

Two Examples

• DrScheme: a modern, pedagogic programming environment

• The PLT Web Server: an experiment in serving dynamic Web content

DrScheme

• What does it take to build a PDE from scratch?

• How about extensions?

DrScheme scope-sensitive syntax checker

interactive evaluator

algebraic stepper

static debugger

syntax-sensitive editor

Thank You to a Fantastic Team

Cormac, SRC

Shriram Krishnamurthi, Brown

Matthew Flatt, Utah

John Clements, NEU

Robby Findler, UChicago

Paul Steckler, NEU

Philippe Meunier, NEU

Paul Graunke, NEU

DrScheme

Editor

Read-Eval-Print-Loop

model

GUI

view

mutual references

DrScheme: Plugging in Tools

Editor

Read-Eval-Print-LoopTools -- plug-ins

Syntax Checker

Algebraic Stepper

DrScheme: Plugging in Student Programs

Editor

Read-Eval-Print-LoopTools -- plug-ins

Syntax Checker

Algebraic StepperStudent Programs

DrScheme: The Big Picture

Editor

Read-Eval-Print-Loop

a single execution environment: the VM

DrScheme

• powerful plug-ins that interact with editor and read-eval-print loop

• student programs become dynamically plugged in components

• easy communication among all components

A Web Server From Components

• What does it take to build a Web server?

• What about dynamic content?

A Web Server From Components

A server is basically a read-eval-print loop. Upon a TCP connect:

read a request: GET /uniquelylara/paintings/index.html HTTP/1.0 … headers …

interpret the request, the path, and the headers: dispatch on file path, check for defaults, …

print the file: send the requested file back via a TCP write

A Web Server From Components

pattern matching TCP connections URL parsing HTML

Dispatcher

REPL aka Web Server

These components are good old libraries.

A Web Server From Components

pattern matching TCP connections URL parsing HTML

Dis: remote

REPL aka Web Server

Dis: local Dis: dynamic

code duplication

A Web Server From Components

pattern matching TCP connections URL parsing HTML

Dispatch

REPL aka Web Server

Dispatch Dispatchlocal

remote

dynam

these components are parameterized and instantiated

Web Server: Dynamic Content

• dynamic content generation is becoming more and more important

• dynamic content generation is inefficient and slow

• dynamic content generation is difficult to manage

Web Server: Plug-in Scripts

Dispatcher

Server Loop

Scripts

Web Server: Scripts as Components

Dispatcher

Server Loop

dynamically linked

exchange of valuesnot bit streams

Web Server: Scripts as Components

Dispatcher

Server Loop

store script as first-class value

Web Server: Scripts as Components

Dispatcher

Server Loop

link in once

Web Server: Scripts as Components

Dispatcher

Server Loop

link in once, link in twice

Web Server: Scripts as Components

Dispatcher

Server Loop

make the internals of the server available: stack, environment, store

A Web Server From Components

• Our Web server’s performance for dynamic content generation is four times better than that of Apache/Perl and 20 to 40% than mod-Perl.

• It is the only Web server whose scripts are automatically safe for back buttons and cloning: control state.

Other Examples

• Linux:– kernel modules– applications

• Web browsers:– plug-ins– applets

• Multimedia players:– drivers– formats

• MS Office– component: COM

• Java Beans

• .NET

From Component Systems to Programming Languages

From Component Systems to PLs

• component-oriented programming:– different than OOP– different than modules

• VM (virtual machines) for extensible systems of components– more than a virtual machine

• contracts for components

Component Software, Szyperski

• A component is …

– a unit of independent deployment – a unit of third-party composition – a blueprint without state

Szyperski, page 30

COP and Languages: Linguistic Support

• parameterization over– imported components– exchange of values, including critical pieces

• linking– graph-based (mutual references)– hierarchical – dynamic (first-class values in core)

• multi-lingual components

COP and Languages: Virtual Machines

• a VM for many languages

• a VM for dynamic extensions

• a VM for “non-cooperative” extensions

COP and Languages: Quality Assurance

• components exchange values, including objects and functions

• components must protect these values and must promise certain properties– types– (partial) functional specifications– concurrency specifications– QoS guarantees

Linguistic Support for COP

with Matthew Flatt, Robby Findler, Shriram Krishnamurthiand a little bit of help from Dan Friedman

Existing Language Support for Components

Which language constructs should we use to support programming with components?

Existing Language Support for Components

Proposal: Use first-class procedures and arrange them in patterns.

Pro: hierarchical linking, dynamic invoking, by-reference passing work

Cons: the granularity is too far off. Maintaining “linking patterns” by hand is error prone.

Existing Language Support for Components

Proposal: Use classes and objects as components.

Pro: larger granularity, contains many functions, protection, dynamic loading

Cons: The super-class is hard-wired, i.e., no parameterization. It is impossible to wire-in the same class many times. Inheritance poses programming and contract checking problems (Szyperski).

Existing Language Support for Components

Proposal: Use Modula-style modules, Java-style packages, or C#-style assemblies.

Pro: proper granularity

Cons: modules are too static (compile-time). Importing is hard-wired. They are not first-class values.

Existing Language Support for Components

Proposal: Use ML-style functors. They find good uses in SML and OCAML.

Pro: They are parameterized over their importing modules, by definition.

Cons: they are not values. It is not possible to link them in a graphical manner. Like plain modules, they are static and not dynamically linkable.

PLT’s Components: Units

• Units are parameterized over– imported components– references to values

• Units are linked in – graph-based– hierarchical – dynamic manner

Units are first-class values:

Units: A Language Constructs for COP

imports

definitions

exports

Units: A Language Constructs for COP

type t , val g : int -> t

fun f (an-int, a-t) = …

val f : int t -> string

example:

Units: Graphical Linking

type t

fun f (an-int, a-t) = …

f : int t -> string

type t

fun f (an-int, a-t) = …

f : int t -> string

type t

fun f (an-int, a-t) = …

f : int t -> string

Units: Hierarchical Linking

type t

fun f (an-int, a-t) = …

f : int t -> string

type t

fun f (an-int, a-t) = …

f : int t -> string

type t

fun f (an-int, a-t) = …

f : int t -> string

Units: Dynamic Linking

type t

fun f (an-int, a-t) = …

f : int t -> string

type t

fun f (an-int, a-t) = …

f : int t -> string

type t

fun f (an-int, a-t) = …

f : int t -> string

type t

fun f (an-int, a-t) = … f : int t -> string

Units: Flatt & Felleisen

• PLT’s units support COP well.

• … linking together components is flexible, with rich control

• … widely used in Version 103

• But there are open problems

Units: A Program Design Problem

• Given: a component with – a datatype T with N variants– a collection F of M functions on T

• Wanted: extend T with variants and F with additional functions without access to the source of the component

Krishnamurthi, Felleisen, Friedman

Units: A Program Design Problem

• Why does it matter?– abstract reasons– examples

• How do we do it? – where units work – where we need more

Units Extensibility - Why?

• maintenance: fix once, fix everywhere

• extensibility: add functionality once, add it everywhere

Unit Extensibility: Example 1

• DrScheme: Krishnamurthi et alli the language hierarchy and the set of tools

• datatype is a grammar

• tools: syntax check, type check, stepper, … and more to come

Unit Extensibility: Example 2

• Geometry server: Hudak, Darpa productivity study

• datatype describes arrangements of geometric shapes

• tools: containment, drawing, moving and many others

Unit Extensibility: Example 3

• San Francisco project: Bohrer, IBM

• datatype describes customer, manager, account hierarchies

• tools: back office maintenance tools

Unit Extensibility: The Picture

f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Core Component

Unit Extensibility: The Picture

f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Variant Extension

f, g, h : extended

Unit Extensibility: The Picture

f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Functionality Extension

f, g, h : extended

j : … t … -> … t ..

k : … t … -> … t ..

Unit Extensibility: The Picture

f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Full Component

f, g, h : extended

j : … t … -> … t ..

k : … t … -> … t ..

self-reference (has-a)

self call

Unit Extensibility: The Experience

• Units work well for extensibility

• … just re-link for each extension

• … each layer around a given component is just an adapter unit (Lieberherr & Lorenz ‘01)

Unit Extensibility: The Experience

• Units alone are not enough

• … need lasses to specify datatypes

• … and mixins for extensions

Unit Extensibility: The Experience

f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Full Component

f, g, h : extended

j : … t … -> … t ..

k : … t … -> … t ..

Interfaces don’t knowsuperclass -- mixin

Findler & Flatt

Unit Extensibility: The Experience

• Linking with units is complex

• Units do not support multiple languages

• PLT also introduced modules.

Component-Oriented Programming

• modules for multi-lingual programs

• units for complex linking

• mixins for specifying and working with data hierarchies plus implementation inheritance

Components in Languages: Research Problems

• still complex: can we simplify this world?

• practical experience:– units and modules “feel” inside out

• type systems: what will a flexible type system for units and modules look like? – mixins (Krishnamurthi)– sharing by specification (Flatt)

Virtual Machines for COP: Resource Administration

with Matthew Flatt, Robby Findler, Shriram Krishnamurthi

Resource Administration: The Situation

• a dynamic plug-in component is probably third-party code with unknown properties

• several plug-ins may run in parallel

• plug-ins compete for finite resources:– files, database servers– network connections– GUI items

Resource Administration: Example 1

• a Web server responds to many requests simultaneously; each may spawn a Web program:– sharing (communicating) with server– sharing among “scripts”

• a script consumes– time– memory– (references to) server stacks– ports, files, databases, …

Resource Administration: Example 2

• DrScheme runs student programs as dynamic extensions so that tools can directly communicate with executing programs

• a program “consumes”– time– memory– GUI windows, … – ports, files, databases, …

Resource Administration: Example 3

• Netscape runs Java applets as dynamic extensions so that …

• a program “consumes”– time– memory– GUI windows, … – ports, files, databases, …

Resource Administration: PLs as OSes

• an extensible component system must play OS for its plug-ins

• the underlying VM must provide mechanisms for resource control

• the underlying programming language must “dress” these mechanisms as language constructs

Resource Administration: PLT’s Solution

• dynamic components as threads

• threads are a resource

• all resources are managed in a custodian

Resource Administration: Custodians

current custodian:

thread 1

thread 2

thread 3

network port 1

network port 2

custodianthread 4 file port 3

GUI 1

custodian shutdown

Resource Administration: Experience

• custodians work well in DrScheme and the Web server

• “special” custodians for GUI because we need to manage GUI events (event spaces)

Resource Administration: Problems

• Custodians don’t deal with memory. (Flatt)

• Custodian actions are fixed. Can we specify them?

• Other OS services?

• Update code dynamically? (Krishnamurthi)

Resource Administration

• In the past, advanced languages provided memory administration (garbage collection). Java has introduced GC into the mainstream.

• In the future, advanced languages will provide other resource administration tools. .NET will provide meta-data with assemblies.

Enforcing Invariants in COP

with Robby Findler, Mario Latendresse

Contract for Components: The Problem

The System A Plugin

invariants

Contract for Components: The Problem

The System A Plugin

invariants

The System consists of many components.

If it blows up, who did it? Economics!

Contracts for Components: The Idea

• need behavioral contracts in addition to type signatures for external wires of components

• must monitor these assertions at run-time (or partially verify them)

• assign blame to faulty component

Contracts for Components:Eiffel

• design by contract a la– Eiffel, iContract, jContractor, HandShake,

JVMAssert, …

• get it right, scale to other constructs

• then scale to COP

Contracts for Components: Example

double square_root(double x) { … } @pre: x >= 0 @post: fn result -> result >= 0

double square_root(double x) { … } @pre: x >= 0 @post: fn x -> fn result -> x *x >= result >= 0

partial behavior:

total behavior:

Contracts for Components: Meaning and Blame

• test pre-conditions for method call, blame caller

• test post-condition for method return, blame callee

• deal with invariants for variables similarly

Contracts for Components: Classes and Objects

• classes (or mixins) are in an inheritance hierarchy

• method contracts must be related

• test implications for pre/post conditions at method call & return, blame class extension

Contracts for Components: Functions

• testing predicates on functions is impossible

• use proxy functions that perform tests for first-order types, blame current co-variant party for bad arguments and contra-variant party for bad result

• switch blame positions with applications

Contracts for Components: Experience

• contracts are as important as types

• contracts complement types:– properties of arguments (subset of types)– relating two arguments– relating two functions in a component

Contracts for Components

• OOP and contracts

• FP and contracts

• COP and contracts: – types– linking: graphs, dynamic, hierarchical

Summary and Advice

Software Systems

• build large systems from components– in-house– third party– different sizes

• plug-in components statically

• plug-in components at run-time

Software Systems

• systems have complex interactions

• systems act as quasi-OS’es for dynamic plug-ins

• such systems consist of components in many languages

Programming Languages

• provide VMs that support high-level values and value sharing

• provide VMs that support resource administration for non-collectible resources

• provide VMs that accommodate many languages

Programming Languages

• provide constructs for constructing and linking components of all sizes and all shapes

• provide constructs for dynamic extensions, support mini-OSes for COP

Programming Languages

• provide interfaces with contracts

• check contracts in a well-founded manner

• assign blame properly

PLT’s Solutions

• modules, units, & mixins are a good solution, but not the last word

• custodians are a good start toward resource administration, but not the last word

• and this slide isn’t the last word

Advice for Academics: PL Research

• build systems, not (just) compilers:• useful and team-sized, e.g., DrScheme • useful and pair-sized, e.g., Web server• useful individual scripts• get “customers”

• test academic languages with all kinds of system; use feedback for PL design

Advice for Industry

• the JVM was a start; it does not accommodate other languages easily, it does not administrate resources

• the .NET world is made for other languages; it still lacks resource administration

• these machines are not the last word, they are just a start for COP

now go forth and do some more type research