merijn de jonge ipa spring days 2005 april 1, 2005 build-level components

Merijn de JongeIPASpring Days 2005April 1, 2005

Build-level Components

IPA Spring Days Merijn de Jonge 2Build-level ComponentsApril 1, 2005

Overview

•Introduction/motivation

•Build-level components

•Build-level composition

•Decoupling into build-level components

•Concluding remarks

IPA Spring Days Merijn de Jonge 3Build-level ComponentsApril 1, 2005

Overview

•Introduction/motivation

•Build-level componentsBuild-level components

•Build-level compositionBuild-level composition

•Decoupling into build-level componentsDecoupling into build-level components

•Concluding remarksConcluding remarks

IPA Spring Days Merijn de Jonge 4Build-level ComponentsApril 1, 2005

Mozilla

IPA Spring Days Merijn de Jonge 5Build-level ComponentsApril 1, 2005

Mozilla Directory Structure

•One very large directory tree•26,000 files•2,500 subdirectories•240 file types

•One very large directory tree•26,000 files•2,500 subdirectories•240 file types

IPA Spring Days Merijn de Jonge 6Build-level ComponentsApril 1, 2005

Mozilla Directory References

•1,521 directory references•Avg. fan-in/fan-out 12•Max fan-in=100•Max fan-out=42•Median fan-in=5•Median fan-out=12

•1,521 directory references•Avg. fan-in/fan-out 12•Max fan-in=100•Max fan-out=42•Median fan-in=5•Median fan-out=12

IPA Spring Days Merijn de Jonge 7Build-level ComponentsApril 1, 2005

Build-level

•File (atomic entity)– Source– Documentation

•Directory tree (container) = source tree

•Build process– To build/install software– Driven by make/ANT/…

•Configuration process– To control build process– Driven by configure/configuration files/…

•Build-level interfaces– Build interface– Configuration interface

IPA Spring Days Merijn de Jonge 8Build-level ComponentsApril 1, 2005

Mozilla Build Level

•1,350 Makefiles•40,000 LOC build instructions•16,000 LOC configuration•Cyclic dependencies•Two-phase build process•Centralized build/configuration knowledge•Code duplication•Component implementations scattered

•1,350 Makefiles•40,000 LOC build instructions•16,000 LOC configuration•Cyclic dependencies•Two-phase build process•Centralized build/configuration knowledge•Code duplication•Component implementations scattered

IPA Spring Days Merijn de Jonge 9Build-level ComponentsApril 1, 2005

Wishes

•Independent deployment (of build-level artifacts)–No circular dependencies–Individual build processes–Individual configuration processes

•Automated composition–Standardized build interface–Standardized configuration interface–Standardized binding mechanism for build-level variability and dependencies

IPA Spring Days Merijn de Jonge 10Build-level ComponentsApril 1, 2005

Idea

Apply CBSE principles to the build level

•Make build-level artefacts available for reuse by decomposing software systems into build-level components

•Make build-level component usable by assembling them into software systems using automated composition techniques

IPA Spring Days Merijn de Jonge 11Build-level ComponentsApril 1, 2005

Overview

•Introduction/motivationIntroduction/motivation

•Build-level components

•Build-level compositionBuild-level composition

•Decoupling into build-level componentsDecoupling into build-level components

•Concluding remarksConcluding remarks

IPA Spring Days Merijn de Jonge 12Build-level ComponentsApril 1, 2005

Component Definition (Szyperski)

A software component is

•A unit of composition …

•with contractually specified interfaces …

•and explicit context dependencies only.

•It is subject to independent deployment …

•and to third-party composition.

IPA Spring Days Merijn de Jonge 13Build-level ComponentsApril 1, 2005

Standardized Build Interface

•Different build systems exist, e.g., – make, ANT, shell scripts, IDE

•Different software systems require different build actions, e.g.,

– make– make bootstrap, make, make install

make cleanmake allmake checkmake (un)installmake dist

make cleanmake allmake checkmake (un)installmake dist

•Standardized build interface to enable that software construction can be controlled via a standard set and sequence of build actions:

IPA Spring Days Merijn de Jonge 14Build-level ComponentsApril 1, 2005

Standardized Configuration Interface

•Different mechanisms exist to control software construction, e.g.,–configuration files, configuration tools, Makefile editing

•Standardized configuration interface to enable uniform compile-time configuration:

configure --helpconfigure --prefix=/usrconfigure --with-aterm=/usr/libconfigure --with-optimization=trueconfigure --with-debug=false

configure --helpconfigure --prefix=/usrconfigure --with-aterm=/usr/libconfigure --with-optimization=trueconfigure --with-debug=false

IPA Spring Days Merijn de Jonge 15Build-level ComponentsApril 1, 2005

Explicit Context Dependencies

•Dependencies on build-level components are declared in configuration interfaces

> configure --help … --with-aterm=DIR use ATerm Library at DIR --with-sglr=DIR use SGLR Parser at DIR …

> configure --help … --with-aterm=DIR use ATerm Library at DIR --with-sglr=DIR use SGLR Parser at DIR …

IPA Spring Days Merijn de Jonge 16Build-level ComponentsApril 1, 2005

Independent Deployment

•Build-level components are deployed as packages

•A package is a versioned release of a build-level component, e.g.,

firefox-1.0.tar.gz

•Packages are published on web/ftp sites

IPA Spring Days Merijn de Jonge 17Build-level ComponentsApril 1, 2005

Third-party Composition

•A configuration interface enables late-binding of dependencies

•Compositions are thus not predefined

•and can be defined by a third party

> configure --with-aterm=/usr/local/aterm --with-sglr=/usr/local/sglr --with-…=

> configure --with-aterm=/usr/local/aterm --with-sglr=/usr/local/sglr --with-…=

IPA Spring Days Merijn de Jonge 18Build-level ComponentsApril 1, 2005

Overview

•Introduction/motivationIntroduction/motivation

•Build-level componentsBuild-level components

•Build-level composition

•Decoupling into build-level componentsDecoupling into build-level components

•Concluding remarksConcluding remarks

IPA Spring Days Merijn de Jonge 19Build-level ComponentsApril 1, 2005

Architectural Description Language

•Package Definition Language (PDL)–Domain-specific language designed specially for build-level component composition

–Used for initial implementations–No explicit provides interfaces

•Koala–Re-use of existing composition language–Implementation is work in progress–Multiple provides interfaces–No version resolution

IPA Spring Days Merijn de Jonge 20Build-level ComponentsApril 1, 2005

Koala Concepts for Build-level Composition

•Interface definitions–To group build-level artefacts into types

•Explicit provides interfaces–To declare the type(s) of a component

•Diversity interfaces–To declare build-level variation points

•Module as atomic entity–To declare the implementation of a component

IPA Spring Days Merijn de Jonge 21Build-level ComponentsApril 1, 2005

Koala Definition for Build-level Componentcomponent Sglr{ provides ISglr p; requires IATerm aterm; IToolbusLib libtb; IErrorSupport error_support; IPtSupport pt_support; contains module m url="http://www.cwi.nl/sglr-3.13.tar.gz"; connects p = m; m = aterm; m = libtb; m = error_support; m = pt_support;}

component Sglr{ provides ISglr p; requires IATerm aterm; IToolbusLib libtb; IErrorSupport error_support; IPtSupport pt_support; contains module m url="http://www.cwi.nl/sglr-3.13.tar.gz"; connects p = m; m = aterm; m = libtb; m = error_support; m = pt_support;}

IPA Spring Days Merijn de Jonge 22Build-level ComponentsApril 1, 2005

Mapping Koala to Build-level Concepts

component Sglr{ provides ISglr p; requires IATerm aterm; IToolbusLib libtb; IErrorSupport error_support; IPtSupport pt_support; contains module m url="http://www.cwi.nl/sglr-3.13.tar.gz"; connects p = m; m = aterm; m = libtb; m = error_support; m = pt_support;}

component Sglr{ provides ISglr p; requires IATerm aterm; IToolbusLib libtb; IErrorSupport error_support; IPtSupport pt_support; contains module m url="http://www.cwi.nl/sglr-3.13.tar.gz"; connects p = m; m = aterm; m = libtb; m = error_support; m = pt_support;}

> configure --help

…

--with-aterm=DIR use ATerm component at DIR

--with-libtb=DIR use LIBTB component at DIR

--with-error_support=DIR use ERROR_SUPPORT component at DIR

--with-pt_support=DIR use PT_SUPPORT component at DIR

…

> configure --help

…

--with-aterm=DIR use ATerm component at DIR

--with-libtb=DIR use LIBTB component at DIR

--with-error_support=DIR use ERROR_SUPPORT component at DIR

--with-pt_support=DIR use PT_SUPPORT component at DIR

…

> curl http://www.cwi.nl/sglr-3.13.tar.gz | tar xfz -> curl http://www.cwi.nl/sglr-3.13.tar.gz | tar xfz -

> configure --with-aterm=<aterm-binding> \

--with-libtb=<libtb-binding> \

--with-error_support=<error_support-binding> \

--with-pt_support=<pt_support-binding>

> configure --with-aterm=<aterm-binding> \

--with-libtb=<libtb-binding> \

--with-error_support=<error_support-binding> \

--with-pt_support=<pt_support-binding>

IPA Spring Days Merijn de Jonge 23Build-level ComponentsApril 1, 2005

•Koala source tree composition

•Checkout www.program-transformation.org/Tools/KoalaCompiler

Implementation

ParsingParsing Realization

RealizationNormalizationNormalization

generic

Component

definition

PackageComponent

definition

IPA Spring Days Merijn de Jonge 24Build-level ComponentsApril 1, 2005

Unbound Diversity/Requires Interfaces

•Propagate upwards to top-level configuration interface

r

interface IConf {

string foo=“some_value”;

string bar;

}

interface IConf {

string foo=“some_value”;

string bar;

}

> configure --help

--with-bar=VAL bind bar to VAL

--with-r=DIR use R component at DIR

> configure --help

--with-bar=VAL bind bar to VAL

--with-r=DIR use R component at DIR

> configure --with-foo=“some_value”> configure --with-foo=“some_value”

> configure --with-foo=“some_value” \

--with-bar=bar_binding() \

--with-r=r_binding()

> configure --with-foo=“some_value” \

--with-bar=bar_binding() \

--with-r=r_binding()

rr

IPA Spring Days Merijn de Jonge 25Build-level ComponentsApril 1, 2005

Online Package Base

www.program-transformation.org/package-basewww.program-transformation.org/package-base

IPA Spring Days Merijn de Jonge 26Build-level ComponentsApril 1, 2005

Overview

•Introduction/motivationIntroduction/motivation

•Build-level componentsBuild-level components

•Build-level compositionBuild-level composition

•Decoupling into build-level components

•Concluding remarksConcluding remarks

IPA Spring Days Merijn de Jonge 27Build-level ComponentsApril 1, 2005

Three Phases of Source Tree Decoupling

1. Source tree analysis

2. Source tree transformation

3. Online package base creation

Performed successfully for graphviz (www.graphviz.org)

IPA Spring Days Merijn de Jonge 28Build-level ComponentsApril 1, 2005

Source Tree Analysis

•Find components

•Find component references

•Fine tune–Add missing arrows–Remove cycles–Combine nodes with strong cohesion

IPA Spring Days Merijn de Jonge 29Build-level ComponentsApril 1, 2005

•Create components

•Create component definitions

•Fine tune

Source Tree Transformation

sglr-3.13.tar.gzsglr-3.13.tar.gz

IPA Spring Days Merijn de Jonge 30Build-level ComponentsApril 1, 2005

Decoupling Mozilla

•Cyclic dependencies

•Centralized build/configuration knowledge

•Cyclic dependencies

•Centralized build/configuration knowledge

IPA Spring Days Merijn de Jonge 31Build-level ComponentsApril 1, 2005

Dealing with Cyclic Dependencies

•108 potential build-level components•4 cyclic-dependent clusters•37 non-cyclic components

•108 potential build-level components•4 cyclic-dependent clusters•37 non-cyclic components

IPA Spring Days Merijn de Jonge 32Build-level ComponentsApril 1, 2005

Dealing with Cyclic Dependencies

•For each cluster create a composite component and a Koala component definition

•One provides interface for each sub-component

•On-demand refactoring to remove cyclic dependencies

•If all cyclic dependencies for a sub-component are removed, the component can be separated

IPA Spring Days Merijn de Jonge 33Build-level ComponentsApril 1, 2005

Dealing with centralized build/configuration

•Build knowledge–Duplicate build system for each component, or …

–Generate (Automake) build process, or …–Dedicated component with build knowledge

•Configuration knowledge–Duplicate for each component,–Prune to remove irrelevant code

IPA Spring Days Merijn de Jonge 34Build-level ComponentsApril 1, 2005

Overview

•Introduction/motivationIntroduction/motivation

•Build-level componentsBuild-level components

•Build-level compositionBuild-level composition

•Decoupling into build-level componentsDecoupling into build-level components

•Concluding remarks

IPA Spring Days Merijn de Jonge 35Build-level ComponentsApril 1, 2005

Concluding Remarks

•Reusable functionality is often not independently deployable due to build-level deficiencies

•CBSE principles help to decouple source trees into build-level components

•Composition of build-level components can be automated

•Introducing CBSE principles can be (partly) automated