June 1998 CHAIMS 1

Compiling High-level AccessInterfaces for Multi-site Software Stanford University

Objective: Investigate revolutionary approaches to large-scale software composition.

Approach: Develop and validate a composition-only language.

Planned contributions: • Asynchrony by splitting up CALL-statement.• Hardware and software platform independence.• Potential for multi-site dataflow optimization.• Performance optimization by invocation scheduling.

CHAIMS: Mega-Programming Research

CHAIMS

June 1998 CHAIMS 2

Participants• Support

– DARPA ISO EDCS program (1996-1999)– Siemens Corporate Research (1996-1998)– DoD AFOSR AASERT student support (1997-1999)– Sloan Foundation - computer industry study (1996-97)

• People - Gio Wiederhold PI -- Marianne Siroker: Administration– Dorothea Beringer (postdoc EPF Lausanne) since Dec.1997– Ron Burback (CS PhD cand.)– Laurence Melloul (CS MS)– Woody Pollack (CS MS)– MS and BS CS graduated: Joshua Hui, Gaurav Bhatia, Prasanna

Ramaswami, Kirti Kwatra, Pankaj Jain, Mehul Bastawala, Catherine Tornabene, Wayne Lim, Connan King

– Louis Perrochon (postdoc ETH Zurich) Fall quarter 1996

June 1998 CHAIMS 3

My Personal Background• Masters in Computer Science: hybrid-monitoring tool

for debugging and software performance analysis for distributed software

• Software engineer: telecommunication systems

• Consultant: software methodologies, quality assurance, project management, CASE-tools

• PhD: Modeling scenarios in object-oriented analysis

• Teaching: Fusion

• Now: CHAIMS -- large-scale software composition, distributed systems

June 1998 CHAIMS 4

Presentation• Motivation and Objectives

– changes in software production– basis for new visions and education

• Concepts of CHAIMS– CHAIMS language– CHAIMS architecture and composition process– Scheduling– Dataflow optimization

• Status, Plans, Conclusions

June 1998 CHAIMS 5

Coding

Integration

1970 1990 2010

Shift in Programming Tasks

June 1998 CHAIMS 6

Typical Scenario: LogisticsA general has to ship troops and/or various material from L.A. to Washington DC:

– different kind of material: criteria for preferred transport differ– not every airport equally suited– congestion, prices– weather constraints– exact due or ready dates

Today: calling different companies, looking up information on the web, reservations by hand

Future: system proposes possibilities that take into account various conditions

» hand-coded systems» composition of processes

June 1998 CHAIMS 7

Scaling alternatives

?

June 1998 CHAIMS 8

C H A I M S

Megamodules

Megaprogram for composition, written by domain programmer

CHAIMS system automates generation of client for

distributed system

Megamodules, provided by various megamodule

providers

CHAIMS

June 1998 CHAIMS 9

Megamodules - DefinitionMegamodules are large, autonomous, distributed,

heterogeneous services or processes.• large: computation intensive, data intensive, ongoing processes

(monitoring services)• distributed: to be used by more than one client• heterogeneous: accessible by various distribution protocols (not

only different languages and systems)• autonomous: maintenance and control over recourses remains

with provider, differing ontologies ( ==> SKC)

Examples:– logistics: “find best transportation route from A to B”,

reservation systems– genomics: easier framework for composing various processing

tools than ad-hoc coding

June 1998 CHAIMS 10

Challenge: Thin Clients (1) Domain expert

Client computer

Control &Computation

Services

I/O

a bcd

e

Wrappers to resolve

differences

I/O

DataResources

June 1998 CHAIMS 11

Challenge: Thin Clients (2) Domain expert

Client workstation

ComputationServices

IO module

MEGA modules

IO module

ab

cd

e

DataResources

Sites RT

S U T

C

June 1998 CHAIMS 12

Challenge: Heavy-weight Services

Services are not free for a client:• execution time of a service• transfer time for data• fees for services

What we need:==> monitoring progress of a service==> possibility to choose among equivalent services

based on estimated waiting time and fees==> parallelism among services==> preliminary overview results, choosing level of

accuracy / number of results for complex processes==> novel optimization techniques

June 1998 CHAIMS 13

Challenge:Non-technical Domain Experts

Company providing services:• domain experts of domain of service (e.g. weather)• technical experts for programming for distribution

protocols, setting up servers in a middleware system• marketing experts

“Megaprogrammer”:• is domain expert of domain that uses these services• is not technical expert of middleware system or

experienced programmer,• wants to focus on problem at hand (=results of using

megaprogram)• e.g. scientist, assistant of a general

June 1998 CHAIMS 14

Challenge: Purely Compositional Language Possible?

Which languages did succeed?– Algol, ADA: integrated composition and computation– C, C++: focus on computation

Why new language?– complexity: not all facilities of a common language (compare

to approach of Java), – inhibiting traditional computational programming (compare C+

+ and Smalltalk concerning object-oriented programming)– focus on issue of composition, parallelism by asynchrony,

and optimization

June 1998 CHAIMS 15

CHAIMS “Logical” Architecture

Customer

Megaprogramclients(in CHAIMS)

Network/Transport(DCE, CORBA,...)

Megamodules(Wrapped or Native)

June 1998 CHAIMS 16

CHAIMS Physical Architecture

NetworkDCE, CORBA, JAVA RMI, DCOM...

MegaprogramClients in CHAIMS

Megamodules (wrapped, native) each supportingsetup, estimate, invoke, examine, extract, and terminate.

June 1998 CHAIMS 17

Decomposing CALL statements

Copying

Code sharing

Parameterized computation

Objects with overloaded method names

Remote procedure calls to distributed modules

Constrained (black box) access to encapsulated data

progressin

scale ofcomputing

ExtractInvokeEstimate ExamineSetup

CHAIMSdecomposes CALL functions

CALL gainedfunctionality

June 1998 CHAIMS 18

CHAIMS PrimitivesPre-invocation:

SETUP: set up the connection to a megamoduleSET-, GETATTRIBUTES: set global parameters in a megamoduleESTIMATE: get estimate of execution time for optimization

Invocation and result gathering:INVOKE: start a specific methodEXAMINE: test status of an invoked methodEXTRACT: extract results from an invoked method

Termination:TERMINATE: terminate a method invocation or a connection to

a megamodule

Control: Utility:WHILE, IF GETPARAM: get default parameters

June 1998 CHAIMS 19

Megaprogram Example: Overview

InputOutput- Input- Output

RouteInfo- AllRoutes- CityPairList- ...

AirGround- CostForGround- CostForAir- ...

Routing- BestRoute- ...

RouteOptimizer- Optimum- ...

General I/O-megamodule» Input function takes as parameter a default

data structure containing names, types and default values for expected input

Travel information:» Computing all possible routes between two

cities» Computing the air and ground cost for each

leg given a list of city-pairs and data about the goods to be transported

Two megamodules that offer equivalent functions for calculating optimal routes

» Optimum and BestRoute both calculate the optimum route given routes and costs

» Global variables: Optimization can be done for cost or for time

June 1998 CHAIMS 20

Megaprogram Example: Codeio_mmh = SETUP ("InputOutput")route_mmh = SETUP ("RouteInfo")...best2_mmh.SETATTRIBUTES (criterion = "cost")

cities_default = route_mmh.GETPARAM(Pair_of_Cities)input_cities_ih = io_mmh.INVOKE ("input”, cities_default)WHILE (input_cities_ih.EXAMINE() != DONE) {}cities = input_cities_ih.EXTRACT()...route_ih = route_mmh.INVOKE ("AllRoutes", Pair_of_Cities = cities)WHILE (route_ih.EXAMINE() != DONE) {}routes = route_ih.EXTRACT() …

IF (best1_mmh.ESTIMATE("Best_Route") < best2_mmh.ESTIMATE("Optimum") ) THEN {best_ih = best1_mmh.INVOKE ("Best_Route", Goods = info_goods, Pair_of_Cities = cities, List_of_Routes = routes, Cost_Ground = cost_list_ground, Cost_Air = cost_list_air)}ELSE {best_ih = best2_mmh.INVOKE ("Optimum", Goods = info_goods, …...best2_mmh.TERMINATE()

// Setup connections to megamodules.

// Set global variables valid for all invocations // of this client.

// Get information from the megaprogram user // about the goods to be transported and about// the two desired cities.

// Get all routes between the two cities.

//Get all city pairs in these routes.//Calculate the costs of all the routes.

// Figure out the optimal megamodule for// picking the best route.

//Pick the best route and display the result.

// Terminate all invocations

June 1998 CHAIMS 21

Operation of one Megamodule

• SETUP

• SETATTRIBUTES provides context

• ESTIMATE serves scheduling

• INVOKE initiates remote computation

• EXAMINE checks for completion

• EXTRACT obtains results

• TERMINATE I / ALL

M handle

M handle

M handle

M handle

I handle

I handle

I handle

M handle

I handle

June 1998 CHAIMS 22

CHAIMS Megaprogr. Language

Purely compositional:– no primitives for arithmetic ==> math megamodules– no primitives for input/output ==> general and problem-

specific I/O megamodules

Splitting up CALL-statement:– parallelism by asynchrony in sequential program– novel possibilities for optimizations– reduction of complexity of invoke statements

• higher-level language (assembler => HLLs, HLLs => composition/megamodule paradigm)

June 1998 CHAIMS 23

Architecture: Runtime

ed

a

b

c

Distribution System (CORBA, RMI…)

CSRT(compiled megaprogram)

MEGA modules

June 1998 CHAIMS 24

Architecture: Composition Process

ed

a

b

cMEGA modules

CHAIMS Repository

adds information to

MegamoduleProvider

wraps non-CHAIMScompliant megamodules

Wrapper Templates

June 1998 CHAIMS 25

writes

Architecture: Composition Process

Megaprogrammer

CSRT(compiled megaprogram)

Megaprogram(in CHAIMS language)

CHAIMS Compiler

generates

CHAIMS Repository

information

information

June 1998 CHAIMS 26

writes

Architecture: Overview

e

Megaprogrammer

d

a

b

c

Distribution System (CORBA, RMI…)

CSRT(compiled megaprogram)

Megaprogram(in CHAIMS language)

CHAIMS Compiler

generates

MEGA modules

CHAIMS Repository

adds information to

MegamoduleProvider

wraps non-CHAIMScompliant megamodules

information

information

Wrapper Templates

June 1998 CHAIMS 27

writes

Architecture: Overview

e

Megaprogrammer

d

a

b

c

Distribution System (CORBA, RMI…)

CSRT(compiled megaprogram)

Megaprogram(in CHAIMS language)

CHAIMS Compiler

generates

MEGA modules

CHAIMS Repository

adds information to

MegamoduleProviders

wraps non-CHAIMScompliant megamodules

information

information

Wrapper Templates

run-time

June 1998 CHAIMS 28

Architecture: CHAIMS-Language and CHAIMS-Protocols

Megaprogram

Megaprogrammer

M e g a m o d u l e s

CHAIMS-language

CHAIMS-protocolsCORBA-idl DCE-idl Java-class

CHAIMS API defines interface between megaprogrammer and megaprogram; the megaprogram is

written in the CHAIMS language.

The CHAIMS protocols define the calls the mega-modules have to understand. These protocols are slightly different for the different distribution protocols, and are defined by an idl for CORBA, another idl for DCE, and a Java class for RMI.

June 1998 CHAIMS 29

Name of Person

Architecture: GentypeA Gentype is a triple of name, type and value, where

value is either a simple type or a list of other gentypes (i.e. a complex type).

Possible simple types: given by ASN.1, the ASN.1-conversion library for C++, our own conversion routines.

Example: Person_Information complex

First Name string Joe Last Name string Smith

Personal Data complex Address

Date of Birth date 6/21/54 Soc.Sec.No string 345-34-345

June 1998 CHAIMS 30

Wrapper: CHAIMS Compliance

• CHAIMS protocol - knowing all CHAIMS primitives

• State management and asynchrony: » clientId (megamodule handle in CHAIMS language)» callId (invocation handle in CHAIMS language)» results must be stored for possible extraction(s) until

termination of the invocation

• Data transformation: » all parameters of type blob (BER-encoded Gentype) must

be converted into the megamodule specific data types (combination hand-coding/decoding routines

June 1998 CHAIMS 31

Architecture: Three Views

Transportation View moving around data blobs and CHAIMS messages

Composition View (megaprogram)

- composition of megamodules

- directing of opaque data blobs

Data View - exchange of data - interpretation of

data - in/between

megamodules

CHAIMS Layer

Distribution Layer

Objective: Clear separation between composition of services, computation of data, and transport

June 1998 CHAIMS 32

execution of a remote method

synchronous

invoke a methodie extract results

setup / set attributes s

s

e

i

time

decomposed(no benefit for one module)

asynchronous

s,i

time

etim

e

available for other methods

e

s,i

Scheduler: Decomposed Execution

June 1998 CHAIMS 33

Scheduler: Optimized Execution

M1 M4(<M1+M2)

M5

M2M3

(>M1+M2)

i1e1

e4e3

e2

i3i4

i5

i2

e5

time

M1

M4

M5

M2

M3

i1

e1

e2

e3

e4

e5

i2

i3

i4

i5

time

data dependenciesexecution of a method

non-optimized, orhandprogrammed

optimized by scheduleraccording to estimates

invoke a methodie extract results

set up / set attributes (not shown)s

June 1998 CHAIMS 34

Scheduling: Simple Example

1 cost_ground_ih = cost_mmh.INVOKE ("Cost_for_Ground", 1 List_of_City_Pairs = city_pairs,Goods = info_goods)

2 WHILE (cost_ground_ih.EXAMINE() != DONE) {} 3cost_list_ground = cost_ground_ih.EXTRACT()

3 cost_air_ih = cost_mmh.INVOKE ("Cost_for_Air", 2 List_of_City_Pairs = city_pairs,Goods = info_good)

4 WHILE (cost_air_ih.EXAMINE() != DONE) {} 4cost_list_air = cost_air_ih.EXTRACT()

order inunscheduledmegaprogram

order in automaticallyscheduled megaprogram

June 1998 CHAIMS 35

Scheduling: Possible ActionsINVOKES: call INVOKE’s as soon as possible

» may depend on other data» moving it outside of an if-block: depending on cost-

function (ESTIMATE of this and following functions concerning execution time, dataflow and fees (resources).

EXTRACT: move EXTRACT’s to where the result is actually needed

» no sense of checking/waiting for results before they are needed

» instead of waiting, polling all invocations and issue next possible invocation as soon as data could be extracted

TERMINATE: terminate invocations that are no longer needed (save resources)

» not every method invocation has an extract (e.g. print-like functions)

June 1998 CHAIMS 36

Compiling into a NetworkMega Program

Module A

Module B

Module CModule E

Module DModule F

current CHAIMS systemMega Program

Module DModule F

control flow data flow

with automatic dataflow optimization

Mega Program

Module A

Module B

Module C Module E

Module DModule F

June 1998 CHAIMS 37

CHAIMS Implementation• Specify minimal language

– minimal functions: CALLs, While, If *– minimal typing {boolean, integer, string, handles, object}

» objects encapsulated using ASN.1 standard– type conversion in wrappers, service modules*

• Compiler for multiple protocols (one-at-time, mixed*)• Wrapper generation for multiple protocols

• Native modules for I/O, simple mathematics*, other• Implement API for CORBA, Java RMI, DCE usage

• Wrap / construct several programs for simple demos

• Schedule optimization *• Demonstrate use in heterogeneous setting *• Define full-scale demonstration * in process

June 1998 CHAIMS 38

Concept Status• Definition of architecture for Megaprogramming

– bottom up assessment of code to be generated• examples: room reservation, shipping

– primitives– handles for parallel operation– heterogeneity -- common features of distribution protocols

• Minimal language that can generate the code– no versus very few types -- ASN.1 for complex types– natural parallelism -- still a major research issue

• Awareness of novel optimizations– information flow constraints -- scheduling– direct data flow between megamodules

June 1998 CHAIMS 39

Focus for Future

• Finishing basic infrastructure and demo examples.

• CHAIMS interpreter instead of CHAIMS compiler.

• Scheduling of invocations and extractions.

• Flexible interaction with megamodules; extracting and handling overview results.

• Direct dataflows between megamodules (future project).

June 1998 CHAIMS 40

CHAIMS

Conclusion: Research Questions• Is a Megaprogramming language focusing only on

composition feasible?• Can it exploit on-going progress in client-server

models and be protocol independent? • Can natural parallelism for distributed services be

effectively scheduled?• Can high-level dataflow among distributed modules

be optimized?• Can CHAIMS express clearly a high-level distributed

SW architecture?• Can the approach affect SW process concepts and

practice?

June 1998 CHAIMS 41

Paying for SW ServicesYou can not run an effective (SW) business and not be

reimbursed for it. How? Four approaches:– Sell Softwaresell oilfield to customer– Lease copy / usage rights lease well– Time / user limited access fill tank– Charge by use instance provide bus

General problems, effects differ– IP protection? – keeping SW updated – billing for est.value – performance effect

poor some fair good

poor ok good good

simple simple awkw. hard

no no little some

Buy Lease Limit Use

protect

update

bill

perform

June 1998 CHAIMS 42

Conclusion: Questions not addressed

• Will one Client/Server protocol subsume all others?– distributed optimization remains an issue

• Synchronization / Concurrency Control– autonomy of sources negates current concepts– if modules share databases, then database locks may

span setup/terminate all for a megaprogram handle.

• Will software vendors consider moving to a service paradigm?

– need CHAIMS demonstration for evaluation

June 1998 CHAIMS 43

Composition of Processes...• versus composition and integration of Data

» data-warehouses» wrapping data available on web

• versus composition of Components» reusing small components via copy/paste or shared

libraries locally installed» large distributed components within same “domain” as

composition, e.g. within one bank or airline

CHAIMS: » processed information

» composing autonomous execution threads

June 1998 CHAIMS 44

Summary• CHAIMS requires rethinking of many common

assumptions– gain understanding via simple examples

• Work focused on CALL statement decomposition– to accomplish integration of large services – exploit inherent asynchrony

• First version of architecture and language drafts are completed; basic infrastructure partially available (compiler, wrapper templates).

• More demos will come soon. Half-way through a four year project.

http://www-db.stanford.edu/CHAIMS

June 1998 CHAIMS 45

CHAIMS

June 1998 CHAIMS 46

Other systems (1)

Darwin/Regis:» configuration language, architecture definition language,

allowing various communication mechanisms (yet all in C++ and under local control)

» local (distributed) architecture, non-autonomous, system not only specifies, generates and controls composition, but also basic components

» only for homogeneous environment (C++), no legacy

Polylith/Polygen:» changes source-files (C, Ada, Pascal, Lisp) according to

annotated design data (architecture information) for distribution

» only for local distribution (access to source code) and only for TCP/IP

June 1998 CHAIMS 47

Other systems (2)Hadas:

» network centric framework with peer-to-peer communication» Java-based (RMI) with Java-wrappers for non-compliant

objects» concept of ambassador objects which are copied to the

peer object» allows administrative and design autonomy for objects

(heterogeneity, changes of methods)

KQML:» protocol for asynchronous communication among agents» allows explicitly different ontologies» offers primitives for getting and updating data, but also for

advertising available knowledge

June 1998 CHAIMS 48

Other systems (3)Differences:

» autonomous megamodules, legacy modules» higher-level (just invoke and extract, no specification of

various kinds of communication; no specification of module content)

» heavy-weight services: ESTIMATE, parallelism by automatic scheduling of invocations according to run-time estimates and cost-functions, EXAMINE, SETPARAM

» heterogeneity in distribution protocols

focus so far: heterogeneity in distribution protocol on client side

focus in future: using ESTIMATE and EXAMINE, automatic optimization and invocationscheduling

June 1998 CHAIMS 49

CHAIMS proves that...• We can do composition in a high-level language.

» same language for Java-RMI-invocations and CORBA-invocations (and DCE, DCOM, TCP/IP protocols)

» (single megaprogram can deal with multiple protocols simultaniously)

» multiple megamodules can run in parallel

• Large-scale composition can be automated.» in contrast to manual non-software composition (e.g. telephone,

cut&paste) » in contrast to fixed programs for one specific problem (e.g.

transporting military goods within US)

• We can do schedulings of programs in a way right now only smart logistics officers can do, avoiding unnecessary waits.

» Scheduling of invocations can be optimized.

June 1998 CHAIMS 50

Long-term Objectives of CHAIMS1 Implementing a system for a simple and purely

compositional language hiding differences of diverse protocols

2 Automatic optimized scheduling of invocations (taking advantage of inherent parallelism and estimate-capabilities of megamodules, hence splitting up of CALL-statement)

3 Decision-making support (direct) interaction with megamodules, based on overview and incremental results (fixed flow, not yet interactive changes to megaprogram)

4 Automatic dataflow optimization (direct dataflows between megamodules), not yet

June 1998 CHAIMS 51

Assumptions, Additional Constraints• Heterogenous legacy modules ==> wrapping of modules, mixing

protocols on client side or in wrappers.• Parallelism of megamodule-methods not through multithreading on

client side but through splitting up CALL-statement (==> sequential program on client side); this leads to useful parallelism because we deal with coarse-grain parallelism.

• CHAIMS-compliancy for megamodules is achieved by wrapper-templates, for new native megamodules as well as for legacy ones (CHAIMS-compliancy is more than just knowing CHAIMS-protocol!).

• No reliance on existence of one specific higher level protocol like CORBA, DCOM, RMI ==> implementing an independent data-encoding and marshalling with ASN.1, instead of using one of them and then having converters in the wrappers.

• Interfaces of megamodules match <==> no investigation into opaque datablobs on client side necessary.

• Thin client, client should be able to run anywhere (not quite fulfilled right now - we need local ORB, DCE, JavaVirtual-machine).

• Clear seperation client - server, minimal repository.

June 1998 CHAIMS 52

Non- (not yet)-Objectives of CHAIMS• No commercial product.• No specific controls over ilities (security, name-

serving, etc.) that they are normally present in distributed systems.

• No sophisticated front-end, no graphical programming/composition, no browser for repository, no higher-level language as input (not yet).

• Not solving all problems of megamodule composition that are mentioned in the various CHAIMS-papers (e.g. differing ontologies, non-matching interfaces of megamodules), only the ones mentioned in objectives and additional conditions.

June 1998 CHAIMS 53

Short-term Objectives of CHAIMSRest of 1998:

» Basic infrastructure (fixing most severe flaws, moving to consistent architecture, all primitives, types, associative lists with handling it, having CORBA) ==> conceptual and implementation work -- CONSOLIDATION

» More examples (descriptions of scenarios as well as implemented demos), wrapping one (maybe two) additional suites of megamodules. ==> implementation work -- CONSOLIDATION

» Mixing of protocols in client (CORBA, RMI) or/and TCP/IP-three-tier architecture

» Preparing for more capable scheduler (examples with current scheduler, reading about other scheduler-problems and implementations, redesigning architecture of compiler (interpreter?), designing scheduler algorithm and architecture, writing paper about all this…) ==> lots of conceptual work, some implementation -- looking ahead for better Scheduler

1999 (depending on where we are at the end of 1998):» Scheduler

June 1998 CHAIMS 54

Upcoming Changes to Architecture:Other Approach to Heterogeneity

Client (megaprogram)

nativeserver

1

nativeserver

3

nativeserver

2

chaimscompliantmodule

chaimsI/O

module

RMIwrapper

TCP/IP socketsCHAIMS protocol

CORBA RMIserver-specific protocols

site

s o

f se

rver

s

clientsite

differentwrapper site

RMIwrapperCORBAwrapper

June 1998 CHAIMS 55

Reasons Alternative ArchitectureOverall:• Simpler architecture: less wrappers, just one protocol on client side

Server-side:• No direct linking with legacy code also for CORBA-wrappers, different sites for wrapper

and legacy megamodule possible• All native CHAIMS-megamodules will be built using wrapper templates ==> no reason for

several protocols, they can all use TCP/IP.• Dataflow-optimization: direct messages between megamodules/their wrappers necessary

(without bridges)

Client-side:• Thin client that could run everywhere (TCP/IP is available everywhere, but not CORBA or

DCE, RMI also is easily available everywhere).• CSRT could be implemented by interpreter instead of compiler, maybe also possible with

current architecture, but more complex.• We use just transport-facility (really true? what about native CHAIMS-types like string,

integer, boolean?) of CORBA, RMI, DCE (for data we have ASN.1); this is already offered by TCP/IP ==> no unnecessary overkill

Drawback: missing one of the current funding objectives (heterogeinity on client side).

June 1998 CHAIMS 56

Upcoming Changes to Architecture:Interpreter instead of Compiler

megaprogramin CHAIMS language

client code in C, C++, Java,

stub code

executableclient (CSRT)

userCompiler:

CHAIMS executionmachine

(interpreter andscheduler)

userInterpreter:

networkCHAIMS-protocol

completemegaprogram

in CHAIMS language

some CHAIMSstatements

user

serves as input to

serve as input to

CHAIMS compiler,simple scheduler

Idl-file generatorand compiler

C++, Java compiler

and linkernetwork

June 1998 CHAIMS 57

Interpreter• Dynamic scheduler:

» Input is parsed and stored in a dependancy graph.» Execution machine (interpreter / scheduler) works through

the graph and makes appropriate calls:– estimate-calls are added in order to get necessary run-time information

for scheduling (cost-function)– every invocation is issued as soon as possible (data-flow) and

reasonable (according to cost-function)– all invocations for which the CSRT waits for results are polled regularly,

and results extracted and new invocations issued as soon as possible

CSRT would still be sequential!

• Overview results, flexible interactions:» megaprogrammer can program statement by statement and get results

immediately; results will influence what he/she does next» like ftp, web

June 1998 CHAIMS 58

Example: parameters of method xz

general default valuesclient-specific settings for client A

method xz, client A:invocation-specific settings

client-specific settings for client B

from deststart-date

end-date number

1JAN1998 1JAN1998

6OCT19984OCT1998

7OCT19984OCT1998

LAS BWI 1

2

actual values used in invocation xz1

SJO ZRH

SFO

1

1

SFO

SJO6OCT19984OCT1998 ZRH

ZRHactual values used in invocation xz2

FRA

7OCT1998

9DEC19981DEC1998

9DEC19981DEC1998 2LAS FRAactual values used in invocation xz1

method xz, client A:invocation-specific settings

method xz, client B:invocation-specific settings

Hierarchical Setting of Parameters

June 1998 CHAIMS 59

Using the General I/O-Megamodulefor Input

Corresponding code in CLAM:ex = SETUP("Example")io = SETUP("io") // io is the general I/O megamodule availabe in CHAIMS(myadress = adress) = ex.GETPARAM() io.INVOKE("ask", value = adress,text =

"Please enter adress:")ih1 = io.INVOKE("ask", value = myadress, text = "Please enter adress:") (myadress = result) = ih1.EXTRACTWHENDONE() ih2 = ex.INVOKE("method", adress = myadress)…..

Megaprogram General I/0Megamodule

MegamoduleExample

2. Invoke method "ask" with default values and type information

3. Extract user-input

4. Invoke desired method with user-input

1. Get default type and value for desired parameter

June 1998 CHAIMS 60

Using the General I/O-Megamodulefor Output

Corresponding code in CLAM:…………... // Setups, Invocations, ih1 is the handle of a

// method of megamodule examples(myadress = adress_result) = ih1.EXTRACTWHENDONE() ih2 = io.INVOKE("write", value = myadress, text ="Currently the adress is:")

Every element of the data structure in parameter adress_result has besides the actual value also type information and descriptive name. Therefore, the general I/O-megamodule can display the parameter in a decent format without any further formatting information.

Megaprogram General I/0Megamodule

MegamoduleExample

2. Invoke method "write" with result to be displayed

1. Extract result to be displayed

June 1998 CHAIMS 61

Primitives in PAM

Pre-invocation:SETUP: set up the connection to a megamoduleSET-, GETPARAM: preset parametersESTIMATE: get cost estimates for a specific method

Invocation and result gathering:INVOKE: start a specific methodEXAMINE: test status of an invoked methodEXTRACT: extract results from an invoked method

Termination:TERMINATE: terminate a method invocationTERMINATEALL: terminate the connection to a megamodule

June 1998 CHAIMS 62

Name of Person

GentypeA Gentype is a triple of descriptive name, type and

value, where value is either a simple type or a list of other gentypes (i.e. a complex type).

Possible simple types: given by ASN.1-library and our own conversion routines (integer, real, string, boolean, date-time, sequence of bytes)

Example: Person_Information complex

First Name string Joe Last Name string Smith

Personal Data complex Address

Date of Birth date 13JUN1934 Soc.Sec.No string 345-34-345

June 1998 CHAIMS 63

INVOKE in various protocolsRMI: public long INVOKE(long clientID, String methodName,

ChaimsAttrValContainer cont) throws RemoteException;

CORBA: void INVOKE ( in CHAIMSCLIENTID clientid,in CHAIMSMETHODNAME methodname, in CHAIMSATTRVALSEQ attrvalseq, out CHAIMSCALLID callid );

DCE: ...local C++: long Invoke (char* methodName, AttrValList * plist) local Java: public long INVOKE (String methodName,

ChaimsAttrValContainer cont);

• AttrValList, CHAIMSATTRVALSEQ and ChaimsAttrValContainer are sequences of name-value pairs, adapted to the various languages and systems.

• INVOKE for local protocols does not need a client ID because for each client there is one instance of the local megamodule.

June 1998 CHAIMS 64

Setting up connection in various protocols (client code RMI, Java)

RMI:System.setSecurityManager(new RMISecurityManager()); ChaimsCompliantObject remote;try{remote=(ChaimsCompliantObject)Naming.lookup(url+mmname);}catch (Exception e){}try {remote.SETUP(clientID);}catch (Exception e){}

local Java: Class cl; //Class to be loaded, for which an object will be instantiatedtry {cl = Class.forName(full_class_name);}catch (Exception e) {}ChaimsCompliantLocal local;try {local = cl.newInstance();}catch (Exception e) {}local.SETUP();

June 1998 CHAIMS 65

Setting up connection in various protocols (client code CORBA)

CORBA (Orbix, C++ client):nameMM_var _pObj0; try {_pObj0=nameMM::_bind(":mmname","site");} catch (...) {} CHAIMSMMID mmid = _pObj0; pObj0->SETUP(mmid); CORBA (Omnibroker, C++ client):CORBA_ORB_var orb = CORBA_ORB_init(argc,argv);// first get reference from a file and then convert it to an objectconst char* refFile_pObj0 = "io.ref"; ifstream in_pObj0;in_pObj0.open(refFile_pObj0); // error handling omitted herechar stre_pObj0[1000]; in_pObj0 >> stre_pObj0;CORBA_Object_var obj_pObj0 = orb -> string_to_object(stre_pObj0);assert(!CORBA_is_nil(obj_pObj0)); // bind now to megamoduleio_var _pObj0 = io::_narrow(obj_pObj0); assert(!CORBA_is_nil(_pObj0));CHAIMSMMID mmid = _pObj0;pObj0->SETUP(mmid);

June 1998 CHAIMS 66

Alternative Architecture:Other Approach to Heterogeneity

Client

nativeserver

1

nativeserver

3

nativeserver

2

chaimscompliantmodule

chaimsI/O

module

RMIwrapper

e.g. TCP/IP socketsPAM

CORBA RMIserver-specific protocols

site

s o

f se

rver

s

clientsite

differentwrapper site

RMIwrapperCORBAwrapper

June 1998 CHAIMS 67

Current Architecture: Several Implementations of PAM

Client

P A M - protocolsCORBA-idl DCE-idl Java-class

M e g a m o d u l e s

W r a p p e r s

June 1998 CHAIMS 68

CHAIMSDomain expert (composer)

writes megaprogram for composition.

CHAIMS automates generation of client for

distributed system and optimized scheduling.

Megamodule provider provides processed

knowledge.

CHAIMS

Megamodules

June 1998 CHAIMS 69

Client program

Domain expert / Composer

DistributedComputation

Services

IO module

Infrastructure: Remote, distributed, parallel computation of a, b, c, d, e

IO module

Wrapper

Legacy Module

ab c

de

Native CHAIMS Module

Control FlowData Flow

MEGA modules

June 1998 CHAIMS 70

e

Composer (Domain Expert)

d

a

b

c

Distribution System (CORBA, RMI…)

CSRT, incl. run-time scheduling

Megaprogram (written in CLAM)

writes

CHAIMS Compiler

generates

MEGA modules

CHAIMS Repository

adds information to

MegamoduleProvider

Wrapper Templates

wraps non-CHAIMScompliant megamodules

information

information

run-time execution

composition process

CPAM protocol

June 1998 CHAIMS 71

Exploiting parallelism of megamodule pieces

M1

M4(<M1+M2)

e

M3 (>M1+M2)

a

d(<a+b)

e

b

c (>a+b)

invoke a method

set up / setattributes

time

prior time

extract results

before piecewisedoptimization

June 1998 CHAIMS 72

Primitives of CLAM• Pre-invocation:• SETUP: set up the connection to a megamodule• SET-, GETPARAM: preset parameters in a megamodule• ESTIMATE: get cost estimation for optimization

• Invocation and result gathering:• INVOKE: start a specific method• EXAMINE: test status and progress of an invoked method• EXTRACT: extract results from an invoked method

• Termination:• TERMINATE: terminate a method invocation• TERMINATEALL: terminate a connection to a megamodule

• Control:• WHILE, IF

June 1998 CHAIMS 73

Exploit parallelisma d

(<a+b)

e

bc

(>a+b)

i1e1

e4e3

e2

i3i4

i5

i2

e5

time

a

d

e

b

c

i1

e1

e2

e3

e4

e5

i2

i3

i4

i5

time

data dependenciesexecution of a method

non-optimized, orhandprogrammed

exploring parallelismaccording to estimates and based on dataflow dependencies

invoke a methodie extract results

June 1998 CHAIMS 74

Dataflow optimizationClient program

DistributedComputation

Services

IO moduleIO module

a

b c

de

Control FlowData Flow

June 1998 CHAIMS 75

ChaimsCallRecord

callID clientID paramContmethodID resCont ObjectWrapperThreadstatus MethodWrapper

~ ~~ ~ ~ ~ ~~~

key

~ ~~ ~ ~ ~ ~~~n-1 ~~ ~ ~ ~ ~~~n

INVOKE( clientID, methodName,attrValContainer)

ChaimsThread

LocalObjectWrapper

ServerObject

MethodXWrapper

createNewCallEntry1

LocalMethodWrapper

start2

callLocalMethod3 callMethod4

methodX5 -> result6

setPartialResult7

setResultContainer8

setResultDone9

setStatus( DONE )10

June 1998 CHAIMS 76

SETUP( clientID )

addClientEntry

ChaimsClientRecord

clientID paramCont (default values)

~

key

~“PA58”

DEFAULT

(A, ) (B, ) (Z, ) (B, ) (C, )

(A, ) (B, ) (C, ) (Z, )

written by SETPARAM, read by GETPARAM

June 1998 CHAIMS 77

What’s the problem?

C++ Client

Java Client

Modules written inC++ or Java

Modules written inJava

CORBA

RMI

June 1998 CHAIMS 78

What’s the problem?

Client

RMI

CORBA

CORBA

Modules

June 1998 CHAIMS 79

Possible Solution: JNI

C++ MasterClient

JavaCode

CORBA

CORBA

RMI

JNI

June 1998 CHAIMS 80

Possible Solution: JNI

JavaMasterClient

C++Code

CORBA

CORBA

RMI

JNI

C++Code

JNI

June 1998 CHAIMS 81

CHAIMS Repository• CHAIMS repository data presently an ASCII file.• Each line conveys information about a specific

megamodule, method, parameter, or “synonym” pair.• C - style commenting ( /* … */ ).• Basically hierarchical model with certain exceptions.

June 1998 CHAIMS 82

CHAIMS Repository Model

June 1998 CHAIMS 83

CHAIMS repository browser: Window Layout #1

June 1998 CHAIMS 84

CHAIMS repository browser: Window Layout #2