©1998-2003, michael aivazis pyre an introduction to the integration framework michael aivazis...

©1998-2003, Michael Aivazis

Pyre

An introduction to the integration framework

Michael Aivazis

California Institute of Technology

DANSE Software Workshop

September 3-8, 2003

2


Outline

Overview of the framework

Crash course on python

Brief description of key infrastructure

3


Pyre: the integration framework

A simulation environment thatenables the non-expertdoes not hinder the expert

Target features for “end-users”:complete and intuitive simulation specificationreasonable defaults, consistency checks of input, good diagnosticseasy access to remote facilitiesstatus monitoring

Target features for “developers”:easy access to user inputshorter development cycle, good debugging support

Status as of March 1, 2003:Python scripts: 1300 classes, 70,000 lines of codeC++: 30000 lines of bindings and infrastructure

Largest run to-date:1764 processors, 2.7 mel solid fluid, 256x256x512 fluid~3k timesteps/24hrs~1.5 Tb of data

4


Distributed services

Workstation Front end Compute nodes

launcher

journal

monitoradlib

arm3d

5


Pyre Architecture

component

bindings

engine

solver

component

bindings

library

infrastructure

service

framework

service

serviceservice

component

bindings

engine

solver

abstract classabstract class

specializationspecialization

packagepackage

The integration framework is a set of co-operating abstract services

FORTRAN/C/C++FORTRAN/C/C++

pythonpython

6


Flexibility through the use of scripting

Scripting enables us toOrganize the large number of simulation parameters

Allow the simulation environment to discover new capabilities without the need for recompilation or relinking

The python interpreterThe interpreter

modern object oriented language

robust, portable, mature, well supported, well documented

easily extensible

rapid application development

Support for parallel programmingtrivial embedding of the interpreter in an MPI compliant manner

a python interpreter on each compute node

MPI is fully integrated: bindings + OO layer

No measurable impact on either performance or scalability

7


Solver integration

custom application driver

properties

geometry

materials

meshing

adv. features viz. support

checkpointsfem

Python bindings

pyadlib.so

MPDb VTFApplication

Controller Application

Mesher Solver StrengthModel

Pyre

8


Framework services

Problem specificationcomponents and their properties

Solid modelingoverall geometrymodel constructiontopological and geometrical information

Boundary and initial conditionshigh level specificationaccess to the underlying solver data structures in a uniform way

Materials and constitutive modelsmaterials properties databasestrength models and EOSassociation with a region of space

Computational enginesselection and association with geometry

solver specific initializations

coupling mechanism specification

Simulation driverinitialization

appropriate time step computation

orchestration of the data exchange

checkpoints and field dumps

Active monitoringinstrumentation: sensors, actuators

real-time visualization

Full simulation archiving

9


Writing python bindings

Given a “low level” routine, such as

and a wrapper

double adlib::StableTimeStep();

char pyadlib_stableTimestep__name__[] = "stableTimestep";PyObject * pyaldib_stableTimestep(PyObject *, PyObject * args){ double dt = adlib::StableTimeStep();

return Py_BuildValue("d", dt);}

dt = pyadlib.stableTimestep()

• one can place the result of the routine in a python variable

• The general case is not much more complicated than this

10


VTFApplication

Integrated applications in Pyre

MaterialModel

arm3d

pyarm3d

GrACE

pygrace

ACIS

pyacis

ARM3d

solid engine

pyadlib

Adlib

options

GUI

script

InterfaceManager

Geometry

Solver

Sensor/Probe

Monitor

Controller

Application

MPI

pympi

11


Pythia layout

pythia

pyre

journal

mpi

acis

simpleviz

the framework

simulation monitoring

bindings for MPI

solid modelling

integrated visualization

12


Pythia layout II – under construction

pythia

contact

cpt

elc

contact detection

fast level set generation

an Eulerian-Lagrangian coupler

hdf5 support for reading and writing hdf5 files

pulse

rigid

a trivial Eulerian solver

a trivial Lagrangian solver

plexus scalable meshing

13


Pyre layout I

pyre

applications

components

facilities

filesystem

geometry

application support

base classes for components

base classes for facilities

support for manipulating file systems

abstract operations on geometrical objects

graph support for building and visualizing graphs

handbook physical constants

ipa authentication daemons

14


Pyre layout II

pyre

monitors

network

parsing

parsers

properties

timers and other simulation monitors

network server and client support

infrastructure for lexical parsing

some parsers for text files

user settable parameters

units dimensional quantities

util miscellaneous functions

xml framework for parsing XML files

15


VTF layout II

vtf

io

metis

pulse

rigid

shells

abstract IO layer for checkpoints

graph partitioning

a simple Eulerian solver

a simple Lagrangian solver

shell finite elements

tetra scalable meshing

vtf base classes for VTF simulations

16


Overview of selected services

Services described hereapplication structure

properties: types and units

parallelism and staging

facilities, components

application monitoring

geometry specification

simulation control: controller, solver

support for integrated visualization

17


HelloApp

from pyre.application.Application import Application

class HelloApp(Application):

def run(self): print "Hello world!" return

def __init__(self): Application.__init__(self, "hello") return

# mainif __name__ == "__main__": app = HelloApp() app.main()



def run(self): print "Hello world!" return



access to the base classaccess to the base class

18


Properties

Named attributes that are under direct user controlautomatic conversions from strings to all supported types

Properties havenamedefault valueoptional validator functions

Accessible from pyre.propertiesfactory methods: string, bool, int, float, sequence, scalar, vectorvalidators: less, greater, range, choice

The user can derive from Property to add new types

import pyre.properties as props

flag = props.bool("flag", True)name = props.string("name", "pyre")scale = props.scalar("scale", 1.0, props.greater(0))

19


Units

Properties can have units:

framework provides: scalar, vector

Support for units is in pyre.unitsfull support for all SI base and derived units

support for common abbreviations and alternative unit systems

correct handling of all arithmetic operations

addition, multiplication, functions from math

import pyre.properties as propsfrom pyre.units.time import s, hourfrom pyre.units.length import km, mile

speed = props.scalar("speed", 50*mile/hour, pyre.units.velocity)

velocity = props.vector("velocity", (0.0*m/s, 0.0*m/s, 10*km/s), pyre.units.velocity)

20


HelloApp: adding properties


class HelloApp(Application): …

class Properties(Application.Properties):

import pyre.properties

__properties__ = Application.Properties.__properties__ + ( pyre.properties.string("name", "world"), )

…


class HelloApp(Application): …

class Properties(Application.Properties):

import pyre.properties

__properties__ = Application.Properties.__properties__ + ( pyre.properties.string("name", "world"), )

…framework property factoriesframework property factories

21




def run(self): print "Hello %s!" % self.name return

def __init__(self): Application.__init__(self, "hello") self.name = self.properties.name return

…



def run(self): print "Hello %s!" % self.name return

def __init__(self): Application.__init__(self, "hello") self.name = self.properties.name return

…

HelloApp: using properties

data member tied to propertydata member tied to property

./hello.py --name="Michael"

• Now the name can be set by the user interface

22


Support for concurrent applications

Python as the driver for concurrent applications thatare embarassingly parallelhave custom communication strategies

sockets, ICE, shared memory

Excellent support for MPImpipython.exe: MPI enabled interpreter (needed only on some platforms)

mpi: package with python bindings for MPIsupport for staging and launchingcommunicator and processor group manipulationsupport for exchanging python objects among processors

mpi.Application: support for launching and staging MPI applicationsdescendant of pyre.application.Applicationauto-detection of parallelismfully configurable at runtimeused as a base class for user defined application classes

23


Parallel Python

Enabling parallelism in Python is implemented by:embedding the interpreter in an MPI application:

constructing an extension module with bindings for MPI

providing an objected oriented veneer for easy access

int main(int argc, char **argv) { int status = MPI_Init(&argc, &argv); if (status != MPI_SUCCESS) {

std::cerr << argv[0] << ": MPI_Init failed! Exiting ..." << std::endl;

return status; } status = Py_Main(argc, argv); MPI_Finalize(); return status;}

int main(int argc, char **argv) { int status = MPI_Init(&argc, &argv); if (status != MPI_SUCCESS) {

std::cerr << argv[0] << ": MPI_Init failed! Exiting ..." << std::endl;

return status; } status = Py_Main(argc, argv); MPI_Finalize(); return status;}

24


Python bindings for MPI

// return the communicator rank (MPI_Comm_rank)char pympi_communicatorRank__doc__[] = "";char pympi_communicatorRank__name__[] = "communicatorRank";PyObject * pympi_communicatorRank(PyObject *, PyObject * args){

PyObject * py_comm; int ok = PyArg_ParseTuple(args, "O:communicatorRank", &py_comm);

if (!ok) { return 0; }

// cast the Python object into a Communicator Communicator * comm = (Communicator *) PyCObject_AsVoidPtr(py_comm); int rank = comm->rank();

// return return PyInt_FromLong(rank);}

// return the communicator rank (MPI_Comm_rank)char pympi_communicatorRank__doc__[] = "";char pympi_communicatorRank__name__[] = "communicatorRank";PyObject * pympi_communicatorRank(PyObject *, PyObject * args){

PyObject * py_comm; int ok = PyArg_ParseTuple(args, "O:communicatorRank", &py_comm);

if (!ok) { return 0; }

// cast the Python object into a Communicator Communicator * comm = (Communicator *) PyCObject_AsVoidPtr(py_comm); int rank = comm->rank();

// return return PyInt_FromLong(rank);}

25


Access to MPI through Pyre

creates a new communicator bymanipulating the communicator group

creates a new communicator bymanipulating the communicator group

import mpi

# get the world communicatorworld = mpi.world()

# compute processor rank in MPI_COMM_WORLDrank = world.rank

# create a new communicatornew = world.include([0])if new: print “world: %d, new: %d” % (rank, new.rank)else: print “world: %d (excluded from new)” % rank

import mpi

# get the world communicatorworld = mpi.world()

# compute processor rank in MPI_COMM_WORLDrank = world.rank

# create a new communicatornew = world.include([0])if new: print “world: %d, new: %d” % (rank, new.rank)else: print “world: %d (excluded from new)” % rank

26


Parallel HelloApp

from mpi.Application import Application


def run(self): import mpi world = mpi.world() print "[%03d/%03d] Hello world" % (world.rank, world.size) return



from mpi.Application import Application


def run(self): import mpi world = mpi.world() print "[%03d/%03d] Hello world" % (world.rank, world.size) return



new base classnew base class

27


Staging

from pyre.application.Application import Application as Base

class Application(Base):

… class Facilities(Base.Facilities):

import StagingMPICH import pyre.facilities

__facilities__ = Base.Facilities.__facilities__ + ( pyre.facilities.facility("staging", StagingMPICH()), ) …

from pyre.application.Application import Application as Base

class Application(Base):

… class Facilities(Base.Facilities):

import StagingMPICH import pyre.facilities

__facilities__ = Base.Facilities.__facilities__ + ( pyre.facilities.facility("staging", StagingMPICH()), ) …

The new base class mpi.Applicationoverrides the initialization protocol

gives the user access to the application staging details

excerpt from mpi/Application/Application.pyexcerpt from mpi/Application/Application.py

28


Facilities and components

A design pattern that enables the assembly of application components at run time under user control

Facilities are named abstract application requirements

Components are concrete named engines that satisfy the requirements

Dynamic control:the application script author provides

a specification of application facilities as part of the Application definition

a component to be used as the default

the user can construct scripts that create alternative components that comply with facility interface

the end user can configure the properties of the component

select which component is to be bound to a given facility

29


Staging properties

Staging defines the following propertiesnodes: the number of processors (int)

dry: do everything except the actual call to mpirun (bool)

command: specify an alternative to mpirun (string)

extra: additional command line arguments to mpirun (string)

Running the parallel version of HelloApp:./hello.py --staging.nodes=4 –staging.dry=on

./hello.py --staging=asap --staging.nodes=4

Specifying an alternate staging configurationassuming that a script asap.py is in the current directory

or the equivalent

./hello.py --staging=asap --asap.nodes=4

30


Simulation monitoring

journal: a component for managing application diagnosticserror, warning, infodebug, firewall

Named categories under global dynamic control forpython, C, C++, FORTRAN

import journal

debug = journal.debug("hello")debug.activate()debug.log("this is a diagnostic")

Customizablemessage contentmeta-dataformattingoutput devices

console, files, sockets for remote transport

journal is a component!

31


Geometry specification

def geometry(): from pyre.units.length import mm side = 50.0*mm diameter = 25.0*mm scale = 5

from pyre.geometry.solids import block,cylinder from pyre.geometry.operations import rotate,subtract,translate

cube = block((side, side, side)) cube = translate(cube, (-side/2, -side/2, -side/2)) hole = cylinder(height=2*side, radius=diameter/2) z_hole = translate(hole, (0*side, 0*side, -side) y_hole = rotate(z_hole, (1,0,0), pi/2) x_hole = rotate(z_hole, (0,1,0), pi/2)

body = subtract(body, x_hole) body = subtract(body, y_hole) body = subtract(body, z_hole)

ils = min(radius, side – diameter)/scale

return pyre.geometry.body(body, ils)

def geometry(): from pyre.units.length import mm side = 50.0*mm diameter = 25.0*mm scale = 5

from pyre.geometry.solids import block,cylinder from pyre.geometry.operations import rotate,subtract,translate

cube = block((side, side, side)) cube = translate(cube, (-side/2, -side/2, -side/2)) hole = cylinder(height=2*side, radius=diameter/2) z_hole = translate(hole, (0*side, 0*side, -side) y_hole = rotate(z_hole, (1,0,0), pi/2) x_hole = rotate(z_hole, (0,1,0), pi/2)

body = subtract(body, x_hole) body = subtract(body, y_hole) body = subtract(body, z_hole)

ils = min(radius, side – diameter)/scale

return pyre.geometry.body(body, ils)

32


Geometry specification graph

Difference

Rotation

BlockCylinder

Translation

Difference

Difference

Translation

Rotation

body

,

,

33


def mesh(model): body = createBody(model)

import acis faceter = acis.faceter()

properties = faceter.properties properties.gridAspectRatio = 1.0 properties.maximumEdgeLength = body.ils boundary = faceter.facet(acis.create(body.geometry)) bbox = boundary.boundingBox()

return boundary, bbox

def mesh(model): body = createBody(model)

import acis faceter = acis.faceter()

properties = faceter.properties properties.gridAspectRatio = 1.0 properties.maximumEdgeLength = body.ils boundary = faceter.facet(acis.create(body.geometry)) bbox = boundary.boundingBox()

return boundary, bbox

Creating the model

the Python bindings for the solid modelerthe Python bindings for the solid modeler

convert the abstract geometrical description into an actual instance of the ACIS class BODY

convert the abstract geometrical description into an actual instance of the ACIS class BODY

abstract representation of the modelabstract representation of the model

34


The finished model

35


Simulation control

Achieved through the collaboration of two componentsSimulationController (controller)

Solver (solver)

SimulationController expects Solver to conform to the following interfaceinitialize(), launch()

startTimestep(), endTimestep()

applyBoundaryConditions()

stableTimestep(), advance()

save: publishState(), plotFile(), checkpoint()

Both components provide (trivial but usable) default implementations

The facility/component pattern enables the selection and initialization of solvers by the end user

36


Advancing the solution in time

from pyre.components.Component import Component

class SimulationController(Component):

def march(self, totalTime=0, steps=0): while 1: solver.startTimestep() solver.applyBoundaryConditions() solver.save() dt = solver.stableTimestep() solver.advance(dt)

self.clock += dt self.step += 1 if totalTime and self.clock >= totalTime: break if steps and self.step >= step: break

solver.endSimulation() return

from pyre.components.Component import Component

class SimulationController(Component):

def march(self, totalTime=0, steps=0): while 1: solver.startTimestep() solver.applyBoundaryConditions() solver.save() dt = solver.stableTimestep() solver.advance(dt)

self.clock += dt self.step += 1 if totalTime and self.clock >= totalTime: break if steps and self.step >= step: break

solver.endSimulation() return

37


Visualization

Support for integrated remote visualization is provided by simpleviz

It is a default facility for vtf.Application

Three tier system: a data source embedded in the simulation

a data server

a client: visualization tools (such as IRIS Explorer)

The serveris a daemon that listens for socket connections from

data sources (mostly pyre simulations)

visualization tools

The clientis a default facility for vtf.Application

has the hostname and port number of the server

Demonstration later this afternoon

38


Summary

We covered many framework aspectsapplication structure

support for parallelism

properties, facilities and components

geometry specification

the controller-solver interface

support for embedded visualization

There is support for much more…

©1998-2003, michael aivazis pyre an introduction to the integration framework michael aivazis...

Documents

michael aivazis pyre

python interpreter

scalability slide

lines of bindings

simulation environment

tb of data slide

simulation archiving

python scripts