b rockmann c onsult 1 20. – 22.01.2009 finnish environment institute, syke helsinki programming...

BROCKMANN CONSULT

1

20. – 22.01.200920. – 22.01.2009Finnish Environment Institute, SykeFinnish Environment Institute, Syke

HelsinkiHelsinki

Programming with Programming with

BEAMBEAMNorman FomferraNorman Fomferra

Carsten BrockmannCarsten Brockmann

BROCKMANN CONSULT

2

Day 1: IntroductionDay 1: Introduction

1. Software OverviewAn introduction to VISAT, its user interface elements and the BEAM command line tools. Includes a detailed presentation of the BEAM standard I/O format.

2. Architecture and APIsA high level introduction into the BEAM 4.x architecture including the diverse application programming interfaces.

3. Runtime ConfigurationA detailed presentation of the BEAM installation directory structure, the standard modules, the 3rd party libraries used, the main entry points into the BEAM code.

4. Development and Build ProcessAn introduction to Java development with BEAM. The goal is to compile and run a simple program that uses the BEAM API with your preferred Java IDE.

Programming BEAM, Syke, 20.-22.01.2010

BROCKMANN CONSULT

3

Day 2: Using the BEAM APIDay 2: Using the BEAM API

1. A simple Data ProcessorWe develop a simple data processor, which will ingest an AATSR Level 1 data product and write a trivial cloud mask. The processor will use a simple command line interface. We will then review the output in VISAT.

2. BEAM Graph Processing Framework (GPF)A detailed introduction to the new BEAM graph processing framework.

3. Developing a GPF OperatorWe convert the simple data processor into a GPF operator. The processor will use the standard command line interface provided by the GPF.

4. BEAM ScriptingAn introduction to the new Scripting API introduced in Java 1.6 and a discussion of its relevance for BEAM.


BROCKMANN CONSULT

4

Day 3: Extending BEAMDay 3: Extending BEAM

1. BEAM Module DevelopmentAn introduction to BEAM modules (plug-ins), the module descriptor file and module activators.

2. The Maven Build SystemMaven is a highly configurable build system which allows us to quickly create a module for BEAM.

3. Creating a BEAM ModuleWe add a module descriptor to the code developed so far in order to create a versioned, deployable and updateable BEAM module. The GPF operator becomes part of BEAM.

4. Integration with VISATWe develop a simple graphical user interface for the GPF operator.We integrate the GPF operator and its GUI into VISAT by adding a command which allows the users to invoke the operator from the VISAT Tools menu.


BROCKMANN CONSULT

5

Wish list Wish list of participantsof participants

• Writing plug-ins (like a simple algorithm) • Batch processing• Automatic retrieval of new satellite products

(wouldn't that be a nice feature) • Command-line interfacing, interfacing with

MATLAB• Automatically producing graphics or

georeferenced sub-products • Pixel value retrieval, coordinate-based retrieval

for a large sample set • Overlaying transect data from external sources

with beam-supported productsProgramming BEAM, Syke, 20.-22.01.2010

BROCKMANN CONSULT

6Programming BEAM, ESRIN, 22.-26.09.2008

Why develop with BEAM?Why develop with BEAM?

• Access EO data, metadata, no-data, flags

• Simple & effective programming models

• Reuse a rich software infrastructure• It is free, it is open source, its Java • Review and reuse source code• BEAM code is widely used, it is in use• BEAM is supported by ESA

BROCKMANN CONSULT


Development Use CasesDevelopment Use Cases1. Use it

Write batch-mode scripts Program stand-alone applications Develop web-services

2. Extend it Add EO data readers, writers Add EO data processing nodes Add map projections, DEMs Add VISAT actions, tool bars and tool

windows3. Clone it

Customize and brand VISAT

BROCKMANN CONSULT

8

Architecture OverviewArchitecture Overview

Programming BEAM, ESRIN, 20.-22.01.2010

BROCKMANN CONSULT

9

Module-based ArchitectureModule-based Architecture

• BEAM module:– Name– Description– Version– Authors– Changelog– Dependencies– Extension points– Extensions

• Simple extension model:– Host module provides

extension point, e.g. “product-reader”

– Client module provides extension to host's extension point, e.g. “seawifs-reader”

– Clients can be hosts

Programming BEAM, ESRIN, 20.-22.01.2010

BROCKMANN CONSULT

10

VISAT Module ManagerVISAT Module Manager


BROCKMANN CONSULT

11

Development Platform Usages Development Platform Usages


BROCKMANN CONSULT


GPF - Graph Processing GPF - Graph Processing FrameworkFramework

Read

AtmCorr

NoiseRed

CloudMask

Write

SourceProduct

TargetProduct

Tile reuse!Tile reuse!

Tile reuse!Tile reuse!

Tile (0,1)Tile (0,1)

Source product:3 bands,3 x 4 tiles

Source product:3 bands,3 x 4 tiles

• Rapid processor development Forget about processing environment Concentrate on data and algorithm

BROCKMANN CONSULT

13

BEAM Development Process at BEAM Development Process at BCBC

• 3-4 core developers

• 1-2 application developers

• 1 permanent tester

• Developers also do user support

• Agile Development Process

• Feature list planning with ESA every 2 months

• Weekly iteration meetings

• Daily stand-up meetings


BROCKMANN CONSULT

14

Tools used for BEAM Tools used for BEAM Development Development

• Development IntelliJ IDEA (10) and Eclipse (2) IDEs Version control system is Subversion Maven - Project model & dependency

management• Testing

JUnit – achieve good tested(10) code coverage QF Test – Automatic application & GUI testing

• Build Maven – The Java make Install4j – Installer builder TeamCity build server


BROCKMANN CONSULT


Getting startedGetting started

• Online programming tutorial and guideswww.brockmann-consult.de/beam/wiki

• From BEAM website Download & install BEAM 4.6.1 Download BEAM API documentation Download BEAM source code

• Install a Java IDE IntelliJ IDEA, Eclipse, NetBeans

BROCKMANN CONSULT

16

Naming Conventions in Java Naming Conventions in Java CodeCode

• Interface / Class names CamaleCase, first character upper-case GeoCoding, Product, RasterDataNode

• Variable names CamelCase, first character lower-case backgroundColor, threshold, minValue

• Constant names All letters upper-case, underscore DIALOG_TITLE, BUFFER_SIZE_LIMIT

• Method / Function names CamelCase, first character lower-case getSampleValue, computeTile , isClosed

• Package names all lower-case beam, syke, binning


BROCKMANN CONSULT

17

BEAM Installation DirectoryBEAM Installation Directory

• beam-4.x.y BEAM installation directory

bin Application binary and script files visat.exe, gpt.bat, ceres-launcher.jar, ...

config Application configuration file(s) beam.config

lib Common, 3rd -party libraries xtream.jar, jdom.jar, gt-*.jar, ...

modules Application and plug-in BEAM modules ceres-*.jar beam-*.jar


BROCKMANN CONSULT

18

Invocation of Command-line Invocation of Command-line ToolsTools

Invocation of gpt.bat via a dedicated launcher program (ceres-launcher.jar). All JARs found in BEAM’s lib and modules directories are put on the classpath:

@echo off

set BEAM4_HOME=C:\Program Files\beam-4.6.1

"%BEAM4_HOME%\jre\bin\java.exe" ^ -Xmx1024M ^ -Dceres.context=beam ^ "-Dbeam.mainClass=org.esa.beam.framework.gpf.main.Main" ^ "-Dbeam.home=%BEAM4_HOME%" ^ "-Dncsa.hdf.hdflib.HDFLibrary.hdflib=%BEAM4_HOME%\...\jhdf.dll" ^ "-Dncsa.hdf.hdf5lib.H5.hdf5lib=%BEAM4_HOME%\...\jhdf5.dll" ^ -jar "%BEAM4_HOME%\bin\ceres-launcher.jar" %*

exit /B 0


BROCKMANN CONSULT

19

The BEAM The BEAM Graph Processing Graph Processing Framework (GPF)Framework (GPF)


BROCKMANN CONSULT

20

TermsTerms

• ((EO) Data) Processing System A system that utilizes hardware and manages

processes running on it • ((EO) Data) Processor

A program that transforms source products into target products

• ((EO) Data) Processing Framework A software and API with an architecture designed for

developing processors• Framework

“Don’t call us, we call you.” Abstract interfaces comprising callback functions

clients must implement


BROCKMANN CONSULT

21

Processing System (1)Processing System (1)

• Main objective: Translate a job into a number of tasks running in parallel on a cluster of machines

• Scalability– Add CPUs– Add more storage

space

• Data locality– Avoid network I/O– Force local data I/O

• Load balancing– Assign tasks to idle

compute nodes

• Backup tasks– Handle tasks taking

unusually long time to complete

• Fault tolerance– Recover from HW

failures

• Monitoring– Statistics &

Performance

BROCKMANN CONSULT

22

Processing System (2)Processing System (2)

• A processing system places some requirements on the processes (processors) it is running: A processor shall be configurable

Input and output files Processing parameters Runtime behaviour (e.g. memory, tiling, buffer sizes)

A processor shall be observable Progress Status Logging Exit code


BROCKMANN CONSULT

23

A Processing GraphA Processing Graph


BROCKMANN CONSULT

24

Processing Framework Processing Framework RequirementsRequirements

• Create processing chains or even graphs composed of processing nodes. Processing nodes shall be reusable in other graphs, e.g. MERIS Cloud Screening be easily configurable in terms of processing parameters be easily configurable in terms of the algorithm

implementation• Processor implementation code shall abstract from

physical file format of EO data file I/O configuration & parameterisation, parameter value access

• Avoid I/O overhead between processing steps• Exploit multi-core CPU architectures• The framework shall be extendible, e.g. it shall recognise

new processing node implementations at runtime


BROCKMANN CONSULT

25

BEAM GPFBEAM GPF

• Allows to construct directed, acyclic processing graphs• Implements the “pull-processing” paradigm

A processor is represented by its processing graph Product processing requests are translated into raster-

tile-requests propagated backwards through the graph Tile requests are automatically parallelised depending on

CPU cores• Simple data processing and programming model

For a particular node, clients (Java developers) implement how a raster tile is being computed

Parameter values are “injected” by the framework• Generated user interfaces: Command-line (gpt) and VISAT

GUI• Operators can be dynamically added to BEAM via plug-in

modulesProgramming BEAM, Syke, 20.-22.01.2010

BROCKMANN CONSULT


Product Data Model (1)Product Data Model (1)

BROCKMANN CONSULT


Product Data Model (2)Product Data Model (2)

BROCKMANN CONSULT


GPF Architecture (1)GPF Architecture (1)

Client Module cloudmask-op

System Module beam-core

System Module beam-gpf

initialize()computeTile(...)computeTileStack(...)dispose()

Operator

getOperatorType(): ClasscreateOperator(): Operator

OperatorSpi

getOperatorSpi(id)addOperatorSpi(spi)removeOperatorSpi(spi)

<<interface>>OperatorSpiRegistry

◄ maintains

Band Product

0..nbands

1targetProduct

0..nsourceProducts

createProduct(op, ...)createProduct(op, ...)createProduct(op, ...)createProductNS(op, ...)

GPF

1instancesetSample(x, y, v)

getSampleInt(x, y): vgetSampleFloat(x, y): v...

<<interface>>Tile

◄ creates

◄ computes target tiles for◄ provides source tiles for

initialize()computeTile(...)

CloudMaskOp

createOperator()

CloudMaskOpSpi

◄ computes target samples for◄ retrieves source samples from

◄ creates target◄ uses as source

◄ knows its

knows its ►

BROCKMANN CONSULT

29

GPF Architecture (2)GPF Architecture (2)

• Clients provide their compiled Operator Java code packed into JAR modules

• A JAR module publishes its operator services via metadata (Service Provider Interface, SPI) META-INF/services/org.esa.beam.framework.gpf.OperatorSpi

• Once GPF is started, e.g. by visat.exe or gpt.bat, it loads all operator services it finds on its (dynamic) classpath %BEAM_HOME%/modules/*.jar %BEAM_HOME%/lib/*.jar


BROCKMANN CONSULT

30

GPF Processing Node Anatomy GPF Processing Node Anatomy (1)(1)

@OperatorMetadata(alias = “NoiseRedOp", version = "1.0", authors = "Ralf Quast", copyright = "(c) 2008 by Brockmann Consult", description = "Performs a noise reduction on”

+ “ CHRIS/Proba images.")

public class NoiseRedOp extends Operator {

@SourceProduct(alias = "source") private Product sourceProduct; @TargetProduct private Product targetProduct; @Parameter(defaultValue=“false”) private boolean slitCorrection; @Parameter(interval=“(0,50]”, defaultValue=“25”) private int smoothingOrder; @Parameter(valueSet={“N2”, “N4”, “N8”}, defaultValue=“N2”) private String neighbourhoodType; // ...} Programming BEAM, Syke, 20.-22.01.2010

BROCKMANN CONSULT

31


If single target bands can be computed independently of each other, e.g. NDVI or algorithms which perform single band filtering:

public class NoiseRedOp extends Operator { public void initialize() throws OperatorException { // validate and process source products and parameter values // create, configure and set the target product

}

public void computeTile(Band targetBand, Tile targetTile, ProgressMonitor pm) throws OperatorException { // Obtain source tiles for used bands of source products

// Process samples of source tiles to samples of target tile // Set samples of single target tile }}


BROCKMANN CONSULT

32


If single target bands cannot be computed independently of each other, e.g. model inversion algorithms based on neural networks:

public class NoiseRedOp extends Operator { public void initialize() throws OperatorException { // validate and process source products and parameter values // create, configure and set the target product

}

public void computeTileStack(Map<Band, Tile> targetTiles, Rectangle targetTileRectangle, ProgressMonitor pm) throws OperatorException {

// Obtain source tiles for used bands of source products // Process samples of source tiles to samples of target tiles // Set samples of all given target tiles }}


BROCKMANN CONSULT


How GPF “drives” Processing How GPF “drives” Processing NodesNodes

1. GPF knows about all registered Operators (Plug-in)2. GPF encounters a required node (GUI, CLI, XML graph)

1. The client‘s Operator class is found by its alias2. The SourceProduct, TargetProduct and Parameter

annotations are analysed3. The client‘s Operator object is created 4. Source products and parameter values are injected

3. GPF calls Operator.initialize(). In this method, the client provides code to1. validate and process source products and parameter

values2. create, configure and set the target product

4. GPF calls either Operator.computeTile() or Operator.computeTileStack()in case raster data is required

BROCKMANN CONSULT

34

gpt – The GPF Command-Line gpt – The GPF Command-Line ToolTool


BROCKMANN CONSULT

35

GPF Processing Node GPF Processing Node ConfigurationConfiguration


BROCKMANN CONSULT

36

GPF Processing Node GUIGPF Processing Node GUI


BROCKMANN CONSULT

37

GPF Programming ModelGPF Programming Model

Product p0 = ProductIO.readProduct(“MER_1P.N1”);

Map<String,Object> params1 = ...;Map<String,Object> params2 = ...;Map<String,Object> params3 = ...;

Product p1, p2, p3;p1 = GPF.createProduct(“SmileCorrOp”, params1, p0);p2 = GPF.createProduct(“CloudMaskOp”, params2, p1);p3 = GPF.createProduct(“NoiseRedOp”, params3,

p1, p2);

WriteOp.writeProduct(p3, new File(“TEST.DIM”), “BEAM-DIMAP”, ProgressMonitor.NULL);


BROCKMANN CONSULT

38

EndEnd

Thanks for using BEAM!Thanks for using BEAM!

http://www.brockmann-consult.de/cms/web/beam/forum


b rockmann c onsult 1 20. – 22.01.2009 finnish environment institute, syke helsinki programming...

Documents