space science and engineering center university of wisconsin-madison space science and engineering...

Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 1

NPP Atmosphere PEATECritical Design ReviewNPP Atmosphere PEATECritical Design Review

Presented by the Atmosphere PEATE TeamSpace Science and Engineering CenterUniversity of Wisconsin-Madison17 April 2008

Presented by the Atmosphere PEATE TeamSpace Science and Engineering CenterUniversity of Wisconsin-Madison17 April 2008

Part 2: Science Processing SystemScott MindockPart 2: Science Processing SystemScott Mindock


DesignSystem SubsystemsSoftware Development and VerificationScience Investigation SupportLogistics

Agenda


UML - Unified Modeling LanguageUse Case - Captures requirements at functional levelActivity - Describes the steps of a use casePackage - Show software structure and dependenciesClass - Show software structure and dependenciesWeb Services - SOAP XML based communications

Definitions


Sample Activity Diagram

Like FlowchartOvals = ActivityRectangle = DataAction A and B Decoupled Dot = startCircle = end


Maintainability - System lifetime spans years

Reusability - Subsystems and design patterns

Testability - Subsystem dependencies are managed

Scalability - Design must scale to larger and smaller systems

Design: Goals (1 of 3)


Learn from others - IDPS, Clusters, Ocean PEATE, Disney Parkwide

Leverage existing proven technologies and standards• Java• Web Services - Provide communications between subsystems • Apache Tomcat - Web Application Server• Apache Axis2 - Web Application providing web services• JAXB - Java XML Binding• ANT - XML Structured make• Eclipse - Java IDE• Subversion - Revision control

Prototype key features - Demonstration Projects

Loosely Coupled System - Manage dependencies, Define interfaces

Design: Methodology (2 of 3)


All systems use XML configuration.All configurations have a XML schema.System persistence leverages schema based code generation.

All systems can persist to DB or XML (scalable)

Major components are deployed web services. (decouple, scalable)

Test patterns used for scenario based testing / system verification

Design: Patterns ( 3 of 3 )


External RelationshipsSubsystemsSystem Activity

System


System: External Relationships


DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages

Subsystem Relationships

Arrows denote dependency

System design minimizes dependencies

Eases maintenance


System Activity Diagram


DMS: Data Management System Stores DataCRG: Computational Resource Grid Processes DataARM: Algorithm Rule Manage Applies Product Rules to DataING : Ingest System Brings Data into System

Subsystem Relationships @ PDR

@PDR@PDR


System Activity Diagram @ PDR

@PDR@PDR


ING - Ingest brings data into the Atmosphere PEATE

DMS - Data Management System, stores data in the form of files. Provides a web service interface to locate and retrieve files. CRGALGARMRUL

Subsystems



System Components, Java Packages


Supports pull ingestsSupports push ingestsVerification functions localizedNotices missing or late data

Dedicated MachineDeploy Multiple Copies

ING


Schema based persistenceXML ingest type configurationSupport for FTP, RSYNC

ING Configuration


Provides well-defined interface deployed as a web service.DMS is autonomousProvides storageProvides catalogSpans multiple file servers on a networkDMS = Linux File system + Postgres database

DMS


Each file in DMS has associated entry in catalogImportant file characteristics are trackedFile are distributed through system on entry

DMS File Properties


Schema defines DMS directory structureDMS creates directories on installationDMS uses round robin method to fill file systemsConfiguration not utilizing DB are restricted to one machineMax disk utilization is specified

DMS Configuration


File system - hold filesDatabase - holds file informationPublic Access - implement DMS interfaceWorker - manages file system

Two flavorsWith DB - spans multiple machinesWithout DB - single machine

DMS Component and Deployment


Provides well-defined interface deployed as a web service.Accepts job requestProvides Job StatusMonitor Job StateScalableTestable

CRG


One or more per nodeRetrieves data from DMSRetrieves and runs software packagesSaves results to DMSConsumes user jobs by using CRG Web Service

ALG


Provides well-defined interface deployed as a web service.Assigns jobs to CRGMonitors data in DMSMonitors the status of jobs in CRGApplies rules to dataGenerates notificationsVolatile logic lives hereProvides extension point for rules

Rules can be added or removed dynamically

ARM


Mechanism to deploy algorithms into APSPSRules have namesRules use regular expressionsURL of software package specifiedDestination DMS can be specifiedDays of lease can be specified

RUL


Used by ARMSpecified by user

Specifies software package to run (SoftwarePackage)Specifies data of interest (FilterRegex)Specifies product output location (DmsUrl)Specifies product lifetime (DaysOfLease)

Rules and Software Packages


Development processTesting StrategyDefect ReportingDefect Correction

Configuration Management

Nightly BuildUnit and Regression Testing Testing ScenariosRequirements Mapping

Software Development and Verification


Spiral Method

DesignImplement TestDeploy

Development Process


We use standard software industry practices• Automate with ANT• Test with JUNIT

Saves Money and Increases Quality• Tests are reproducible• Tests can be run more often than they would be if they were manual• Tests are improved over time

We don’t just build, the process includes testing and verification

Testing Strategy


Anyone with an account can report a bugReporter chooses software systemReporter enters bug informationOwner of project is notified automatically by emailOwner prioritizes

Defect Reporting


Verify software before startingVerify problem existsFix and VerifyAdd back to code base

Defect Correction


Leverages open source softwareProvides versioned access to softwareProvides defect-tracking facilitiesUsed for source codeUsed for test scriptsUsed for software packagesBackup is made on regular basisAccounts are password protected

Configuration Management


Provides versioned access to softwareProvides defect-tracking facilitiesUsed for source codeUsed for test scriptsUsed for binaries

Configuration Management @ PDR

@PDR@PDR


Source code management

Software packages distributed to cluster as subversion repositoriesSoftware packages may contain large static data sets

Issue:

Subversion retrieval of large binary datasets slowSubversion stores two copies for diff purposes

Solution:

Use two repositoriesCheckout executing software packageExport static binary dataVerified solution with subversion contributor

Subversion


Builds systemTests subsystemsTests scenariosUpdates repositoriesLogs results

Scenarios demonstraterequirements

Nightly Build


May use internal knowledge interfaces for testingTest and exercise public interfacesStress test interfacesEvolve to test and verify bugsTests run in EclipseTest run from command lineFixed defects have specific tests added

Tests run in nightlyTests verify release

Layered approach to testingEverything tested, Every Night

Unit and Regression Testing


Test ingest functionTest forward and reprocess functions Tests rule executionTest user extension points

Testing Scenarios - Activity Diagram


Single Rule - Simplest test, connects data with software

Double Rule - Chain products togetherDouble Rule Reprocess Half - Chains products, reprocess second half of chain

MODIS L1->CM - real world SingleMODIS L1-> CM and LEOCAT - real world Double

Scenario / Use Case Test Instances


Level 3 requirements are mapped to level 4 requirements in requirements specification, 3600-0003.080402.doc

Operations Concepts illustrates requirements, 3600-0004.060911.doc

Scenario tests implement Operations Concepts use cases

Scenario tests have several instances

Generic instances exercise system.

Specific instances implement Ops Con use cases

Specific instances currently use MODIS and LEOCAT

Requirements Mapping


Algorithm LifecycleAlgorithm IngestAlgorithm QualificationLEOCAT

Science Investigation Support


Algorithm can come from anywhere.Once qualified, the algorithm can be applied from ARM.

Algorithm Lifecycle Use Case Diagram


Algorithm entered into subversionProduct created in bugzillaAlgorithm is ported and wrappedTests are created

Algorithm Ingest


Write a script to execute algorithmScript manages execution environmentAlgorithm name, inputs and outputs entered into ARM

Algorithm Qualification


APSPS executes bash scriptScript is passed arguments

• Binary Directory• Working / Output directory• Static Ancillary Directory• Dynamic Ancillary Directory

• Inputs as specified by rule• Outputs as specified by rule

Binary is executed from line in script

Results are stored by process that started script.

Qualified Algorithm Structure


Create slide content

LEOCAT



LEOCAT Structure


Org ChartStaffingHardwareFloor PlanSecuritySoftware Milestones

Logistics


Includes sourceIncludes generated persistence code (JAXB) > 600 lines in rulesDoes not include axis generated web service code ~ 30000 lines

ALG increase reflects subversion issueIncreased leveraging of java and third party libraries resulted in code size decrease.

Lines of code

Subsystem Lines Code (Sept 07) Predicted Total ( Sept 07 ) Line @ PDR ING 1200 2500 2515 ALG 600 1500 2101 CRG 2600 5000 1965 ARM 2300 8000 2046 DMS 5000 8000 4802 Rules 989 Total 11700 25000 14418


Not finished configuration control management plan - need to have backup systems

Write goals

APSPS accomplishments since PDR


Includes sourceIncludes generated persistence code (JAXB) > 600 lines in rulesDoes not include axis generated web service code ~ 30000 lines

ALG increase reflects subversion issueIncreased leveraging of java and third party libraries resulted in code size decrease.

Prototype system reuse



Organizational Chart



Staffing



Hardware


PEATE shares space with other projects including GOES archiveOperators located in adjoining roomRoom for expansion

PEATE Operations Floor Plan



Security



Build Plan


Summary

space science and engineering center university of wisconsin-madison space science and engineering...

Documents