nasa sensorweb: from v2.0 to v3.0 workflows and wcps for ... · nasa sensorweb: from v2.0 to v3.0...

NASA SensorWeb: From v2.0 to v3.0Workflows and WCPS for Space 

ApplicationsApplications

Pat Cappelaere (Vightel)

D M dl (GSFC)

Open Geospatial Consortium  TC Meeting

W kfl & C W ki G Dan Mandl (GSFC)

Stu Frye (SGT)Workflow & Coverages Working GroupsJune 2010, Silver Spring MD

1

https://ntrs.nasa.gov/search.jsp?R=20100028151 2020-04-21T21:56:32+00:00Z

AgendaAgenda

• Our Standard Compliance Evolution

• v2 0 Accomplishments & Lessons Learnedv2.0 Accomplishments & Lessons Learned

• v3.0 Architecture Migration– Towards A Unified Interface

– Examplesp

22

SatellitesSatellitesInt’lInt’l

UsersUsers

Int’lInt’lDisastersDisasters NASANASA

SensorWebSensorWebSensorWebSensorWebProjectProject

3

Big Software Project: Cost is Big Software Project: Cost is f(codef(code))

Cos

tC

ost

Lines of CodeLines of Code

4

Our Reality...Our Reality...

Fund

ing

Fund

ing

2005 ... 20102005 ... 2010

FF

5

But we have made some good progress...But we have made some good progress...

ties

ties

Cap

abili

tC

apab

ilit

2005 ... 20102005 ... 2010

6

Number of OGC Standards Has Increased SignificantlyNumber of OGC Standards Has Increased Significantly

ndar

dsnd

ards

OG

C S

tan

OG

C S

tan

which wouldwhich wouldincrease our cost!increase our cost!

2005 ... 20102005 ... 2010

# O

# O

7

S O OGC C li H Si ifi tl D dS O OGC C li H Si ifi tl D d

ance

ance

So Our OGC Compliance Has Significantly DecreasedSo Our OGC Compliance Has Significantly Decreased

C C

ompl

iaC

Com

plia

2005 ... 20102005 ... 2010

OG

CO

GC

8

v2.0 Architecture

9

NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

EO 1ASTER

FORMOSATSPOT...

Sensor Planning Service10

NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

You’ve Got Data EO 1ASTER

FORMOSATSPOT...

WNSWNSYou ve Got Data

Web Notification Service11

NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

You’ve Got Data EO 1ASTER

FORMOSATSPOT...

OPSBOPSBYou ve Got Data

Publish/Subscribe Basic Service12

NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

ASTERFORMOSAT

SPOT...OPSBOPSB

Raw Data (L1R, L1G)SOSSOS

DataData

Sensor Observation Service13

NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

ASTERFORMOSAT

SPOT...OPSBOPSB

Raw Data (L1R, L1G)GeoTorrentGeoTorrent

DataData

GeoTorrent Service14

NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

EO 1ASTER

FORMOSATSPOT...

OPSBOPSB

GeoTorrentGeoTorrentDataData

Derived Products CompositeCompositeCompositeCompositeDerived Products

WPS

pp

BurnscarBurnscar......

(JPL NG )

pp

WPS(JPL, NG...)

Web Processing Service15

NASA SensorWeb 2.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

ASTERFORMOSAT

SPOT...OPSBOPSB

GeoTorrentGeoTorrentDataData

Execute Workflow

Data Atom Feed

WfCS

Create/Visualize Workflows

FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlows

CompositeCompositeCompositeComposite

Create/Visualize Workflows

WPS

pp

BurnscarBurnscar......

(JPL NG )

pp

WPS (JPL, NG...)

Workflow Chaining Service16

NASA SensorWeb 2.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1

Task

ASTERFORMOSAT

SPOT...OPSBOPSB

GeoTorrentGeoTorrentDataData

Execute Workflow

Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows

WfCS

CompositeCompositeBB

CompositeComposite

Create/Visualize Workflows

WPSpp

BurnscarBurnscar......pp

WCPSAccess Raster Data

WPSWPS WCSWCS

Execute Algorithm Access Raster Data

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgorithmsAlgorithmsCustom Products

Web Coverage Processing ServiceCreate/Visualize Algorithms

17

NASA SensorWeb 2.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO 1

Task

EO-1HyspIRITestbed

OPSBOPSB

GeoTorrentGeoTorrentDataData

Execute Workflow

Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows

WfCS

CompositeCompositeBB

CompositeComposite

Create/Visualize Workflows

GCC GCC Compiler/Compiler/

LinkerLinker

Dynamic SwamoAgent Upload

WPSpp

BurnscarBurnscar......pp

WCPS

LinkerLinker

AMQP

WPSWPSParser /Parser /CodeCode

GeneratorGenerator

Execute AlgorithmAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgorithmsAlgorithmsCustom Products ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL

WCPS With a TwistCreate/Visualize Algorithms

ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL

18

NASA SensorWeb 2.0

WCPS: A SQL-like Query Language

for c in ( NIR )return

encode((char) ( ((c.0 / ((float)c.0 + c.1))-(c.1 / ((float)c.0 + c.1)))> 0.6 ) * 255,

"png" )

WCPS

WPSWPS

Execute AlgorithmAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgorithmsAlgorithmsCustom Products

WCPS LanguageCreate/Visualize Algorithms

19

NASA SensorWeb 2.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

Task

Visual EditorT t EditOPSBOPSB

GeoTorrentGeoTorrent

define name: flow1

Text Editor

Execute Workflow

Data Atom Feed FlowsFlowsFlowsFlows

FlowsFlowsFlowsFlows

FlowsFlowsCreate/Visualize Workflows

WfCSdefine name: flow1task_eo1process_alinotify_user

CompositeCompositeBB

CompositeComposite

Create/Visualize Workflows y_end

WPSBurnscarBurnscar......

WCPSBPMN 1.0

WPSWPS

Execute AlgorithmAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgorithmsAlgorithmsCustom Products

WfCS LanguageCreate/Visualize Algorithms

20

v2.0 Lessons Learned

Too Many Ways to Do Similar Things

Execute Task, Process, Workflow, Algorithm...

Hard Learning Curve on Users and ImplementorsHard Learning Curve on Users and Implementors

Two Languages: g g

For Workflows and Algorthms Developments

What About Web Service Security?

21

NASA SensorWeb v3.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO 1

Task

EO-1HyspIRITestbed

OPSBOPSB

GeoTorrentGeoTorrentDataData

Execute Workflow

Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows

WfCS

CompositeCompositeBB

CompositeComposite

Create/Visualize Workflows

GCC GCC Compiler/Compiler/

LinkerLinker

Dynamic SwamoAgent Upload

WPSpp

BurnscarBurnscar......pp

WCPS

LinkerLinker

AMQP

WPSWPSParser /Parser /CodeCode

GeneratorGenerator

Execute AlgorithmAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgorithmsAlgorithmsCustom Products ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL

v 3.0 Simplification #1Create/Visualize Algorithms

ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL

22

NASA SensorWeb v3.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO 1

Task

EO-1HyspIRITestbed

OPSBOPSB

GeoTorrentGeoTorrentDataData

Execute Workflow

Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows

WfCS

CompositeCompositeBB

CompositeComposite

Create/Visualize Workflows

WPSBurnscarBurnscar......

pp

WCPS

SEXPWPSWPS

Execute AlgorithmAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosCustom Products

SEXP

Create/Visualize Algos23

SEXP?SEXP?

24

S ES-Exp or S b li E iSymbolic Expressions

Think LISP or Parse TreesThink LISP or Parse Trees(= 4 (+ 2 2))

25

NASA SensorWeb v3.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO 1

Task GenericGenericOnboard SWAMO Onboard SWAMO

AgentAgent

EO-1HyspIRITestbed

OPSBOPSB

GeoTorrentGeoTorrent

Execute Workflow

Data Atom Feed FlowsFlowsFlowsFlows

FlowsFlowsFlowsFlows

FlowsFlowsCreate/Visualize Workflows

WfCS

CompositeCompositeBB

CompositeComposite

Create/Visualize Workflows

WPSBurnscarBurnscar......

WCPS

WPSWPS

Execute AlgorithmAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgorithmsAlgorithmsCustom Products

SEXP

Create/Visualize Algorithms26

EO 1

GenericGenericOnboard SWAMO Onboard SWAMO

AgentAgent

EO-1HyspIRITestbed

TextText VisualVisualCreate/Update

TextText VisualVisual

Domain

SEXP or JSONExecute

FlowsFlows

SpecifcLanguage (DSL) AlgorithmsAlgorithms

SEXP

RMagickRMagick

APIAPI

SecuritySecurity RubyRuby

GDALGDAL

gg

DataDataSecuritySecurity

2-factor Auth27

DSL E lDSL Example

define name:‘ali_l1g_visible’, author:‘Pat C.’, revision:1.0scene asset:EO1, instrument:‘ALI’, data:‘L1g’b5 = band(5)b4 = band(4)b3 = band(3)mask = b5 > 0composite_image red:b5, green:b4, blue:b3, alpha:mask

t t h t 2stretch percent:2compression quality:50 # 0 to 100, 100% being highest qualityencode type:‘jpg‘ #could do tif, png, kmz

endendend

end

28

DSL E lDSL Example

define name:‘ali_ndvi’, author:‘Pat C.’, revision:1.0scene asset:EO1, instrument:‘ALI’, data:‘L1g’b7 = band(7)b5 = band(5)mask = b7 > 0 and b5 > 0normalized_difference_ratio first:b7, second:b5

stretch percent:2l i ‘ i ’colorize map:‘prism’

compression quality:50 # 0 to 100, 100% being highest qualityencode type:‘kmz‘ #could do tif, png, jpg, kmz

endendend

end

29

NASA SensorWeb v3.0SPS

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO 1

Task

EO-1HyspIRITestbed

OPSBOPSB

GeoTorrentGeoTorrentDataData

Execute Workflow

Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows

WfCS

CompositeCompositeBB

CompositeComposite

Create/Visualize Workflows

GCC GCC Compiler/Compiler/

LinkerLinker

Dynamic SwamoAgent Upload

WPSpp

BurnscarBurnscar......pp

WCPS

LinkerLinker

AMQP

WPSWPSParser /Parser /CodeCode

GeneratorGenerator

Execute AlgorithmAlgosAlgos

AlgosAlgosAlgosAlgos

AlgosAlgosAlgosAlgos

AlgorithmsAlgorithmsCustom Products ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL

v3.0 Simplification #1Create/Visualize Algorithms

ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL

30

NASA SensorWeb v3.0OPSBOPSB

GeoTorrentGeoTorrent

Notifications

Data Atom Feed

EO 1Execute Task

Feature ServerFeature ServerProfileProfile

GetMetada

DataDataGetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1HyspIRITestbed

Execute Workflow

TaskingTaskingProfileProfile

GetFeasibilities

Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlows

WorkflowsWorkflowsProfileProfile

CompositeCompositeBurnscarBurnscar......

CompositeComposite

Execute ProcessProcessingProcessing

ProfileProfileCustom Products

AlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgorithmsAlgorithmsCustom Products

Coverage Coverage ProcessingProcessing

ProfileProfile

Execute Algos

Dynamic Upload

Create/Visualize Flows and Algos

31

NASA SensorWeb v3.0OPSBOPSB

GeoTorrentGeoTorrent

Notifications

Data Atom Feed

GetFeasibilitiesGetFeasibilities EO 1

DataDataExecute Task

GetMetada Feature ServerFeature ServerProfileProfile

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1HyspIRITestbed

Execute Workflow

GetFeasibilities TaskingTaskingProfileProfile

Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlows

WorkflowsWorkflowsProfileProfile

CompositeCompositeBurnscarBurnscar......

CompositeComposite

Execute ProcessProcessingProcessing

ProfileProfileCustom Products

CoverageCoverage AlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgorithmsAlgorithmsCustom Products

Coverage Coverage ProcessingProcessing

ProfileProfile

Execute Algorithms OtherOtherNodesNodes

32

Security / Security / OpenIDOpenID / / OAUthOAUth SecuritySecurity

Distributed Security Providers

2-Factor Auth

NASA SensorWeb v3.0

GetFeasibilitiesGetFeasibilities EO 1

DataDataResources

GetFeasibilitiesGetFeasibilities

SubmitTaskSubmitTask

EO-1HyspIRITestbed

Service Document

FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsGet, Post, Put, DeleteRESTFul /

SecureAPI

CompositeCompositeBurnscarBurnscar......

CompositeCompositeAtom Feeds

API

AlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgorithmsAlgorithms

OtherOtherNodesNodes

33

Security / Security / OpenIDOpenID / / OAUthOAUth SecuritySecurity

Distributed Security Providers

2-Factor Auth