nasa sensorweb: from v2.0 to v3.0 workflows and wcps for ... · nasa sensorweb: from v2.0 to v3.0...
TRANSCRIPT
NASA SensorWeb: From v2.0 to v3.0Workflows and WCPS for Space
ApplicationsApplications
Pat Cappelaere (Vightel)
D M dl (GSFC)
Open Geospatial Consortium TC Meeting
W kfl & C W ki G Dan Mandl (GSFC)
Stu Frye (SGT)Workflow & Coverages Working GroupsJune 2010, Silver Spring MD
1
https://ntrs.nasa.gov/search.jsp?R=20100028151 2020-04-21T21:56:32+00:00Z
AgendaAgenda
• Our Standard Compliance Evolution
• v2 0 Accomplishments & Lessons Learnedv2.0 Accomplishments & Lessons Learned
• v3.0 Architecture Migration– Towards A Unified Interface
– Examplesp
22
SatellitesSatellitesInt’lInt’l
UsersUsers
Int’lInt’lDisastersDisasters NASANASA
SensorWebSensorWebSensorWebSensorWebProjectProject
3
Big Software Project: Cost is Big Software Project: Cost is f(codef(code))
Cos
tC
ost
Lines of CodeLines of Code
4
Our Reality...Our Reality...
Fund
ing
Fund
ing
2005 ... 20102005 ... 2010
FF
5
But we have made some good progress...But we have made some good progress...
ties
ties
Cap
abili
tC
apab
ilit
2005 ... 20102005 ... 2010
6
Number of OGC Standards Has Increased SignificantlyNumber of OGC Standards Has Increased Significantly
ndar
dsnd
ards
OG
C S
tan
OG
C S
tan
which wouldwhich wouldincrease our cost!increase our cost!
2005 ... 20102005 ... 2010
# O
# O
7
S O OGC C li H Si ifi tl D dS O OGC C li H Si ifi tl D d
ance
ance
So Our OGC Compliance Has Significantly DecreasedSo Our OGC Compliance Has Significantly Decreased
C C
ompl
iaC
Com
plia
2005 ... 20102005 ... 2010
OG
CO
GC
8
v2.0 Architecture
9
NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
EO 1ASTER
FORMOSATSPOT...
Sensor Planning Service10
NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
You’ve Got Data EO 1ASTER
FORMOSATSPOT...
WNSWNSYou ve Got Data
Web Notification Service11
NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
You’ve Got Data EO 1ASTER
FORMOSATSPOT...
OPSBOPSBYou ve Got Data
Publish/Subscribe Basic Service12
NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
ASTERFORMOSAT
SPOT...OPSBOPSB
Raw Data (L1R, L1G)SOSSOS
DataData
Sensor Observation Service13
NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
ASTERFORMOSAT
SPOT...OPSBOPSB
Raw Data (L1R, L1G)GeoTorrentGeoTorrent
DataData
GeoTorrent Service14
NASA SensorWeb 2.0Orbit/STKOrbit/STKSPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
EO 1ASTER
FORMOSATSPOT...
OPSBOPSB
GeoTorrentGeoTorrentDataData
Derived Products CompositeCompositeCompositeCompositeDerived Products
WPS
pp
BurnscarBurnscar......
(JPL NG )
pp
WPS(JPL, NG...)
Web Processing Service15
NASA SensorWeb 2.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
ASTERFORMOSAT
SPOT...OPSBOPSB
GeoTorrentGeoTorrentDataData
Execute Workflow
Data Atom Feed
WfCS
Create/Visualize Workflows
FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlows
CompositeCompositeCompositeComposite
Create/Visualize Workflows
WPS
pp
BurnscarBurnscar......
(JPL NG )
pp
WPS (JPL, NG...)
Workflow Chaining Service16
NASA SensorWeb 2.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1
Task
ASTERFORMOSAT
SPOT...OPSBOPSB
GeoTorrentGeoTorrentDataData
Execute Workflow
Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows
WfCS
CompositeCompositeBB
CompositeComposite
Create/Visualize Workflows
WPSpp
BurnscarBurnscar......pp
WCPSAccess Raster Data
WPSWPS WCSWCS
Execute Algorithm Access Raster Data
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgorithmsAlgorithmsCustom Products
Web Coverage Processing ServiceCreate/Visualize Algorithms
17
NASA SensorWeb 2.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO 1
Task
EO-1HyspIRITestbed
OPSBOPSB
GeoTorrentGeoTorrentDataData
Execute Workflow
Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows
WfCS
CompositeCompositeBB
CompositeComposite
Create/Visualize Workflows
GCC GCC Compiler/Compiler/
LinkerLinker
Dynamic SwamoAgent Upload
WPSpp
BurnscarBurnscar......pp
WCPS
LinkerLinker
AMQP
WPSWPSParser /Parser /CodeCode
GeneratorGenerator
Execute AlgorithmAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgorithmsAlgorithmsCustom Products ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL
WCPS With a TwistCreate/Visualize Algorithms
ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL
18
NASA SensorWeb 2.0
WCPS: A SQL-like Query Language
for c in ( NIR )return
encode((char) ( ((c.0 / ((float)c.0 + c.1))-(c.1 / ((float)c.0 + c.1)))> 0.6 ) * 255,
"png" )
WCPS
WPSWPS
Execute AlgorithmAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgorithmsAlgorithmsCustom Products
WCPS LanguageCreate/Visualize Algorithms
19
NASA SensorWeb 2.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
Task
Visual EditorT t EditOPSBOPSB
GeoTorrentGeoTorrent
define name: flow1
Text Editor
Execute Workflow
Data Atom Feed FlowsFlowsFlowsFlows
FlowsFlowsFlowsFlows
FlowsFlowsCreate/Visualize Workflows
WfCSdefine name: flow1task_eo1process_alinotify_user
CompositeCompositeBB
CompositeComposite
Create/Visualize Workflows y_end
WPSBurnscarBurnscar......
WCPSBPMN 1.0
WPSWPS
Execute AlgorithmAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgorithmsAlgorithmsCustom Products
WfCS LanguageCreate/Visualize Algorithms
20
v2.0 Lessons Learned
Too Many Ways to Do Similar Things
Execute Task, Process, Workflow, Algorithm...
Hard Learning Curve on Users and ImplementorsHard Learning Curve on Users and Implementors
Two Languages: g g
For Workflows and Algorthms Developments
What About Web Service Security?
21
NASA SensorWeb v3.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO 1
Task
EO-1HyspIRITestbed
OPSBOPSB
GeoTorrentGeoTorrentDataData
Execute Workflow
Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows
WfCS
CompositeCompositeBB
CompositeComposite
Create/Visualize Workflows
GCC GCC Compiler/Compiler/
LinkerLinker
Dynamic SwamoAgent Upload
WPSpp
BurnscarBurnscar......pp
WCPS
LinkerLinker
AMQP
WPSWPSParser /Parser /CodeCode
GeneratorGenerator
Execute AlgorithmAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgorithmsAlgorithmsCustom Products ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL
v 3.0 Simplification #1Create/Visualize Algorithms
ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL
22
NASA SensorWeb v3.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO 1
Task
EO-1HyspIRITestbed
OPSBOPSB
GeoTorrentGeoTorrentDataData
Execute Workflow
Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows
WfCS
CompositeCompositeBB
CompositeComposite
Create/Visualize Workflows
WPSBurnscarBurnscar......
pp
WCPS
SEXPWPSWPS
Execute AlgorithmAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosCustom Products
SEXP
Create/Visualize Algos23
SEXP?SEXP?
24
S ES-Exp or S b li E iSymbolic Expressions
Think LISP or Parse TreesThink LISP or Parse Trees(= 4 (+ 2 2))
25
NASA SensorWeb v3.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO 1
Task GenericGenericOnboard SWAMO Onboard SWAMO
AgentAgent
EO-1HyspIRITestbed
OPSBOPSB
GeoTorrentGeoTorrent
Execute Workflow
Data Atom Feed FlowsFlowsFlowsFlows
FlowsFlowsFlowsFlows
FlowsFlowsCreate/Visualize Workflows
WfCS
CompositeCompositeBB
CompositeComposite
Create/Visualize Workflows
WPSBurnscarBurnscar......
WCPS
WPSWPS
Execute AlgorithmAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgorithmsAlgorithmsCustom Products
SEXP
Create/Visualize Algorithms26
EO 1
GenericGenericOnboard SWAMO Onboard SWAMO
AgentAgent
EO-1HyspIRITestbed
TextText VisualVisualCreate/Update
TextText VisualVisual
Domain
SEXP or JSONExecute
FlowsFlows
SpecifcLanguage (DSL) AlgorithmsAlgorithms
SEXP
RMagickRMagick
APIAPI
SecuritySecurity RubyRuby
GDALGDAL
gg
DataDataSecuritySecurity
2-factor Auth27
DSL E lDSL Example
define name:‘ali_l1g_visible’, author:‘Pat C.’, revision:1.0scene asset:EO1, instrument:‘ALI’, data:‘L1g’b5 = band(5)b4 = band(4)b3 = band(3)mask = b5 > 0composite_image red:b5, green:b4, blue:b3, alpha:mask
t t h t 2stretch percent:2compression quality:50 # 0 to 100, 100% being highest qualityencode type:‘jpg‘ #could do tif, png, kmz
endendend
end
28
DSL E lDSL Example
define name:‘ali_ndvi’, author:‘Pat C.’, revision:1.0scene asset:EO1, instrument:‘ALI’, data:‘L1g’b7 = band(7)b5 = band(5)mask = b7 > 0 and b5 > 0normalized_difference_ratio first:b7, second:b5
stretch percent:2l i ‘ i ’colorize map:‘prism’
compression quality:50 # 0 to 100, 100% being highest qualityencode type:‘kmz‘ #could do tif, png, jpg, kmz
endendend
end
29
NASA SensorWeb v3.0SPS
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO 1
Task
EO-1HyspIRITestbed
OPSBOPSB
GeoTorrentGeoTorrentDataData
Execute Workflow
Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsCreate/Visualize Workflows
WfCS
CompositeCompositeBB
CompositeComposite
Create/Visualize Workflows
GCC GCC Compiler/Compiler/
LinkerLinker
Dynamic SwamoAgent Upload
WPSpp
BurnscarBurnscar......pp
WCPS
LinkerLinker
AMQP
WPSWPSParser /Parser /CodeCode
GeneratorGenerator
Execute AlgorithmAlgosAlgos
AlgosAlgosAlgosAlgos
AlgosAlgosAlgosAlgos
AlgorithmsAlgorithmsCustom Products ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL
v3.0 Simplification #1Create/Visualize Algorithms
ENVI/IDLENVI/IDL ENVI/IDLENVI/IDL
30
NASA SensorWeb v3.0OPSBOPSB
GeoTorrentGeoTorrent
Notifications
Data Atom Feed
EO 1Execute Task
Feature ServerFeature ServerProfileProfile
GetMetada
DataDataGetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1HyspIRITestbed
Execute Workflow
TaskingTaskingProfileProfile
GetFeasibilities
Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlows
WorkflowsWorkflowsProfileProfile
CompositeCompositeBurnscarBurnscar......
CompositeComposite
Execute ProcessProcessingProcessing
ProfileProfileCustom Products
AlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgorithmsAlgorithmsCustom Products
Coverage Coverage ProcessingProcessing
ProfileProfile
Execute Algos
Dynamic Upload
Create/Visualize Flows and Algos
31
NASA SensorWeb v3.0OPSBOPSB
GeoTorrentGeoTorrent
Notifications
Data Atom Feed
GetFeasibilitiesGetFeasibilities EO 1
DataDataExecute Task
GetMetada Feature ServerFeature ServerProfileProfile
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1HyspIRITestbed
Execute Workflow
GetFeasibilities TaskingTaskingProfileProfile
Data Atom Feed FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlows
WorkflowsWorkflowsProfileProfile
CompositeCompositeBurnscarBurnscar......
CompositeComposite
Execute ProcessProcessingProcessing
ProfileProfileCustom Products
CoverageCoverage AlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgorithmsAlgorithmsCustom Products
Coverage Coverage ProcessingProcessing
ProfileProfile
Execute Algorithms OtherOtherNodesNodes
32
Security / Security / OpenIDOpenID / / OAUthOAUth SecuritySecurity
Distributed Security Providers
2-Factor Auth
NASA SensorWeb v3.0
GetFeasibilitiesGetFeasibilities EO 1
DataDataResources
GetFeasibilitiesGetFeasibilities
SubmitTaskSubmitTask
EO-1HyspIRITestbed
Service Document
FlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsFlowsGet, Post, Put, DeleteRESTFul /
SecureAPI
CompositeCompositeBurnscarBurnscar......
CompositeCompositeAtom Feeds
API
AlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgosAlgorithmsAlgorithms
OtherOtherNodesNodes
33
Security / Security / OpenIDOpenID / / OAUthOAUth SecuritySecurity
Distributed Security Providers
2-Factor Auth