![Page 1: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/1.jpg)
FACE-IT:EarthscienceworkflowsmadeeasywithGlobusandGalaxytechnologies
(Providemorecapabilityformorepeopleatlowercostbydelivering“ScienceasaService”)
RaffaeleMontella1,2,AlisonBrizius2,JoshuaElliott2,DavidKelly2,RaviMadduri2,3,KetanMaheshwari3,CherylPorter4,PeterVilter2,MichaelWilde2,WeiXiong4,MengZhang4 andIanFoster2,3,5
1DepartmentofScienceandTechnologies,UniversityofNaplesParthenope,Naples,ITALY;2ComputationInstitute,ArgonneNationalLaboratoryandUniversityofChicago,Chicago,Illinois,USA;3MathematicsandComputerScienceDivision,ArgonneNationalLaboratory,Argonne,Illinois,USA;
4UniversityofFlorida,DepartmentofAgriculturalandBiologicalEngineering,Gainsville,Florida,USA;5DepartmetofComputerScience,UniversityofChicago,Chicago,Illinois,USA;
DepartmentofScienceandTechnologiesUniversityofNaplesParthenope
MathematicsandComputerScience
Division
DepartmentofAgriculturaland
Biological Engineering
faceit-portal.org usefaceit.org
![Page 2: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/2.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
FacingrealproblemswithInformationTechnology
NobuzzwordRealthings!Anopenplaygroundforthenextgenerationofearthsystemscientists
What’sinaname…
ScienceGatew
ays
Data+Workflows=Re
sults
Theuserprofile…Scientists
Expertsoftheir fieldsLimited programmingskills
Complex experiments
Effectiveandefficientsolutions torealproblemsExperts indesign andabstraction
Inform
ation
Techno
logy
Development-experts (inwizardry)…
BuiltonwidelyusedGalaxy,
Globus, andSwift systems
faceit-portal.org
…lessmagicwands,moremagicminds...
![Page 3: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/3.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
ServiceOrientedScience
• Peoplecreateservices(dataorfunctions)…• whichIdiscover(&decidewhethertouse)…• &composetocreateanewfunction...• &thenpublishasanewservice.
→ Ifind“someoneelse”tohost services, soIdon’thavetobecomeanexpertinoperatingservices&computers!→ Ihopethatthis“someoneelse”canmanage security,reliability, scalability,… !!
2005
![Page 4: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/4.jpg)
IaaS
PaaS
SaaS
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
TheScienceStack
• Galaxy– Interactive execution– Creation,Execution, Sharing,DiscoveringWorkflows
• Globus– Datamanagement– IdentityManagement
• AWS– HTCondor,Chef,EC2,EBS,S3,SNS,NEWT– Spot,Route53,CloudFormation
Makingcomplexthingsstraightforward
Infrastructure
Platform
Software
![Page 5: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/5.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
FromgenestoEarthin6steps
• Datatypes
• Tools
• Toolparameters
TeachingGalaxytospeakEarthScience
• Aggregateddatatypes
• Dataproviders
• Visualizers
![Page 6: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/6.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
StepONEof6:earthsystemdatatypes
• Colorscheme data.Data
data.Text
tabular.Tabular xml.Generic_Xml
enhanced_xml.EnhancedXml generic_json.GenericJson
enhanced_xml.DsRef(RAFT)
geojson.GeoJson geojson.GeometryJsongeojson.BBoxJson geojson.CrsJson
netcdf.NCML
netcdf.NetCDF
ecmwfera40.ECMWFERA40gcm.GCM
acmo.Acmo
RegularGalaxy Extensionsofgeneraldatatypes
Newdatatypesforearthsciences
FACE-ITapplicationsdatatypes
OtherFACE-ITappsdatatypes
TeachingGalaxytospeakEarthScience
![Page 7: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/7.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
StepTWOof6:newtools
• Tool:Isacomputingprocess fedbyoneormoredatasets producing oneormoredatasets
• Itiswrappedoverany kindofexecutable
• Runningbynaïve localscheduler,super-computers,virtualmachinessomewhere inthecloud.
• Eachinputandoutput isdatatyped• Itisdefined usingXML Thetoolspalette
Thesametoolinaworkflow
Atoolindataanalysis
TeachingGalaxytospeakEarthScience(“Globusally”)
![Page 8: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/8.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
StepTHREEof6:toolparameters
• Toolparameters:Definetheuserinterfaceelements foratool
• Regulartoolparameterswraptextfields, radiobuttonsanddropdrownlists.
• Customtoolparameters forGlobus,OpenDap,datepeakingandfeatureselection onmaps.
TeachingGalaxytospeakEarthScience
![Page 9: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/9.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
StepFOURof6:aggregateddatatypes(RAFT*)
• DatasetReferences:XMLbaseddatatypegroupingreferences todifferentdatasets inthesamehistory.
• TheregularGalaxyworksonsingle filedatasets orcomposite filedatasets.
• Actsasa‘struct’oran‘array’oramixofboth.
• Supportsschemas andtranslators.
DsRef(EnhancedXML)
Usedwhen:• Atoolconsumes and/orproducesavariable
numberofdatasets• Thetoolisimplemented usingaSwiftscript
workinginparallel
TeachingGalaxytospeakEarthScience
Globus HTTPS Endpoints(...future…)
![Page 10: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/10.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
StepFIVEof6:dataproviders
• Dataproviders:softwarecomponentsinterfacing thedatasets withthewebbrowser.
• TheyprovidedataasarrayofJSONobjects
• Key/Values, Columnar,custom• Implemented inDatatypeclasses
WebBrowser GalaxyInstance
HistoryDatabaseAssociation
DataProviders
Datatype
DatafileDataset
DataProvider
Webpage…
…dynamicallygenerated...
…formMakotemplate
(mixofserversidepythoncodewithclientsidewebtechnologies)
request
response
template
TeachingGalaxytospeakEarthScience
![Page 11: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/11.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
StepSIXof6(!):mapvisualizers
• Visualizers:client-sidesoftwarecomponentsforinteractivedatavisualization
• Quasi-GIS!
• Map:VisualizesvectordataproducedasGeoJsonobjectsbyadataprovider
• Wms (WorldMapServer):VisualizesrasterdatafromNetCDFdatatypes.
TeachingGalaxytospeakEarthScience
![Page 12: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/12.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
Deliverondemandweathersimulations
• wrf-model.org
• +30kpeoplecommunity
• 150countries
• highcomputingdemanding
WeatherResearchandForecast@AWS:a(real)application.
![Page 13: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/13.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
WeatherForecastonChicagoarea(333m)
• WRF• 4nested
domains• 9kmto333m
Temperature (C°)Cumulated Rain(mm)
WeatherResearchandForecast@AWS
![Page 14: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/14.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
NetCDFscavenging,discoveryandprovisioning
• NetCDFScavenger• Indexing/Ingestion• Tiling• Spatial/metadata
query• Selection• Usage
Breakingthefourthwall
Craweler
Ingester
Tiler
FACE-ITGalaxyToolBox
Discover
Publish
Share
NetCDFdata
![Page 15: Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things](https://reader031.vdocuments.us/reader031/viewer/2022030407/5a8493ca7f8b9a14748b79fd/html5/thumbnails/15.jpg)
FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology
Conclusionsand[now]futureworks
• Face-ITGalaxyisacreativeplaygroundforthenextgenerationofearthscientistspoweredbyGlobus fordatamovementandmore.
• Propose yourapplication,writeyourcodeandshareit!
• Spin-offprojects:extremeweathersimulationsintheBayofNapoli,IT(UniParthenope)
http://www.faceit-portal.org
Mustbe...…continued!