PowerPoint Presentation

Earth System CoG and the Earth System Grid Federation: A Partnership for Improved Data Management and Project Coordination

BESSIG March 18, 2014Boulder, CO

Sylvia Murphy (NOAA/CIRES) (sylvia.murphy@noaa.gov), Luca Cinquini (JPL/NOAA), Cecelia DeLuca (NOAA/CIRES), Allyn Treshansky (NOAA/CIRES)

Presentation OutlineOverview of ESGFESGF UsageOverview of CoGCoG CapabilitiesFuture WorkOverview of ESGFThe Earth System Grid Federation (ESGF) is a multi-agency, international collaboration of people and institutions working together to build an open source software infrastructure for the management and analysis of Earth Science data on a global scale.ESGF is a system of distributed and federated Nodes that interact dynamically through a Peer-To-Peer (P2P) paradigm.A client (browser or program) can start from any Node in the federation and discover, download and analyze data from multiple locations as if they were stored in a single central archive.

Distributed: data and metadata are published, stored and served from multiple centers (Nodes)Federated: Nodes interoperate because of the adoption of common services, protocols and APIs, and establishment of mutual trust relationshipsDynamic: Nodes can join/leave the federation dynamically - global data and services will change accordingly

Data Node : secure data publication and accessIndex Node : metadata indexing and searching (w/ Solr)web portal UI to drive human interactiondashboard suite of admin applicationsmodel metadata viewer plugin Identity Provider : user authentication and group membership Compute Node : analysis and visualization

3ESGF UsageCMIP5 (Phase 5 of the Climate Modeling Intercomparison Project) possibly the largest coordinated scientific modeling effort of all time.40+ models, 25+ modeling centers, 17 countriesglobal, distributed archive comprising 2.5 PB of dataobs4MIPs: NASA and DoE observations packaged as CMIP5 output for easy comparisonana4MIPs: reanalysis dataCORDEX: regional climate models, 2PB dataTAMIP: atmospheric models intercomparisonGeoMIP: geo-engineering models intercomparisonDCMIP: dynamic core models intercomparison

Overview of CoGCoG is a collaboration environment and hub to connect projects in the Earth sciences.It hosts software development projects, model intercomparison projects (MIPS), and university short-courses or workshops. It includes a configurable search to data on ANY ESGF data node.It provides projects with a wiki and customizable navigation to wiki content. It contains an ontology for the description and management of projects and provides a consolidated look at this content across a projects network.It contains a file server for documents and images. It provides services for Earth system model metadata collection and display.

Some of the 74 projects hosted on CoG include:Ana4MIPsObs4MIPsNational Climate Predictions and Projections Platform (NCPP)Climate Informatics (University of Michigan)Earth System Documentation (ES-DOC)NOAAs High Impact Weather Prediction Project (HIWPP)Earth System Prediction Capability (ESPC)Dynamical Core Model Intercomparison Project (DCMIP)Customizable Data ServicesInterfacing with ESGFSearch widget can be turned on/off.Search can be narrowed to any ESGF node and to any project (e.g. CMIP).Search facets can be created, deleted, and grouped.Help text can be added to the top of the search page.Search results can be saved to a Data Cart associated with a user. Items in the Data Cart persist. Search results can be:Forwarded to the Live Access Server (LAS) for simple visualization. Downloaded directly via a WGET script.Associated with model metadata if it exists.

ESGF Search Customization

Data Cart

Items in the Data Cart can be sent individually or collectively to LAS or WGET.The Data Cart is associated with a user and not a project.Show Metadata

Wiki and Collaboration Tools

The CoG layout is color-coded: The right-hand side (dark yellow) is where services (data, news, project connectivity) are located.The Upper Navigation bar (dark teal) contains links to project-level metadata. On the left (light teal) is an auto-generated navigation system created when projects develop freeform content. The central portion of the site is a wiki that allows projects to create their own content. Screenshot of the CoG project workspace for the 2012 Dynamical Core Model Intercomparison (DCMIP) Workshop.Project Networks and the Project BrowserProjects in CoG are arranged in a hierarchy of Parents, Peers, and Children.The Project Browser displays the network and allows for inter-project navigation. Projects can be tagged with keywords and projects can be searched for using keywords.

Project-level Metadata Roll-upManagement of information is a major problem in projects that involve many sub-projects, partners, multiple leads, and many resources. CoG acts as an index into project information that is necessary for coordination and collaboration and enables people responsible for overall coordination to quickly get consolidated views of information.

This example shows the Partners feature that allows projects to list their project partners and include a logo for each. Below the list for ED-DOC is a consolidated view of the partners for ES-DOCs peer projects. CoG SchemaThe CoG schema contains classes to describe software development projects, short-courses or meetings, and overall project coordination. Projects select which metadata to display via a simple web form.

Project-level metadata is linked in standardized locations via the upper navigation bar.

UML Diagramhttps://earthsystemcog.org/site_media/projects/cog/cog_ontology.pngResourcesResources are pointers to data, files, and URLs.Resources folders can be created, moved, and deleted.Projects can turn on a set of standardized Resources folders (e.g. Presentations, Minutes).Saved data searches can be saved as a Resource.Each Resource can have a private wiki-based notes page to facilitate discussions.

NewsNews is a way to send announcements across a project network.News is visible in the news widget on any targeted project.News will be added to social media (Google+, Facebook, Twitter, RSS) in a future release.

Model Metadata ServicesThe CoG Team is partnering with the international Earth System Documentation (ES-DOC) project to develop and use an Earth System Model metadata entry and view capability.The ES-DOC Viewer is a lightweight JavaScript plugin that will display any Common Information Model (CIM) record. The ES-Questionnaire collects standardized CIM metadata through a high-customizable web form. The output is saved to a community CIM repository.

Future WorkCoG ESGF Integration (through summer 2014):CoG is going to replace the ESGF web front end.CoG will be federated so that projects hosted on one CoG-ESGF instance will be visible on others.OpenID access added.Look and feel will be more customizable to meet institutional branding requirements.

Possible other features:Be able to export the CoG ontology. Be able to list non-hosted projects the Project Browser. Be able to version the wiki. Enable RSS and social networking for news. Enable non-wiki links in the left navigation bar.

Questions?Questions and contacts:cog_support@list.woc.noaa.gov

CoG: https://earthsystemcog.org/ESRL ESGF data node: http://hydra.fsl.noaa.gov/esgf-web-fe/PCMDI ESGF data node: http://pcmdi9.llnl.gov/esgf-web-fe/JPL ESGF data node: http://esg-datanode.jpl.nasa.gov/esgf-web-fe/