Download - NEESGrid Data and MetaData Technology
![Page 1: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/1.jpg)
NEESGrid Data and MetaDataTechnology
Kincho Law, Jun Peng, Jim Eng, Terry Weymouth, Paul Hubbard, Charles
Severance
![Page 2: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/2.jpg)
Goals Data is online and persistent Data and Metadata are supported together Data migrates transparently including security, and metadata Data is completely secure with access controls but security does not
get in the way Data provenance - how was it gathered, how has it been manipulated? Data in support of research publication Support for repeatable experiments Data oriented research computation support Support for workflow
![Page 3: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/3.jpg)
The Slide
gas
mile
gallon
car
model
ratio
mileage
fluid
volume
denominator
numerator
measurementdistance
rate
travel
consumption
fuel
unit
efficiency
vehicle
estimate
Metadata
Data
Data Viewers
Data
Map
pers
Data Ingestors
There is a layer is where we develop tools which take advantage and begin to depend on of the “meaning” of the data – where we begin to depend on the meaning of a second.
Where we make a viewer capable of viewing a certain type of object.
This is where we build things which make use of knowledge.
This layer will never be complete but it is a large focus of the coming months.
Concept
s
Search
09/2003
![Page 4: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/4.jpg)
NEESgrid Data – Core Elements Local Repository Central Repository JAVA APIs – Run locally on the same system as a
repository or over OGSA Web Services– NEES File Management Services– NEES Meta Data Services
Data Viewers– Streaming (numeric, X/Y graph)– Stored (X/Y graph, 2-D structure, video)
![Page 5: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/5.jpg)
NEESdata
NEESpop
LocalRepository
Core Elements
AP
I
CentralRepository
DataTeamlets
Data Acquisition
Workstation
AP
I
DataTeamlets
AP
I
Data/MDIngestTools
Data tools
Data viewers
Grid and Web Services
![Page 6: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/6.jpg)
A Simple Experimental Scenario
DAQ System
Glue
Test S
pe
cimen
Lab
view
Developer System
Researcher System
![Page 7: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/7.jpg)
Simulation System
Code
Simulation System
Code
A Simulation ScenarioDeveloper System
Simulation System
Code
![Page 8: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/8.jpg)
NCSA NEESPop (1.1)
Colorado
NEESPop (1.1)
Incoming FTP
NEESMost (Win XP)
UIUC/Newmark
NEESPop (2.0)
LabView DAQ
MatLab Host And Real-Time Target Control
SystemSim Controller
CO
LabView DAQ
NSDS
UIUC
Test Specimen
Matlab Computational
Model
Shore-Western
Test Specimen
Incoming FTP
NSDSRepository
CO
NCSA
UIUC
Meta
Series of files
Complete file (aggregated)
NTCP
Site / LocationComputer Process
NCSA
Ingest
Ingest
NTCP
Matlab
NTCP
Ftp
NTCP
Wires
NFMS/NMDS
NSDS
File I/O
Plug In
Ingest
UIUC
MOST Scanario
![Page 9: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/9.jpg)
Overview of tools and technologies Model development in RDF Project Browser Repository Browser Electronic Notebook Data Mappings Data Viewer Data Turbine Data As Video Still Cameras
11/2003
![Page 10: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/10.jpg)
RDF Integration
Some of the data and meta data task force members are using Protégé-2000 to develop their models and expressing them in RDF.
RDF and NEESML are very similar but not identical so it may be challenging to ingest any arbitrary RDF
We expect that we will be able to map a subset of RDF to NEESML for ingestion or adapt an RDF parser (Jena or Raptor) to ingest that subset directly into the repository
11/2003
![Page 11: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/11.jpg)
Models + Data Model
Repo
Models
Configure
Data
Load
Con
figur
e
RDF/OWL
RDF<owl:ObjectProperty rdf:ID="hasPublications"> <rdfs:domain> <owl:Class> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Project"/> <owl:Class rdf:about="#Task"/> </owl:unionOf> </owl:Class> </rdfs:domain> <rdfs:range rdf:resource="#Publications"/> </owl:ObjectProperty>
![Page 12: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/12.jpg)
Protégé - 2K
Models + Data Model
Repo
Models
Configure
Data
Load
Con
figur
e
RDF/OWL
RDF
![Page 13: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/13.jpg)
Repository Browser
![Page 14: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/14.jpg)
Electronic Notebook
Collaborative effort with the DOE SciDAC– Electronic notebook - metadata entry– Data mapping– Data provenance– Data display– Slide data/metadata jakarta.apache.org/slide/
Ultimate integration will be via JSR-170 www.scidac.org/SAM/ collaboratory.emsl.pnl.gov/docs/collab/sam/samtechoverview.html
![Page 15: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/15.jpg)
DOE ELN / Example
![Page 16: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/16.jpg)
DOE ELN / NEESgrid Integration (to date)
SAM /Slide
Repository
ChefGrid
Security
NEESgridRepository
MyProxy
Technology Celebration
![Page 17: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/17.jpg)
DOE ELN / NEESgrid Integration (ultimate)
ChefGrid
Security
NEESgridRepository
MyProxy
![Page 18: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/18.jpg)
Mappings and the Data Viewer NSDS (ISO 8601 Time channel) Column data with time recorded as a column Column – generate time Column – generate time – trigger filter
Channel units: g,g,in,kipTime ATL1 ATT12002-11-13T15:48:55.26499 -0.006409 0.004272 2002-11-13T15:48:55.36499 -0.005798 -0.003662
100.0000.435 0.161 -1.016 -0.981 0.430 0.161 -1.016 -0.9770.435 0.161 -1.016 -0.977
public class NEESDataMap{ public static boolean repoMap(File mainFile,
File mappingFile, String mapping) {
// Code here }}
![Page 19: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/19.jpg)
![Page 20: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/20.jpg)
Data Turbine
Commercial, free data streaming toolkit
![Page 21: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/21.jpg)
Data Turbine (cont)
Existing data viewers will be adapted to access and display data from data turbine
Data acquisition software will be adapted to place information in Data Turbine Channels
Metadata elements will be developed to represent data turbine live, stored, and derived channels
New efforts (video as data) will be developed from the ground up using Data Turbine
outlet.creare.com/rbnb/
11/2003
![Page 22: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/22.jpg)
Video as Data
Follow on to initial demonstration at ORST Experiment based development: Minnesota Design phase complete Joint effort, NEESGrid SI, ORST, Minnesota,
UC Davis, Texas, Buffalo, and others
![Page 23: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/23.jpg)
DT Main System
CTLNTCPPlugin
Data Turbine - Today
NTCPControl
DAQData
CaptureDT Client
BT848 rbnbjcap
DT Client
NEESNSDSDriver
ControlPlugin
AXIS / DTGateway
Axis
![Page 24: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/24.jpg)
DT Main System
Data Turbine
NTCPControl
ControlPlugin
Thumbs
Control
Make Smoothie
Technology Celebration
![Page 25: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/25.jpg)
DT Main System
PTZ/USB
StillCapture
DT Client
DT Capturing
CameraControl
DAQData
CaptureDT Client
AudioAudio
EncoderDT Client
BT848 rbnbjcap
DT Client
Still Capture - Minnesota / Paul HubbardVideo capture - From CreareAudio capture - From Creare (TBD)Data Capture - From sites (upwards compatible)NEES NSDS Driver - Paul HubbardCamera Control Plugin - Mich / Minn
NEESNSDSDriver
Each still capture produces two channels - Small 1-5fps stream + large single images when picture is taken
ControlPlugin
![Page 26: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/26.jpg)
DT Main System
User Views / Still Camera
Quicktime Storage System
QuicktimeSlicingTool
Still Image / Camera Control
~
< >^
^
< >
ControlPlugin
Data Viewer
ThumbnailProcess
Thumbnail - uBuffalo / UmichiganThumbnail viewer - Creare / MichCamera Control Applet - Minn / MichQuicktime Slicing tool - Mich (low)Stored Data Viewer - MichJPEG Viewer - CreareQuickime Viewer - Apple
Thumbnail + Audio + Data
< > +
![Page 27: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/27.jpg)
MinnesotaMock - up
If you area developer and interested in following / helping / participating in this activity, join the mailing list
![Page 28: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/28.jpg)
Tool List - To Do
Next release of repository Integrate ELN into repository DAQ Control Panel in CHEF
– Set/Retrieve Metadata– Start / Stop– Ingest data from staging space
Data Turbine Control Panel in CHEF– Start / Stop / Configure Sources
• Video | Audio | Data | Thumbnail– Control permanent storage of video
![Page 29: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/29.jpg)
To Do (cont)
NTCP Debugging and Monitoring in CHEF– Needed Data Turbine
Data as Video Client Tools in CHEF– New Monitor Tool– Still Image– Camera Control
Data Turbine Audio Capture We may need to support XML Schema QuickTime Capabilities
– Archive, retrieve, slice, dice, convert, present– Probably will not be completed as part of SI effort
![Page 30: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/30.jpg)
Data Model Work
Data Curation Summit– Understand issues form go-forward plan– Meeting 3/18/2004
Data and Metadata Task Force– Finish the tsunami and centrifuge models
DSAC Committee– Meeting 3/19/2004
![Page 31: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/31.jpg)
Summary
In September 2003, we met and “re-visioned” data
A bunch of requirements gathering and development has been done– The “high risk” elements are working now
There is more to do - We will run out of time– Evolutionary development approach - there will
always be usable working code - we will stop when we run out of time
![Page 32: NEESGrid Data and MetaData Technology](https://reader035.vdocuments.us/reader035/viewer/2022062221/568148b5550346895db5cc19/html5/thumbnails/32.jpg)
The people…
Gokhan Peckan - Data Models Kincho Law - Data Models / Software design Jun Peng - Data Models / Software design Jim Eng - Parse / Ingest / RDF / Project
Browser Jim Myers - Electronic Notebook Terry Weymouth - Data Turbine Paul Hubbard - DAQ and NTCP Joe Futrelle - Data / Metadata Repository