data and metadata challenges

1
www.walsaip.uprm.edu W ALSAIP WALSAIP Supported By This research aims to provide solution to problems on remote sensing such as lack of direct physical relationship between data and metadata files and the ambiguity of interpretation between different users with heterogeneous systems (software in which data/metadata will be processed), and platforms (Linux, Windows, Unix, ...etc). To address the accessibility and interoperability aspects of data in the Wide Area Large Scale Automated Information Processing Project (WALSAIP) this work explores the encapsulation of data and metadata together in order to create a direct relationship between them ensuring the usefulness of each reading taken from sensors. Data and Metadata Challenges Data and Metadata [1] Manetti Luca, Terribilini Andrea, Knecht Alfredo, “Autonomous Remote Monitoring System for Landslides”, SPIE’s 9th Annual International Symposium on Smart Structures and Materials, 2002, San Diego, CA. [2] Nativi Stefano, Giuli Dino, Innocenti Emilio Bugli, “Interoperability for Multimedia Systems to Support Decision-Makers in the Environment Sector” IEEE International Conference On Multimedia Computing and Systems, Volume 2, June 1999, Pages 338-342 [3] Dong-Jun Won, Il-Yop Chung, Joong-Moon Kim, Seung-Il Moon, Jang-Cheol Seo, Jong-Woong Choe, D Won, II-Yop Chung, J. Kim, S. Moon, J. Seo, J. Choe, “Development of Power Quality Monitoring System with Central Processing Scheme”, Power Engineering Society Summer Meeting, IEEE, South Korea, pp. 915-919 vol.2, 21-25 July 2002. [4] MANTIS Project (MultimodAl Networks of In-situ Sensors): http://mantis.cs.colorado.edu/index.php/tiki-index.php Abstract 1 WALSAIP Conceptual Model 2 References 6 Ongoing Work 5 Problem Formulation 3 Proposed Solution 4 Luz V. Acabá-Cuevas – M.S. Student Prof. Domingo Rodríguez – Advisor AIP Group, ECE Department, University of Puerto Rico, Email: [email protected] Mayagüez Campus Automated XML Schema Representation in Sensor Array Processing Systems Data Representation Systems Signal Conditioning System Raw Data Servers Computational Signal Processing Systems Signal Data Post-processing Pre-processing Stage Post-processing Stage Processing Stage INTERNET Computed Data Servers Information Rendering Systems Sensor Array Structures Signal Data - all readings collected directly from sensors. Metadata – data that describes data. Metadata is crucial to provide researchers a concrete idea of the real conditions in which data was collected. Metadata is a determinant of how the environment influenced the measurement in case of abnormal findings. There is a need to ensure direct physical relationship between data and metadata files in order to ensure the significance of readings. Data should be interoperable across heterogeneous users. Heterogeneous users are users with different data architectures, storage systems, and platforms. A mechanism should be design to make data readable and comprehensive across users heterogeneous users and different disciplines. Lack of support for dynamic metadata. Need to incorporate information from “Human Sensors”. Study Mechanisms for data and metadata acquisition Study correspondences of data and metadata files Figure 1. WALSAIP Conceptual Model Figure 3. Example: NERR System Data/Metadata Data + Metadata + Processing Decision Making! Figure 2. Decision Making Input Figure 4. Shannon’s Theory and XML Processing AUTOMATE! U ser Informati on Source XML Coder Communicatio n Channel Rx Receiver Tx Transmitter XML Decoder U ser Informatio n Destinatio n Hazards: Jamming Interferenc e Power Failure Etc…. Data Meta Data XML XML Data Meta Data Using Java, XML, and FTP technologies, will be used to encapsulate data and metadata files together in order to transfer them across multiple server. The encapsulation can be reversed in order to fit information destination. Once encapsulated, files can be transferred using FTP technology. The user can create a Stencil in order to customize XML tags that are going to be used in the encapsulation. Applying software engineering techniques to the design of the solution. Generating the code that implements the solution. Identifying potential test cases to perform functional verification test at the end of the coding. Integrate the solution to the WALSAIP architecture. Implementation Effort 5 Develop a framework for sensor signal acquisition and server storage of raw data. Implement solution to force direct and specific relationship between data and metadata files. Analyze dynamic metadata management. <?xml version="1.0" encoding="iso-8859-1" ?> <walsaip> <metadata> <research> <researchName>Wide Area Large Scale Automated Information Processing</researchName> <intitution>University of Puerto Rico at Mayaguez</intitution> <department>Department of Electrical and Computer Engineering</department> <address>PO Box 9000, Mayagüez, P.R. 00681-9000</address> <phone>787-832-4040</phone> <contact>Domingo Rodriguez</contact> <email>[email protected]</email> </research> <sensingInfo> <InitialTimeStamp>2006-07-05 22:23:00.14</InitialTimeStamp> <endingTimeStamp>2006-07-06 22:23:00.14</endingTimeStamp> <nodeID>0</nodeID> <samplingRate>138</samplingRate> <dataType>humidity</dataType> </sensingInfo> </metadata> <data> 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 </data> </walsaip> 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 Metadata Data 2006-07-05 22:23:00.14 2006-07-06 22:23:00.14 0 138 humidity Figure 5. Data and Metadata Encapsulation Process N ote: Static Local Inform ation Input Server Application Output Input 1 K D 1 K M K D ~ K M ~ Internet K D K M K K K M M M ~ 1 K K K D D D ~ 1 - - Internet Figure 5. Information Flow Using Shannon’s Theory The solution contemplates dynamic metadata management. Data and metadata can be enhanced with user observations. User can edit the obtained data by adding comments and parameters.

Upload: katoka

Post on 15-Jan-2016

45 views

Category:

Documents


1 download

DESCRIPTION

5. 1. 2. 6. 5. 3. 4. Implementation Effort. Abstract. WALSAIP Conceptual Model. References. Ongoing Work. Problem Formulation. Proposed Solution. Study Mechanisms for data and metadata acquisition. Data + Metadata + Processing Decision Making!. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Data and Metadata Challenges

www.walsaip.uprm.eduWALSAIP

WALSAIPSupported By

This research aims to provide solution to problems on remote sensing such as lack of direct physical relationship between data and metadata files and the ambiguity of interpretation between different users with heterogeneous systems (software in which data/metadata will be processed), and platforms (Linux, Windows, Unix, ...etc). To address the accessibility and interoperability aspects of data in the Wide Area Large Scale Automated Information Processing Project (WALSAIP) this work explores the encapsulation of data and metadata together in order to create a direct relationship between them ensuring the usefulness of each reading taken from sensors.

Data and Metadata Challenges

Data and Metadata

[1] Manetti Luca, Terribilini Andrea, Knecht Alfredo, “Autonomous Remote Monitoring System for Landslides”, SPIE’s 9th Annual International Symposium on Smart Structures and Materials, 2002, San Diego, CA.

[2] Nativi Stefano, Giuli Dino, Innocenti Emilio Bugli, “Interoperability for Multimedia Systems to Support Decision-Makers in the Environment Sector” IEEE International Conference On Multimedia Computing and Systems, Volume 2, June 1999, Pages 338-342

[3] Dong-Jun Won, Il-Yop Chung,   Joong-Moon Kim,  Seung-Il Moon,   Jang-Cheol Seo,  Jong-Woong Choe, D Won, II-Yop Chung, J. Kim, S. Moon, J. Seo, J. Choe, “Development of Power Quality Monitoring System with Central Processing Scheme”, Power Engineering Society Summer Meeting, IEEE, South Korea, pp. 915-919 vol.2, 21-25 July 2002.

[4] MANTIS Project (MultimodAl Networks of In-situ Sensors): http://mantis.cs.colorado.edu/index.php/tiki-index.php

Abstract1

WALSAIP Conceptual Model2

References6

Ongoing Work5

Problem Formulation3

Proposed Solution4

Luz V. Acabá-Cuevas – M.S. Student Prof. Domingo Rodríguez – AdvisorAIP Group, ECE Department, University of Puerto Rico, Email: [email protected] Mayagüez Campus

Automated XML Schema Representation in Sensor Array Processing Systems

Data Representation Systems

Signal Conditioning System

Raw Data Servers

Computational Signal Processing Systems

Signal Data Post-processing

Pre-processing Stage Post-processing StageProcessing Stage

INTERNETComputed Data

Servers

Information Rendering

Systems

Sensor Array Structures

Signal Data - all readings collected directly from sensors.

Metadata – data that describes data. Metadata is crucial to provide researchers a concrete idea of the real conditions in which data was collected. Metadata is a determinant of how the environment influenced the measurement in case of abnormal findings.

There is a need to ensure direct physical relationship between data and metadata files in order to ensure the significance of readings.

Data should be interoperable across heterogeneous users. Heterogeneous users are users with different data architectures, storage systems, and platforms.

A mechanism should be design to make data readable and comprehensive across users heterogeneous users and different disciplines.

Lack of support for dynamic metadata.

Need to incorporate information from “Human Sensors”.

Study Mechanisms for data and metadata acquisition

Study correspondences of data and metadata files

Figure 1. WALSAIP Conceptual Model

Figure 3. Example: NERR System Data/Metadata

Data + Metadata + Processing

Decision Making!

Figure 2. Decision Making Input

Figure 4. Shannon’s Theory and XML Processing

AUTOMATE!

User

Information Source

XMLCoder

Communication

Channel

RxReceiver

TxTransmitter

XMLDecoder

User

Information Destination

Hazards:

JammingInterferencePower FailureEtc….

DataMetaData

XML

XMLData

MetaData

Using Java, XML, and FTP technologies, will be used to encapsulate data and metadata files together in order to transfer them across multiple server.

The encapsulation can be reversed in order to fit information destination.

Once encapsulated, files can be transferred using FTP technology.

The user can create a Stencil in order to customize XML tags that are going to be used in the encapsulation.

Applying software engineering techniques to the design of the solution.Generating the code that implements the solution.Identifying potential test cases to perform functional verification test at the end of the coding.Integrate the solution to the WALSAIP architecture.

Implementation Effort5Develop a framework for sensor signal acquisition and server storage of raw data.Implement solution to force direct and specific relationship between data and metadata files.Analyze dynamic metadata management.

  <?xml version="1.0" encoding="iso-8859-1" ?> <walsaip> <metadata> <research>  <researchName>Wide Area Large Scale Automated Information Processing</researchName>   <intitution>University of Puerto Rico at Mayaguez</intitution>   <department>Department of Electrical and Computer Engineering</department>   <address>PO Box 9000, Mayagüez, P.R. 00681-9000</address>   <phone>787-832-4040</phone>   <contact>Domingo Rodriguez</contact>   <email>[email protected]</email>   </research> <sensingInfo>  <InitialTimeStamp>2006-07-05 22:23:00.14</InitialTimeStamp>   <endingTimeStamp>2006-07-06 22:23:00.14</endingTimeStamp>   <nodeID>0</nodeID>   <samplingRate>138</samplingRate>   <dataType>humidity</dataType>   </sensingInfo>  </metadata>  <data> 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 </data>   </walsaip>

65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535 65535

65535 65535 65535

MetadataData

2006-07-05 22:23:00.142006-07-06 22:23:00.140138humidity

Figure 5. Data and Metadata Encapsulation Process

Note:

Static Local Information

Input

Server Application

Output

Inpu

t

1KD 1KM

KD~

KM ~

Internet

KD KM

KKK MMM ~1

KKK DDD ~1

-

--

Internet

Figure 5. Information Flow Using Shannon’s Theory

The solution contemplates dynamic metadata management.

Data and metadata can be enhanced with user observations.

User can edit the obtained data by adding comments and parameters.