portable data management cloud for field science

PORTABLE DATA MANAGEMENT CLOUDFOR FIELD SCIENCE

UC San Diego, Calit2Yuma Matsui, Aaron Gidding, Thomas E. Levy, Falko Kuester, Thomas A. DeFanti

IEEE Cloud 2012, 6/24/2012

CONTENTS

Managing Big Data in Archaeology

Heterogeneous data

Need for data management system

Portability of Data Management Cloud

System in the Wild

DATA-DRIVENFIELD SCIENCE

DATA-DRIVENFIELD SCIENCE

I need data management infrastructure... but no fancy datacenter

and broadband network here.

PORTABLE DATA MANAGEMENT CLOUDNeed data management system that runs both on

Campus: powerful computers, high-speed network

Field sites: small computers, limited network

Need data management system that runs both on

Campus: powerful computers, high-speed network

Field sites: small computers, limited network

Cloud provides flexible computer infrastructure

virtualized environment, ease of deployment, scalability

Need data management system that runs both on

Campus: powerful computers, high-speed network

Field sites: small computers, limited network

Cloud provides flexible computer infrastructure

virtualized environment, ease of deployment, scalability

PORTABLE DATA MANAGEMENT CLOUDNeed data management system that runs both on

Campus: powerful computers, high-speed network

Field sites: small computers, limited network

Portable data management infrastructurebetween field sites and campus with cloud!

Cloud provides flexible computer infrastructure

virtualized environment, ease of deployment, scalability

Need data management system that runs both on

Campus: powerful computers, high-speed network

Field sites: small computers, limited network

Cloud provides flexible computer infrastructure

virtualized environment, ease of deployment, scalability

Need data management system that runs both on

Campus: powerful computers, high-speed network

Field sites: small computers, limited network

Cloud provides flexible computer infrastructure

virtualized environment, ease of deployment, scalability

Managing Big Data in Archaeology

Portability of Data Management Cloud

Virtualized environment

Data access

System in the Wild

PORTABILITY IN THE SYSTEM

Goal: streamline data processes over field sites and campus

Data collection

Data management

Data analysis

What is portability?

Portability of whole system environment

Portability of collected data

Data Collection Data Management

Data Analysis and

Visualization

Field Sites Campus DatacenterPortability

PORTABLE SYSTEM WITH CLOUD

IaaS

Fully controllable virtualized environment

Makes whole environment (data and programs) portable

Suitable for our field science needs

PaaS

SaaS

DATA ACCESS

Structured data: artifact/site metadata, artifact inventory data, and total station geo-data

Stored in a database

Accessible with JSON REST API

Raw measurement data: Photos, XRF (X-Ray Fluorescence), FTIR (Fourier Transform Infrared Spectroscopy), and LiDAR

Stored in an object storage

Accessible with S3-compatible REST API

Web-based data management application

All data are accessible with the web application or REST API. This makes data portable and consumable.

Managing Big Data in Archaeology

Between Cloud and Ground

System in the Wild

System components

System workflow

SYSTEM COMPONENTS

register

TotalStation

LIDAR

Artifact Data

copy

NetworkAttachedStorage

Cloud Storage

Photos

Field Sites Campus Datacenter

VisualizationFacility

(CAVE,OptIPortal)

Small Server

WebApp DB

Virtualization

IaaS Cloud

WebApp DB

Virtualization

SYSTEM WORKFLOW

Field sites

Various data are collected with insruments.

Structured data are put into the database through the web application.

Raw file data are temporarily stored in network-attached storage.

Campus

Data and programs from fields are moved to campus cloud infrastructure.

Data analyses and visualizations are executed with the collected data on high-performance computers.

Synchronize environments(VM copy and object storage registration)

register

TotalStation

LIDAR

Artifact Data

copy

NetworkAttachedStorage

Cloud Storage

Photos

Field Sites Campus Datacenter

VisualizationFacility

(CAVE,OptIPortal)

Small Server

WebApp DB

Virtualization

IaaS Cloud

WebApp DB

Virtualization

CONCLUSION AND FUTURE WORK

We developed a portable data management infrastructure for digital archaeology.

It is based on IaaS virtualized hosting environments and equipped with unified data access methods.

We used the system in an excavation in 2011.

Integration of the system with large-scale analysis and visualization is in progress.

Thank you!

contact: yumatsui@ucsd.edu