a design of distributed storage system over http for collecting sensor data

1

A Design of Distributed Storage System

over HTTP for Collecting Sensor Data

Sayed Ahmad NaweedSupervisor: Dr. Yokoyama Teruaki

4th June 2016

2

Environmental Monitoring

Transportation

Industry

Weather

Military

Medical

Intelligent Transportation

System

Background

3

Application of Sensor Technology

Environmental Monitoring

Transportation

Industry

Weather

Military

Medical

4

Collecting Sensor Data

5

Collecting Sensor Data

6

Collecting Sensor Data

7

Collecting Sensor Data

Data Collection

8

Afghanistan IT Infrastructure

Afghanistan National Data Center (ANDC)

e-Government

e-Commerce

m-Government

Cloud Computing

Sources: http://andc.gov.af/en/page/1098

9

Possible Image of Sensor Network in Afghanistan

Sensors

ANDC Location

App

App

ANDC

10

Sensors

DC Location

Sensors

DC Location

App

App

ANDC

Challenges of Data Collection

① Storing sensor data to a far distant server

Long Distant

11

Sensors

DC Location

App

App

ANDC

Challenges of Data Collection

② Single Point of Failure

1212

Sensors

DC Location

App

App

ANDC

Challenges of Data Collection

③ Traffic Congestion

13

Our Proposed Design

Copying Data to the Core Storage

14

15

Nature of Sensor Data

SensorData

Key

Time

Value

1

2

3

Usually sensor data is stored in time series format.

(Key, Time, Value)

16

Data Accessing Mechanism

1

17

Data Accessing Mechanism

2

1

2

1

18

Data Accessing Mechanism

2

1

2

1

Data Verification Respond

3

19

Data Accessing Mechanism

2

1

2

1

Data Verification Respond

3

Send Data4

20

Advantage of the Proposed System

1

Data Availability ① Link disconnection

between Core and Edge Node

21

Advantage of the Proposed System

1

Data Availability ① Link disconnection

between Core and Edge Node

② Core Storage failure

2

22

Advantage of the Proposed System

1

Data Availability ① Link disconnection

between Core and Edge Node

② Core Storage failure

2

23

Our Contributions

Transferring sensor data to the far-distant Core Node

1

Core Storage

Edge Storage

SensorNetwork

24

Our Contributions

Developers design applications by accessing sensor data

2

Core Storage

Edge Storage

SensorNetwork

25

Originality

Available Systems

Distributed Database System • Mostly in one location

inside a SAN• Requires Consistency

Our System

Distributed Storage System• Distributed

geographically in far-distance places

• No consistency because of sensor data

26

Limitation

• The CAP theorem: – Consistency– Availability – Partitioning

Trade-off

Availability

Consistency

27

Challenge

How do clients find or switch to a nearest edge storage?

28

Experiment

InternetCore Storage

JapanAfghanistan

cURL

1

29

Experiment

Internet

Edge Storage

Core Storage

JapanAfghanistan

cURL

cURL

1

2

30

Result

Internet

Edge Storage

Core Storage

JapanAfghanistan

cURL

cURL

8:00 9:00 10:00 11:00 12:000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

4.3

2.5

3.5

4.5

2.22

1.21

2

0.5

Data from core Storage

Data from Edge Storage

Monday 23rd May 2016

31

Future Work

32

Future Work

P2P Network

33

Thank you“Things should be made as simple as possible, but no simpler.”

Albert Einstein