eco-friend: data-drive environment protection eco-friend: data …€¦ · protection platform with...

www.scse.ntu.edu.sg Project Objective Overall System Architecture Data Engineering and Dataset Update Django + NGINX Server Performance Testing The objective of this project is to implement an environment protection platform with React Native. Datasets from publicly available sources are used and users would be able to add their own data entries, collectively forming a more comprehensive database. Users will be able to create and join teams and events and share their activities and pictures on social medias in the aim of creating an interactive and engaging experience, making environment protection an interesting option for friend gathering. Server optimizations were implemented with NGINX load balancer and an automated data pipeline for data processing and regular update of datasets. The raw format of datasets from different data sources includes comma separated values (CSV), JSON and KML/KMZ formats. In order to achieve the fastest processing speed of our data pipeline, all formats are converted to csv formats. Front-end User Interface References Q. F. Shao, T. C. Yang, and W. Hou, “The Design of High Available Single Sign-On Server of Nginx-Based,” Applied Mechanics and Materials, vol. 241-244, pp. 2411–2416, 2012. “January 2018 Web Server Survey,” January 2018 Web Server Survey | Netcraft, 19-Jan-2018. [Online]. Available: https://news.netcraft.com/archives/2018/01/19/january-2018- web-server-survey.html. [Accessed: 05-Oct-2018]. [1] [2] Raw Format JSON Populate MongoDB In order to keep datasets updated and our application relevant, datasets stored in central MongoDB database are updated on a pre- defined basis. Update process is streamlined as only new entries are updated or outdated entries deleted, with majority of data unchanged. Initial Data Extraction Regular Query for Changes in data Sources Insertion/Deletion in MongoDB User-created Trashpoints User-created Events Locust Interface Technology Suite Max Concurrent User Average Response Time Django + NGINX 50 89 ms Django 680 99 ms Server Performance Improvement Result Eco-Friend: Data-Drive Environment Protection with React Native Implementation Student: Xiong Hengjie Supervisor: Dr Smitha K G, Dr Sharad SINHA

Upload: others

Post on 20-Jun-2020

1 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Eco-Friend: Data-Drive Environment Protection Eco-Friend: Data …€¦ · protection platform with React Native. Datasets from publicly available sources are used and users would

www.scse.ntu.edu.sgwww.ntu.edu.sg

Project Objective

Overall System Architecture

Data Engineering and Dataset Update

Django + NGINX Server Performance Testing

The objective of this project is to implement an environmentprotection platform with React Native. Datasets from publiclyavailable sources are used and users would be able to add theirown data entries, collectively forming a more comprehensivedatabase. Users will be able to create and join teams and eventsand share their activities and pictures on social medias in the aimof creating an interactive and engaging experience, makingenvironment protection an interesting option for friend gathering.Server optimizations were implemented with NGINX load balancerand an automated data pipeline for data processing and regularupdate of datasets.

The raw format of datasets from different data sources includescomma separated values (CSV), JSON and KML/KMZ formats. Inorder to achieve the fastest processing speed of our data pipeline,all formats are converted to csv formats.

Front-end User Interface

Student: Xiong Hengjie Supervisor: Dr. Smitha K G, Dr. Sharad SINHA

ReferencesQ. F. Shao, T. C. Yang, and W. Hou, “The Design of High AvailableSingle Sign-On Server of Nginx-Based,” Applied Mechanics andMaterials, vol. 241-244, pp. 2411–2416, 2012.

“January 2018 Web Server Survey,” January 2018 Web ServerSurvey | Netcraft, 19-Jan-2018. [Online]. Available:https://news.netcraft.com/archives/2018/01/19/january-2018-web-server-survey.html. [Accessed: 05-Oct-2018].

[1]

[2]

Raw Format JSON Populate MongoDB

In order to keep datasets updated and our application relevant,datasets stored in central MongoDB database are updated on a pre-defined basis. Update process is streamlined as only new entriesare updated or outdated entries deleted, with majority of dataunchanged.

Initial Data Extraction

Regular Query for Changes in data

Sources

Insertion/Deletion in MongoDB

User-created Trashpoints

User-created Events

Locust Interface

Technology Suite Max Concurrent User

Average Response Time

Django + NGINX 50 89 msDjango 680 99 ms

Server Performance Improvement Result

Eco-Friend: Data-Drive Environment Protection with React Native Implementation