environmental data management and analytics

PowerPoint Presentation

Environmental Data Analytics

Dr Prasad Modak, Tausif Farooqui

1

Structure of Presentation

2

EMCs Experiences


3

EMC

Experiences on Environmental Data analytics

Dr Prasad Modak had setup the first data base management system in India for environmental data for CPCB in 1986 Which included Air Data managementWater data management, Consent managementsCess calculationsWater Quality data analysis for Godavari Monthly data for 7 years

Indian RegulatorsInternational Financing institutionManufacturing IndustriesInfrastructure and Transport IndustryOur Strengths

4

EMC


Indian RegulatorsInternational Financing institutionManufacturing IndustriesInfrastructure and Transport IndustryOur Strengths5

EMC


Industries are responsible to collect data for mandatory submission to regulator, by law.Industries also use these data sets for disclosuresAnd in sustainability reportsIndian RegulatorsInternational Financing institutionManufacturing IndustriesInfrastructure and Transport IndustryOur Strengths6

EMC


Egyptian Pollution Abatement Programme-phase 2Dr Prasad Modak had setup an online environmental monitoring system which was to serve regulator and financer


7

EMC

Experiences on Environmental Data analytics Analysis of real-time water quality monitoring data of River Ganga 2015Data generated by sensors installed at 10 stations.Geo-Database for Punjab water supply department 2016Analysis and Dashboard for National Ground Water Management Improvement program- India 2016Indian RegulatorsInternational Financing institutionManufacturing IndustriesInfrastructure and Transport IndustryOur Strengths

8

8

EMC


Analysis of air quality in a air shed with 2 power plants - 2016Data from Ambient monitoring, stack measurements, automatic real-time monitoring, and metrological real time monitoring were analyzed Air quality model was built for spatial prediction of pollutant concentrationIndian RegulatorsInternational Financing institutionManufacturing IndustriesInfrastructure and Transport IndustryOur Strengths

9

9

EMC


EMC had setup an EMIS for Rourkela steel plant in 2004


EMC


Corporate Sustainability Report for Glenmark Pharmaceuticals -2015


EMC


Web based GIS for management for the Ahmedabad-Mehsana Toll Road.Data of Maintenance and toll Cause-effect analysis for accidents Showcased by ESRI an one of the first application of ARC on the web


EMC


Designing the data structureDesigning the MIS, EMISOnline/offline applicationsEnvironmental Data analysisEnvironmental modelingGIS applicationsCommunity applications crowd sourcing



14

Characteristics of Environmental Data


15


MultivariateBIG DataIrregular

FuzzyPrimary Data and Meta Data16


BIG DATA

Summing up all the characteristics of environmental dataEssentially a BIG DATA management and processing system is neededTo generate meaningful and actionable reports from the data


Fuzzy17

Link to image

http://www.centrodeinnovacionbbva.com/sites/default/files/bigdata_ejemplos_cibbva.jpg17


Various data sets have different frequency of measurementsAmbient Air Quality parameters are monitored twice a week while stack and metrological data is monitored in real time (15 min period). Other data sets like heath, agriculture are recorded and published monthly or seasonally.

It creates a complicated situation to merge and consolidate all data sets.MultivariateBIG DataIrregular

Fuzzy18


Data from different types of instrumentationMultivariateBIG DataIrregular

Fuzzy19

http://www.sviva.gov.il/PhotoAlbum/Air%20Quality/3air-mana.jpg

http://3.imimg.com/data3/HK/AI/MY-1830445/250-500x500.jpg

19


Large volumes of dataUncertainties are very highFrequent and large Data gapsAccuracy of measurement is questionable at some instances

It makes the data not ready for consumption of decision makers


Fuzzy20

To derive value from these data sets, advanced application of DATA analytics is required

21

212121

Role Based reports

Characteristics of Environmental Data22

22


23

Dashboard


24

Dashboard

Visual representation of the DataAll the information at one placeKey IndicatorsA single screen for insights and analyticsObjective oriented or goal basedAdvanced statistical/mathematical models What is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau25

DashboardWhat is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau

26

https://www.thebookdesigner.com/wp-content/uploads/2015/04/bigstock-Car-dashboard-background-75463333.jpg

26

DashboardDo you record data or measurements ? YesWhere do you use the recorded data ?For decision makingFor complianceInformation dissemination mass communication

How do you use the recorded data ?Print ?Excel, pivot tables ? Or Visualize ?

For BIG data which method would you prefer? What is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau27

27

Dashboard

BIG DATA AnalyticsTo understand the dataTo manage the dataTo process the captured data in a hierarchical mannerTo visually communicate dataTo generate Role based reports instantlyTo make data driven business or operations decisionsTo create alerts

28What is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau

28

Dashboard

http://www.forbes.com/What is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau29

29


30

30


31

31

DashboardWhat is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau32

More about comparison

https://bitipsblog.wordpress.com/2016/10/30/test/https://comparisons.financesonline.com/sas-business-intelligence-vs-tableau-software32

DashboardWhat is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau33

More about comparison

https://bitipsblog.wordpress.com/2016/10/30/test/https://comparisons.financesonline.com/sas-business-intelligence-vs-tableau-software33

Dashboard Tableau is a business intelligence (BI) tool

Tableau connects easily to nearly any data source

Tableau allows transforming data into visually appealing, interactive visualizations

Compatibility across Multiple Platforms - desktop tool, web browser, iPad or mobile phone.

What is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau34

DashboardTableau's Data Analytic and Visualization Features Interactive visualizationsSimultaneous Connections with multiple data sourcesAllows to create custom calculationsCharts, Graphs, Heat maps etc.Time series functionsMap visualizations (point, line, polygon)

Access controlMultiple simultaneous dashboards

What is a Dashboard?Why Dashboard?Sample DashboardsToolsTableau

35


36

Dashboard for Air Quality Management


37

What does Real Time Environmental Data Analytics Involve?


Allows Quick and Dynamic Visualizations

Mines the BIG data to Optimize Sampling Frequencies and get More Value

Uses Real Time Algorithms with Advanced Statistical Tools

Detects Trends, Patterns and allows Forecasting

Helps in establishing association between Emissions and Ambient Quality or understand Source Influences

FeaturesInputParametersillustrationsNew ConceptsThe Future38

Emissions

Ambient Air Quality

Meteorology

Ambient, Stack and Automatic monitoring and metrological data in a single dashboard for effective interpretation, reporting and taking action

What are the important factors for air quality management?

Dashboard for Air Quality Management FeaturesInputParametersillustrationsNew ConceptsThe Future39

Analytical features for Ambient Air


Missing Values treatmentOutliers Basic Statistics (Mean, Deviations, Percentiles, Type of Distribution)Box Whisker PlotsHistograms and Frequency Diagrams Compliance with standards with timestamps Detecting simultaneous ViolationsLocation Importance Index Correlations Inter parameter and Inter stationTrend Analysis (magnitude, direction and statistical significance)Air Quality Index


Time Date of observation/ measurementObserved/Measured values of Concentration of Particulate Matter (PM10, PM2.5)Concentration of Gasses ( CO2, NOx )Location of the station Meta data like up time, calibration date, battery level, sensor make, sensor accuracy etc.,. Parameters for Ambient Air


Analytical features for Stack Emission

Dashboard for Air Quality Management Missing Values Outliers Basic Statistics (Mean, Deviations, Percentiles, Type of Distribution)Box Whisker PlotsHistograms and Frequency Diagrams Compliance with standards with timestamps Detecting simultaneous ViolationsStack Importance Index Predictive Emission Modelling


Parameters for Stack Emission

Dashboard for Air Quality Management Time Data of observation/ measurementObserved/Measured values of Concentration of SPM, other pollutantsTemperatureGenerationLocation of the stack Meta data like up time, calibration date, battery level, sensor make, sensor accuracy etc.,.


Analytical features for Metrological data

Dashboard for Air Quality Management Wind rose, Pollution rose Persistent wind roseNon-Parametric Wind Regression (NWR) for Source IdentificationStack-station Influence analysisFeaturesInputParametersillustrationsNew ConceptsThe Future44

Parameters for Meteorological data

Dashboard for Air Quality Management Time Data of observation/ measurementObserved/Measured values of TemperatureWind SpeedWind DirectionRelative HumiditySolar RadiationFeaturesInputParametersillustrationsNew ConceptsThe Future45

Missing Value

Dashboard for Air Quality Management an observation not recorded

Treatment of Missing ValueReplaced by Mean of Window or the selected time spanReplaced by Mean of NeighborsFeaturesInputParametersillustrationsNew ConceptsThe Future46

Demo Dashboard

Visualize parameters NOxSOxPM10PM2.5

Can be extended to include other parameters

Demo is prepared for AIR, a similar dashboard can be developed for any other Environmental Data

Dashboard for Air Quality ManagementFeaturesInputParametersillustrationsNew ConceptsThe Future47

Options in dropdown boxesSelect Visualization Type Maps, Statistics, Charts

MapChartStats

Dashboard for Air Quality ManagementFeaturesInputParametersillustrationsNew ConceptsThe Future48


User can only download the PDF and Image of the screen.

User cannot however download the raw data behind it.


Station 1

Station 2

Parameter 1Parameter 2AveragingPeriod

User options


Selected stations are located on the map,

Map is sourced from ISRO- Bhuvan

Click to open satellite view of the location in Google maps

Map Visualization


Green band permitted values in designated water quality class

Black lines higher and lowerStatistically acceptable permitted values, beyond this value is an outlier

Blue line Chart of selected parameter

Red dots Both of the selected parameters are violating applicable standards simultaneously

StationsParameter 1

Parameter 2

-Charts Visualization


Basic Statistics Number of RecordsMax, Min, Mean, MedianSD, CoV, Number of OutliersPercentage violations, and Longest Contiguous violation for both the selected parameters

StationsParameter 1

Parameter 2

Percentage simultaneous violations by both parameters against relevant standard. Longest Contiguous and Simultaneous violation -Crosstab Visualization


An outlier is an observation point that is distant from other observations.User option to select method of treating outliersUser option: Show/Hide outlierOutlier is removed and treated as missing value

Outlier


Basic StatisticsBox Whisker Plots

Dashboard for Air Quality Management Annual Maximum Annual Minimum Annual Mean CPCB standardStandard DeviationFeaturesInputParametersillustrationsNew ConceptsThe Future55

Compliance with standards with timestamps


Dates of violationsFeaturesInputParametersillustrationsNew ConceptsThe Future56

Radial ChartDistribution of Wind speeds Vs Direction

Wind Rose


Inter-station correlations


Trend Analysis (magnitude, direction and statistical significance)



Radial ChartDistribution of Pollutant Concentration Vs Direction

Pollution Rose


Compliance Histogram


Detecting simultaneous Violations



Percent exceedance (how many times is the standard for a parameter violated in relation to the length of the data chain)Extent of contiguous exceedance (how long have been the violation over standard on a continuous basis in relation to the length of the data chain)Magnitude of exceedance (how high are the violations when there is an exceedance over standard, expressed in sum of squares of deviations)

Location Importance Index (LII)


Location Importance Index (LII)

Dashboard for Air Quality Management Locations could be a receptor, ambient station or a stackFeaturesInputParametersillustrationsNew ConceptsThe Future64

Distribution of Consistency in wind direction Vs Direction in the number of hours in a Radial ChartData Quality check for automatic metereology stationsPersistent wind rose


Stack-station Influence analysis


Influence of emissions from a particular stack on a particular ambient stationInstances of influence are plotted as time-series


Stack-station Influence analysis


Air Quality measurements are taken at Stack and Ambient

Estimates influence of a stack emission on a specific ambient monitoring station

Longest contiguous duration of a receptor under influence of stack


Non-Parametric Wind Regression (NWR) for Source Identification


NWR has been applied for better understanding of emission sources of influence since early 2000

Technique of NWR is used to identify presence and locations of major emission influence on the observed pollutant concentrations

NWR requires high frequency (e.g. one hour) concentration and meteorological data at the location where emission influences are to be assessedFeaturesInputParametersillustrationsNew ConceptsThe Future68

Non-Parametric Wind Regression (NWR) for Source Identification



69

Source Diagnostics or Environmental Forensics


Data science + Environmental scienceApplication of Sophisticated modelling tools, AERMOD for air quality Receptor ModellingSource identification using Met and Pollution Concentration DataAnalyzing of special events Technique of Intervention analyses Predictive emissions modelling system


AERMOD

Air Dispersion Modelling71US EPA based ModelGaussian plume air dispersion modelPredict downwind pollutant concentrations based on source emissions, meteorological field, and site parameters (land use, terrain features etc.)Model can predict accurately for 50km

FeaturesInputParametersillustrationsNew ConceptsThe Future

Source apportionment

Air Dispersion Modelling72

FeaturesInputParametersillustrationsNew ConceptsThe FutureClick on the box to see Percentage contribution of two power plants of ambient PM10 at a sensitive receptor

Modelled percentage contribution of two power plants of ambient PM10 at a sensitive receptor

Period% contribution of A% contribution of BWinter36.14.7Summer14.313.9Post monsoon 45.37.9Annual32.28.5

73

Integration of Models and Dashboard

Integrated Approach74Air Emission and Air Quality Dashboard integrated with the Dispersion models

Model could be fired from the dashboard itself and required data on emissions, air quality and meteorology will be picked up automatically from data available.

Model will keep re-calibrating itself with actual real time data and this will increase the accuracy of the model outputs.

Models will also help to validate the data collected, thereby improving the quality of monitored data.

FeaturesInputParametersillustrationsNew ConceptsThe Future

PEMS

Predictive EmissionsMonitoring System75FeaturesInputParametersillustrationsNew ConceptsThe Future

PEMS

Predictive EmissionsMonitoring System76Developed on Artificial Neural Network (ANN) conceptsThe model automatically predict emissions data in real time while exploiting the inherent correlation between process variables (flow, temperature and pressure) and emission properties (NOx, SO2, CO, CO2)PEMS can replace CEMS and hence save on costs

FeaturesInputParametersillustrationsNew ConceptsThe FutureClick on the box to see Comparison between CEMS & PEMS

CEMSPEMSCostLarge capital & operational expenditureLess expensive (50% to 75%) than CEMS.MaintenancePeriodic maintenance and calibration of the sensor and analyser is requiredNo additional instrument maintenance (other thannormal process sensors) is requiredAccuracyCEMS are the standard to which PEMS are measuredPEMS are initially calibrated using CEMS. The CEMS are removed when the PEMS becomes operational. PEMS are less accurate than CEMS when the latter is well maintained and calibratedUsesCEMS alarms when emissions are approaching or exceeding an environmental standardPEMS can alarm, but also predict future outcomes and optimize processes in real-time. PEMS therefore leads to higher production efficiency and hence lower cost of production.

77Comparison of CEMS and PEMS

Thank you

Please visit http://dev-data-portal.pantheonsite.io/

Dr Prasad Modak [email protected] Farooqui [email protected]

78


79