dataops toolchain for continuous control...

#PIWorld ©2018 OSIsoft, LLC

DataOps Toolchain for Continuous Control Monitoring

Mahyar SEPEHR

1


Who is apoIO?

2

We work with our clients to optimize their Digital Journey

•  Cloud Architecture •  Internet Of Things •  Industrial Big Data:

PI System


A Typical Client and her Use-Cases

3

Laura is a P&U Project Manager and Business Analyst

Water Network

issues

Water leak detection

Energy consumption optimization

Smart Meters Average Consumption

Water Network Quality Control

Behaviour of the consumers, etc.


Sensors from the Water Network send by GSM values in a non chronological way : Invalidates all rules based on events

Quantity optimisation of substances used on a supply chain : Many timeranges contain exceptions because of Data Quality

4

Business Challenges on Data Quality


5

Data has imperfections : Welcome to Reality!

03:00 03:10 03:20 03:30 03:40 03:50

Speed

Oil Temp

Oil Temp Speed

Torque 03:00

03:10

03:20

03:30

03:40

03:50

Oil

Tem

p

60

60

60

120

60

Oil

Tem

p

50

50

50

50

45

Spe

ed

10.0

10.2

9.8

10.0

10.1

Spe

ed

9.8

10

10

9.9

9.9

Torq

ue

25

24

25

25

25

Comm Failure

Different start times

Additional Measure

Uneven Spacing

Spike / Out of Range


Analysis of the problem

6

Lack of direct communication : Business & the Team

More direct and frequent communication between actors

Cycle from conception to execution too long

Organize some short iteration to avoid loosing time on failure

Issues with Data quality discovered far after conception & implementation

Integrate the data quality monitoring

Organisation

Process

Toolchain

Wha

t wen

t wro

ng?

How

can

we

fix it

? ?

?

?


DevOps Approach : Continuous Improvement

7 © https://en.wikipedia.org/wiki/DevOps_toolchain


Process : DevOps toolchain

8

Source Code Management

Source Code

Test / Build Continuous Integration

Code Quality Review

Deploy To

Dev. Env.

Deploy To

Staging Env.

Acceptance Testing

Deploy To

Production Env.

Deployment Environment

Configuration Deployment tools


PI System Project life-cycle: Data Driven Processes

9

10101010101010110011010 0101010101010101010010101010101010101010101010

10101010101010110011010 0101010101010101010010101010101010101010101010 DATA


Organization Chart

Laura has a new role: Product Owner

10

Security

System Administrator

Data Architect

Product Owner Tech Lead

Ops

Data Engineer

Data Scientist

End-Users

Enterprise Architect


Agile Management : Rethink the Journey

11

z

DESIG

N

TEST

REVI

EW Hotfix

z z

DESIG

N

TEST

REVIEW

DESIG

N

z

DESIG

N

TEST

REVIEW

TEST

REVIEW

Iteration1 3w

Iteration2 3w

Iteration3 3w

Dem

o Fe

atur

es

Dem

o Fe

atur

es Writes User Stories

Core Features including Data Quality

Launch Launch Launch


11!

1

2!

3!

4!

5!

16!

Simple Excel DataLink

Design with Asset Analytics

Design an AF SDK based Custom Program

Design a PI WEB API Custom Program

STAR : Self Test And Repair by Code

STAR : Self Test And Repair with PI Integrator

Continuous Control Monitoring : MVP Toolchains

Minimum Viable

Product (MVP)

12


MVP : STAR with Machine Learning

13

PULL

CLEANSE Data quality

AUGMENT Data aggregation

SHAPE

Model normalization

TRANSMIT Data compatibility

PUSH

PI Integrator for Business Analytics

2 3 4 5 6 Analyse Transform Visualise

Write in to PI System

Report


14

Risks of DataOps Approach

Count on a clear Data Governance & Support?

Data Exploration managed by Business need

Tasks in iterations depending on other Teams

Does DataOps Team have Testing Background?


15

CCM bringing Business values From Data Quality to Data Exploration

Data Quality Statistics helping optimize Business rules

Threshold Optimization

Data Exploration helping in Prediction

Data Exploration cleaning useless Monitoring Alerts


16

Call for Participation! Let’s contribute to a Common Goal: Answer to some sharp questions about PI System Data Exploration

pi-data-exploration

PI WEB API Get Data

Data Preparation

Analysis & Transform

Test

Store into

PI System

Report

1

2

3

4

5

6


pi-data-exploration Description

Sharp Questions: •  Classification Modeling as Leak Detection (no threshhold) •  Autofill on Missed Data •  Statistics over the data as Missed Data Python Dependencies: •  PI-Web-API-Client-Python : PI Client for Python •  pandas : Data structure manipulation •  numpy : perform calculations over the data

https://github.com/msepehr/pi-data-exploration

17


Key Takeaways Key Insights Lessons learned from our journey

01Blueprint for success Do not let your transformation initiative fail because of a lack of anticipation on the project due to the data

02Data Exploration, Data Discovery more & more vital in Businesses in any sector

03Agility with a clear Roadmap Iteration on core features is completed with a real vision in a long term

04Empowering End-Users Better and faster communication, interactions with DataOps Team

05New Roles to Integrate PO, Data preparation, Data Scientists, Business Intelligence, Data Visualization

06Automate, automate, automate! From Data Exploration to Business Rules, Industrialize as much as you can

18


19

Data Exploration makes us grow up…

Let’s grow up together! Mahyar SEPEHR IT Architect [email protected]


Questions?

Please wait for the microphone

State your name & company

Please rate this session in the mobile app!

Search “OSIsoft” in

your app store

20


21

dataops toolchain for continuous control...

Documents