dataops toolchain for continuous control monitoring · #piworld ©2018 osisoft, llc dataops...
TRANSCRIPT
#PIWorld ©2018 OSIsoft, LLC
DataOps Toolchain for Continuous Control Monitoring
Mahyar SEPEHR
1
#PIWorld ©2018 OSIsoft, LLC
Who is apoIO?
2
We work with our clients to optimize their Digital Journey
• Cloud Architecture
• Internet Of Things
• Industrial Big Data: PI System
#PIWorld ©2018 OSIsoft, LLC
A Typical Client and her Use-Cases
3
Laura is a P&U Project Manager
and Business Analyst
Water
Network
issues
Water leak
detection
Energy
consumption
optimization
Smart Meters
Average
Consumption
Water Network
Quality Control
Behaviour of the
consumers, etc.
#PIWorld ©2018 OSIsoft, LLC
Sensors from the Water Network send by GSM values in a non chronological way : Invalidates all rules based on events
Optimisation of substances use on a supply chain :
Many timeranges contain exceptions because of
Data Quality
4
Business Challenges on Data Quality
#PIWorld ©2018 OSIsoft, LLC
5
Data has imperfections : Welcome to Reality!
03:00 03:10 03:20 03:30 03:40 03:50
Speed
Oil
Temp
Oil
Temp
Speed
Torque
03:00
03:10
03:20
03:30
03:40
03:50
Oil
Te
mp
60
60
60
120
60
Oil
Te
mp
50
50
50
50
45
Sp
ee
d
10.0
10.2
9.8
10.0
10.1
Sp
ee
d
9.8
10
10
9.9
9.9
To
rqu
e
25
24
25
25
25
Comm Failure
Different
start times
Additional
Measure
Uneven Spacing
Spike /
Out of Range
#PIWorld ©2018 OSIsoft, LLC
Analysis of the problem
6
Lack of direct
communication :
Business & the Team
More direct and frequent
communication between
actors
Cycle from
conception to
execution too long
Organize some short
iteration to avoid
loosing time on failure
Issues with Data
quality discovered far
after conception &
implementation
Integrate the data
quality monitoring
Organisation
Process
Toolchain
Wh
at
wen
t w
ron
g?
Ho
w c
an
we f
ix it?
?
?
?
#PIWorld ©2018 OSIsoft, LLC
DevOps Approach : Continuous Improvement
7
© https://en.wikipedia.org/wiki/DevOps_toolchain
#PIWorld ©2018 OSIsoft, LLC
Process : DevOps toolchain
8
Source Code
Management
Source
Code
Test / Build
Continuous
Integration
Code
Quality
Review
Deploy
To
Dev. Env.
Deploy
To
Staging Env.
Acceptance
Testing
Deploy
To
Production
Env.
Deployment Environment
Configuration Deployment tools
#PIWorld ©2018 OSIsoft, LLC
PI System Project life-cycle: Data Driven Processes
9
10101010101010110011010 0101010101010101010010101010101010101010101010
10101010101010110011010 0101010101010101010010101010101010101010101010
DATA
#PIWorld ©2018 OSIsoft, LLC
Organization Chart
Laura has a new role:
Product Owner
10
Security
System
Administrator
Data
Architect
Product
Owner Tech
Lead
Ops
Data
Engineer
Data
Scientist
End-Users
Enterprise
Architect
#PIWorld ©2018 OSIsoft, LLC
Agile Management : Rethink the Journey
11
z Hotfix
z z z Iteration1
3w
Iteration2
3w
Iteration3
3w D
em
o
Fe
atu
res
De
mo
Fe
atu
res
Writes User Stories
Core Features
including Data Quality
Launch Launch Launch
#PIWorld ©2018 OSIsoft, LLC
1 1
1
2
3
4
5
1 6
Simple Excel DataLink
Design with Asset Analytics
Design an AF SDK based Custom Program
Design a PI WEB API Custom Program
STAR : Self Test And Repair by Code
STAR : Self Test And Repair with PI Integrator
Continuous Control Monitoring : MVP Toolchains
Minimum Viable
Product (MVP)
12
#PIWorld ©2018 OSIsoft, LLC
MVP : STAR with Machine Learning
13
PULL
CLEANSE
Data quality
AUGMENT Data
aggregation
SHAPE
Model
normalization
TRANSMIT Data
compatibility
PUSH
PI Integrator for
Business Analytics
2 3 4 5 6
Analyse Transform Visualise Write
in to
PI System
Report
#PIWorld ©2018 OSIsoft, LLC
14
Risks of DataOps Approach
Count on a clear Data Governance & Support?
Data Exploration managed by Business need
Tasks in iterations depending on other Teams
Does DataOps Team have Testing Background?
#PIWorld ©2018 OSIsoft, LLC
15
CCM bringing Business values From Data Quality to Data Exploration
Data Quality Statistics helping optimize Business rules
Threshold Optimization
Data Exploration helping in Prediction
Data Exploration cleaning useless Monitoring Alerts
#PIWorld ©2018 OSIsoft, LLC
16
Call for Participation!
Let’s contribute to
a Common Goal:
Answer to
some sharp
questions about
PI System Data Exploration
pi-data-exploration
PI WEB API
Get Data
Data
Preparation
Analysis &
Transform
Test
Store
into
PI System
Report
1
2
3
4
5
6
#PIWorld ©2018 OSIsoft, LLC
pi-data-exploration Description
Sharp Questions:
• Classification Modeling as Leak Detection (no threshhold)
• Autofill on Missed Data
• Statistics over the data as Missed Data
Python Dependencies:
• PI-Web-API-Client-Python : PI Client for Python
• pandas : Data structure manipulation
• numpy : perform calculations over the data
https://github.com/msepehr/pi-data-exploration
17
#PIWorld ©2018 OSIsoft, LLC
Key Takeaways Key Insights Lessons learned from our journey
01 Blueprint for success Do not let your transformation initiative
fail because of a lack of anticipation on
the project due to the data
02 Data Exploration, Data Discovery more & more vital in Businesses in any
sector
03 Agility with a clear Roadmap Iteration on core features is completed
with a real vision in a long term
04 Empowering End-Users Better and faster communication,
interactions with DataOps Team
05 New Roles to Integrate PO, Data preparation, Data Scientists,
Business Intelligence, Data Visualization
06 Automate, automate, automate! From Data Exploration to Business
Rules, Industrialize as much as you can
18
#PIWorld ©2018 OSIsoft, LLC
19
Data Exploration makes us grow up…
Let’s grow up together!
Mahyar SEPEHR
IT Architect
#PIWorld ©2018 OSIsoft, LLC
Questions?
Please wait for
the microphone
State your
name & company
Please rate this session
in the mobile app!
Search
“OSIsoft” in
your app store
20
#PIWorld ©2018 OSIsoft, LLC
21