recurrent auto-encoder model for sensor signal analysis
TRANSCRIPT
![Page 1: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/1.jpg)
Recurrent Auto-encoder Model for Sensor Signal Analysis
TIMOTHY WONG
SENIOR DATA SCIENTIST, CENTRICA PLC
![Page 2: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/2.jpg)
About Us
We supply energy and services to over 27 million customer accounts
Supported by around 12,000 engineers and technicians
Our areas of focus are Energy Supply
& Services, Connected Home, Distributed Energy & Power, Energy Marketing & Trading
![Page 3: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/3.jpg)
Gas TerminalEast Irish Sea
•Morecambe (Barrow) gas terminal
•Produces roughly 6% of GB gas supply
•Onshore and offshore operations
•Comprised of many subsystemsApprox. 6% of
GB gas supply
Slugcatcher
CO2
Removal Train
Condensate
Stabilisation
Dewpoint
Control
Product
Gas
Compressor
Methanol
Still
Nitrogen
Rejection
Unit
Central
Processing
Platform Drilling
Platform 1
Accommodation
Platform
![Page 4: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/4.jpg)
Compression Sub-system
• Two centrifugal compressors driven on a single shaft
• Powered by aeroderivative gas turbine• Increases and regulate gas pressure at a
certain level
Impeller Suction scrubber
RB-211 Industrial Compressor
![Page 5: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/5.jpg)
Compressor
![Page 6: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/6.jpg)
ProblemLarge-scale industrial processes contains many sensors
◦ Sensor records continuous stream of real-valued measurements(e.g. temperature, pressure, rotary speed… etc.)
Can we determine the underlying operating states of the process?
![Page 7: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/7.jpg)
Data PreprocessingReal-valued measurements recorded by sensor are just time series data (Unbounded & unlabelled)
◦ Unevenly-spaced time series (inconsistent interval)
◦ Can be convert into regularly-spaced time series through downsampling.
◦ 𝑃 sensors can form a 𝑃-dimensional multivariate time series:
ℝ𝑡𝑃: 𝑡 ∈ [1, 𝑇]
![Page 8: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/8.jpg)
Recurrent Auto-encoderRNN Encoder-decoder model (‘Seq2seq’, 2014)
◦ Can be converted into recurrent auto-encoder by aligning input/output time steps
◦ Performs partial reconstruction on the decoder side (Decoder dims smaller than Encoder dims)
◦ Deep structure learns abstract temporal patternso (Train parameters by gradient descent. Choice of optimiser is important!)
RNN encoder
RNN decoder
Sensor measurements as output
Feature representation
Sensor measurements as input
![Page 9: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/9.jpg)
Long-short term memory (LSTM, 1997)
Forget gate:𝑓𝑡 = 𝜎(𝑊𝑓 ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑓)
Input gate:𝑖𝑡 = 𝜎 𝑊𝑖 ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑖
ሚ𝐶𝑡 = 𝑡𝑎𝑛ℎ(𝑊𝑐 ℎ𝑡−1, 𝑥𝑡 + 𝑏𝐶)
Update recurrent state:𝐶𝑡 = 𝑓𝑡 × 𝐶𝑡−1 + 𝑖𝑡 × ሚ𝐶𝑡
Output gate:𝑂𝑡 = 𝜎 𝑊𝑖 ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑜
ℎ𝑡 = 𝑂𝑡 × 𝑡𝑎𝑛ℎ𝐶𝑡
![Page 10: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/10.jpg)
RegularisationPrevents network overfittingDropout wrapper (2014)o Masks neuron inputs
o Non-recurrent connections only
L1 / L2 regularisation
![Page 11: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/11.jpg)
SamplingSamples of length 𝑇 can be generated from any dataset with total size 𝑇′ ≥ 𝑇
◦ Generate sequences with fixed length 𝑇
◦ Samples can be generated recursively by shifting time step by fixed number of unit
◦ 𝑇′ − 𝑇 samples can be generated
![Page 12: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/12.jpg)
Sequence ReconstructionSpecimens were selected randomly for qualitative appreciation
𝑲 output dimensions
𝑻 sequence length
Reconstructed sequences
Original sequences
![Page 13: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/13.jpg)
Context vectorPairwise Pearson correlation
◦ Diagonal shows strong correlation among successive context vectors
◦ i.e. Operating state drifting gradually
![Page 14: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/14.jpg)
Context Vectors (Dimensional Reduction)
Dimensionality reduction (PCA) to 2D◦ Additional clustering algorithms can be applied (e.g. 𝐾-means)
◦ Decision boundary calculated using Support Vector Machine (+RBF)
◦ Can be applied to on-line setting for operational state recognition
◦ e.g. IF cluster1 cluster2 THEN TRIGGER ALERT
Neighbourhoods reflect similar operating states
![Page 15: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/15.jpg)
Context Vectors (Dimensional Reduction)
Alternative example◦ Drifting context vector (i.e. the operating state)
HP/LP discharge pressure only
![Page 16: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/16.jpg)
SummaryUnsupervised clustering method for multidimensional time series
◦ Neural network-based time series feature selection
◦ Works with any real-valued temporal measurement (e.g. temperature, pressure… etc.)
◦ Identifies underlying operating stateso Drifts around same neighbourhood No change
o Drift across boundary changed@article{anonymous,
title={Recurrent Auto-Encoder Model for Multidimensional Time Series Representation},author={Anonymous},journal={International Conference on Learning Representations},year={2018},url={https://openreview.net/forum?id=r1cLblgCZ}
}
The technical method described in this presentation is the subject of British patent application GB1717651.2.
![Page 17: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/17.jpg)
Tools Used
![Page 19: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/19.jpg)
Neural Nets
![Page 20: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/20.jpg)
Recurrent Neural Nets•Powerful tool for handling temporal data
•Learns past information in a given sequence
ℎ𝑡 = 𝑓(ℎ𝑡−1, 𝑥𝑡)
•Back-propagation Through Time (BPTT)◦ RNNs can be unrolled into deep forward-feeding network (FNN)
◦ Yet prone to vanishing gradient problem
Unroll
![Page 21: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/21.jpg)
Non-linear Activation Functions
Commonly-used activation functions
![Page 22: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/22.jpg)
SGD Optimiser
Minibatch SGD
Batch SGD
![Page 23: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/23.jpg)
SGD with Momentum
![Page 24: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/24.jpg)
Commonly-used OptimisersAdagrad
RMSProp
Adam
![Page 25: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/25.jpg)
Effects of Learning Rate
![Page 26: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/26.jpg)
Sequence Reversal
![Page 27: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/27.jpg)
Effects of Sequence Length
![Page 28: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/28.jpg)
Effects of Various Optimisers
![Page 29: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/29.jpg)
Effects of Network Topology
![Page 30: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/30.jpg)
Effects of Dropout
![Page 31: Recurrent Auto-encoder Model for Sensor Signal Analysis](https://reader031.vdocuments.us/reader031/viewer/2022012502/617b5f42d7f78f07982398d6/html5/thumbnails/31.jpg)
Effects of Minibatch size