![Page 1: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/1.jpg)
Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data using Unsupervised Learning using Unsupervised Learning using Unsupervised Learning using Unsupervised Learning
SONG CUISONG CUISONG CUISONG CUI
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 2: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/2.jpg)
Outline
• Concept of compressed sensing
• Basic theories with interpretations
• Case studies:
Medical imaging (Considered as a sensor network problem)
Wearable electroencephalography (EEG)
Recommendation system
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 3: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/3.jpg)
Concept of compressed sensing
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Sensor (limited computational capability, transmission power et al.)
Learning algorithms
Data compression
CPU or GPU (excellent computational capability)
Data analytics
• Feature extraction
• Prediction
• Decision making
Data acquisition
![Page 4: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/4.jpg)
Compression techniques
• Traditional lossy compression techniques: JPEG, wavelet et al.
• Traditional compression techniques are not efficient for sparse data.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Sparse data Non-sparse data
![Page 5: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/5.jpg)
Why JPEG is ineffective for sparse data
• Discrete cosine transform converts the image (2-D matrix) from spatial domain to frequency domain in JPEG.
• Insignificant coefficients in frequency domain are discarded in JPEG.• Sparse data have comparable coefficients in all frequencies.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Keep the first 5 coefficients in
frequency domain and
recover the signal
![Page 6: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/6.jpg)
Data compression
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 7: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/7.jpg)
Data compression
• Data compression is through underdetermined linear system.• It is a dimension reduction process in machine learning.
• Compression matrix C is pre-determined where the compression ratio is: m/n.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
y Cx= , Where n<m
![Page 8: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/8.jpg)
Comparison with PCA
• The compression matrix C in compressed sensing is pre-determined with some restrictions.
• PCA is more computational demanding:
Normalizing means and variances in training samples .
Finding the eigenvalues and eigenvectors for .
Picking up the first n principle eigenvectors to form C.
• Data compression is efficient and requires limited or no computational power which favors applications (e. g. mobile sensors) that have limited power, data storage, transmission, and computational capabilities.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
1 2 3 4, , , ..., mx x x x x
T
1
( ( )) /m
i i
i
x x m=
∑
![Page 9: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/9.jpg)
Data reconstruction
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 10: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/10.jpg)
Data reconstruction
Can we fully recover the original signal x from its compressed version y?
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
y Cx= , Where n<m
1 2y x x= +The answer is no in general:
![Page 11: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/11.jpg)
Data reconstruction
• However, the signal can be fully recovered if:
The sparsity of the original signal x is s (the maximum number of non-zero entry is s).
C must satisfy restricted isometry property (RIP) which means any s columns in matrix C are independent.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Sparsity is the key!
![Page 12: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/12.jpg)
Data reconstruction method
y Cx=
• Ill imposed inverse Problem: find the x from y:
• L1 norm minimization is used in practice:
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
0min || || . t .y Cxx s =ɶ ɶ
If RIP holds and x is sparse
2 1|| || || ||y Cx xλ− +ɶ ɶMinimizing
E. Candès and T. Tao, IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203-4215 (2005).
1 2min || || . t . || y Cx ||x s δ− ≤ɶ ɶ
![Page 13: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/13.jpg)
Why L1 regularization?
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
L1 norm minimization
L2 norm minimization
Original sparse signal
Numerical examples are from Prof. W.
K. Ma’s lecture notes
(http://www.ee.cuhk.edu.hk/~wkma).
Analogy to help
understand L1 and L2
regularizations:
Personal income tax
rate
2|| || || || (p 1or 2)py Cx xλ− + =ɶ ɶ
![Page 14: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/14.jpg)
Parallel computing for large-scale datasets
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
1 2 1 1
2 2 1 2
3 2 1 3
|| || || ||
|| || || ||
|| || || ||
......
y Cx x x
y Cx x x
y Cx x x
λ
λ
λ
− + ⇒
− + ⇒
− + ⇒
ɶ ɶ ɶ
ɶ ɶ ɶ
ɶ ɶ ɶ
y Cx= 1 1 2 2 3 3( , ), ( , ), ( , ),...x y x y x yData compression:
Data recovery:
1 2 1 1|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ
1 CPU Multi-thread computing in GPU or CPU clusters
2 2 1 2|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ
3 2 1 3|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ
…. ….
Distributed methods are available for single L1 regularization problem.
![Page 15: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/15.jpg)
Case study 1: Medical imaging
• Positron emission tomography (PET) is capable of measuring positron-emitting radionuclides.
• It is a medical diagnostic instrument for oncology, neuroimaging, and cardiology applications.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 16: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/16.jpg)
Background
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 17: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/17.jpg)
Compressed sensing multiplexing circuits
• Data is sparse in spatial domain.
• Data compression is implemented on PCB circuit boards with pre-amplifiers, resistors and capacitors.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
P. D. Olcott et al., IEEE NSS-MIC Conference Record
p. 3224 (2011).
![Page 18: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/18.jpg)
General applications to wireless sensor network
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
The figure is from J. Haupt et al., IEEE Signal Processing Mag.
pp. 92 Mar. 2008.
![Page 19: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/19.jpg)
Case study 2: Wearable electroencephalography (EEG)
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Figures are from: AJ. Casson et al., IEEE Eng. Med. Biol. Mag 29:44–56 (2010)
![Page 20: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/20.jpg)
Trend of wearable EEG
• Long-term monitoring capability is needed and large-scale data can be generated.
• Data are processed with machine learning algorithms in the remote end with strong computational capabilities and large database.
• Wearable data transmission enables device miniaturization and body area network applications.
• Applications include sleep disorders and augmented cognition.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 21: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/21.jpg)
Challenges in wearable EEG
• Electrode Technology
• Battery power consumption
Data acquisition
Data transmission
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
One possible solution: Compressed sensing!
![Page 22: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/22.jpg)
Compressed sensing for scalp EEG
The data is not sparse in time domain.
The data has sparse representations in terms of other basis functions.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
A. M. Abdulghani et al., Med. Biol. Eng. Comput. 50:1137–1145 (2012).
![Page 23: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/23.jpg)
Case study 3: Recommendation system
Item 1 Item 2 Item 3 Item 4
User 1 5 ? 3
User 2 2
User 3 4 3 2
User 4 5 4 ?
• Matrix factorization methods
• Baseline methods
…………
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
User – item rating matrix: MThink about how you build up
a recommendation system!
T
ui i ur u q p= +
ui i ur u b b= + +
![Page 24: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/24.jpg)
Interpretation of matrix completion
• We assume that there are correlations among some users rating on the same items (dependency in rows).
• We assume there are correlations among ratings on some items from the same user.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Item 1 Item 2 Item 3 Item 4
User 1 5 ? 3
User 2 2
User 3 4 3 2
User 4 5 4 ?
User – item rating matrix: M
We hypothesis that rank (M) is small!
![Page 25: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/25.jpg)
Problem formulation
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
iu iu iurank(X),s. t .X M (M )= ∈Ω
Item 1 Item 2 Item 3 Item 4
User 1 5 ? 3
User 2 2
User 3 4 3 2
User 4 5 4 ?
User – item rating matrix: MFind a matrix X which fills
up the unknown user-
item ratings and satisfy:
Minimize
E. Candès, et al., Foundations of Computational Mathematics,
vol. 9, pp. 717 (2009).
NP hard
problem
0rank(X) || || ,X U *V= Σ = Σwhere
* iu iu iu|| X || ,s. t .X M (M )= ∈ΩMinimize
*
1
|| X || ,r
i i
i
σ σ=
=∑where is the singular value
![Page 26: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/26.jpg)
Connection with compressed sensing
• A more robust and computational efficient approach:
• The method has been tested on Netflix dataset.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
2
iu iu
1
(X M )r
i
i u i
λ σ=
− +∑∑ ∑ 2 1|| || || ||y Cx xλ− +
Minimize
Minimize
Previous results:
N. Srebro and R. Salakhutdinov., Advances in Neural Information Processing Systems, vol. 23, pp. 2056
(2010).
*|| X M || || ||F Xλ− +
Interpretation: M is a compressed version
of X!
![Page 27: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/27.jpg)
Challenges
• Does the hypothesis min rank (X) always match with the truth?
• How to deal with non-uniform sampling (e. g. Some user have much more ratings than others)?
Weighted regularization
• How to deal with cold start problems in the model?
• How to incorporate additional information such as sex, time drifting, and geography location in the model?
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Personal opinions!
![Page 28: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/28.jpg)
Summary
• Compressed sensing is a new data compression and recovery method.
• It is effective for sparse data.Sparse in time domain.
Sparse in frequency domain.
Sparse in other representations.
•It is useful for mobile sensors which has large-scale data transmission, limited battery power, computational capabilities and requires device miniaturization.
• It has seen applications in machine learning such as recommendation systems.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 29: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/29.jpg)
Extra materials and links
• Stanford course website: http://statweb.stanford.edu/~candes/stats330/index.shtml
• A very resourceful website: Nult Blanche’s blog: http://nuit-blanche.blogspot.com/p/teaching-compressed-sensing.html covers updates from theories and applications such as MRI and machine learning.
• You can also add me on Linkedin: http://www.linkedin.com/profile/view?id=71703589 or contact [email protected] if you want to have further discussions.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
![Page 30: Compressed sensing techniques for sensor data using unsupervised learning](https://reader033.vdocuments.us/reader033/viewer/2022052602/55999f441a28abe5758b4633/html5/thumbnails/30.jpg)
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED