compressed sensing techniques for sensor data using unsupervised learning
Post on 06-Jul-2015
547 Views
Preview:
DESCRIPTION
TRANSCRIPT
Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data using Unsupervised Learning using Unsupervised Learning using Unsupervised Learning using Unsupervised Learning
SONG CUISONG CUISONG CUISONG CUI
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Outline
• Concept of compressed sensing
• Basic theories with interpretations
• Case studies:
Medical imaging (Considered as a sensor network problem)
Wearable electroencephalography (EEG)
Recommendation system
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Concept of compressed sensing
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Sensor (limited computational capability, transmission power et al.)
Learning algorithms
Data compression
CPU or GPU (excellent computational capability)
Data analytics
• Feature extraction
• Prediction
• Decision making
Data acquisition
Compression techniques
• Traditional lossy compression techniques: JPEG, wavelet et al.
• Traditional compression techniques are not efficient for sparse data.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Sparse data Non-sparse data
Why JPEG is ineffective for sparse data
• Discrete cosine transform converts the image (2-D matrix) from spatial domain to frequency domain in JPEG.
• Insignificant coefficients in frequency domain are discarded in JPEG.• Sparse data have comparable coefficients in all frequencies.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Keep the first 5 coefficients in
frequency domain and
recover the signal
Data compression
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Data compression
• Data compression is through underdetermined linear system.• It is a dimension reduction process in machine learning.
• Compression matrix C is pre-determined where the compression ratio is: m/n.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
y Cx= , Where n<m
Comparison with PCA
• The compression matrix C in compressed sensing is pre-determined with some restrictions.
• PCA is more computational demanding:
Normalizing means and variances in training samples .
Finding the eigenvalues and eigenvectors for .
Picking up the first n principle eigenvectors to form C.
• Data compression is efficient and requires limited or no computational power which favors applications (e. g. mobile sensors) that have limited power, data storage, transmission, and computational capabilities.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
1 2 3 4, , , ..., mx x x x x
T
1
( ( )) /m
i i
i
x x m=
∑
Data reconstruction
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Data reconstruction
Can we fully recover the original signal x from its compressed version y?
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
y Cx= , Where n<m
1 2y x x= +The answer is no in general:
Data reconstruction
• However, the signal can be fully recovered if:
The sparsity of the original signal x is s (the maximum number of non-zero entry is s).
C must satisfy restricted isometry property (RIP) which means any s columns in matrix C are independent.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Sparsity is the key!
Data reconstruction method
y Cx=
• Ill imposed inverse Problem: find the x from y:
• L1 norm minimization is used in practice:
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
0min || || . t .y Cxx s =ɶ ɶ
If RIP holds and x is sparse
2 1|| || || ||y Cx xλ− +ɶ ɶMinimizing
E. Candès and T. Tao, IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203-4215 (2005).
1 2min || || . t . || y Cx ||x s δ− ≤ɶ ɶ
Why L1 regularization?
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
L1 norm minimization
L2 norm minimization
Original sparse signal
Numerical examples are from Prof. W.
K. Ma’s lecture notes
(http://www.ee.cuhk.edu.hk/~wkma).
Analogy to help
understand L1 and L2
regularizations:
Personal income tax
rate
2|| || || || (p 1or 2)py Cx xλ− + =ɶ ɶ
Parallel computing for large-scale datasets
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
1 2 1 1
2 2 1 2
3 2 1 3
|| || || ||
|| || || ||
|| || || ||
......
y Cx x x
y Cx x x
y Cx x x
λ
λ
λ
− + ⇒
− + ⇒
− + ⇒
ɶ ɶ ɶ
ɶ ɶ ɶ
ɶ ɶ ɶ
y Cx= 1 1 2 2 3 3( , ), ( , ), ( , ),...x y x y x yData compression:
Data recovery:
1 2 1 1|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ
1 CPU Multi-thread computing in GPU or CPU clusters
2 2 1 2|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ
3 2 1 3|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ
…. ….
Distributed methods are available for single L1 regularization problem.
Case study 1: Medical imaging
• Positron emission tomography (PET) is capable of measuring positron-emitting radionuclides.
• It is a medical diagnostic instrument for oncology, neuroimaging, and cardiology applications.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Background
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Compressed sensing multiplexing circuits
• Data is sparse in spatial domain.
• Data compression is implemented on PCB circuit boards with pre-amplifiers, resistors and capacitors.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
P. D. Olcott et al., IEEE NSS-MIC Conference Record
p. 3224 (2011).
General applications to wireless sensor network
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
The figure is from J. Haupt et al., IEEE Signal Processing Mag.
pp. 92 Mar. 2008.
Case study 2: Wearable electroencephalography (EEG)
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Figures are from: AJ. Casson et al., IEEE Eng. Med. Biol. Mag 29:44–56 (2010)
Trend of wearable EEG
• Long-term monitoring capability is needed and large-scale data can be generated.
• Data are processed with machine learning algorithms in the remote end with strong computational capabilities and large database.
• Wearable data transmission enables device miniaturization and body area network applications.
• Applications include sleep disorders and augmented cognition.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Challenges in wearable EEG
• Electrode Technology
• Battery power consumption
Data acquisition
Data transmission
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
One possible solution: Compressed sensing!
Compressed sensing for scalp EEG
The data is not sparse in time domain.
The data has sparse representations in terms of other basis functions.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
A. M. Abdulghani et al., Med. Biol. Eng. Comput. 50:1137–1145 (2012).
Case study 3: Recommendation system
Item 1 Item 2 Item 3 Item 4
User 1 5 ? 3
User 2 2
User 3 4 3 2
User 4 5 4 ?
• Matrix factorization methods
• Baseline methods
…………
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
User – item rating matrix: MThink about how you build up
a recommendation system!
T
ui i ur u q p= +
ui i ur u b b= + +
Interpretation of matrix completion
• We assume that there are correlations among some users rating on the same items (dependency in rows).
• We assume there are correlations among ratings on some items from the same user.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Item 1 Item 2 Item 3 Item 4
User 1 5 ? 3
User 2 2
User 3 4 3 2
User 4 5 4 ?
User – item rating matrix: M
We hypothesis that rank (M) is small!
Problem formulation
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
iu iu iurank(X),s. t .X M (M )= ∈Ω
Item 1 Item 2 Item 3 Item 4
User 1 5 ? 3
User 2 2
User 3 4 3 2
User 4 5 4 ?
User – item rating matrix: MFind a matrix X which fills
up the unknown user-
item ratings and satisfy:
Minimize
E. Candès, et al., Foundations of Computational Mathematics,
vol. 9, pp. 717 (2009).
NP hard
problem
0rank(X) || || ,X U *V= Σ = Σwhere
* iu iu iu|| X || ,s. t .X M (M )= ∈ΩMinimize
*
1
|| X || ,r
i i
i
σ σ=
=∑where is the singular value
Connection with compressed sensing
• A more robust and computational efficient approach:
• The method has been tested on Netflix dataset.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
2
iu iu
1
(X M )r
i
i u i
λ σ=
− +∑∑ ∑ 2 1|| || || ||y Cx xλ− +
Minimize
Minimize
Previous results:
N. Srebro and R. Salakhutdinov., Advances in Neural Information Processing Systems, vol. 23, pp. 2056
(2010).
*|| X M || || ||F Xλ− +
Interpretation: M is a compressed version
of X!
Challenges
• Does the hypothesis min rank (X) always match with the truth?
• How to deal with non-uniform sampling (e. g. Some user have much more ratings than others)?
Weighted regularization
• How to deal with cold start problems in the model?
• How to incorporate additional information such as sex, time drifting, and geography location in the model?
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Personal opinions!
Summary
• Compressed sensing is a new data compression and recovery method.
• It is effective for sparse data.Sparse in time domain.
Sparse in frequency domain.
Sparse in other representations.
•It is useful for mobile sensors which has large-scale data transmission, limited battery power, computational capabilities and requires device miniaturization.
• It has seen applications in machine learning such as recommendation systems.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
Extra materials and links
• Stanford course website: http://statweb.stanford.edu/~candes/stats330/index.shtml
• A very resourceful website: Nult Blanche’s blog: http://nuit-blanche.blogspot.com/p/teaching-compressed-sensing.html covers updates from theories and applications such as MRI and machine learning.
• You can also add me on Linkedin: http://www.linkedin.com/profile/view?id=71703589 or contact andycui1@hotmail.com if you want to have further discussions.
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED
top related