mining sparse representations: theory, algorithms, and ... · mining sparse representations:...
TRANSCRIPT
![Page 1: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/1.jpg)
Center for Evolutionary Functional Genomics
Mining Sparse Representations: Theory, Algorithms, and Applications
Jun Liu, Shuiwang Ji, and Jieping YeComputer Science and Engineering
The Biodesign Institute Arizona State University
![Page 2: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/2.jpg)
Center for Evolutionary Functional Genomics
What is Sparsity?• Many data mining tasks can be represented using a
vector or a matrix.• “Sparsity” implies many zeros in a vector or a matrix.
![Page 3: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/3.jpg)
Center for Evolutionary Functional Genomics
Sparsity in Data Mining• Regression• Classification• Multi-Task Learning• Collaborative Filtering• Network Construction
![Page 4: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/4.jpg)
Center for Evolutionary Functional Genomics
Regression
• Select a small number of features– Lasso
![Page 5: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/5.jpg)
Center for Evolutionary Functional Genomics
Application: Genetics• Find the locations on the genome that influence a
trait: e.g. cholesterol level
• Model: y = Ax + z (x is very sparse)
Number of columns: about 500,000, number of rows: a few thousands
![Page 6: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/6.jpg)
Center for Evolutionary Functional Genomics
Application: Neuroscience
• Neuroimages for studying Alzheimer’s Disease
![Page 7: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/7.jpg)
Center for Evolutionary Functional Genomics
Multi-Task/Class Learning
• Key Challenge: How to capture the shared information among multiple tasks?– Shared features (group Lasso)– Shared low-dimensional subspace (trace norm)
X1 Y1 X2 Y2 Xk Yk…
wkw2w1 … = W
![Page 8: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/8.jpg)
Center for Evolutionary Functional Genomics
Application: Biological Image
Annotation
• Document expression patterns over 6,000 genes with over 70,000 images
• Annotated with body part keywords
http://www.fruitfly.org/cgi-bin/ex/insitu.pl
![Page 9: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/9.jpg)
Center for Evolutionary Functional Genomics
Collaborative Filtering
• Customers are asked to rank items• Not all customers ranked all items• Predict the missing rankings
? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?
Customers
Items
![Page 10: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/10.jpg)
Center for Evolutionary Functional Genomics
Application: The Netflix Challenge
• About a million users and 25,000 movies• Known ratings are sparsely distributed• Predict unknown ratings
? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?
Users
Movies
![Page 11: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/11.jpg)
Center for Evolutionary Functional Genomics
Network Construction
Equivalent matrix representation
Sparsity: Each node is linked to a small number of neighbors in the network.
![Page 12: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/12.jpg)
Center for Evolutionary Functional Genomics
Application: Brain Network
Brain Connectivity of different regions for Alzheimer's Disease(Sun et al., KDD 2009)
![Page 13: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/13.jpg)
Center for Evolutionary Functional Genomics
A Unified Model for Sparse Learning • Let x be the model parameter to be estimated. A
commonly employed model for estimating x is
min loss(x) + λ penalty(x) (1)
• (1) is equivalent to the following model:
min loss(x)s.t. penalty(x) ≤z (2)
![Page 14: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/14.jpg)
Center for Evolutionary Functional Genomics
Loss Functions• Least squares• Logistic loss• Hinge loss• …
![Page 15: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/15.jpg)
Center for Evolutionary Functional Genomics
Penalty Functions: Nonconvex• Zero norm
penalty(x)=the number of nonzero elements• Advantages
The sparsest solution• Disadvantages:
Not a valid norm, nonconvex, NP-hard
• Extension to the matrix case: rank
![Page 16: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/16.jpg)
Center for Evolutionary Functional Genomics
Penalty Functions: Convex• L1 norm
penalty(x)=||x||1=∑i|xi|• Advantages:
Valid normConvexComputationally tractableTheoretical propertiesMany applicationsVarious Extensions
In this tutorial, we discuss sparse representation based
on L1 and its extensions.
![Page 17: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/17.jpg)
Center for Evolutionary Functional Genomics
Why does L1 Induce Sparsity?Analysis in 1D (comparison with L2)
0.5×(x-v)2 + λ|x| 0.5×(x-v)2 + λx2
Nondifferentiable at 0 Differentiable at 0
If v≥ λ, x=v- λIf v≤ -λ, x=v+λElse, x=0
x=v/(1+2 λ)
![Page 18: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/18.jpg)
Center for Evolutionary Functional Genomics
Why does L1 Induce Sparsity?Understanding from the projection
min loss(x)s.t. ||x||2 ≤1
min 0.5||x-v||2s.t. ||x||2 ≤1
min loss(x)s.t. ||x||1 ≤1
min 0.5||x-v||2s.t. ||x||1 ≤1
Sparse
![Page 19: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/19.jpg)
Center for Evolutionary Functional Genomics
Why does L1 Induce Sparsity?Understanding from constrained optimization
(Bishop, 2006)
min loss(x)s.t. ||x||1 ≤1
min loss(x)s.t. ||x||2 ≤1
![Page 20: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/20.jpg)
Center for Evolutionary Functional Genomics
Sparsity via L1
Lasso, Dantzig selector
Summation of the absolute values
Application: Regression/Classification
![Page 21: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/21.jpg)
Center for Evolutionary Functional Genomics
Sparsity via L1/Lq
Group Lasso, Group Dantzig Application: Multi-Task Learning
![Page 22: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/22.jpg)
Center for Evolutionary Functional Genomics
Sparsity via L1+ L1/Lq
Sparse Group Lasso Application: Multi-Task Learning
![Page 23: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/23.jpg)
Center for Evolutionary Functional Genomics
Sparsity via Fused Lasso
Fused Lasso, Time Varying Network Application: Regression with Ordered Features
![Page 24: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/24.jpg)
Center for Evolutionary Functional Genomics
Sparsity via Trace Norm
Application: Multi-Task Learning, Matrix Completion
![Page 25: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/25.jpg)
Center for Evolutionary Functional Genomics
Sparse Inverse Covariance
Connectivity Study
![Page 26: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/26.jpg)
Center for Evolutionary Functional Genomics
Goal of This Tutorial
• Introduce various sparsity-induced norms– Map sparse models to different applications
• Efficient implementations– Scale sparse learning algorithms to large-size problems
![Page 27: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/27.jpg)
Center for Evolutionary Functional Genomics
Outline• Sparsity via L1
• Sparsity via L1/Lq
• Sparsity via Fused Lasso• Sparse Inverse Covariance Estimation• Sparsity via Trace Norm• Implementations and the SLEP package• Trends in Sparse Learning
![Page 28: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/28.jpg)
Center for Evolutionary Functional Genomics
Compressive Sensing(Donoho, 2004; Candes and Tao, 2008; Candes and Wakin, 2008)
Principle:
P0 P1
![Page 29: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/29.jpg)
Center for Evolutionary Functional Genomics
Basis Pursuit(Chen, Donoho, and Saunders, 1999)
Linear Programming
min <1, t>s.t. y=Φx, -t≤x≤t
Question:When can P1 obtain the same result as P0?
![Page 30: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/30.jpg)
Center for Evolutionary Functional Genomics
Sparse Recovery and RIP
for every K-sparse vector x.
satisfies the K-restricted isometry property with constant if is the smallest constant satisfying
![Page 31: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/31.jpg)
Center for Evolutionary Functional Genomics
Extensions to the Noisy Casenoise
Basis pursuit De-Noising (Chen, Donoho, and Saunders, 1999)
Lasso (Tibshirani, 1996)
Dantzig selector (Candes and Tao, 2007)Regularized counterpart of Lasso
![Page 32: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/32.jpg)
Center for Evolutionary Functional Genomics
Lasso(Tibshirani, 1996)
![Page 33: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/33.jpg)
Center for Evolutionary Functional Genomics
Theory of Lasso(Zhao and Yu, 2006; Wainwright 2009;
Meinshausen and Yu, 2009; Bickel, Ritov, and Tsybakov, 2009)
• Support Recovery
• Sign Recovery
• L1 Error
• L2 Error
sup(x)=sup(x*)?
sign(x)=sign (x*)?
||x-x*||1
||x-x*||2
![Page 34: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/34.jpg)
Center for Evolutionary Functional Genomics
L1 Error(Bickel, Ritov, and Tsybakov, 2009)
![Page 35: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/35.jpg)
Center for Evolutionary Functional Genomics
Dantzig Selector
![Page 36: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/36.jpg)
Center for Evolutionary Functional Genomics
Theory of Dantzig Selector (Candes and Tao, 2007; Bickel, Ritov, and Tsybakov, 2009)
![Page 37: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/37.jpg)
Center for Evolutionary Functional Genomics
Face Recognition(Wright et al. 2009)
![Page 38: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/38.jpg)
Center for Evolutionary Functional Genomics
1-norm SVM(Zhu et al. 2003)
Hinge Loss
1-norm constraint
![Page 39: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/39.jpg)
Center for Evolutionary Functional Genomics
Outline• Sparsity via L1
• Sparsity via L1/Lq
• Sparsity via Fused Lasso• Sparse Inverse Covariance Estimation• Sparsity via Trace Norm• Implementations and the SLEP package• Trends in Sparse Learning
![Page 40: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/40.jpg)
Center for Evolutionary Functional Genomics
From L1 to L1/Lq (q>1)?
L1
L1/LqL1/Lq
Most existing work focus on q=2, ∞
![Page 41: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/41.jpg)
Center for Evolutionary Functional Genomics
Group Lasso(Yuan and Lin, 2006; Meier et al., 2008)
![Page 42: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/42.jpg)
Center for Evolutionary Functional Genomics
Splice Site Detection
![Page 43: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/43.jpg)
Center for Evolutionary Functional Genomics
Multi-task Learning via L1/Lq Regularization
![Page 44: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/44.jpg)
Center for Evolutionary Functional Genomics
Face Recognition(Liu, Yuan, Chen, and Ye, 2009)
![Page 45: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/45.jpg)
Center for Evolutionary Functional Genomics
Writer-specific Character Recognition(Obozinski, Taskar, and Jordan, 2006)
![Page 46: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/46.jpg)
Center for Evolutionary Functional Genomics
Sparse Group Lasso(Peng et al., 2010; Friedman, Hastie, and Tibshirani, 2010; Liu and Ye, 2010)
![Page 47: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/47.jpg)
Center for Evolutionary Functional Genomics
Simulation Study(Liu and Ye, 2010)
![Page 48: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/48.jpg)
Center for Evolutionary Functional Genomics
Simulation Study(Liu and Ye, 2010)
![Page 49: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/49.jpg)
Center for Evolutionary Functional Genomics
Integrative Genomics Study of Breast Cancer(Peng et al., 2010)
![Page 50: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/50.jpg)
Center for Evolutionary Functional Genomics
Integrative Genomics Study of Breast Cancer(Peng et al., 2010)
![Page 51: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/51.jpg)
Center for Evolutionary Functional Genomics
Outline• Sparsity via L1
• Sparsity via L1/Lq
• Sparsity via Fused Lasso• Sparse Inverse Covariance Estimation• Sparsity via Trace Norm• Implementations and the SLEP package• Trends in Sparse Learning
![Page 52: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/52.jpg)
Center for Evolutionary Functional Genomics
Fused Lasso(Tibshirani et al., 2005; Tibshirani and Wang, 2008; Friedman et al., 2007)
Fused LassoL1
![Page 53: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/53.jpg)
Center for Evolutionary Functional Genomics
Illustration of Fused Lasso(Rinaldo, 2009)
Noise
![Page 54: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/54.jpg)
Center for Evolutionary Functional Genomics
Fused Lasso
![Page 55: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/55.jpg)
Center for Evolutionary Functional Genomics
Asymptotic Property(Tibshirani et al., 2005)
![Page 56: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/56.jpg)
Center for Evolutionary Functional Genomics
Prostate Cancer(Tibshirani et al., 2005)
![Page 57: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/57.jpg)
Center for Evolutionary Functional Genomics
Leukaemia Classification Using Microarrays(Tibshirani et al., 2005)
![Page 58: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/58.jpg)
Center for Evolutionary Functional Genomics
Arracy CGH
![Page 59: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/59.jpg)
Center for Evolutionary Functional Genomics
Outline• Sparsity via L1
• Sparsity via L1/Lq
• Sparsity via Fused Lasso• Sparse Inverse Covariance Estimation• Sparsity via Trace Norm• Implementations and the SLEP package• Trends in Sparse Learning
![Page 60: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/60.jpg)
Center for Evolutionary Functional Genomics
Sparse Inverse Covariance EstimationThe pattern of zero entries in the inverse covariance matrix of a
multivariate normal distribution corresponds to conditional
independence restrictions between variables (Meinshausen &
Buhlmann, 2006).
![Page 61: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/61.jpg)
Center for Evolutionary Functional Genomics
The SICE Model
Maximum Likelihood Estimation
When S is invertible, directly maximizing the likelihood gives
X=S-1
![Page 62: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/62.jpg)
Center for Evolutionary Functional Genomics
Example: Senate Voting Records Data (2004-06)
Republican senatorsDemocratic senators
Chafee (R, RI) has only Democrats as his neighbors, an observation that supports media statements made by and about Chafee during those years.
![Page 63: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/63.jpg)
Center for Evolutionary Functional Genomics
The Monotone Property(Huang et al., NIPS 2009)
Intuitively, if two nodes are connected (either directly or
indirectly) at one level of sparseness, they will be
connected at all lower levels of sparseness.
)( 1kC )( 2kC
kX 1 2
21 )()( 21 kk CC
Monotone Property
Let and be the sets of all the connectivity components of with and respectively. If , then .
![Page 64: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/64.jpg)
Center for Evolutionary Functional Genomics
Brain Network for Alzheimer’s Disease(Huang et al., 2009)
Small λLarge λ λ3 λ2 λ1
AD
![Page 65: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/65.jpg)
Center for Evolutionary Functional Genomics
Small λLarge λ
Brain Network for Alzheimer’s Disease(Huang et al., 2009)
MD
![Page 66: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/66.jpg)
Center for Evolutionary Functional Genomics
NC
Brain Network for Alzheimer’s Disease(Huang et al., 2009)
![Page 67: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/67.jpg)
Center for Evolutionary Functional Genomics
Brain Network for Alzheimer’s Disease(Huang et al., 2009)
NCMDAD
Strong Connectivity
![Page 68: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/68.jpg)
Center for Evolutionary Functional Genomics
Brain Network for Alzheimer’s Disease(Huang et al., 2009)
NCMDAD
Mild Connectivity
![Page 69: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/69.jpg)
Center for Evolutionary Functional Genomics
Brain Network for Alzheimer’s Disease(Huang et al., 2009)
NCMDAD
Weak Connectivity
![Page 70: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/70.jpg)
Center for Evolutionary Functional Genomics
Brain Network for Alzheimer’s Disease(Huang et al., 2009)
•Temporal: decreased connectivity in AD, decrease not significant in MCI.• Frontal: increased connectivity in AD (compensation), increase not significant in MCI.
• Parietal, occipital: no significant difference.• Parietal-occipital: increased weak/mild con. in AD.• Frontal-occipital: decreased weak/mild con. in MCI.• Left-right: decreased strong con. in AD, not MCI.
![Page 71: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/71.jpg)
Center for Evolutionary Functional Genomics
Outline• Sparsity via L1
• Sparsity via L1/Lq
• Sparsity via Fused Lasso• Sparse Inverse Covariance Estimation• Sparsity via Trace Norm• Implementations and the SLEP package• Trends in Sparse Learning
![Page 72: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/72.jpg)
Center for Evolutionary Functional Genomics
Collaborative Filtering
• Customers are asked to rank items• Not all customers ranked all items• Predict the missing rankings
? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?
Customers
Items
![Page 73: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/73.jpg)
Center for Evolutionary Functional Genomics
The Netflix Problem
• About a million users and 25,000 movies• Known ratings are sparsely distributed• Predict unknown ratings
? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?
Users
Movies
Preferences of users are determined by a small number of factors low rank
![Page 74: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/74.jpg)
Center for Evolutionary Functional Genomics
Matrix Rank
• The number of independent rows or columns• The singular value decomposition (SVD):
= × ×
}rank
![Page 75: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/75.jpg)
Center for Evolutionary Functional Genomics
The matrix Completion Problem
![Page 76: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/76.jpg)
Center for Evolutionary Functional Genomics
The Alternating Approach
• Optimize over U and V iteratively• Solution is locally optimal
![Page 77: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/77.jpg)
Center for Evolutionary Functional Genomics
Fundamental Questions
• Can we recover a matrix M of size n1 by n2 from m sampled entries, m << n1 n2?
• In general, it is impossible.
• Surprises (Candes & Recht’08):– Can recover matrices of interest from incomplete
sampled entries– Can be done by convex programming
![Page 78: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/78.jpg)
Center for Evolutionary Functional Genomics
Multi-Task/Class Learning
• The multiple tasks/classes are usually related• The matrix W is of low rank
X1 Y1 X2 Y2 Xk Yk…
wkw2w1 … = W
)(*),(1 1
WrankxwylT
i
n
j
j
i
T
i
j
i
i
![Page 79: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/79.jpg)
Center for Evolutionary Functional Genomics
Matrix Classification
x1
[Tomioka & Aihara (2007), ICML]
x2
xn
…
y1
y2
yn
…
)())(,(min1
WrankXWTryln
i
i
T
iW
![Page 80: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/80.jpg)
Center for Evolutionary Functional Genomics
Other Low-Rank Problems
• Image compression• System identification in control theory• Structure-from-motion problem in computer vision• Other settings:
– low-degree statistical model for a random process– a low-order realization of a linear system– A low-order controller for a plant– a low-dimensional embedding of data in Euclidean
space
![Page 81: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/81.jpg)
Center for Evolutionary Functional Genomics
Two Formulations for Rank
Minimization
min f(W) + λ*rank(W)min rank(W)
subject to f(W)=b
Rank minimization is NP-hard
![Page 82: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/82.jpg)
Center for Evolutionary Functional Genomics
Trace Norm (Nuclear Norm)
• trace norm ⇔ 1-norm of the vector of singular values• Trace norm is the convex envelope of the rank function over the unit
ball of spectral norm ⇒ a convex relaxation• In some sense, this is the tightest convex relaxation of the NP-hard
rank minimization problem
![Page 83: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/83.jpg)
Center for Evolutionary Functional Genomics
Two Formulations for Nuclear Norm
min f(W) + λ* ||W|| min ||W||
subject to f(W)=b
Nuclear norm minimization is convex
**
• Consistent estimation can be obtained
• Can be solved by
• Semi-definite programming
• Gradient-based methods
![Page 84: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/84.jpg)
Center for Evolutionary Functional Genomics
Sparsity with Vectors and Matrices
[Recht et al. (2010) SIAM Review]
Parsimony concept Cardinality RankHilbert space norm Euclidean Frobenius
Sparsity inducing norm Trace normConvex optimization Linear programming Semi-definite programming
![Page 85: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/85.jpg)
Center for Evolutionary Functional Genomics
Rank Minimization and CS
Rank minimizationmin rank(W)s.t. f(W) = b
Convex relaxationmin ||W||
s.t. f(W) = b*
W is diagonal, linear constraint
Rank minimizationmin ||w||0
s.t. Aw = b
Convex relaxationmin ||w||1
s.t. Aw = b
![Page 86: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/86.jpg)
Center for Evolutionary Functional Genomics
Theory of Matrix Completion
Candès and Recht (2008)
![Page 87: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/87.jpg)
Center for Evolutionary Functional Genomics
Semi-definite programming (SDP)
))()((21min 21,, 21
ATrATrAAX
bWf
AX
XAT
)(
02
1
*min W
W
bWf )( s.t.s.t.
• SDP is convex, but computationally expensive
• Many recent efficient solvers: • Singular value thresholding (Cai et al, 2008 )
• Fixed point method (Ma et al, 2009)
• Accelerated gradient descent (Toh & Yun, 2009, Ji & Ye, 2009)
![Page 88: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/88.jpg)
Center for Evolutionary Functional Genomics
Outline• Sparsity via L1
• Sparsity via L1/Lq
• Sparsity via Fused Lasso• Sparsity via Trace Norm• Sparse Inverse Covariance Estimation• Implementations and the SLEP package• Trends in Sparse Learning
![Page 89: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/89.jpg)
Center for Evolutionary Functional Genomics
Optimization Algorithm
• Smooth Reformulation – general solver• Coordinate descent• Subgradient descent• Gradient descent• Accelerated gradient descent• Online algorithms• Stochastic algorithms• …
min f(x)= loss(x) + λ×penalty(x)min loss(x)s.t. penalty(x) ≤z
loss(x) is convex and smooth, penalty(x) is convex but nonsmooth
![Page 90: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/90.jpg)
Center for Evolutionary Functional Genomics
Smooth Reformulations: L1
Linearly constrained quadratic programming
![Page 91: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/91.jpg)
Center for Evolutionary Functional Genomics
Smooth Reformulation: L1/L2
Second order cone programming
![Page 92: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/92.jpg)
Center for Evolutionary Functional Genomics
Smooth Reformulation: Fused Lasso
Linearly constrained quadratic programming
![Page 93: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/93.jpg)
Center for Evolutionary Functional Genomics
Summary of Reformulations
Advantages:• Easy to incorporate existing solvers• Fast and high precision for small size problems
Disadvantages:• Does not scale well for large size problems, due to 1) many additional
variables and constraints are introduced; 2) the computation of Hessian is demanding
• Does not utilize well the “structure” of the nonsmooth penalty• Not applicable to all the penalties discussed in this tutorial, say, L1/L3.
![Page 94: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/94.jpg)
Center for Evolutionary Functional Genomics
Coordinate Descent(Tseng, 2002)
![Page 95: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/95.jpg)
Center for Evolutionary Functional Genomics
Coordinate Descent: Example(Tseng, 2002)
![Page 96: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/96.jpg)
Center for Evolutionary Functional Genomics
Coordinate Descent: Convergent?(Tseng, 2002)
• If f(x) is smooth, then the algorithm is guaranteed to converge.• If f(x) is nonsmooth, the algorithm can get stuck.
![Page 97: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/97.jpg)
Center for Evolutionary Functional Genomics
Coordinate Descent: Convergent?(Tseng, 2002)
• If f(x) is smooth, then the algorithm is guaranteed to converge.• If f(x) is nonsmooth, the algorithm can get stuck.• If the nonsmooth part is separable, convergence is guaranteed.
min f(x)= loss(x) + λ×penalty(x) penalty(x)=||x||1
![Page 98: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/98.jpg)
Center for Evolutionary Functional Genomics
Coordinate Descent
• Can xnew be computed efficiently?
min f(x)= loss(x) + λ×penalty(x) penalty(x)=||x||1
loss(x)=0.5×||Ax-y||22
![Page 99: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/99.jpg)
Center for Evolutionary Functional Genomics
CD in Sparse Representation• Lasso (Fu, 1998; Friedman et al., 2007)
• L1/Lq regularized least squares & logistic regression (Yuan and Lin, 2006,Liu et al., 2009; Argyriou et al., 2008; Meier et al., 2008)
• Sparse group Lasso for q=2 (Peng et al., 2010; Friedman et al., 2010)
• Fused Lasso Signal Approximator (Friedman et al., 2007; Hofling, 2010)
• Sparse inverse covariance estimation (Banerjee et al., 2008; Friedman et al., 2007)
![Page 100: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/100.jpg)
Center for Evolutionary Functional Genomics
Summary of CDAdvantages:• Easy for implementation, especially for the least
squares loss• Can be fast, especially the solution is very sparse
Disadvantages:• No convergence rate• Can be hard to derive xnew for general loss• Can get stuck when the penalty is non-separable
![Page 101: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/101.jpg)
Center for Evolutionary Functional Genomics
Subgradient Descent(Nemirovski, 1994; Nesterov, 2004)
Repeat
Until “convergence”
G
Subgradient: one element in the subdifferential set
![Page 102: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/102.jpg)
Center for Evolutionary Functional Genomics
Subgradient Descent: Subgradient(Nemirovski, 1994; Nesterov, 2004)
h(x)=0.5×(x-v)2 + λ|x| g(x)=0.5×(x-v)2 + λx2
g(x) is differentiable for all x h(x) is non-differentiable at x=0
![Page 103: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/103.jpg)
Center for Evolutionary Functional Genomics
Subgradient Descent: Convergent?(Nemirovski, 1994; Nesterov, 2004)
Repeat
Until “convergence”
If f(x) is Lipschitz continuous with constant L(f), the set G is closed convex, and the step size is set as follows
then, we have
![Page 104: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/104.jpg)
Center for Evolutionary Functional Genomics
• L1 constrained optimization (Duchi et al., 2008)
• L1/L∞ constrained optimization (Quattoni et al., 2009)
Advantages: • Easy implementation• Guaranteed global convergence
Disadvantages• It converges slowly• It does not take the structure of the non-smooth term into consideration
SD in Sparse Representation
![Page 105: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/105.jpg)
Center for Evolutionary Functional Genomics
Gradient Descent
Repeat
Until “convergence”
Repeat
Until “convergence”
f(x) is smooth
![Page 106: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/106.jpg)
Center for Evolutionary Functional Genomics
Gradient Descent: The essence of the gradient step
Repeat
Until “convergence”
Repeat
Until “convergence”
1st order Taylor expansion Regularization
![Page 107: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/107.jpg)
Center for Evolutionary Functional Genomics
Gradient Descent: Extension to the composite model (Nesterov, 2007; Beck and Teboulle, 2009)
min f(x)= loss(x) + λ×penalty(x)
1st order Taylor expansion Regularization Nonsmooth part
Repeat
Until “convergence”
Convergence rate
Much better than Subgradient descent
![Page 108: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/108.jpg)
Center for Evolutionary Functional Genomics
Gradient Descent: Extension to the composite model (Nesterov, 2007; Beck and Teboulle, 2009)
Repeat
Until “convergence”
![Page 109: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/109.jpg)
Center for Evolutionary Functional Genomics
Accelerated Gradient Descent: unconstrained version (Nesterov, 2007; Beck and Teboulle, 2009)
The lower complexity bound shows that, the first-order methods can not achieve a better convergence rate than O(1/N2).Can we develop a method that can achieves the optimal convergence rate?
Repeat
Until “convergence”
GD
O(1/N)
Repeat
Until “convergence”
AGD
O(1/N2)
![Page 110: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/110.jpg)
Center for Evolutionary Functional Genomics
Accelerated Gradient Descent: constrained version (Nesterov, 2007; Beck and Teboulle, 2009)
Repeat
Until “convergence”
GD
O(1/N)
Repeat
Until “convergence”
AGD
O(1/N2)
Can the projection be computed efficiently?
![Page 111: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/111.jpg)
Center for Evolutionary Functional Genomics
Accelerated Gradient Descent: composite model (Nesterov, 2007; Beck and Teboulle, 2009)
Repeat
Until “convergence”
GD
O(1/N)
min f(x)= loss(x) + λ×penalty(x)Repeat
Until “convergence”
AGD
O(1/N2)
Can the proximal operator be computed efficiently?
![Page 112: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/112.jpg)
Center for Evolutionary Functional Genomics
Accelerated Gradient Descent in Sparse Representations
• Lasso (Nesterov, 2007; Beck and Teboulle, 2009)
• L1/Lq (Liu, Ji, and Ye, 2009; Liu and Ye, 2010)
• Sparse group Lasso (Liu and Ye, 2010)
• Trace Norm (Ji and Ye, 2009; Pong et al., 2009; Toh and Yun, 2009; Lu et al., 2009)
• Fused Lasso (Liu, Yuan, and Ye, 2010)
Advantages:• Easy for implementation• Optimal convergence rate• Scalable to large sample size problem
![Page 113: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/113.jpg)
Center for Evolutionary Functional Genomics
Accelerated Gradient Descent in Sparse Representations
Key computational cost• Gradient and functional value• The projection (for constrained smooth optimization)• The proximal operator (for composite function)
Advantages:• Easy for implementation• Optimal convergence rate• Scalable to large sample size problem
![Page 114: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/114.jpg)
Center for Evolutionary Functional Genomics
Euclidean projection onto the L1 ball(Duchi et al., 2008; Liu and Ye, 2009)
![Page 115: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/115.jpg)
Center for Evolutionary Functional Genomics
Efficient Computation of the Proximal Operators(Liu and Ye, 2010; Liu and Ye, 2010; Liu, Yuan and Ye, 2010)
min f(x)= loss(x) + λ×penalty(x)
• L1/Lq
• Sparse group Lasso
• Fused Lasso
![Page 116: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/116.jpg)
Center for Evolutionary Functional Genomics
Proximal Operator Associated with L1/Lq
It can be decoupled into the following q-norm regularized Euclidean projection problem:
Optimization problem:
Associated proximal operator:
![Page 117: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/117.jpg)
Center for Evolutionary Functional Genomics
Proximal Operator Associated with L1/Lq
Convert it to two simple zero finding algorithmsMethod:
1. Suitable to any2. The proximal plays a key building block in quite a few
methods such as the accelerated gradient descent, coordinate gradient descent(Tseng, 2008), forward-looking subgradient(Duchi and Singer, 2009), and so on.
Characteristics:
![Page 118: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/118.jpg)
Center for Evolutionary Functional Genomics
Achieving a Zero SolutionFor the q-norm regularized Euclidean projection
We have 1 1 1q q
We restrict our following discussion to
![Page 119: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/119.jpg)
Center for Evolutionary Functional Genomics
Fixed Point Iteration?One way to compute is to apply the following fixed point iteration
Let us consider the two-dimensional case.
Fixed point iteration is not guaranteed to converge.
We start from
![Page 120: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/120.jpg)
Center for Evolutionary Functional Genomics
Fixed Point Iteration?
0 20 40 60 80 1000.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
iteration
x1
![Page 121: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/121.jpg)
Center for Evolutionary Functional Genomics
Solving Two Zero Finding Problems The methodology is also based on the following relationship
Construct three auxiliary functions.
Solve the above equation by two simple one-dimensional zero finding problems (associated with the constructed auxiliary functions).
![Page 122: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/122.jpg)
Center for Evolutionary Functional Genomics
Efficiency(compared with spectral projected gradient)
![Page 123: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/123.jpg)
Center for Evolutionary Functional Genomics
Efficiency(compared with spectral projected gradient)
![Page 124: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/124.jpg)
Center for Evolutionary Functional Genomics
Effect of q in L1/Lq
RAND RANDN
Multivariate linear regression
![Page 125: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/125.jpg)
Center for Evolutionary Functional Genomics
Proximal Operator Associated with L1+L1/Lq(Liu and Ye, 2010)
Optimization problem:
Associated proximal operator:
![Page 126: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/126.jpg)
Center for Evolutionary Functional Genomics
Proximal Operator Associated with L1+L1/L2(Liu and Ye, 2010)
![Page 127: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/127.jpg)
Center for Evolutionary Functional Genomics
Proximal Operator Associated with L1+L1/Linf(Liu and Ye, 2010)
t* is the root of
![Page 128: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/128.jpg)
Center for Evolutionary Functional Genomics
Effective Interval for Sparse Group Lasso(Liu and Ye, 2010)
![Page 129: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/129.jpg)
Center for Evolutionary Functional Genomics
Efficiency (comparison with remMap via coordiante descent)
![Page 130: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/130.jpg)
Center for Evolutionary Functional Genomics
Efficiency (comparison with remMap via coordiante descent)
![Page 131: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/131.jpg)
Center for Evolutionary Functional Genomics
Proximal Operator Associated with Fused Lasso
Optimization problem:
Associated proximal operator:
![Page 132: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/132.jpg)
Center for Evolutionary Functional Genomics
Fused Lasso Signal Approximator(Liu, Yuan, and Ye, 2010)
Let
We have
![Page 133: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/133.jpg)
Center for Evolutionary Functional Genomics
Fused Lasso Signal Approximator(Liu, Yuan, and Ye, 2010)
Method:• Subgradient Finding Algorithm (SFA)---looking for an appropriate and unique subgradient of at the minimizer (motivated by the proof of Theorem 1).
![Page 134: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/134.jpg)
Center for Evolutionary Functional Genomics
Efficiency(Comparison with the CVX solver)
![Page 135: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/135.jpg)
Center for Evolutionary Functional Genomics
Efficiency(Comparison with the CVX solver)
![Page 136: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/136.jpg)
Center for Evolutionary Functional Genomics
Summary of Implementations• The accelerated gradient descent, which is an optimal
first-order method, is favorable for large-scale optimization.
• To apply the accelerated gradient descent, the key is to develop efficient algorithms for solving either the projection or the associated proximal operator.
http://www.public.asu.edu/~jye02/Software/SLEP
![Page 137: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/137.jpg)
Center for Evolutionary Functional Genomics
Outline• Sparsity via L1
• Sparsity via L1/Lq
• Sparsity via Fused Lasso• Sparsity via Trace Norm• Sparse Inverse Covariance Estimation• Implementations and the SLEP package• Trends in Sparse Learning
![Page 138: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/138.jpg)
Center for Evolutionary Functional Genomics
New Sparsity Induced Penalties?min f(x)= loss(x) + λ×penalty(x)
Sparse
Fused LassoSparse inverse covariance
![Page 139: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/139.jpg)
Center for Evolutionary Functional Genomics
Overlapping Groups?min f(x)= loss(x) + λ×penalty(x)
Sparse
Sparse group LassoGroup Lasso
• How to learn groups?• How about the overlapping groups?
![Page 140: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/140.jpg)
Center for Evolutionary Functional Genomics
Efficient Algorithms for Huge-Scale Problems
Algorithms for p>108, n>105?
It costs over 1 Terabyte to store the data.
![Page 141: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/141.jpg)
Center for Evolutionary Functional Genomics
References(Compressive Sensing and Lasso)
![Page 142: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/142.jpg)
Center for Evolutionary Functional Genomics
References(Compressive Sensing and Lasso)
![Page 143: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/143.jpg)
Center for Evolutionary Functional Genomics
References(Compressive Sensing and Lasso)
![Page 144: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/144.jpg)
Center for Evolutionary Functional Genomics
References(Compressive Sensing and Lasso)
![Page 145: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/145.jpg)
Center for Evolutionary Functional Genomics
References(Compressive Sensing and Lasso)
![Page 146: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/146.jpg)
Center for Evolutionary Functional Genomics
References(Compressive Sensing and Lasso)
![Page 147: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/147.jpg)
Center for Evolutionary Functional Genomics
References(Compressive Sensing and Lasso)
![Page 148: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/148.jpg)
Center for Evolutionary Functional Genomics
References(Group Lasso and Sparse Group Lasso)
![Page 149: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/149.jpg)
Center for Evolutionary Functional Genomics
References(Group Lasso and Sparse Group Lasso)
![Page 150: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/150.jpg)
Center for Evolutionary Functional Genomics
References(Group Lasso and Sparse Group Lasso)
![Page 151: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/151.jpg)
Center for Evolutionary Functional Genomics
References(Group Lasso and Sparse Group Lasso)
![Page 152: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/152.jpg)
Center for Evolutionary Functional Genomics
References(Group Lasso and Sparse Group Lasso)
![Page 153: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/153.jpg)
Center for Evolutionary Functional Genomics
References(Group Lasso and Sparse Group Lasso)
![Page 154: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/154.jpg)
Center for Evolutionary Functional Genomics
References(Fused Lasso)
![Page 155: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/155.jpg)
Center for Evolutionary Functional Genomics
References(Fused Lasso)
![Page 156: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/156.jpg)
Center for Evolutionary Functional Genomics
References(Trace Norm)
![Page 157: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/157.jpg)
Center for Evolutionary Functional Genomics
References(Trace Norm)
![Page 158: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/158.jpg)
Center for Evolutionary Functional Genomics
References(Trace Norm)
![Page 159: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/159.jpg)
Center for Evolutionary Functional Genomics
References(Sparse Inverse Covariance)
![Page 160: Mining Sparse Representations: Theory, Algorithms, and ... · Mining Sparse Representations: Theory, Algorithms, and Applications. Jun Liu, Shuiwang Ji, and Jieping Ye. ... • Many](https://reader035.vdocuments.us/reader035/viewer/2022062918/5edb128509ac2c67fa68c39f/html5/thumbnails/160.jpg)
Center for Evolutionary Functional Genomics
References(Sparse Inverse Covariance)