convolutional lstm networks for subcellular localization of proteins
DESCRIPTION
Protein sorting in eukaryotes Various compartments have different functions and different sets of proteins. Nobel Prize to Günter Blobel in 1999.TRANSCRIPT
![Page 1: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/1.jpg)
1
Convolutional LSTM Networks for SubcellularLocalization of Proteins
Søren Kaae Sønderby, Casper Kaae Sønderby, Henrik Nielsen*, and Ole Winther
*Center for Biological Sequence AnalysisDepartment of Systems Biology
Technical University of Denmark
![Page 2: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/2.jpg)
2
Protein sorting in eukaryotes
![Page 3: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/3.jpg)
3
Feed-forward Neural Networks
Problems for sequence analysis:• No builtin concept of
sequence• No natural way of
handling sequences of varying length
• No mechanism for handling long range correlations (beyond input window size)
![Page 4: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/4.jpg)
4
LSTM networks
xt: input at time tht-1: previous outputi : input gate, f : forget gate, o: output gate, g: input modulation gate, c: memory cell.
An LSTM (Long Short Term Memory) cell
The blue arrow head refers to ct−1.
LSTM networks• are easier to train than
other types of recurrent neural networks
• can process very long time lags of unknown size between important events
• are used in speech recognition, handwriting recognition, and machine translation
![Page 5: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/5.jpg)
5
“Unrolled” LSTM network
Each square represents a layer of LSTM cells at a particular time (1, 2, ... t).
The target y is presented at the final timestep.
![Page 6: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/6.jpg)
6
Regular LSTM networks
Bidirectional: one target per position
Double unidirectional: one target per sequence
![Page 7: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/7.jpg)
7
Attention LSTM networks
Bidirectional, but with one target per sequence.
Align weights determine where in the sequence the network directs its attention.
![Page 8: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/8.jpg)
8
Convolutional Neural Networks
A convolutional layer in a neural network consists of small neuron collections which look at small portions of the input image, called receptive fields.
Often used in image processing, where they can handle translation invariance.
First layer convolutional filters learned in an image processing network, note that many filters are edge detectors or color detectors
![Page 9: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/9.jpg)
9
Our basic model
Conv.
LSTM
Conv.
LSTM
Conv.
LSTM
FFN
……t t+1 T
Target prediction at t=T
Softmax
xt xt+1 xT
Note that conv. weights are shared across sequence steps for the convolutional filters
1D convolution(variable width)
Y K P WAxtxt-1xt-2 xt+1 xt+2
Conv. weights
![Page 10: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/10.jpg)
10
Conv.
LSTM
Conv.
LSTM
Conv.
LSTM……t T
xt xt+1 xT
Encoder
……ht ht+1 hT
Vectors containing the activations in each LSTM unit at each time step
Attention
Attention
Attention
Att. Weighting over sequence positions𝛼t 𝛼t+1 𝛼T
Decoder
t+1
Weighted hidden average
Softmax
Target prediction
FFN
Our model, with attention
![Page 11: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/11.jpg)
11
Our model, specifications
– Input encoding: Sparse, BLOSUM80, HSDM and profile (R1×80)
– Conv. filter sizes: 1, 3, 5, 9, 15, 21 (10 of each)– LSTM layer: 1×200 units– Fully connected FFN layer: 1×200 units– Attention model: Wa (R200×400), va (R1×200)
![Page 12: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/12.jpg)
12
MultiLoc architectureMultiLoc is an SVM-based based predictor using only sequence as input
![Page 13: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/13.jpg)
13
MultiLoc2 architecture
MultiLoc2 corresponds to MultiLoc + PhyloLoc + GOLoc.
Thus, its input is not only sequence, but also metadata derived from homology searches.
![Page 14: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/14.jpg)
14
SherLoc2 architecture
SherLoc2 corresponds to MultiLoc2 + EpiLoc
EpiLoc = a prediction system based on features derived from PubMed abstracts found through homology searches
![Page 15: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/15.jpg)
15
Results: performance
![Page 16: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/16.jpg)
16
Learned Convolutional Filters
![Page 17: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/17.jpg)
17
Learned Attention Weights𝛼1 . . . . . . . . . 𝛼t . . . . . . . . . . 𝛼T
![Page 18: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/18.jpg)
18
t-SNE plot of LSTM representation
![Page 19: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/19.jpg)
19
Contributions1. We show that LSTM networks combined with convolutions
are efficient for predicting subcellular localization of proteins from sequence.
2. We show that convolutional filters can be used for amino acid sequence analysis and introduce a visualization technique.
3. We investigate an attention mechanism that lets us visualize where the LSTM network focuses.
4. We show that the LSTM network effectively extracts a fixed length representation of variable length proteins.
![Page 20: Convolutional LSTM Networks for Subcellular Localization of Proteins](https://reader036.vdocuments.us/reader036/viewer/2022062504/5a4d1b3a7f8b9ab05999e492/html5/thumbnails/20.jpg)
20
AcknowledgmentsThanks to:• Søren & Casper Kaae Sønderby,
for doing the actual implementation and training
• Ole Wintherfor supervising Søren & Casper
• Søren Brunakfor introducing me to the world of neural networks
• The organizersfor accepting our paper
• Youfor listening!