Download - Introduction to Recurrent neural networks (RNN), Long short-term memory (LSTM) Wenjie Pei
Introduction to Recurrent neural networks (RNN), Long short-
term memory (LSTM)
Wenjie Pei
Artificial Neural Networks
• Feedforward neural networks– ANNs without cycle connections between nodes
• (Feedback) Recurrent neural networks– ANNs with cycle connections between nodes
Feedforward Neural Networks
• Multilayer perceptron (MLP)
Universal function approximation theory:Sufficient nonlinear hidden units approximate any continuous mapping function
Drawback:Output depends only on the current inputNo temporal information dependencies
Recurrent Neural Networks
Advantage: Memory of previous inputsIncorporate contextual information
Feedback from hidden unit activation of last time step to current time step
Universal approximation theory:An RNN with sufficient hidden unitsAny measurable sequence-to-sequence mappingor dynamic system
Recurrent Neural Networks
• Bidirectional RNNs
Recurrent Neural Networks
• Vanishing gradient problem
Sensitivity decay exponentially over the time
Long Short-Term Memory (LSTM)Input gate [0, 1]: How much information from input could go into the cell
Forget gate [0, 1]: How much information from last time step could enter the cell
Output gate [0, 1]: How much information to output
Long Short-Term Memory
• Advantage: long-period time memory
Applications
• Applications to sequence labeling problems:– Handwritten character recognition– Speech recognition– Protein secondary structure prediction– …
Want to know more about the latest papers:Waiting for my next coffee talk