Introduction to Artificial IntelligenceNeural Networks - Deep Learning for

NLP

Janyl JumadinovaNovember 21, 2016

Neural Networks

2/20

Neural Networks

3/20

Neural NetworksNeural computing requires a number of neurons, to be connectedtogether into a neural network.

Neurons are arranged in layers.

4/20

Activation Functions

I The activation function is generally non-linear.

I Linear functions are limited because the output is simplyproportional to the input.

5/20

Activation Functions

6/20

Network structures

Feed-forward networks:

I Single-layer perceptrons

I Multi-layer perceptrons

7/20

Feed-forward example

8/20

Single-layer Perceptrons

Output units all operate separately – no shared weights.

Adjusting weights moves the location, orientation, and steepness ofcliff. 9/20

Multi-layer Perceptrons

I Layers are usually fully connected.I Numbers of hidden units typically chosen by hand.

10/20

A neural network for learning word vector

I Idea: A word and its context is a posiGve training sample

I A random word in that same context gives a negative trainingsample:

11/20

A neural network for learning word vector

12/20

A neural network for learning word vector

These are the word features we want to learn .13/20

A neural network for learning word vector

14/20

Deep Learning

I Most current machine learning works well because ofhuman-designed representations and input features .

I Machine learning becomes just optimizing weights to best makea final prediction.

I Deep learning algorithms attempt to learn multiple levels ofrepresentation of increasing complexity/abstraction.

15/20

Deep Learning

I Most current machine learning works well because ofhuman-designed representations and input features .

I Machine learning becomes just optimizing weights to best makea final prediction.

I Deep learning algorithms attempt to learn multiple levels ofrepresentation of increasing complexity/abstraction.

15/20

A Deep Architecture

16/20

The Need for Distributed RepresentationsCurrent NLP systems are incredibly fragile because of their atomicsymbol representations

17/20

Handling the recursivity of human language

18/20

Recursive Deep Learning: Building on Word

Vector Space Models

19/20

How should we map phrases into a vector

space?

20/20