More biological plausible neural network on image recognition

Chengyuan Zhuang

Power of ANN

• Artificial neural networks (ANNs) are computational models inspired by an animal's central nervous systems (in particular the brain)

Universal approximation theorem

• Standard multilayer feedforward networks are capable of approximating any measurable function to any desired degree of accuracy (universal approximators)

• Recurrent architecture with rational valued weights has the full power of a Universal Turing Machine

• There are no theoretical constraints for the success of feedforward networks

• Lack of success is due to inadequate learning, insuffcient number of hidden units or the lack of a deterministic relationship between input and target

Limitation of Deep Learning

• Researchers argue that there is no evidence of such process as back propagation of error or gradient descent in the brain, not so biological plausible

• There are many training parameters to be considered with a DNN, such as the size (number of layers and number of units per layer), the learning rate and initial weights. Sweeping through the parameter space for optimal parameters may not be feasible due to the cost in time and computational resources.

Biological findings

• Each of the 1011 (one hundred billion) neurons has on average 7,000 synaptic connections to other neurons

• Two types synapses: promote firing and inhibit it

Hebbian rule

• When an axon of a cell A is near enough to excite cell B or repeatedly or persistently takes part in firing it, some growth or metabolic change takes place in both cells such that A's efficiency, as one of the cells firing B, is increased (Hebb, D.O. 1949).

• Hebb emphasized that cell A needs to 'take part in firing' cell B, and such causality can only occur if cell A fires just before, not at the same time as, cell B

Long-term potentiation (LTP)

• Long-term potentiation (LTP) is a persistent increase in synaptic strength following high-frequency stimulation of a chemical synapse.

• Produce a long-lasting increase in signal transmission between two neurons

• A few seconds of highfrequency electrical stimulation can enhance synaptic transmission in the rabbit hippocampus for days or even weeks (Timothy Bliss, Mill Hill,1970s)

• As memories are thought to be encoded by modification of synaptic strength, LTP is widely considered one of the major cellular mechanisms that underlies learning and memory

Long-term depression (LTD)

• Long-term depression (LTD) is an activity-dependent reduction in the efficacy of neuronal synapses lasting hours or longer following a long patterned stimulus

• The opposite process to LTP, produce long-lasting decrease in synaptic strength

• Serves to selectively weaken specific synapses

• If allowed continue increasing in strength, synapses would ultimately reach a ceiling level of efficiency, which would inhibit the encoding of new information (Purves D,2008)

Competitive Learning

• A form of unsupervised learning in artificial neural networks, in which nodes compete for the right to respond to a subset of input data

• Inhibitory connections. Only the winning unit is "rewarded" (by having its weights increased)

Genetic algorithm

• A search heuristic that mimics the process of natural evolution, such as inheritance, mutation, selection, and crossover.

• In each generation (iteration), the fitness of every individual is evaluated; the more fit individuals are stochastically selected, evolve toward better solutions.

Self organizing map

• Using unsupervised learning to produce a low-dimensional (typically two-dimensional), discretized representation of the input space of the training samples, called a map.

• Using a neighborhood function to preserve the topological properties of the input space

Adaptive Resonance Theory

• A theory developed by Stephen Grossberg and Gail Carpenter on aspects of how the brain processes information.

• Introduced to overcome the stability-plasticity dilemma. The basic idea is that learning in a parallel and distributed system requires plasticity for the integration of new knowledge, but also stability in order to prevent the forgetting of previous knowledge.

• Too much plasticity will result in previously encoded data being constantly forgotten, whereas too much stability will impede the efficient coding of this data at the level of the synapses.

Our approach

Short-Term Memory Layer

Long-Term Memory Layer

TrainingPattern

Label

TestingPattern

The MNIST hand-written digits database

• 60,000 training images and 10,000 test images

• A subset of a larger set available from NIST

• Digits have been size-normalized and centered in a fixed-size image (Y LeCun, 1998)

• 0~9, equally distributed

• Popular dataset

• Comparing results when Y LeCun published his famous paper in 1998

• Performance up till now

Results

• Due to several design and redesign, we currently only have trained on 10,000 images equally distributed, and also tested on 1,000 images equally distributed

• Accuracy is 93.2%

• We just use a portion of data, we believe with more training data, the performance still has room for improvement

Future Work

• Would incorporate design more like Self-Organizing Map

• Extend this approach to colored and high resolution images as in ImageNet

• Explore more complex, more demanding tasks such as object detection or automatic driving