stamatios cheirdaris, dmitry nikelshpur, charles tappert, alexander cipully, roberto rodriguez,...

Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and Robert Zucker

Human Visual System Neural Network

The Visual System

• Hubel and Wiesel• 1981 Nobel Prize for work in early 1960s

• Cat’s visual cortex

• cats anesthetized, eyes open with controlling muscles paralyzed to fix the stare in a specific direction

• thin microelectrodes measure activity in individual cells

• cells specifically sensitive to line of light at specific orientation

• Key discovery – line and edge detectors in the visual cortex of mammals

The Study

• Compare Two Neural Networks• One without vertical and horizontal line detectors

• One with vertical and horizontal line detectors

• Objective• Show that the neural network with line detectors

is superior to the one without on the six vertical-horizontal line-segment letters E, F, H, I, L, T

• Also, experiment with the full alphabet• Without line detectors

Uppercase 5x7 Bit-map AlphabetHorizontal-vertical line-segment letters are E, F, H, I, L, T

Neural Network Without Line Detectors

Neural Network SpecificationWithout Line Detectors

Layers

1. Input layer: 20x20 retina of binary units

2. Hidden layer: 50 units (other numbers explored)

3. Output layer: 6 units for letters E, F, H, I, L, T

Weights

• 20,000 (400x50) between input and hidden layer

• 300 (50x6) between hidden and output layer

• Total of 20,300 variable weights, no fixed weights

Neural Network With Line Detectors

Neural Network SpecificationWith Vertical and Horizontal Line Detectors

Layers

1. Input layer: 20x20 retina of binary units

2. 576 simple vertical and horizontal line detectors

3. 48 complex vertical and horizontal line detectors

4. Hidden layer: 50 units (other numbers explored)

5. Output layer: 6 units for letters E, F, H, I, L, T

Weights

• 6336 (576x11) fixed weights from input to simple detectors

• 576 fixed weights from simple and complex detectors

• 2400 (48x50) variable weights from complex detectors to hidden layer

• 300 (50x6) variable weights from hidden to output layer

• Total of 6912 fixed weights and 2700 variable weights

DETECTORS OVERLAP COVERING EACH POSSIBLE RETINAL POSITION FOR A TOTAL OF 288 (18x16) VERTICAL LINE DETECTORS

EACH DETECTOR HAS 5 EXCITATORY AND 6 INHIBITORY INPUTS (11 FIXED WEIGHTS),WITH A THRESHOLD OF 3

Vertical Line Detectors

Horizontal Line Detectors are Similar

Retina Image – Letter “E” in Upper Left Area

Region of possible upper-left corners is shown in green.

Retina Image – Letter “E” in Upper Right Area

Region of possible upper-left corners is shown in green.

Retina Image – Letter “E” in Lower Right Area

Region of possible upper-left corners is shown in green.

Example of Vertical Line Detectoron Line Segment of “E” – Detector Activated

Example of Shifted Vertical Line Detectoron Letter “E” – Detector Not Activated

Example of Shifted Vertical Line Detectoron Letter “E” – Detector Not Activated

24 Vertical Complex Line Detector RegionsAny Simple Line Detector in a RegionActivates the Complex Line Detector

24 Horizontal Complex Line Detector RegionsAny Simple Line Detector in a RegionActivates the Complex Line Detector

Complex Horizontal and Vertical Line Detector Matrix

The Corresponding 48 Complex Horizontal and Vertical Line Detectors

Experiments

Experiment 1o 6 Line-Segment Letters without Line Detectorso 26 Letters without Line Detectors

Experiment 2o 6 Line-Segment Letters with Line Detectors

Experimental Parameter Combinations• Epochs:

50

100

200

400

800

1600

32000 (occasionally)

• Hidden Layer Units:

10

18*

50

100

200 *

300*

500*

*Selected cases

• Noise:

0%

2%

5%

10%

15%

20%

Experiment 1

Simulation View – Peltarion’s Synapse Product

• Weight Layer: Forward Rule: No rule Back Rule: Levenberg-

Marquardt Propagator: Weight Layer

• Function Layer:

• Function: Tanh Sigmoid Forward Rule: No rule Back Rule: Levenberg-

Marquardt Propagator: Function Layer

Simulation Settings Experiment 2 – Line Detectors

6 Line-Segment Letters: E, F, H, I, L, T

6 Letters: no line detectors

50 Hidden layer units

50 Epochs

0% noise

Exp 1 – 6 Letters, No Line Detectors – 35.42% Accuracy

6 Letters: no line detectors

50 Hidden layer units

1600 Epochs

0% noise

Exp 1 – 6 Letters, No Line Detectors – 36.25% Accuracy

6 Letters: with line detectors

50 Hidden layer units

50 Epochs

0% noise

Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy

6 Letters: with line detectors

50 Hidden layer units

50 Epochs

0% noise

Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy

6 Letters: no line detectors

10 Hidden layer units

1600 Epochs

0% noise

Exp 1 – 6 Letters, No Line Detectors – 27.69% Accuracy

6 Letters: with line detectors

10 Hidden layer units

1600 Epochs

0% noise

Exp 1 – 6 Letters, With Line Detectors – 82.5% Accuracy

6 Letters: with line detectors

10 Hidden layer units

1600 Epochs

0% noise

Exp 2 – 6 Letters, With Line Detectors – 82.5% Accuracy

26 Letters: no line detectors

50 Hidden layer units

1600 Epochs

0% noise

Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy

26 Letters: no line detectors

50 Hidden layer units

1600 Epochs

0% noise

Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy

Percent Noise

Epochs 0% 2% 5% 10% 15% 20%

50 35.42 20.83 24.17 20 20.42 20.83

100 36.25 21.67 25 18.75 20.83 20.42

200 35.42 21.67 23.75 18.75 20.83 20.83

400 36.25 22.50 23.75 20.42 20.42 20.83

800 36.67 22.50 23.75 20.42 20.00 20.83

1600 36.25 18.33 23.75 20.42 20.42 20.83

Exp 1 – 6 Letters, No Line DetectorsEpochs versus Percent Added Noise

Exp 1 – 26 Letters, No Line DetectorsEpochs versus Percent Added Noise

Percent Noise

Epochs 0% 2% 5% 10% 15% 20%

50 23.08 8.17 7.21 7.50 6.25 6.83

100 23.56 9.04 7.69 7.31 6.15 4.62

200 25.00 9.81 7.60 5.58 6.06 4.52

400 27.12 9.81 6.92 5.02 5.87 4.62

800 28.37 8.96 6.92 5.48 5.58 5.19

1600 27.69 10.10 7.31 5.67 7.21 6.15

Exp 2 – 6 Letters, With Line DetectorsEpochs versus Percent Added Noise

Epochs 0% 2% 5% 10% 15% 20%

50 67.50 58.75 46.25 56.67 16.67 42.50

100 72.92 59.17 46.25 56.67 17.08 28.00

200 77.08 59.58 50.00 62.50 48.75 43.75

400 75.83 58.33 50.00 62.92 34.58 28.33

800 78.33 60.42 52.08 59.17 32.92 57.08

1600 83.33 59.17 56.67 60.00 35.00 32.08

3200 85.00 66.67 56.67 57.92 52.50 62.92

Comparison of Line / No-Line Detector Networks6 letters, 50 hidden layer units, 1600 epochs, no noise

Experiment PerformanceFixed

WeightsVariable Weights

Total Weights

No Line Detectors 36.3% 0 20,300 20,300

Line Detectors 83.3% 6,912 2,700 9,612

• Character recognition performance and efficiency of the neural network using Hubel-Wiesel-like line detectors in the early layers is superior to that of a network using adjustable weights directly from the retina

• Recognition performance more than doubled

• Line detector network was much more efficient

• order of magnitude fewer variable weights and half as many total weights

• training time decrease of several orders of magnitude

Main Conclusion

• Increasing the number of hidden layer units does not translate to better accuracy, it actually reduces it.

• Increasing the number of epochs increased the accuracy but not always

• For Experiment 2 (6 letters with line detectors) we can achieve perfect training accuracy and very good validation accuracy

• Training time varied from a few minutes to many hours with Experiment 1 – 26 Letters taking the longest out of all, i.e. for 500 hidden layer units it required up to 9 hours.

• When noise is added to the retina image the accuracy of the system drops significantly, even for Experiment 2 with the line detectors

Additional Conclusions