artificial neural networksdataanalysis.vsb.cz/data/vyuka/navy/01 ann-history.pdfartificial neural...
TRANSCRIPT
NAVY Research Group Department of Computer Science
Faculty of Electrical Engineering and Computer Science VŠB-‐TUO 17. listopadu 15
708 33 Ostrava-‐Poruba Czech Republic
Artificial Neural Networks History and Basic Info
Ivan Zelinka
MBCS CIPT, www.bcs.org/ http://www.springer.com/series/10624
Department of Computer Science
Faculty of Electrical Engineering and Computer Science, VŠB-TUO 17. listopadu 15 , 708 33 Ostrava-Poruba
Czech Republic www.ivanzelinka.eu
navy.cs.vsb.cz 2
Topics
• Our perception • History of ANNs and basic terms • Difference between ANNs and computer • Biological and technical neuron • Transfer functions • ANNs classification • ANN topology • Linear and nonlinear separable problems • Hilbert's 13th problem and the Kolmogorov theorem
navy.cs.vsb.cz 3
Objectives
The objectives of the lesson are: • Explain effect of our perception by our brain • Show historical background of ANNs • Clarify the main difference between ANNs and computer • Describe basic processes in biological and technical neuron • Explain what are transfer functions and its impact on ANNs
classification and linear and nonlinear separable problems • Define ANN topology anf its conjunctions with Hilbert's 13th problem
and the Kolmogorov theorem
navy.cs.vsb.cz 4
Artificial Neural Networks Our Perception
• Is our perception reality or illusion? • Neural networks are derived from the basic patterns - links observed
in the human brain. This provides both advantages and disadvantages of the human brain.
Memory-switch effect What do you see?
navy.cs.vsb.cz 5
Artificial Neural Networks Complexity
• The human brain is very complicated neural system, which consists of biological neural networks, which are in terms of the primitive function at the lowest level.
• Brain and complexity
navy.cs.vsb.cz 6
Artificial Neural Networks Stanislaw Lem the Futurologist
Stanisław Lem, 12 September 1921 – 27 March 2006) was a Polish writer of science fiction, philosophy and satire. His books have been translated into 41 languages and have sold over 45 million copies. From the 1950s to 2000s he published many books, both science fiction and philosophical/futurological. He is best known as the author of the 1961 novel Solaris, which has been made into a feature film three times.
navy.cs.vsb.cz 7
Artificial Neural Networks Does the Ocean Think?
• Solaris (1961) by the Stanislaw Lem • Non-human thinking system • Planetary ocean as a brain
• Fantasy and science fiction??? • Really?
navy.cs.vsb.cz 8
Artificial Neural Networks Does the Ocean Think?
• Our brains has 100 billions interconnected cells
• Microbiologist Yuri Gorby (http://faculty.rpi.edu/node/1179) – Yuri is a pioneer in the emerging field of “electromicrobiology” and has
been heavily cited for his publications on electrically conductive protein filaments called ‘bacterial nanowires’. He established the first Electromicrobiology Group during his 5 years with the J. Craig Venter Laboratory in San Diego, CA, and was a founding member of the Electromicrobiology Group with his colleagues at the University of Southern California.
navy.cs.vsb.cz 9
Artificial Neural Networks Does the Ocean Think?
• Microbiologist Yuri Gorby – Ocean bacteria has 100 trillions trillions interconnected cells – Bacterial nanowires are conducting electricity – Firing is 5x faster than in human brain – Estimation of size and time is 140 000 000 km2 and more than 100000
years – Living organism included into this network – This activity is running too much long to be only random event
navy.cs.vsb.cz 10
Artificial Neural Networks Does the Ocean Think?
Does the Ocean Really Think?
See all at
Through the Wormhole with Morgan Freeman Does the Ocean Think? Season 05 Episode 05
https://www.youtube.com/watch?v=qCxgTzG0OTQ
navy.cs.vsb.cz 11
Artificial Neural Networks Does the Swarm System Think?
• Invincible by the Stanislaw Lem • Non-human thinking system • Swarm system as a brain • Fantasy and science fiction??? • Really?
navy.cs.vsb.cz 12
Artificial Neural Networks Invincible by the Stanislaw Lem
• Swarm robotic • Collective memory • Decentralized control
• See also – http://news.bbc.co.uk/2/hi/science/nature/8044200.stm – http://imr.ciirc.cvut.cz/Swarm/Swarm – http://mrs.felk.cvut.cz/research/swarm-robotics
navy.cs.vsb.cz 13
Artificial Neural Networks History
• Ancient Egypt
navy.cs.vsb.cz 14
Artificial Neural Networks The Difference Between ANN and Computer
ANN Computer
It is learned by adjusNng of weights, thresholds and structures
It is programmed instrucNons (if, then, go to ...)
Memory and execuNve elements are arranged altogether
The process and memory for him are separated
Parallelism SequenNal
Tolerate deviaNons from the original informaNon
Does not tolerate deviaNons
Self-‐organizaNon during learning Unchanged program structure
navy.cs.vsb.cz 15
Artificial Neural Networks ANN Use
The use of neural networks is really wide and becomes more and more important. They can be used for example to • Identification of data structures, radar or sonar signals,… • Prediction of behavior • Classification • Optimization • Filtration
navy.cs.vsb.cz 16
Artificial Neural Networks ANN History
The history of a continual development of neural networks in the first half of the 20th century, when it was first published work of neurons and their American model. W. S. McCulloch. In the 40 years of this century, with his student W. Pitts neuron model developed, which is practically used today. • 1920s W. S. McCulloch • 1958 F. Rosenblatt • 1969 M. Minsky spolu & S. Papert - Perceptron • 1980s D. Rumelhart, G. Hinton a R. Wiliams „Learning Internal
Representation by Error Propagation” • ANNs: Hopfield, Kohonen (SOM), Grossberg (ART) • ANN and unconventional hybridization with bio-inspired algorithms
and fuzzy logic
navy.cs.vsb.cz 17
Artificial Neural Networks ANN – a Few Facts
Definition: The ability of neural networks to generalize results come from their own topology. Neural network is connection of a large number of simple units and it is this large number of their mutual connections that gives them this property. • Human brain contain 1013-15 neurons • 10 000 neurons died per day (0.00025 % of total amount) • Each neuron has an average of 6000 somatic and dendritic
connections covering about 40% of its surface.…
navy.cs.vsb.cz 18
Artificial Neural Networks ANN – a Few Facts
Technicall neuron is basically a simple unit that evaluate-multiply all inputs to their weights (weights change during learning - hence the adaptation of the network), and the resulting values are added together. The resulting value is entered into the transfer function of the neuron (generally speaking, is a function which governs the response to input stimuli) and the output of this function is the output of the neuron, which serves as input to the next neuron.
That is the whole "miracle."
navy.cs.vsb.cz 19
Artificial Neural Networks ANN – Scheme of Biological Neuron
Biological neuron
navy.cs.vsb.cz 20
Artificial Neural Networks ANN – Scheme of Technical Neuron
Technical neuron
x1 w1
x2 w2
x3 w3
xn wn
...
y … output
1
n
i iix w
=∑
k wb … threshold
Inpu
t
navy.cs.vsb.cz 21
Artificial Neural Networks ANN – a Few Facts
• Biologically neuron is "a little more" complicated. In the description of the block also consists of inputs (dendrites), the body (soma) and outputs (axons). Its functions and is more complex. Each neuron is covered by the bilayer membrane lipids of the thickness of about 20 nm. In this membrane are embedded proteins that act as channel carrying ions. When channels open there is a change of membrane potential and the potential creation of a wave - the excitement is spreading through the body of the neuron dendrites to the axon.
• Synapse acts as a valve and that in two ways - excitation (excitation) and inhibitory (damping). What will be the response (excitation, inhibition) depends on the transmitters that are responsible for our memory.
navy.cs.vsb.cz 22
Artificial Neural Networks ANN - Classification
Definition: All types of neural networks are composed of the same structural units - neurons. May contain different transfer functions, connected with each other and adaptive - learning algorithm. All this determines kind of network. ANNs are classified according to basic three criteria. • According to the number of ANN layers:
– One layer (Hopfield, Kohonen network, ...) – Multiple layer (ART network, Perceptron, classical multilayer network
with backpropagation algorithm)
navy.cs.vsb.cz 23
Artificial Neural Networks ANN - Classification
• Depending on the type of ANN learning algorithm: – Supervised (network with backpropagation algorithm, ...) – Unsupervised (Hopfield, ...)
• According to the learning style of the ANN learning: – Deterministic (e.g. Backpropagation algorithm) – Stochastic (random setting their weights)…
navy.cs.vsb.cz 24
Artificial Neural Networks ANN - Transfer functions
Definition: Transfer function of the neuron is a function that transforms the input signal to the output signal between [0, 1] or [-1, +1]. This feature can be jump or combined and must be monotonic – i.e. that the associated responses of output to input is clear. The choice of function depends on the problem you want to solve. Among the most popular transfer function are: • Perceptron • Binary • Logistic (It is also used the term "sigmoid") • Hyperbolic tangent
navy.cs.vsb.cz 25
Artificial Neural Networks ANN - Perceptron transfer functions
It is a linear function, which was used in the first Rosenblatt neural network, and which because of its linearity was able to solve only linearly separable problems. This ultimately led to the decline of interest in neural networks.
( ) 0 x = 0F P x x else= ∀ >
navy.cs.vsb.cz 26
Artificial Neural Networks ANN - Binary transfer functions
As seen from the registration image and, this is a two valued function, which can have the value 0 or 1.
( ) 1 0 x = 0F B x else= ∀ >
navy.cs.vsb.cz 27
Artificial Neural Networks ANN - Logistic transfer functions
It is one of the most used functions, which was derived as an approximation of the transfer function of a biological neuron.
1( )1 xF Se−
=+
navy.cs.vsb.cz 28
Artificial Neural Networks ANN - Hyperbolic tangent transfer functions
Hyperbolic tangent function is equivalent to the logistic function, with the difference that can take values from -1 to +1, which means that it provides, inter alia, greater linear portion around the origin, which also has its importance.
( )x x
x xe eTanh xe e
−
−
−=+
navy.cs.vsb.cz 29
Artificial Neural Networks ANN – All transfer functions
( )x x
x xe eTanh xe e
−
−
−=+
1( )1 xF Se−
=+
( ) 1 0 x = 0F B x else= ∀ >
( ) 0x = 0
F P x xelse
= ∀ >
navy.cs.vsb.cz 30
Artificial Neural Networks ANN – Topology and parameters
Definition: The term topology (sometimes structure) networks understand how they are interconnected individual neurons, layers and the number of inputs and outputs of the network. In the network topology can include parameters such as the type of transfer functions in layers, learning parameter, momentum, etc.
Um�lá inteligence I Úvod.
-18-
U vícevrstvých sítí platí, =e první vrstva je v=dy v�tvící, co= znamená, =e neurony ve vstupní
vrstv� pouze distribuují vstupní hodnoty do
dalCí vrstvy. Vzhledem k tomu, =e se jedná
obecn� o vícebodový vstup do sít�, tak se
mluví o vstupních (výstupních) vektorech
informací. Jak je z Obr 1.5-2 vid�t, tak po�et
neuron� v jednotlivých vrstvách je variabilní a
zále=í na eCeném problému. Pozd�ji uká=eme,
=e lze zhruba odhadnout �i pípadn� vypo�ítat
po�et neuron� pro ka=dou vrstvu, nicmén� ani
pozd�ji uvedené vzorce na výpo�et vhodného
po�tu neuron� nejsou vCelék.
Mnohem vhodn�jCí zp�sob, jak ur�it po�et
neuron�, je pou=ít sí, která si sama tento
po�et m�ní podle vývoje globální chyby. Tato metoda je podrobn�ji objasn�na v kapitole 3.2.
Ka=dý vstup do neuronu má piazenu tzv. váhu Wxy, co= je bezrozm�rné �íslo, které UíkáV,
jaký význam má daný vstup pro písluCný neuron (ne pro sí �i problém).
U�ící schopnost neuronových sítí spo�ívá práv� v mo=nosti m�nit vCechny váhy v síti podle
vhodných algoritm� na rozdíl od sítí biologických, kde je schopnost se u�it zalo=ena na mo=nosti
tvorby nových spoj� mezi neurony. Fyzicky jsou tedy ob� schopnosti se u�it zalo=eny na
rozdílných principech, nicmén� z hlediska logiky ne. V pípad� vzniku nového spoje-vstupu u
biologického neuronu je to stejné, jako kdy= v technické síti je spoj mezi dv�ma neurony
ohodnocen vahou s hodnotou 0 a tudí= jako vstup pro neuron do kterého vstupuje, neexistuje. V
okam=iku kdy se váha zm�ní z 0 na libovolné �íslo, tak se daný spoj zviditelní - vznikne.
N11 N12 N13
N21 N22 N23
N31 N32 N33
W11 W33
Vstupní vrstva
Skrytá vrstva
Výstupní vrstva
Výstupní vektor
Vstupní vektor
Toto je klasická vícevrstvá sí (konkrétn� 3 vrstvy po 3neuronech). Po�et vrstev a neuron� v nich m�=e býtr�zný, nap. 4 vrstvy a v ka=dé jiný po�et neuron�.
Obr. 1.5-1 Vícevrstvá sí�
Um�lá inteligence I Úvod.
-19-
1.6 Jak sí� funguje
V�ta 1.6-1 O fázích sít� a tídách.
Jeden ze základních pedpoklad pro funkci sít� je její nau�ení - adaptace na daný
problém. Adapta�ní - u�ící proces se skládá ze dvou fází - adapta�ní a aktiva�ní, b�hem
kterých se nastavují váhy sít�.
Tídou rozumíme mnoFinu, která zahrnuje X jedinc se spole�nou vlastností.
Nov� vytvoenou, ale také jakoukoliv nenau�enou neuronovou sí lze povaBovat za jakéhosi
technického novorozence, který nic neumí. Neumí rozeznávat, klasifikovat,... Aby se mohla
pouBívat, musí být nau�ena stejn� jako kterýkoliv Bivý jedinec (samozejm�, Be zde se mnoBství
informací a délka u�ení nedá porovnávat). Z tohoto d�vodu byly vyvinuty algoritmy, pomocí
kterých se písluKná sí dokáBe nau�it na danou mnoBinu informací. Algoritmus se obvykle d�lí
na dv� fáze a to na fázi aktiva�ní (vybavovací) a adapta�ní (u�ící), které ke své �innosti
potebují trénovací mnoBinu. Trénovací mnoBina je skupina vektor� obsahujících informace o
daném problému pro u�ení. Pokud u�íme sí s u�itelem, pak jsou to dvojice vektor� vstup -
výstup. JestliBe u�íme sí bez u�itele, pak trénovací mnoBina obsahuje jen vstupní vektory. Pokud
pouBíváme jen fázi aktiva�ní, pak mluvíme o vybavování. Tuto fázi pouBíváme samostatn� jen
tehdy, kdyB je sí nau�ena. Cyklické stídání obou fází je vlastní u�ení.
W11 W32
Vstupní vrstva(v�tvící)
Skrytá vrstva
Výstupní vrstva
W12
a)
W11 W33
Vstupní vrstva(v�tvící)
Skrytá vrstva
Výstupní vrstva
b)
Obr. 1.5-2 Rzné topologie sítí
Um�lá inteligence I Úvod.
-19-
1.6 Jak sí� funguje
V�ta 1.6-1 O fázích sít� a tídách.
Jeden ze základních pedpoklad pro funkci sít� je její nau�ení - adaptace na daný
problém. Adapta�ní - u�ící proces se skládá ze dvou fází - adapta�ní a aktiva�ní, b�hem
kterých se nastavují váhy sít�.
Tídou rozumíme mnoFinu, která zahrnuje X jedinc se spole�nou vlastností.
Nov� vytvoenou, ale také jakoukoliv nenau�enou neuronovou sí lze povaBovat za jakéhosi
technického novorozence, který nic neumí. Neumí rozeznávat, klasifikovat,... Aby se mohla
pouBívat, musí být nau�ena stejn� jako kterýkoliv Bivý jedinec (samozejm�, Be zde se mnoBství
informací a délka u�ení nedá porovnávat). Z tohoto d�vodu byly vyvinuty algoritmy, pomocí
kterých se písluKná sí dokáBe nau�it na danou mnoBinu informací. Algoritmus se obvykle d�lí
na dv� fáze a to na fázi aktiva�ní (vybavovací) a adapta�ní (u�ící), které ke své �innosti
potebují trénovací mnoBinu. Trénovací mnoBina je skupina vektor� obsahujících informace o
daném problému pro u�ení. Pokud u�íme sí s u�itelem, pak jsou to dvojice vektor� vstup -
výstup. JestliBe u�íme sí bez u�itele, pak trénovací mnoBina obsahuje jen vstupní vektory. Pokud
pouBíváme jen fázi aktiva�ní, pak mluvíme o vybavování. Tuto fázi pouBíváme samostatn� jen
tehdy, kdyB je sí nau�ena. Cyklické stídání obou fází je vlastní u�ení.
W11 W32
Vstupní vrstva(v�tvící)
Skrytá vrstva
Výstupní vrstva
W12
a)
W11 W33
Vstupní vrstva(v�tvící)
Skrytá vrstva
Výstupní vrstva
b)
Obr. 1.5-2 Rzné topologie sítí
navy.cs.vsb.cz 31
Artificial Neural Networks ANN – Topology and parameters
navy.cs.vsb.cz 32
Artificial Neural Networks ANN – Topology and parameters
Beside standard structures, one can get very natural one by means of Darwinian evolution in computers…
navy.cs.vsb.cz 33
Artificial Neural Networks ANN – The dynamics of the network learning and classification
Definition: One of the basic assumptions of the network is its learning - adaptation to the problem. Adaptation - Learning process consists of two phases - the adaptation and activation during which the adjusted weights of the network. By term class we understood set that includes X individuals with common properties. • Activation phase is the process by which the presentation of vector
information for input to recalculate networks over all joints including their ranking weight up to the exit, where they appear on the network response vector in the format of the output vector. When learning this vector is compared with the original vector (required, output) and the difference between the two vectors (local variations - error) is stored in memory variables.
navy.cs.vsb.cz 34
Artificial Neural Networks ANN – The dynamics of the network learning and classification
• Adaptation phase is the process by which local error is minimized network so that the recalculated weights of individual joints direction from output to input for making the most similarities output response to the original vector.
• Cyclic repetition of both phases is called epoch. ANN are learned in epochs.
navy.cs.vsb.cz 35
Artificial Neural Networks ANN – Distance measurement
Definition: Distance measurement is used to determine to which class of the currently processed vector belongs. There is a few methods how to measure distances: • Hamming • Euclid • Manhattan • Square
navy.cs.vsb.cz 36
Artificial Neural Networks ANN – Hamming distance measurement
Calculation by this method is mainly used for binary vectors (Hopfield network). This distance is given by the sum of the elements of the two vectors.
[ ] [ ]1 2 1 2
1
... ...n n
n
i ii
A a a a B b b b
H a b=
= =
= −∑
navy.cs.vsb.cz 37
Artificial Neural Networks ANN – Euclid distance measurement
It is a distance calculation based on triangle. Formulas for calculating distances in 2D and ND (N-dimensional space) are.
2 21 2
2
1
2
( ( ) ( ))n
i
E X X for D
E A i B i for ND=
= Δ + Δ
= −∑
navy.cs.vsb.cz 38
Artificial Neural Networks ANN – Manhattan distance measurement
It's a simple modification of the Euclidean distance, which means a quick calculation at the cost of larger errors.
1
( ) ( )n
iM A i B i
=
= −∑
navy.cs.vsb.cz 39
Artificial Neural Networks ANN – Square distance measurement
This is again a simplification Manhattan distance, which in turn result in a higher rate, but even bigger mistake.
( ) ( )i
C MAX A i B i= −
navy.cs.vsb.cz 40
Artificial Neural Networks ANN – Training Set
Carrot
Cucumber
Apple
Pear
41 navy.cs.vsb.cz
Artificial Neural Networks ANN – Separable Problems
Um�lá inteligence I Úvod.
-20-
Aktiva�ní fáze je proces, pi kterém se pedlo>ený vektor informací na vstup sít� pepo�ítá pes
vCechny spoje v�etn� jejich ohodnocení vahami a> na výstup, kde se objeví odezva sít� na tento
vektor ve form� výstupního vektoru. Pi u�ení se tento vektor se porovná s vektorem originálním
(po>adovaným, výstupním) a rozdíl mezi ob�ma vektory (lokální odchylka - chyba) se ulo>í do
pam�ové prom�nné .
Adapta�ní fáze je proces, pi kterém je minimalizovaná lokální chyba sít� tak, >e se
pepo�ítávají váhy jednotlivých spoj� sm�rem z výstupu na vstup za ú�elem co nejv�tCí
podobnosti výstupní odezvy s originálním vektorem.
Po té se op�t opakuje aktiva�ní fáze. DalCí získaný rozdíl (lokální odchylka) se pi�te
k pedchozímu, atd... Pokud se tímto
postupem projde celá trénovací mno>ina,
je hotová jedna epocha. Celé sum�
odchylek za jednu epochu se íká
globální odchylka - chyba. Pokud je
globální odchylka menCí ne> námi
po>adovaná chyba, pak proces u�ení
skon�í.
Z výCe popsaného je vid�t, >e proces
u�ení není nic jiného, ne> pelévání
informací ze vstupu na výstup a naopak.
Pi u�ení se v tzv. vstupním prostoru
vytváejí shluky bod�, které pedstavují
jednotlivé �leny tíd, pi�em> ka>dý shluk
pedstavuje tídu (Obr. 1.6-1).
Pedstavte si, >e máte n�kolik tíd a to nap. tídu kol, aut, lodí, atd. Z ka>dé tídy vybereme
mno>inu reprezentativních zástupc� (vzor� pro u�ení) a vCechny popíCeme vhodným �íselným
zp�sobem ve form� vektor�. Pro ka>dou mno>inu vektor� jedné tídy vytvoíme vzorový vektor,
který bude zastupovat mateskou tídu z reálného sv�ta. V tomto okam>iku máme vytvoené
skupiny vektor� popisující jednotlivé �leny a jim písluCející pedstavitele tíd ve form� vektor�.
Napíklad máme vektory, které popisují vybrané �leny z tídy kol a vektor, který íká Sjá jsem
tída kolT. V tomto pípad� u�ení znamená to, >e se u�ící algoritmus sna>í najít takovou
Vstup 1
Vst
up 2
Tídy
Hranice tíd
Tída B
Tída C
Tída A
Obr. 1.6-1 Nelineární hranice mezi tídami - ideální pípad (?ádný z �len tíd A,B a C nele?í v prostoru druhé tídy) Z hlediska tvaru hranice je samozejm� ideální pímka.
0 2 4 6
0
2
4
6
Input 1
Border
Inpu
t 2
42 navy.cs.vsb.cz
Artificial Neural Networks ANN – Nonlinear Separable Problems
Um�lá inteligence I Úvod.
-24-
Tabulka 1.8-1 Data pro NLS
problém
1.9 T�ináctý Hilbert�v problém a Kolmogorov�v teorém
Jak u9 bylo nazna�eno, neuronová sí transformuje vstupní vektor na vektor výstupní.
Matematické koeny tohoto problému sahají a9 do roku 1900, kdy matematik D. Hilbert zveejnil
tzv. 13. Hilbert�v problém, který se týká nemo9nosti eNit obecn� rovnice 7. stupn� slo9ením
spojitých funkcí o dvou prom�nných, tzn. 9e koeny rovnice 1.9-1 se nedají vyjádit pomocí
jakékoliv kontinuální funkce jen o dvou parametrech [1].
Vstup 1
Vst
up 2
0
1
1
Výstup má hodnotu 0 Výstup má hodnotu 1
Problém je nelineárn� separabilní -hranicí nem�9e být pímka
Obr. 1.8-1 Problém nelineárn� separabilní
Vstup 1 Vstup 2 Výstup
0 0 0
1 0 1
0 1 1
1 1 0
Vstup 1
Vst
up 2
0
1
1
Výstup má hodnotu 0 Výstup má hodnotu 1
Problém je lineárn� separabilní - hranicí je pímka
Obr. 1.8-2 Problém lineárn� separabilní
Vstup1 Vstup 2 Výstup
0 0 1
1 0 0
0 1 1
1 1 1
Tabulka 1.8-2 - data pro LS problém
The border between two classes cannot be linear line
Input 1 Input 2 Output
0 0 0
1 0 1
0 1 1
1 1 0
43 navy.cs.vsb.cz
Artificial Neural Networks ANN – Linear Separable Problems
Input 1 Input 2 Output
0 0 1
1 0 0
0 1 1
1 1 1
Um�lá inteligence I Úvod.
-24-
Tabulka 1.8-1 Data pro NLS
problém
1.9 T�ináctý Hilbert�v problém a Kolmogorov�v teorém
Jak u9 bylo nazna�eno, neuronová sí transformuje vstupní vektor na vektor výstupní.
Matematické koeny tohoto problému sahají a9 do roku 1900, kdy matematik D. Hilbert zveejnil
tzv. 13. Hilbert�v problém, který se týká nemo9nosti eNit obecn� rovnice 7. stupn� slo9ením
spojitých funkcí o dvou prom�nných, tzn. 9e koeny rovnice 1.9-1 se nedají vyjádit pomocí
jakékoliv kontinuální funkce jen o dvou parametrech [1].
Vstup 1
Vst
up 2
0
1
1
Výstup má hodnotu 0 Výstup má hodnotu 1
Problém je nelineárn� separabilní -hranicí nem�9e být pímka
Obr. 1.8-1 Problém nelineárn� separabilní
Vstup 1 Vstup 2 Výstup
0 0 0
1 0 1
0 1 1
1 1 0
Vstup 1
Vst
up 2
0
1
1
Výstup má hodnotu 0 Výstup má hodnotu 1
Problém je lineárn� separabilní - hranicí je pímka
Obr. 1.8-2 Problém lineárn� separabilní
Vstup1 Vstup 2 Výstup
0 0 1
1 0 0
0 1 1
1 1 1
Tabulka 1.8-2 - data pro LS problém
The border between two classes is linear line.
Input 1
Inpu
t 2
navy.cs.vsb.cz 44
Artificial Neural Networks ANN – Hilbert's 13th problem and the Kolmogorov theorem
As already indicated, the neural network transforms the input vector to the vector output. Mathematical roots of the problem date back to 1900 when the mathematician D. Hilbert published the so-called Hilbert's 13th problem, which refers to the inability to solve the general equation of the seventh degree of the composition of continuous functions of two variables.
( )2 1
1 21 1
( , ,..., )n n
n j k jk j
f x x x h g xλ+
= =
⎛ ⎞= ⎜ ⎟
⎝ ⎠∑ ∑
navy.cs.vsb.cz 45
Artificial Neural Networks ANN – Hilbert's 13th problem and the Kolmogorov theorem
The neural network (three layers) that transforms input to output in the simple case of two inputs and one output contains the above equation in the first hidden layer (direction from output to input) 2n +1 neurons and in the next lower layer of n (2n +1 ) neurons. Other modifications it was found that by Kolmogorov theorem on problems of neural networks only results existential evidence that any solution to the problem of sufficient network of three layers.
navy.cs.vsb.cz 46
Artificial Neural Networks ANN – Questions
1. Who and when created the first model of the neuron. 2. When and by whom it was first constructed as a neural network was
called. 3. Why has fallen interested in neural networks. 4. What is the difference between the traditional PC and neural
networks. 5. What passes generalization ability of neural networks. 6. Explain the similarities between biological and technical neuron. 7. How is divided into neural networks and what they mean different
labels. 8. What types of transfer functions familiar to you, your relationships
and draw them.
navy.cs.vsb.cz 47
Artificial Neural Networks ANN – Questions
9. Describe the stage of network learning and equipping know how and ongoing.
10. Why network terminates learning. 11. Explain in your own words what it is: era, the global error class. 12. What types of measuring distances in neural networks, you know,
describe and explain their significance. 13. Explain what does it means the term linear separable. 14. Explain in what follows Kolmogorov's theorem and its consequences.
navy.cs.vsb.cz 48
Conclusions
• Principles and impact of our perception has been introduced • History of ANNs and basic terms were explained • Difference between ANNs and computer was described as well as
between biological and technical neuron • Different kind of basic transfer functions were demonstrated • Classification ANNs was explained • ANN topology and its variability was discussed • Linear and nonlinear separable problems were introduced in
connection with history and stagnation of ANNs • Hilbert's 13th problem and the Kolmogorov theorem was mentioned
as well as its impact on ANNs history
navy.cs.vsb.cz 50
Copyright
This didactic material is meant for the personal use of the student only, and is copyrighted. Its reproduction, even for a partial utilization, is strictly forbidden in compliance with and in force of the law on Authors rights.
Copyright©NAVY.CS.VSB.CZ