an introduction to artificial neural networks...an introduction to artificial neural networks yipeng...
TRANSCRIPT
![Page 1: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/1.jpg)
An introduction to artificial neural networks
Yipeng SunLLRF 2019 workshop, ChicagoOct 3, 2019
Thanks to: Google Colab computing; Coursera.org for online courses.Work supported by the U.S. Department of Energy, Office of Science, Office of Basic Energy Sciences, under Contract No. DE-AC02-06CH11357.
![Page 2: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/2.jpg)
Outline
2
● Overview on artificial intelligence and Deep learning● Artificial neural networks code developments● Applications of this code:
– Supervised and unsupervised learning– Classifier– Regression
● Conclusion
![Page 3: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/3.jpg)
Machine learning examples
kNN: k nearest neighborsk=1
NO explicit instructions; NO need math/physics model; needs data
k-means clustering (unsupervised)k=4
![Page 4: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/4.jpg)
Neural network: forward and backward propagation
Input layer Output layer
forward
BackwardChain rule
Cost function(MSE, MAE, CE)
Z A Z(n) = W*A(n-1) + bA(n) = F(Z(n))
dZ dAdA(n) = W*dZ(n+1)
dZ(n) = dA(n)*F’(Z(n))dW, db from dZ(n) & A(n-1)
![Page 5: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/5.jpg)
Neural network as arbitrary function approximator
Activations
Rectified linear unit ReLu: A = max(0,X)SoftPlus: A = log(1+exp(X))
Only one layer
2 ReLu: x^2 (top); 50 ReLu: Radial-basis functions (bottom)
![Page 6: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/6.jpg)
Multiple layers: function composition g(f(x))
One layer (top); two layers (bottom)
Z = cos(X^2*0.1 +sin(Y^3*0.1)+0.5)
![Page 7: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/7.jpg)
Python codes in development for neural network
7
● Motivation: to understand the algorithms and apply to our work● Why Python
– ‘default’ choice of deep learning: computation speed, user community, existing algorithms (TensorFlow, Keras, Caffe, PyTorch...)
● What has been done:– Initialization (0, random, random normalised) – Forward and backward propagation (activations, gradients)– Cost functions (MSE, MAE, CE)– Optimization algorithms (SGD, Rprop, momentum, ImpliMom, AdaGrad,
RMSprop, adam)– Variational auto encoder, generative adversarial networks
● To be developed: – More activation and cost functions– regularization algorithms (dropout...), CNN, RNN...
![Page 8: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/8.jpg)
Application: rose function etc.
Training data: red-blue dots from rose functionPrediction: contour plotAnimation on iterations
Training data: sklearn.datasets.make_moonsPrediction: contour plotAnimation on iterations
Higher than 90% ‘accuracy’
![Page 9: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/9.jpg)
Application: Hand writing, 10-class [1]
Training/val data: 60k images (28 by 28 pixels)Test data: 10k imagesAccuracy: 99% (training data), ~98% (test/val)
[1] MNIST database, http://yann.lecun.com/exdb/mnist/
Predictions (selected blocks with wrong labels)
![Page 10: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/10.jpg)
Generative models: variational auto encoder
2D latent (hidden) space, Z 2D latent space, Y^
Training with MNIST database, http://yann.lecun.com/exdb/mnist/
![Page 11: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/11.jpg)
Generative models: variational auto encoder
Generated numbers
Generated synthetic cat images
Generated cats animation
* Cat data, https://ajolicoeur.wordpress.com/cats/
Training data
![Page 12: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/12.jpg)
GAN: Generative Adversarial Networks
Training: discriminator and generator GAN Generated synthetic imagesThis code (left); Keras/TF (right)
![Page 13: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/13.jpg)
Application: R matrix for accelerator elements*5-class: sbend, quad, drift, skew quad, bendEdge
(Accuracy: Training 99.8% validation 99.6% test 99.1%)
*Yipeng Sun, NAPAC19
![Page 14: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/14.jpg)
Benchmark on optimization algorithms*
‘adam’ seems to be most robustexample with R matrix classifications, showing average accuracy
*Yipeng Sun, NAPAC19
![Page 15: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/15.jpg)
Application: 2nd order map*
ANN predictionsANN training
Training data input Training data output
ANN particle tracking
*Yipeng Sun, NAPAC19
![Page 16: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/16.jpg)
Top up injection efficiency v.s. loss monitors etc.
16
Input: APS operation data: 01-22-2019 to 06-28-2019
Training / Cost 2%
Predictions on injection efficiency (output)
![Page 17: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/17.jpg)
Lifetime
Depends on:● Beam current● No. of bunches● Coupling● Chromaticity● Bunch length
● RF voltage● Mom. compaction
● ID energy loss● Physical apertures
● ID4 orbit
Results in:● Beam loss monitors
Predictions on lifetime (bottom). Training data (top)
Training data: APS operation 01-2019 to 06-2019
![Page 18: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/18.jpg)
Lifetime
Predictions on lifetime Analyzing trained NN model
![Page 19: An introduction to artificial neural networks...An introduction to artificial neural networks Yipeng Sun LLRF 2019 workshop, Chicago Oct 3, 2019 Thanks to: Google Colab computing;](https://reader030.vdocuments.us/reader030/viewer/2022040609/5ecd305ae4765603a703f717/html5/thumbnails/19.jpg)
APS-U Beam Physics Review, May 16-17, 2018 19
Copyright
The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan.
http://energy.gov/downloads/doe-public-access-plan