i. systems biology of circadian rhythms & ii. neural network …€¦ · bio_cycle...
TRANSCRIPT
![Page 1: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/1.jpg)
I. Systems Biology of Circadian Rhythms
II. Neural Network Capacity
P. BaldiUniversity of California, Irvine
Department of Computer ScienceInstitute for Genomics and BioinformaticsCenter for Machine Learning and Intelligent Systems
1
![Page 2: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/2.jpg)
Circadian (and Other) Rhythms are Pervasive in Biological Systems
Circadian (≈24 hours)
Ultradian (<24 hours)
Seasonal (>24 hours)
EEG activity during sleep
Sleep/wake
Strumwasser, F. (1960) Some physiological principles governing hibernation. Bulletin of the Museum of Comparative Zoology, 124, Harvard University
Hibernation
EEG activity during sleep
2
![Page 3: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/3.jpg)
Circadian Rhythms are Self-Perpetuating
Observation: plant leaves continued to fold rhythmically,
even in constant darkness
Observation: Plant leaves continued to fold rhythmically but rhythms slightly deviant from the
24-hour time span. This was indicative of endogenous, free-running clock
Jean-Jacques d'Ortous de Mairan
Augustin Pyramus de Candolle
Mimosa pudica
3
![Page 4: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/4.jpg)
4
![Page 5: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/5.jpg)
5
Astronauts in Caves: Michel Siffre
![Page 6: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/6.jpg)
Circadian Clocks are Tethered to the Environment via the SCN
6
![Page 7: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/7.jpg)
7
![Page 8: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/8.jpg)
Cell, 155, 7, 1464-1478, 2013.8
![Page 9: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/9.jpg)
Autonomous Core Clock
9
![Page 10: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/10.jpg)
Consequences of the Clock Breaking Down
• Sleep disorders (FASPS, etc.)• Depression• Obesity/metabolic disorders• Aging
10
![Page 11: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/11.jpg)
Towards Personalized Medicine
11
Zhang et al. PNAS, 11, 45, 2014.
![Page 12: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/12.jpg)
BIO_CYCLE
• Given a circadian time series for a transcript, metabolite, protein etc determine if it is periodic or not with some statistical significance.
• If periodic, estimate the period, the phase, the amplitude.
12
![Page 13: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/13.jpg)
BIO_CYCLE Architecture
13
3 Hidden Layers 100 Units Each
Sigmoidal: Periodic/AperiodicLinear: Period
Additional Computation of Amplitudes, Phases, p-values, q-values
![Page 14: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/14.jpg)
Sample of Training Signals
14Training set 1-10M examples
![Page 15: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/15.jpg)
Evaluation
15
![Page 16: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/16.jpg)
BIO_TIME
• Given measurements for transcripts, metabolites, proteins etc taken at a single time point, determine the time (or phase).
• Initially use only core clock transcripts in mouse.
16
![Page 17: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/17.jpg)
17
Circadian AutoencoderNeural Network
![Page 18: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/18.jpg)
BIO_TIME Architecture
18
10-16 Core Clock Genes
Cos=U1/sqrt(U12+U22)Sin=U2/sqrt(U12+U22)
![Page 19: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/19.jpg)
BIO_TIME EVALUATION
• For WT, predicts time with 90 minutes fairly robustly across tissues.
• Used to impute a time for all mouse Gene Expression Omnibus (GEO) experiments.
• Challenges for extending to other species.
19
![Page 20: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/20.jpg)
Examples of High-Throughput Experiments
20
S. Masri, T. Papagiannakopoulos, K. Kinouchi, Y. Liu, M. Cervantes, P. Baldi, T. Jacks, and P. Sassone-Corsi. Lung Adenocarcinoma Distally Rewires Hepatic Circadian Homeostasis. Cell, 165, 4, 896—909, (2016).
S. Masri, P. Rigor, M. Cervantes, N. Ceglia, C. Sebastian, C. Xiao,M. Roqueta-Rivera, C. Deng, T. F. Osborne, R. Mostoslavsky, P. Baldi, and P. Sassone-Corsi. Partitioning Circadian Transcription by SIRT6 Leads to Segregated Control of Cellular Metabolism. Cell, 158, 3, 659—672, (2014).
K. L. Eckel-Mahan, V. R. Patel, S. de Mateo, N. J. Ceglia, S. Sahar, S. Dilag, K. A. Dyar, R. Orozco-Solis, P. Baldi, and Paolo Sassone-Corsi. Reprogramming of the Circadian Clock by Nutritional Challenge. Cell, 155, 7, 1464-1478, (2013).
M. M. Bellet, E. Deriu, J. Liu, B. Grimaldi, C. Blaschitz, M. Zeller, R. A. Edwards, S. Sahar, S.Dandekar, P. Baldi, M. D.George, M. Raffatellu, and P. Sassone-Corsi. The Circadian Clock Regulates the Host Response to Salmonella. PNAS, 110, 24, 9897-9902, (2013).
K. L. Eckel-Mahan, V. R. Patel, K. S.Vignola, R. P. Mohney, P. Baldi, and P. Sassone-Corsi. Coordination of Metabolome and Transcriptome by the Circadian Clock. PNAS, 109 (14) 5541-5546, (2012).
![Page 21: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/21.jpg)
http://circadiomics.igb.uci.eduNature Methods 9, 8, 772-773, 2012.Nuclei Acids Research, Web Server Issue, in press, (2018).
21
![Page 22: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/22.jpg)
Cell, 155, 7, 1464-1478, 2013.22
![Page 23: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/23.jpg)
23
Trends in Cell Biology, 24, 329-331, 2104.Bioinformatics, 31, 19, 2015.
At p = 0.05, 68% oscillate.95% with more recent data sets.
![Page 24: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/24.jpg)
24
At p = 0.05, 67% oscillate.
![Page 25: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/25.jpg)
25
![Page 26: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/26.jpg)
26
![Page 27: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/27.jpg)
Main Findings1. The core clock contains only a dozen genes.2. In any tissue/condition 10% (± 5%) of transcripts
or metabolites oscillates. 3. The overlap across tissues/conditions is small
(2%).4. Genetic or environmental perturbations result in
massive changes:– Amplitude changes (including suppression)– Phase changes– New oscillations
27
![Page 28: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/28.jpg)
Explanation
1. In general, molecular species in isolation do not oscillate
2. Loops of interacting (regulatory, metabolic, PPI) species can oscillate
3. Many oscillator loops in the cell4. Why do they tend to have an intrinsic period
of 24h?
28
![Page 29: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/29.jpg)
29
Physical objects have intrinsic vibration frequencies….
![Page 30: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/30.jpg)
30
![Page 31: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/31.jpg)
3.5x109x365= 1.3 x 1012
More like: 2 x 1012
(period has increased due to tidal effects) 31
Number of Revolutions since the Origin of Life
![Page 32: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/32.jpg)
Cyanobacteria
32
![Page 33: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/33.jpg)
Network of Coupled Circadian Oscillators: Spectrum of Models
• At one extreme, completely centralized. The core clock controls all the oscillators.
• At the other extreme, completely decentralized. The oscillators compete and self-organize.
• Biology is somewhere in between. Where?
33
![Page 34: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/34.jpg)
34
![Page 35: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/35.jpg)
Circadian Regulatory Control (CRC)
• TF protein coding transcript if and only if:1) TF and transcript are circadian at some p-
value (BIO_CYCLE);2) TF has binding sites in the promoter of
transcript (MotifMap, MotifMap-RNA);3) TF and transcript have the “right” phase
lag;Similarly for RBPs (taking the introns or UTRs of the target transcript).
35
![Page 36: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/36.jpg)
Empirical Distribution of Lags
36
![Page 37: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/37.jpg)
Tables showing the ranking of circadian TFs and RBPs by CRC E-score in different tissue types. The leftmost table shows ranking in mouse transcriptome across all datasets.
RBPs are labeled in red while TFs are labeled in black. Core clock TFs have been removed from the listing.
37
![Page 38: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/38.jpg)
38
![Page 39: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/39.jpg)
39
Highly enriched in olfactory GPCRs.
![Page 40: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/40.jpg)
40
Highly enriched in olfactory GPCRs.
![Page 41: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/41.jpg)
Hierarchical Organization
• Core Clock at the apex (level 0)• Level 1 (35%, distance 1)• Level 2 (70%, distance 2)• Level 3 (80%, distance 3)• Fan out decreases with distance.• There is feedback between levels.• Most of cellular reprogramming must occur at level 1.• Small set of transcripts that do no oscillate in any
experiment: highly enriched in olfactory GPCRs.
41
![Page 42: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/42.jpg)
42
![Page 43: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/43.jpg)
Summary• Roughly 10% of all molecular species oscillate in any
cell/tissue/condition with small overlaps beyond the core clock.• Genetic, epigenetic, and environmental conditions (e.g. diet) have a
profound effect on which species oscillate and lead to cellular reprogramming.
• Tools: BIO_CYCLE, BIO_TIME, CircadiOmics.• Coupled-circadian-oscillator networks provides a general
framework.• Hierarchical organization emanating at the core clock. • Precision medicine: monitor and optimize health by monitoring
and optimizing oscillations. • Precision medicine: New diagnostic tools. Optimize timing of
therapeutic interventions (drugs). • Even a 5% increase in efficacy could have significant impact.
43
![Page 44: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/44.jpg)
II. Neural Network Capacity
44
![Page 45: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/45.jpg)
Neural Network Capacity
• h = target function (typically known from examples)• A = class of hypothesis or approximating functions (typically
associated with a NN architecture)
h
A
![Page 46: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/46.jpg)
Neural Network Capacity
• h = target function (typically known from examples)• A = class of hypothesis or approximating functions (typically
associated with a NN architecture)
h
A
C(A) = log2 |A|
![Page 47: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/47.jpg)
Neural Network Capacity• Can we compute C(A) for specific, interesting, neural
networks?
h
A
C(A) = log2 |A|
![Page 48: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/48.jpg)
Neural Network Capacity
• Assume neural networks of linear or polynomial threshold gates (Boolean functions)
f = sign
![Page 49: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/49.jpg)
Threshold Gates
• Linear Threshold Gatesy = sign [∑i wi xi]
• Polynomial Threshold Gatesy = sign [Pd(x)]
• Variations:– Homogenous– Binary weights– Positive weights
![Page 50: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/50.jpg)
Network Capacity
• Given a network of linear or polynomial threshold gates, |A| is finite.
• We define the capacity as:
C(network) = log2 |A|=log2(#number of Boolean functions that can be
implemented by the network)
![Page 51: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/51.jpg)
ANDORNOTGEQLEQSINGLE
PARITYCONNECTEDPAIR
22𝑁𝑁
?
Capacity of a Single Linear Threshold Gate
![Page 52: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/52.jpg)
Capacity Of Linear Threshold Gates
C[LTG(N)] ≤ N2
T. Cover 1965
![Page 53: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/53.jpg)
Capacity Of Linear Threshold Gates
C[LTG(N)] ≤ N2
cN2 ≤ C[LTG(N)] (c<1) T. Cover 1965
S. Muroga (1965)
![Page 54: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/54.jpg)
Capacity Of Linear Threshold GatesC[LTG(N)] ≤ N2
cN2 ≤ C[LTG(N)] (c<1) T. Cover 1965
S. Muroga (1965)
C[LTG(N)] = N2 (1 + o(1))
Yu. A. Zuev (1989)
![Page 55: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/55.jpg)
ANDORNOTGEQLEQSINGLE
PARITYCONNECTEDPAIR
22𝑁𝑁
2𝑁𝑁2
Capacity of a Single Linear Threshold Gate
![Page 56: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/56.jpg)
Capacity Of Polynomial Threshold Gates
C[PTG(N,d)] ≤ 𝑁𝑁𝑑𝑑+1
𝑑𝑑!P.B. 1988
![Page 57: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/57.jpg)
Capacity Of Polynomial Threshold Gates
C[PTG(N,d)] ≤ 𝑁𝑁𝑑𝑑+1
𝑑𝑑!P.B. 1988
𝑁𝑁𝑑𝑑 + 1 ≤ C[PTG(N,d)] M. Saks 1993
![Page 58: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/58.jpg)
Capacity Of Polynomial Threshold Gates
C[PTG(N,d)] ≤ 𝑁𝑁𝑑𝑑+1
𝑑𝑑!P.B. 1988
𝑁𝑁𝑑𝑑 + 1 ≤ C[PTG(N,d)] M. Saks 199
C[PTG(N,d)] = 𝑁𝑁𝑑𝑑+1
𝑑𝑑!(1 + o(1))
P.B. and R.V. 2018
![Page 59: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/59.jpg)
Additional Results
• Binary weights (wi= -1 or +1):
C(Binary-Weight LTG) = N
• Positive weights (wi ≥ 0): C(Positive-Weight LTG) = N2 – N
• ReLUC(ReLU) = N2 + N
![Page 60: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/60.jpg)
𝑑𝑑 = 1 𝑁𝑁2
𝑑𝑑 = 2 𝑁𝑁3/2
⁄𝑁𝑁𝑑𝑑+1 𝑑𝑑!
𝑑𝑑 = 1𝑁𝑁
𝑑𝑑 = 1 𝑁𝑁2−𝑁𝑁
Linear threshold functions with binary weights
Linear threshold functions with positive weights
Linear threshold functions (d=1)
Polynomial threshold functions of degree d
All Boolean functions of N variables 2N
ReLU N2 + N
![Page 61: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/61.jpg)
What about Networks?
• Focus on LTG but everything can be extended to PTG.• Fully connected RNN with N linear threshold gates:
C(RNN) = N3
• More generally, for any neural network NN with N linear threshold gates:
C(NN) ≤ ∑ capacities = ∑ (fan-ini)2 ≤ N (max fan-ini)2
![Page 62: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/62.jpg)
Neural Network Capacity
• The capacity satisfies: C(NN) ≤ ∑ capacities
• The capacity of a polynomial-size network can be expressed as a polynomial. The capacity of a network with N LTG units is at most N3.
• Can we get estimates on the capacity?• Can we get lower bounds on the capacity?• Can we compare different architectures?
![Page 63: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/63.jpg)
Neural Network Capacitywith Single Hidden-Layer
N
M
Theorem: C[N,M,1] = MN2 (1 + o(1))
For instance, if M = α N: C ≈ α N3
![Page 64: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/64.jpg)
Conclusions• Precise definition of capacity C=log2[#functions].• C [PTG(N,d)] = Nd+1/d! (1+o(1)).• C[LTG(N,d)] = N2(1+o(1)).• Extensions to special cases (binary weights, positive
weights).• The capacity of a fully connected network with N units
is N3.• The capacity of feedforward networks can be
estimated. It is a low degree polynomial in the relevant variables (fan-ins, layer sizes).
• C[N,M,1] = MN2(1+o(1))
![Page 65: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/65.jpg)
Conclusions
• The capacity can be used to compare different architectures.
• Ongoing work: deep networks versus shallow networks, finite size versus asymptotic.
![Page 66: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/66.jpg)
THANK YOU
66
![Page 67: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/67.jpg)
A
67
![Page 68: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/68.jpg)
Diet Change
Normal Chow (2221) (1517) High-Fat (1110)
68
![Page 69: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/69.jpg)
Clock-Bmal1 form a complex
And bind to tandem E-boxes in the promoter of Upp2
Driving the rhythmic expression of Upp2
And in turn the rhythmic expression of Uracil and Uridine
Example:
PNAS, 109 (14) 5541-5546, 2012. 69
![Page 70: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/70.jpg)
Coupled-Circadian-Oscillators Framework
• Many oscillatory loops• Intrinsic periodicity close to 24 h (evolution)• Coupled-oscillators• Many coupling mechanisms:
– ≈10% of genes are in a directed loop containing Clock or Bmal1
– ≈60% of genes are within two hops from Clock or Bmal1
– odd/even loops• Amplitude limited by energy balance
(homeostasis)
70
![Page 71: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/71.jpg)
71
Highly enriched in olfactory GPCRs.
![Page 72: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/72.jpg)
72
Highly enriched in olfactory GPCRs.
![Page 73: I. Systems Biology of Circadian Rhythms & II. Neural Network …€¦ · BIO_CYCLE Architecture. 13. 3 Hidden Layers 100 Units Each Sigmoidal: Periodic/Aperiodic. Linear: Period](https://reader034.vdocuments.us/reader034/viewer/2022050304/5f6d2b71502c792a7b1fac63/html5/thumbnails/73.jpg)
73
Highly enriched in olfactory GPCRs.