estimating mutual information kenneth d. harris 25/3/2015
TRANSCRIPT
![Page 1: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/1.jpg)
Estimating mutual information
Kenneth D. Harris25/3/2015
![Page 2: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/2.jpg)
Entropy
• Number of bits needed to communicate , on average
![Page 3: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/3.jpg)
Number of bits saved communicating X if you know YNumber of bits saved communicating Y if you know X
• If , • If and are independent,
Mutual information
![Page 4: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/4.jpg)
“Plug in” measure
• Compute histogram of X and Y, .• Estimate
• Biased above
![Page 5: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/5.jpg)
No information
• X and Y independent and random binary variables
• True information is zero
• Histogram is rarely
![Page 6: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/6.jpg)
Bias correction methods
• Not always perfect
• Only use them if you truly understand how they work!
Panzeri et al, J Neurophys 2007
![Page 7: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/7.jpg)
Cross-validation
• Mutual information measures how many bits I save telling you about the spike train, if we both know the stimulus
• Or how many bits I save telling you the stimulus, if we both know the spike train
• We agree a code based on the training set
• How many bits do we save on the test set? (might be negative)
![Page 8: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/8.jpg)
Strategy
• Use training set to estimate
Compute
Codeword length when we don’t know stimulus Codeword length when we do know stimulus
![Page 9: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/9.jpg)
This underestimates information
• Can show expected bias is negative of plug-in bias
![Page 10: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/10.jpg)
Two choices:
• Predict stimulus from spike train(s)
• Predicted spike train(s) from stimulus
![Page 11: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/11.jpg)
Predicting spike counts
• Single cell
Problem: variance is higher than PoissonSolution: use generalized Poisson or negative binomial distribution
Likelihood ratio
![Page 12: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/12.jpg)
Unit of measurement
“Information theory is probability theory with logs taken to base 2”
• Bits / stimulus• Bits / second (Bits/stimulus divided stimulus length)• Bits / spike (Bits/second divided mean firing rate)
• High bits/second => dense code• High bits/spike => sparse code.
![Page 13: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/13.jpg)
Bits per stimulus and bits per spike
6 bits if spike bits if no spikebits/stimulus spikes/stimulus7.4 bits/spike
1 bit if spike1 bit if no spike1 bit/stimulus.5 spikes/stimulus2 bits/spike
![Page 14: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/14.jpg)
Measuring sparseness with bits/spike
Sakata and Harris, Neuron 2009
![Page 15: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/15.jpg)
Continuous time
• is intensity function
• If when there is a spike, this is • Must make sure predictions are never too close to 0
• Compare against where is training set mean rate
Itskov et al, Neural computation 2008
![Page 16: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/16.jpg)
Likelihood ratio
Constants cancel! Good thing, since they are both infinite.Remember these are natural logs. To get bits, divide by .
![Page 17: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/17.jpg)
Predicting firing rate from place
Cross-validation finds best smoothing width
Without cross-validation, appears best with least smoothing
Harris et al, Nature 2003
![Page 18: Estimating mutual information Kenneth D. Harris 25/3/2015](https://reader034.vdocuments.us/reader034/viewer/2022051516/56649d0a5503460f949dc52c/html5/thumbnails/18.jpg)
Comparing different predictions
Harris et al, Nature 2003