jochen triesch, uc san diego, triesch 1 information theory recommended readings: simon haykin:...
Post on 16-Jan-2016
233 Views
Preview:
TRANSCRIPT
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 1
Information TheoryInformation TheoryRecommended readings:Simon Haykin: Neural Networks: a comprehensive foundation,Prentice Hall, 1999, chapter “Information Theoretic Models”.
Hyvärinen, J. Karhunen, E. Oja: Independent Component AnalysisWiley, 2001.
What is the goal of sensory coding?Attneave (1954): “A major function of the perceptual machinery isto strip away some of the redundancy of stimulation, to describeor encode information in a form more economical than that in whichit impinges on the receptors.”
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 2
MotivationMotivation
Observation: Brain needs representations that allow efficient access to information and do not waste precious resources. We want efficientrepresentation and communication.
A natural framework for dealing with efficient coding of information is information theory, which is based on probability theory.
Idea: the brain may employ principles from information theory to code sensory information from the environment efficiently.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 3
Observation: (Baddeley, 1997) Neurons in lower (V1) and higher(IT) visual cortical areas, show approximately exponential distributionof firing rates in response to natural video (panel A). Why may this be?
Dayan and A
bbott
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 4
What is the goal of sensory coding?Sparse vs. compact codes:
two extremes: PCA (compact) vs. Vector Quantization
Advantages of sparse codes (Field, 1994):
• signal-to-noise ratio:small set of very active units above “sea of inactivity”
• correspondence and feature detection:small number of active units facilitates finding the rightcorrespondences, higher order correlations are rarer and thus moremeaningful
• storage and retrieval with associative memory: storage capacity greater in sparse networks (e.g. Hopfield)
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 5
What is the goal of sensory coding?Another answer:
Find independent causes of sensory input.
Example: image of a face on the retina depends on identity, pose,location, facial expression, lighting situation, etc. We can assumethat these causes are close to independent.
Would be nice to have a representationthat allows easy read out of (some of)the individual causes.
Example: Independence: p(a,b)=p(a)p(b)p(identity, expression)=p(identity)p(expression)(factorial code)
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 6
What is the goal of sensory coding?More answers:
• Fish out information relevant for survival (markings of prey and predators).
• Fish out information that lead to rewards and punishments.
• Find a code that allows good generalizations. Situations requiring similar actions should have similar representations
• Find representations that allow different modalities to talk to one another, facilitate sensory integration.
Information theory gives us a mathematical framework for discussing some but not all of these issues. Only speaks about probabilities ofsymbols/events but not their meaning.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 7
Efficient coding and compressionEfficient coding and compressionSaving space on your computer hard disk:- use “zip” program to compress files, store images and movies in “jpg” and “mpg” formats- lossy versus loss-less compression
Example: RetinaWhen viewing natural scenes, the firing of neighboring rods orcones will be highly correlated. Retina input: ~108 rods and conesRetina output: ~106 fibers in optic nerve, reduction by factor ~100Apparently, visual input is efficiently re-coded to save resources.
Caveat: Information theory just talks about symbols, messages and their probabilities but not their meaning, e.g., how survival relevant they are for the organism.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 8
Communication over noisy channelsCommunication over noisy channelsInformation theory also provides a framework for studying information transmission across noisy channels where messages may get compromised due to noise.
Examples:
modem → phone line → modem
Galileo satellite → radio waves → earth
daughter cellparent cell daughter cell
computer memory → disk drive → computer memory
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 9
Note: base of logarithm is matter of convention, in information theory base 2 is typically used: information measured in bits in contrast to nats for base e (natural logarithm).
Recall:
xa
xx
xyx
yxyx
yxxy
xa
x
xx
x
y
bb
a
a
log
2
))exp(ln(
)log()log(
)log()log()/log(
)log()log()log(
loglog
1log
)2ln(
)ln()(log
Matlab:LOG, LOG2, LOG10
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 10
How to Quantify Information?How to Quantify Information?
consider discrete random variable X taking values from theset of “outcomes” {xk, k=1,…,N} with probability Pk: 0≤Pk≤1; ΣPk=1
Question: what would be a good measure for the information gainedby observing outcome X= xk ?
Idea: Improbable outcomes should somehow provide more informationLet’s try: )log()/1log()( kkk PPxI Properties:I(xk) ≥ 0 since 0≤Pk≤1
I(xk) = 0 for Pk=1 (certain event gives no information)
I(xk) > I(xi) for Pk < Pi
logarithm is monotonic, i.e:if a<b then log(a)<log(b)
“Shannon information content”
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 11
Information gained for sequenceInformation gained for sequenceof independent eventsof independent events
Consider observing the outcomes xa, xb, xc in succession;the probability for observing this sequence is P(xa,xb,xc) = PaPbPc
Let’s look at the information gained:
I(xa,xb,xc) = - log(P(xa,xb,xc)) = -log(PaPbPc)
)log()/1log()( kkk PPxI
= -log(Pa)-log(Pb)-log(Pc)
= I(xa) + I(xb) +I(xc)
Information gained is just the sum of individual
information gains
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 12
EntropyEntropy
Question: What is the average information gained when observinga random variable over and over again?
Answer: Entropy!
k
kk PPXIEXH )log()()(
Notes:• entropy always bigger than or equal to zero• entropy is a measure of the uncertainty in a random variable• can be seen as generalization of variance• entropy related to minimum average code length for variable• related concept in physics and physical chemistry: there entropy is a measure of the “disorder” of a system.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 13
ExamplesExamples
1. Binary random variable: outcomes are {0,1} whereoutcome ‘1’ occurs with probability P andoutcome ‘0’ occurs with probability Q=1-P.
Question: What is the entropy of this random variable?
k
kk PPXH )log()(
Answer: (just apply definition))log()log()( QQPPXH
Note: entropy zero if one outcomecertain, entropy maximized if bothoutcomes equally likely (1 bit)
0)0log(0 :Note
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 14
2. Horse Race: eight horses are starting and their respectiveodds of winning are: 1/2, 1/4, 1/8, 1/16, 1/64, 1/64, 1/64, 1/64
What is the entropy?H = -(1/2 log(1/2) + 1/4 log(1/4) + 1/8 log(1/8) + 1/16 log(1/16) + 4 * 1/64 log(1/64) = 2 bits
What if each horse had chance of 1/8 of winning?H = -8 * 1/8 log(1/8) = 3 bits (maximum uncertainty)
3. Uniform: for N outcomes entropy maximized if all equally likely:
k
kk PPXIEXH )log()()(
NNN
NPPXHN
kkk log
1log
1)log()(
1
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 15
Entropy and Data CompressionEntropy and Data CompressionNote: Entropy is roughly the minimum average code length required for coding a random variable.
Idea: use short codes for likely outcomes and long codes for rare ones. Try to use every symbol of your alphabet equally often on average (e.g. Huffman coding described in Ballard book).
This is basis for data compression methods.
Example: consider horse race again: Probabilities of winning: 1/2, 1/4, 1/8, 1/16, 1/64, 1/64, 1/64, 1/64
Naïve code: 3 bit combination to indicate winner: 000, 001, …, 111
Better code: 0, 10, 110, 1110, 111100, 111101, 111110, 111111
requires on average just 2 bits (the entropy), 33% savings !
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 16
Differential EntropyDifferential EntropyIdea: generalize to continuous random variables described by pdf:
Notes:• differential entropy can be negative, in contrast to entropy of discrete random variable• but still: the smaller differential entropy, the “less random” is X
Example: uniform distribution
dxxpxpXH ))(log()()(
)log()log()(
otherwise ,0
0for ,)(
0
11
1
adxXH
axxp
a
aa
a
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 17
Maximum Entropy DistributionsMaximum Entropy DistributionsIdea: maximum entropy distributions are “most random”For discrete RV, uniform distribution has maximum entropy
For continuous RV, need to consider additional constraints on the distributions:
Neurons at very different ends of the visual system show the same exponentialdistribution of firing rates in response to “video watching” (Baddeley et al., 1997)
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 18
Why exponential distribution?
Exponential distribution maximizes entropy under the constraint of a fixedmean firing rate µ.
Maximum entropy principle: find density p(x) with maximum entropy thatsatisfies certain constraints, formulated as expectations of functions of x:
rrp exp1
)(
Interpretation of mean firing rate constraint: each spike incurs certain metaboliccosts: goal is to maximize transmitted information given a fixed average energyexpenditure
nicdxxFxp ii ,,1 , )()( Answer:
iii xFaAxp )(exp)(
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 19
ExamplesExamples
Two important results:1) for a fixed variance, the Gaussian distribution has the highest
entropy (another reason why the Gaussian is so special)
2) for a fixed mean and p(x)=0 if x ≤0, the exponential distribution has the highest entropy. Neurons in brain may have exponential firing rate distributions because this allows them to be most “informative” given a fixed average firing rate, which corresponds to a certain level of average energy consumption
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 20
Intrinsic PlasticityIntrinsic Plasticity
Not only the synapses are plastic! Now surge of evidence for adaptation of dendritic and somatic excitability. Typically associated with homeostasis ideas, e.g.:
• neuron tries to keep mean firing rate at desired level
• neuron keeps variance of firing rate at desired level
• neuron tries to attain particular distribution of firing rates
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 21
Question 2: how do intrinsic and synaptic learning processes interact?
Stemmler&Koch (1999): derived learning rule for two compartment Hodgkin-Huxleymodel:
Question: how do intrinsic conductance properties need to change to reach specific firing rate distribution?
where gi is the (peak) value of the i-th conductance
… of uniform firing rate distribution
adaptation of gi leads to learning...
Question 1: formulations of intrinsic plasticity for firing rate models?
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 22
Kullback Leibler DivergenceKullback Leibler Divergence
Idea: Consider you want to compare two probability distributions P and Q that are defined over the same set of outcomes.
k k
kkP xQ
xPxP
XQ
XPEQPD
)(
)(log)(
)(
)(log)||(
A “natural” way of defining a “distance” between two distributionsis the so-called Kullback-Leibler divergence (KL-distance),or relative entropy:
Pk Qk
1 2 3 4 5 6 1 2 3 4 5 6 unfair dice
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 23
D(P||Q) ≥ 0 and D(P||Q)=0 if and only if P=Q, i.e., if two distributionsare the same, their KL-divergence is zero otherwise it’s bigger.D(P||Q) in general is not equal to D(Q||P) (i.e. D(.||.) is not a metric)
The KL-divergence is a quantitative measure of how “alike” twoprobability distributions are.
k k
kkP xQ
xPxP
XQ
XPEQPD
)(
)(log)(
)(
)(log)||(
Properties of KL-divergence:
Generalization to continuous distributions:
dx
xq
xpxp
xq
xpExqxpD p )(
)(log)(
)(
)(log))(||)((
The same properties as above hold.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 24
Mutual InformationMutual Information
Goal: KL-divergence is quantitative measure of “alikeness” ofdistributions of two random variables. Can we find a quantitative measure of independence of two random variables?
Idea: recall definition of independence of two random variables X,Y:
We define as the mutual information the KL-divergence betweenthe joint distribution and the product of the marginal distributions:
Consider two random variables X and Y with a joint probability mass function P(x,y), and marginal probability mass functions P(x) and P(y).
)()()()(),(),( yPxPyYPxXPyYxXPyxP
x y yPxP
yxPyxPYPXPYXPDYXI
)()(
),(log),()()(||),();(
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 25
1. I(X;Y)≥0, equality if and only if X and Y are independent
2. I(X;Y) = I(Y;X) (symmetry)
3. I(X;X) = H(X), entropy is “self-information”
x y ypxp
yxpyxpYpXpYXpDYXI
)()(
),(log),()()(||),();(
Properties of Mutual Information:
Generalization to multiple RVs:
)()...()(||),...,,(),...,,( 212121 nnn XpXpXpXXXpDXXXI
),...,,(log),...,,(
where, ),...,,()(),...,,(
2121
211
21
nn
n
n
iin
XXXpEXXXH
XXXHXHXXXI
deviation fromindependence
savings inencoding
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 26
Mutual Information as Objective FunctionMutual Information as Objective FunctionHaykin
Note: trying to maximize or minimize MI in a neural network architecturesometimes leads to biologically implausible non-local learning rules
Linsker’sInfomax
Imax byBecker and
Hinton
ICA
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 27
Cocktail Party ProblemCocktail Party Problem
Motivation: cocktail party with many speakers as sound sources and array of microphones, where each microphone is picking up a different mixture of the speakers.
Question: can you “tease apart” the individual speakers just given x, i.e. without knowing how the speakers’ signals got mixed? (Blind source separation (BSS))
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 28
Blind Source Separation ExampleBlind Source Separation Example
Demos of blind audio source separation: http://www.cnl.salk.edu/~tewon/Blind/blind_audio.html
original time series of source s linear mixture x
unmixed signals:
)(1 ts
)(2 ts )(2 tx
)(1 tx
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 29
Definition of ICADefinition of ICANote: there are several, this is the simplest and most restrictive:• sources are independent and non-gaussian: s1, …, sn
• sources have zero mean• n observations are linear mixtures:
• the inverse of A exists:
Goal: find this inverse. Sine we do not know the original, can cannot compute it directly but we will have to estimate it:
• once we have our estimate, we can compute the sources:
)()( tt Asx
1AW
)(ˆ)(ˆ tt xWs
)(ˆ)( tt xWy
Relation of BSS and ICA:• ICA is one method of addressing BSS, but not the only one• BSS is not the only problem where ICA can be usefully applied
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 30
Restrictions of ICARestrictions of ICANeed to require: (it’s surprisingly little we have to require!)• sources are independent• sources are non-gaussian (at most one can be gaussian)
Ambiguities of solution:a) sources can be estimated only up to a constant scale factorb) may get permutation of the sources, i.e. sources may not be
recovered in their right order
b) switching these columns leaves x unchanged
a) multiplying source with constant anddividing that source’s matrix entries by same constant leaves x unchanged
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 31
Principles for Estimating Principles for Estimating W:W:
Several possible objectives (contrast functions):• requiring outputs yi to be uncorrelated is not sufficient• outputs yi should be maximally independent• outputs yi should be maximally non-gaussian, (central limit theorem), (projection pursuit)• maximum likelihood estimation
Algorithms:• various different algorithms based onthese different objectives with interestingrelationships between them• to evaluate criteria like independence (MI), need to make approximations since distributions unknown• typically, whitening is used as a pre-processing stage (reduces number of parameters that need to be estimated)
)(ˆ)( tt xWy
Projection Pursuit:find projection along which data looks most “interesting”
PCA
Projection Pursuit
y
W
x
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 32
Artifact Removal in MEG DataArtifact Removal in MEG Data
original MEG data
ICs found with ICA algorithm
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 33
ICA on Natural Image PatchesICA on Natural Image Patches
The basic idea:• consider 16 by 16 gray scale patches of natural scenes• what are the “independent sources” of these patches?
Result:• they look similar to V1 simple cells: localized, oriented, bandpass
Hypothesis: Finding ICs may be principle of sensory coding in cortex!
Extensions:stereo/color images, non-linear methods, topographic ICA
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 34
Discussion: Information TheoryDiscussion: Information Theory
Several uses:• Information theory is important for understanding sensory coding and information transmission in nervous systems
• Information theory is a starting point for developing new machine learning and signal processing techniques such as ICA
• Such techniques can in turn be useful for analyzing neuroscience data as seen in the EEG example
Caveat:• Typically difficult to derive biologically plausible learning rules from information theoretic principles. Beware of non-local learning rules.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 35
Some other limitationsSome other limitationsThe “standard approach” for understanding visual representations:The visual system tries to find efficient encoding of random natural image patches or video fragments that optimizes statistical criterion: sparseness, independence, temporal continuity or slowness
Argument 1: the visual cortex doesn’t get tosee random image patches but actively shapesthe statistics of its inputs
Argument 3: vision has to serve action: the brain may be more interested in representations thatallow efficient acting, predict rewards etc.
Is this enough for understanding higher level visual representations?
Yarbus (1950s): vision isactive and goal-directed
Argument 2: approaches based on trace rulesbreak down for discontinuous shifts caused bysaccades
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 36
Anthropomorphic robot head:• 9 degrees of freedom, dual 640x480 color images at 30 Hz
Building Embodied Computational Models Building Embodied Computational Models of Learning in the Visual Systemof Learning in the Visual System
Autonomous Learning:• unsupervised learning, reinforcement learning (active), innate biases
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 37
Simple illustration: clustering of image patches an “infant vision system” looked at
The biases, preferences, and goals of the developing system may be as important as the learning mechanism itself:
the standard approach needs to be augmented!
b: system learning from random image patchesc: system learning from “interesting” image patches (motion, color)
top related