lecture 12 equality and inequality constraints
DESCRIPTION
Lecture 12 Equality and Inequality Constraints. Syllabus. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/1.jpg)
Lecture 12
Equality and Inequality Constraints
![Page 2: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/2.jpg)
SyllabusLecture 01 Describing Inverse ProblemsLecture 02 Probability and Measurement Error, Part 1Lecture 03 Probability and Measurement Error, Part 2 Lecture 04 The L2 Norm and Simple Least SquaresLecture 05 A Priori Information and Weighted Least SquaredLecture 06 Resolution and Generalized InversesLecture 07 Backus-Gilbert Inverse and the Trade Off of Resolution and VarianceLecture 08 The Principle of Maximum LikelihoodLecture 09 Inexact TheoriesLecture 10 Nonuniqueness and Localized AveragesLecture 11 Vector Spaces and Singular Value DecompositionLecture 12 Equality and Inequality ConstraintsLecture 13 L1 , L∞ Norm Problems and Linear ProgrammingLecture 14 Nonlinear Problems: Grid and Monte Carlo Searches Lecture 15 Nonlinear Problems: Newton’s Method Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals Lecture 17 Factor AnalysisLecture 18 Varimax Factors, Empirical Orthogonal FunctionsLecture 19 Backus-Gilbert Theory for Continuous Problems; Radon’s ProblemLecture 20 Linear Operators and Their AdjointsLecture 21 Fréchet DerivativesLecture 22 Exemplary Inverse Problems, incl. Filter DesignLecture 23 Exemplary Inverse Problems, incl. Earthquake LocationLecture 24 Exemplary Inverse Problems, incl. Vibrational Problems
![Page 3: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/3.jpg)
Purpose of the Lecture
Review the Natural Solution and SVD
Apply SVD to other types of prior informationand to
equality constraints
Introduce Inequality Constraints and theNotion of Feasibility
Develop Solution Methods
Solve Exemplary Problems
![Page 4: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/4.jpg)
Part 1
Review the Natural Solutionand
SVD
![Page 5: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/5.jpg)
subspaces
model parameters mp can affect datam0 cannot affect data
data dp can be fit by modeld0 cannot be fit by any model
![Page 6: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/6.jpg)
natural solution
determine mp by solving dp-Gmp=0set m0=0
![Page 7: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/7.jpg)
natural solution
determine mp by solving dp-Gmp=0set m0=0
error reduced to its minimum E=e0Te0
![Page 8: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/8.jpg)
natural solution
determine mp by solving dp-Gmp=0set m0=0
solution length reduced to its minimum L=mpTmp
![Page 9: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/9.jpg)
Singular Value Decomposition (SVD)
![Page 10: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/10.jpg)
singular value decomposition
UTU=I and VTV=I
![Page 11: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/11.jpg)
suppose only p λ’s are non-zero
![Page 12: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/12.jpg)
suppose only p λ’s are non-zero
only first p columns of U
only first p columns of V
![Page 13: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/13.jpg)
UpTUp=I and Vp
TVp=Isince vectors mutually pependicular
and of unit length
UpUpT≠I and VpVp
T≠Isince vectors do not span entire space
![Page 14: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/14.jpg)
The Natural Solution
![Page 15: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/15.jpg)
The Natural Solution
natural generalized inverse G-g
![Page 16: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/16.jpg)
resolution and covariance
![Page 17: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/17.jpg)
Part 2
Application of SVD to other types of prior information
and toequality constraints
![Page 18: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/18.jpg)
general solution to linear inverse problem
![Page 19: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/19.jpg)
2 lectures ago
general minimum-error solution
![Page 20: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/20.jpg)
general minimum-error solution
natural solution
plus amount α of null vectors
![Page 21: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/21.jpg)
you can adjust α to match whatevera priori information you want
for examplem=<m> by minimizing L=||m-<m>||2 w.r.t. α
![Page 22: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/22.jpg)
you can adjust α to match whatevera priori information you want
for examplem=<m> by minimizing L=||m-<m>||2 w.r.t. α
get α =V0T<m> so m = VpΛp-1UpTd + V0V0T<m>
![Page 23: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/23.jpg)
equality constraints
minimize E with constraint Hm=h
![Page 24: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/24.jpg)
Step 1find part of solution constrained by Hm=h
SVD of H (not G)H = VpΛpUpTsom=VpΛp-1UpTh + V0α
![Page 25: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/25.jpg)
Step 2convert Gm=d
into and equation for α
GVpΛp-1UpTh + GV0α = dand rearrange
[GV0]α = [d - GVpΛp-1UpTh]G’α= d’
![Page 26: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/26.jpg)
Step 3solve G’α= d’
for αusing least squares
![Page 27: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/27.jpg)
Step 4 reconstruct m from αm=VpΛp-1UpTh + V0α
![Page 28: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/28.jpg)
Part 3
Inequality Constraints and theNotion of Feasibility
![Page 29: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/29.jpg)
Not all inequality constraints provide new information
x > 3x > 2
![Page 30: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/30.jpg)
Not all inequality constraints provide new information
x > 3x > 2
follows from first constraint
![Page 31: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/31.jpg)
Some inequality constraints are incompatible
x > 3x < 2
![Page 32: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/32.jpg)
Some inequality constraints are incompatible
x > 3x < 2
nothing can be both bigger than 3 and smaller than 2
![Page 33: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/33.jpg)
every row of the inequality constraint
Hm ≥ hdivides the space of m
into two parts
one where a solution is feasibleone where it is infeasible
the boundary is a planar surface
![Page 34: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/34.jpg)
when all the constraints are considered togetherthey either create a feasible volume
or they don’t
if they do, then the solution must be in that volume
if they don’t, then no solution exists
![Page 35: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/35.jpg)
(A) (B)
m1 m1
m2 m2
feasible region
![Page 36: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/36.jpg)
now consider the problem of minimizing the error E
subject to inequality constraints Hm ≥ h
![Page 37: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/37.jpg)
if the global minimum isinside the feasible region
then
the inequality constraintshave no effect on the solution
![Page 38: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/38.jpg)
butif the global minimum is
outside the feasible region
then
the solution is on the surfaceof the feasible volume
![Page 39: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/39.jpg)
butif the global minimum is
outside the feasible region
then
the solution is on the surfaceof the feasible volume
the point on the surface where E is the smallest
![Page 40: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/40.jpg)
Hm≥h
m1
m2
-Emest
Emininfeasible
feasible
![Page 41: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/41.jpg)
furthermore
the feasible-pointing normal to the surfacemust be parallel to ∇E
elseyou could slide the point along the surface
to reduce the error E
![Page 42: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/42.jpg)
Hm≥h
m1
m2
-Emest
Emin
![Page 43: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/43.jpg)
Kuhn – Tucker theorem
![Page 44: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/44.jpg)
it’s possible to find a vector y with yi≥0 such that
![Page 45: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/45.jpg)
it’s possible to find a vector y with y≥0 such thatfeasible-pointing normals
to surface
![Page 46: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/46.jpg)
it’s possible to find a vector y with y≥0 such that
the gradient of the error
feasible-pointing normals to surface
![Page 47: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/47.jpg)
it’s possible to find a vector y with y≥0 such that
the gradient of the error
is a non-negative combination of feasible normals
feasible-pointing normals to surface
![Page 48: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/48.jpg)
it’s possible to find a vector y with y≥0 such that
the gradient of the error
is a non-negative combination of feasible normals
feasible-pointing normals to surface
y specifies the combination
![Page 49: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/49.jpg)
it’s possible to find a vector y with y≥0 such that
for linear case with Gm=d
![Page 50: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/50.jpg)
it’s possible to find a vector y with y≥0 such that
some coefficients yi are positive
![Page 51: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/51.jpg)
it’s possible to find a vector y with y≥0 such that
some coefficients yi are positive
the solution is on the corresponding constraint
surface
![Page 52: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/52.jpg)
it’s possible to find a vector y with y≥0 such that
some coefficients yi are zero
![Page 53: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/53.jpg)
it’s possible to find a vector y with y≥0 such that
some coefficients yi are zero
the solution is on the feasible side of the
corresponding constraint surface
![Page 54: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/54.jpg)
Part 4
Solution Methods
![Page 55: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/55.jpg)
simplest case
minimize E subject to mi>0(H=I and h=0)
iterative algorithm with two nested loops
![Page 56: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/56.jpg)
Step 1
Start with an initial guess for mThe particular initial guess m=0 is feasible
It has all its elements in mE
constraints satisfied in the equality sense
![Page 57: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/57.jpg)
Step 2
Any model parameter mi in mE that has associated with it a negative gradient [∇E]i can be changed both to decrease the error and to remain feasible.
If there is no such model parameter in mE, the Kuhn – Tucker theorem indicates that this m is the solution to the problem.
![Page 58: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/58.jpg)
Step 3
If some model parameter mi in mE has a corresponding negative gradient, then the solution can be changed to decrease the prediction error.
To change the solution, we select the model parameter corresponding to the most negative gradient and move it to the set mS.
All the model parameters in mS are now recomputed by solving the system GSm’S=dS in the least squares sense. The subscript S on the matrix indicates that only the columns multiplying the model parameters in mS have been included in the calculation.
All the mE’s are still zero. If the new model parameters are all feasible, then we set m = m′ and return to Step 2.
![Page 59: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/59.jpg)
Step 4If some of the elements of m’S are infeasible, however,
we cannot use this vector as a new guess for the solution.
So, we compute the change in the solution and add as much of this vector as possible to the solution mS without causing the solution to become infeasible.
We therefore replace mS with the new guess mS + α δm, where is the largest choice that can be made without some mS becoming infeasible. At least one of the mSi’s has its constraint satisfied in the equality sense and must be moved back to mE. The process then returns to Step 3.
![Page 60: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/60.jpg)
In MatLab
mest = lsqnonneg(G,dobs);
![Page 61: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/61.jpg)
example
gravitational field depends upon density
via the inverse square law
![Page 62: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/62.jpg)
example
observations
gravitational force depends upon densitymodel
parameters
via the inverse square lawtheory
![Page 63: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/63.jpg)
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 30
50
100
150
200
y
dobs
(y)
(A)
(B)
(C)
(D)
(E)
(F)
(xi,yi)
(xj,yj)
grav
ity
forc
e, d i
distance, yi
θijR ij
![Page 64: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/64.jpg)
more complicated case
minimize ||m||2 subject to Hm≥h
![Page 65: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/65.jpg)
this problem is solved by transformation to the previous
problem
![Page 66: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/66.jpg)
solve by non-negative least squares
then compute mi as
with e’=d’-G’m’
![Page 67: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/67.jpg)
In MatLab
![Page 68: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/68.jpg)
m2
m1
m2
m1
m2
m1
m2
m1feasible
(A) m1-m2≥0 (B) m½ 1-m2≥1 (C) m1≥0.2 (D) Intersection
feasible feasible feasible
infeasible infeasible infeasible infeasible
mest
![Page 69: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/69.jpg)
yet more complicated case
minimize ||d-Gm||2 subject to Hm≥h
![Page 70: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/70.jpg)
this problem is solved by transformation to the previous
problem
![Page 71: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/71.jpg)
minimize ||m’|| subject to H’m’≥h’and
where
![Page 72: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/72.jpg)
m1
m2
m1
feasible
(A) (B) (C)
feasible
infeasible
2
1
0
2
1
0 1 2 m20 1 2
0 0.5 10
0.5
1
1.5
2
d
z
![Page 73: Lecture 12 Equality and Inequality Constraints](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815887550346895dc5e9e3/html5/thumbnails/73.jpg)
In MatLab[Up, Lp, Vp] = svd(G,0);lambda = diag(Lp);rlambda = 1./lambda;Lpi = diag(rlambda); % transformation 1Hp = -H*Vp*Lpi;hp = h + Hp*Up'*dobs; % transformation 2Gp = [Hp, hp]';dp = [zeros(1,length(Hp(1,:))), 1]';mpp = lsqnonneg(Gp,dp);ep = dp - Gp*mpp;mp = -ep(1:end-1)/ep(end); % take mp back to mmest = Vp*Lpi*(Up'*dobs-mp);dpre = G*mest;