a new method for searching an l1 solution of an...

LAD ProblemBasic properties

New methodAplication

A new method for searching an L1 solution of anoverdetermined system of linear equations and

applications

Goran Kusec1 Ivana Kuzmanovic2 Kristian Sabo2

Rudolf Scitovski2

1Faculty of Agriculture, University of Osijek2Department of Mathematics, University of Osijek

Kusec, Kuzmanovic, Sabo, Scitovski A new method for searching an L1 solution of an overdetermined system of linear equations and applications



LAD Problem

A ∈ Rm×n, (m� n), b ∈ Rm

A =

? ? . . . ?? ? . . . ?? ? . . . ?? ? . . . ?...

......

......

......

...? ? . . . ?? ? . . . ?? ? . . . ?? ? . . . ?

︸︷︷︸

, b =

????......????

m

n




system of linear equations Ax = b

b ∈ R(A)

b R(A)




overdetermined system of linear equations Ax ≈ b

b /∈ R(A)

b

R(A)




‖b− Ax‖ → minx∈Rn

Euclidean l2-norm (Least Square (LS) solution)l1-norm (Least absolute deviation (LAD) solution)l∞-norm (Tschebishev solution)

applications in various fields of applied researchJ. A. Cadzow, Minimum l1, l2 and l∞ Norm Approximate Solutions to an Overdetermined System

of Linear Equations, Digital Signal Processing 12(2002) 524–560

S. Dasgupta, S. K. Mishra, Least absolute deviation estimation of linear econometric models: A

literature review, Munich Personal RePEc Archive, 1-26, 2004

C. Gurwitz, Weighted median algorithms for L1 approximation, BIT 30(1990) 301–310

Dodge Y. (Ed.), Statistical Data Analysis Based on the L1-norm and Related Methods, Proceedingsof The Thrid International Conference on Statistical Data Analysis Based on the L1-norm andRelated Methods, Elsevier, Neuchatel, 1997




Vector x∗ ∈ Rn such that

minx∈Rn‖b− Ax‖1 = ‖b− Ax∗‖1

is called the best LAD-solution of overdetermined system Ax ≈ b.

operational research literature - a parameter estimationproblem of a hyperplane on the basis of a given set of points

N. M. Korneenko, H. Martini, Hyperplane approximation and related topics, in: New Trends inDiscrete and Computational Geometry, (J. Pach, Ed.), Springer-Verlag, Berlin, 1993.

A. Schobel, Locating Lines and Hyperplanes: Theory and Algorithms, Springer Verlag, Berlin, 1999.

statistics literature - a parameter estimation problem for linearregression

S. Dasgupta, S. K. Mishra, Least absolute deviation estimation of linear econometric models: A

literature review, Munich Personal RePEc Archive, 1-26, 2004

E. Z. Demidenko, Optimization and Regression, Nauka, Moscow, 1989, in Russian

Dodge Y. (Ed.) (1997): Statistical Data Analysis Based on the L1-norm and Related Methods,Proceedings of The Thrid International Conference on Statistical Data Analysis Based on theL1-norm and Related Methods, Elsevier, Neuchatel




principle is considered to have been proposed by the Croatianmathematician J. R. Boskovic in the mid-eighteenth century

D. Birkes, Y. Dodge, Alternative Methods of Regression, Wiley, New York, 1993

P. Bloomfield, W. Steiger, Least Absolute Deviations: Theory, Applications, and Algorithms,Birkhauser, Boston, 1983

Dodge Y. (Ed.) (1997): Statistical Data Analysis Based on the L1-norm and Related Methods,Proceedings of The Thrid International Conference on Statistical Data Analysis Based on theL1-norm and Related Methods, Elsevier, Neuchatel

Numerical methodsclassical nondifferentiable minimization methods cannot beapplied directly-unreasonably long computing timespecialized algorithms

J. A. Cadzow, Minimum l1, l2 and l∞ Norm Approximate Solutions to an Overdetermined System of Linear

Equations, Digital Signal Processing 12(2002) 524–560

E. Castillo, R. Mınguez, C. Castillo, A. S. Cofinno, Dealing with the multiplicity of solutions of the l1 and l∞regression models, European J. Oper. Res. 188(2008), 460–484.

Y. Li, A maximum likelihood approach to least absolute deviation regression, EURASIP Journal on Applied

Signal Processing 12(2004), 1762–1769

N. T. Trendafilov, G. A. Watson, The l1 oblique procrustes problem, Statistics and Computing 14(2004)

39–51




LAD problem ⇔

On the basis of the given set of points

Λ = {Ti (x(i)1 , . . . , x

(i)n−1, z

(i)) ∈ Rn : i ∈ I}, I = {1, . . . ,m}, vectora∗ = (a∗0, a

∗1, . . . , a

∗n−1) ∈ Rn of optimal parameters of hyperplane

f (x; a) = a0 +∑n−1

j=1 ajxj should be determined such that:

G (a∗) = mina∈Rn

G (a), G (a) =m∑

i=1

∣∣∣∣∣∣z(i) − a0 −n−1∑j=1

ajx(i)j

∣∣∣∣∣∣ .




n = 2

Data-points Λ = {(x (i)1 , z(i)), i = 1, . . . ,m}

LAD line z = a∗0 + a∗1x1

min(a0,a1)∈R2

m∑i=1

|z(i) − a0 − a1x(i)1 | =

m∑i=1

|z(i) − a∗0 − a∗1x(i)1 |




n = 2

Data-points Λ = {(x (i)1 , z(i)), i = 1, . . . ,m}

LAD line z = a∗0 + a∗1x

min(a0,a1)∈R2

m∑i=1

|z(i) − a0 − a1x(i)1 | =

m∑i=1

|z(i) − a∗0 − a∗1x(i)1 |




LAD solution always exists

N. M. Korneenko, H. Martini, Hyperplane approximation and related topics, in: New Trends in Discrete andComputational Geometry, (J. Pach, Ed.), Springer-Verlag, Berlin, 1993.

A. Pinkus, On L1-approximation, Cambridge University Press, New York, 1993.

G. A. Watson, Approximation Theory and Numerical Methods, John Wiley & Sons, Chichester, 1980.




LAD solution is generally not unique




n=2, LAD solution is not unique

0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1




Definition

Matrix A ∈ Rm×n (m ≥ n) is said to satisfy the Haar condition ifevery n × n submatrix is nonsingular.




Theorem 1

Let A ∈ Rm×n, m > n, be a matrix of full column rank andb = (b1, . . . , bm)T ∈ Rm a given vector. Then there exists a permutationmatrix Π ∈ Rm×m such that

ΠA =

[A1

A2

], Πb =

[b1

b2

], A1 ∈ Rn×n, A2 ∈ R(m−n)×n, b1 ∈ Rn, b2 ∈ Rm−n,

whereby A1 is a nonsingular matrix, and there exists a LAD-solution x∗ ∈ Rn

such that A1x∗ = b1.Furthermore, if matrix [A b] satisfies the Haar condition, then the solutionx∗ ∈ Rn of the system A1x = b1 is a solution of LAD problem if and only ifvector

v = (A1T )−1A2

T s, si := sign (b2 − A2x∗)i

satisfies ‖v‖∞ ≤ 1.

Furthermore, x∗ is a unique solution of LAD problem if and only if ‖v‖∞ < 1.

A. Pinkus, On L1-approximation, Cambridge University Press, New York, 1993.

G. A. Watson, Approximation Theory and Numerical Methods, John Wiley & Sons, Chichester, 1980.




parameter estimation problem of a hyperplane for a set ofpoints in a plane

n = 2 for the given set of points in a plane, with condition thatnot all of them lie on some line parallel to the second axis,there exists the best LAD-line passing through at least twodifferent data pointsn = 3 for the given set of points in the space, with conditionthat not all of them lie on some plane parallel to the third axis,there exists the best LAD-plane passing through at least threedifferent data points

Y. Li, A maximum likelihood approach to least absolute deviation regression, EURASIP Journal on Applied

Signal Processing 12(2004), 1762–1769

R. Scitovski, K. Sabo, I. Kuzmanovic, I. Vazler, R. Cupec, R. Grbic, Three points method for searching the

best least absolute deviations plane, 4th Croatian Mathematical Congress, Osijek, June 17 – 20, 2008

A. Schobel, Locating Lines and Hyperplanes: Theory and Algorithms, Springer Verlag, Berlin, 1999.




New method

I. Kuzmanovic, K. Sabo, R. Scitovski, Method for searching the best least absolute deviations solution of an

overdetermined system of linear equations, submitted




numerical methods for searching the best LAD-solution:Hooke and Jeeves Method, Differential Evolution, NelderMead, Random Search, Simulated Annealing

J. A. Nelder, R. Mead, A simplex method for function minimization, Comput. J. 7(1965)

308–313

C. T. Kelley, Iterative Methods for Optimization, SIAM, Philadelphia, 1999.

various specializations of the Gauss-Newton method

E. Z. Demidenko, Optimization and Regression, Nauka, Moscow, 1989, in Russia

M. R. Osborne, Finite Algorithms in Optimization and Data Analysis, Wiley, Chichester,1985.

Linear Programming methods:

A. Barrodale, F. D. K. Roberts, An improved algorithm for discrete l1 linear approximation,

SIAM J. Numer. Anal. 10(1973) 839–848

N. M. Korneenko, H. Martini, Hyperplane approximation and related topics, in: New Trendsin Discrete and Computational Geometry, (J. Pach, Ed.), Springer-Verlag, Berlin, 1993.

some of specialized methods: Barrodale-Roberts,Bartels-Conn-Sinclair, Bloomfield-Steiger

N. T. Trendafilov, G. A. Watson, The l1 oblique procrustes problem, Statistics and

Computing 14(2004) 39–51

P. Bloomfield, W. Steiger, Least Absolute Deviations: Theory, Applications, and Algorithms,Birkhauser, Boston, 1983

Y. Dodge, Statistical Data Analysis Based on the L1-norm and Related Methods,Proceedings of The Thrid International Conference on Statistical Data Analysis Based onthe L1-norm and Related Methods, Elsevier, Neuchatel, 1997




Lemma 1

Let y1 ≤ y2 ≤ . . . ≤ ym be real numbers with corresponding weights wi > 0,W :=

∑mi=1 wi , and I = {1, . . . ,m}, m ≥ 2 a set of indices. Denote

ν0 =

{max

{ν ∈ I :

∑νi=1 wi ≤ W

2

}, if

{ν ∈ I :

∑νi=1 wi ≤ W

2

}6= ∅

0, otherwise

Furthermore, let F : R→ R be a function defined by the formulaF (α) =

∑mi=1 wi |yi − α|. Then

(i) if∑ν0

i=1 wi <W2

, then the minimum of the function F is attained at thepoint α? = yν0+1;

(ii) if∑ν0

i=1 wi = W2

, then the minimum of the function F is attained at everypoint α? from the segment [yν0 , yν0+1].




Lemma 2

Let x0 be a unique solution of the system Ax = b, where A ∈ Rn×n is a squarenonsingular matrix and b ∈ Rn a given vector.If the j-th equation of this system aT

j x = bj is replaced with equation uT x = β,whereby uT dj 6= 0, where dj ∈ Rn is the j-th column of matrix A−1, then thesolution of a new system is given by

x = x0 + αdj , α =β − uT x0

uT dj.




Definition

Vector x∗Jk∈ Rn, Jk = {i1, . . . , ik} ⊆ I , k ≤ n is called the best

Jk -LAD-solution of problem

minx∈Rn‖b− Ax‖1 = ‖b− Ax∗‖1

if on x∗Jkthe minimum of functional x 7→ ‖r(x)‖1 is attained with

conditionri (x∗Jk

) := bi − (Ax)i = 0, i ∈ Jk .

the best Jn-LAD-solution is obtained by solving the system oflinear equations

ri (x) = 0, i = i1, . . . , in,

the best Jn−1-LAD-solution is obtained as a solution of oneweighted median problem.




Lemma 3

Let A ∈ Rm×n , (m > n) be a matrix of full column rank, b ∈ Rm a given vector and e = (1, . . . ,m)T a vector

of indices. Furthermore, let Π ∈ Rm×m be a permutation matrix such that

ΠA =

[A−1A+

2

]=

[A11 a1k A12A21 a2k A22

], Πb =

[b−1b+

2

],

where A11 ∈ R(n−1)×(k−1), A12 ∈ R(n−1)×(n−k), A21 ∈ R(m−n+1)×(k−1), A22 ∈ R(m−n+1)×(n−k),

a1k , b−1 ∈ Rn−1, a2k , b+2 ∈ Rm−n+1, whereby index k is such that A1k := [A11 A12] is a nonsingular matrix.

Let A2k := [A21 A22], I− = {(Πe)k : k = 1, . . . , n − 1} and I + = {(Πe)k : k = n, n + 1, . . . ,m}.

Then there exists i0 ∈ I + and the best Jn-LAD-solution x∗ ∈ Rn , such that ri (x∗) = 0 for all i ∈ I− ∪ {i0}.The k-th component x∗k of vector x∗ is thereby a solution of the weighted median problem

‖A2k η − b+2 − α(A2k ξ − a2k )‖1 → min

α, (1)

whereby ξ, i.e. η, is a unique solution of systems

A1k ξ = a1k , i.e. A1k η = b−1 , (2)

whereas other components of vector x∗ are obtained by solving the system

A1k (x1, . . . , xk−1, xk+1, . . . , xn)T = b−1 − x∗k a1k . (3)




matrices A and [A b] - satisfy the Haar condition.




Step 0 (Input) m, n, e = (1, . . . ,m)T , A ∈ Rm×n, b ∈ Rm , Π0 ∈ Rm×m .

Step 1 (Defining the first submatrix) Determine index k such that A1k := [A11 A12] ∈ R(n−1)×(n−1) is anonsingular matrix, where

H0 := Π0A =

[A−1A+

2

]=

[A11 a1k A12A21 a2k A22

], Π0b =

[b−1b+

2

].

Set I + = {(Π0e)i : i = n, n + 1, . . . ,m} and e := Π0e.

Step 2 (Searching for a new equation) According to Lemma 3, determine i0 ∈ I + and solution x(0). Calculate

G0 = G(x(0)) := ‖b− Ax(0)‖1

Let A1 =

[A−1aT

i0

]∈ Rn×n , b1 =

[b−1bi0

]∈ Rn .

Furthermore, let A2 ∈ R(m−n)×n be a matrix obtained from A+2 by dropping the i0-th row, b2 ∈ Rm−n

a vector obtained from b+2 by dropping the i0-th component

Solve system

AT1 v = AT

2 s, si := sign (b2 − A2x)i , i = 1, . . . , n,

and denote the solution by v(0)

Determine j0 such that |v (0)j0| = max

i=1,...,n|v (0)

i | =: vM

If vM ≤ 1, STOP; otherwise go to Step 3

Step 3 (Equations exchange) Let Π1 ∈ Rm×m be a permutation matrix which in matrix H0 replaces the j0-th

row with the i0-th row. Determine index k such that A1k := [A11 A12] ∈ R(n−1)×(n−1) is a nonsingularmatrix, where

H1 := Π1H0 =

[A−1A+

2

]=

[A11 a1k A12A21 a2k A22

], Π1b =

[b−1b+

2

],

Set I + = {(Π1e)i : i = n, n + 1, . . . ,m} and e := Π1e.




Step 4 (Searching for a new equation) According to Lemma 2, determine i0 ∈ I + and solution x(1). Calculate

G1 = G(x(1)) := ‖b− Ax(1)‖1;

Step 5 (Preparation for a new loop) Let A1 =

[A−1aT

i0

]∈ Rn×n , b1 =

[b−1bi0

]∈ Rn .

Furthermore, let A2 ∈ R(m−n)×n be a matrix obtained from A+2 by dropping the i0-th row, b2 ∈ Rm−n

a vector obtained from b+2 by dropping the i0-th component.

Solve system

AT1 v = AT

2 s, si := sign (b2 − A2x)i , i = 1, . . . , n,

and denote the solution by v(1)

Determine j0 such that |v (1)j0| = max

i=1,...,n|v (1)

i | =: vM

If vM ≤ 1, STOP; otherwise put H0 = H1 and go to Step 3




Initialization

The initial matrix A−1matrix A satisfies the Haar conditionthe first (n − 1) rows of matrix AΠ0 = I- identity matrix

A nonsingular submatrix A1k of matrix

H0 := Π0A =

[A−1A+

2

]=

[A11 a1k A12

A21 a2k A22

]in Step 1 - by applying QR factorization with column pivoting

A−1 Π := [A11 a1k A12] Π = Q[R ρ],

Π ∈ Rn×n - permutation matrixQ ∈ R(n−1)×(n−1) - an orthogonal matrixR ∈ R(n−1)×(n−1) - an upper triangular nonsingular matrixρ ∈ R(n−1)




A1k := [A11 A12] = Q R ΠT1 , a1k = Qρ,

Π1 is the matrix - from Π by dropping the kth row and thenth column.




Theorem 2

Let x be approximation of LAD-problem obtained in Step 2, i.e. Step 4, of the

Algorithm as a solution of the system A1x = b1, where A1 =

[A−1aT

i0

]∈ Rn×n,

b1 =

[b−1bi0

]∈ Rn. Furthermore, let A2 ∈ R(m−n)×n be a matrix obtained from

A+2 by dropping the i0-th row, b2 ∈ Rm−n a vector obtained from b+

2 bydropping the i0-th component, and v ∈ Rn a solution of the system

AT1 v = AT

2 s, si := sign (b2 − A2x)i , i = 1, . . . , n.

Denote|vj0 | = max

i=1,...,n|vi |.

I. If |vj0 | ≤ 1, then x is the best LAD-solution;

II. If |vj0 | > 1, then j0 6= n, and maximum decreasing of minimizing functionvalues is attained such that in Step 3 the j0-th row of matrix A1 isreplaced.




Convergence Theorem 4

Let A ∈ Rm×n be a matrix and b ∈ Rm a given vector, m ≥ n,such that matrices A and [A b] satisfy the Haar condition. Thensequence (x(n)) defined by the iterative procedure in the Algorithmconverges in finitely many steps to the best LAD-solution.




Example 1

Table 1.

i A b1 42 7 -28 34 13 -49 62 31 46 -31 -12 17 18 -423 13 -11 -1 -35 31 -33 484 45 -35 43 -9 -36 -8 -165 25 -34 37 -30 -2 32 136 -37 -24 -9 -35 -35 21 -167 -12 26 -49 -49 -11 -22 -328 -42 31 28 -5 -45 47 -289 19 2 -29 -33 0 -46 35

10 -21 -26 -45 -44 5 26 4711 28 26 -38 -39 -8 -16 -612 32 27 -7 9 -43 -22 -38

G0 = 236.346Kusec, Kuzmanovic, Sabo, Scitovski A new method for searching an L1 solution of an overdetermined system of linear equations and applications



Example 1

Table 1.

i A b1 42 7 -28 34 13 -49 62 31 46 -31 -12 17 18 -423 13 -11 -1 -35 31 -33 484 45 -35 43 -9 -36 -8 -165 25 -34 37 -30 -2 32 136 -37 -24 -9 -35 -35 21 -167 -12 26 -49 -49 -11 -22 -328 -42 31 28 -5 -45 47 -289 19 2 -29 -33 0 -46 35

10 -21 -26 -45 -44 5 26 4711 28 26 -38 -39 -8 -16 -612 32 27 -7 9 -43 -22 -38




Application

Data-points Ti = (x(i)1 , x

(i)2 , z(i)), i = 1, . . . ,m

x(i)1 -fat thickness

x(i)2 -lumbar muscle thickness

z (i)- muscle percent of the i th pig

z = a0 + a1x1 + a2x2

outliers




Root-Mean-Square Error of Prediction (RMSEP)

D. Causeur, G. Daumas, T. Dhorne, B. Engel, M. Font, I. Furnols, S. Hojsgaard, Statistical handbook forassessing pig classification methods: Recommendations from the EUPIGCLASS project group

EU reference method

P. Walstra, G. S. M. Merkus, Procedure for assessment of the lean meat percentage as a consequence of thenew EU reference dissection method in pig carcass classification, DLO-Research Institute for Animal Scienceand Health (ID - DLO). Research Branch Zeist, P.O. Box 501, 3700 AM Zeist, The Netherlands, 1995




Example 2 m=145

EU reference method - RMSEP

a∗2 = 0.71259, a∗1 = 0.02312, a∗0 = 67.77137

LAD method

a∗2 = 0.696471, a∗1 = 0.0176915, a∗0 = 67.8906


a new method for searching an l1 solution of an...

Documents