a balanced approach to multichannel blind deconvolution

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: REGULAR PAPERS, VOL. 55, NO. 2, MARCH 2008 599

A Balanced Approach to MultichannelBlind Deconvolution

Ah Chung Tsoi and Liangsuo Ma

Abstract—In some general state-space approaches to the mul-tichannel blind deconvolution problem, e.g., the informationbackpropagation approach (Zhang and Cichocki 2000), an im-plicit assumption is usually involved therein, viz., the dimension ofthe state vector of the mixer is known a priori. In general, if thenumber of states in the state space is not known a priori, Zhangand Cichocki (2000) suggested using a maximum possible numberof states; this procedure will introduce additional delays in therecovered source signals. In this paper, our aim is to relax thisassumption. The objective is achieved by using balanced parame-terization of the underlying discrete-time dynamical system. Sincethere are no known balanced parameterization algorithms for dis-crete-time systems, we need to go through a “circuitous” route, byfirst transforming the discrete-time system into a continuous-timesystem using a bilinear transformation, perform the balanced pa-rameterization on the resulting continuous-time system, and thentransform the resulting system back to discrete-time balanced pa-rameterized system using an inverse bilinear transformation. Thenumber of states can be determined by the number of significantsingular values in the ensuing singular value decomposition stepin the balanced parameterization.

Index Terms—Balanced canonical form, blind deconvolution,blind source separation, independent component analysis (ICA),linear dynamical system.

I. INTRODUCTION

BLIND deconvolution or separation of source signals whichhad been mixed by a dynamical system has been exten-

sively studied in recent years [1]–[6]. In this paper we wish tostudy the multichannel blind deconvolution (MBD) problem ofthe following form:

(1)

(2)

where is the source signal, is the sensor outputof this linear time-invariant (LTI) system, is the stateof the dynamical system, is the number of states. Usually, wecall the LTI system described by (1) and (2) the mixer in theMBD context.

Note that here we will study the fully determined MBDproblem, in other words, we assume that there are as many

Manuscript received September 2, 2004; revised November 30, 2006. Thiswork was supported in part by the Australian Research Council Strategic Part-nership with Industry Research and Training (SPIRT) and in part by MotorolaAustralia Research Centre. This paper was recommended by B. C. Levy.

A. C. Tsoi is with Hong Kong Baptist University, Hong Kong (e-mail:[email protected]).

L. Ma was with the University of Iowa, Iowa City, IA 52246 USA. He is nowwith the University of Texas Health Science Center at Houston, Houston, TX77030 USA (e-mail: [email protected]).

Digital Object Identifier 10.1109/TCSI.2007.910752

sensors as there are the number of sources, both of them are. This assumption does not affect our presentation in this

paper because the general case where the sensor number isgreater than the source number, can be converted to the fullydetermined MBD problem, thus is solved using the algorithmdedicated to the fully determined MBD problem [7]. We do notconsider the relatively more complicated situation when thenumber of sensors is less than the number of sources in thispaper.

The problem can be stated as follows: given the sensor output, is it possible to recover the source signals . In general, this

is not possible. However, if we impose a number of assump-tions on the source signals then the answer becomes confirma-tive. The usual assumptions placed on the source signals are asfollows [2], [3].

1) The source signals are independent and identically dis-tributed (i.i.d) sequences.

2) At the most one of the source signals is Gaussiandistributed.

3) It is only possible to recover the source signals moduloscale invariance, and polarity.

Since we assume that the source signals are mixed by a dy-namical system, we need to place some further assumptions onthe dynamical system (mixer) itself. Common assumptions onthe dynamical systems [8] are as follows.

1) The dynamical system must be causal.2) The dynamical system is assumed to be time invariant.3) The dynamical system is assumed to be linear.In other words, in (1) and (2), the matrices , , , and are

assumed to be constant matrices. The problem then becomes:can we recover the source signals from the sensor measure-ments , assuming that the underlying mixer is a LTI system.This is commonly referred as the problem of blind separation/deconvolution. An associated question is: can we estimate theunknown constant parameter set ( )from the sensor measurements, if we assume that the numberof states is known. Note that using the LTI system [see (1)and (2)], the total number of parameters will be .

This problem set in a state-space formulation has been studiedby Zhang and Cichocki [5], who tackled the problem by de-vising the following LTI demixer with the parameter set

, assuming that the number of states in thedemixer is known

(3)

(4)

where and is the recovered signal vector,and is the state vector, is the number of states.

1549-8328/$25.00 © 2008 IEEE

600 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: REGULAR PAPERS, VOL. 55, NO. 2, MARCH 2008

To reduce the number of parameters to be estimated, they [5]have assumed a controller canonical form for the parameters

. This is a valid assumption as it is known thata LTI dynamical system is invariant under coordinate transfor-mations [8]. Hence, the general LTI system as expressed in (3)and (4) can be transformed into equivalent representations, e.g.,the controller canonical form. The controller canonical form isgiven as follows [8]:

(5)

(6)

(7)

(8)

where is an ma-trix, is an null matrix, and are,respectively, and identity ma-trices. , are constant matrices. Thereare a total of parameters. They have derived a pa-rameter estimation algorithm which they called an informationbackpropagation algorithm [5] to estimate the parameters ,

, and , using the mutual in-formation as a cost criterion.

However, there are a number of issues associated with the in-formation backpropagation MBD algorithm [5]; the main pointscan be summarized as follows.

• For successful separation and deconvolution of the sourcesignals, a necessary condition is the dimension of thedemixer is not less than that of the mixer; in other words,

. However, the information backpropagationalgorithm requires the dimension to be known a priori.Zhang and Cichocki [5] indicated that such value can beestimated using, e.g., Akaike information criterion (AIC)or the final prediction error (FPE) criterion [5]. They fur-ther indicated that if the dimension is over estimated thenit will introduce additional delays in the recovered signals.Such additional delays would not cause detrimental effectson the recovered signals.

• From system theory, it is known that the controller canon-ical form has some difficulties, in particular, when it is usedas a parameterization in system identification studies [9].These difficulties include the possibility of ill conditioning,i.e., the parameter estimation process may become unstabledue to pole-zero cancellation.

It is also known in system theory that the difficulties encoun-tered in the application of a controller canonical form can beovercome to a large extent by the use of balanced parameteri-zation [10]–[12], which is another possible parameterization ofLTI systems [8]. In this case, we need to assume that the LTIsystem is stable. This is a reasonable assumption in most prac-tical systems.

In this paper, our aim is to explore the possibility of using bal-anced parameterization of the LTI dynamical system in MBDproblem. Such method, if successful, will provide a means todetermine the number of states in the demixer. While this maynot seem to add much to the controller canonical form approach

as indicated by [5], theoretically, our approach gives a means todetermine the total number of states in the demixer, and prac-tically, it will be computationally more robust than the corre-sponding controller canonical form based algorithms.

The structure of the paper is as follows. In Section II, wewill give a brief introduction to the balanced parameterizationof linear systems. In Section III, we will give a derivation ofthe estimation algorithm of the unknown parameters in the bal-anced parameterized model, while in Section IV, we will givesome comparative results using the proposed technique and thetechnique developed by [5]. It is shown that using balanced pa-rameterization, the results appear to be more robust. Some con-clusions are drawn in Section V.

II. BALANCED PARAMETERIZATION OF LTI SYSTEMS

Canonical forms for LTI systems are attractive in the field ofsystem identification because they provide a state-space repre-sentation of the linear systems [13]. The most commonly usedcanonical forms are controller, observer, and balanced canonicalforms [10]. Among the three candidates, the first two are rela-tively simple. In addition, most noticeably in single input singleoutput systems, the parameters in the state mixing matrix in boththe controller and observer canonical forms are related in a verysimple manner to the coefficients of the denominator of the cor-responding transfer function representation. This provides a di-rect relationship between the controller (or observer) canonicalform state-space representation and the corresponding transferfunction representation. However, the controller canonical formdoes have some shortcomings; the most relevant one to our cur-rent studies is that the parameter set in the controller canonicalform which leads to a minimal1 system representation is verycomplicated [13].

Apart from controller and observer canonical forms, the bal-anced canonical form is another possible way to parameterizea LTI system. In [15], it is argued there are mainly two mo-tives for studying balanced parameterization of linear systems:1) their close relation to model reduction, thus closely relatedto robust control theory and 2) the potential usefulness of bal-anced parameterization to system identification. The balancedparameterization has the advantage that the parameter estima-tion process is less dependent on the correct initial choice ofmodel order. For continuous-time systems, the parameters ofa balanced realization change continuously when the systemorder is increased. The situation with discrete-time systems issimilar when the order is increased from the true (minimal)order [16].

1Minimal in the sense that the total number of parameters used in the repre-sentation is a minimum. It was shown in [9] that minimal representation is notnecessarily good for system identification studies. In general, it is convenient to“slightly” over parameterize the system for system identification studies, ratherthan in striving to use the minimal system representation. The balanced param-eterization to be introduced later in this paper presents one of a number possible“slight” over parameterizations of the system, so as to facilitate an easier systemidentification and parameter estimation process. It was shown by Overbeek andLjung [14] that “slight” over parameterization actually assists the parameter es-timation process. The intuition is that by allowing a few more parameters thanabsolutely necessary, it gives some extra degrees of freedom to the parameterestimation process so as to overcome some of the “ravines” in the estimated pa-rameter landscape. Without these extra parameters, the minimal representationhas some difficulties to overcome these “ravines” computationally.

TSOI AND MA: BALANCED APPROACH TO MBD 601

Consider a discrete LTI system described in (3) and (4). Ifthe system is asymptotically stable, the controllability Gram-mian and the observability Grammian , both ma-trices, are, respectively, given by the following dual discrete-time Lyapunov equations:

(9)

(10)

Discrete-time balanced realization is defined when, where is a

diagonal matrix, and denotes the diag-onal values of a matrix. The quantities , arecalled the Hankel singular values [8], [9]. Since linear systemsare invariant under a coordinate transformation, this impliesthat in general it is possible to find a coordinate transformation

such that a general LTI system can be transformed into abalanced realization.

In a similar fashion, it is possible to define a balancedrealization of continuous LTI systems. In this case, con-sider a continuous-time LTI system with the parameter set

(11)

(12)

where , , and . The controllabilityGrammian and observability Grammian , bothmatrices, are, respectively, given by the solution to the dual Lya-punov equations as follows:

(13)

(14)

The system is said to be balanced if.

Note that in both the continuous-time case and the discrete-time case, the balanced realization is defined through the simul-taneous diagonalization of the controllability Grammian andthe observability Grammian. If the system or

are given, this can be performed quite readily.However, if the system is not known and needs to be estimatedfrom input-output data, then it is quite difficult to obtain the un-known system as well as satisfying the dual Lyapunov equationswith a diagonal matrix. As a result, a number of researchersproposed balanced parameterization models [10]–[12]. Theseare parameterized models which will yield a balanced realizedstate-space model, i.e., it allows us to find the particular set ofparameters from the given input output data which will give adiagonal matrix solution to the dual Lyapunov equations. Thebalanced parameterization is formulated only for the continuousLTI systems [11], [12]. There is no corresponding balanced pa-rameterization for discrete LTI systems.

A major motivation for balanced realization is model reduc-tion. If we make further assumptions on the Hankel singularvalues such that , and that isrelatively small, or , then it is possible to show thatthe reduced order model of dimension is asymptotically stableand minimal [9]. In addition, it can be demonstrated that the be-haviour of the reduced order model “approximates” that of the

original LTI system [9]. Taking advantage of these two points,the estimation of the number of the states ( ) can be performedas follows. Define , where , if thereexists a minimal value , satisfying the condition of

(15)

where is a positive constant close to 1, then is consideredas an estimate of the true number of the states ( ). Note, thequalification of the estimation is dependent on the value of .The closer is to 1, the more accurate is the estimation.

In this paper, we will not give the most general balanced pa-rameterization model for continuous LTI systems. This is givenin [10]. Instead we will give a balanced parameterization used in[12], which overcomes a number of limitations, e.g., the need tosatisfy a set of nonlinear constraints, of the parameterized modelgiven in [10].

Consider a linear continuous-time invariant system withstates, and inputs and outputs, parameterized by the set ofparameters , where [12]

, the set of singular values,satisfying , and ;

, and is an columnvector, its element ;

, and is row vectorwith real values, the first element in the row is positive;

real matrix.

Note that here we have taken a simplifying assumption: allthe Hankel singular values are distinct. This assumption simpli-fies a number of the structural parameters, which are usually as-sociated with multiple Hankel singular values [12]. In practice,this assumption is usually satisfied in practical systems, as thereare always numerical errors which prevent one Hankel singularvalue to be exactly the same as another Hankel singular value.

The state-space matrices can be obtainedfrom this parameter set as follows [12]:

for

is an matrix

where the th column of is constructed as

(16)

and the th column vector of is given as follows:

(17)

where the details of , for , are given as


...

(17a)

The elements of the matrix is given by the following[12]:

(18a)

(18b)

Under the above parameterization, the continuous LTIsystem with parameter set is balanced with Grammian

. Observe that in (17), is con-structed through an indirect vector , which is used toovercome the problem that is required to be quadratic insome general balanced forms, e.g., [10]. This is the mainadvantage of the balanced form that was introduced in [12]compared with the one introduced in [10].

This is the parameterization of the continuous LTI system.There are a total of

parameters. Note that the number of unknown parametersis more than the corresponding ones in the controller canonicalform. Intuitively speaking, it is these extra parameters whichallow the parameter estimation algorithm to perform better thanthe parameterization using a controller canonical form.

As indicated, the parameterization is derived from thebalanced realization techniques. It has internal constraints,for instance, the satisfaction of the dual Lyapunov equationswhich arises from controllability and observability studies.These equations must be internally satisfied. If we tackle theLyapunov equations directly, this will result in a constrainedoptimization problem, which may not have a solution. Oneway in which we can avoid the formulation of the constrainedoptimization problem is to transform the balanced parameter-ized form into the general state-space formulation as indicatedpreviously. In other words, we will parameterize a generalcontinuous LTI system in state-space form in terms of theparameterization .

III. DERIVATION OF PARAMETER ESTIMATION ALGORITHM

It is noted that the balanced parameterization is given only interms of a continuous LTI system. However, the sensor outputsare assumed to be in discrete time, as a result, we will need toconvert the continuous-time model using a bilinear transforma-tion into a corresponding discrete LTI model. This will permitus to use the measurement data to estimate the parameters of thecorresponding discrete LTI model.

The parameters of the discrete LTI system can be estimatedusing the general parameter estimation algorithms provided by[5].

Now once the parameters are updated using the algorithmsgiven in [5], we will need to perform the inverse bilinear trans-formation to convert it into a corresponding continuous LTIsystem setting. Through the relationship between the continuous

LTI system parameters and the balanced parameterization, it ispossible to obtain a set of new parameter values for the balancedparameterization which can be used to update them. Once theparameters are updated, the cycle can begin again until the al-gorithm converges.

The question one wish to ask is: why do we need to go throughthis circuitous route in order to obtain an updating algorithmfor the parameters of the balanced parameterization form. Theanswer to this question lies in two aspects.

1) The development of the theory on system identificationusing balanced parameterization is only available in thecontinuous-time setting. The corresponding theory ofsystem identification using discrete-time balanced param-eterization is not yet developed. This is the reason why weneed to perform the bilinear transformations so that wecan make use of the input output data which is assumed tobe available in discrete time.

2) Had we tackled the parameter estimation problem of thebalanced realization directly, we will need to satisfy thedual Lyapunov equations at every step. Developing suchan algorithm is a challenge, as it is not immediately clearhow this can be performed.

Hence we have chosen this rather circuitous route to obtainthe parameter estimation algorithm.

A general structure of the parameter estimation can be de-scribed as follows.

Step 1) From the set of parameters , obtain the contin-uous-time balanced parameterization.

Step 2) Use bilinear transformation to transform this con-tinuous LTI system into the corresponding discrete-time parameterization.

Step 3) Estimate the new parameters for the discrete-timeparameterization.

Step 4) Convert this new set of parameters for discrete-timeparameterization back to continuous-time settingusing the inverse bilinear transformation.

Step 5) Convert the new parameters in compatible form tothe balanced parameterization.

Step 6) Cycle through Steps 1) to 5) until convergence.The first step was addressed in Section II already, in the fol-

lowing subsections, we will consider the remaining steps of theparameter estimation procedures.

A. Converting a Continuous-Time System to Discrete-TimeSystem Using a Bilinear Transformation

Construct a set of discrete LTI system from the set ofcontinuous LTI system using the following standard bilineartransformation [8]:

(19a)

(19b)

(19c)

(19d)

B. Updating of Parameters of the Discrete-Time System

Since the parameterization of the discrete LTI system is gen-eral, we can use the algorithm provided in [5] for their update.In this subsection, we will briefly indicate how the parameterestimation algorithm is derived.


The algorithm is derived by minimizing the following costcriterion [5]:

(20)

where , is the determinantof the matrix , is an approximation of the probabilitydistribution function of th source signal.

From the cost function (20) we can easily obtain the followingderivatives:

(21)

(22)

where is a vector of nonlinear activation functions relatedto the source distribution, the th element of can be de-scribed as

(23)

From (21) and (22), it is possible to obtain the following gra-dient-based updating rules for and [5]:

(24)

(25)

Note the learning rate and are functions of time, andthey are not necessary identical. Also note the natural gradienttechnique [1] is used in the updating rule of to improve theperformance of the learning process.

The updating rules for , and are relatively complicated,as we need to work through the state equations. From the costfunction and the state equation, we can obtain [5]

(26)

Applying chain rule, we have [5]

(27)

(28)

where and can be obtained as follows[5]:

(29)

(30)

for , and . is the Kro-necker delta function. Using above results, we can obtain the

following update equations for , the elements of , andthe elements of :

(31)

(32)

where is the th column of matrix .The (24), (25), (31) and (32) allow us to update the values of

the elements ( ) of the parameter set .

C. Converting Discrete-Time System to Continuous-TimeSystem

We obtain new values of the elements ( ) of theparameter set through the inverse bilinear transformation asfollows:

(33a)

(33b)

(33c)

(33d)

D. Updating the Parameter Set

In the parameter set , the matrices and are alreadyobtained. Hence, we only need to work out how to update theparameters and .

From the construction of matrix , i.e., (16), (17) and (17a),we can obtain

(34)To proceed, we need the following definition for

:

(35)

Combining (34) and (35), we have the following relation:

(36)

where we have defined

(37)

(38)

(39)


.... . .

...

(40)

To express with and in (42), we require thatexists. In other words, . Observe

, hence we need to satisfy for all . Fromthe definition of , we know that this is equivalent to satis-fying . In other words, we need

(41)

In Section II, we defined ; this is to guar-antee that (41) is satisfied.

Finally, we have the following result:

(42)

Note, to obtain , we only need to consider the 2-nd to thcolumn of matrix . Also note, in (42), we need to computean inverse matrix , however it is trivial because is anupper triangular matrix.

For the parameters , a similar derivation can be obtained.Consider the diagonal elements of matrix in (18), we caneasily obtain

(43)

combining the following chain rule:

(44)

we obtain the following result:

(45)

Once the relationships (42) and (45) are established, the up-dates of and can be performed accordingly as follows:

(46)

(47)

where , is a reasonably selected positive timeduration.

To summarize, the details of the proposed discrete-time bal-anced MBD algorithm are given in Algorithm 1.

Algorithm 1 Discrete-time balanced MBD algorithm

1: Initialize the values of the unknown parameter set

2: Construct the parameter set of a continuous LTI systemfrom the parameter set using (16), (17), (17a), and (18)

3: while do

4: Construct the parameter set of a corresponding discreteLTI system using the bilinear transformation (19)

5: Learn the elements ( , , , and ) of the parameterset using the parameter updating rules (24), (25), (31), and(32)

6: Obtain updates of the parameter set using the inversebilinear transformation (33)

7: Obtain new values of the parameter set

8: Obtain new values of the balanced parameterization fromthe new values of

9: Calculate the update of the cost function ( )

10: end while

E. Remarks

• The proposed algorithm consists of two parts. The first partis the balanced parameterization, which keeps the demixera balanced dynamical LTI system. The second part is theinformation backpropagation algorithm, the task of whichis to achieve the objective of MBD. These two parts work incontinuous-time domain and discrete domain, respectively.They are linked by the bilinear transformations.

• The learning rules (24) and (25) in the general algorithmare required to satisfy some stability conditions: a set ofsuch stability conditions is given in [5]. Another concern ishow to keep the demixer stable during the learning process.This may be a problem when the learning rule (31) is usedto update the matrix . There is no general proof that thisis globally stable. However, the following heuristic appearsto ensure that the updating algorithm stays within the sta-bility region [22]. At each step of the iteration, compute theeigenvalues of the estimated value of the matrix . If any ofthe eigenvalues are outside the stability region, then reflectthem inside the stability region, and use these as the ini-tial values for the next iterated values. On the other hand,if all the eigenvalues are within the stability region, thenthese will be used as the initial values for the next iteration.Using such a heuristic appears to alleviate any stability is-sues related to the learning algorithm. Once the values ofthe estimated parameters are close to the converged values,then, the algorithm will converge, as the balanced param-eterized LTI system is stable in the neighborhood of con-vergence. This innate stability may be viewed as one of theadvantages of the proposed algorithm.

• In [6], a filter decomposition method is proposed to solvethe MBD problem when the mixer is assumed to be a non-minimum-phase system. The advantage of this approach isthat a doubly finite-impulse response (FIR) filter can be de-composed into a cascaded form of a causal FIR filter andan anticausal FIR filter. This approach is different from theone proposed in this paper, in that we are more interestedin obtaining the number of states in a state-space formu-lation, and then use such estimated value in the estima-tion of the parameters of the demixer, while [6] is moreinterested in finding a set of FIR filters which can recoverthe sources. Superficially the two methods are not related


as they use quite different approaches. One notes howeverthat our proposed approach as it stands cannot be appliedto the non-minimum phase mixer MBD problem. This isbecause the information propagation algorithm has an im-plicit assumption of a minimum phase mixer, as it essen-tially attempts to find an inverse of the mixer. However,the balanced parameterization process does not have animplicit assumption of non-minimum phase. Hence, oneway in which the proposed method may be modified tohandle a non-minimum phase mixer would be to com-bine the balanced parameterization approach with the de-composition of the doubly FIR filter approach proposedin [6] as follows: use the balanced parameterization tofind the balanced realization form, and use the doubly FIRfilter approach to estimate the unknown parameters. How-ever, whether this suggestion will work or not is yet to bestudied. This is suggested as an area for future studies.

IV. EXPERIMENTAL RESULTS

In this section, we will illustrate the issues discussed inthis paper through computer simulations. In Section IV-Awe will give two performance measurement indices, viz. themean-squared error (MSE) and the average mutual information,both will be used to evaluate the performance of different algo-rithms. In Section IV-B we will examine the proposed balancedalgorithm using a set of artificial mixed speech observations.In particular, we will compare the performance of the proposedbalanced algorithm with the general state-space algorithm [5];then we will examine the algorithm’s ability of estimatingthe number of states of the mixer; the motivation of the pro-posed balanced algorithm is demonstrated by the improvedperformance of the general state-space algorithm [5] withappropriate number of states. In Section IV-C, we will show apossible way of applying the proposed balanced algorithm toa set of real sensor measurements, viz. electroencephalograph(EEG) data. We will compare the performance of the balancedalgorithm with a public domain Matlab toolbox for EEG signalprocessing; using the estimated number of states of the mixer( ), we will show that, after is obtained, the average mutualinformation given by the general state-space algorithm [5] isless than that given by the EEG toolbox. These indicate thatthe proposed balanced parameterization method work well inpractice.

A. Performance Measurements

In this paper, we compare different MBD algorithms to eval-uate their relative performances using some performance in-dices. There exist many possible performance indexes [17], [18]in the literature. We choose the MSE criterion from such set ofpossible candidates. The MSE criterion evaluates the similaritybetween the waveforms of the source signals and their recov-ered version. Given samples of -channel source signalsand their recovered signals , the MSE criterion is defined asfollows [19]:

for

The MSE requires the source signals to be available, however,generally, this cannot be satisfied in practice. Since the objectiveof MBD is usually accomplished through minimizing the mu-tual information (MI) between the outputs of the demixer, there-fore mutual information is a valid criterion for measuring theperformance of MBD algorithms. Normally mutual informationis difficult to compute. Moddemeijer [20] provided a practicalapproach to compute the mutual information between two sig-nals. In this paper, we also adopt mutual information as our per-formance index. Moddemeijer’s approach [20] only allows tocompute the mutual information between two waveforms, andin our case we need to compute the average mutual informa-tion for the situation where the source number is . Giventhe number of sources , the average mutual information(AMI) can be defined as follows:

where is the absolute value of the mutual informationbetween the th signal and the th signal.

B. Speech Data

The observation data used in this subsection is obtained bypassing 10, 000 samples of three digital speech source signalsthrough a stable discrete-time LTI dynamical system with in-ternal states. The parameters of the mixer is chosen such that:1) exists; 2) the eigenvalues of the are all within theunit circle in the complex plane and 3) and are randomlyselected. Note, we will use speech data to evaluate MBD algo-rithms in this subsection, however, speech signals are generallynot identical and independent distributed in nature. The prepro-cessing method proposed in [21] is used to avoid this problem.

1) Performance Comparison: First, we will compare the pro-posed balanced MBD algorithm with the general discrete-timestate-space MBD algorithm, i.e., the information backpropa-gation approach [5], under the following structure parameters:

, . The details of the four chosen systemmatrices of the mixer are given as follows:

The original signals and the mixed signals are shown inFig. 1(a) and (b), respectively. In both algorithms, we employthe hyperbolic tangent as the nonlinear activation function. Thenonlinearity is a valid choice because the sources are speechsignals, which are normally super-Gaussian distributed. Inboth algorithms, we use a demixer with a sufficiently highnumber of states ( ) to separate the signals. We run thetwo algorithms ten times with different initial conditions each


Fig. 1. (a) Original source signals. (b) Mixed signals. (c) Signals recovered from the general MBD algorithm [5] with the estimated number of states (N = 2).(d) Signals recovered from the proposed MBD algorithm with the estimated number of states (N = 2).

time. The overall MSE performance of the two algorithms in aparticular run is given in Fig. 2.

The comparison of MSE criterion and the mutual infor-mation criterion are given in Fig. 3. From Figs. 2 and 3 weobserve that, in the sense of MSE and mutual information,the performance of the two algorithms are comparable forthis simple MBD problem, although the proposed balancedparameterization algorithm performs slightly better. However,regarding the speed of convergence, the proposed algorithmis much faster.

2) Determining the Dimension of States: Next, we will esti-mate the number of states of the mixer using the proposed bal-anced parameterization algorithm. We assume that the numberof states ( ) of the mixer is unknown, but we believe that thecurrent number of states in the demixer satisfies the conditionof . We consider five situations: to 5, re-spectively. In each situation, we run the proposed algorithm tentimes with different initial conditions each time. In this simula-tion, we choose , i.e., the reduced order system con-tains at least 80% of the total energy of the original system.


Fig. 2. Iterative curves of MSE: (a) given by information backpropagation algorithm [5] and (b) given by the proposed balanced parameterization algorithm inthis paper.

Fig. 3. Comparison of the MSEs and the average mutual information between the general MBD algorithm (N = 6), the proposed algorithm (N = 6), theproposed MBD algorithm (N = 2) and the general MBD algorithm (N = 2). Note: The unit of theY axis is 10 in Subplot (a). “Propose”denotes the proposedalgorithm, “General” denotes the general algorithm [5], ’O’ denotes that an algorithm with an overestimated number of states, and “E” denotes that an algorithmwith the estimated number of states from the proposed algorithm in this paper. (a) MSE. (b) AMI.

The average values of are listed in Table I. As an example,the evolution of the singular values for is shown inFig. 4. From Table I and Fig. 4, we observe that the proposeddiscrete-time balanced parameterization algorithm identifies thecorrect number of states quite well.

Further, to compare with the algorithms with the overesti-mated number of states ( ), we run the general MBD al-gorithm [5] and the proposed MBD algorithm ten times, eachtime with different initial conditions, after the number of states( ) is obtained. The MSE criterion and the mutual infor-mation criterion are shown in Fig. 3. The recovered signals in aparticular run are plotted in Fig. 1(c) and (d), respectively. FromFig. 3 we observe that, once the number of states is estimated,the performance of both the general MBD algorithm [5] and theproposed algorithm is improved compared to the cases of over-estimated number of states.

In summary, there are a number of observations which can bemade from this experiment.

• The performance of the proposed algorithm and the in-formation backpropagation algorithm [5] are quite similar,though on an average the proposed algorithm has a slightedge over the information backpropagation algorithm interms of MSEs, and average mutual information measures.This is irrespective if the number of states of the mixer isknown or not.

• The proposed method can determine the number of statesof the mixer.

• The performance of both the proposed method and theinformation backpropagation algorithm are better (as mea-sured by the MSE and average mutual information mea-sures) if the number of mixer states is known.


Fig. 4. Evolution of singular values in the proposed MBD algorithm with assumed state number N = 6.

Fig. 5. Original seven channels of EEG recordings.

TABLE IMEAN VALUES OF AVERAGE MUTUAL INFORMATION FOR

DIFFERENT VALUES OF N

C. EEG Data

In this section, we will show the application of our proposedmethodology to a class of biometric signals, viz., electroen-

cephalograph (EEG) signals. When applying independentcomponent analysis (ICA) or MBD to processing the biomed-ical signals, a practical difficulty is the lack of ideal performanceindexes to measure the performance of the algorithm as theoriginal source signals are not available for comparisons. Thismakes it difficult to evaluate the algorithms or interpret therecovered signals. In this paper, we use average mutual infor-mation criterion as an alternative. This measures the averagemutual information in the recovered outputs. Thus, the onewith the lowest average mutual information will be consideredto provide the ‘best’ recovered signals. However, such indirectinference, though necessary, is tenuous at best.


Fig. 6. Comparison of the average mutual information between ICA, the proposed algorithm (N = 6), the general MBD algorithm (N = 3) and the proposedMBD algorithm (N = 3). Note: The unit of the Y axis is 10 . ’Propose’ denotes the proposed algorithm, “General” denotes the general algorithm [5], “O”denotes that an algorithm with an overestimated number of states, and “E” denotes that an algorithm with the estimated number of states from the proposedalgorithm in this paper.

Fig. 7. Recovered EEG signals obtained from the proposed MBD algorithm for N = 6.

The 2500 samples (10 s) of the seven-channel EEG data (seeFig. 5) were recorded by Z. Keirn at Purdue University usinga sampling frequency of 250 Hz.2 To realize our objective, weassume that the given set of observation data is generated by amixer with seven inputs and seven outputs. The number of states

2The EEG recordings and the corresponding information can be found at thewebsite of http://www.cs.colostate.edu/~anderson/res/eeg/.

of the mixer is unknown, which needs to be estimated. We as-sume that is less than 6, with different initial parameters, werun the proposed balanced parameterization algorithm with thenumber of states ten times. The AMIs are given in Fig. 6.The recovered signals and the evolution of the singular values ina particular run are shown in Figs. 7 and 8, respectively. To eval-uate the performance of the proposed balanced parameterization


Fig. 8. Evolution of singular values in the proposed MBD algorithm with an assumed number of states N = 6.

Fig. 9. Recovered EEG signals obtained by using the extended infomax ICA algorithm in EEG toolbox 3.61.

algorithm, we employ the EEG toolbox 3.613 as a basis of com-parison. Similarly, we run the ICA algorithm in the toolbox tentimes, the corresponding average mutual informations demon-strated in Fig. 6. The recovered signals obtained from the EEGtoolbox in a particular run are plotted in Fig. 9. From Fig. 6,we observe that the mean average mutual information given bythe balanced parameterization algorithm is slightly smaller thanthat given by the EEG toolbox. From Fig. 8, we observe that the

3The EEG toolbox was released by S. Makeig et. al. in CNL/Salk Institute,US. It can be downloaded from the world wide web at http://www.cnl.salk.edu/~scott/icabib.html.

proposed balanced parameterization algorithm draw the conclu-sion that the number of states is 3.

To examine the validity of the above estimation, we run thegeneral state-space algorithm [5] and the proposed algorithm tentimes, both with the number of states . The average mu-tual informations are depicted in Fig. 6. The recovered signalsin a particular run are plotted in Figs. 10 and 11, respectively.From Fig. 6, we observe that, compared to the EEG toolbox andthe MBD algorithms with a higher number of states, the meanaverage mutual informations given by the MBD algorithms withestimated number of states are smaller.


Fig. 10. Recovered EEG signals obtained from the general MBD algorithm for N = 3.

Fig. 11. Recovered EEG signals obtained from the proposed MBD algorithm for N = 3.

This experiment confirms that the proposed algorithm can es-timate the total number of states in the mixer. Armed with thisinformation, it is shown that the information backpropagationalgorithm, and our proposed algorithm both perform well, betterthan the one using an ICA algorithm (with no dynamics assumedin the mixer) again our proposed algorithm performs slightlybetter than the information backpropagation algorithm as mea-sured by the average mutual information criterion.

V. CONCLUSION

In this paper, we have considered the possibility of obtaininga balanced realization of the deconvolution of latent source sig-nals mixed by unknown LTI dynamical systems. Based on thework [5] and [11], [12], we have derived a discrete-time bal-anced MBD algorithm. The contributions of this paper are asfollows: 1) this is the first time that a balanced parameteriza-tion approach is applied to the MBD problem; 2) we have suc-


cessfully eliminated one of the assumptions in the general state-space MBD algorithms, viz., we do not require the number ofstates to be known a priori. Instead, the number of states canbe obtained as part of our parameter estimation procedure. Allwe need to do is to impose an upper bound on the number ofdemixer states, and then our approach will be able to estimatethe number of states. Note that this is not an online algorithm, inthat the algorithm will find the “correct” number of states online.One way in which this algorithm can be applied is to imposea sufficiently high upper bound on the total number of states.Use the proposed algorithm on the set of input output data. Theappropriate value of the total number of states will emerge bytracking the converged Hankel singular values and by decidingon which ones can be neglected. Then this estimated number ofstates can be used either in the general information backpropa-gation algorithm [5] or the proposed balanced parameterizationalgorithm. This will yield an improved result. It is noted that onboth examples shown in the experimental section, our proposedmethod performs slightly better than the information backprop-agation algorithm as measured by the MSEs, and average mu-tual information criterion.

The advantages of the proposed algorithm over the infor-mation backpropagation algorithm proposed in [5] are as fol-lows: 1) the proposed algorithm gives a numerically robust al-gorithm for the determination of the unknown parameters and2) it allows a practical method for the determination of thenumber of states. Compared with the information backprop-agation algorithm, the proposed method is numerically moreexpensive, as it requires the computation of singular values ineach time step. An interesting future work is to alleviate thisproblem by exploring more efficient methods for performingthe singular value decomposition, or by using more efficientrobust numerical computational methods, e.g., QR decomposi-tion. The outcomes of MBD algorithms are generally identicaland independent distributed. Therefore, another interesting fu-ture work is to test the proposed algorithm using data with i.i.ddistribution.

ACKNOWLEDGMENT

The authors acknowledge useful discussions withDr. A. Madievski, Motorola Australian Research Centre,leading to much improved results; useful discussions withDr. J. Xi and Dr. C. T. Chou in the course of this research; andconstructive comments of a number of anonymous reviewersin improving the readability of this paper. The work reportedin this paper was performed while both authors were with Fac-ulty of Informatics, University of Wollongong, Wollongong,Australia.

REFERENCES

[1] S. Amari, “Natural gradient working efficiently in learnin,” NeuralComput., vol. 10, no. 2, pp. 251–276, Feb. 1998.

[2] A. J. Bell and T. J. Sejnowski, “An information maximization approachto blind separation and blind deconvolution,” Neural Comput., vol. 7,no. 6, pp. 1129–1159, Nov. 1995.

[3] A. Cichocki and S. Amari, Adaptive Blind Signal and Image Pro-cessing: Learning Algorithms and Applications. West Sussex, U.K.:Wiley, 2003.

[4] K. Waheed and F. M. Salam, “Blind source recovery: A framework inthe state space,” J. Mach. Learning Res., vol. 4, pp. 1411–1446, Dec.2003.

[5] L. Zhang and A. Cichocki, “Blind deconvolution of dynamical systems:A state-space approach,” J. Signal Process., vol. 4, no. 2, pp. 111–130,2000.

[6] L. Zhang, A. Cichocki, and S. Amari, “Multichannel blind de-convolution of nonminimum-phase systems using filter decomposi-tion,” IEEE Trans. Signal Process., vol. 52, no. 5, pp. 1430–1442,May 2004.

[7] L. Zhang, S. Amari, and A. Cichocki, “Natural gradient approach toblind separation of over- and under-complete mixtures,” in Proc. 1stInt. Workshop Independent Compon. Anal. Signal Separation, 1999,pp. 455–460.

[8] T. Kailath, Linear Systems. Upper Saddle River, NJ: Prentice-Hall,1980.

[9] K. Glover, “All optimal Hankel norm approximations of linear multi-variable systems and L error bounds,” Int. J. Contr., vol. 39, no. 6,pp. 1115–1193, 1984.

[10] R. Ober, “Balanced parametrization of classes of linear systems,” SIAMJ. Contr. Optim., vol. 29, no. 6, pp. 1251–1287, 1991.

[11] C. T. Chou, “Geometry of Linear Systems and Identification,” Ph.D.dissertation, Dept. Elect. Eng., Cambridge Univ., Cambridge, U.K.,1994.

[12] C. T. Chou and J. M. Maciejowski, “System identification using bal-anced parametrizations,” IEEE Trans. Autom. Contr., vol. 42, no. 7, pp.956–974, Jul. 1997.

[13] R. Ober, “Balanced canonical forms (Lectures given at the NATO-ASIschool, from identification to learning held in como italy Aug. 1994),”in Identification, Adaptation, Learning. New York: Springer-Verlag,1996.

[14] A. J. M. van Overbeek and L. Liung, “On-line structure selectionfor multivariable state-space models,” Automatica, vol. 18, no. 5, pp.529–543, Sep. 1982.

[15] B. Hanzon and R. J. Ober, “Overlapping block-balanced canonicalforms and parametrizations: The stable SISO case,” SIAM J. Contr.Optim., vol. 35, no. 1, pp. 228–242, 1997.

[16] J. M. Maciejowski and R. J. Ober, “Balanced parameterizations andcanonical forms for system identification,” IFAC Identification andSystem Parameter Estimation, pp. 701–708, 1988.

[17] A. K. Nandi, Blind Estimation Using Higher Order Statistics. Nor-well, MA: Kluwer Academic, 1999.

[18] K. Waheed and F. M. Salam, “A data-derived quadratic independencemeasure for adaptive blind source recovery in practical applications,”in Proc. 45th IEEE Midwest Symp. Circuits Syst., 2002, vol. 3, pp.473–476.

[19] R. A. Choudrey, “Variational methods for Bayesian independent com-ponent analysis,” Ph.D., Dept. Elect. Eng., Univ. of Oxford, Oxford,U.K., 2002.

[20] R. Moddemeijer, “On estimation of entropy and mutual information ofcontinuous distributions,” Signal Process., vol. 16, no. 3, pp. 233–246,1989.

[21] J. Solé-Casals and E. Monte-Moreno, “Source separation techniquesapplied to blind deconvolution of real world signals,” in Proc. ISCA Tu-torial and Research Workshop on Non-Linear Speech Processing (NO-LISP), Barcelona, Spain, 2005.

[22] G. C. Goodwin and K. S. Sin, Adaptive Filtering Prediction and Con-trol. Englewood Cliffs, NJ: Prentice-Hall, 1984.

Ah Chung Tsoi received the higher diploma in elec-tronic engineering from the Hong Kong TechnicalCollege, Hong Kong, in 1969, and the M.Sc. degreein electronic control engineering, and Ph.D. degreein control engineering from the University of Sal-ford, Salford, U.K., in 1970 and 1972, respectively.

Since graduation, he worked as a PostdoctoralFellow at the Inter-University Institute of Engi-neering Control, University College of North Wales,Bangor, U.K., a Lecturer at Paisley College ofTechnology, Paisley, U.K., before immigrating to


New Zealand. He worked as a Senior Lecturer in Electrical Engineering inthe Department of Electrical Engineering, University of Auckland, Auckland,New Zealand, before immigrating to Australia. He worked as a Senior Lecturerin Electrical Engineering, University College University of New South Wales,Canberra, Australia, for five years. He then served as Professor of ElectricalEngineering at University of Queensland, Queensland, Australia; Dean, andsimultaneously Director of Information Technology Services, and then foun-dation Pro-Vice Chancellor (Information Technology and Communications)at University of Wollongong, Wollongong, Australia; before joining the Aus-tralian Research Council as an Executive Director, Mathematics, Informationand Communications Sciences. He was Director, Monash e-Research Centre,Monash University, Melbourne, Australia. In April 2007, he took up the posi-tion of Vice President (Research and Institutional Advancement), Hong KongBaptist University, Hong Kong. He works in the area of artificial intelligence inparticular neural networks and fuzzy systems in recent years. He has publishedin neural network literature. In more recent years, he has been working in theapplication of neural networks to graph domains, with applications to the worldwide web searching, and ranking problems, sub-graph matching problem.

Prof. Tsoi is a Fellow of the Institute of Engineers and Technologists, U.K.He is also a Fellow of the Australian Institute of Company Directors, Australia.

Dr. Liangsuo Ma received the B.E. degree in elec-trical engineering from Tsinghua University, Beijing,China, in 1995, and the M.E. and Ph.D. degrees inelectrical, computer telecommunication engineeringfrom University of Wollongong, Wollongong,, Aus-tralia in 2000 and 2005, respectively.

From December 2004 to December 2007, he wasa Postdoctoral Research Fellow at the University ofIowa, Iowa City. Since January 2008, he has beenwith the University of Texas Health Science Centerat Houston.

a balanced approach to multichannel blind deconvolution

Documents