Rong Guo and Jimin Ye

(School of Mathematics and Statistics,Xidian University,Xi’an 710071,China)

Abstract:Independent component analysis (ICA) can reveal the essential underlying structure of data, and independent component regression (ICR) methods usually obtain better performance than other regression methods such as principal component regression. However, when existing ICR methods separate or extract independent components using prewhitened data, the backward propagation of inevitable prewhitened errors deteriorates the final linear prediction accuracy. To overcome this weakness, first, we proposed using weighted orthogonal constraint condition to replace the prewhitening of the data in ICA. Next, the statistical independence of ICs and the close relationship between ICs and quality variables are considered at the same time. Then, by combining the merits of improved ICR and ensemble ICR algorithm which solved the problem of selecting an appropriate nonquadratic function in ICA iteration procedure, a modified independent component regression (MICR) method that directly used the measured process data was proposed. Finally, three experimental results were used to validate excellent performance of modified algorithm.

Keywords:independent component analysis; weighted orthogonal constraint; independent component regression; prewhitened data

1 Introduction

In recent years, regression analyses had been applied more and more widely[1-3]. Common regression analysis includes: principal component regression (PCR), partial least squares regression (PLSR) and multiple linear regression (MLR). However, because of the high dimension and linear correlation between process variables and quality variable, it is hard to deal with data in MLR, and neither PCR nor PLSR will recover true latent variables. Additionally, PCR/PLSR can only tackle the first and second moment of data, ignoring higher moment. ICA can recover independent source signals from mixed signals. Independence was a condition that can reveal the essential underlying data structure and made full use of higher-order moments information[4-6].

Because of the advantages of ICA, ICR was developed for regression in recent years[6-9]. Although the conventional ICR model is meaningful for regression analysis, it still has some disadvantages. First, the random initialization of the demixing matrix can bring about different optimization solutions, which might generate uncertain model results. Second, there is no particular criterion to determine which nonquadratic function used in ICA algorithms is optimal. Third, even if the extracted ICs are independent of each other, they may not carry the information of the quality variable and do not contribute to predictions and interpretations. If we employ all separated ICs to interpret the quality variables, it will lead to complex modeling and overfitting easily in regression analysis. Lee and Qin[10]solved the first problem with modified ICA (MICA). MICA replaces the random initialization condition of the original ICA with fixing initialization condition to produce a specific ICA solution. With respect to the second weakness, the ensemble modified ICR (EMICR) method presented by Tong[11-12]built an ensemble model that combined the results of multiple regression models with three different nonquadratic functions. This integrated method possesses better performance than a single-model method. A dual-objective cost function is built by Zhao[13]to solve the third issue. Nevertheless, the above approaches apply a whitening procedure before separating the independent components.

Whitening procedures remove second-order dependence from dataXand make it easier to solve the separation problem. Generally, the whitening error cannot be avoided and will reduce the accuracy of model. When some source signals are weak or the mixing matrix is ill-conditioned, this phenomenon will become more serious.

To overcome above issues, substituting the prewhitening step in ICA with weighted orthogonal constraint, combining the advantages of the improved ICR algorithm raised by Zhao[13]with those of the EMICR algorithm suggested by Tong[11], a modified independent component regression (MICR) method was put forward, which estimated the regression coefficients from measure data directly, and had better performance.

2 Theory

2.1ICRMethodBasedonPrewhitenedData

2.1.1ConventionalICRmethod

z=Vx

(1)

whereVis the whitening matrix. The relationship between these variables is given by:

z=VAs+e

(2)

(3)

(4)

(5)

The regression coefficient can be obtained by the least square method as:

(6)

(7)

2.1.2EnsemblemodifiedICRmethod

In EMICR method, the cost function is based on the following approximation of negentropy[5]:

J=(E{G(y)}-E{G(v)})2

(8)

wherev～N(0,1), the EMICR model aims to build three ICR models base on three different nonquadratic functions instead employs a single ICR model, Hyv?rinen et al.[5]suggested three nonquadratic functions listed as follows:

(9)

(10)

wherei=1,2,3, and the regression coefficient (according to theith reconstructed latent variables) is:

(11)

(12)

Therefore, from Eqs.(7), (11) and (12), the prediction of quality variableyis:

(13)

Then, combine the three regression formulas as a whole,

(14)

wheregiis obtained by optimizing the following Eq. (15):

(15)

Note that all of the existing ICR methods use data whiteningzto separate the latent variabless. Prewhitening process will lead to error accumulation, and may deteriorate prediction accuracy, particularly when an online whitening process is used.

2.2 Modified ICR Method Based on Measured Data

In both conventional ICRs and ensemble modified ICRs, latent variables are separated or extracted from prewhitened process variables. It is well known that prewhitening will introduce errors, and the backward propagation of those errors will reduce the prediction accuracy.

Replacing the prewhitening process with weighted orthogonal constraint condition on the separating matrix or extracting vectors, the latent variables can be directly separated or extracted from the measured process data[15], the error propagation is avoid. Additionally, in order to make the latent variables meaningful with respect to the quality variables, negentropy and the covariance between ICs and quality variables are simultaneously adopted for the iterative extraction of ICs. We propose the following dual-objective cost function based on measured process data:

(16)

whereαandβare the weight coefficients of each sub-optimization, and are defined to satisfyα+β=1.Gis a nonquadratic function defined by Eq.(9). Because ICs are extracted individually, the weighted orthogonal constraint on extracting vectors

(17)

is used. Wherewidenotes the extraction vector of theith ICs. Simple calculation yields the gradient and Hessian matrix ofJwith respect tow

(18)

(19)

wheregandg′are respectively the first-order and second-order derivative ofG. Using Newton’s algorithm to solve the cost function (16), and applying a generalized Gram-Schmidt algorithm to implement the weighted orthogonal constraint on extracting vectors, the following iteration process is obtained:

(20)

(21)

(22)

(23)

(24)

(25)

Replacing prewhitening with weighted orthogonal constraint, error backward propagation is avoided. Using the covariance between ICs and quality variables, the extracted latent variables become meaningful to the quality variables. The solution produced by an ensemble approach is more effective and robust. We term the ICR method mentioned above as the modified ICR (MICR).

For the selection of optimized parametersαandβin Eq.(16), special criteria cannot be given easily which depend heavily on current practical applications. When prior knowledge is unavailable, the weight parameters can be determined by cross-validation and try and error method. Ifα>β, the correlation between ICs and quality variable is mainly considered. Ifα<β, the statistical independence of ICs is more important than the correlation between ICs and quality variable.Note, the prediction error is the most important factor in regression, generally,αis set bigger thanβ.

3 Experimental Results and Analysis

In this chapter, we use three examples from different fields to verify the performance of the raised MICR approach. A rooted mean squared error (RMSE) index is employed to determine prediction accuracy:

(26)

3.1 Numerical Data

Consider three source signals with the following distribution:

s1(i)=3cos(0.02i)sin(0.09i)

(27)

s2(i)=sin(0.3i)+2cos(0.6i)

(28)

s3(i)=[rem(i,30)-13]/9

(29)

The rem function of Eq.(29) returns a remainder after division, while the mixing signals are generated by linear modelx=As+e. The mixing matrixAis defined as:

(30)

Noiseeis a Gaussian variable of zero mean and standard deviationσ={0.1,0.5,1}. Settingy=3s1+s2+2s3, whereyis the quality variable applied to regression analysis.The coefficients ofs1,s2, ands3can be randomly selected, and generally, the size of them are different, which means that the contribution of each IC to the quality variable is different. We can also employ two or one source signal to generatey. In MICR, the IC that does not contribute to the quality variable is extracted in the end, and will not be used as an explanatory variable. Even if it is used, from Eq. (25), we know that the estimated coefficient of it equals zero.

Choose one variance value (σ=0.1) of noise as an example. A total of 1 000 mixes are generated for model training, and 1 000 testing samples are generated for model prediction. A plot of the source signals can be seen in Fig.1. Then, we use the numerical data to carry out the MICR method and conventional ICR with differentGfunctions, and MICR method with EMICR method, using both measured and prewhitening data.

Fig.1 Source signals

Noteαshould be bigger thenβ. Through try and error method we setα=0.7 andβ=0.3. Fig.2 and Fig.3 describe the prediction results of the quality variable when selecting different number of ICs.

Fig.2 RMSE values vs. numbers of ICs(MICR and conventional ICR with different function G)

Obviously, Fig.2 and Fig.3 illustrate that the MICR algorithm possesses better performance than conventional ICR and EMICR. It is easy to explain: in MICR, the error propagation is avoided because of dropping the prewhitening process, thus, the regression accuracy is improved, and MICR retains the merits of EMICR and improved ICR.

Fig.3 RMSE values vs. numbers of ICs(MICR and EMICR using whitening and measured data)

3.2 TE Process Data

Tennessee Eastman (TE) process is a realistic industrial process[16]proposed by the Eastman Chemical Company. In the past few decades, the TE process has wide application in process control and detection[17-19]. The process is composed of a reactor, a condenser, a vapor-liquid separator, a recycle compressor, and a stripper, and measures 52 process variables in total. The process adopts 33 continuous variables with remaining 19 variables being composition measurements which are sampled less frequently. Table 1 lists some of the variables.

In this case, the first 33 variables in Table 1 are the input variables and the last one is the output variable. We got the simulation datasets at http://web.mit.edu/braatzgroup/, and divided it into two parts: a training dataset contains 480 samples and a test dataset contains 960 samples. Suppose that the sampling intervals for input and output were 3 min and 6 min, respectively. The sub-optimization parameters were set to beα=0.8 andβ=0.2.The selection ofαandβis similar to the method used in chapter 2.2.

The quality prediction results (using the RMSE index) from the MICR algorithm can be seen in Fig.4. Clearly, the proposed MICR algorithm yields a lower RMSE value than EMICR that uses prewhitening data and measured data. Additionally, only approximately 15 ICs require the use of regression analysis, successfully simplifying the model and improving system performance. The description of the actual value and the predicted value of the quality variable are shown in Fig.5, which gives a visual impression of the quality variable and prediction. We can see that the predicted values are very close to the real values.

Table 1 Parts of variables for the TE process

Fig.4 RMSE values vs. numbers of ICs(MICR and EMICR using whitening and measured data)

Fig.5 The comparison of quality variable and prediction

3.3 Fetal Electrocardiogram Data(FECG)

In present case study, to test the performance of the MICR method, we measured eight-channel cutaneous potential recordings on one pregnant woman's skin. It is difficult to place an electrode directly on a baby because of several factors. Therefore, ECG recordings are usually measured on the mother's skin[20]. We download related data at the website:http://homes.esat.kuleuven.be/～smc/daisy/daisydata.html. Here, using the first 1500 samples as the training set, and an additional 1000 samples are used for evaluating the performance of model. Fig.6 shows the potential test data recordings. In this example, recordings from channels 1 and 8 are used as quality variablesy1andy2for regression analysis, and the signals from channels 2-7 are selected as the process variablesx1-x6.

Fig.7 shows and compares the quality prediction results (using the RMSE index) from different algorithms in the case of whitening and non-whitening, respectively. The sub-optimization parameters are set asα=0.8 andβ=0.2. They are calculated using 1000 samples and different numbers of ICs. The proposed MICR algorithm yields lower RMSE values than the EMICR algorithm using measured and prewhitening data. Additionally, only two ICs are needed to interpret variabley1and one IC is needed to interpret variabley2. Thus, interpretation ability is improved and computational complexity is reduced. After model testing, we use different algorithms to calculate the RMSE values against different numbers of ICs and list final results in Table 2. In summary, the proposed MICR model predicts quality are more accurate than the EMICR model. Finally, the profiles of MICR method and conventional ICR with different functionsGabout quality variablesy1andy2are given in Fig.8. We can also see the advantages of ensemble learning in the ICR method.

Fig.6 Recordings of FECG data

Fig.7DescriptionofRMSEvaluesvs.numbersofICs

Table 2 Results of the FECG dataset

Fig.8DescriptionofRMSEvaluesvs.numbersofICs

4 Conclusions

The paper suggests an MICR method with non prewhitening process and considers the advantages of the EMICR and improved ICR algorithms. The proposed model is superior to the EMICR and improved ICR models because it uses three optimization objectives based on measured data in the IC extraction, which can separate more effective ICs. According to the experimental results, all simulations mentioned in this work have demonstrated that MICR method has effectiveness and better prediction performance than other methods.

While the present approach (without prewhitening) increases quality variable predictive capabilities and decreases data errors, the computational complexity of the model is sometimes high and holds promise for future research using the MICR algorithm.