XU Zhigang,MA Qiang,and YUAN Feixiang

School of Computer and Communication,Lanzhou University of Technology,Lanzhou 730050,China

Abstract: Color image super-resolution reconstruction based on the sparse representation model usually adopts the regularization norm (e.g., L1 or L2). These methods have limited ability to keep image texture detail to some extent and are easy to cause the problem of blurring details and color artifacts in color reconstructed images. This paper presents a color super-resolution reconstruction method combining the L2/3 sparse regularization model with color channel constraints. The method converts the low-resolution color image from RGB to YCbCr. The L2/3 sparse regularization model is designed to reconstruct the brightness channel of the input low-resolution color image. Then the color channel-constraint method is adopted to remove artifacts of the reconstructed highresolution image.The method not only ensures the reconstruction quality of the color image details, but also improves the removal ability of color artifacts. The experimental results on natural images validate that our method has improved both subjective and objective evaluation.

Keywords: color image, sparse representation, super-resolution,L2/3 regularization norm,color channel constraint.

1.Introduction

Single image super-resolution(SR)reconstruction restores the missing high-frequency details of the original image with a degraded and low-resolution(LR)observation image. The SR reconstruction offers a promise to overcome the inherent resolution limitations of LR images, so that the reconstructed high-resolution(HR) image can present better visual effects[1,2].

The single image SR is an underdetermined inverse problem because its solution is not unique. This problem can be mitigated by adding strong prior information to the solution space.To learn the prior,the learning-based strategies for SR have got a lot of attention in recent years[3,4].The learning-based SR reconstruction method can make full use of the prior information of LR images and the prior information obtained by learning. The better reconstruction results can be obtained even when the magnification factor is high.Therefore,the learning-based SR becomes one of the most promising methods for many practical applications.There are some representative methods based on learning, such as the methods based on neural network[5,6],the methods based on sparse coding[7–9]and the methods based on anchored neighborhood regression[10,11].

Conventional SR methods concentrated on grayscale images. However, in the past several years, the demands for SR of single color image are increasing with the rapid development of color imaging and display devices. Thus,the color image SR has important application prospects[6,12], such as high-definition digital TV, remote sensing monitoring,mobile Internet,medical,cultural relic protection and display.It makes the color image SR reconstruction become an urgent problem to be solved.

There are two categories of color image reconstruction methods used in SR tasks:the methods based on joint color channels and the methods based on converted color space.

1.1 SR methods based on joint color channels

The methods connect three channels of the color image as a whole to be reconstructed,which expect to reduce color artifacts of the reconstructed color image. Xu et al. [12]adopted a quaternion matrix to represent the color image.They presented a quaternion-based sparse dictionary learning algorithm which was used in SR. Yang et al. [13] regarded RGB channels as a single vector,and strengthened the connection among RGB channels through the new defined inner product. Cheng et al. [14] proposed a singleimage SR algorithm based on group sparse representation.When training the SR dictionaries, the algorithm considered the three color channels of an image patch as a group composed of three atoms.The whole group is selected simultaneously when representing an image patch so that the correlations between the color channels can be retained.

The implementation of these methods is complex.Such methods usually use the same sparse representation coefficient for each color channel when the sparse dictionaries are trained, which may result in the obtained sparse dictionary containing more gray atoms.As a result,it causes some color artifacts in those reconstructed HR color images.

1.2 SR methods based on converted color space

According to the characteristic that human eyes are more sensitive to luminance component of color images, these methods converted an RGB image into other color space that can separate luminance and chrominance channels(e.g.,YCbCr,Lab,or HSI).Yang et al.[7]converted color images into YCbCr color space.They used the sparse SR algorithm to reconstruct only the brightness layer. Yang et al.[15]established the generation model of HR and LR image patch pairs based on self-learning and data-driven methods, using support vector regression to generate the luminance channel of the HR images. Xu et al. [16] proposed an SR method which combined color channels adaptively to obtain significant texture information.They integrated the information into the luminance component and adopted the improved total variation regularization method to suppress artifacts of the reconstructed HR image.

The SR reconstruction methods based on converting color space process the luminance information of the color image by using SR reconstruction methods of the gray image.It can reduce the complexity of the SR problem while basically guarantee the quality of color image reconstruction. However, these methods may completely ignore the inter-relationship among multiple color channels,which is likely to lose some image details of the reconstruction results.

In this paper, we present a blended sparse SR reconstruction method for color images.The major contributions of the proposed method include two aspects. Firstly, theL2/3sparse regularization model is adopted to reconstruct the luminance component of the HR image. This strategy will result in more sparse solution of the SR problem.Secondly,the boundary similarity constraint among RGB color channels is used to eliminate color artifacts of the reconstructed color image, so that the reconstructed color image is rendered to have better visual effects.The effectiveness of our method is verified by the experimental results.

The rest of this paper is organized as follows.Section 2 describes the details of the proposed method. Section 3 gives the experimental results for color image SR reconstruction.Finally,we draw our conclusions in Section 4.

2.The proposed single color image super-resolution algorithm

This section will analyze the proposed blended sparse reconstruction algorithm.The algorithm can be divided into two steps. In the first step, the LR color image is transformed into the YCbCr color space.TheL2/3sparse regularization model is utilized to reconstruct the Y component of the HR color image.The Cb and Cr components of the HR image are interpolated by Bicubic interpolation.In the second step,the boundary similarity constraints among RGB color channels is used to found a color artifact removal model which is based on sparse representation.

2.1 Sparse representation model based on L2/3 regularization norm

The purpose of color SR reconstruction is to reconstruct an HR color image with an input LR color image. Based on the sparse representation model, for each input LR patchy,a sparse representation vector will be found with respect to a dictionaryDlwhich is trained from LR patches.The problem of finding the sparsest representation vector ofycan be expressed as

Then,the output HR patchxwill be generated according to the vectorαandDhwhich is trained from HR patches.

The optimization (1) is an NP-hard problem. Solving the problem generally is intractable. Therefore, it is well known that theL1regularization method is widely used for learning SR sparse dictionaries and SR reconstruction.TheL1sparse optimization problem is formulated as follows:

TheL1regularization problem can be transformed into an equivalent convex quadratic optimization problem and can be efficiently solved. Nevertheless, theL1regularization may yield additional deviations in sparse representation coefficient estimation. It tends to result in solutions that are usually not sparse enough,and easily fails to achieve good sparse approximation results for input image data affected by noise.

TheL2/3regularization[17]was proved that it can induce the sparsity of vectors further effectively. TheL2/3regularization is more robust to noise in sparse representation and more stable than the solution ofL1regularization.Therefore,our algorithm constructs a joint sparse SR dictionary model based onL2/3regularization,as follows:

In order to train the HR/LR dictionary pairs, a closedform thresholding method [17] is adopted in the sparse coding phase, and the K-singular value decomposition method[18]is used in the dictionary atom updating phase.

2.2 Removing color artifacts of the color HR reconstruction image

Since the algorithm proposed in Section 2.1 only reconstructs the Y channel of the color LR image, it does not consider the correlation among the chromaticity channels,which may make color artifacts appear in the reconstructed HR image.

In fact, most of the high-frequency spatial components of natural images have a strong correlation in RGB channels, so this paper utilizes the boundary consistency method [19] to obtain the boundary information of each color channel. We intend to enforce the correlation between color channels of color images, thereby eliminating the color artifacts of the reconstructed HR image.The boundary similarity model among RGB color channels can be expressed as

wherer,gandbindicate R, G and B channels.F(Fr,Fg,Fb) is high-pass filters. These constraints can essentially reinforce the edge information similarity across color channels of an HR patch.Moreover,these constraints corresponding to the sparse representation vectors of each independent color channel can be expressed as

The problem(5)for different color channels are not independent and they can be jointly estimated by solving the following optimization problem:

where the subscriptkrepresents the R,G and B color channels.βandγare the regularization parameters.

The problem(6)can be minimized into a convex sparse constrained optimization problem. We adopt the fast iterative shrinkage-thresholding algorithm [20] to solve the problem.Then,the color HR image patch can be obtained

2.3 Color LR image SR reconstruction

This paper combines the sparse model and the color channel constraint model for color image SR reconstruction.The entire SR process is summarized as Algorithm 1.

Algorithm 1 Color image SR reconstruction based on sparse representation and color constraint.

YCbCr space:

(i)Input: trained dictionariesDlandDh,the color LR imageYl.

(ii) The color LR imageYlis transformed into the YCbCr color space.

(iii) The luminance layer is divided intopatches,and adjacent patches have overlap ofels.

a.Compute the mean pixel valuemof patchy.

b.Solve the optimization problem as follows:

The sparse coefficientsαcan be obtained.

c.The HR patch is generated by(8).

(iv)Fusing all the HR patches,the HR luminance layer is generated. Bicubic interpolation is adopted to reconstruct the chrominance layers Cb and Cr. Then,the initial HR reconstruction imagewill be obtained.

RGB space:

(i)Input:color restoration dictionaryDC,and the initial HR reconstruction image√

b.Solving the optimization problem(6)andis got.

(ii)Fusing all the color image patches,the HR color imageXwill be generated.

(iii)Output:HR reconstruction imageX.

3.Experimental results and analysis

In this section, we firstly discuss two influential factors for the proposed algorithm including the size of the training dictionary and the regularization parameterλ. Then,we compare the proposed method with several SR methods, namely bicubic interpolation (BI), sparse coding SR(ScSR) [7], nonlocally centralized SR (NCSR) [21] and multiple color channels SR(MCcSR)[19].BI represents a most common reconstruction method.ScSR is a sparse reconstruction method which adopts learned joint dictionaries to reconstruct HR images.NCSR is a sparse SR method based on a novel nonlocally centralized sparse representation.MCcSR is a color SR method by learning color dictionaries that encourage edge similarities.

The SR joint dictionaries are trained with the same training set as[7].Test images in Fig.1 are selected from commonly used data sets, such as Set5, Set14 and so on. The reconstructed images of various methods are evaluated by being used visually and qualitatively in the peak signal-tonoise ratio(PSNR)and the structural similarity(SSIM).In order to get the quantitative results, we only apply PSNR and SSIM to the luminance channel of the reconstructed HR images and the original HR images.

Fig.1 Test images

Fig.2 shows the influence of regularization parameterλon the reconstruction results of the butterfly image. From Fig.2,we can easily get that PSNRs increase rapidly asλvaries from 1.5 to 2.0,and then reach the maximum value near 1.8.Therefore,we set the value ofλas 1.8 in the subsequent experiments.

Fig.2 Change curve of PSNR value

Fig. 3 evaluates the effects of different dictionary sizes on the reconstruction results. The reconstructed results of the bike image using dictionaries of different sizes are shown. We train four dictionaries with the sizes of 128,256,512,and 1 024.We apply the dictionaries to the same input image.We also use the 100 000 image patches as the dictionaries for comparison.As we can see,with the rapid increase of the dictionary size,the increase of PSNR is not high.Comparing the training time of the joint dictionaries with the quality of the reconstructed image,we choose the sparse dictionary of the size 512.

Fig.3 PSNR curve and the training time histogram

In order to verify the validity of the proposed method,we compare our method with several methods. We use 100 000 HR/LR patches of the size 6×6 with 4-pixel overlaps to train the joint dictionaries. We set that the upsampling factor is 2 and the iterations are 20.The experiment results are shown in Table 1. From Table 1, we can see that our method achieves better results on PSNR and SSIM values than other methods.

Table 1 PSNR and SSIM results of five color images for ×2 scale factor

Fig. 4 is one of the reconstruction results of the above five experiments. As can be observed, the BI algorithm produces smooth edges and loses some details,making the reconstructed image look very blurry.The ScSR algorithm achieves better reconstruction results, but produces some obvious color artifacts in the reconstructed image. The NCSR algorithm reconstructs a good result without obvious color artifacts, but the result is excessive by smooth for more detail-rich areas so that some details are lost.The MCcSR algorithm can reconstruct the color information of the color image very well,but there are still some artifacts in the reconstruction image. It is worthwhile to point out that the reconstructed textures by our method are the most natural.Furthermore,the reconstructed image edge sharpness improves a lot.Some obvious artifacts are effectively removed in the edge region of the HR image.

Fig.4 Comparison of SR results on the flowers image

4.Conclusions

The SR reconstruction of the color image based on the sparse representation model adopts the regularization norm(e.g.,L1orL2), which results in sparse solutions that are usually not sparse enough. Furthermore, the prior information of color channel correlation is often neglected.In this paper, we propose an SR reconstruction algorithm based on theL2/3sparse regularization model and the color channel constraints.This method not only adopts the sparse model based on theL2/3regularization norms to reconstruct the LR images,but also eliminates the color artifacts in the reconstructed image by using the color channel correlation constraint model.Thus, the SR reconstruction effect and the fidelity of the reconstructed color image are improved. Experimental results indicate that our method can achieve highly competitive performance to other methods.