ZHANG Yan,ZHANG Jinghua,SHI Zhiguang,ZHANG Yu,and LING Feng

National Key Laboratory of Science and Technology on Automatic Target Recognition,College of Electronic Science and Engineering,National University of Defense Technology,Changsha 410073,China

Abstract: Specific to the reflected light problem on the surface of transparent body,the polarization characteristics of the reflection region are analyzed,and a polarization characterization model combining the reflection and transmission effects is established.On the basis of the polarization characteristic analysis,the minimum value of normalized cross-correlation (NCC)coefficient between transmission and reflection images is solved through the gradient descent method,and their polarization degrees under the minimum correlation are acquired.According to the distribution relations of the transmitted and reflected lights in perpendicular and parallel directions,reflection separation is realized via the polarized orthogonality difference algorithm based on the degree of reflection polarization and the degree of transmission polarization.

Keywords: reflection separation,transparent object,correlation,polarization characteristics.

1.Introduction

When the scenery behind transparent objects is imaged by a camera,the acquired image consists of two parts,namely,transmitted light and reflected light [1,2].As the reflection and transmission simultaneously take place on the surface of the transparent object,the reflected and transmitted lights are usually mixed and influence each other [3-6].With the modernized urban construction,materials with transmission and reflection effects,such as glass and plastics,have been applied in quantity in interior home decoration.The separation of transmitted light and reflected light on the surface of transparent object and the inhibition problem of reflected light have aroused extensive attention of scholars and have become the research hotspots.The reflected light presents a virtual image on the object surface,covering information such as color sand details of scenery behind glass [7-9].When the intensity of incident light source is strong,a highlighted area is formed on the object surface,which influences the image quality and brings about great problems to computer processing and human eye recognition[10,11].In addition,information,such as intensity,position,and surrounding environment of reflected light source can be acquired by extracting and analyzing reflected light,with a certain application value [12-14].Therefore,the separation of reflected light and transmitted light on the surface of transparent object is challenging in the field of computer vision;it is important for applications such as image segmentation,target recognition,and stereo matching [15-17].

The existing reflection separation algorithms can be divided into two types as follows: (i) Separation algorithms based on image features,such as blurred edge features of reflected scenery caused by camera defocus [10],and double image features generated by the simultaneous reflection of front and rear glass faces [18].Pure reflection separation based on image features can not achieve ideal effect because the reflection and transmission images present mutual interference and overlapping;(ii)Separation algorithms based on physical properties,such as polarization characteristics generated in the reflection process of light wave.The reflected and transmitted lights on the surface of transparent objects have an evident polarization effect,and their polarization characteristics are evidently different.Thus,the reflection light separation can be realized using the polarization characteristics.In addition,relative to the pure separation algorithm based on image features,better separation effect and stability are obtained.The reflection separation technology based on polarization characteristics was investigated by Wolff in early phase.Wolff et al.[19,20] conducted a detailed analysis and quantitative description of mirror reflection effect and diffuse reflection polarization effect on the object surface according to Fresnel law of reflection,laying a foundation for the reflected component separation based on polarization characteristics.In 2002,with the assumption that the diffuse reflected light on object surface was natural light,the mirror reflected light was partially polarized light,and the polarization degree did not change with the observation angle of object surface.Wolff et al.[21] used the Fresnel reflectivity of reflected light to separate the mirror reflected light component and diffuse reflected light component.The experimental results show that compared with the traditional reflection separation method based on color information,the reflection separation method based on polarization information can separate the reflected light not only on the surface of dielectric medium surface but also on the surface of metal material.In addition,it has greater applicability because it does not limit the light source or color information on the surface of the reflected object.Schechner et al.[22] proposed a reflected light and transmitted light separation method based on polarization characteristics for such transparent objects as glass.They discovered that the reflected and transmitted lights belong to polarized light by analyzing the light intensity received by the detector using this method.Under the known refractive index,the ratio of reflected light to transmitted light is only related to the observation angle of the glass surface.The observation angle is estimated through the polarization information to realize the separation of reflected light and transmitted light.Ohnishi et al.[11]put forward a reflection separation method based on rotary polarizing films.In accordance with the principle that the light intensity varies with the incident polarizing angle,this approach acquires the polarization methods at different incident polarizing angles through rotary polarizing films,selects the pixel point with the minimum intensity as the transmitted light component,and takes the value by deducting the minimum light intensity from the maximum light intensity as the reflected light part.Thus,the separation of reflected light is realized.The polarization image acquired under a single polarizing angle cannot easily completely eliminate the reflected light.Thus,Kong et al.[23] put forward a method of eliminating reflected light using multiple polarization images.This method can separate and repress the reflected light on glass surface through the optimization method,considering the continuous changes in polarization image at adjacent incident polarizing angle and the principle of gradient mutual exclusion between reflecting-layer and transmitting-layer images.

Although the reflection separation technology based on polarization characteristics has aroused extensive attention and has been deeply probed with gratifying research results,most existing reflection separation algorithms are based on strict experimental constraint conditions and assumptions;they can realize favorable separation of reflected light only under specific environment or on the precondition of prior information.For example,the method in [11] can only eliminate the reflected light of incident angle nearby Brewster angle.The method in [22]requires that the reflective index of transparent texture is known.The application condition for the method in [21]is that the diffuse reflection is unbiased.For the algorithm in [23],the polarization image is assumed to have gradient sparsity,and manually setting the gradient threshold is necessary.These methods have certain limitations in the practical application process because these experimental conditions cannot be easily satisfied in reality.In addition,most present reflection separation methods based on polarization only consider the polarization effect of reflected light,while not taking full advantage of the polarization characteristics of transmitted light.The reflected and transmitted lights on the transparent surface are polarized lights,and they co-exist and exert a combined action.The optimal separation of reflected component and transmitted component can be realized only by comprehensively considering their interaction.Given these problems in the present reflection separation methods based on polarization characteristics,the comprehensive polarization effect of reflected and transmitted lights is combined.A reflection separation algorithm based on the minimum correlation between reflected image and transmitted image is proposed,following the orthogonal decomposition principle of polarization.This algorithm does not require strict experimental assumptions or known prior information such as refractive index.It can realize the separation of reflected light directly by using the polarization images acquired by a split focal plane camera.The polarization phenomena of reflected and transmitted lights on object surface are analyzed,and a polarization characterization model,in consideration of their interaction,is established and used to calculate the polarization state of light wave on object surface.Subsequently,the orthogonality difference algorithm of polarized light is used to solve the reflected light and transmitted light components via their polarization degrees.The gradient descent algorithm is used to solve the minimum value of normalized crosscorrelation (NCC) coefficient between reflection image and transmission image to acquire their corresponding polarization degrees and achieve their best separation because the polarization degrees of reflected component and transmitted component could not be directly measured by the detector.

2.Polarization modeling of transparent object surface

2.1 Orthogonal decomposition of reflected and transmitted lights

The reflected light on the object surface belongs to partial polarized light,which can be decomposed into the sum of light intensity in the direction perpendicular to the incident plane and that in the direction parallel to the incident plane [24-26]:

The polarization degree generated in the reflection process [27,28] is as follows:

Similarly,the following orthogonal decomposition can be implemented for the transmission polarized light:

The polarization degree generated in the transmission process is as follows:

2.2 Calculation model of polarization degree combining reflection and transmission effects

The reflected and transmitted lights co-exist and interact with each other on the surface of the transparent object,and the total light intensity received by the detector is as follows:

whereI⊥andI||respectively denote the total light intensity received by the detector in perpendicular direction and parallel direction,which can be expressed as

The polarization degree on the surface of the target object is as follows:

whereαis the ratio of transmitted light intensityPTto r eflected light intensityPR.αcan be calculated as follo ws:

Differentαvalues (0.4,0.6,0.8,1,1.2,1.5,and 2) are selected to simulate the polarization degree of glass surface at different observation angles,as shown in Fig.1.

Fig.1 Representation of polarization combining specular reflection and diffuse reflection

Fig.1 shows that the polarization state on object surface is not only related to its own refractive index but also to the reflected light intensity and transmitted light intensity.The polarization direction of reflected light is perpendicular to that of transmitted light;thus,their interaction would result in the depolarization phenomenon of light wave.When the reflected light is greater than the transmitted light,the polarization direction is perpendicular to the reflector;otherwise,it is parallel to the reflector.When the reflectivity α fluctuates to approximately 1,the polarization direction is changed by 90°.

3.Separation of reflected and transmitted lights

Previous analysis indicates that the polarization state on the object surface is jointly decided by reflected and transmitted lights.Therefore,the polarization information of light wave could realize the separation of reflected and transmitted lights.In this section,the distribution relations of reflected light intensity and transmitted light intensity in perpendicular and parallel directions are used to solve the reflected and transmitted light components on the surface of the transparent object through the polarization degrees of reflection and transmission to realize the separation of reflected light.

3.1 Component extraction in perpendicular and parallel directions of light intensity

According to the representation form of polarized light[29,30],the calculation formula of polarized light intensity at different incident polarizing angles is as follows:

where φκisthe incident polarizing angle,κisthevalue of theincident polarizing angle,and φ⊥istheincident polarizing angle in the perpendicular direction. φ0is set to 0°,and the incident polarizing angles are set to φ0=φ0,φ45=φ0+45°,and φ90=φ0+90°to acquire the corresponding polarization imagesI0,I45,andI90,respectively;they are s ubstituted into the equation to solve the light intensitiesI⊥andI||in the perpendicular and parallel directions,respectively,as follows:

3.2 Reflection separation based on orthogonal decomposition of polarization

According to the orthogonal decomposition principle of polarization,the components of light intensity received by the detector in the perpendicular and parallel directions,are as follows:

The reflected and transmitted lights on the surface of the transparent object belong to polarized right.The polarization degrees of the reflected and transmitted lights are set to γ and χ,respectively,as follows:

Then,we obtain the following equations:

Equations (15) and (16) are substituted into (12) to solve the components of transmitted light intensity in the perpendicular and parallel directions as follows:

Furthermore,the components of reflected light intensity in the perpendicular and parallel directions are as follows:

The total light intensity is equivalent to the sum of light intensity in the perpendicular direction and that in the parallel direction.Thus,the reflected and transmitted light components on the glass surface can be solved as follows:

3.3 Solving polarization degrees of reflection and transmission based on the minimum correlation

3.3.1 NCC

Under the known polarization degrees of reflected and transmitted lights as well as light intensities in the perpendicular and parallel directions,(19) can be used to solve the reflected light intensity and transmitted light intensity at each point on the transparent object surface.However,due to the fact that the transparent object simultaneously reflects and transmits light,the polarization degrees of reflected and transmitted lights can not be directly acquired by the detector.

The reflection and transmission images acquired under the ideal separation have the minimum correlation because the reflected and transmitted lights on the surface of the transparent object contain different image contents.Therefore,the reflection and transmission polarizations under the minimum correlation could be obtained to realize the separation of reflected light.The correlation between the two images could be denoted by the NCC.The NCC coefficient(i,j) at any pixel pointr(i,j) in the image is as follows:

whereUandVexpress the window sizes;uandvrepresent the positions of pixel points within the window;andare the mean values of gray-level pixels in the windows of the reflection and transmission images,respectively.f(γ,χ) is the sum of the absolute values of NCC coefficients of all pixel points in the reflection image and transmission image after separation,as follows:

3.3.2 Solving the minimum cross-correlation coefficient The visible images acquired in real world are shown in Fig.2(a) and Fig.2(b),and the perpendicular and parallel polarization images simulated according to the principle of reflected light are shown in Fig.2(c) and Fig.2(d).

Fig.2 Reflection scenes

The transmission and reflection images separated through (19) under the transmission polarization of χ=0.1 and reflection polarization of γ=0.2,0.4,0.8,as shown in Fig.3.Fig.4 display the transmission and reflection images acquired under the reflection polarization of γ =0.4 and transmission polarization of χ=0.01,0.15,0.3.

Fig.3 Separation results of reflected light when χ=0.1 and γ=0.2,0.4,0.8

Fig.4 Reflection separation results when γ=0.4 and χ=0.01,0.15,0.3

Fig.4 shows that the decomposed reflected and transmitted light components vary with the reflection polarization.Under a small reflection polarization,the decomposed reflected light component is greater than the actual reflected light component,the decomposed reflection image contains partial transmitted light component,which is called “over-decomposition”.When the reflection polarization is large,the circumstance is called“under decomposition”.Regardless of “under decomposition” or “over decomposition”,the decomposed reflection image is highly correlated with the transmission image.

The results in Fig.4 indicate that the decomposed reflected and transmitted light components vary under different transmission polarizations.When the transmission polarization is greater than or smaller than the actual value,the over-decomposition and under-decomposition also appear,and a high correlation is still manifested between the reflection and transmission images.

Fig.5 shows the change laws of NCC coefficientf(γ,χ)between the reflection image and transmission image with the degree of reflection polarization (DoRP)and degree of transmission polarization (DoTP).

Fig.5 Change laws of NCC with DoRP and DoTP

Fig.5 shows that after the separation,the cross-correlation between the reflection image and transmission image reaches a minimal value at pointP,where the corresponding DoRP and DoTP are the closest to the actual values,and the separation effect of reflected and transmitted lights is the best at the moment.

3.3.3 Extraction of dominant pixel points

In the actual separation process,two problems are triggered if the sum of the cross-correlation coefficients of all pixel points in the image,such as large calculated quantity,low calculation speed,and susceptibility to noise interference is directly solved.A method of extracting dominant pixel points is introduced to inhibit the noise interference and elevate the calculation speed.The dominant pixel points refer to pixel points dominated by the transmitted or reflected light component in a hybrid image,similar to the pixel points in the blue box region in Fig.2(c) and Fig.2(d),in which the edges,textures,and details of the reflection image are clearly shown;these pixel points are called dominant pixel points of reflection.Similarly,the clear edges,textures,and details of transmission image are found in the pixel points (dominant pixel points of transmission) in the red box region in Fig.2(c) and Fig.2(d).The proposed polarization model combining the transmitted and reflected lights in the previous section indicates that the polarization degree is associated with the ratio of transmitted light to reflected light,and the greater difference value indicates greater polarization degree,and the smaller difference value indicates smaller polarization degree.The dominant pixel points of reflection and transmission in the image are greatly different in the transmitted and reflected light intensity.Thus,these pixel points are featured by high polarization degree and strong anti-interference performance.

For the pixel points dominated by reflected light,such as the pixel points in the blue box shown in Fig.3,asγ continuously increases,the separation process of the reflected light is transformed from over-decomposition into under-decomposition,and the correlation between transmission and reflection images is transformed from negative correlation into positive correlation.For the pixel points dominated by transmitted light,such as the pixel points in the red box shown in Fig.3,the correlation change is not evident when γ is elevated,without the“positive to negative” change.Therefore,the pixel points dominated by reflected light could be extracted through the positive-negative change relation in the under-separation and over-separation processes.Subsequently,the pixel points dominated by reflected light can be used to accurately and efficiently solve the actual γ.

The specific extraction steps of pixel points dominated by reflected light are as follows: First,γ=0.01 and χ=0.2 are set to over-separate the hybrid image and obtain the over-separated transmission imageIover-tafter the separation;Second,γ=0.99 and χ=0.2 are set to perform the under-separation of the hybrid image and obtain the under-separated transmission imageIunder-t(Note:According to the Fresnel law,the χ of glass ranges from 0 to 0.4;thus,the median 0.2 is selected to realize the under-separation and over-separation solution.According to (19),χ only influences the intensity of transmission image after separation,but the textures and details of the separated transmission image remain unchanged.The value of χ at 0.1 or 0.4 has no remarkable influence on the correlation between the under-separated and over-separated transmission image.Thus,setting χ=0.2 is reasonable).Then,the correlation betweenIover-tandIunder-tis solved to obtain the related imageRt.In the end,the pixel points with negative correlation are set to 1,and those with positive correlation are set to 0 to extract the reflection-dominated pixel points.

At the moment,the NCC sumfR(γ,χ) of the reflectiondominated pixel points is solved as follows:

Similarly,the actual DoTP is solved by selecting the transmission-dominated pixel points.First,γ =0.5 and χ=0.01 (Note: The selection basis for γ is similar to that for χ) are set to over-separate the hybrid image and obtain the over-separated reflection imageIover-r.Subsequently,γ=0.5 and χ=0.99 are selected to under-separate the hybrid image and obtain the under-separated transmission imageIunder-r;their cross-correlation is solved to obtainRr,followed by binarization,as follows:

At the moment,the NCC sumfT(γ,χ) of transmissiondominated pixel points is solved as follows:

3.3.4 Gradient descent method

The gradient descent algorithm is adopted to solve the minimum value of NCC between the reflected and transmitted light components,and the NCC coefficientsfR(γ,χ) andfT(γ,χ) of the reflection and transmission images could be regarded as functions of γ and χ.Next,their partial derivatives are solved to decline alternately along the gradient direction.Through multiple iterations,the minimum values offR(γ,χ) andfT(γ,χ) can be simultaneously obtained.

where η is the learning rate.The iteration ends when the convergence conditions are satisfied;thus,the minimum values of NCC coefficientsfR(γm,χm) andfT(γm,χm),as well as the corresponding χmand γmare obtained to realize the optimal separation of reflected and transmitted lights.

4.Experimental results and analysis

4.1 Analysis of reflection separation results

The reflection separation algorithm based on polarization characteristics is tested using the images acquired under different reflection scenes,which include different transmission backgrounds (outdoor and indoor environments).The reflector,reflected light source,and reflection angle vary from scene to scene.Therefore,the reflection images acquired from these scenes could be used to comprehensively and effectively evaluate the proposed algorithm.

Fig.6(a) presents the images acquired by the detector in the real world,where the odd-numbered lines show the background images without placement of glass,and the even-numbered lines show the reflected light hybrid images acquired after glass are placed in front of the detector.

Fig.6 shows that the reflection-and transmission-dominated pixel points exist in the hybrid image.The separation results acquired through the proposed algorithm are displayed in Fig.6(b).DoRP and DoTP are combined to realize the effective separation of reflected light under different scenes on the basis of the physical characteristics of polarized propagation of reflected light and transmitted light on the glass surface as well as the parallel and perpendicular distribution characteristics of reflected light and transmitted light.From the visual effect,the transmission images (odd-numbered lines) separated through the proposed algorithm are approximate to the actual background image.The reflected light is effectively separated and removed in the intense reflection region or weak reflection region.Meanwhile,the reflection images (even-numbered lines) separated through the algorithm shows that the details and textures of reflection images are effectively extracted and reserved after the separation.

The proposed algorithm is compared with the existing reflection separation algorithms to comprehensively evaluate its separation effect.Fig.6(c) displays the results acquired by the reflection separation algorithm in [10].This reflection separation algorithm employs a strategy that the reflection layer in an image is assumed to be smoother than the transmission layer.Fig.6(c) shows that this algorithm could easily obtain extremely smooth transmission image and missing texture of the reflection image.Fig.6(d) presents the separation results obtained through the algorithm in [11],which considered the minimum value of light intensity as the transmitted light component,and the difference between the maximum and minimum values as the reflected light component.The separation results in Fig.6(d) show that as the minimum light intensity usually containes some reflected light components,this method could hardly realize the complete separation of the reflected light.The separation results obtained by the reflection separation algorithm combining independent component analysis (ICA) and polarization characteristics in [24] are depicted in Fig.6(e).Fig.6(e) shows that this algorithm could easily lead to the“over-separation” and “under-separation” of reflected light.Thus,the reflected and transmitted light components still co-exist after the separation,especially a large number of transmitted light components exist in the separated reflection image.

Fig.6 Separation results of the reflected light under different scenes

The comparative results indicate that the proposed algorithm fully considers the interaction between the reflected and transmitted lights on the glass surface and realizes their separation by following the principle of polarized light orthogonality difference,with a separation effect superior to the existing algorithms.

4.2 Quantitative comparison

The structural similarity (SSIM),peak signal to noise ratio (PSNR),image cross-correlation (CORR),and the three common evaluation indexes of image correlation,are used to compare the transmission images separated with the actual image,as shown in Table 1 (black bold data mean that this index is the highest) to quantitatively compare the reflection separation effects of these algorithms.Table 1 shows that relative to the other algorithms,the proposed algorithm achieves the highest SSIM,PSNR,and CORR between the separated transmission image and actual image,indicating that the reflection image separated through the proposed algorithm is more approximate to the actual image,with better reflection separation effect.

Table 1 Quantitative comparison of reflection separation effects under different methods

5.Conclusions

A separation technology of reflected and transmitted light on the surface of transparent object based on polarization characteristics is explored as an emphasis in this study.According to the Fresnel’s law of reflection,a polarization characterization model combining the comprehensive action of reflection and transmission is established to simulate and analyze the polarization state in the reflection region.After analyzing the polarization characteristics,the light intensity distribution relations of the reflected and transmitted lights in the perpendicular and parallel directions are used on the basis of their interaction characteristics.The reflection and transmission images are solved using the polarized orthogonality difference algorithm through the DoRP and DoTP.Finally,following the principle of minimum correlation between the reflection and transmission images under the optimal separation,the gradient descent algorithm is used to solve the DoRP and DoTP corresponding to the minimum value of NCC between reflection image and transmission image to realize the optimal separation of reflected and transmitted lights on the surface of transparent object.The experimental results show that the proposed algorithm can separate the reflected light under different scenes,and its separation effect is superior to other existing algorithms.