Contourlet-based non-local mean via Retinex theory for robot infrared image enhancement

Aiming at the problems of fuzzy details and excessive enhancement in traditional robot infrared image enhancement algorithms, a robot infrared image enhancement method based on Retinex theory and contourlet-based non-local mean is proposed. Firstly, the single-scale Retinex method is used to adjust the gray level of the over-dark and over-bright parts of the image. Then, the contourlet-based non-local mean is used to decompose the image to obtain the basic layer and detail layer. Histogram equalization is used to stretch the contrast of the basic layer, and nonlinear function is used to enhance the detail layer. Finally, the results of different levels are fused to obtain the contrast and detail enhanced robot infrared image. The proposed method is used to simulate several groups of robot infrared images in different scenes, and compared with other enhancement methods for subjective and objective analysis. The results show that the proposed method achieves better performance in detail and contrast enhancement of infrared images.


Introduction
The rapid development of robot infrared technology has made infrared cameras more and more widely used in some areas such as automatic driving, night search and rescue, and security [1]. However, the thermal radiation signals received by infrared cameras are easily absorbed, scattered or scattered during atmospheric transmission. Reflected by clouds and water vapor, the obtained robot infrared images have low contrast and lack detailed texture information. Such low-quality infrared images are difficult to meet the needs of practical applications. Therefore, the contrast and detail enhancement of infrared images become the most important thing in infrared imaging systems.
To improve the visual effect of the original robot infrared image, the image enhancement algorithm must highlight the details in the image and stretch the image contrast while retaining most of the information of the original infrared image. The existing image enhancement algorithms are mainly divided into spatial domain and transform domain [2]. The image enhancement algorithm based on the histogram is a very important spatial image enhancement algorithm, and the histogram equalization (HE) [3] will be unevenly distributed. The histogram is transformed into a uniform distribution after mapping, so that the contrast of the image is significantly enhanced. Xi Zhang and Jiyue Wang 2 However, this algorithm is a global operation of lowquality infrared images with limited gray-scale distribution, which is prone to loss of detail and excessive enhancement. Some researchers have proposed various improved algorithms for HE, including platform histogram equalization (PHE) [4], contrast-limited adaptive histogram equalization (CLAHE) [5], adaptive platform histogram equalization (APHE) [6] and other algorithms [7][8][9]. These improved HE enhancement algorithms improve the problem of image detail loss and over-enhancement by segmenting the histogram or setting thresholds, and have achieved better results. However, based on the histogram enhancement, these algorithms only pay attention to the global or local gray distribution, and do not consider the structural characteristics of the input image, which often causes the image visual effect to be stiff and the noise is enhanced. Another type of image enhancement algorithm based on the spatial domain is adaptive image enhancement, including improved Unsharp masking method [10], Retinex [11], homomorphic filtering (HF) [12] and mathematical morphology [13]. The unsharp masking method uses lowpass filter decomposition. For the image, different methods are used to enhance the details and contrast of each part of the decomposed image, and a better enhancement effect is obtained. However, the result of the image decomposition by the filter determines the enhancement result to a large extent. Retinex image enhancement can better adjust the gray distribution in the image, but it is not very ideal in the enhancement of image details. Image enhancement based on the transform domain mainly includes Fourier transform, wavelet transform, contourlet transform and other multi-scale Geometric analysis methods [14,15]. This type of enhancement method first maps the original infrared image to the transform domain, and then classifies the decomposed sub-band coefficients, enlarges useful components, reduces useless components, and finally inversely transforms to the spatial domain to achieve image enhancement. The method can realize the enhancement of the obvious structure in the image, but it ignores the weak detailed information.

EAI Endorsed Transactions on Scalable Information Systems
Probabilistic Non-Local Means Filtering (PNLM) is an image smoothing filter proposed on the basis of improving the accuracy of non-local filtering weights [16]. It has good edge preservation and denoising performance. It is applied to the unsharp mask. In the image enhancement framework, images can be decomposed more accurately. In view of this, this paper proposes a robot infrared image enhancement method based on Retinex theory and modified non-local mean for the problem of robot infrared image contrast and detail enhancement. This method makes full use of Retinex, the adjustment effect of theory on image gray distribution and the good performance of probabilistic non-local mean filtering in image detail extraction have achieved good results in image contrast and detail enhancement.

Retinex theory
Retinex is composed of two phrases: Retina and cerebral Cortex. Retinex theory, as a brightness and color perception model of human vision, reveals that the color changes of objects are not affected by changes in the lighting environment, but it is caused by the reflection of the surface of the object and has consistency. According to this theory, the observation image F(x,y) can be decomposed into two different images, namely the reflection image R(x,y) and the illuminance image L(x,y), where L(x,y) can be regarded as uneven illumination, which is the main factor affecting image quality, while R(x,y) is considered to be the original image that is not affected by the outside world. Its mathematical model is expressed as: From the mathematical model of Retinex theory, the illuminance image L(x,y) can be regarded as the multiplicative noise attached to the ideal image R(x,y). The purpose of image enhancement is to obtain from the observation image F(x,y), remove the influence of L(x,y), and restore the essential information of the scene, that is, to achieve the purpose of enhancing the image. Singlescale Retinex (SSR) image enhancement algorithm will convert formula (1) to the logarithmic domain to calculate the incident image R(x,y). SSR can be expressed as: In the formula, * is the convolution operator, G(x,y) is the standard surround function, , . Using the observation image F(x,y) to estimate L(x,y). The infrared image enhancement method L(x, y) based on Retinex theory and probabilistic non-local mean, by processing the ratio between the pixel in F(x, y) and its surrounding weighted mean to obtain an estimate of the illumination The Gaussian function used in SSR is expressed as: where, σ represents the standard deviation of the surrounding neighborhood, which controls how much detailed information in the image is retained. It directly affects image enhancement. ω is the normalization factor. Finally, R(x,y) can be obtained from equation (4). Contourlet-based non-local mean via Retinex theory for robot infrared image enhancement

Non-local mean (NLM)
NLM uses the local self-similarity existing in the image to calculate the weight of the neighboring pixels, which can be expressed as: In the formula, i and j represent the pixel points in the image.
j v is the pixel value of pixel point j .
Traditional NLM uses an exponential function to calculate the weight. Due to the defects of the exponential function, the weight calculation will be inaccurate. In response to this problem, new NLM abandons the Gaussian weight function and proposes a new probability weight function, namely, In the formula, j i f , are the probability density functions of random variables j i R , , the distance between the two pixels is recorded as is the estimated distance of estimated variance 2 2 .  is the tuning parameter, and the degree of smoothing of the probability non-local means can be controlled by adjusting the size of  .  is approximately large, and the image is smoother, but the edge details in the image will become blurred. When two image regions have overlapping parts, j i R , does not obey independent and identical distribution. It can be considered that the distribution is approximate: In the formula, where, ) ( E and ) var( represent the mean and variance, respectively. When cause the probability density function j i f . to be different. According to the above analysis, the probability density function is established by constructing the Euclidean distance of similar blocks, and finally the probability distribution function determines the size of the weight. It is precisely because the calculation of the NLM weight does not consider the irrelevance of the space, the weight value is inaccurate. And the probability distribution function can better reflect the similarity between image blocks and achieve better smooth filtering results.

Proposed robot infrared image enhancement
At present, the existing image enhancement methods can amplify the noise while enhancing the weak edge of the image. This is because they cannot distinguish between weak edges and noise. In the frequency domain, both weak edges and noise get low amplitude coefficients. Contourlet transform can not only provide multiresolution analysis, but also obtain geometric information in the image [17,18].
In order to enhance the image transform coefficient and suppress the noise, it is necessary to determine which transform coefficient is generated by the noise. However, noise can not be determined only by the value of wavelet coefficients, because the transformation coefficients of some details in the image are very small. The size is about the same as the noise, and if you blindly suppress all the wavelet coefficients below the set threshold, some details in the image will be lost. Contourlet transform coefficients satisfy the edge non-Gaussian statistical characteristics, and the correlation between adjacent coefficients is strong. The high frequency coefficients generated by noise in Contourlet transform domain usually have little correlation with the high frequency transform coefficients in adjacent subband layers. The correlation coefficient between high frequency Xi Zhang and Jiyue Wang 4 coefficients of wavelet generated by details is large. Therefore, correlations between coefficients can be used to determine which are generated by noise and which are generated by detailed features in the image. High frequency coefficients generated by noise and detail are processed differently. According to the fact that signal energy forms clustering in Contourlet transform domain. The detail coefficient and noise coefficient can be separated by morphological expansion operator.
A nonlinear mapping function is used to modify the Contourlet transform coefficients.
is the multiple of amplification. The c value is related to the noise content in the image. The following formula can be used to estimate c and obtain the estimated standard deviation of noise in the image.
This function preserves the value of the large coefficient in the transformation, amplifies the detail coefficient, and sets the noise to zero.
By analyzing single-scale Retinex method (SSR) and Contourlet-based non-local mean filtering method (CNLM), a robot infrared image enhancement method combining SSR and CNLM is proposed. Aiming at the deficiency of low contrast and unsatisfactory visual effect of infrared image, the method uses SSR algorithm and CNLM filter to decompose and process the original infrared image at different levels, which improves the level of infrared image and improves the visual effect of infrared image. The specific operation flow of the algorithm is shown in figure 1. The algorithm is mainly divided into three parts: image gray scale adjustment based on SSR, image detail and contrast enhancement and result fusion. Infrared imaging belongs to temperature difference imaging. Generally, the temperature difference between the background and the target in the scene is small. Therefore, the gray distribution area of the infrared image is often narrow, the image contrast is low, and some scenes are even submerged in the dark background. In the Retinex theory, the image degradation is caused by uneven illumination. Applying this theory to infrared images, the factors that affect image quality can be regarded as the illuminance image L(x,y) superimposed on the ideal image. Therefore, these factors can be removed by using the SSR algorithm. Thereby, it adjusts the grayscale distribution of the dark and bright parts of the image, highlights the scene details in the dark part, and compresses the grayscale of the bright part, so that the image meets the requirements of the human eye. Suppose the input image is A(x,y), ) ( SSR is a singlescale Retinex operator, then the infrared image obtained by SSR processing can be expressed as: In the formula, (x,y) represents the coordinates of the pixels in the image. Since the Retinex theory assumes that the illumination changes are gentle, Gaussian filtering is usually used in the SSR algorithm to estimate the illumination image. So that the strong boundary in the image is correct. The estimation of illumination will produce deviations, and then produce halo phenomenon at the edge of the scene. Therefore, the anisotropic diffusion equation is used to estimate the illumination image. In the process of smoothing the image, the anisotropic diffusion equation is for the image in the smoothing process. The boundary part and the stable area of the image can be adaptively adjusted to the smooth scale, so the boundary of the image is better maintained. The use of anisotropic diffusion equation to estimate the illuminance image can more accurately estimate the illumination at the boundary in the image and avoid the image with the halo phenomenon. The anisotropic diffusion equation is defined as: In the formula, div(·) represents the divergence operator,  and Δ represent the gradient operator and the Laplacian operator, respectively.
where, k is the adjustment parameter, which can adjust the size of the diffusion coefficient to control the smoothness of the anisotropic diffusion equation for the image. The larger k value denotes the smoother obtained image, but at the same time, the details in the image are also vaguer. Figure 2 shows the processing results of the SSR algorithm in this paper on an infrared image. It can be seen from the figure that the illumination image obtained by the anisotropic diffusion equation takes into account the sudden change of the image edge illumination and avoids the halo phenomenon. It can be seen from  Xi Zhang and Jiyue Wang 6 At this time, the extracted detail layer is mainly the highfrequency information in the image. In addition to the image detail texture and edge information, this information also contains a part of noise. In order to suppress the noise information in the detail layer, At the same time, the detailed texture and edges are more prominent, and the non-linear function is selected to process the detail layer. The edge of the image texture is further enhanced and the noise is suppressed. The nonlinear function is: Figure 3 shows the use of CNLM to decompose images, obtain detailed information, and enhance details. Figure  3(b) is the basic layer. Through the processing of CNLM filtering, the noise in the image is filtered out, and the texture details are also filtered out to a certain extent. But due to the better edge retention performance of CNLM filtering, there is no serious blurring in the filtered image. The contrast of figure 3(b) is not much different from the original image, so further contrast is required to enhance. We use the histogram equalization algorithm to process D(x,y) and get the basic layer after contrast enhancement.   (19) and the enhanced detail layer obtained by equation (20). It can be seen from the figure that the detail texture is enhanced by nonlinear function.

Image fusion
The contrast-enhanced image where  is the adjustment parameter, which is used to adjust the degree of influence of the illuminance image on the enhanced image. In this new algorithm,  =0.2 can make the thermal radiation distribution of the enhanced image consistent with that of the original infrared image and avoid excessive enhancement.

Experiment and analysis
In order to objectively evaluate the performance of the proposed algorithm in this paper, four sets of robot infrared images are used to test several image enhancement algorithms. The four sets of infrared images are: 1) the training images in the school; 2) the road, people and trees; 3) House and car; 4) Bridge. The size of the first two and fourth infrared images is 256×256, and the size of the third image is 240×320. Both subjective and objective methods are used to evaluate the robot infrared images. The software operating environment of the experiment is MATLAB 2017a and Windows10. The hardware environment is a notebook computer with Inter(R)Core(TM)i3-3220 CPU@3.3GHz and 2GB memory. We select histogram equalization (HE), homomorphism filtering (HF), Single-scale Retinex (SSR) and Contrast-Limited Histogram Equalization (CLAHE) image enhancement methods to make comparison in this article.

Subjective evaluation
Figures 4~7 are the enhancement results of the four images. Figure 4 is the enhancement result of the character shooting map. From figure 4(a), it can be seen that the original image contains people, distant trees and houses, and the overall gray of the image is low. The trees in the distant house are blurred. From figure 4(b)~(f), it can be seen that the overall gray level of the image enhanced by the HE method has been greatly improved, and the contrast of the ground part has been significantly improved. However, the image processed by this method has been over-enhanced. After the shooting person is processed by the HE algorithm, only the highlighted outline is left, and the details such as the texture of the clothes and the hair disappear. This reason is that the gray level of the details by HE method with a lower probability of occurrence is inhibited, so when the contrast of the entire image is enhanced, the texture details in the image are suppressed. The HF filter enhancement effect is softer, and the image brightness is improved overall, but the overall image contrast and contour details have not been significantly improved. The SSR algorithm has a certain effect on the detail enhancement of the image, especially for the background of trees and houses in the dark. But the overall brightness improvement is not obvious. The CLAHE method is the improvement of the HE method, it can also be seen from the enhancement effect that this method has better overcome the defects of the HE method of excessive enhancement and suppressed details in the image. It has a good performance in image contrast and detail enhancement. After the method is enhanced, the texture details of the trees, houses, and the clothes of the characters in the image background are more abundant, and the contrast of the image has also been greatly improved, and the visual effect is better.
The scene in figure 5 includes cars and trees. The original infrared image is generally dark, and the gray levels of trees and cars are low, the details cannot be distinguished. After processing by several enhancement algorithms, it can be seen that the HE algorithm also has excessive enhancement. Especially when the front wheel of the car is too bright, the visual effect is not good. HF and SSR algorithms have also achieved a certain enhancement effect, but the enhancement effect on details is not obvious. The new algorithm in this paper and the CLAHE enhancement algorithm have the best effect. The comparison clearly shows that the new algorithm in this paper is better in enhancing image details. The enhancement results of figure 6 and figure 7 are more consistent with the results of the first two images. From the simulation experiment results of these four infrared images, it can be seen that this new algorithm has shown good performance in the details and contrast enhancement of scenes such as characters, sky clouds, houses and trees.

Objective evaluation
In order to objectively compare the performance of different enhancement methods on the enhancement effect of infrared images, two evaluation indicators (Enhancement by Entropy (EME) and Average Gradient (AVG)) are used to measure the enhancement effect of the image [19][20][21][22]. EME is a commonly used evaluation index in enhancement. It uses the block method to obtain the approximate contrast of the image. The larger value denotes that the image has a better visual effect. It is defined as: When using EME to measure image quality, it is necessary to divide the image into blocks, divide the image into K1×K2 blocks, and find the maximum  Table 1 and Table 2 are the results of EME and AVG, respectively. From the table, it can be seen that the two indicators with the new method in this paper have achieved better results, so the image contrast and visual effect after processing by the new method in this paper are better. The image quality is better.

Conclusion
This paper proposes an infrared image detail enhancement method based on contourlet-based non-local mean and Retinex theory. The original infrared image is decomposed at different levels through the single-scale Retinex algorithm and contourlet-based non-local mean.
And different enhancement methods are used for different levels. The new method is used to enhance the contrast and details of the image, and finally the enhancement results of different levels are fused to obtain the final enhancement result. Experimental comparison and analysis with a variety of methods in terms of the subjective and objective aspects, the results show that the processed image by the new proposed method has richer contents, more obvious details and textures. However, the new algorithm in this paper is more computationally complex, and the real-time performance of the algorithm needs to be further improved. In the follow-up work, realtime performance should be improved to adapt to the needs of practical applications.