1. Introduction

Different sensors have different descriptions for the same scene. Infrared sensors are sensitive to high heat radiation within the region. They can extract targets according to the infrared energy difference between target and background, which can pass through a certain thickness of soil layers and even concrete layers. However, compared to Synthetic Aperture Radar (SAR) imaging, infrared imaging is vulnerable to the influence of clouds, rain, and fog. SAR is an irreplaceable reconnaissance tool due to its advantages of all day, all weather, and long detection range. However, in some cases, the image information obtained by a single SAR is not enough to be used for better analysis and understanding of the target or scene^[1]. Therefore, combined with the advantages of SAR reconnaissance and infrared reconnaissance, the research of SAR image and infrared image fusion can greatly improve the reconnaissance efficiency. The “LANTIRN” pod on the American F-16 fighter takes infrared reconnaissance as the main means of low altitude reconnaissance, and combines it with SAR reconnaissance to play a good effect. The Ref. [2] takes the fusion between the SAR data and the infrared data as one of the core issues of missile multi-mode guidance. The fusion between infrared image and SAR image can help to output a fused image which is more suitable for human visual perception or computer processing and analysis. It can significantly improve the lack of information obtained by a single sensor, improving the clarity of the resulting image and information content, which is conducive to more accurate, more reliable, more comprehensive access to the target or scene information. It is mainly used in military operation, national defense, resource survey, and other fields.

In recent years, the methods based on multi-scale decomposition have received extensive attentions in image fusion^[3–6]. However, there are some drawbacks in these methods. Firstly, some multi-scale decomposition tools lack of shift invariance, or some do have the shift invariance, but their computational complexities are quite high. Secondly, the low-frequency components obtained by multi-scale decomposition tools are the approximate representation of the source images, in which the number of pixel grayscales close to zero is small, as a result, the low-frequency information of the source images cannot be described sparsely, and it is not convenient to capture the salient features of the source images. Therefore, in this paper, we apply the Complex Contourlet Transform (CCT) proposed in Ref. [7] to the remote sensing image fusion. This multi-scale decomposition tool is fast and shift-invariant, which can reduce the influence of the low accuracy of image registration on the fusion results. In Ref. [8], the complex contourlet transform is applied to image denoising and has achieved relatively good results, but the use of complex contourlet transform in image fusion is still in the exploratory stage. In recent years, Sparse Representation (SR) has been applied to image fusion as a new signal processing model. The image fusion method using the sparse representation model or the joint sparse representation in the Refs. [9,10] improves the image fusion effect. But the two methods directly carry out the fusion in the sparse representation domain. Considering that multi-scale decomposition tool can describe the details of the image from multiple scales, if the multi-scale decomposition tool is not used, the fusion image cannot inherit the detailed information of the source images well. In Ref. [11], a method of fusion between an infrared image and a visible image based on Non-Subsampled Contourlet Transform (NSCT) and sparse representation is proposed. However, the combined use of sparse representation and NSCT has a high computational complexity. In addition, in view of the grayscale difference between the infrared image and the SAR image and the interference of the speckle noise in SAR image, if the low-frequency component is directly fused without sparse representation, it may result in confusion of pixels and the target in the fusion image is not significant.

To this end, an image fusion method in CCT domain based on joint sparse representation is proposed to fuse the SAR image and the infrared image. The fused image via the proposed method combines well the advantages of the SAR and infrared images and has a better visual quality.

2. Complex Contourlet Transform and Joint Sparse Representation

2.1 Complex contourlet transform

Complex contourlet transform is obtained by combining contourlet transform with double-tree complex wavelet transform. The principle of this transform is that: after the original image being decomposed by double-tree complex wavelet transform, the double-tree structure is formed. Then the 2-dimensional Directional Filter Banks (DFB) are used to separate the high-frequency components in six directions, hence the sub-bands can be expanded to the numbers of 2ⁿ. The essence of CCT is to replace the Laplacian Pyramid (LP) filter structure in the contourlet transform with the double-tree structure in the Dual-Tree Complex Wavelet Transform (DT-CWT), so as to replace the original single high-frequency component with the high-frequency components in the six directions, thus the high-frequency components can better capture the details of the image. CCT takes into account the amplitude and phase information of the original signal, and the decomposition speed is fast. Meanwhile, it retains the property of shift invariance. The principle of CCT is shown in Fig. 1.

Figure  1.  Schematic diagram of CCT

DownLoad: Full-Size Img PowerPoint

2.2 Joint sparse representation

By using an over-complete dictionary matrix that contains M atoms, a signal can be represented as a sparse linear combination of these atoms, thus revealing the essential features of the original image more sparsely. The mathematical definition of the sparse representation model is:

where ${{α}}$ represents the sparse representation coefficient, ${{x}} \in {{{R}}^M}$ is the original signal, ${\left\| \cdot \right\|_0}$ denotes the l₀ norm counting the non-zero entries of a vector, $\varepsilon$ refers to the error tolerance and $\varepsilon \ge 0$ .

The Joint Sparse Model (JSM) has been developed from the theory of sparse representation. Then JSM-1, JSM-2, and JSM-3 were proposed^[12] in succession. These models consider that each original signal contains both a sparse portion common to all signals and a unique sparse portion of each signal. Each signal in the signal ensemble ${{Γ}}$ can be expressed as:

where ${{{V}}_{\rm{c}}}$ represents the common part of the entire signal ensemble, ${{{V}}_{{u_i}}}$ denotes the unique part of the i-th signal, ${{{s}}_{\rm{c}}}$ refers to the common sparse representation coefficient of the entire signal ensemble, and ${{{s}}_i}$ is the sparse representation coefficient of the unique part of the i-th signal. The joint sparse representation model of the signal ensemble is:

where ${{V}} \!=\! \left[\!\! {\begin{array}{*{20}{c}} {{{{V}}_1}}\\ {{{{V}}_2}}\\ \vdots \\ {{{{V}}_K}} \end{array}}\!\! \right],{{{D}}_{{\rm{JSR}}}} \!= \!\left[\!\! {\begin{array}{*{20}{c}} {{D}} & 0 & ·\!·\!· & 0\\ 0 & {{D}} & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0\\ 0 & ·\!·\!· & 0 & {{D}} \end{array}}\!\! \right],$ ${{S}} = \left[ {\begin{array}{*{20}{c}} {{{{s}}_{\rm{c}}}}\\ {{{{s}}_1}}\\ \vdots \\ {{{{s}}_K}} \end{array}} \right]$ .

3. Fusion Method of SAR Image and Infrared Image

3.1 Fusion of low-frequency components

The low-frequency components obtained by CCT are the approximate representation of the source images, but their sparseness are not enough. Considering that the acquired multi-source remote sensing images are the descriptions of the same scene from different aspects, there exists a certain correlation between the low-frequency components of the two source images, i.e. there is joint sparsity between the low-frequency components of the images to be fused, while there are some differences between them. Therefore, for the fusion of low-frequency components of the original images, the joint sparse representation is implemented on them. Thus the common features and the unique features of the low-frequency components of the image to be fused are distinguished, the fusion is performed by selecting the unique features with a larger l₁ norm, while the common features remain unchanged. Specific steps are as follows.

Step 1 　Create the training sample set. Given the low-frequency components of the two images to be fused are L₁ and L₂, the sliding window (the step size is 1) is used to form a series of 4×4 image blocks in a row-first manner. Then, all image blocks are reorganized into column vectors V₁, V₂ in a row-first manner, and the training sample set is chosen from them randomly.

Step 2 　Joint sparse representation. The matrix V₁ and V₂ obtained in Step 1 are merged into a union matrix V₃. The K-SVD^[13] method is used to train the samples to construct the dictionary of V₃. According to the joint sparse representation model:

Then OMP method^[14] is used to find the sparse representation coefficients for Eq. (4).

Step 3  Fusion of sparse representation coefficients of low-frequency components. The fusion consists of two parts: the selection of the activity evaluation index and the design of the fusion rule. The l₁ norm of is used as the evaluation index of the activity degree. Let the sparse representation coefficient after fusion be S_F, then S_F = ${{{s}}_{\rm{c}}} + \mathop {\arg \;\max }\limits_{{{{s}}_i}} \left( {{{\left\| {{{{s}}_i}} \right\|}_1}} \right){\rm{ }}(i = 1,2)$ , namely, the fusion is performed by selecting the unique features with a larger l₁ norm, while the common features remain unchanged, so the fusion vector matrix of the low-frequency components of the two source images is ${{{V}}_{\rm{F}}} = {{D}}{{{S}}_{\rm{F}}}$ .

Step 4 　Reconstruction. Reconstructing the low-frequency component by V_F is an inverse sliding window process, namely, the column vectors of the fusion vector matrix V_F are restored into the image blocks. Since the step size of the sliding window is 1, there is a partial overlap between the adjacent image blocks. Thus the overlapped parts of the adjacent image blocks are subjected to weighted averaging to obtain the fused components of the low-frequency components.

3.2 Fusion of high-frequency components

The high-frequency components of the image contain details of the source images, such as textures, edges. The larger the coefficients of the high-frequency components, the richer the information of the region where the central pixel is located. When the central pixel of the image local region is the target pixel, the grayscales of the local region are more discrete and the region information entropy is larger. When the central pixel of the image local region is the background pixel, the grayscales of the local region are less discrete and the region information entropy is smaller. However, when the background information remains in all directions of the high-frequency components, the region information entropy is larger, and the region energy is larger as well. It is possible to distinguish the background pixels from the target pixels using the visual sensitivity coefficient based on the fact that the human eye is more sensitive to local changes in the image^[15]. In addition, considering the fact that the discrete degree of the grayscales of the local area where the target pixels are located is generally larger than that of the background pixels, the fusion rule of the high-frequency component is designed by combining the advantages of the visual activity coefficients with the energy matching degree. The fusion rules of high-frequency components are designed so that the high-frequency components after fusion can better inherit the detail information of the source images and improve the visual effect. The visual sensitivity coefficient ${{η}}$ and the energy matching degree ${{ρ}}$ are defined as:

where ${{{C}}^x}\left( {i, j} \right)$ represents the low-frequency component coefficient, ${{C}}_{k, l}^x\left( {i, j} \right)$ denotes the high-frequency component coefficient, $\overline {{{{C}}^x}\left( {i, j} \right)} =$ $\displaystyle\frac{1}{{M \times N}}\sum\limits_{m = 0}^M {\sum\limits_{n = 0}^N {{{{C}}^x}\left( {i + m, j + n} \right)} }$ , and x is the infrared image A or the SAR image B.

Let the energy matching degree threshold be T. When ${{ρ}}\left( {i, j} \right) \le T \ $ , the high-frequency components still remain the background information, then the fusion rules are as follows:

when ${{ρ}}\left( {i, j} \right) > T \ $ , the fusion rules are as follows:

The fused procedure of the proposed image fusion method based on CCT and joint sparse representation is shown in Fig. 2.

Figure  2.  The procedure of the proposed image fusion method

DownLoad: Full-Size Img PowerPoint

4. Experimental Results and Analysis

To evaluate the performance of the proposed image fusion method, the SAR images and infrared images of the same scene which were from the SAHARA project of the Royal Military Academy in Belgium are fused, as shown in Figs. 3(a)–3(f). Source images are in 256×256 size. The proposed image fusion method is compared with the method based on LP, the method based on Wavelet Transform (WT), the method based on NSCT, the method based on DT-CWT, and the method based on sparse representation in Ref. [11]. The experimental results by six methods are shown in Fig. 4, Fig. 5, and Fig. 6.

Figure  3.  Three groups of infrared images and SAR images

DownLoad: Full-Size Img PowerPoint

Figure  4.  Fusion results 1 by six methods

DownLoad: Full-Size Img PowerPoint

Figure  5.  Fusion results 2 by six methods

DownLoad: Full-Size Img PowerPoint

Figure  6.  Fusion results 3 by six methods

DownLoad: Full-Size Img PowerPoint

From Fig. 4, Fig. 5, and Fig. 6, it can be seen that the fusion image obtained by the LP fusion method is blurred, the overall brightness is relatively dim, the image local contrast is slightly low, and the target is not too salient. The WT fusion method has improved the overall brightness in the fusion image, but the edge is still blurred, and some parts of the target and background are mixed together. The fusion image obtained by the NSCT fusion method can better retain the contours of source images, but there is still a problem that the contrast is relatively low. In addition, the overall brightness of the fusion image is relatively dark. The result of DT-CWT fusion method is slightly worse than that of NSCT fusion method. Compared with the above-mentioned four methods, the overall brightness of the fused image is further improved by the method in Ref. [11]. Meantime, the contrast between the target and the background is improved. However, some obvious haloes appear in the image. For instance, there are obvious artifacts appearing in the right bottom part of Figs. 4(e), 5(e), and 6(e). The fusion image obtained by the method proposed in this paper has the best visual effect and no obvious artifacts. The overall brightness is more coincident with the human eye perception, and the image texture is continuous and the image details are clear. The image contrast is higher and the fused image inherits the original contour information of objects in the source images.

In this paper, six objective evaluation indices^[16], such as Information Entropy (IE), Mutual Information (MI), Correlation Coefficient (CC), Spatial Frequency (SF), Average Gradient (AG), Standard Deviation (SD), and running time (Time) are used to compare the experimental results of six different fusion methods. Tab. 1 gives the quantitative evaluation results of the six methods.

Table  1.  Quantitative evaluation of six fusion methods

Image group	Fusion method	IE	MI	CC	SF	AG	SD	Time (s)
Group 1	LP method	6.839	11.405	0.687	32.852	22.406	35.499	2.126
	WT method	6.724	11.119	0.869	25.667	17.947	29.863	1.827
	NSCT method	6.821	11.295	0.813	26.881	18.373	33.828	50.332
	DT-CWT method	6.759	11.236	0.719	25.691	17.881	32.614	3.765
	Method in Ref. [11]	6.887	11.411	0.788	24.312	16.955	32.022	67.782
	Proposed method	7.091	11.827	0.829	30.276	20.294	43.965	27.232
Group 2	LP method	6.973	12.070	0.739	29.114	21.005	33.825	1.728
	WT method	7.097	12.181	0.883	25.643	19.298	35.330	0.898
	NSCT method	6.895	11.808	0.729	23.456	17.272	33.062	39.590
	DT-CWT method	6.902	11.848	0.784	22.633	16.745	32.184	3.371
	Method in Ref. [11]	7.140	12.843	0.7833	30.390	22.565	37.019	53.945
	Proposed method	7.305	12.615	0.869	28.648	20.887	44.539	22.733
Group 3	LP method	6.633	12.141	0.834	28.629	18.687	33.825	1.351
	WT method	6.797	12.1241	0.803	25.643	19.298	35.330	0.557
	NSCT method	6.801	11.973	0.864	26.105	17.539	40.052	26.433
	DT-CWT method	6.665	11.723	0.846	23.539	16.745	32.184	1.371
	Method in Ref. [11]	6.796	12.234	0.812	25.518	17.244	37.274	37.274
	Proposed method	6.864	11.8401	0.836	28.689	18.982	40.084	18.613

 | Show Table

DownLoad: CSV

From the experiments, the fusion time of the proposed method compared with the fusion method in Ref. [11] is obviously reduced. Although compared with other classical fusion methods, this method has no great advantage in time, but the improvement of fusion accuracy must be at the expense of fusion time. As can be seen from Tab. 1, the information entropy and standard deviation of the proposed method are always higher than those of the other five methods, while other indexes are sometimes slightly lower than other methods. It shows that the robustness and overall performance of the proposed method are the best, which is consistent with the subjective analysis. The proposed method is superior to other five methods in terms of information entropy and standard deviation. It reflects that the fusion image contains more detail information and has a higher local contrast. Remarkably, the sparse representation fusion method in Ref. [11] has higher spatial frequency and average gradient for the fusion results of the second group of infrared image and SAR image. But actually the reason is that the method cannot discriminate the common features and the unique features of the low-frequency components of the source images, resulting in image distortion. In the proposed method, the low-frequency components of the infrared image and the SAR image are decomposed by complex contourlet transform, and the common features of the low-frequency components of the source images are distinguished from each other by the joint sparse representation. By combining the visual sensitivity coefficient and the energy matching degree to fuse the high-frequency components, the rich detail information of the two source images is captured. The fusion result can highlight the target and enhance the background, texture, and other details. On the whole, the proposed method is superior to the other five methods in the subjective visual effect and objective quantitative evaluation index.

5. Conclusion

A novel fusion method between the SAR and infrared image in complex contourlet domain based on joint sparse representation is proposed in this paper. The method can take full advantage of SAR and infrared image. Experimental results demonstrate that the proposed fusion method has a higher performance and a better visual quality.