Airport localization based on contextual knowledge complementarity in large scale remote sensing images

Localization of airport object region is an important task of object detection, which also is a significant application in remote sensing image processing. Because a single feature cannot fully describe the object, some non-airport images with flat line characteristics (such as road) or similar texture characteristics with the airport can be detected as the existing airport regions, which brings difficulties to subsequent object recognition and change detection. To address the above problems, a new airport localization method based on contextual knowledge complementarity in large scale remote sensing images is proposed. This new method first makes the utmost of the contextual information of the airport region in remote sensing image to construct a feature dictionary base. This dictionary base contains shallow vision knowledge and high semantic knowledge. Then the saliency maps of extracted knowledge are obtained and fused. A simple Ostu segmentation method is adopted to remove the false alarms and obtain the final airport region. It also gives the relative position coordinate and acreage of the detected airport. Compared with two state-of-the-art airport extraction methods through abundant experiments, the results show that the proposed airport localization method can fast and accurate locate the airport region, which has better effectiveness and robustness.


Introduction
Recently, remote sensing technology has been developed rapidly. It has a closely relation with the national economy, ecological protection and national defense security. For example, remote sensing has played an important role in land resource survey, ecological environment monitoring, agricultural monitoring and crop yield assessment, disaster prediction and disaster assessment, marine environment survey, weather forecast, air quality monitoring, electronic map and navigation [1][2][3]. Especially, region localization can highlight the progress of remote sensing technology.
Airport, as an important traffic facility, has great significance in both military and civil fields. The automatic detection of airport has an eminent impact on remote sensing applications and plenty of scholars have paid perceptive attention to it. However, airport detection also faces many difficulties, such as complex and changing background and distinct shape of airports. The airport detection performance is also affected by road signs and other ground objects interference.
Many researchers have proposed advanced methods for airport detection mainly containing three detection strategies: edge contour-based, region segment-based and model-based methods. The edge contour-based airport detection deals with line features of runway. Tang et al. [4] proposed a novel method for airport detection where the entire algorithm based on line segment classification and texture classification. Firstly, a fast line segment detector was applied to extract the line segments in images and computed the features of these line segments. Then, the line segments were discriminated by a trained runway line classifier, and the regions of interest (ROIs) were extracted from the line segments, which were classified as runway lines. Finally, whether the ROI was actually an airport was determined by analyzing the classification results of the image blocks. Budak et al. [5] presented a two-stage method for airport detection for remote sensing images. In the first stage, a new algorithm composed of several line-based processing steps was used for extraction of candidate airport regions. In the second stage, the scale-invariant feature transformation (SIFT) and Fisher vector coding were used for efficient representation of the airport and non-airport regions and support vector machines employed for classification. Meng et al. [6] proposed runway detection based on line feature detection of partitioned remote sensing images. Initially, overlapped partitions were carried out on these images, and line feature detection was performed on each part by introducing the line segment detector(LSD). Then, based on the mathematical features of airport runways,the extracted line features were divided into several parallel line groups according to angles [7,8]. After line growing and merging, candidate airport runway areas were obtained with the help of Radon transform. It has several advantages such as simplicity of method and fast speed. But it does not make optimal use of line features and other non-airport targets with line features will also be detected at the same time. The region segment-based airport detection method mainly utilizes the texture difference between the airport and surrounding landscape. Halici et al. [9] proposed a runway detection method based on textural properties. Since the best discriminative features for airport runways could not be trivially predicted, the Adaboost algorithm was employed as a feature selector over a large set of features. Thus, the Adaboost-based selected feature subset could be used for both detecting runways and identifying their textural characteristics. Temizkan et al. [10] presented a simple and efficient airport detection method using both the geometric and texture features. Because of the image segmentation method, the description of the airport area is more accurate and has higher positioning accuracy, but the complexity is higher and the computing speed is slower.
The airport detection based on model aims to build a system, extracting local invariant feature points from the corresponding positions of the original image and the image to be tested, then matching them to detect the existence of the airport. Zhao et al. [11] designed a reinforcement learning structure to suppress background and highlight interesting airport regions level by level. To generate a final saliency map, the bottom-up region features with top-down line feature based on target attribute was fused, which could restrain other salient regions except for airports. Moreover, a learning stop criterion based on Latent Dirichlet Allocation (LDA) model was proposed at each level to judge the state of saliency detection, thus learning process could be adaptively controlled. Besides, a back-level propagation mechanism was employed to reinforce airport target between levels. Yin et al [12] proposed an improved faster region-based convolutional neural network (RCNN) detection method for airport detection in large scale remote sensing images. Multi-scale training was applied to faster RCNN to enhance the robustness of network for detecting airport with different sizes. Meanwhile, the modified multitask loss function was adopted to improve the accuracy of airport detection. Online hard example mining strategy was introduced to decrease the redundant negative samples in the training process. Then the non-maximum suppression method was used to remove the redundant boxes of the detected airport. Yin et al. [13] proposed an accurate and fast region search in optical remote sensing images under cloud computing environment, which was based on hybrid convolutional neural network. The proposed region search method was partitioned into four processes. First, fully convolutional network was adopted to produce all the candidate regions that contain the possible object regions. This process avoided exhaustive search for input images. Then, the features of all candidate regions were extracted by a fast region-based convolutional neural network structure. Third, he designed a new difficult sample mining method for the training process. At the end, in order to improve the region search precision, an iterative bounding box regression algorithm was used to normalize the detected bounding boxes, in which the regions contain candidate objects. The proposed algorithm was evaluated on optical remote sensing images acquired from Google Earth.
Currently, some researchers developed more effective models based on convolutional neural networks (CNNs) for detecting airport in optical remote sensing images such as [14][15][16][17]. However, a large amount of training samples is required which is difficult to collect needs severe efforts for remote sensing imagery. Aiming to obtain the accurate location of airports, a new airport localization method based on contextual knowledge complementarity in large scale remote sensing images is presented. A feature dictionary based on the contextual information of the airport region in remote sensing image is constructed. Features of different knowledge of shallow vision and high semantic are extracted. Then the saliency maps of extracted knowledge are obtained and fused. Finally, a simple Ostu segmentation method is utilized to remove the false alarms and obtain the final airport region. We also give the relative position coordinate and acreage of the detected airport. The rest of this paper is organized as follows. In Section 2, we elaborate the methodology in detail. Experiments and analysis are described in Section 3. Finally, the paper is concluded in Section 4.

Establishing prior knowledge of airport region
It can be seen from the visual perception characteristics of human eyes that people's cognition of scenes is usually based on the present things in the scene. But for the discrimination of scenes, the detection of target is particularly important [18]. The saliency detection algorithm mainly focuses on the description of the target contour information and does not distinguish the region saliency of the target. Furthermore, it only focuses on the extraction of the target itself, not on the correlation between the targets, which exhibits several limitations. Moreover, simple target detection and combination cannot accurately represent the content reflected by the image scene. Therefore, it extracts the saliency region that is not only significantly different from the surrounding adjacent regions, but also different from the image as a whole fully considering the semantic relationship between contexts in the image. In general, if the color difference between image blocks is higher and the distance is closer, then the saliency is stronger. On the basis of extracting the saliency region of scene image, the context relationship between significant objects is emphasized.
According to the previous study on airport detection, this paper can summary the following knowledge. According to the above knowledge, it constructs the airport localization model. Figure 1 is the proposed airport localization framework.

Shallow vision saliency extraction
Brightness is related to the luminous intensity of the object being observed. It looks brighter or darker depending on its strength. For the same object, if the irradiation light is stronger, the reflection light is stronger too, then it will be brighter. For different objects, if the reflection is stronger, it looks brighter under the same irradiation. Meanwhile, color features account for a large proportion in the process of calculating image saliency. a. Extracting brightness feature and brightness feature saliency map. The brightness map is extracted using the following formula.
Where R, G and B are the pixel values of the three single-color channels of red, green and blue, respectively. I represents the brightness eigenvalue. The feature saliency is obtained by calculating the Difference of Gaussian (DOG) between the central region c and the surrounding region s of the image.

High semantic saliency extraction
According to previous observations, runway, as an important part of the airport, can be used for assisting airport localization. LSD (line segment detector) algorithm is used for line detection of image [19]. LSD has the advantages of fast calculation speed and without parameter dependence. The disadvantage is that the straight line will break at the intersection. In order to make up this defect, the straight line connection method is adopted to connect the straight lines with similar directions and close distances. When the remote sensing image resolution is constant, the straight length of the runway is within a certain range. The abnormal lines contains too short, too long and multiple straight lines. In order to avoid their interference, these straight lines are removed after straight line connection. If the two lines are closer, then they are more parallel. In this paper, the fuzzy value between [0,1] is used to measure the parallelism of two lines. Assuming that the inclination angles of the two lines are a  and b  , respectively. The angle between the two lines is: We define the parallelism metric function as:

Ostu segmentation
Ostu [20] divides all the pixels into two categories, the background pixel and the target pixel. According to the principle of probability statistics, variance is a measure of the uniformity of gray distribution. If the variance value is higher then the difference between the two parts of the image will be greater. If the variance between the two groups of segmented images is maximized, then the error of the two groups of data is minimized. In a simple way, the optimal threshold is determined by maximizing the inter-class variance of the two groups of segmented images, so as to achieve the best separation between different classes. Firstly, the occurrence probability of each gray level is calculated according to the histogram, and the gray level is divided into two categories by threshold variables, and then the inter-class variance of each category are calculated. The threshold variables make the outer-class variance the largest and the intraclass variance the smallest which will be selected as the optimal threshold.

Experiment and analysis
To illustrate the effectiveness and robustness of the proposed airport detection method, we conduct experiments on remote sensing images with Intel(R),Core(TM) i7-8750H, CPU 2.20GHz,32.0 GB MEMORY, Windows 10 and GEFORCE GTX1060 [21][22][23][24]. The experiment data is taken from Google Earth with the resolution ranging from 5 to 40m. The size of images is from 600×600 pixels to 2048×2048 pixels. These images include both large international airports with complex structures and small airports with simple structures. The total number of images is 200.
In the experiment, the detection rate (DR), false alarm rate(FAR) and average detection time (AT) are used to evaluate the algorithm, and the DR and FAR are defined as:

Saliency map analysis
Based on contextual knowledge complementarity, we obtain the saliency map at different stages as shown in figure 2. As can be seen from figure 2, the outline of the airport can be well depicted.

Airport localization result
We conduct abundant experiments and obtain better results, but due to the limited space, only some results are displayed in figure 3.  Experimental results show that this proposed airport localization algorithm can accurately locate the location of the airport and has strong applicability. It has achieved good detection results for airports with different complex scenes, different types and different shapes. Figure 4 is the accuracy rate curve of the detected airports. There are some false alarms and missed detections in the results, which are mainly caused by the simple structure of some airports and the weak linear characteristics of runways, so the effective localization and detection cannot be carried out according to the runways. In addition, some urban areas have a high similarity with the airports, and their interference cannot be completely excluded, so there are some false alarms.

Comparison with other airport localization methods
In this subsection, we make comparison with other stateof-the-art airport detection method including SOACM [25] and HRL [11]. SOACM is an efficient airport extraction framework for RSIs. It combined vision-oriented saliency and knowledge-oriented saliency for the airport position estimation. And a saliency-oriented active contour model (SOACM) for airport contour tracking, where a saliency orientation term was incorporated into the level-set-based energy functions. The average comparison results are given in table 2. Figure 5 is the detection rate within 400 iterations. This table and figure describe that our proposed method has better results than other methods.

Discussion
The airport localization results are exhibited in figure 3.
In here, the red boxes are detected airports and the yellow boxes are false detected airports. Overall, the proposed method with contextual knowledge complementarity method can precisely locate the airports under complex scenes such as multilane roads, occlusion and cross track. Although the non-airports cannot be completely avoided, we significantly reduce the number of false airports by our proposed method in this paper. Table 2 is the comparison result with HRL, SOACM and our proposed method. The detection rate of proposed method increases by 0.9\% and 2.8\% respectively than HRL, SOACM. The time declines to 6.38s than HRL, SOACM. Meanwhile, the line feature dictionary is applied into the fusion model to verify the effectiveness of our proposed method, which then achieved a better result than the traditional saliency-based methods. Finally, through the rich experiments, a conclusion can be concluded that the saliency module can significantly improve the detection rate and decrease the false rate.

Conclusion
To improve the airport localization result, we propose a new airport localization method based on contextual knowledge complementarity in large scale remote sensing images. First, the contextual information of the airport region in remote sensing image is adopted to construct a feature dictionary base. This dictionary base contains shallow vision knowledge and high semantic knowledge. Then the saliency maps of extracted knowledge are obtained and fused. Furthermore, a simple Ostu segmentation method is used to remove the false alarms and obtain the final airport region. Experimental results show that the proposed algorithm has stronger robustness, higher detection rate and lower false alarm rate than other methods. In the future, we will improve the accuracy of saliency detection and combine the latest neural networksbased method to enhance the airport detection efficiency.