Self-Controllable Super-Resolution Deep Learning Framework for Surveillance Drones in Security Applications

This paper proposes a self-controllable super-resolution adaptation algorithm in drone platforms. The drone platforms are generally used for surveillance in target network areas. Thus, super-resolution algorithms which are for enhancing surveillance video quality are essential. In surveillance drone platforms, generating video streams obtained by CCTV cameras is not static, because the cameras record the video when abnormal objects are detected. The generation of streams is not predictable, therefore, this unpredictable situation can be harmful to reliable surveillance monitoring. To handle this problem, the proposed algorithm designs super-resolution adaptation. With the proposed algorithm, the shallow model which is fast and low-performance will be used if the stream queue is near overﬂow. On the other hand, the deep model which is high-performance and slow will be used if the queue is idle to improve the performance of super-resolution.


Introduction
Recently, many research results have been introduced to deal with the use of mobile stations for enhancing communications and networking performance, e.g., (i) mobile computing for surveillance monitoring and (ii) cellular network coverage extension using drones. Among them, the use of drone wireless communications and networking is promising and a lot of related research contributions are now available for various applications and settings [1][2][3][4][5][6][7][8][9].
This paper considers the case where drone mobile platforms are used for surveillance. The drone mobile platforms are equipped with CCTV cameras and the cameras record the environment and transmit the recorded video streams to ground surveillance monitoring centers over wireless channels such as 5G millimeter-wave channels or LTE. In drone mobile platforms, CCTV cameras are equipped with computer vision functions, i.e., object detection and superresolution [10,11]. The super-resolution stands for the technology which is for enhancing the resolution of given images or video streams. Previously, various signal processing techniques are used for this purpose. However, various modern deep learning methodologies such as convolutional neural networks (CNN) and generative adversarial networks (GAN) have been used nowadays. With the function of object detection, the CCTV cameras are able to record video streams when abnormal objects are detected. Then, the recorded video streams are stored in the queues with the drone mobile platforms. Thus, the arrivals of the queues are not static. In order to improve the qualities of the video streams with the queues, super-resolution algorithms are used.
In this system setting, multiple super-resolution algorithms are implemented. Among these algorithms, the deep learning-based models are for high-performance learning whereas they take a longer time in terms of their computation. On the other hand, the shallow learning-based models are for fast computation whereas they take low-performance learning.
The proposed algorithm in this paper selects an optimal super-resolution algorithm among a set of candidate super-resolution algorithms depending on the queue-backlog size. If the system has a larger queue-backlog (near overflow), the super-resolution computation should be accelerated in order to speed up the departure processes. Thus, shallow models should be selected while sacrificing certain amounts of learning accuracy. On the other hand, if the system has an idle queue-backlog, the super-resolution computation can take more time in order to improve the performance of learning accuracy. In this case, the longer computation times with the deep models do not need to be strictly considered because the queue-backlog is idle. Note that the main objective of this paper is for improving the performance of super-resolution deep learning-based image processing in order to increase surveillance performance [12]. In this paper, a dynamic novel super-resolution adaptation control algorithm is designed for time-average learning performance maximization subject to queue stability by selecting the super-resolution algorithm among the given set, inspired by the Lyapunov optimization framework. Note that the Lyapunov optimization framework provides theoretical underpinnings for time-average utility function maximization subject to system stability.
Therefore, the motivation of this paper can be illustrated as follows. The proposed Lyapunov-optimization based dynamic super-resolution deep learning control algorithm maximizes time-average object recognition performance via super-resolution model selection subject to queue stability. Thus, our scheme achieves maximum performance while makes our systems be stabilized. Finally, it is obvious that our scheme guarantees optimal performance as well as is suitable for real-time computing platforms.
The rest of this paper is organized as follows. Sec. 2 presents the preliminary knowledge for our research presented in this work. Sec. 3 proposes the dynamic algorithm (to control) which is for super-resolution deep learning adaptation in drone mobile platforms. Sec. 4 verifies the proposed algorithm via simulations, and then, Sec. 5 concludes this paper.

Preliminaries
This section serves two objectives: 1) introducing a reference system model characterizing the context of deployment of our work and 2) introducing the time-average utility maximization via Lyapunov optimization, which is a key technique in this paper.

Reference System Model
Our considered reference system model is illustrated in Fig. 1. As presented in Fig. 1, a drone mobile platform using its own camera exists and its camera conducts object detection. If abnormal objects are detected, the streams will be recorded and the results are stored in the queue. Then, a super-resolution algorithm is used in order to improve the quality of streams. In Fig. 1, the super-resolution framework is illustrated and it consists of multiple super-resolution algorithms. If a super-resolution algorithm is deep, it takes more time and (often) achieves high learning performance, as presented in Table 1. Note that our considering super-resolution deep learning algorithm is based on VSDR [11] which is one of the most well-known CNN-based super-resolution algorithms.
On the other hand, if a super-resolution algorithm is shallow, it consumes less computation time whereas its learning performance is (often) not as good. Thus, a trade-off exists between learning performance and delays. In order to address this state of trade-off under constraints, this paper designs an adaptive control algorithm that can flexibly select super-resolution deep learning algorithms depending on the queue-backlog.

Lyapunov Optimization
The Lyapunov optimization framework, which is based on stochastic optimization (time-average utility maximization while achieving queue stability), is widely applied in the literature [24], with applications that include the following: • Adaptive video streaming: Kim et al. [13,14] and Bethanabhotla et al. [15] design a novel control algorithm for time-average streaming quality (modelled with peak-signal-to-noise-ratio (PSNR)) maximization subject to transmit buffer stability in wireless video networks. In addition, Koo et al. [16] propose a novel dynamic adaptive streaming over HTTP (DASH) mechanism for video streaming quality optimization under the consideration of battery status, LTE data quota, and buffer stability in integrated LTE and IEEE 802.11 wireless networks.
• Wireless networks: Neely, et al. [17,18] propose a novel dynamic routing algorithm which is for energy-efficient and power-aware multi-hop data transmission while maintaining queue stability.
• Surveillance monitoring applications: Mo et al. [19] design a secure learning framework for CCTVbased surveillance applications. In the system, multiple artificial neural network (ANN) frameworks exist; and each ANN is with its own parameters and setup. In this situation, there exists a tradeoff between computational complexity and performance. Therefore, the proposed algorithm dynamically and adaptively selects an ANN module depending on buffer size for recognition performance maximization subject to CCTV buffer 2 EAI Endorsed Transactions on Security and Safety 05 2020 -06 2020 | Volume 7 | Issue 23 | e5   [20] design a novel face identification machine learning framework for CCTV-based surveillance platforms. Instead of multiple ANN frameworks, this architecture has one learning system (based on Open-Face open-source software library) and controls the sampling rates of the CCTV camera. Eventually, the proposed adaptation algorithm dynamically and adaptively controls CCTV sampling rates for recognition performance maximization subject to CCTV buffer stability.
• Others: The application of Lyapunov optimization framework to dynamic reinforcement learning policy design is presented in [21]. In addition, the use of Lyapunov optimization for stock market pricing is well described in [22]. Finally, the stabilized adaptive control in smart and micro grid systems is discussed in [23] as yet another application.
Clearly, the applications above provide a great insight into the parallels of the existing literature and our application domain, which we explore through our proposed algorithm in the subsequent section.

Proposed Algorithm
This section describes the proposed super-resolution deep learning model adaptation for time-average quality maximization subject to queue stability, inspired by Lyapunov optimization framework [24].
First of all, the system queue dynamics are modeled as follows [24], where Q(t), µ(t), and λ(t), respectively, denote the queue-backlog at drone mobile platforms at time t, the number of enhanced video streams via selected superresolution algorithm (and then left from the queue) at time t, and the number of images arriving at the queue (i.i.d. random arrivals from the drone CCTV cameras) at time t. Then, we formulate the mathematical program for maximizing the time-average super-resolution 3 EAI Endorsed Transactions on Security and Safety 05 2020 -06 2020 | Volume 7 | Issue 23 | e5 performance maximization subject to queue stability, which is presented as: where P (α(τ)) is a peak signal-to-noise ratio (PSNR) value when current selected super-resolution algorithm at time τ is α(τ) and its system queue stability can be formulated as: Based on the Lyapunov optimization [24], which is for the time-average performance maximization subject to queue stability, this proposed program can be reformulated as following (where α * (t) is the selected optimal super-resolution algorithm for time-average PSNR maximization subject to queue stability).
where S is the set of all possible super-resolution algorithms; and V is the trade-off coefficient between the performance and queue stability, respectively. Therefore, our proposed closed-form equation (5) should be computed in each unit time while observing Q(t), and then our proposed algorithm guarantees time-average PSNR maximization subject to stability. Based on this characterization and property, our proposed algorithm is self-controllable because it can dynamically and adaptively select one superresolution algorithm automatically. Moreover, the proposed algorithm is reliable according to the fact that the self-adaptation is for maximizing its utility while maintaining stability.
The proposed algorithm can be presented in the form of a pseudo-code (Algorithm 1). As shown in Algorithm 1, the computation procedure is for solving the closed-form equation, i.e., (5). Thus, the complexity of this algorithm is polynomial-time.

Performance Evaluation
The performances of the super-resolution algorithm computation depending on the different numbers of hidden layers are presented in Table 1 and Fig. 2. Note that simulation settings are equivalent to the settings in VSDR [11]). As can be seen in the table and demonstrated by examples in the figures, the superresolution models with more hidden layers show better performance in terms of PSNR, i.e., video quality [26,27].

Algorithm 1 Proposed Super-Resolution Deep Learning Adaptation
Initialize: Observe Q(t) 6: for α(t) ∈ S do 8: if T ≥ T * then 10: T * ← T The simulation-based evaluation for the proposed super-resolution adaptation control algorithm is performed and then the results are presented in Fig. 3.
As illustrated in Fig. 3, if the models are static (i.e., Deep or Shallow in the plotting), the curves say that the two models are not efficient. The deep model cannot handle the overflow case, thus the queue diverges. On the other hand, the shallow model is fast, thus the queue is always empty. This is obviously good for stability, although the performance in terms of PSNR is the lowest. Thus, it might be better if the algorithm allows certain amounts of delays in order to enhance the quality of super-resolution. The proposed dynamic algorithm via Lyapunov optimization (i.e., Proposed (Lyapunov)) initially follows the Deep model because the queue is idle during the initial phases. If the queue is filled with images (up to a threshold), then it starts the control. Thus, the proposed algorithm starts to select models that can handle delays. Thus, it is clear that the proposed dynamic algorithm is better than the other two static algorithms.

Concluding Remarks and Future Work
This paper proposes a dynamic and stabilized superresolution deep learning adaptation algorithm for surveillance drone mobile platforms in security applications. In the reference surveillance model, drone mobile platforms detect abnormal objects and then the detected video streams will be stored into the queue. Then, each stream in the queue needs super-resolution computation in order to enhance the video stream quality improvement. If the queue is idle, deep superresolution algorithms can be used for better performance. On the other hand, shallow super-resolution algorithms should be used for fast computation while 4 EAI Endorsed Transactions on Security and Safety 05 2020 -06 2020 | Volume 7 | Issue 23 | e5 sacrificing certain amounts of performance if the queue is near overflow. Therefore, in order to hand the tradeoff between performance and delay, the proposed algorithm for time-average super-resolution performance maximization subject to queue stability is designed inspired by Lyapunov optimization theory. Simulationbased performance evaluation verifies that the proposed algorithm is better than static model selection algorithms. As a future research direction, a multi-drone scenario will be considered, since such a scenario will also be subject to multiple scheduling problems. 5 EAI Endorsed Transactions on Security and Safety 05 2020 -06 2020 | Volume 7 | Issue 23 | e5